Posts Tagged DataOps

Implementing an Effective and Repeatable Approach for Capturing Data Analytics System Requirements

Implement an effective, consistent, and repeatable strategy for documenting data analytics workflows and capturing system requirements

Audio version of this blog post


Data analytics applications involve more than just analyzing data, particularly on advanced analytics projects. Much of the required work takes place upfront, in collecting, integrating, and preparing data and then developing, testing, and revising analytical models to ensure that they produce accurate results. In addition to data scientists and other data analysts, analytics teams often include data engineers, who create data pipelines and help prepare data sets for analysis.” — TechTarget

Successful consultants, project managers, and product owners take a well-defined and systematic approach to achieve desired outcomes. They use frameworks, templates, and proven patterns to accomplish their goals — consistently successful engagements, project outcomes, and product and service launches.

Given the complexity of modern data analytics workflows, the goal of this platform-agnostic discovery process is to lead an organization, department, team, or customer through an effective, efficient, consistent, and repeatable approach for capturing data analytics workflows and systems requirements. This process will result in a clear understanding of existing analytics workflows, current and desired future business and technical outcomes, and existing and anticipated future business and technical constraints.

If applicable, the discovery process serves as a foundation for architecting new data analytics workflows. Starting with a set of business and technical constraints, the approach facilitates the development of new data analytics workflows to achieve desired business and technical outcomes. Outcomes and constraints determine the requirements.

Analytics Workflow Stages

There are many patterns organizations use to delineate the stages of their analytics workflows. However, the granularity of the stages is not critical to this process as long as all major functionality is considered. This process utilizes six stages of a typical analytics workflow, from left to right: Generate, Collect, Prepare, Store, Analyze, and Deliver.

Visualization of the discovery process

The discovery process starts by working backward and from the outside in. First, the process identifies current and desired future outcomes (right side of the diagram above). Then, it identifies existing and anticipated future constraints (left-side of the diagram). Next, it identifies the inputs and the outputs for the existing and re-architected workflows. Finally, the process identifies the current and re-architected analytics workflows — collect, prepare, store, and analyze — the steps required to get from data sources to deliverables.

Collect, prepare, store, and analyze — the steps required to get from data sources to deliverables.

Specifically, the process identifies and documents the following:

  1. Current business and technical outcomes
  2. Desired future business and technical outcomes
  3. Existing business and technical constraints
  4. Anticipated future business and technical constraints
  5. Inputs required to achieve desired outcomes
  6. Outputs needed to achieve desired outcomes
  7. Data producers and consumers
  8. Existing analytics tools, procedures, and people
  9. Measures of success for analytics workflows


Capture desired business and technical outcomes (goals and objectives), for example:

  • Re-architect analytics workflows to modernize, reduce costs, increase speed, reduce complexity, and add capabilities
  • Move analytics workflows to the Cloud (review 6 R’s strategies)
  • Migrate from another Cloud or SaaS provider
  • Shift away from proprietary platforms to an open-source analytics stack
  • Migrate away from custom in-house analytics tools
  • Add DataOps — CI/CD automation to existing workflows
  • Integrated hybrid on-prem, multi-cloud, and SaaS-based analytics architectures
  • Develop new greenfield analytics workflows, products, and services
  • Standardization of analytics workflows
  • Develop machine learning models from the data
  • Provide key stakeholders with real-time business KPIs dashboards


Identify the existing and potential future business and technical constraints that impact analytics workflows, for example:

  • Budget
  • Timeline
  • People, including lack of specific skills (need for training)
  • Internal and external regulatory, data governance, and data lineage requirements
  • SLAs and KPIs (measures of workflow effectiveness)
  • Current vendor, partner, Cloud-provider, and SaaS relationships
  • Licensing and contractual obligations
  • Must-keep aspects of current analytics workflows


Capture sources of data that are required to produce the outputs, for example:

  • Batch sources such as databases and data feeds
  • Streaming sources such as IoT device telemetry, operational metrics, logs, clickstreams, and gaming stats
  • Database engines, including relational, key-value, document, in-memory, graph, time series, wide column, and ledger
  • Data warehouses
  • Data feeds, such as flat files from legacy or third-party systems
  • API endpoints
  • Public and licensed datasets

Use the 5 V’s of data, including Volume, Velocity, Variety, Veracity, and Value, to dive deep into each input. Capture existing and anticipated data access and usage patterns.


Identify the deliverables required to meet the desired outcomes. For example, prepare and make data available for:

  • Further analysis
  • Business Intelligence (BI), visualizations, and dashboards
  • Machine Learning (ML) and Artificial Intelligence (AI)
  • Data exports, such as Excel or CSV-format flat files
  • Hosted datasets for external or internal consumption
  • Expose data via APIs

Analytics Workflows

Capture existing analytics workflows using the four stages of Collect, Prepare, Store, and Analyze as a way to organize this part of the process:

  • High- and low-level architecture, process flow, patterns, and external dependencies
  • Analytics tools, including hardware, commercial, custom, and OSS software, libraries, modules, source code
  • DataOps, CI/CD, DevOps, and Infrastructure-as-Code (IaC) automation
  • People, including Managed Service Providers (MSPs), roles and required skills
  • Partners, including consultants, vendors, SaaS-providers
  • Overall effectiveness and customer satisfaction with existing analytics workflows
  • Deficiencies with current workflows

Use checklists for each stage to ensure all potential features, functions, and capabilities of typical analytics workflows are considered and captured. For example, review a checklist of all possible ways data is typically collected to ensure nothing is missed.

Measures of Success

Identify how success is measured with existing analytics workflows as well as with new workflows, and by whom, for example:

  • Key Performance Indicators (KPIs)
  • Service Level Agreements (SLAs)
  • Customer Satisfaction Score (CSAT)
  • Net Promoter Score (NPS)
  • How measurements are determined, calculated, and weighted

Data Producers and Consumers

Capture the data producers and data consumers:

  • Data producers
  • Data consumers


The output of the data analytics discovery process is a clear and concise document that captures all elements described herein. Additionally, the document contains any customer-supplied artifacts, such as architectural and process flow diagrams. The document is thoroughly reviewed for accuracy and completeness by the customer. This document serves as a record of current data analytics workflows and a basis for architecting new workflows.

This blog represents my own viewpoints and not of my employer, Amazon Web Services (AWS). All product names, logos, and brands are the property of their respective owners.

, , ,

Leave a comment

Amazon Managed Workflows for Apache Airflow — Configuration: Understanding Amazon MWAA’s Configuration Options


For anyone new to Amazon Managed Workflows for Apache Airflow (Amazon MWAA), especially those used to managing their own Apache Airflow platform, Amazon MWAA’s configuration might appear to be a bit of a black box at first. This brief post will explore Amazon MWAA’s configuration — how to inspect it and how to modify it. We will use Airflow DAGs to review an MWAA environment’s airflow.cfg file, environment variables, and Python packages.

Amazon MWAA

Apache Airflow is a popular open-source platform designed to schedule and monitor workflows. According to Wikipedia, Airflow was created at Airbnb in 2014 to manage the company’s increasingly complex workflows. From the beginning, the project was made open source, becoming an Apache Incubator project in 2016 and a top-level Apache Software Foundation project in 2019.

With the announcement of Amazon MWAA in November 2020, AWS customers can now focus on developing workflow automation while leaving the management of Airflow to AWS. Amazon MWAA can be used as an alternative to AWS Step Functions for workflow automation on AWS.

The Amazon MWAA service is available using the AWS Management Console, as well as the Amazon MWAA API using the latest versions of the AWS SDK and AWS CLI. For more information on Amazon MWAA, read my last post, Running Spark Jobs on Amazon EMR with Apache Airflow.

Image for post
Apache Airflow UI

Source Code

The DAGs referenced in this post are available on GitHub. Using this git clone command, download a copy of this post’s GitHub repository to your local environment.

git clone --branch main --single-branch --depth 1 --no-tags \

Accessing Configuration

Environment Variables

Environment variables are an essential part of an MWAA environment’s configuration. There are various ways to examine the environment variables. You could use Airflow’s BashOperator to simply call the command, env, or the PythonOperator to call a Python iterator function, as shown below. A sample DAG, dags/, is included in the project.

def print_env_vars():
keys = str(os.environ.keys().replace("', '", "'|'").split("|")
for key in keys:
get_env_vars_operator = PythonOperator(
view raw hosted with ❤ by GitHub

The DAG’s PythonOperator will iterate over the MWAA environment’s environment variables and output them to the task’s log. Below is a snippet of an example task’s log.

[2020-12-25 23:59:07,170] {{}} INFO – Job 272: Subtask get_env_vars_task
[2020-12-25 23:59:08,423] {{}} INFO – 'AIRFLOW_CONN_AWS_DEFAULT': 'aws://'
[2020-12-25 23:59:08,516] {{}} INFO – 'AIRFLOW_CONSOLE_LOGS_ENABLED': 'false'
[2020-12-25 23:59:08,689] {{}} INFO – 'AIRFLOW_CONSOLE_LOG_LEVEL': 'WARNING'
[2020-12-25 23:59:08,777] {{}} INFO – 'AIRFLOW_CTX_DAG_EMAIL': ''
[2020-12-25 23:59:08,877] {{}} INFO – 'AIRFLOW_CTX_DAG_ID': 'get_env_vars'
[2020-12-25 23:59:08,970] {{}} INFO – 'AIRFLOW_CTX_DAG_OWNER': 'airflow'
[2020-12-25 23:59:09,269] {{}} INFO – 'AIRFLOW_CTX_TASK_ID': 'get_env_vars_task'
[2020-12-25 23:59:09,357] {{}} INFO – 'AIRFLOW_DAG_PROCESSING_LOGS_ENABLED': 'false'
[2020-12-25 23:59:09,552] {{}} INFO – 'AIRFLOW_DAG_PROCESSING_LOG_LEVEL': 'WARNING'
[2020-12-25 23:59:09,647] {{}} INFO – 'AIRFLOW_ENV_NAME': 'MyAirflowEnvironment'
[2020-12-25 23:59:09,729] {{}} INFO – 'AIRFLOW_HOME': '/usr/local/airflow'
[2020-12-25 23:59:09,827] {{}} INFO – 'AIRFLOW_SCHEDULER_LOGS_ENABLED': 'false'
[2020-12-25 23:59:12,915] {{}} INFO – 'AIRFLOW__CORE__DAG_CONCURRENCY': '10000'
[2020-12-25 23:59:12,986] {{}} INFO – 'AIRFLOW__CORE__EXECUTOR': 'CeleryExecutor'
[2020-12-25 23:59:13,136] {{}} INFO – 'AIRFLOW__CORE__LOAD_EXAMPLES': 'False'
[2020-12-25 23:59:13,217] {{}} INFO – 'AIRFLOW__CORE__PARALLELISM': '10000'
[2020-12-25 23:59:14,531] {{}} INFO – 'AWS_DEFAULT_REGION': 'us-east-1'
[2020-12-25 23:59:14,565] {{}} INFO – 'AWS_EXECUTION_ENV': 'AWS_ECS_FARGATE'
[2020-12-25 23:59:14,616] {{}} INFO – 'AWS_REGION': 'us-east-1'
[2020-12-25 23:59:14,647] {{}} INFO – 'CELERY_LOG_FILE': ''
[2020-12-25 23:59:14,679] {{}} INFO – 'CELERY_LOG_LEVEL': '20'
[2020-12-25 23:59:14,711] {{}} INFO – 'CELERY_LOG_REDIRECT': '1'
[2020-12-25 23:59:14,747] {{}} INFO – 'CELERY_LOG_REDIRECT_LEVEL': 'WARNING'

Airflow Configuration File

According to Airflow, the airflow.cfg file contains Airflow’s configuration. You can edit it to change any of the settings. The first time you run Apache Airflow, it creates an airflow.cfg configuration file in your AIRFLOW_HOME directory and attaches the configurations to your environment as environment variables.

Amazon MWAA doesn’t expose the airflow.cfg in the Apache Airflow UI of an environment. Although you can’t access it directly, you can view the airflow.cfg file. The configuration file is located in your AIRFLOW_HOME directory, /usr/local/airflow (~/airflow by default).

There are multiple ways to examine your MWAA environment’s airflow.cfg file. You could use Airflow’s PythonOperator to call a Python function that reads the contents of the file, as shown below. The function uses the AIRFLOW_HOME environment variable to locate and read the airflow.cfg. A sample DAG, dags/, is included in the project.

def print_airflow_cfg():
with open(f"{os.getenv('AIRFLOW_HOME')}/airflow.cfg", 'r') as airflow_cfg:
file_contents =
get_airflow_cfg_operator = PythonOperator(

The DAG’s task will read the MWAA environment’s airflow.cfg file and output it to the task’s log. Below is a snippet of an example task’s log.

[2020-12-26 00:02:57,163] {{}} INFO – Job 274: Subtask get_airflow_cfg_task
[2020-12-26 00:02:57,583] {{}} INFO –
# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository
# This path must be absolute
dags_folder = /usr/local/airflow/dags
# The folder where airflow should store its log files
# This path must be absolute
base_log_folder = /usr/local/airflow/logs
# Airflow can store logs remotely in AWS S3, Google Cloud Storage or Elastic Search.
# Set this to True if you want to enable remote logging.
remote_logging = True
# Users must supply an Airflow connection id that provides access to the storage
# location.
remote_log_conn_id = aws_default
remote_base_log_folder = cloudwatch://arn:aws:logs:::log-group:airflow-logs:*
encrypt_s3_logs = False
# Logging level
logging_level = INFO
# Logging level for Flask-appbuilder UI
fab_logging_level = WARN
# Logging class
# Specify the class that will specify the logging configuration
# This class has to be on the python classpath
# Example: logging_config_class = my.path.default_local_settings.LOGGING_CONFIG
logging_config_class = log_config.LOGGING_CONFIG
# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
parallelism = 32
# The number of task instances allowed to run concurrently by the scheduler
dag_concurrency = 16
redirect_url =
session_duration_minutes = 720
# The base url of your website as airflow cannot guess what domain or
# cname you are using. This is used in automated emails that
# airflow sends to point links to the right web server
base_url = http://localhost:8080
# Default timezone to display all dates in the RBAC UI, can be UTC, system, or
# any IANA timezone string (e.g. Europe/Amsterdam). If left empty the
# default value of core/default_timezone will be used
# Example: default_ui_timezone = America/New_York
default_ui_timezone = UTC
# The ip specified when starting the web server
web_server_host =
# The port on which to run the web server
web_server_port = 8080

Customizing Airflow Configurations

While AWS doesn’t expose the airflow.cfg in the Apache Airflow UI of your environment, you can change the default Apache Airflow configuration options directly within the Amazon MWAA console and continue using all other settings in airflow.cfg. The configuration options changed in the Amazon MWAA console are translated into environment variables.

To customize the Apache Airflow configuration, change the default options directly on the Amazon MWAA console. Select Edit, add or modify configuration options and values in the Airflow configuration options menu, then select Save. For example, we can change Airflow’s default timezone (core.default_ui_timezone) to America/New_York.

Image for post
Amazon MWAA’s Airflow configuration options

Once the MWAA environment is updated, which may take several minutes, view your changes by re-running the DAG,dags/ Note the new configuration item on both lines 2 and 6 of the log snippet shown below. The configuration item appears on its own (AIRFLOW__CORE_DEFAULT__UI_TIMEZONE), as well as part of the AIRFLOW_CONFIG_SECRETS dictionary environment variable.

[2020-12-26 05:00:57,756] {{}} INFO – Job 293: Subtask get_env_vars_task
[2020-12-26 05:00:58,158] {{}} INFO – 'AIRFLOW_CONFIG_SECRETS': '{"AIRFLOW__CORE__DEFAULT_UI_TIMEZONE":"America/New_York"}'
[2020-12-26 05:00:58,190] {{}} INFO – 'AIRFLOW_CONN_AWS_DEFAULT': 'aws://'
[2020-12-26 05:01:00,537] {{}} INFO – 'AIRFLOW__CORE__DAG_CONCURRENCY': '10000'
[2020-12-26 05:01:00,578] {{}} INFO – 'AIRFLOW__CORE__DEFAULT_UI_TIMEZONE': 'America/New_York'
[2020-12-26 05:01:00,630] {{}} INFO – 'AIRFLOW__CORE__EXECUTOR': 'CeleryExecutor'

Using the MWAA API

We can also make configuration changes using the MWAA API. For example, to change the default Airflow UI timezone, call the MWAA API’s update-environment command using the AWS CLI. Include the --airflow-configuration-option parameter, passing the core.default_ui_timezone key/value pair as a JSON blob.

aws mwaa update-environment \
–name <your_environment_name> \
–airflow-configuration-options """{
\"core.default_ui_timezone\": \"America/Los_Angeles\"

To review an environment’s configuration, use the get-environment command in combination with jq.

aws mwaa get-environment \
–name <your_environment_name> | \
jq -r '.Environment.AirflowConfigurationOptions'

Below, we see an example of the output.

"core.default_ui_timezone": "America/Los_Angeles"

Python Packages

Airflow is written in Python, and workflows are created via Python scripts. Python packages are a crucial part of an MWAA environment’s configuration. According to the documentation, an ‘extra package’, is a Python subpackage that is not included in the Apache Airflow base, installed on your MWAA environment. As part of setting up an MWAA environment, you can specify the location of the requirements.txt file in the Airflow S3 bucket. Extra packages are installed using the requirements.txt file.

Image for post
Amazon MWAA environment’s configuration

There are several ways to check your MWAA environment’s installed Python packages and versions. You could use Airflow’s BashOperator to call the command, python3 -m pip list. A sample DAG, dags/, is included in the project.

list_python_packages_operator = BashOperator(
bash_command='python3 -m pip list'
view raw hosted with ❤ by GitHub

The DAG’s task will output a list of all Python packages and package versions to the task’s log. Below is a snippet of an example task’s log.

[2020-12-26 21:53:06,310] {{}} INFO – Temporary script location: /tmp/airflowtmp2whgp_p8/list_python_packagesxo8slhc6
[2020-12-26 21:53:06,350] {{}} INFO – Running command: python3 -m pip list
[2020-12-26 21:53:06,395] {{}} INFO – Output:
[2020-12-26 21:53:06,750] {{}} INFO – Package Version
[2020-12-26 21:53:06,786] {{}} INFO – ———————- ———
[2020-12-26 21:53:06,815] {{}} INFO – alembic 1.4.2
[2020-12-26 21:53:06,856] {{}} INFO – amqp 2.6.1
[2020-12-26 21:53:06,898] {{}} INFO – apache-airflow 1.10.12
[2020-12-26 21:53:06,929] {{}} INFO – apispec 1.3.3
[2020-12-26 21:53:06,960] {{}} INFO – argcomplete 1.12.0
[2020-12-26 21:53:07,002] {{}} INFO – attrs 19.3.0
[2020-12-26 21:53:07,036] {{}} INFO – Babel 2.8.0
[2020-12-26 21:53:07,071] {{}} INFO – billiard
[2020-12-26 21:53:07,960] {{}} INFO – boto3 1.16.10
[2020-12-26 21:53:07,993] {{}} INFO – botocore 1.19.10
[2020-12-26 21:53:08,028] {{}} INFO – cached-property 1.5.1
[2020-12-26 21:53:08,061] {{}} INFO – cattrs 1.0.0
[2020-12-26 21:53:08,096] {{}} INFO – celery 4.4.7
[2020-12-26 21:53:08,130] {{}} INFO – certifi 2020.6.20
[2020-12-26 21:53:12,260] {{}} INFO – pandas 1.1.0
[2020-12-26 21:53:12,289] {{}} INFO – pendulum 1.4.4
[2020-12-26 21:53:12,490] {{}} INFO – pip 9.0.3
[2020-12-26 21:53:12,522] {{}} INFO – prison 0.1.3
[2020-12-26 21:53:12,550] {{}} INFO – prometheus-client 0.8.0
[2020-12-26 21:53:12,580] {{}} INFO – psutil 5.7.2
[2020-12-26 21:53:12,613] {{}} INFO – pycparser 2.20
[2020-12-26 21:53:12,641] {{}} INFO – pycurl
[2020-12-26 21:53:12,676] {{}} INFO – Pygments 2.6.1
[2020-12-26 21:53:12,710] {{}} INFO – PyGreSQL 5.2.1
[2020-12-26 21:53:12,746] {{}} INFO – PyJWT 1.7.1


Understanding your Amazon MWAA environment’s airflow.cfg file, environment variables, and Python packages are all important for proper Airflow platform management. This brief post learned more about Amazon MWAA’s configuration — how to inspect it using DAGs and how to modify it through the Amazon MWAA console.

, , , ,

Leave a comment