Implement an effective, consistent, and repeatable strategy for documenting data analytics workflows and capturing system requirements

“Data analytics applications involve more than just analyzing data, particularly on advanced analytics projects. Much of the required work takes place upfront, in collecting, integrating, and preparing data and then developing, testing, and revising analytical models to ensure that they produce accurate results. In addition to data scientists and other data analysts, analytics teams often include data engineers, who create data pipelines and help prepare data sets for analysis.” — TechTarget

Introduction

Successful consultants, project managers, and product owners use well-proven and systematic approaches to achieve desired outcomes, including successful customer engagements, project results, and product and service launches. Modern data stacks and analytics workflows are increasingly complex. This technology-agnostic discovery process aims to help an organization efficiently and repeatably capture a concise record of existing analytics workflows, business and technical goals and constraints, and measures of success. If applicable, the discovery process is also used to compile and clarify requirements for new data analytics workflows.

Analytics Workflow Stages

There are many patterns organizations use to delineate the stages of their analytics workflows. This process utilizes six stages of a typical analytics workflow:

1. Generate: All the ways data is generated and the systems of record where it is stored or originates from, also referred to as data ingress
2. Collect: All the ways data is collected or ingested
3. Prepare: All the ways data is transformed, including ETL, ELT, reverse ETL, and ML
4. Store: All the ways data is stored, organized, and secured for analytics purposes
5. Analyze: All the ways data is analyzed
6. Deliver: All the ways data is delivered and how it is consumed, also referred to as data egress or data products

The precise nomenclature is not critical to this process as long as all major functionality is considered.

The Process

The discovery process starts by working backward. It first identifies existing goals and desired outcomes. It then identifies existing and anticipated future constraints. Next, it breaks down the current analytics workflows, examining the four stages of collect, prepare, store, and analyze, the steps required to get from data sources to deliverables. Finally, it captures the inputs and the outputs for the workflows and the data producers and consumers.

Collect, prepare, store, and analyze — the steps required to get from data sources to deliverables.

Specifically, the process identifies and documents the following:

1. Business and technical goals and desired outcomes
2. Business and technical constraints also referred to as limitations or restrictions
3. Analytics workflows: tools, techniques, procedures, and organizational structure
4. Outputs also referred to as deliverables, required to achieve desired outcomes
5. Inputs, also referred to as data sources, required to achieve desired outcomes
6. Data producers and consumers
7. Measures of success
8. Recommended next steps

Outcomes

Capture business and technical goals and desired outcomes driving the necessity to rearchitect current analytics processes. For example:

1. Re-architect analytics processes to modernize, reduce complexity, or add new capabilities
2. Reduce or control costs
3. Increase performance, scalability, speed
4. Migrate on-premises workloads, workflows, processes to the Cloud
5. Migrate from one cloud provider or SaaS provider to another
6. Move away from proprietary software products to open source software (OSS) or commercial open source software (COSS)
7. Migrate away from custom-built software to commercial off-the-shelf (COTS), OSS, or COSS solutions
8. Integrate DevOps, GitOps, DataOps, or MLOps practices
9. Integrate on-premises, multi-cloud, and SaaS-based hybrid architectures
10. Develop new analytics product or service offerings
11. Standardize analytics processes
12. Leverage the data for AI and ML purposes
13. Provide stakeholders with a real-time business KPIs dashboard
14. Construct a data lake, data warehouse, data lakehouse, or data mesh

If migration is involved, review the 6 R’s of Cloud Migration: Rehost, Replatform, Repurchase, Refactor, Retain, or Retire.

Constraints

Identify the existing and potential future business and technical constraints that impact analytics workflows. For example:

1. Budgets
2. Cost attribution
3. Timelines
4. Access to skilled resources
5. Internal and external regulatory requirements, such as HIPAA, SOC2, FedRAMP, GDPR, PCI DSS, CCPA, and FISMA
6. Business Continuity and Disaster Recovery (BCDR) requirements
7. Architecture Review Board (ARB), Center of Excellence (CoE), Change Advisory Board (CAB), and Release Management standards and guidelines
8. Data residency and data sovereignty requirements
9. Security policies
10. Service Level Agreements (SLAs); see ‘Measures of Success’ section
11. Existing vendor, partner, cloud-provider, and SaaS relationships
12. Existing licensing and contractual obligations
13. Deprecated code dependencies and other technical debt
14. Must-keep aspects of existing processes
15. Build versus buy propensity
16. Proprietary versus open source software propensity
17. Insourcing versus outsourcing propensity
18. Managed, hosted, SaaS versus self-managed software propensity

Analytics Workflows

Capture analytics workflows using the four stages of collect, prepare, store, and analyze, as a way to organize the discussion:

1. High- and low-level architecture, process flow diagrams, sequence diagrams
2. Recent architectural assessments such as reviews based on the AWS Data Analytics Lens, AWS Well-Architected Framework, Microsoft Azure Well-Architected Framework, or Google Cloud Architecture Framework
3. Analytics tools, including hardware and commercial, custom, and open-source software
4. Security policies, processes, standards, and technologies
5. Observability, logging, monitoring, alerting, and notification
6. Teams, including roles, responsibilities, and skillsets
7. Partners, including consultants, vendors, SaaS providers, and Managed Service Providers (MSP)
8. SDLC environments, such as Local, Sandbox, Development, Testing, Staging, Production, and Disaster Recovery (DR)
9. Business Continuity Planning (BCP) policies, processes, standards, and technologies
10. Primary analytics programming languages
11. External system dependencies
12. DataOps, MLOps, DevOps, CI/CD, SCM/VCS, and Infrastructure-as-Code (IaC) automation policies, processes, standards, and technologies
13. Data governance and data lineage policies, processes, standards, and technologies
14. Data quality (or data assurance) policies, processes, standards, technologies, and testing methodologies
15. Data anomaly detection policies, processes, standards, and technologies
16. Intellectual property (IP), the ‘secret sauce’ that differentiates the organization’s processes and provides a competitive advantage, such as ML models, proprietary algorithms, datasets, highly specialized knowledge, and patents
17. Overall effectiveness and customer satisfaction with existing analytics platform (document sources of customer feedback)
18. Known deficiencies with current analytics processes

Outputs

Identify the deliverables required to meet the desired outcomes. For example, prepare and provide data for:

1. Data analytics purposes
2. Business Intelligence (BI), visualizations, and dashboards
3. Machine Learning (ML) and Artificial Intelligence (AI)
4. Data exports and data feeds, such as Excel or CSV-format files
5. Hosted datasets for external or internal consumption
6. Data APIs for external or internal consumption
7. Documentation, Data API guides, data dictionaries, example code such as Notebooks
8. SaaS-based product offering

Inputs

Capture sources of data that are required to produce the outputs. For example:

1. Batch sources such as flat files from legacy systems, third-party providers, and enterprise platforms
2. Streaming sources such as message queues, change data capture (CDC), IoT device telemetry, operational metrics, real time logs, clickstream data, connected devices, mobile, and gaming feeds
3. Databases, including relational, NoSQL, key-value, document, in-memory, graph, time series, and wide column (OLTP data stores)
4. Data warehouses (OLAP data stores)
5. Data lakes
6. API endpoints
7. Internal, public, and licensed datasets

Use the 5 V’s of big data to dive deep into each data source: Volume, Velocity, Variety, Veracity (or Validity), and Value.

Data Producers and Consumers

Capture all producers and consumers of data:

1. Data producers
2. Data consumers
3. Data access patterns
4. Data usage patterns
5. Consumer and producer requirements and constraints

Measures of Success

Identify how success is measured for the analytics workflows and by whom. For example:

1. Key Performance Indicators (KPIs)
2. Service Level Agreements (SLAs)
3. Customer Satisfaction Score (CSAT)
4. Net Promoter Score (NPS)
5. SaaS growth metrics: churn, activation rate, MRR, ARR, CAC, CLV, expansion revenue (source: appcues.com)
6. Data quality guarantees
7. How are measurements determined, calculated, and weighted?
8. What are the business and technical actions resulting from missed measures of success?

Results

The immediate artifact of the data analytics discovery process is a clear and concise document that captures all feedback and inputs. In addition, the document contains all customer-supplied artifacts, such as architectural and process flow diagrams. The document should be thoroughly reviewed for accuracy and completeness by the process participants. This artifact serves as a record of current data analytics workflows and a basis for making workflow improvement recommendations or architecting new workflows.

---

This blog represents my own viewpoints and not of my employer, Amazon Web Services (AWS). All product names, logos, and brands are the property of their respective owners.