Use DataHub’s data catalog capabilities to collect, organize, enrich, and search for metadata across multiple platforms

Introduction

According to Shirshanka Das, Founder of LinkedIn DataHub, Apache Gobblin, and Acryl Data, one of the simplest definitions for a data catalog can be found on the Oracle website: “Simply put, a data catalog is an organized inventory of data assets in the organization. It uses metadata to help organizations manage their data. It also helps data professionals collect, organize, access, and enrich metadata to support data discovery and governance.”

Another succinct description of a data catalog’s purpose comes from Alation: “a collection of metadata, combined with data management and search tools, that helps analysts and other data users to find the data that they need, serves as an inventory of available data, and provides information to evaluate the fitness of data for intended uses.”

Working with many organizations in the area of Analytics, one of the more common requests I receive regards choosing and implementing a data catalog. Organizations have datasources hosted in corporate data centers, on AWS, by SaaS providers, and with other Cloud Service Providers. Several of these organizations have recently gravitated to DataHub, the open-source metadata platform for the modern data stack, originally developed by LinkedIn.

In this post, we will explore the capabilities of DataHub to build a centralized data catalog on AWS for datasources hosted in multiple AWS accounts, SaaS providers, cloud service providers, and corporate data centers. I will demonstrate how to build a DataHub data catalog using out-of-the-box data source plugins for automated metadata ingestion.

Data Catalog Competitors

Data catalogs are not new; technologies such as data dictionaries have been around as far back as the 1980’s. Gartner publishes their Metadata Management (EMM) Solutions Reviews and Ratings and Metadata Management Magic Quadrant. These reports contain a comprehensive list of traditional commercial enterprise players, modern cloud-native SaaS vendors, and Cloud Service Provider (CSP) offerings. DBMS Tools also hosts a comprehensive list of 30 data catalogs. A sampling of current data catalogs includes:

Open Source Software

• LinkedIn’s DataHub
• Lyft’s Amundsen
• WeWork’s Marquez
• Apache Atlas

Commercial

• Acryl Data (based on LinkedIn’s DataHub)
• Atlan
• Stemma (based on Lyft’s Amundsen)
• Talend
• Alation
• Collibra
• data.world

Cloud Service Providers

• Google Cloud Data Catalog
• Microsoft Azure Data Catalog
• AWS Glue Data Catalog

Data Catalog Features

DataHub describes itself as “a modern data catalog built to enable end-to-end data discovery, data observability, and data governance.” Sorting through vendor’s marketing jargon and hype, standard features of leading data catalogs include:

• Metadata ingestion
• Data discovery
• Data governance
• Data observability
• Data lineage
• Data dictionary
• Data classification
• Usage/popularity statistics
• Sensitive data handling
• Data fitness (aka data quality or data profiling)
• Manage both technical and business metadata
• Business glossary
• Tagging
• Natively supported datasource integrations
• Advanced metadata search
• Fine-grain authentication and authorization
• UI- and API-based interaction

Datasources

When considering a data catalog solution, in my experience, the most common datasources that customers want to discover, inventory, and search include:

• Relational databases and other OLTP datasources such as PostgreSQL, MySQL, Microsoft SQL Server, and Oracle
• Cloud Data Warehouses and other OLAP datasources such as Amazon Redshift, Snowflake, and Google BigQuery
• NoSQL datasources such as MongoDB, MongoDB Atlas, and Azure Cosmos DB
• Persistent event-streaming platforms such as Apache Kafka (Amazon MSK and Confluent)
• Distributed storage datasets (e.g., Data Lakes) such as Amazon S3, Apache Hive, and AWS Glue Data Catalogs
• Business Intelligence (BI), dashboards, and data visualization sources such as Looker, Tableau, and Microsoft Power BI
• ETL sources, such as Apache Spark, Apache Airflow, Apache NiFi, and dbt

DataHub on AWS

DataHub’s convenient AWS setup guide covers options to deploy DataHub to AWS. For this post, I have hosted DataHub on Kubernetes, using Amazon Elastic Kubernetes Service (Amazon EKS). Alternately, you could choose Google Kubernetes Engine (GKE) on Google Cloud or Azure Kubernetes Service (AKS) on Microsoft Azure.

Conveniently, DataHub offers a Helm chart, making deployment to Kubernetes straightforward. Furthermore, Helm charts are easily integrated with popular CI/CD tools. For this post, I’ve used ArgoCD, the declarative GitOps continuous delivery tool for Kubernetes, to deploy the DataHub Helm charts to Amazon EKS.

According to the documentation, DataHub consists of four main components: GMS, MAE Consumer (optional), MCE Consumer (optional), and Frontend. Kubernetes deployment for each of the components is defined as sub-charts under the main DataHub Helm chart.

External Storage Layer Dependencies

Four external storage layer dependencies power the main DataHub components: Kafka, Local DB (MySQL, Postgres, or MariaDB), Search Index (Elasticsearch), and Graph Index (Neo4j or Elasticsearch). DataHub has provided a separate DataHub Prerequisites Helm chart for the dependencies. The dependencies must be deployed before deploying DataHub.

Alternately, you can substitute AWS managed services for the external storage layer dependencies, which is also detailed in the Deploying to AWS documentation. AWS managed service dependency substitutions include Amazon RDS for MySQL, Amazon OpenSearch (fka Amazon Elasticsearch), and Amazon Managed Streaming for Apache Kafka (Amazon MSK). According to DataHub, support for using AWS Neptune as the Graph Index is coming soon.

DataHub CLI and Plug-ins

DataHub comes with the datahub CLI, allowing you to perform many common operations on the command line. You can install and use the DataHub CLI within your development environment or integrate it with your CI/CD tooling.

DataHub uses a plugin architecture. Plugins allow you to install only the datasource dependencies you need. For example, if you want to ingest metadata from Amazon Athena, just install the Athena plugin: pip install 'acryl-datahub[athena]'. DataHub Source, Sink, and Transformer plugins can be displayed using the datahub check plugins CLI command.

Secure Metadata Ingestion

Often, datasources are not externally accessible for security reasons. Further, many datasources may not be accessible to individual users, especially in higher environments like UAT, Staging, and Production. They are only accessible to applications or CI/CD tooling. To overcome these limitations when extracting metadata with DataHub, I prefer to perform my DataHub-related development and testing locally but execute all DataHub ingestion securely on AWS.

In my local development environment, I use JetBrains PyCharm to author the Python and YAML-based DataHub configuration files and ingestion pipeline recipes, then commit those files to git and push them to a private GitHub repository. Finally, I use GitHub Actions to test DataHub files.

To run DataHub ingestion jobs and push the results to DataHub running in Kubernetes on Amazon EKS, I have built a custom Python-based Docker container. The container runs the DataHub CLI, required DataHub plugins, and any additional Python dependencies. The container’s pod has the appropriate AWS IAM permissions, using IAM Roles for Service Accounts (IRSA), to securely access datasources to ingest and the DataHub application.

Schedule and Monitor Pipelines

Scheduling and managing multiple metadata ingestion jobs on AWS is best handled with Apache Airflow with Amazon Managed Workflows for Apache Airflow (Amazon MWAA). Ingestion jobs run as Airflow DAG tasks, which call the EKS-based DataHub CLI container. With MWAA, datasource connections, credentials, and other sensitive configurations can be kept secure and not be exposed externally or in plain text.

When running the ingestion pipelines on AWS with DataHub, all communications between AWS-based datasources, ingestion jobs running in Airflow, and DataHub, should use secure private IP addressing and DNS resolution instead of transferring metadata over the Internet. Make sure to create all the necessary VPC peering connections, network route table configurations, and VPC endpoints to connect all relevant services.

SaaS services such as Snowflake or MongoDB Atlas, services provided by other Cloud Service Providers such as Google Cloud and Microsoft Azure, and datasources in corporate datasources require alternate networking and security strategies to access metadata securely.

Markup or Code?

According to the documentation, a DataHub recipe is a configuration file that tells ingestion scripts where to pull data from (source) and where to put it (sink). Recipes normally contain a source, sink, and transformers configuration section. Mark-up language-based job automation written in YAML, JSON, or Domain Specific Languages (DSLs) is often an alternative to writing code. DataHub recipes can be written in YAML. The example recipe shown below is used to ingest metadata from an Amazon RDS for PostgreSQL database, running on AWS.

YAML-based recipes can also use automatic environment variable expansion for convenience, automation, and security. It is considered best practice to secure sensitive configuration values, such as database credentials, in a secure location and reference them as environment variables. For example, note the server: ${DATAHUB_REST_ENDPOINT} entry in the sink section below. The DATAHUB_REST_ENDPOINT environment variable is set ahead of time and re-used for all ingestion jobs. Sensitive database connection information has also been variablized and stored separately.

	# Purpose: DataHub example recipe for PostgreSQL datasource
	# Author: Gary A. Stafford
	# Date: March 2022
	# see https://datahubproject.io/docs/metadata-ingestion/source_docs/postgres
	source:
	type: postgres
	config:
	# Coordinates
	host_port: ${DB_HOST_PORT}
	database: tickit
	# Credentials
	username: ${DB_USERNAME}
	password: ${DB_PASSWORD}
	# Options
	profiling:
	enabled: true
	# Environment
	env: DEV
	# see https://datahubproject.io/docs/metadata-ingestion/transformers/#adding-a-set-of-tags
	transformers:
	– type: "simple_add_dataset_tags"
	config:
	tag_urns:
	– "urn:li:tag:AWS"
	– "urn:li:tag:${ACCOUNT_ID}"
	– "urn:li:tag:us-east-1"
	– type: "pattern_add_dataset_terms"
	config:
	term_pattern:
	rules:
	".*users.*": ["urn:li:glossaryTerm:Classification.Sensitive"]
	– type: "simple_add_dataset_ownership"
	config:
	owner_urns:
	– "urn:li:corpuser:Database Administrators"
	ownership_type: "DATAOWNER"
	# see https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub for complete documentation
	sink:
	type: "datahub-rest"
	config:
	server: ${DATAHUB_REST_ENDPOINT}
	# see https://datahubproject.io/docs/metadata-ingestion/source_docs/reporting_telemetry/
	pipeline_name: "postgres-pipeline-tickit"
	reporting:
	– type: "datahub"
	config:
	datahub_api:
	server: ${DATAHUB_REST_ENDPOINT}

view raw postgresql_to_datahub_rest_tickit.yml hosted with ❤ by GitHub

Using Python

You can configure and run a pipeline entirely from within a custom Python script using DataHub’s Python API as an alternative to YAML. Below, we see two nearly identical ingestion recipes to the YAML above, written in Python. Writing ingestion pipeline logic programmatically gives you increased flexibility for automation, error checking, unit-testing, and notification. Below is a basic pipeline written in Python. The code is functional, but not very Pythonic, secure, scalable, or Production ready.

	# Purpose: Simple programmatic DataHub pipline example
	# Author: Gary A. Stafford
	# Date: March 2022
	# Reference: https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/programatic_pipeline.py
	from datahub.ingestion.run.pipeline import Pipeline
	# The pipeline configuration is similar to the recipe YAML files provided to the CLI tool.
	pipeline = Pipeline.create(
	{
	"run_id": "postgres-run",
	"source": {
	"type": "postgres",
	"config": {
	"host_port": "demo-instance.abcd1234.us-east-1.rds.amazonaws.com:5432",
	"database": "tickit",
	"username": "datahub",
	"password": "My5up3r53cr3tPa55w0rd",
	"env": "DEV",
	"profiling": {
	"enabled": "true"
	}
	}
	},
	"transformers": [
	{
	"type": "simple_add_dataset_tags",
	"config": {
	"tag_urns": [
	f"urn:li:tag:AWS",
	f"urn:li:tag:111222333444",
	f"urn:li:tag:us-east-1"
	]
	}
	},
	{
	"type": "pattern_add_dataset_terms",
	"config": {
	"term_pattern": {
	"rules": {
	".*users.*": [
	"urn:li:glossaryTerm:Classification.Sensitive"
	]
	}
	}
	}
	},
	{
	"type": "simple_add_dataset_ownership",
	"config": {
	"owner_urns": [
	f"urn:li:corpuser:Database Administrators"
	],
	"ownership_type": "DATAOWNER"
	}
	}
	],
	"sink": {
	"type": "datahub-rest",
	"config": {
	"server": "http://192.168.111.222:33333"
	}
	}
	}
	)
	# Run the pipeline and report the results.
	pipeline.run()
	pipeline.pretty_print_summary()

view raw postgresql_to_datahub_rest.py hosted with ❤ by GitHub

The second version of the same pipeline is more Production ready. The code is more Pythonic in nature and makes use of error checking, logging, and the AWS Systems Manager (SSM) Parameter Store. Like recipes written in YAML, environment variables can be used for convenience and security. In this example, commonly reused and sensitive connection configuration items have been extracted and placed in the SSM Parameter Store. Additional configuration is pulled from the environment, such as AWS Account ID and AWS Region. The script loads these values at runtime.

	# Purpose: Programmatic DataHub pipline example
	# Author: Gary A. Stafford
	# Date: March 2022
	import json
	import logging
	import boto3
	from botocore.exceptions import ClientError
	from datahub.ingestion.run.pipeline import Pipeline
	logging.basicConfig(
	format="[%(asctime)s] %(levelname)s – %(message)s", level=logging.INFO
	)
	def main():
	sts_client = boto3.client("sts")
	params = get_parameters()
	params["owner"] = "Database Administrators"
	params["environment"] = "DEV"
	params["database"] = "tickit"
	params["region"] = sts_client.meta.region_name
	params["account"] = sts_client.get_caller_identity()["Account"]
	logging.info(f"Params: {json.dumps(params, indent=4, sort_keys=True)}")
	ingestion_pipeline = create_pipeline(params)
	run_pipeline(ingestion_pipeline)
	def create_pipeline(params) -> Pipeline:
	"""Constructs a Pipeline for a PostgreSQL Source and a DataHub Sink
	:return: instance of datahub.ingestion.run.pipeline
	"""
	pipeline = Pipeline.create(
	{
	"run_id": "postgres-run",
	"source": {
	"type": "postgres",
	"config": {
	"host_port": params.get("/datahub_demo/postgres_host_port_tickit"),
	"database": params.get("database"),
	"username": params.get("/datahub_demo/postgres_username_tickit"),
	"password": params.get("/datahub_demo/postgres_password_tickit"),
	"profiling": {
	"enabled": "true"
	},
	"env": params.get("environment"),
	}
	},
	"transformers": [
	{
	"type": "simple_add_dataset_tags",
	"config": {
	"tag_urns": [
	f"urn:li:tag:{params.get('account')}",
	f"urn:li:tag:{params.get('region')}"
	]
	}
	},
	{
	"type": "pattern_add_dataset_terms",
	"config": {
	"term_pattern": {
	"rules": {
	".*users.*": [
	"urn:li:glossaryTerm:Classification.Sensitive"
	]
	}
	}
	}
	},
	{
	"type": "simple_add_dataset_ownership",
	"config": {
	"owner_urns": [
	f"urn:li:corpuser:{params.get('owner')}"
	],
	"ownership_type": "DATAOWNER"
	}
	}
	],
	"sink": {
	"type": "datahub-rest",
	"config": {
	"server": params.get("/datahub_demo/datahub_rest_endpoint_public")
	}
	}
	}
	)
	return pipeline
	def run_pipeline(pipeline):
	"""Runs the ingestion pipeline and prints summary of the results
	:param pipeline: instance of datahub.ingestion.run.pipeline
	:return:
	"""
	pipeline.run()
	pipeline.pretty_print_summary()
	def get_parameters() -> dict:
	"""
	Load parameter values from AWS Systems Manager (SSM) Parameter Store
	:return: dict of parameter k/v's
	"""
	ssm_client = boto3.client("ssm")
	params: dict = {}
	try:
	# make a single SSM API call for all parameters
	response = ssm_client.get_parameters_by_path(
	Path="/datahub_demo"
	)
	# create a dictionary of parameter k/v's
	for param in response.get("Parameters"):
	params[param["Name"]] = param["Value"]
	logging.debug(f"Params: {params}")
	except ClientError as e:
	logging.error(e)
	exit(1)
	return params
	if __name__ == '__main__':
	main()

view raw postgresql_to_datahub_rest_tickit.py hosted with ❤ by GitHub

Sinking to DataHub

When syncing metadata to DataHub, you have two choices, the GMS REST API or Kafka. According to DataHub, the advantage of the REST-based interface is that any errors can immediately be reported. On the other hand, the advantage of the Kafka-based interface is that it is asynchronous and can handle higher throughput. For this post, I am DataHub’s REST API.

Column-level Metadata

In addition to column names and data types, it is possible to extract column descriptions and key types from certain datasources. Column descriptions, tags, and glossary terms can also be input through the DataHub UI. Below, we see an example of an Amazon Redshift fact table, whose table and column descriptions were ingested as part of the metadata.

Business Glossary

DataHub can assign business glossary terms to entities. The DataHub Business Glossary plugin pulls business glossary metadata from a YAML-based configuration file.

	# see sample: https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/bootstrap_data/business_glossary.yml
	version: 1
	source: DataHub
	owners:
	users:
	– datahub
	url: "https://github.com/datahub-project/datahub/"
	nodes:
	– name: Classification
	description: A set of terms related to Data Classification
	terms:
	– name: Sensitive
	description: Sensitive Data
	custom_properties:
	is_confidential: false
	– name: Confidential
	description: Confidential Data
	custom_properties:
	is_confidential: true
	– name: HighlyConfidential
	description: Highly Confidential Data
	custom_properties:
	is_confidential: true
	– name: PersonalInformation
	description: All terms related to personal information
	owners:
	users:
	– datahub
	terms:
	– name: ID
	description: An individual's unqiue identifier
	inherits:
	– Classification.Sensitive
	– name: Name
	description: An individual's Name
	inherits:
	– Classification.Sensitive
	– name: SSN
	description: An individual's SSN
	inherits:
	– Classification.Confidential
	– name: DriverLicense
	description: An individual's Driver License ID
	inherits:
	– Classification.Confidential
	– name: Email
	description: An individual's email address
	inherits:
	– Classification.Confidential
	– name: Address
	description: A physical address
	– name: Gender
	description: The gender identity of the individual
	inherits:
	– Classification.Sensitive

view raw business_glossary.yml hosted with ❤ by GitHub

Business glossary terms can be reviewed in the Glossary Terms tab of the DataHub’s UI. Below, we see the three terms associated with the Classification glossary node: Confidential, HighlyConfidential, and Sensitive.

We can search for entities inventoried in DataHub using their assigned business glossary terms.

Finally, we see an example of an AWS Athena data catalog table with business glossary terms applied to columns within the table’s schema.

SQL-based Profiler

DataHub also can extract statistics about entities in DataHub using the SQL-based Profiler. According to the DataHub documentation, the Profiler can extract the following:

• Row and column counts for each table
• Column null counts and proportions
• Column distinct counts and proportions
• Column min, max, mean, median, standard deviation, quantile values
• Column histograms or frequencies of unique values

In addition, we can also track the historical stats for each profiled entity each time metadata is ingested.

Data Lineage

DataHub’s data lineage features allow us to view upstream and downstream relationships between different types of entities. DataHub can trace lineage across multiple platforms, datasets, pipelines, charts, and dashboards.

Below, we see a simple example of dataset entity-to-entity lineage in Amazon Redshift and then Apache Spark on Amazon EMR. The fact table has a downstream relationship to four database views. The views are based on SQL queries that include the upstream table as a datasource.

DataHub Analytics

DataHub provides basic metadata quality and usage analytics in the DataHub UI: user activity, counts of datasource types, business glossary terms, environments, and actions.

Conclusion

In this post, we explored the features of a data catalog and learned about some of the leading commercial and open-source data catalogs. Next, we learned how DataHub could collect, organize, enrich, and search metadata across multiple datasources. Lastly, we discovered how easy it is to catalog metadata from datasources spread across multiple CSP, SaaS providers, and corporate data centers, and centralize those results in DataHub.

In addition to the basic features reviewed in this post, DataHub offers a growing number of additional capabilities, including GraphQL and Timeline APIs, robust authentication and authorization, application monitoring observability, and Great Expectations integration. All these qualities make DataHub an excellent choice for a data catalog.

---

This blog represents my own viewpoints and not of my employer, Amazon Web Services (AWS). All product names, logos, and brands are the property of their respective owners.