Knowledge Sharing - Data Management

Overview

This project intends to collect, analyze and synthetize referential material about data management, i.e., how to build and operate so-called Modern Data Stack (MDS) and Modern Metadata Platform (MMP).

Even though the members of the GitHub organization may be employed by some companies, they speak on their personal behalf and do not represent these companies.

Other repositories of Data Engineering helpers

• Data Engineering Helpers - Knowledge Sharing - Data products
• Data Engineering Helpers - Knowledge Sharing - Data contracts
• Data Engineering Helpers - Knowledge Sharing - Data quality
• Data Engineering Helpers - Knowledge Sharing - Architecture principles
• Data Engineering Helpers - Knowledge Sharing - Data life cycle
• Data Engineering Helpers - Knowledge Sharing - Data lakehouse
• Data Engineering Helpers - Knowledge Sharing - Metadata
• Data Engineering Helpers - Knowledge Sharing - Data pipeline deployment
• Data Engineering Helpers - Knowledge Sharing - Semantic layer

Articles

Envisioning LakeDB: The Next Evolution of the Lakehouse Architecture

• Title: Envisioning LakeDB: The Next Evolution of the Lakehouse Architecture
• Date: Jan. 2025
• Author: Ananth Packkildurai (Ananth Packkildurai on LinkedIn, Ananth Packkildurai on Substack)
• Link to the article: https://open.substack.com/pub/dataengineeringweekly/p/envisioning-lakedb-the-next-evolution

The data product marketplace

• Title: The Data Product Marketplace: A Single Interface for Business
• Date: Ot. 2024
• Author: Arielle Rolland (Arielle Rolland on LinkedIn, Arielle Rolland on Substack)
• Link to the article on Substack: https://moderndata101.substack.com/p/the-data-product-marketplace-a-single

Data Platform Philosophy

• Title: Issue #22 – Deciding on your Data Platform Philosophy
• Date: Sep. 2024
• Authors: Dylan Anderson (Dylan Anderson on LinkedIn, Dylan Anderson on Substack)
• Link to the article: https://thedataecosystem.substack.com/p/issue-22-deciding-on-your-data-platform
• Publisher: Substack

Composable Data Management

• Title: Composable data management at Meta
• Date: May 2024
• Authors: Pedro Pedreira, Amit Purohit
• Link to the article: https://engineering.fb.com/2024/05/22/data-infrastructure/composable-data-management-at-meta/
• Publisher: Meta

Openhouse

• Title: Open sourcing Openhouse
• Author: Sumedh Sakdeo
• Date: March 2024
• Link to the article: https://www.linkedin.com/blog/engineering/open-source/open-sourcing-openhouse

Metaphor and DataHub

• The Grand Rewrite of DataHub, by Mars Lan et al, Sep. 2023 - https://metaphor.io/blog/the-grand-rewrite-of-datahub
• The Modern Metadata Platform (MMP): What, Why, and How? by Mars Lan et al, Jan. 2022 - https://metaphor.io/blog/the-modern-metadata-platform-what-why-and-how
• DataHub: A generalized metadata search & discovery tool, by Mars Lan et al, Aug. 2019 - https://engineering.linkedin.com/blog/2019/data-hub

Delta universal format (UniForm)

• Delta Lake Universal Format (UniForm) for Iceberg compatibility, now Generally Available (GA): https://www.databricks.com/blog/delta-lake-universal-format-uniform-iceberg-compatibility-now-ga
• Authors: Jonathan Brito, Fred Liu and Susan Pierce
• Date: June 2024

Frameworks

See:

• GitHub - Material for the Data platform - Data product frameworks
• GitHub - Material for the Data platform - Metadata frameworks