OpenLineage: A Common Observability Framework for Data Pipelines — Datadog Illuminated with Julien Le Dem

ep05-board-gist.md

OpenLineage: A Common Observability Framework for Data Pipelines

Associated Datadog Illuminated episode: OpenLineage: A Common Observability Framework for Data Pipelines

---

Problems

• Isolated teams
• Bad data propagates (errors, privacy)
• Finding bad data / sources is hard
• No common language

---

Data Lineage

Data Lineage is the record of:

1. where data originated
2. how data is moved around
3. how it is transformed

---

How OpenLineage Works

RunEvents

• RunEvents = Start Event + End Event
• Job: a task that transforms data
• Run: an instance of that task
• Datasets (input/output): data in / data out

flowchart TD
    subgraph OrgA["Org A"]
        A1[Dataset] --> A2[Dataset]
    end
    subgraph OrgB["Org B"]
        B1[Dataset] --> B2[Dataset]
        B1 --> B3[Dataset]
    end
    subgraph OrgC["Org C"]
        C1[Dataset]
    end
    A2 --> B1
    B2 --> C1
    A1 -.->|RunEvent| Repo[Data Lineage Repo]
    A2 -.->|RunEvent| Repo
    B1 -.->|RunEvent| Repo
    B2 -.->|RunEvent| Repo
    B3 -.->|RunEvent| Repo
    C1 -.->|RunEvent| Repo

Loading

Facets

• Facets: metadata spec that's specific to a technology
• Job facet / Run facet / Dataset facet

---

OpenLineage

An open observability framework:

1. Specification
2. Libraries
3. Integrations

For data processing. Multi-vendor, wide adoption.

Before OpenLineage — every producer connects directly to every consumer:

flowchart TB
    subgraph Producers["Data Producers — ex. Postgres, Spark, Snowflake"]
        P1[Tool 1]
        P2[Tool 2]
        P3[Tool 3]
        P4[Tool 4]
    end
    subgraph Consumers["Data Consumers — ex. Marquez, DataHub, Datadog"]
        C1[Tool 1]
        C2[Tool 2]
        C3[Tool 3]
        C4[Tool 4]
    end
    P1 --> C1
    P1 --> C2
    P1 --> C3
    P1 --> C4
    P2 --> C1
    P2 --> C2
    P2 --> C3
    P2 --> C4
    P3 --> C1
    P3 --> C2
    P3 --> C3
    P3 --> C4
    P4 --> C1
    P4 --> C2
    P4 --> C3
    P4 --> C4

Loading

With OpenLineage — hub and spoke:

flowchart TB
    subgraph Producers["Data Producers — ex. Postgres, Spark, Snowflake"]
        P1[Tool 1]
        P2[Tool 2]
        P3[Tool 3]
        P4[Tool 4]
    end
    OL[OpenLineage]
    subgraph Consumers["Data Consumers — ex. Marquez, DataHub, Datadog"]
        C1[Tool 1]
        C2[Tool 2]
        C3[Tool 3]
        C4[Tool 4]
    end
    P1 --> OL
    P2 --> OL
    P3 --> OL
    P4 --> OL
    OL --> C1
    OL --> C2
    OL --> C3
    OL --> C4

Loading

---

OpenLineage Unlocks Cool Stuff

• Try different tools easily!
• Easier onboarding (people, tools!)
• Root cause identification
• Data reliability
• Compliance
• Impact analysis

---

OpenLineage at Datadog

• Data job monitoring: faster time to detect issues
• Data quality monitoring: helps find bad data
• Lineage Graph: priorities

---

Resources

• openlineage.io
• marquezproject.ai

---