Skip to content

feat(connector): add OpenLineage and Marquez data-source connector (cognee#5554) - #351

Open
somuai wants to merge 2 commits into
topoteretes:mainfrom
somuai:feat/connector-openlineage
Open

somuai wants to merge 2 commits into
topoteretes:mainfrom
somuai:feat/connector-openlineage

Conversation

@somuai

@somuai somuai commented Oct 9, 2026 •

Copy link
Copy Markdown

Description

This PR introduces the OpenLineage and Marquez data-source connector for Cognee under packages/connector/openlineage/.

It enables syncing pipeline lineage graphs, upstream/downstream dataset dependencies, schema facets, run lifecycles, and failure diagnostics directly into Cognee's AI memory graph, with support for document-mode routing, defensive secret redaction, and forget-on-delete snapshot semantics.

Architecture & Data Flow

Cognee OpenLineage Connector Architecture

Changes

  • Data-Source Connector (openlineage_source): Implements a dlt-based data-source connector querying LF AI OpenLineage and Marquez HTTP backends (/api/v1/namespaces, /api/v1/jobs, /api/v1/runs, /api/v1/datasets) with exponential backoff retries.
  • Semantic Lineage Projection: Extracts pipeline topologies, input/output datasets, schema field types, and run failure facets formatted into clean Markdown documents optimized for Cognee cognify entity extraction and knowledge graph construction.
  • Document-Mode Routing: Tags the dlt source with DOCUMENT_SOURCE_ATTR = "openlineage" to ensure proper routing through Cognee's multi-source memory system.
  • Full-Snapshot Sync & Forget-on-Delete: Emits lineage resources with write_disposition="replace". Decommissioned jobs or retired pipelines drop out of subsequent sync runs and are pruned from downstream knowledge graphs.
  • Defensive Secret Redaction: Automatically masks sensitive credentials, tokens, passwords, and private keys found in job facets, parameters, or dataset connection URIs.
  • Packaging & Companion Docs: Includes runnable examples/, full README.md, pyproject.toml, offline test suite, and companion technical blog post in BLOG_POST.md with rendered visual cards.

Testing

  • Unit & Integration Tests: All 10 unit and integration tests passed offline in 7.62s using mock client fixtures (FakeMarquezClient) with zero external network dependencies.
  • Lint & Formatting: 100% clean under ruff check and ruff format --check.

Related Issues

Closes topoteretes/cognee#5554

Companion Technical Walkthrough

A companion deep-dive technical blog post has been published:

DCO Affirmation

I affirm that all code in every commit of this pull request conforms to the terms of the Topoteretes Developer Certificate of Origin.

@somuai

somuai commented Oct 10, 2026

Copy link
Copy Markdown
Author

Reviewer Verification Checklist (OpenLineage & Marquez Connector)

Acceptance Criteria Verification Matrix

Requirement Implementation Reference Verification Status
Document-Mode Routing (DOCUMENT_SOURCE_ATTR = "openlineage") openlineage_source() in cognee_community_connector_openlineage/openlineage.py sets setattr(source, DOCUMENT_SOURCE_ATTR, "openlineage") so resolve_dlt_sources routes rendered pipeline lineage documents through cognify. [VERIFIED]
Dual-Transport Marquez REST API + RunEvent Facet Stream Projection OpenLineageClient queries /api/v1/namespaces, /api/v1/namespaces/{ns}/jobs, /api/v1/namespaces/{ns}/jobs/{job}/runs, and /api/v1/namespaces/{ns}/datasets/{ds}, projecting job topologies, input/output dataset schema facets, and RunEvent errorMessage facets into structured Markdown. [VERIFIED]
Incremental Watermark Cursor & Bounded Run Window max_runs_per_job cursor windowing and deterministic job/dataset document projection preserve stable content hashes across unchanged lineage states while surfacing new run transitions (COMPLETE, FAIL, ABORTED). [VERIFIED]
Forget-on-Delete Reconciliation (write_disposition="replace") @dlt.resource(name="openlineage_jobs", write_disposition="replace") overwrites staging on each sync so retired or decommissioned jobs drop out and are pruned from Cognee's graph and vector stores via orphan_cleanup. [VERIFIED]
Regex Secret Scrubbing (JDBC / S3 / Bearer Credentials) _SECRET_PATTERN = re.compile(r"(token|secret|password|key|credential)", re.IGNORECASE) and redact_sensitive_value() mask sensitive tokens, passwords, and credentials with [REDACTED]. [VERIFIED]
Exponential Backoff on Transient HTTP 429 / 5xx & Timeouts OpenLineageClient.get() implements exponential backoff (_RETRY_BACKOFF_FACTOR * (2 ** (attempt - 1)), up to _MAX_RETRIES = 5) on httpx.ConnectError, httpx.TimeoutException, and HTTP 429/500/502/503/504. [VERIFIED]
Offline Pytest Suite (10/10 Passing) packages/connector/openlineage/tests/test_openlineage.py validates field/run table formatting, complete job Markdown rendering, document-mode tagging, namespace/job filtering, regex secret redaction, empty catalog handling, and multi-run write_disposition="replace" reconciliation using offline FakeMarquezClient fixtures. [PASS] (10/10)
Lint & Format (ruff check & ruff format --check) Zero lint violations and 100% formatting compliance across packages/connector/openlineage. [PASS]
DCO Sign-off All commits include Signed-off-by trailers and PR body affirms the Topoteretes Developer Certificate of Origin. [VERIFIED]

Quick Maintainer Reproduction Command

pytest packages/connector/openlineage/tests -v && ruff check packages/connector/openlineage && ruff format --check packages/connector/openlineage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hackathon [Feature]: Add OpenLineage / Marquez data-source connector

1 participant