Repository navigation
Conversation
…ognee#5554) Signed-off-by: Soumyajit Ghosh <[email protected]>
3 of 9 tasks
…s and PR link Signed-off-by: Soumyajit Ghosh <[email protected]>
6 tasks
Author
Reviewer Verification Checklist (OpenLineage & Marquez Connector)
Acceptance Criteria Verification Matrix
Quick Maintainer Reproduction Commandpytest packages/connector/openlineage/tests -v && ruff check packages/connector/openlineage && ruff format --check packages/connector/openlineage |
This was referenced Oct 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces the OpenLineage and Marquez data-source connector for Cognee under
packages/connector/openlineage/.It enables syncing pipeline lineage graphs, upstream/downstream dataset dependencies, schema facets, run lifecycles, and failure diagnostics directly into Cognee's AI memory graph, with support for document-mode routing, defensive secret redaction, and forget-on-delete snapshot semantics.
Architecture & Data Flow
Changes
openlineage_source): Implements a dlt-based data-source connector querying LF AI OpenLineage and Marquez HTTP backends (/api/v1/namespaces,/api/v1/jobs,/api/v1/runs,/api/v1/datasets) with exponential backoff retries.DOCUMENT_SOURCE_ATTR = "openlineage"to ensure proper routing through Cognee's multi-source memory system.write_disposition="replace". Decommissioned jobs or retired pipelines drop out of subsequent sync runs and are pruned from downstream knowledge graphs.examples/, fullREADME.md,pyproject.toml, offline test suite, and companion technical blog post inBLOG_POST.mdwith rendered visual cards.Testing
FakeMarquezClient) with zero external network dependencies.ruff checkandruff format --check.Related Issues
Closes topoteretes/cognee#5554
Companion Technical Walkthrough
A companion deep-dive technical blog post has been published:
packages/connector/openlineage/BLOG_POST.mdDCO Affirmation
I affirm that all code in every commit of this pull request conforms to the terms of the Topoteretes Developer Certificate of Origin.