Skip to content

feat(connector): add Apache Iceberg data-source connector (cognee#5553) - #348

Open
somuai wants to merge 5 commits into
topoteretes:mainfrom
somuai:feat/connector-iceberg
Open

somuai wants to merge 5 commits into
topoteretes:mainfrom
somuai:feat/connector-iceberg

Conversation

@somuai

@somuai somuai commented Oct 9, 2026 •

Copy link
Copy Markdown

Description

This PR introduces the Apache Iceberg data-source connector for Cognee under packages/connector/iceberg/.

It enables syncing Apache Iceberg lakehouse catalog metadata (table schemas, nested fields, partition specs, and snapshot commit histories) directly into Cognee's AI memory graph, with support for incremental sync, secret redaction, and forget-on-delete semantics.

Architecture & Data Flow

Cognee Apache Iceberg Connector Architecture

Changes

  • Data-Source Connector (iceberg_source): Implements a dlt-based data-source connector using pyiceberg to inspect REST, AWS Glue, Hive Metastore, or Nessie catalogs.
  • Metadata-First Architecture: Ingests architectural schemas, column descriptions, partition transforms (identity, bucket, truncate, day, month, year), and snapshot commit history into Cognee's cognitive memory graph without dumping raw lakehouse data or leaking PII.
  • Document-Mode Routing: Tags source with DOCUMENT_SOURCE_ATTR = "iceberg" to route lakehouse metadata into Cognee's cognify entity extraction and graph completion pipeline.
  • Full-Snapshot Sync & Forget-on-Delete: Emits tables with write_disposition="replace". Decommissioned or dropped tables disappear from subsequent sync runs and are automatically pruned from Cognee knowledge graph and vector stores via orphan_cleanup.
  • Secret Redaction: Defensively redacts sensitive properties (matching token, secret, password, key) prior to ingestion.
  • Packaging & Companion Docs: Includes runnable examples/, README.md, pyproject.toml, uv.lock, and comprehensive companion technical blog post in BLOG_POST.md with rendered visual assets.

Testing

  • Unit & Integration Tests: All 8 unit and integration tests passed offline using in-memory mock catalogs (FakeIcebergCatalog, FakeIcebergTable).
  • Lint & Formatting: 100% clean under ruff check and ruff format --check.

Related Issues

Closes topoteretes/cognee#5553

Companion Technical Walkthrough

A companion deep-dive technical blog post has been published:

DCO Affirmation

I affirm that all code in every commit of this pull request conforms to the terms of the Topoteretes Developer Certificate of Origin.

…isual assets to blog post

Signed-off-by: Soumyajit Ghosh <[email protected]>
@somuai

somuai commented Oct 10, 2026

Copy link
Copy Markdown
Author

Reviewer Verification Checklist (Apache Iceberg Connector)

Acceptance Criteria Verification Matrix

Requirement Implementation Reference Verification Status
Document-Mode Routing (DOCUMENT_SOURCE_ATTR = "iceberg") iceberg_source() in cognee_community_connector_iceberg/iceberg.py sets setattr(source, DOCUMENT_SOURCE_ATTR, "iceberg") so resolve_dlt_sources routes table metadata documents directly through cognify. [VERIFIED]
Incremental Snapshot Watermark Sync _table_to_row() and _render_snapshots() in cognee_community_connector_iceberg/iceberg.py capture current_snapshot_id and reverse-chronological snapshot commit histories (snapshot_id, UTC timestamp, operation, added-records), ensuring deterministic content hashing across unchanged snapshots. [VERIFIED]
Forget-on-Delete Reconciliation (write_disposition="replace") @dlt.resource(name="iceberg_tables", primary_key="id", write_disposition="replace") paired with _is_gone() handling (NoSuchTableError, NoSuchNamespaceError) prunes dropped tables via orphan_cleanup. [VERIFIED]
Regex / Key-Pattern Secret Scrubbing (S3 / Bearer / Catalog Tokens) _render_properties() filters sensitive property keys matching token, secret, password, or key (e.g., s3.secret-access-key, catalog bearer tokens) prior to document emission. [VERIFIED]
Exponential Backoff on Transient HTTP 429 / 5xx & Timeouts _request() and _is_transient() implement bounded exponential backoff (2**attempt, up to _MAX_RETRIES = 5) on rate limits, HTTP 502/503/504, and connection timeouts. [VERIFIED]
Offline Pytest Suite (8/8 Passing) packages/connector/iceberg/tests/test_iceberg.py covers schema rendering, partitioned/unpartitioned specs, snapshot formatting, secret redaction, row structure, transient/gone error classification, and multi-run write_disposition="replace" reconciliation using in-memory FakeIcebergCatalog fixtures. [PASS] (8/8)
Lint & Format (ruff check & ruff format --check) Zero lint violations and 100% formatting compliance across packages/connector/iceberg. [PASS]
DCO Sign-off All commits include Signed-off-by trailers and PR body affirms the Topoteretes Developer Certificate of Origin. [VERIFIED]

Quick Maintainer Reproduction Command

pytest packages/connector/iceberg/tests -v && ruff check packages/connector/iceberg && ruff format --check packages/connector/iceberg

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hackathon [Feature]: Add Apache Iceberg data-source connector

1 participant