You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
cognee has five data-source connectors today (confluence, gmail, google-drive, notion, slack) against roughly 120 planned. Data sources are the thinnest surface in the project.
Amazon Redshift is on the roadmap and has no connector.
Proposed Solution
Add a redshift connector under packages/connector/redshift/ in cognee-community, following the shape of the five existing connectors.
Auth: IAM / database credentials
Ingest: query results and table comments
Incremental sync: timestamp column cursor
Forget-on-delete: deleting upstream must remove the record from the graph on the next sync
Watch out for
Reachability is the hard part — usually inside a VPC. Document the tunnel.
Hi @NMZivkovic, I'd like to take this one. Please assign it to me.
Approach
I'll build packages/connector/redshift/ in cognee-community, following the layout and patterns of the existing notion connector.
Auth: IAM (temporary cluster credentials) or standard database user/password. The user provides cluster/workgroup details and selects which schemas, tables, or queries to ingest.
Ingest: query results from the selected tables or custom SQL, plus table and column comments, turned into documents that cognee can add and cognify.
Incremental sync: a configurable timestamp column is used as the cursor. Each run fetches only rows newer than the last stored value.
Forget-on-delete: the connector tracks the keys of synced rows and compares them against the source on each sync. Rows or tables removed upstream are deleted from the graph.
Reachability: since Redshift usually sits inside a VPC, the README will include a step-by-step guide for connecting through an SSH tunnel / bastion host, and the connector will be tested over that setup.
Deliverables: README with setup steps, a runnable example under examples/, and tests for the ingest path and the incremental cursor.
This will be tested against a real Redshift instance, not just mocks. I'll follow the DCO/commit rules, keep the PR solo, and publish a blog post on a real-world use case once the PR is open.
Hi @NMZivkovic, I’d like to work on this for Mergetober.
How I’d build it:
Source: Add packages/connector/redshift/ in cognee-community, following Notion’s layout. Use boto3’s Data API by default for provisioned clusters and Serverless, with IAM authorization and temporary database credentials or Secrets Manager. Support direct redshift_connector connections with IAM/database credentials, documenting SSH/SSM tunneling with TLS hostname verification.
Data: Ingest selected table/query rows and table/column comments as separately keyed Cognee documents, preserving types, nulls and provenance. Validate unique result keys and refresh comments independently. Use parameterized timestamp filters with overlap and content hashes, including corrections at the saved timestamp. For native tables, read changes and complete key inventories in one transaction every sync; fetch newly discovered keys even below the cursor. Queries without a reliable update timestamp use full keyed snapshots. Poll to FINISHED and consume every NextToken page before staging merges and hard-delete tombstones. Failed, expired or oversized results preserve state; retained staging supports downstream replay. Scope state and cleanup by warehouse, database, query and Cognee dataset.
Testing: Cover pagination, retries, timestamp ties, late arrivals, comment-only edits, failed inventories and final-row deletion. After assignment, validate ingestion
For #4773, I’d strengthen your existing comment by making the Data API the default, validating result keys, and refreshing comments independently of row timestamps. AWS’s Data API provides HTTPS access, and Redshift does not enforce declared primary-key uniqueness—so the connector needs its own key validation.
Verification: Five focused Cognee checks and four SDK probes passed, covering timestamp corrections, metadata updates, pagination, retries and deletion handling. These were offline checks; live AWS access, tunneling and physical graph/vector deletion remain to be verified.
@dexters1 could you please assign this issue to me ? would also write a blog post of my integration
Hey @NMZivkovic, I am interested in working on this issue.
I read the community connectors, cognee's relational dlt path on dev, and the AWS docs for the Redshift Data API. The main design choice follows from the "reachability" note in the issue.
Transport: Data API by default, direct driver as fallback
The Redshift Data API is an HTTPS endpoint called through boto3 (redshift-data). It does not need a network path into the VPC, a driver, or a persistent connection. That removes the tunnel problem for most users.
It works with provisioned clusters (ClusterIdentifier + DbUser for IAM temporary credentials) and Serverless workgroups (WorkgroupName), or with a Secrets Manager SecretArn. No password is passed in the call.
For users who cannot use the Data API, a direct mode uses redshift_connector with IAM auth. The README will document the SSH tunnel through a bastion host for this mode.
What gets ingested
Rows: users select tables, or custom SQL with a declared primary_key and cursor_column. Rows go through cognee's relational path, so they become graph nodes without LLM extraction.
Table comments: read from SVV_ALL_TABLES.remarks and SVV_ALL_COLUMNS.remarks. Each table becomes a short "table card" document on the document path: the table name, its comment, and its columns with types and comments. This adds the meaning of each table and column to the graph, which the relational path alone does not carry.
Incremental sync
Per table, WHERE cursor_column >= :cursor is passed as a Data API named parameter, never formatted into the SQL string.
Rows that share the cursor value are handled with the Linear connector's floor pattern: keep the cursor plus the keys seen at that exact value, and skip them next run. An optional lookback covers late-arriving loads (for example COPY jobs with older timestamps).
Large tables are read in ordered batches, so no single result goes over the Data API's 500 MB limit. Results are paged with NextToken.
The cursor only advances after every statement succeeds.
Forget-on-delete
Merge with a _deleted hard-delete column, like Drive, Gmail and Linear.
If a table has a soft-delete column (deleted_at or is_deleted, configurable), deletions come through the normal cursor as tombstones.
Otherwise each run runs SELECT <primary_key> FROM <table>. Redshift is columnar, so this only reads the key column and stays cheap. Keys that disappeared since the last run get tombstones, but only after a clean full key scan.
Tests and docs
Tests stub boto3 with botocore's Stubber: statement polling, NextToken paging, a failed statement leaving state unchanged, cursor ties, lookback, soft-delete and key-scan tombstones, and comment cards.
A README covering IAM policy setup, Data API vs direct mode, and the tunnel steps, plus a runnable examples/example.py.
Hi @NMZivkovic and @dexters1 I’m an active contributor in the Cognee community and would like to work on #4773 – Amazon Redshift data-source connector.
I reviewed the existing packages/connector/notion/ implementation and the current Redshift proposals. I’d implement this as a DLT-based connector, but with a deliberate transport and synchronization design to address the main Redshift-specific challenges.
Proposed implementation
1. Connectivity
Use the Redshift Data API as the default transport so users do not need direct VPC connectivity for the common case.
Support:
provisioned clusters
Redshift Serverless workgroups
IAM authentication
Secrets Manager credentials
For users that require direct database connectivity, provide a direct mode using the Redshift database driver and document SSH/bastion tunneling in the README.
2. Selected ingestion
Expose explicit selection for schemas, tables and custom read queries:
Raw rows will stay on the structured/relational ingestion path, while table and column descriptions will be normalized into searchable metadata documents.
3. Table metadata
In addition to row data, ingest table and column comments with:
database/schema/table
table description
column names
data types
nullability
column comments
A metadata hash will be used so comment/schema changes are detected even when row timestamps have not changed.
4. Incremental synchronization
Each selected table/query will have an independent cursor:
cursor_column
cursor_value
cursor_keys_at_cursor
The query will use a parameterized timestamp condition with an overlap/lookback window and stable primary-key ordering.
Timestamp ties will be handled using:
timestamp + keys already processed at that timestamp
The cursor will only be committed after every result page succeeds, so a partially failed run cannot skip records.
5. Result pagination
For Data API execution I’ll explicitly handle:
ExecuteStatement
→ poll statement state
→ consume ResultSet
→ follow NextToken
→ finish only after all pages are read
HTTP success will not be treated as SQL success without checking the Redshift statement status.
6. Forget-on-delete
For tables with a configured soft-delete column such as deleted_at / is_deleted, emit deletion tombstones through the normal incremental path.
For tables without soft deletion, maintain a stable primary-key inventory and periodically run a lightweight key-only query:
SELECT customer_id FROMpublic.customers
After a complete successful inventory:
previous_keys - current_keys
becomes the confirmed deletion set.
A failed, partial, timed-out, or permission-limited inventory will never trigger deletion or advance the checkpoint.
This ensures an upstream deleted row is removed from Cognee on the next successful sync without treating connectivity failures as deletions.
The README will document both Data API connectivity (recommended) and direct-driver + SSH/bastion connectivity, along with AWS/IAM setup, environment variables, table/query selection, incremental behavior, deletion semantics and known limitations.
I’m especially interested in solving the VPC reachability and safe deletion requirements rather than implementing Redshift as only a basic SQL wrapper.
Could you please assign #4773 to me? I’m ready to begin implementation.
Maintainer Note: In case this issue is considered for reassignment due to assignee inactivity / lack of PR, I am fully prepared to take ownership and deliver the Amazon Redshift connector within 24–48 hours for Mergetober following the complete technical blueprint below:
Here is my technical design and implementation plan, modeled directly after packages/connector/notion/ and cognee/tasks/ingestion/resolve_dlt_sources.py:
Structure: packages/connector/redshift/ in cognee-community with standard layout (redshift.py, README.md, examples/example.py, tests/test_redshift.py, pyproject.toml).
Deterministic Document IDs: Documents get deterministic IDs (redshift:<entity>:{id}) so re-syncs update in place and never duplicate in Cognee graph/vector memory.
Fail-Fast Auth: API credentials configured via dlt secrets or environment variables. On startup, the client makes one cheap validation call (DescribeDatabases call) to verify token validity and scopes before ingesting, failing fast with an actionable error if unauthenticated or under-scoped. Credentials are never logged.
Selection & Explicit Field Allowlist: Resources configurable via resources=["catalogs", "schemas", "tables", "views"]. Only named, schema-allowlisted fields are projected into structured Markdown documents (metadata preserved as columns for citation). Ingests data catalog schemas, column descriptions, and view definitions via Amazon Redshift Data API. Bulk table data rows are excluded.
Document-Mode Ingestion (cognify): Declares DOCUMENT_SOURCE_ATTR = "redshift", routing formatted Markdown documents through Cognee's entity extraction and knowledge graph memory layer.
Incremental Sync via State Cursor: Tracks high-watermark timestamp cursor (updated_at / last_modified) inside dlt.current.resource_state()["last_synced_at"]. The cursor advances only after a page loads successfully so interrupted runs resume safely.
Forget-on-Delete & Deletion Safety Guard: Uses write_disposition="replace" so deleted/archived upstream items drop out of the snapshot and are purged via Cognee's orphan_cleanup. Includes a max_delete_ratio safety threshold to abort and alert if an anomalous/empty API response attempts to wipe a large share of the graph.
Resilience & Backoff: Maximum page size pagination with duplicate-page guards, handling HTTP 429 and 5xx errors with exponential backoff and strict Retry-After delay validation (protecting against negative, NaN, and infinite header values).
Docs & Testing: Comprehensive README, runnable example, and offline unit test suite (tests/test_redshift.py) with httpx.MockTransport and a temporary DuckDB destination covering multi-page pagination, cursor advancement, deletion cleanup, and retry backoff.
I have verified the developer API specifications and can open the tested PR within 24–48 hours if this issue becomes available for reassignment. Thanks!
Hi @NMZivkovic, I'd like to work on the Amazon Redshift connector for Mergetober if this issue is still available. I have reviewed AWS Redshift documentation and Cognee's connector patterns. This is proposed work; I have not started implementation and will wait for assignment.
Here is my proposed approach:
Scope, authentication and boundaries
Package:packages/connector/redshift/ in cognee-community, following the packaging structure of existing database and Notion connectors under Hatchling.
Auth: Support official AWS IAM authentication using the Redshift Data API (boto3.client("redshift-data")) for serverless/managed execution, as well as direct cluster credentials via redshift_connector. The adapter validates connection parameters upfront using a lightweight probe against cluster catalog metadata.
Security & SSRF: Enforce SSL connection encryption (sslmode=verify-full) and bound connection hosts to authorized AWS endpoints. Document required least-privilege IAM actions (redshift-data:DescribeTable, redshift-data:ExecuteStatement, redshift-data:GetStatementResult); never log database passwords or temporary IAM session tokens.
Scope Boundaries: Allow users to specify target database schemas (schema_names) and tables (table_names). The connector prioritizes catalog metadata and schema structure; table row extraction is opt-in and bounded by configurable row limits to prevent memory exhaustion on massive warehouses.
Schema introspection and table rendering
Introspect catalog metadata via standard Redshift system views: SVV_TABLE_INFO, SVV_ALL_COLUMNS, and pg_catalog.pg_description to extract table descriptions, column data types, distribution keys, sort keys, and column comments.
Map schema relationships into Cognee's knowledge graph: tables as primary document entities, with foreign keys and join paths as explicit relational graph edges.
Render table schemas, comments, and constraints into clean Markdown prose for downstream LLM semantic retrieval.
For selected table row data, serialize records into document-mode prose with stable composite IDs ({schema}.{table}.{pk}), sanitizing binary, geometry, and sensitive raw formats.
Incremental sync and deletion policy
Cursor: For table data ingestion, track monotonic timestamp or serial ID watermark columns (e.g. updated_at) with overlapping query windows to handle clock skew across distributed cluster slices.
Forget-on-Delete: Dropped tables or retired schemas upstream must not leave dangling graph nodes. The connector implements snapshot replacement for catalog metadata: calling remember(..., write_disposition="replace", run_in_background=False, max_rows_per_table=0) after successful catalog sweeps so removed tables drop out via Cognee's orphan_cleanup.
Zero-Row Edge Case: Test authoritative zero-row replacement so dropping the final table in a monitored schema purges the table and vector embeddings completely rather than leaving stale vector records in LanceDB.
Pin and test against a compatible Cognee version with DOCUMENT_SYNC_VERSION >= 1 (supporting table-scoped cleanup and confirmed empty-table handling). Use DOCUMENT_SOURCE_ATTR = "redshift" and PIPELINE_SCOPE_ATTR on the selected core.
Intercept Redshift cluster pause states, query timeouts, and AWS API throttling with exponential backoff and jitter.
Testing Standard: 100% offline pytest suite mocking Redshift Data API responses, SQL schema introspection, query execution pagination, and last-table deletion. Live validation will be demonstrated after assignment using a test database.
Deliver pyproject.toml, mocked unit tests, README with minimum IAM policies and setup steps, a runnable examples/example.py, and the required usage blog post with DCO sign-offs.
Could you assign this issue to me if this approach fits? Thanks!
I’ve reviewed the issue and the existing Cognee connector patterns. My approach would be to use the Redshift Data API as the primary transport through boto3, with direct database connectivity as a documented fallback for environments that require it.
For incremental sync, I’ll support a configurable cursor column with stable key handling for timestamp ties and only advance the checkpoint after all result pages have completed successfully. I’ll also make deletion handling conservative so failed or incomplete source reads never cause valid Cognee data to be removed.
I’ll include table/column metadata, the required package structure, mocked tests for pagination, cursor state, failures and deletion handling, a runnable example, and documentation covering IAM permissions and connectivity options.
I’m ready to work on this immediately and will wait for assignment before starting implementation. Thanks!
Hi @NMZivkovic, I would like to take on #4773 for Mergetober. I have thoroughly reviewed the existing community connectors (like Notion) and the architectural constraints specific to Redshift.
Here is how I plan to implement it cleanly and robustly:
Transport & Connectivity:
Use the Redshift Data API (boto3) as the primary transport layer to bypass complex VPC network/bastion tunnel setup for standard users.
Support standard IAM authentication, temporary cluster credentials, and Secrets Manager, with direct driver fallback documented clearly.
Ingestion & Metadata:
Ingest structural data rows into Cognee's relational path, ensuring stable primary-key identities (redshift:{db}:{schema}:{table}:{pk}).
Extract table and column remarks from system catalogs (SVV_ALL_TABLES / SVV_ALL_COLUMNS) and turn them into searchable "table card" documents so LLMs understand schema semantics.
Fully manage NextToken pagination and statement polling so that large query results never exceed memory limits.
Forget-on-Delete & Safety:
Use soft-delete columns where available, or perform a lightweight key-only inventory scan (SELECT primary_key FROM table) to safely handle hard deletions without risking data loss on network timeouts.
The cursor and state will only advance after every statement/page successfully commits.
Deliverables:
Clean package layout under packages/connector/redshift/ containing pyproject.toml, a comprehensive README.md with IAM/tunneling setups, a runnable examples/example.py, and a robust pytest suite using mocked responses.
I am ready to start immediately and will ensure a clean, solo PR with a companion blog post. Please assign this issue to me!
Hi @dexters1@NMZivkovic, I would like to request assignment for #4773 to implement the Amazon Redshift data-source connector in topoteretes/cognee-community.
Having recently shipped and verified two production data-platform connectors in cognee-community following the exact packages/connector/ architecture:
Supports both Redshift Data API (transport="data_api", default, zero VPC tunnel needed) and Direct DB-API 2.0 (transport="direct", via redshift-connector over VPC/SSH tunnel) across Provisioned Clusters and Redshift Serverless Workgroups:
Queries SVV_TABLE_INFO, SVV_ALL_TABLES, SVV_COLUMNS, PG_DESCRIPTION, and PG_CONSTRAINT to synthesize structured Markdown table cards containing:
Table & column REMARKS (semantic business descriptions).
Physical layout diagnostics: diststyle (KEY, EVEN, ALL, AUTO), sortkey1, skew_rows, unsorted percentage, pct_used, tbl_rows, and size (MB).
Primary key, foreign key, and uniqueness constraints from PG_CONSTRAINT.
Redshift Spectrum federated table definitions from SVV_EXTERNAL_TABLES and SVV_EXTERNAL_COLUMNS.
Operational Lineage & Load Diagnostics:
Optionally extracts bounded summaries from SYS_QUERY_HISTORY (recent DDL/DML transformations, execution duration, queue wait times) and STL_LOAD_ERRORS (S3 COPY ingestion failures with sanitized error reasons) so Cognee can answer operational warehouse debugging questions.
Parameterized Incremental Sync with Tie-Breaking Floor:
Executes parameterized queries via Data API Parameters=[{"name": "cursor_watermark", "value": ...}] (never string-interpolated) using WHERE <cursor_column> >= :cursor_watermark ORDER BY <cursor_column> ASC, <primary_key> ASC.
Applies a configurable lookback_seconds window for late-arriving COPY / micro-batch loads and deduplicates boundary ties via cursor_tie_keys.
Handles ExecuteStatement -> exponential backoff polling on DescribeStatement (FINISHED vs FAILED / ABORTED) -> full GetStatementResultNextToken pagination. The checkpoint advances only after all pages succeed.
Regex Secret & Credential Scrubbing:
Automatically scrubs IAM_ROLE 'arn:aws:iam::...', aws_access_key_id=..., aws_secret_access_key=..., aws_session_token=..., SECRET_ARN, S3 pre-signed query strings, and JDBC/ODBC password parameters from all query texts, table DDL/comments, and load error records prior to document rendering.
C. ingest.py — Forget-on-Delete Graph Reconciliation (write_disposition="replace")
Computes a deterministic catalog & primary-key inventory snapshot (redshift:{database}:{schema}:{table}:{pk}) and detects:
Schema/Table Drops or DDL Mutations: When an upstream table is dropped from SVV_ALL_TABLES or its schema/comment fingerprint changes.
Row Hard-Deletes & Soft-Deletes: Evaluates configured soft_delete_column (deleted_at / is_deleted) or runs a lightweight columnar key scan (SELECT <primary_key> FROM <schema>.<table>). Because Redshift is columnar, scanning a single sort/primary key column reads minimal blocks.
When upstream deletions or schema removals are confirmed after a complete, non-errored read, ingest() switches the affected dataset sync to write_disposition="replace" in cognee.add(..., dataset_name=dataset_name, data_per_batch=data_per_batch, write_disposition="replace") before running cognee.cognify() and cognee.memify(), ensuring deleted upstream records are purged from Cognee's graph and vector stores without risking false deletions on transient network errors.
We are ready to open a clean, self-contained PR in topoteretes/cognee-community with full documentation (including VPC SSH bastion tunneling and IAM policies) and a companion blog post as soon as this issue is assigned. Thank you!
Hi @NMZivkovic, I'd like to be considered for the Amazon Redshift connector for Mergetober.
I'd use AWS's official Python SDK and Redshift Data API to read selected query results and table comments into Cognee, following the existing community connector structure. The example would use a small sample dataset so someone can try searching their warehouse data and its descriptions.
I'll keep database access read-only, handle paginated results, and use a chosen timestamp column to pick up changes. Since that alone won't catch deleted rows, I'll also add a separate check for removals that only runs after a successful read. I'll add automated tests and validate against a real Redshift test environment before calling it complete, with connection instructions, a runnable example, and the usage blog.
If this is still available, could you assign it to me? I see the earlier requests and will wait for your decision. My other integration requests, including Freshdesk, are alternatives; I'd focus on one assignment at a time. Thanks!
I’d like to contribute to #4773 — Add Amazon Redshift data-source connector as part of Mergetober.
I plan to build the connector around Cognee’s existing integration conventions, with particular attention to reliable incremental synchronization and safe handling of source deletions.
Implementation plan:
Follow the existing architecture: Study packages/connector/notion/ and the integration guide first, then mirror the expected package structure and ingestion lifecycle in packages/connector/redshift/.
Support secure connectivity: Implement configurable Redshift connection settings and IAM/database credential authentication using established AWS and database libraries. Document the required permissions and VPC connectivity options, including SSH tunnelling where appropriate, without hardcoding credentials.
Support selective ingestion: Allow users to select databases, schemas, tables, and relevant content. Ingest query results alongside table comments, preserving stable source identifiers and useful metadata so the content remains searchable in Cognee.
Implement dependable incremental sync: Support a configurable timestamp column for eligible tables, persist the cursor using the project's established conventions, and handle cursor boundaries and failed runs carefully so records are not skipped.
Make deletion handling safe: Track ingested source records and remove corresponding graph data when upstream records are confirmed deleted. Avoid interpreting query failures, permission errors, or incomplete scans as deletions.
Document and validate: Add the requested README.md, examples/, tests, and pyproject.toml, including setup instructions, a runnable example, required permissions, and network-access limitations.
Before finalizing the design, I’ll verify how Cognee currently handles source identifiers, cursor persistence, and graph cleanup so the connector follows established behavior instead of introducing incompatible mechanisms.
Could you please confirm whether this issue is still available and consider assigning it to me? I’ll wait for assignment before starting implementation.
Activity
mohitjeswani01 commentedon Oct 4, 2026
Hi @NMZivkovic, I'd like to take this one. Please assign it to me.
Approach
I'll build
packages/connector/redshift/in cognee-community, following the layout and patterns of the existingnotionconnector.examples/, and tests for the ingest path and the incremental cursor.This will be tested against a real Redshift instance, not just mocks. I'll follow the DCO/commit rules, keep the PR solo, and publish a blog post on a real-world use case once the PR is open.
ANAMASGARD commentedon Oct 5, 2026
Hi @NMZivkovic, I’d like to work on this for Mergetober.
How I’d build it:
Source: Add
packages/connector/redshift/incognee-community, following Notion’s layout. Useboto3’s Data API by default for provisioned clusters and Serverless, with IAM authorization and temporary database credentials or Secrets Manager. Support directredshift_connectorconnections with IAM/database credentials, documenting SSH/SSM tunneling with TLS hostname verification.Data: Ingest selected table/query rows and table/column comments as separately keyed Cognee documents, preserving types, nulls and provenance. Validate unique result keys and refresh comments independently. Use parameterized timestamp filters with overlap and content hashes, including corrections at the saved timestamp. For native tables, read changes and complete key inventories in one transaction every sync; fetch newly discovered keys even below the cursor. Queries without a reliable update timestamp use full keyed snapshots. Poll to
FINISHEDand consume everyNextTokenpage before staging merges and hard-delete tombstones. Failed, expired or oversized results preserve state; retained staging supports downstream replay. Scope state and cleanup by warehouse, database, query and Cognee dataset.Testing: Cover pagination, retries, timestamp ties, late arrivals, comment-only edits, failed inventories and final-row deletion. After assignment, validate ingestion
For #4773, I’d strengthen your existing comment by making the Data API the default, validating result keys, and refreshing comments independently of row timestamps. AWS’s Data API provides HTTPS access, and Redshift does not enforce declared primary-key uniqueness—so the connector needs its own key validation.
Verification: Five focused Cognee checks and four SDK probes passed, covering timestamp corrections, metadata updates, pagination, retries and deletion handling. These were offline checks; live AWS access, tunneling and physical graph/vector deletion remain to be verified.
Akshats-git commentedon Oct 6, 2026
Hey @NMZivkovic, I am interested in working on this issue.
I read the community connectors, cognee's relational dlt path on
dev, and the AWS docs for the Redshift Data API. The main design choice follows from the "reachability" note in the issue.Transport: Data API by default, direct driver as fallback
redshift-data). It does not need a network path into the VPC, a driver, or a persistent connection. That removes the tunnel problem for most users.ClusterIdentifier+DbUserfor IAM temporary credentials) and Serverless workgroups (WorkgroupName), or with a Secrets ManagerSecretArn. No password is passed in the call.redshift_connectorwith IAM auth. The README will document the SSH tunnel through a bastion host for this mode.What gets ingested
primary_keyandcursor_column. Rows go through cognee's relational path, so they become graph nodes without LLM extraction.SVV_ALL_TABLES.remarksandSVV_ALL_COLUMNS.remarks. Each table becomes a short "table card" document on the document path: the table name, its comment, and its columns with types and comments. This adds the meaning of each table and column to the graph, which the relational path alone does not carry.Incremental sync
WHERE cursor_column >= :cursoris passed as a Data API named parameter, never formatted into the SQL string.lookbackcovers late-arriving loads (for example COPY jobs with older timestamps).NextToken.Forget-on-delete
_deletedhard-delete column, like Drive, Gmail and Linear.deleted_atoris_deleted, configurable), deletions come through the normal cursor as tombstones.SELECT <primary_key> FROM <table>. Redshift is columnar, so this only reads the key column and stays cheap. Keys that disappeared since the last run get tombstones, but only after a clean full key scan.Tests and docs
Stubber: statement polling,NextTokenpaging, a failed statement leaving state unchanged, cursor ties, lookback, soft-delete and key-scan tombstones, and comment cards.examples/example.py.I can start working once assigned. Thanks!
Darshan3690 commentedon Oct 7, 2026
Hi @NMZivkovic and @dexters1 I’m an active contributor in the Cognee community and would like to work on #4773 – Amazon Redshift data-source connector.
I reviewed the existing
packages/connector/notion/implementation and the current Redshift proposals. I’d implement this as a DLT-based connector, but with a deliberate transport and synchronization design to address the main Redshift-specific challenges.Proposed implementation
1. Connectivity
Use the Redshift Data API as the default transport so users do not need direct VPC connectivity for the common case.
Support:
For users that require direct database connectivity, provide a
directmode using the Redshift database driver and document SSH/bastion tunneling in the README.2. Selected ingestion
Expose explicit selection for schemas, tables and custom read queries:
Raw rows will stay on the structured/relational ingestion path, while table and column descriptions will be normalized into searchable metadata documents.
3. Table metadata
In addition to row data, ingest table and column comments with:
A metadata hash will be used so comment/schema changes are detected even when row timestamps have not changed.
4. Incremental synchronization
Each selected table/query will have an independent cursor:
The query will use a parameterized timestamp condition with an overlap/lookback window and stable primary-key ordering.
Timestamp ties will be handled using:
The cursor will only be committed after every result page succeeds, so a partially failed run cannot skip records.
5. Result pagination
For Data API execution I’ll explicitly handle:
HTTP success will not be treated as SQL success without checking the Redshift statement status.
6. Forget-on-delete
For tables with a configured soft-delete column such as
deleted_at/is_deleted, emit deletion tombstones through the normal incremental path.For tables without soft deletion, maintain a stable primary-key inventory and periodically run a lightweight key-only query:
After a complete successful inventory:
becomes the confirmed deletion set.
A failed, partial, timed-out, or permission-limited inventory will never trigger deletion or advance the checkpoint.
This ensures an upstream deleted row is removed from Cognee on the next successful sync without treating connectivity failures as deletions.
7. Stable identities
Rows will use a deterministic identity such as:
so updates do not create duplicate graph entities.
Selection and synchronization state will be scoped by dataset/table/query so one connector configuration cannot delete another configuration's data.
Tests
I’ll add mocked tests covering:
Deliverables
The README will document both Data API connectivity (recommended) and direct-driver + SSH/bastion connectivity, along with AWS/IAM setup, environment variables, table/query selection, incremental behavior, deletion semantics and known limitations.
I’m especially interested in solving the VPC reachability and safe deletion requirements rather than implementing Redshift as only a basic SQL wrapper.
Could you please assign #4773 to me? I’m ready to begin implementation.
ADITYASINGH1206 commentedon Oct 8, 2026
Hi @dexters1 and @NMZivkovic,
Maintainer Note: In case this issue is considered for reassignment due to assignee inactivity / lack of PR, I am fully prepared to take ownership and deliver the Amazon Redshift connector within 24–48 hours for Mergetober following the complete technical blueprint below:
Here is my technical design and implementation plan, modeled directly after
packages/connector/notion/andcognee/tasks/ingestion/resolve_dlt_sources.py:packages/connector/redshift/incognee-communitywith standard layout (redshift.py,README.md,examples/example.py,tests/test_redshift.py,pyproject.toml).redshift:<entity>:{id}) so re-syncs update in place and never duplicate in Cognee graph/vector memory.resources=["catalogs", "schemas", "tables", "views"]. Only named, schema-allowlisted fields are projected into structured Markdown documents (metadata preserved as columns for citation). Ingests data catalog schemas, column descriptions, and view definitions via Amazon Redshift Data API. Bulk table data rows are excluded.cognify): DeclaresDOCUMENT_SOURCE_ATTR = "redshift", routing formatted Markdown documents through Cognee's entity extraction and knowledge graph memory layer.updated_at/last_modified) insidedlt.current.resource_state()["last_synced_at"]. The cursor advances only after a page loads successfully so interrupted runs resume safely.write_disposition="replace"so deleted/archived upstream items drop out of the snapshot and are purged via Cognee'sorphan_cleanup. Includes amax_delete_ratiosafety threshold to abort and alert if an anomalous/empty API response attempts to wipe a large share of the graph.Retry-Afterdelay validation (protecting against negative, NaN, and infinite header values).tests/test_redshift.py) withhttpx.MockTransportand a temporary DuckDB destination covering multi-page pagination, cursor advancement, deletion cleanup, and retry backoff.I have verified the developer API specifications and can open the tested PR within 24–48 hours if this issue becomes available for reassignment. Thanks!
Pranjulchaurasiya commentedon Oct 8, 2026
Hi @NMZivkovic, I'd like to work on the Amazon Redshift connector for Mergetober if this issue is still available. I have reviewed AWS Redshift documentation and Cognee's connector patterns. This is proposed work; I have not started implementation and will wait for assignment.
Here is my proposed approach:
Scope, authentication and boundaries
packages/connector/redshift/incognee-community, following the packaging structure of existing database and Notion connectors under Hatchling.boto3.client("redshift-data")) for serverless/managed execution, as well as direct cluster credentials viaredshift_connector. The adapter validates connection parameters upfront using a lightweight probe against cluster catalog metadata.sslmode=verify-full) and bound connection hosts to authorized AWS endpoints. Document required least-privilege IAM actions (redshift-data:DescribeTable,redshift-data:ExecuteStatement,redshift-data:GetStatementResult); never log database passwords or temporary IAM session tokens.schema_names) and tables (table_names). The connector prioritizes catalog metadata and schema structure; table row extraction is opt-in and bounded by configurable row limits to prevent memory exhaustion on massive warehouses.Schema introspection and table rendering
SVV_TABLE_INFO,SVV_ALL_COLUMNS, andpg_catalog.pg_descriptionto extract table descriptions, column data types, distribution keys, sort keys, and column comments.({schema}.{table}.{pk}), sanitizing binary, geometry, and sensitive raw formats.Incremental sync and deletion policy
updated_at) with overlapping query windows to handle clock skew across distributed cluster slices.remember(..., write_disposition="replace", run_in_background=False, max_rows_per_table=0)after successful catalog sweeps so removed tables drop out via Cognee'sorphan_cleanup.References: AWS Redshift Data API, Redshift SVV_TABLE_INFO, Redshift SVV_ALL_COLUMNS, Official redshift-connector driver, and Cognee Community Connector Guidelines.
Cognee contract and deliverables
DOCUMENT_SYNC_VERSION >= 1(supporting table-scoped cleanup and confirmed empty-table handling). UseDOCUMENT_SOURCE_ATTR = "redshift"andPIPELINE_SCOPE_ATTRon the selected core.pytestsuite mocking Redshift Data API responses, SQL schema introspection, query execution pagination, and last-table deletion. Live validation will be demonstrated after assignment using a test database.pyproject.toml, mocked unit tests, README with minimum IAM policies and setup steps, a runnableexamples/example.py, and the required usage blog post with DCO sign-offs.Could you assign this issue to me if this approach fits? Thanks!
Jayesh-Waghmare commentedon Oct 8, 2026
Hi @NMZivkovic @dexters1 , Could you please assign to me this issue?
It would be my first hacktoberfest issue assigned in this repo.
shafiankhan commentedon Oct 8, 2026
Hi @NMZivkovic @dexters1, I’d like to take on #4773 for Mergetober.
I’ve reviewed the issue and the existing Cognee connector patterns. My approach would be to use the Redshift Data API as the primary transport through boto3, with direct database connectivity as a documented fallback for environments that require it.
For incremental sync, I’ll support a configurable cursor column with stable key handling for timestamp ties and only advance the checkpoint after all result pages have completed successfully. I’ll also make deletion handling conservative so failed or incomplete source reads never cause valid Cognee data to be removed.
I’ll include table/column metadata, the required package structure, mocked tests for pagination, cursor state, failures and deletion handling, a runnable example, and documentation covering IAM permissions and connectivity options.
I’m ready to work on this immediately and will wait for assignment before starting implementation. Thanks!
Aditya8369 commentedon Oct 9, 2026
Hi @dexters1 @NMZivkovic,
I am requesting assignment of this issue to build the Amazon Redshift data-source connector.
[Specification: Redshift Connector]
Connection & Auth
Ingestion Scope (
redshift_source)Incremental Sync
updated_at).Deliverables
redshift_sourceconnector module compatible withcognee.remember().example.pyand configuration guide inREADME.md.Waiting until the issue is being assigned. Thanks!
AryaBhatt45 commentedon Oct 9, 2026
Hi @NMZivkovic, I would like to take on #4773 for Mergetober. I have thoroughly reviewed the existing community connectors (like Notion) and the architectural constraints specific to Redshift.
Here is how I plan to implement it cleanly and robustly:
Transport & Connectivity:
boto3) as the primary transport layer to bypass complex VPC network/bastion tunnel setup for standard users.Ingestion & Metadata:
redshift:{db}:{schema}:{table}:{pk}).SVV_ALL_TABLES/SVV_ALL_COLUMNS) and turn them into searchable "table card" documents so LLMs understand schema semantics.Incremental Sync & Pagination:
WHERE cursor_column >= :cursor) with built-in timestamp-tie handling (Linear connector floor pattern).NextTokenpagination and statement polling so that large query results never exceed memory limits.Forget-on-Delete & Safety:
SELECT primary_key FROM table) to safely handle hard deletions without risking data loss on network timeouts.Deliverables:
packages/connector/redshift/containingpyproject.toml, a comprehensiveREADME.mdwith IAM/tunneling setups, a runnableexamples/example.py, and a robustpytestsuite using mocked responses.I am ready to start immediately and will ensure a clean, solo PR with a companion blog post. Please assign this issue to me!
somuai commentedon Oct 10, 2026
Hi @dexters1 @NMZivkovic, I would like to request assignment for
#4773to implement the Amazon Redshift data-source connector intopoteretes/cognee-community.Having recently shipped and verified two production data-platform connectors in
cognee-communityfollowing the exactpackages/connector/architecture:cognee#5553): topoteretes/cognee-community#348 ([PASS] 22/22 offline pytest suite)cognee#5554): topoteretes/cognee-community#351 ([PASS] 22/22 offline pytest suite)Below is our concrete architectural specification for
packages/connector/redshift/that addresses Redshift VPC reachability, physical warehouse metadata (SVV_TABLE_INFO,SVV_COLUMNS,PG_CONSTRAINT), operational query/load lineage (SYS_QUERY_HISTORY,STL_LOAD_ERRORS), parameterized watermark incremental sync, secret scrubbing, and deterministic forget-on-delete graph reconciliation viawrite_disposition="replace".1. System Architecture & Data Flow
flowchart TD subgraph Sources["Amazon Redshift Cluster / Serverless Workgroup"] CAT["System Catalogs & Physical Layout<br/>SVV_TABLE_INFO / SVV_COLUMNS / PG_CONSTRAINT"] EXT["Spectrum External Schemas<br/>SVV_EXTERNAL_TABLES / SVV_EXTERNAL_COLUMNS"] OPS["Operational & Load Lineage<br/>SYS_QUERY_HISTORY / STL_LOAD_ERRORS"] DATA["Incremental Table & Query Projections<br/>Parameterized Cursor + Primary Key Floor"] end subgraph Transport["Dual-Transport Extraction Layer (connector.py)"] DAPI["Primary: AWS Redshift Data API (boto3)<br/>HTTPS Async ExecuteStatement + NextToken Pagination<br/>(Zero VPC Peering / Bastion Required)"] DBAPI["Fallback: redshift-connector (DB-API 2.0)<br/>IAM GetClusterCredentials / Direct TCP over SSH Bastion"] end subgraph Security["Sanitization & Normalization Gate"] SCRUB["Regex Credential & Secret Scrubber<br/>Redacts IAM_ROLE ARNs, aws_access_key_id,<br/>SECRET_ARN, S3 COPY tokens, JDBC/ODBC passwords"] PROJ["Deterministic Markdown & Relational Projector<br/>DOCUMENT_SOURCE_ATTR = 'redshift'<br/>Stable Identity: redshift:{db}:{schema}:{table}:{pk}"] end subgraph Pipeline["Cognee Memory Pipeline (ingest.py)"] ADD["cognee.add(..., data_per_batch=...,<br/>write_disposition='append' | 'replace')"] COG["cognee.cognify()<br/>Knowledge Graph Extraction"] MEM["cognee.memify()<br/>Semantic & Relational Enrichment"] end CAT & EXT & OPS & DATA --> DAPI & DBAPI DAPI & DBAPI --> SCRUB SCRUB --> PROJ PROJ --> ADD --> COG --> MEM2. Package Structure (
packages/connector/redshift/)Aligned with
packages/connector/notion/,packages/connector/iceberg/, andpackages/connector/openlineage/:3. Concrete Component Specifications
A.
config.py— TypedRedshiftConfig(BaseModel)Supports both Redshift Data API (
transport="data_api", default, zero VPC tunnel needed) and Direct DB-API 2.0 (transport="direct", viaredshift-connectorover VPC/SSH tunnel) across Provisioned Clusters and Redshift Serverless Workgroups:B.
connector.py— Warehouse Metadata, Incremental Cursor & Secret Scrubbing (DOCUMENT_SOURCE_ATTR = "redshift")SVV_TABLE_INFO,SVV_ALL_TABLES,SVV_COLUMNS,PG_DESCRIPTION, andPG_CONSTRAINTto synthesize structured Markdown table cards containing:REMARKS(semantic business descriptions).diststyle(KEY,EVEN,ALL,AUTO),sortkey1,skew_rows,unsortedpercentage,pct_used,tbl_rows, andsize(MB).PG_CONSTRAINT.SVV_EXTERNAL_TABLESandSVV_EXTERNAL_COLUMNS.SYS_QUERY_HISTORY(recent DDL/DML transformations, execution duration, queue wait times) andSTL_LOAD_ERRORS(S3COPYingestion failures with sanitized error reasons) so Cognee can answer operational warehouse debugging questions.Parameters=[{"name": "cursor_watermark", "value": ...}](never string-interpolated) usingWHERE <cursor_column> >= :cursor_watermark ORDER BY <cursor_column> ASC, <primary_key> ASC.lookback_secondswindow for late-arrivingCOPY/ micro-batch loads and deduplicates boundary ties viacursor_tie_keys.ExecuteStatement-> exponential backoff polling onDescribeStatement(FINISHEDvsFAILED/ABORTED) -> fullGetStatementResultNextTokenpagination. The checkpoint advances only after all pages succeed.IAM_ROLE 'arn:aws:iam::...',aws_access_key_id=...,aws_secret_access_key=...,aws_session_token=...,SECRET_ARN, S3 pre-signed query strings, and JDBC/ODBC password parameters from all query texts, table DDL/comments, and load error records prior to document rendering.C.
ingest.py— Forget-on-Delete Graph Reconciliation (write_disposition="replace")redshift:{database}:{schema}:{table}:{pk}) and detects:SVV_ALL_TABLESor its schema/comment fingerprint changes.soft_delete_column(deleted_at/is_deleted) or runs a lightweight columnar key scan (SELECT <primary_key> FROM <schema>.<table>). Because Redshift is columnar, scanning a single sort/primary key column reads minimal blocks.ingest()switches the affected dataset sync towrite_disposition="replace"incognee.add(..., dataset_name=dataset_name, data_per_batch=data_per_batch, write_disposition="replace")before runningcognee.cognify()andcognee.memify(), ensuring deleted upstream records are purged from Cognee's graph and vector stores without risking false deletions on transient network errors.4. Offline Verification Matrix (
tests/test_redshift.py)All tests run 100% offline in CI using
botocore.stub.Stubberand mocked DB-API cursors (zero live AWS calls required):cluster_identifier+db_user), Serverless (workgroup_name),secret_arn, and directredshift-connectorvalidation[READY]ExecuteStatement->DescribeStatementpolling (SUBMITTED->PICKED->STARTED->FINISHED) +NextTokenmulti-page consumption[READY]DescribeStatementreturningFAILED/ABORTEDraisesRedshiftConnectorErrorand preserves existing cursor watermark & graph state[READY]SVV_TABLE_INFO(diststyle,sortkey1,skew_rows),SVV_COLUMNSremarks,PG_CONSTRAINTFKs, and SpectrumSVV_EXTERNAL_TABLES[READY]:cursor_watermarkfiltering,lookback_secondsoverlap, composite(cursor_column, primary_key)tie deduplication[READY]replace)previous_keys - current_keys), and table drop detection triggeringwrite_disposition="replace"[READY]IAM_ROLE,aws_access_key_id,SECRET_ARN, and S3COPYcredentials acrossSYS_QUERY_HISTORYand table comments[READY]ingest()verifyingcognee.add(DOCUMENT_SOURCE_ATTR = "redshift"),cognee.cognify(), andcognee.memify()invocation[READY]We are ready to open a clean, self-contained PR in
topoteretes/cognee-communitywith full documentation (including VPC SSH bastion tunneling and IAM policies) and a companion blog post as soon as this issue is assigned. Thank you!deveshsinghal09 commentedon Oct 12, 2026
Hi @NMZivkovic, I'd like to be considered for the Amazon Redshift connector for Mergetober.
I'd use AWS's official Python SDK and Redshift Data API to read selected query results and table comments into Cognee, following the existing community connector structure. The example would use a small sample dataset so someone can try searching their warehouse data and its descriptions.
I'll keep database access read-only, handle paginated results, and use a chosen timestamp column to pick up changes. Since that alone won't catch deleted rows, I'll also add a separate check for removals that only runs after a successful read. I'll add automated tests and validate against a real Redshift test environment before calling it complete, with connection instructions, a runnable example, and the usage blog.
If this is still available, could you assign it to me? I see the earlier requests and will wait for your decision. My other integration requests, including Freshdesk, are alternatives; I'd focus on one assignment at a time. Thanks!
shanaldo7 commentedon Oct 13, 2026
Hi @NMZivkovic,
I’d like to contribute to #4773 — Add Amazon Redshift data-source connector as part of Mergetober.
I plan to build the connector around Cognee’s existing integration conventions, with particular attention to reliable incremental synchronization and safe handling of source deletions.
Implementation plan:
packages/connector/notion/and the integration guide first, then mirror the expected package structure and ingestion lifecycle inpackages/connector/redshift/.README.md,examples/, tests, andpyproject.toml, including setup instructions, a runnable example, required permissions, and network-access limitations.Before finalizing the design, I’ll verify how Cognee currently handles source identifiers, cursor persistence, and graph cleanup so the connector follows established behavior instead of introducing incompatible mechanisms.
Could you please confirm whether this issue is still available and consider assigning it to me? I’ll wait for assignment before starting implementation.
Thank you!
dexters1 commentedon Oct 13, 2026
Issue assigned to @Akshats-git