Skip to content

Add a config to disable CDC store for ClickHouse#4438

Merged
ilidemi merged 7 commits into
mainfrom
cdc-store-maybe-cleaner
Jun 17, 2026
Merged

Add a config to disable CDC store for ClickHouse#4438
ilidemi merged 7 commits into
mainfrom
cdc-store-maybe-cleaner

Conversation

@ilidemi

@ilidemi ilidemi commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Building on #4366

  • Introduce PEERDB_CLICKHOUSE_CDC_STORE_ENABLED to allow separating handling of cdc store enablement for ClickHouse (since other destinations like PG-to-PG and PG-to-Snowflake requires this to be enabled).
  • Still honor existing services with CDC store disabled, to avoid OOM issues resurfacing
  • Currently PEERDB_CLICKHOUSE_CDC_STORE_ENABLED is default to true, can be updated to false once we are ready to flip the switch (PR is currently a no-op)

@ilidemi
ilidemi requested a review from a team as a code owner June 16, 2026 21:36
@ilidemi ilidemi mentioned this pull request Jun 16, 2026
Closed
@claude

claude Bot commented Jun 16, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

1 similar comment
@claude

claude Bot commented Jun 16, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@codecov

codecov Bot commented Jun 16, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
2277 2 2275 213
View the top 1 failed test(s) by shortest run time 
github.com/PeerDB-io/peerdb/flow/e2e::TestGenericBQ/Test_Inheritance_Table_With_Dynamic_Setting
Stack Traces | 64.3s run time 
=== RUN   TestGenericBQ/Test_Inheritance_Table_With_Dynamic_Setting
=== PAUSE TestGenericBQ/Test_Inheritance_Table_With_Dynamic_Setting
=== CONT  TestGenericBQ/Test_Inheritance_Table_With_Dynamic_Setting
2026/06/16 21:40:24 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,good_column,lost_column FROM e2e_test_pgchclg_jc8mzf8d.\"test_lost_column_bug\" ORDER BY id"
2026/06/16 21:40:24 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,good_column,lost_column FROM e2e_test_pgchclg_jc8mzf8d.\"test_lost_column_bug\" ORDER BY id"
2026/06/16 21:40:24 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_500045438243934377 CURSOR FOR SELECT id,good_column,lost_column FROM e2e_test_pgchclg_jc8mzf8d.\"test_lost_column_bug\" ORDER BY id" args=[]
2026/06/16 21:40:24 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,good_column,lost_column FROM e2e_test_pgchclg_jc8mzf8d.\"test_lost_column_bug\" ORDER BY id" channelLen=0
2026/06/16 21:40:24 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_500045438243934377
2026/06/16 21:40:24 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_500045438243934377 records=2 bytes=23 channelLen=1
2026/06/16 21:40:24 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,good_column,lost_column FROM e2e_test_pgchclg_jc8mzf8d.\"test_lost_column_bug\" ORDER BY id" rows=2 bytes=23 channelLen=1
2026/06/16 21:40:24 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_500045438243934377
2026/06/16 21:40:24 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_500045438243934377 records=0 bytes=0 channelLen=0
2026/06/16 21:40:24 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,good_column,lost_column FROM e2e_test_pgchclg_jc8mzf8d.\"test_lost_column_bug\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/06/16 21:40:24 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/06/16 21:40:24 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,good_column,lost_column FROM e2e_test_pgchclg_jc8mzf8d.\"test_lost_column_bug\" ORDER BY id" rows=2 bytes=23 channelLen=0
    generic_test.go:1101: UNEXPECTED STATUS TIMEOUT STATUS_SNAPSHOT
    bigquery.go:86: begin tearing down postgres schema bq_oa7vohzn_20260616214024
2026/06/16 21:41:28 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychclg_oufeldoo.test_simple_schema_changes
--- FAIL: TestGenericBQ/Test_Inheritance_Table_With_Dynamic_Setting (64.31s)
View the full list of 1 ❄️ flaky test(s) 
github.com/PeerDB-io/peerdb/flow/e2e::TestGenericBQ

Flake rate in main: 22.86% (Passed 54 times, Failed 16 times)

Stack Traces | 0s run time 
=== RUN   TestGenericBQ
=== PAUSE TestGenericBQ
=== CONT  TestGenericBQ
--- FAIL: TestGenericBQ (0.00s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@github-actions

github-actions Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: The single failure, TestGenericBQ/Test_Inheritance_Table_With_Dynamic_Setting, timed out waiting for the BigQuery mirror to leave STATUS_SNAPSHOT — a timeout against an external service in a suite the code itself flags as flaky under high concurrency, failing in only 1 of 2490 tests on a single matrix shard.
Confidence: 0.9

✅ Automatically retrying the workflow

View workflow run

@ilidemi
ilidemi merged commit b5c24ef into main Jun 17, 2026
16 checks passed
@ilidemi
ilidemi deleted the cdc-store-maybe-cleaner branch June 17, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jgao54 jgao54 jgao54 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ilidemi @jgao54