bug: CDC Date clamping consistent with initial load limits by pfcoperez · Pull Request #4600 · PeerDB-io/peerdb

pfcoperez · 2026-07-20T15:10:16Z

In #4534 we added logic to proactively clamp dates to CH Date32 limits at the time.

However, the scope of that change didn't include CDC. CDC normalization was therefore constrained by ClickHouse's parseDateTime64BestEffortOrNull limits.

These limits changed with with ClickHouse/ClickHouse#107907 leaving initial loads and cdc supported date ranges inconsistent with each other.

This PR aligns CDC to use the same limits as initial load.

From now this is a constant to [1900-01-01, 2299-12-31] but might want to follow up with a piece of functionality probing destination CH valid ranges to make the most of the target CH version capabilities.

codecov · 2026-07-20T15:28:24Z

❌ 8 Tests Failed:

Tests completed	Failed	Passed	Skipped
3033	8	3025	377

View the top 3 failed test(s) by shortest run time

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH

Stack Traces | 0.01s run time

=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster

Stack Traces | 0.01s run time

=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster
2026/07/20 15:13:25 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH

Stack Traces | 0.01s run time

=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH
2026/07/20 15:13:25 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH_Cluster

Stack Traces | 0.01s run time

=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH_Cluster (0.01s)
2026/07/20 15:26:10 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/20 15:26:10 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/20 15:26:10 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_api_s7ddtuvc.t1

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range

Stack Traces | 20.1s run time

=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range
2026/07/20 15:27:17 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/20 15:27:17 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
    clickhouse_mysql_test.go:2139: WaitFor waiting on snapshot 2026-07-20 15:27:21.78304777 +0000 UTC m=+836.206518636
    clickhouse_mysql_test.go:2143: WaitFor waiting on cdc 2026-07-20 15:27:21.787746356 +0000 UTC m=+836.211217212
    clickhouse_mysql_test.go:2160: 
        	Error Trace:	.../flow/e2e/clickhouse_mysql_test.go:2160
        	            				.../flow/e2e/clickhouse_mysql_test.go:2172
        	            				.../hostedtoolcache/go/1.26.5.../src/runtime/asm_amd64.s:1771
        	Error:      	Not equal: 
        	            	expected: "1900-01-01 05:30:45.000000"
        	            	actual  : "1000-06-15 05:30:45.000000"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-1900-01-01 05:30:45.000000
        	            	+1000-06-15 05:30:45.000000
        	Test:       	TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range (20.13s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range

Stack Traces | 20.2s run time

=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
2026/07/20 15:19:23 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/20 15:19:23 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/20 15:19:23 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_machcl_tmmk739b.test_mysql_ghost_schema
    clickhouse_mysql_test.go:2139: WaitFor waiting on snapshot 2026-07-20 15:19:28.880576877 +0000 UTC m=+363.304047733
    clickhouse_mysql_test.go:2143: WaitFor waiting on cdc 2026-07-20 15:19:28.886969784 +0000 UTC m=+363.310440641
2026/07/20 15:19:28 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/20 15:19:28 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_machcl_tmmk739b.test_mysql_ghost_schema
    clickhouse_mysql_test.go:2160: 
        	Error Trace:	.../flow/e2e/clickhouse_mysql_test.go:2160
        	            				.../flow/e2e/clickhouse_mysql_test.go:2172
        	            				.../hostedtoolcache/go/1.26.5.../src/runtime/asm_amd64.s:1771
        	Error:      	Not equal: 
        	            	expected: "1900-01-01 05:30:45.000000"
        	            	actual  : "1000-06-15 05:30:45.000000"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-1900-01-01 05:30:45.000000
        	            	+1000-06-15 05:30:45.000000
        	Test:       	TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range (20.24s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range

Stack Traces | 20.3s run time

=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
2026/07/20 15:24:53 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/20 15:24:53 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
    clickhouse_mysql_test.go:2139: WaitFor waiting on snapshot 2026-07-20 15:24:58.542636469 +0000 UTC m=+692.966107325
    clickhouse_mysql_test.go:2143: WaitFor waiting on cdc 2026-07-20 15:24:58.55027006 +0000 UTC m=+692.973740926
    clickhouse_mysql_test.go:2160: 
        	Error Trace:	.../flow/e2e/clickhouse_mysql_test.go:2160
        	            				.../flow/e2e/clickhouse_mysql_test.go:2172
        	            				.../hostedtoolcache/go/1.26.5.../src/runtime/asm_amd64.s:1771
        	Error:      	Not equal: 
        	            	expected: "1900-01-01 05:30:45.000000"
        	            	actual  : "1000-06-15 05:30:45.000000"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-1900-01-01 05:30:45.000000
        	            	+1000-06-15 05:30:45.000000
        	Test:       	TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range (20.26s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range

Stack Traces | 20.3s run time

=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range
2026/07/20 15:21:54 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/20 15:21:54 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/20 15:21:54 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mych_ayempky1.test_exclude_ch
    clickhouse_mysql_test.go:2139: WaitFor waiting on snapshot 2026-07-20 15:21:58.190251255 +0000 UTC m=+512.613722121
    clickhouse_mysql_test.go:2143: WaitFor waiting on cdc 2026-07-20 15:21:58.19516076 +0000 UTC m=+512.618631626
2026/07/20 15:21:58 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mych_ayempky1.test_exclude_ch
    clickhouse_mysql_test.go:2160: 
        	Error Trace:	.../flow/e2e/clickhouse_mysql_test.go:2160
        	            				.../flow/e2e/clickhouse_mysql_test.go:2172
        	            				.../hostedtoolcache/go/1.26.5.../src/runtime/asm_amd64.s:1771
        	Error:      	Not equal: 
        	            	expected: "1900-01-01 05:30:45.000000"
        	            	actual  : "1000-06-15 05:30:45.000000"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-1900-01-01 05:30:45.000000
        	            	+1000-06-15 05:30:45.000000
        	Test:       	TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range
2026/07/20 15:22:14 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mych_ekw34gll.test_bit
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range (20.33s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-07-20T15:32:03Z

❌ Test Failure

Analysis: A real bug: the cdc/date-clamping PR fails Test_MySQL_DateTime_ClickHouse_Range deterministically across all 4 ClickHouse suite variants because out-of-range MySQL DATETIME values aren't clamped to the 1900-01-01 boundary (got raw 1000-06-15), indicating incomplete DATETIME64 clamping logic.
Confidence: 0.98

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

…Null` is used for normalization.

claude · 2026-07-20T15:47:59Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

…global strings

pfcoperez added 2 commits July 20, 2026 16:58

Make clamp limits public

fae3a94

Clamp Date32 to stable ClickHouse stable limits

837200c

pfcoperez added 2 commits July 20, 2026 17:37

Add comments on date clamping limits

422dfbe

Clamp CDC dates in DateTime64(6) where `parseDateTime64BestEffortOr…

7379b7f

…Null` is used for normalization.

pfcoperez marked this pull request as ready for review July 20, 2026 15:42

pfcoperez requested a review from a team as a code owner July 20, 2026 15:42

pfcoperez enabled auto-merge (squash) July 20, 2026 17:21

pfcoperez disabled auto-merge July 20, 2026 17:21

pfcoperez enabled auto-merge (squash) July 20, 2026 17:21

serprex reviewed Jul 20, 2026

View reviewed changes

Comment thread flow/connectors/clickhouse/normalize_query.go Outdated

PR Comments: Performance improvement by moving function constants to …

09ca7ae

…global strings

pfcoperez force-pushed the cdc/date-clamping branch from 39f6d63 to 09ca7ae Compare July 20, 2026 20:50

ilidemi approved these changes Jul 21, 2026

View reviewed changes

pfcoperez merged commit a2be15c into main Jul 21, 2026
18 checks passed

pfcoperez deleted the cdc/date-clamping branch July 21, 2026 05:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

bug: CDC Date clamping consistent with initial load limits#4600

bug: CDC Date clamping consistent with initial load limits#4600
pfcoperez merged 5 commits into
mainfrom
cdc/date-clamping

pfcoperez commented Jul 20, 2026

Uh oh!

codecov Bot commented Jul 20, 2026

Uh oh!

github-actions Bot commented Jul 20, 2026

Uh oh!

claude Bot commented Jul 20, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

pfcoperez commented Jul 20, 2026

Uh oh!

codecov Bot commented Jul 20, 2026

❌ 8 Tests Failed:

Uh oh!

github-actions Bot commented Jul 20, 2026

❌ Test Failure

Uh oh!

claude Bot commented Jul 20, 2026

Code review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants