Skip to content

Fix zombie snapshot flows after the process restarts; fix a rare conn leak#4603

Merged
ilidemi merged 2 commits into
mainfrom
snapshot-flow-session-cleanup
Jul 21, 2026
Merged

Fix zombie snapshot flows after the process restarts; fix a rare conn leak#4603
ilidemi merged 2 commits into
mainfrom
snapshot-flow-session-cleanup

Conversation

@ilidemi

@ilidemi ilidemi commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

SnapshotFlowWorkflow holds a Temporal session for the entire workflow duration. It can receive a context cancellation in two cases: legit cancel from the outside and a k8s-initiated process restart. In both cases it runs a deferred cleanup that closes PG connection (important for the legit cancel). But after the process restart it doesn't own the Temporal session anymore, so CloseSlotKeepAlive can't execute (or retry), which is fine for PG as connection was closed anyway but Temporal patitently waits till a session-holding worker is available. Add ScheduleToCloseTimeout that would reap the activity Temporal-side.

Also add the same defer cleanup pattern for cdc-only mirrors as for snapshot - very minor as it's two short activities back to back but AI flagged it so it must be done.

Closes DBI-936

@ilidemi
ilidemi requested a review from a team as a code owner July 21, 2026 05:13
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

❌ 8 Tests Failed:

Tests completed Failed Passed Skipped
3033 8 3025 377
View the top 3 failed test(s) by shortest run time 
github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH
Stack Traces | 0.01s run time 
=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster
Stack Traces | 0.01s run time 
=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster (0.01s)
2026/07/21 05:31:07 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 05:31:07 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 05:31:08 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_gckkwglh.test_schema_as_column

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH
Stack Traces | 0.01s run time 
=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
Stack Traces | 0.01s run time 
=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH_Cluster
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH_Cluster (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range
Stack Traces | 19.1s run time 
=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range
2026/07/21 05:31:44 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 05:31:44 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 05:31:44 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_fo8stnb1.test_bit
    clickhouse_mysql_test.go:2139: WaitFor waiting on snapshot 2026-07-21 05:31:48.580373366 +0000 UTC m=+773.424174870
    clickhouse_mysql_test.go:2143: WaitFor waiting on cdc 2026-07-21 05:31:48.583898588 +0000 UTC m=+773.427700091
2026/07/21 05:31:48 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_c0p8pger.test_date_coercion
    clickhouse_mysql_test.go:2160: 
        	Error Trace:	.../flow/e2e/clickhouse_mysql_test.go:2160
        	            				.../flow/e2e/clickhouse_mysql_test.go:2166
        	            				.../hostedtoolcache/go/1.26.5.../src/runtime/asm_amd64.s:1771
        	Error:      	Not equal: 
        	            	expected: "1900-01-01"
        	            	actual  : "1000-06-15"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-1900-01-01
        	            	+1000-06-15
        	Test:       	TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MySQL_DateTime_ClickHouse_Range (19.11s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
Stack Traces | 19.2s run time 
=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
2026/07/21 05:30:20 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 05:30:20 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 05:30:20 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 05:30:20 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_machcl_jkx4ji8v.test_nullengine
    clickhouse_mysql_test.go:2139: WaitFor waiting on snapshot 2026-07-21 05:30:24.284875867 +0000 UTC m=+689.128677373
    clickhouse_mysql_test.go:2143: WaitFor waiting on cdc 2026-07-21 05:30:24.291484894 +0000 UTC m=+689.135286400
2026/07/21 05:30:24 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_machcl_jkx4ji8v.test_nullengine
    clickhouse_mysql_test.go:2160: 
        	Error Trace:	.../flow/e2e/clickhouse_mysql_test.go:2160
        	            				.../flow/e2e/clickhouse_mysql_test.go:2166
        	            				.../hostedtoolcache/go/1.26.5.../src/runtime/asm_amd64.s:1771
        	Error:      	Not equal: 
        	            	expected: "1900-01-01"
        	            	actual  : "1000-06-15"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-1900-01-01
        	            	+1000-06-15
        	Test:       	TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range (19.24s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range
Stack Traces | 20.2s run time 
=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range
2026/07/21 05:28:05 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 05:28:05 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
    clickhouse_mysql_test.go:2139: WaitFor waiting on snapshot 2026-07-21 05:28:09.871014852 +0000 UTC m=+554.714816359
    clickhouse_mysql_test.go:2143: WaitFor waiting on cdc 2026-07-21 05:28:09.875085492 +0000 UTC m=+554.718887006
2026/07/21 05:28:09 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mych_jgtx44xl.test_mysql_schema_changes
    clickhouse_mysql_test.go:2160: 
        	Error Trace:	.../flow/e2e/clickhouse_mysql_test.go:2160
        	            				.../flow/e2e/clickhouse_mysql_test.go:2166
        	            				.../hostedtoolcache/go/1.26.5.../src/runtime/asm_amd64.s:1771
        	Error:      	Not equal: 
        	            	expected: "1900-01-01"
        	            	actual  : "1000-06-15"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-1900-01-01
        	            	+1000-06-15
        	Test:       	TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_DateTime_ClickHouse_Range (20.19s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
Stack Traces | 20.2s run time 
=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
2026/07/21 05:26:55 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 05:26:55 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 05:26:56 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_huaqhx4k.test_nullengine
    clickhouse_mysql_test.go:2139: WaitFor waiting on snapshot 2026-07-21 05:27:00.956876363 +0000 UTC m=+485.800677858
    clickhouse_mysql_test.go:2143: WaitFor waiting on cdc 2026-07-21 05:27:00.963007162 +0000 UTC m=+485.806808669
2026/07/21 05:27:01 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_q2gxqhry.test_float
2026/07/21 05:27:01 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_huaqhx4k.test_nullengine
    clickhouse_mysql_test.go:2160: 
        	Error Trace:	.../flow/e2e/clickhouse_mysql_test.go:2160
        	            				.../flow/e2e/clickhouse_mysql_test.go:2166
        	            				.../hostedtoolcache/go/1.26.5.../src/runtime/asm_amd64.s:1771
        	Error:      	Not equal: 
        	            	expected: "1900-01-01"
        	            	actual  : "1000-06-15"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-1900-01-01
        	            	+1000-06-15
        	Test:       	TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH_Cluster/Test_MySQL_DateTime_ClickHouse_Range (20.23s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

❌ Test Failure

Analysis: All 8 failures are the same deterministic assertion in Test_MySQL_DateTime_ClickHouse_Range (expected "1900-01-01", got "1000-06-15") that fails only on the ClickHouse 8.0/latest leg because the newest ClickHouse widened its Date32 lower bound below the test's hardcoded clamp boundary — a real, reproducible version-driven breakage, not a flake.
Confidence: 0.88

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@ilidemi
ilidemi merged commit 279caf8 into main Jul 21, 2026
18 checks passed
@ilidemi
ilidemi deleted the snapshot-flow-session-cleanup branch July 21, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@masterashu masterashu masterashu approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ilidemi @masterashu