Affected test:

• /swarms/feature/node failure/check restart clickhouse on swarm node

Affected files:

• swarms/tests/node_failure.py
• swarms/tests/steps/swarm_node_actions.py

Description

The check restart clickhouse on swarm node test fails consistently on ClickHouse Antalya 26.1 (100% fail rate) while passing on Antalya 25.8. The test verifies that a swarm cluster query fails with exitcode=32 (Attempt to read after eof) when a ClickHouse process is killed on a swarm node during query execution.

On 26.1, the query either succeeds when it shouldn't (with SEGV signal) or hangs indefinitely (with KILL signal), instead of failing quickly with an EOF error as it does on 25.8.

Analysis

Test behavior by version and signal

Two separate issues

1. SEGV on 26.1: query survives node failure (AssertionError) — historical, test already updated

With SEGV, the ClickHouse process takes time to die (core dump generation). During this window, the TaskDistributor detects the node going down and redistributes its pending tasks to the surviving node. The query completes successfully with only one node's results. The test code was updated in commit f1827080b to use KILL instead of SEGV, so this failure mode will no longer occur once the regression commit hash is updated on the main branch.

2. KILL on 26.1: query hangs instead of failing (ExpectTimeoutError) — potential ClickHouse bug

With KILL, the process dies instantly (no cleanup). On 25.8, this causes an immediate Attempt to read after eof error (~5s). On 26.1, the initiator does not detect the broken connection and the query hangs for the full 600s bash timeout. This is a behavioral regression in 26.1.

What changed in 26.1

All Altinity swarm-specific PRs (#780, #866, #1014, #1042, #1201, etc.) are labeled antalya-25.8 or earlier — the swarm code is already present in 25.8 where the test passes. The PRs #1395 and #1414 that frontported this code to 26.1 contain only code that was already in 25.8.

The difference is in the upstream ClickHouse base code (26.1 vs. ~25.5). Key files that differ between the two branches:

• src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp — new parallel_replicas_for_cluster_engines, cluster_table_function_split_granularity, changed distributed_processing derivation
• src/Storages/ObjectStorage/StorageObjectStorageStableTaskDistributor.cpp — removed has_concurrent_next() check, reordered handleReplicaLoss() operations, changed file identifier resolution
• src/Storages/IStorageCluster.cpp — metadata handling, join detection changes
• src/Storages/ObjectStorage/StorageObjectStorageSource.cpp/.h — structural changes

Core connection handling code is identical between versions:

• RemoteQueryExecutor.cpp, MultiplexedConnections.cpp, ReadBufferFromPocoSocket.cpp, PacketReceiver.cpp, ConnectionTimeouts.cpp/h

Database evidence

-- 26.1: 100% failure rate
SELECT result, count() FROM `gh-data`.clickhouse_regression_results
WHERE test_name = '/swarms/feature/node failure/check restart clickhouse on swarm node'
  AND clickhouse_version LIKE '26.1%'
GROUP BY result
-- Fail: ~25, Error: ~8

-- 25.8: passes consistently (flaky rare failure)
SELECT result, count() FROM `gh-data`.clickhouse_regression_results
WHERE test_name = '/swarms/feature/node failure/check restart clickhouse on swarm node'
  AND clickhouse_version LIKE '25.8%'
GROUP BY result
-- OK: ~15, Fail: 1

Commit context

The test code was updated in commit f1827080b (2026-03-10, "Update node failure tests to use kill instead segv") to use signal="KILL", delay=30, delay_before_execution=5. The main branch antalya-26.1 still uses the old regression commit a54216bbc which has signal="SEGV", delay=0. PR 1520 is updating the regression commit hash for antalya-26.1, and the failures with KILL (ExpectTimeoutError) can be seen in that PR's CI runs.

Next Steps

• Validate manually that KILL on 26.1 causes a hang (reproduce the ExpectTimeoutError scenario)
• Investigate why the upstream 26.1 base code doesn't propagate EOF from a killed node's connection
• Potentially file a bug in Altinity/ClickHouse if the KILL hang is confirmed as a regression

References

• Failing run: https://github.com/Altinity/ClickHouse/actions/runs/23546722566/job/68549375510
• Commit that changed test signal from SEGV to KILL: f1827080b