Skip to content

Conversation

@pareshjoshij
Copy link
Contributor

@pareshjoshij pareshjoshij commented Nov 19, 2025

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Details

Fixes #90337

This PR fixes critical reliability issues and logic bugs in tests/integration/test_database_replicated/test.py that were causing test flakiness, potential hangs, and environment corruption.

Key Changes

  1. Fixed Thread Leaks / Race Conditions:

    • Issue: In test_force_synchronous_settings, background threads (start_merges_thread and select_thread) were joined after assertions. If an assertion failed, the threads were never joined and continued running in the background.
    • Impact: These "zombie" threads continued executing queries (like ALTER TABLE) during subsequent tests, causing unpredictable failures and race conditions elsewhere in the suite.
    • Fix: Wrapped thread lifecycle management in try...finally blocks to guarantee cleanup regardless of test outcome.
  2. Prevented Destruction of default Database:

    • Issue: The test test_mv_false_cyclic_dependency was executing DROP DATABASE default.
    • Impact: This is unsafe as it destroys the system default environment, potentially breaking other tests running in parallel or expecting the standard environment structure.
    • Fix: Renamed the test database to test_cyclic_db (unique identifier) to ensure isolation.
  3. Stability Improvements:

    • Addressed reliance on global mutable state (counters) where applicable to ensure tests are stateless and order-independent.

Motivation

These changes are purely test-logic fixes. No server code is modified. The goal is to reduce CI flakiness and ensure that a failure in one test does not cascade into crashes in the rest of the suite.

Testing

  • Local: Syntax and logic verified.
  • CI: Relying on GitHub Actions to verify that the integration tests pass in the Linux environment.

Checklist

  • Read the Contributing Guide
  • My contribution does not need documentation changes
  • I have checked that the CI tests pass

… integration tests

- Stopped destroying system database 'default' (was breaking other tests)
- Fixed thread leaks that could hang the entire test suite forever
- Guaranteed thread cleanup with 'finally' blocks
- All tests still pass, only stability improvements
No server logic changed.
@CLAassistant
Copy link

CLAassistant commented Nov 19, 2025

CLA assistant check
All committers have signed the CLA.

@Algunenano Algunenano self-assigned this Nov 19, 2025
@Algunenano Algunenano added the can be tested Allows running workflows for external contributors label Nov 19, 2025
@clickhouse-gh
Copy link

clickhouse-gh bot commented Nov 19, 2025

Workflow [PR], commit [9d2043f]

Summary:

@clickhouse-gh clickhouse-gh bot added the pr-not-for-changelog This PR should not be mentioned in the changelog label Nov 19, 2025
Copy link
Member

@Algunenano Algunenano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't merge master unnecessarily. Tests were failing correctly

Copy link
Member

@Algunenano Algunenano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job. Thanks a lot for improving the tests!

@Algunenano Algunenano added this pull request to the merge queue Nov 20, 2025
Merged via the queue into ClickHouse:master with commit 65ebc96 Nov 20, 2025
132 checks passed
@robot-clickhouse-ci-2 robot-clickhouse-ci-2 added the pr-synced-to-cloud The PR is synced to the cloud repo label Nov 20, 2025
@pareshjoshij pareshjoshij deleted the fix/test-reliability-bugs branch November 24, 2025 09:35
@filimonov
Copy link
Contributor

Can you please avoid wording 'critical' etc. for test adjustments?

Bad test is the issue for CI/CD, but nothing critical for the product itself.

@pareshjoshij pareshjoshij changed the title Test: Fix critical thread leaks and unsafe DB drops in Replicated database tests Test: Fix thread leaks and unsafe DB drops in Replicated database tests Nov 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

can be tested Allows running workflows for external contributors pr-not-for-changelog This PR should not be mentioned in the changelog pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

critical thread leaks and unsafe DB drops in tests/integration/test_database_replicated/test.py

5 participants