Skip to content

critical thread leaks and unsafe DB drops in tests/integration/test_database_replicated/test.py #90337

@pareshjoshij

Description

@pareshjoshij

Company or project name

Individual Contributor (Open Source)

Describe what's wrong

I identified three critical reliability issues in tests/integration/test_database_replicated/test.py that cause test flakiness and environment corruption:

  1. Thread Leakage / Race Condition: In test_force_synchronous_settings, background threads (start_merges_thread and select_thread) are started, but their .join() calls are placed after assertions. If an assertion fails, the test aborts, leaving the threads running. These "zombie" threads continue executing queries against the database while subsequent tests run, leading to race conditions and random failures in the rest of the suite.

  2. Unsafe Environment Modification: The test test_mv_false_cyclic_dependency explicitly executes DROP DATABASE default. This destroys the system default database, which violates test isolation and poisons the environment for other tests running in parallel or subsequently.

  3. Global Mutable State: The test relies on a global variable test_recover_staled_replica_run to track state between runs, which makes the test order-dependent and prevents it from being run multiple times reliably.

Does it reproduce on the most recent release?

Yes

How to reproduce

To observe the logic defects, examine tests/integration/test_database_replicated/test.py.

To simulate the crash/hang:

  1. Modify test_force_synchronous_settings to force an assertion failure before the .join() call.
  2. Run the test suite using pytest.
  3. Observe that the background threads are not cleaned up, potentially causing the next test to fail or the suite to hang.

Command:
pytest tests/integration/test_database_replicated/test.py

Expected behavior

No response

Error message and/or stacktrace

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    potential bugTo be reviewed by developers and confirmed/rejected.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions