Workflow [PR], commit [8335bbc]

Summary: ⏳

I've discovered that a similar problem may occur with preempted threads. For example, a query might have 6 threads: 2 of which are preempted, waiting for 1 1-second preemption timeout to downscale or a brand new granted slot to continue working. In the meantime other 4 threads do the rest of the the work an finish by going to idle state. This creates a hanging query waiting for its preempted threads.

In this PR, I will add the logic to properly wake and shut down preempted threads when the query finishes, and update the pipeline shutdown condition to take preempted threads into account.

There are too many failed tests, but all seem unrelated. Let's rerun one more time.

All failed tests are unrelated:

02177_issue_31009

#86335

02443_detach_attach_partition

Infamously known to be flaky

test_storage_kafka_sasl/test.py::test_kafka_sasl

flaky

test_storage_s3_queue/test_4.py::test_list_and_delete_race

#86506

Okay, I think I can do one with a reduced preemption timeout to trigger the issue more readily

Sanitizer have found related issue. Investigating...

UPD. I was not careful enough and introduced the following data race. TaskQueue itself is not thread-safe and relies on external synchronization (the ExecutorTasks::mutex). In ExecutorTasks::preempt(), it acquires the mutex. If the preempted thread had a local task, it pushes that task back into task_queue and calls tryWakeUpAnyOtherThreadWithTasks(). That helper may call lock.unlock() internally (to wake a thread outside the critical section). Immediately after returning, preempt() checks task_queue.empty() without re-acquiring the lock. That read races with other threads popping tasks (which hold the mutex), exactly as TSan reports. This is fixed in 8335bbc