Avoid `Cannot schedule a task` error when loading parts by kssenii · Pull Request #70257 · ClickHouse/ClickHouse

kssenii · 2024-10-02T08:49:02Z

Fixes https://s3.amazonaws.com/clickhouse-test-reports/70114/7835b66565b481c61e535d2baee362736d2b750a/stress_test__debug_/fatal_messages.txt

/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972445 [ 40816 ] {} <Fatal> BaseDaemon: (version 24.10.1.857, build id: A9203E66AFE91D0AF06E8CA6A1DFF6F0CD0587CE, git hash: d106f46b94d8e82f7c32fc8f6340df57a3f64c39) (from thread 40868) Terminate called for uncaught exception:
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972472 [ 40816 ] {} <Fatal> BaseDaemon: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=24, jobs=0). (CANNOT_SCHEDULE_TASK), Stack trace (when copying this message, always include the lines below):
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972525 [ 40816 ] {} <Fatal> BaseDaemon: 
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972536 [ 40816 ] {} <Fatal> BaseDaemon: 0. /build/contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000167b9dd2
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972546 [ 40816 ] {} <Fatal> BaseDaemon: 1. /build/src/Common/Exception.cpp:109: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c3b9e79
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972556 [ 40816 ] {} <Fatal> BaseDaemon: 2. /build/src/Common/Exception.h:110: DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000070bf16c
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972572 [ 40816 ] {} <Fatal> BaseDaemon: 3. /build/src/Common/Exception.h:128: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x000000000c4721ab
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972584 [ 40816 ] {} <Fatal> BaseDaemon: 4. /build/src/Common/ThreadPool.cpp:202: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x000000000c4741d3
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972593 [ 40816 ] {} <Fatal> BaseDaemon: 5. /build/src/Common/ThreadPool.cpp:218: void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x000000000c46e746
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972604 [ 40816 ] {} <Fatal> BaseDaemon: 6. /build/src/Common/ThreadPool.cpp:325: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::scheduleOrThrowOnError(std::function<void ()>, Priority) @ 0x000000000c46e62e
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972617 [ 40816 ] {} <Fatal> BaseDaemon: 7. /build/src/Common/threadPoolCallbackRunner.h:59: std::function<std::future<DB::IAsynchronousReader::Result> (std::function<DB::IAsynchronousReader::Result ()>&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, std::function<DB::IAsynchronousReader::Result ()>>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>&, String const&)::'lambda'(std::function<DB::IAsynchronousReader::Result ()>&&, Priority)::operator()(std::function<DB::IAsynchronousReader::Result ()>&&, Priority) @ 0x00000000104a129a
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972628 [ 40816 ] {} <Fatal> BaseDaemon: 8. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? @ 0x00000000104a108f
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972635 [ 40816 ] {} <Fatal> BaseDaemon: 9. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x00000000104a02b0
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972648 [ 40816 ] {} <Fatal> BaseDaemon: 10. /build/src/IO/AsynchronousReadBufferFromFileDescriptor.cpp:62: DB::AsynchronousReadBufferFromFileDescriptor::asyncReadInto(char*, unsigned long, Priority) @ 0x0000000010497ee2
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972656 [ 40816 ] {} <Fatal> BaseDaemon: 11. /build/src/IO/AsynchronousReadBufferFromFileDescriptor.cpp:100: DB::AsynchronousReadBufferFromFileDescriptor::nextImpl() @ 0x00000000104981cb
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972664 [ 40816 ] {} <Fatal> BaseDaemon: 12. /build/src/IO/ReadBuffer.h:70: DB::ReadBuffer::next() @ 0x00000000078dfffb
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972672 [ 40816 ] {} <Fatal> BaseDaemon: 13. /build/src/IO/ReadBuffer.h:106: void DB::readEscapedStringUntilEOLInto<String>(String&, DB::ReadBuffer&) @ 0x000000000c42c38e
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972681 [ 40816 ] {} <Fatal> BaseDaemon: 14. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:1016: DB::IMergeTreeDataPart::loadDefaultCompressionCodec() @ 0x0000000012998873
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972688 [ 40816 ] {} <Fatal> BaseDaemon: 15. /build/src/Storages/MergeTree/IMergeTreeDataPart.cpp:738: DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x0000000012993795
/var/log/clickhouse-server/clickhouse-server.final.log:2024.09.30 14:13:32.972701 [ 40816 ] {} <Fatal> BaseDaemon: 16. /build/src/Storages/MergeTree/MergeTreeData.cpp:1472: DB::MergeTreeData::loadDataPart(DB::MergeTreePartInfo const&, String const&, std::shared_ptr<DB::IDisk> const&, DB::MergeTreeDataPartState, std::mutex&) @ 0x0000000012a41d7f

It started reproducing because of #65625

Changelog category (leave one):

Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

...

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

Allow: All Required Checks
Allow: Stateless tests
Allow: Stateful tests
Allow: Integration Tests
Allow: Performance tests
Allow: All Builds
Allow: batch 1, 2 for multi-batch jobs
Allow: batch 3, 4, 5, 6 for multi-batch jobs

Exclude: Style check
Exclude: Fast test
Exclude: All with ASAN
Exclude: All with TSAN, MSAN, UBSAN, Coverage
Exclude: All with aarch64, release, debug

Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
Exclude: AST fuzzers

robot-clickhouse · 2024-10-02T08:51:16Z

This is an automated comment for commit 45cff04 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	❌ failure

Successful checks

Check name	Description	Status
AST fuzzer	Runs randomly generated queries to catch program errors. The build type is optionally given in parenthesis. If it fails, ask a maintainer for help	✅ success
Builds	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
ClickBench	Runs [ClickBench](https://github.com/ClickHouse/ClickBench/) with instant-attach table	✅ success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	✅ success
Docker keeper image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docker server image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docs check	Builds and tests the documentation	✅ success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	✅ success
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
Performance Comparison	Measure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests	✅ success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success
Unit tests	Runs the unit tests for different release types	✅ success
Upgrade check	Runs stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts	✅ success

kssenii · 2024-10-02T08:53:38Z

src/Storages/MergeTree/PartMetadataManagerOrdinary.cpp

-    auto res = part->getDataPartStorage().readFile(file_name, getReadSettings().adjustBufferSize(file_size), file_size, std::nullopt);
+    auto read_settings = getReadSettings().adjustBufferSize(file_size);
+    /// Default read method is pread_threadpool, but there is not much point in it here.
+    read_settings.local_fs_method = LocalFSReadMethod::pread;


P.S. I did not do the same for remote_fs_method, because it does not use threadpool unless prefetch is made, which is not made here (while for local fs we use threadpool unconditionally, sadly this is inconsistent). Plus there is a known issue with remote_fs_method=read which is not fixed.

antonio2368 · 2024-10-02T13:23:38Z

but the exception should be caught, am I missing something?
https://github.com/ClickHouse/ClickHouse/blob/master/src/Storages/MergeTree/MergeTreeData.cpp#L1473

antonio2368 · 2024-10-02T14:32:14Z

ErrorCodes::CANNOT_SCHEDULE_TASK is retryable...

kssenii · 2024-10-02T14:43:43Z

Yes, I see, but I guess we retried this error several times until the reach the limit of retries and then terminated the server

ClickHouse/src/Storages/MergeTree/MergeTreeData.cpp

Lines 2163 to 2165 in 3198aaf

    
           LOG_ERROR(log, "Loading of outdated parts failed. " 
        
               "Will terminate to avoid undefined behaviour due to inconsistent set of parts. " 
        
               "Exception: {}", getCurrentExceptionMessage(true));

https://s3.amazonaws.com/clickhouse-test-reports/70114/7835b66565b481c61e535d2baee362736d2b750a/stress_test__debug_/fatal_messages.txt

antonio2368 · 2024-10-02T14:46:36Z

This shouldn't be the place because we block the fault injection

auto blocker = CannotAllocateThreadFaultInjector::blockFaultInjections();

I'm honestly not a fan of this solution as removing this line of code will break it again.
Adding more descriptive comment should be better but I think we should figure out why it was uncaught and catch it where it's appropriate (even if still terminate in the catch block like in the code you linked above). We can block fault injection in the case of terminate.

antonio2368 · 2024-10-02T14:47:47Z

I'm also okay with adding blocker of fault injection with a TODO comment that a fix is needed.

kssenii · 2024-10-03T10:17:44Z

Integration tests (tsan) [6/6]

test_storage_kafka

Avoid Cannot schedule a task error when loading parts

9682ebc

kssenii mentioned this pull request Oct 2, 2024

Fixed MaterializedPostgreSQL for subset of columns for several similar table names #70114

Merged

20 tasks

robot-ch-test-poll added the pr-not-for-changelog This PR should not be mentioned in the changelog label Oct 2, 2024

kssenii commented Oct 2, 2024

View reviewed changes

antonio2368 self-assigned this Oct 2, 2024

Add blocker inside runner as well because it is thread_local

45cff04

antonio2368 approved these changes Oct 2, 2024

View reviewed changes

filimonov mentioned this pull request Oct 2, 2024

Thread pool: move thread creation out of lock #68694

Merged

filimonov approved these changes Oct 3, 2024

View reviewed changes

kssenii added this pull request to the merge queue Oct 3, 2024

Merged via the queue into master with commit cea52cc Oct 3, 2024

kssenii deleted the fix-cannot-schedule-task branch October 3, 2024 10:43

robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label Oct 3, 2024

pmimanukyan mentioned this pull request Oct 3, 2024

Limit section (INRANGE) #63661

Open

1 task

Algunenano mentioned this pull request Oct 3, 2024

Terminate called for uncaught exception in MergeTreeData::loadOutdatedDataParts #70169

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid `Cannot schedule a task` error when loading parts#70257

Avoid `Cannot schedule a task` error when loading parts#70257
kssenii merged 2 commits intomasterfrom
fix-cannot-schedule-task

kssenii commented Oct 2, 2024

Uh oh!

robot-clickhouse commented Oct 2, 2024 •

edited by robot-clickhouse-ci-2

Loading

Uh oh!

kssenii Oct 2, 2024

Uh oh!

antonio2368 commented Oct 2, 2024

Uh oh!

antonio2368 commented Oct 2, 2024

Uh oh!

kssenii commented Oct 2, 2024

Uh oh!

antonio2368 commented Oct 2, 2024

Uh oh!

antonio2368 commented Oct 2, 2024

Uh oh!

kssenii commented Oct 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

kssenii commented Oct 2, 2024

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI Settings (Only check the boxes if you know what you are doing):

Uh oh!

robot-clickhouse commented Oct 2, 2024 • edited by robot-clickhouse-ci-2 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kssenii Oct 2, 2024

Choose a reason for hiding this comment

Uh oh!

antonio2368 commented Oct 2, 2024

Uh oh!

antonio2368 commented Oct 2, 2024

Uh oh!

kssenii commented Oct 2, 2024

Uh oh!

antonio2368 commented Oct 2, 2024

Uh oh!

antonio2368 commented Oct 2, 2024

Uh oh!

kssenii commented Oct 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

robot-clickhouse commented Oct 2, 2024 •

edited by robot-clickhouse-ci-2

Loading