Skip to content

Fix 01114_database_atomic flakiness#68930

Merged
Algunenano merged 4 commits intoClickHouse:masterfrom
Algunenano:01114_database_atomic
Aug 28, 2024
Merged

Fix 01114_database_atomic flakiness#68930
Algunenano merged 4 commits intoClickHouse:masterfrom
Algunenano:01114_database_atomic

Conversation

@Algunenano
Copy link
Copy Markdown
Member

@Algunenano Algunenano commented Aug 27, 2024

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix 01114_database_atomic flakiness

https://s3.amazonaws.com/clickhouse-test-reports/67983/e8d9fbdd1f9bd23ee9fda60e2068b4a9642e52cd/stateless_tests__asan__[2_2].html

Sleep is a terrible synchronization method

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

  • Allow: All Required Checks
  • Allow: Stateless tests
  • Allow: Stateful tests
  • Allow: Integration Tests
  • Allow: Performance tests
  • Allow: All Builds
  • Allow: batch 1, 2 for multi-batch jobs
  • Allow: batch 3, 4, 5, 6 for multi-batch jobs
  • Exclude: Style check
  • Exclude: Fast test
  • Exclude: All with ASAN
  • Exclude: All with TSAN, MSAN, UBSAN, Coverage
  • Exclude: All with aarch64, release, debug
  • Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
  • Exclude: AST fuzzers
  • Do not test
  • Woolen Wolfdog
  • Upload binaries for special builds
  • Disable merge-commit
  • Disable CI cache

@Algunenano Algunenano added the 🍃 green ci 🌿 Fixing flaky tests in CI label Aug 27, 2024
@robot-clickhouse robot-clickhouse added the pr-not-for-changelog This PR should not be mentioned in the changelog label Aug 27, 2024
@robot-ch-test-poll2
Copy link
Copy Markdown
Contributor

robot-ch-test-poll2 commented Aug 27, 2024 •
edited by robot-ch-test-poll3
Loading

This is an automated comment for commit 0fe8f94 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check nameDescriptionStatus
Stateless testsRuns stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc❌ failure
Successful checks
Check nameDescriptionStatus
BuildsThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Fast testNormally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here✅ success
Flaky testsChecks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc✅ success
Stress testRuns stateless functional tests concurrently from several clients to detect concurrency-related errors✅ success
Style checkRuns a set of checks to keep the code style clean. If some of tests failed, see the related log from the report✅ success
Unit testsRuns the unit tests for different release types✅ success
Upgrade checkRuns stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts✅ success

@Algunenano Algunenano mentioned this pull request Aug 27, 2024
Merged
19 tasks
@Algunenano
Copy link
Copy Markdown
Member Author

Algunenano commented Aug 27, 2024

From failed flaky check there is something unrelated to the test changes but quite intriguing. Why did this query take 165 seconds instead of the expected 33?

$ grep 23890d4e-3387-4c3d-8cca-a79984428744 a.log
2024.08.27 07:42:13.955871 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Debug> executeQuery: (from [::1]:42968) (comment: 01114_database_atomic.sh) SELECT count(col), sum(col) FROM (SELECT n + sleepEachRow(1.5) AS col FROM test_cb36bnb2_1.mt) -- 25279 (stage: Complete)
2024.08.27 07:42:13.977383 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> Planner: Query to stage Complete
2024.08.27 07:42:13.979284 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> Planner: Query to stage Complete
2024.08.27 07:42:13.988626 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> Planner: Query from stage FetchColumns to stage Complete
2024.08.27 07:42:13.991501 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> Planner: Query from stage FetchColumns to stage Complete
2024.08.27 07:42:14.008912 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Debug> test_cb36bnb2_3.mt (SelectExecutor): Key condition: unknown
2024.08.27 07:42:14.015499 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Debug> test_cb36bnb2_3.mt (SelectExecutor): MinMax index condition: unknown
2024.08.27 07:42:14.017124 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> test_cb36bnb2_3.mt (SelectExecutor): Filtering marks by primary and secondary keys
2024.08.27 07:42:14.020040 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Debug> test_cb36bnb2_3.mt (SelectExecutor): Selected 5/5 parts by partition key, 5 parts by primary key, 5/5 marks by primary key, 5 marks to read from 5 ranges
2024.08.27 07:42:14.023495 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> test_cb36bnb2_3.mt (SelectExecutor): Spreading mark ranges among streams (default reading)
2024.08.27 07:42:14.026671 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> test_cb36bnb2_3.mt (SelectExecutor): Reading 1 ranges in order from part 0_1_1_0, approx. 22 rows starting from 0
2024.08.27 07:42:14.028694 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> test_cb36bnb2_3.mt (SelectExecutor): Reading 1 ranges in order from part 1_2_2_0, approx. 22 rows starting from 0
2024.08.27 07:42:14.031261 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> test_cb36bnb2_3.mt (SelectExecutor): Reading 1 ranges in order from part 2_3_3_0, approx. 22 rows starting from 0
2024.08.27 07:42:14.033017 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> test_cb36bnb2_3.mt (SelectExecutor): Reading 1 ranges in order from part 3_4_4_0, approx. 22 rows starting from 0
2024.08.27 07:42:14.041174 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> test_cb36bnb2_3.mt (SelectExecutor): Reading 1 ranges in order from part 4_5_5_0, approx. 22 rows starting from 0
2024.08.27 07:42:47.069202 [ 1435 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> AggregatingTransform: Aggregating
2024.08.27 07:42:47.070030 [ 1435 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> Aggregator: Aggregation method: without_key
2024.08.27 07:43:56.968718 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Debug> TCPHandler: Sending profile events block with 5 rows, 374 bytes took 137 milliseconds
2024.08.27 07:44:18.128161 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Debug> TCPHandler: Sending profile events block with 5 rows, 374 bytes took 105 milliseconds
2024.08.27 07:44:59.126268 [ 1435 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> AggregatingTransform: Aggregated. 110 to 1 rows (from 880.00 B) in 165.079057233 sec. (0.666 rows/sec., 5.33 B/sec.)
2024.08.27 07:44:59.127164 [ 1435 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> Aggregator: Merging aggregated data
2024.08.27 07:44:59.128130 [ 1435 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Trace> HashTablesStatistics: Statistics updated for key=8495912807294640416: new sum_of_sizes=1, median_size=1
2024.08.27 07:44:59.146471 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Debug> executeQuery: Read 110 rows, 880.00 B in 165.222086 sec., 0.6657705556386693 rows/sec., 5.33 B/sec.
2024.08.27 07:44:59.151001 [ 2370 ] {23890d4e-3387-4c3d-8cca-a79984428744} <Debug> TCPHandler: Processed in 165.301555906 sec

Well, 33 is when all partitions are processed in parallel, not when they are all executed in the same thread. And in these cases we have --max_threads 1. I'll adjust it so we use more threads on those queries. It might still be possible to be slower than 33 seconds, which is ok, but it won't take 3 minutes to run the test.

@Avogar Avogar self-assigned this Aug 28, 2024
Avogar
Avogar reviewed Aug 28, 2024
View reviewed changes
Avogar
Avogar reviewed Aug 28, 2024
View reviewed changes
@Algunenano
Copy link
Copy Markdown
Member Author

Algunenano commented Aug 28, 2024

Failure in 01290_max_execution_speed_distributed unrelated. I'll fix in in a different PR

@Algunenano Algunenano added this pull request to the merge queue Aug 28, 2024
Merged via the queue into ClickHouse:master with commit e1291cd Aug 28, 2024
@Algunenano Algunenano deleted the 01114_database_atomic branch August 28, 2024 17:26
@Algunenano Algunenano added the pr-must-backport Pull request should be backported intentionally. Use this label with great care! label Aug 28, 2024
@robot-ch-test-poll robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label Aug 28, 2024
@robot-clickhouse-ci-1 robot-clickhouse-ci-1 mentioned this pull request Aug 28, 2024
Merged
robot-clickhouse added a commit that referenced this pull request Aug 28, 2024
@robot-clickhouse
Backport #68930 to 24.3: Fix 01114_database_atomic flakiness
9b79715
This was referenced Aug 28, 2024
Merged
Merged
robot-clickhouse added a commit that referenced this pull request Aug 28, 2024
@robot-clickhouse
Backport #68930 to 24.5: Fix 01114_database_atomic flakiness
4edb06b
This was referenced Aug 28, 2024
Merged
Merged
robot-clickhouse added a commit that referenced this pull request Aug 28, 2024
@robot-clickhouse
Backport #68930 to 24.6: Fix 01114_database_atomic flakiness
61c5d09
This was referenced Aug 28, 2024
Merged
Merged
robot-clickhouse added a commit that referenced this pull request Aug 28, 2024
@robot-clickhouse
Backport #68930 to 24.7: Fix 01114_database_atomic flakiness
e2b1ae5
This was referenced Aug 28, 2024
Merged
Merged
robot-clickhouse added a commit that referenced this pull request Aug 28, 2024
@robot-clickhouse
Backport #68930 to 24.8: Fix 01114_database_atomic flakiness
c233e73
@robot-clickhouse-ci-1 robot-clickhouse-ci-1 mentioned this pull request Aug 28, 2024
Merged
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Aug 28, 2024
robot-ch-test-poll1 added a commit that referenced this pull request Aug 28, 2024
@robot-ch-test-poll1
Merge pull request #69044 from ClickHouse/backport/24.3/68930
afc0964 
Backport #68930 to 24.3: Fix 01114_database_atomic flakiness
@Algunenano Algunenano removed the pr-must-backport Pull request should be backported intentionally. Use this label with great care! label Aug 30, 2024
Algunenano added a commit that referenced this pull request Aug 30, 2024
@Algunenano
Merge pull request #69052 from ClickHouse/backport/24.8/68930
e55c222 
Backport #68930 to 24.8: Fix 01114_database_atomic flakiness
Algunenano added a commit that referenced this pull request Aug 30, 2024
@Algunenano
Merge pull request #69050 from ClickHouse/backport/24.7/68930
cc72e95 
Backport #68930 to 24.7: Fix 01114_database_atomic flakiness
Algunenano added a commit that referenced this pull request Aug 30, 2024
@Algunenano
Merge pull request #69048 from ClickHouse/backport/24.6/68930
4fa84f7 
Backport #68930 to 24.6: Fix 01114_database_atomic flakiness
Algunenano added a commit that referenced this pull request Aug 30, 2024
@Algunenano
Merge pull request #69046 from ClickHouse/backport/24.5/68930
f6ac209 
Backport #68930 to 24.5: Fix 01114_database_atomic flakiness
@Algunenano Algunenano mentioned this pull request Sep 4, 2024
Closed
@Algunenano Algunenano restored the 01114_database_atomic branch September 4, 2024 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Avogar Avogar Avogar approved these changes

Assignees

@Avogar Avogar

Labels

🍃 green ci 🌿 Fixing flaky tests in CI pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore pr-not-for-changelog This PR should not be mentioned in the changelog pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Algunenano @robot-ch-test-poll2 @Avogar @robot-clickhouse @robot-ch-test-poll @robot-ch-test-poll3