Skip to content

S3Queue rework ordered mode parallel processing#64349

Merged
kssenii merged 20 commits intomasterfrom
s3queue-fix-ordered-mode
Jun 4, 2024
Merged

S3Queue rework ordered mode parallel processing#64349
kssenii merged 20 commits intomasterfrom
s3queue-fix-ordered-mode

Conversation

@kssenii
Copy link
Copy Markdown
Member

@kssenii kssenii commented May 24, 2024 •
edited by Algunenano
Loading

Changelog category (leave one):

  • Backward Incompatible Change

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Rework parallel processing in Ordered mode of storage S3Queue. This PR is backward incompatible for Ordered mode if you used settings s3queue_processing_threads_num or s3queue_total_shards_num. Setting s3queue_total_shards_num is deleted, previously it was allowed to use only under s3queue_allow_experimental_sharded_mode, which is now deprecated. A new setting is added - s3queue_buckets.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings

NOTE: If your merge the PR with modified CI you MUST KNOW what you are doing
NOTE: Checked options will be applied if set before CI RunConfig/PrepareRunConfig step

Run these jobs only (required builds will be added automatically):

  • Integration Tests
  • Fast test
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • All with aarch64
  • All with ASAN
  • All with TSAN
  • All with Analyzer
  • All with Azure
  • Add your option here

Deny these jobs:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64

Extra options:

  • do not test (only style check)
  • disable merge-commit (no merge from master before tests)
  • disable CI cache (job reuse)

Only specified batches in multi-batch jobs:

  • 1
  • 2
  • 3
  • 4

@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-backward-incompatible Pull request with backwards incompatible changes label May 24, 2024
@robot-ch-test-poll3
Copy link
Copy Markdown
Contributor

robot-ch-test-poll3 commented May 24, 2024 •
edited by robot-clickhouse
Loading

This is an automated comment for commit 1790c85 with description of existing statuses. It's updated for the latest CI running

⏳ Click here to open a full report in a separate page

Check nameDescriptionStatus
CI runningA meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR⏳ pending
Successful checks
Check nameDescriptionStatus
A SyncIf it fails, ask a maintainer for help✅ success
Docs checkBuilds and tests the documentation✅ success
Fast testNormally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here✅ success
Integration testsThe integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests✅ success
Mergeable CheckChecks if all other necessary checks are successful✅ success
PR CheckThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Style checkRuns a set of checks to keep the code style clean. If some of tests failed, see the related log from the report✅ success

@kssenii kssenii force-pushed the s3queue-fix-ordered-mode branch from eedf8ec to d8c55d3 Compare May 24, 2024 17:14
@kssenii kssenii marked this pull request as ready for review May 24, 2024 17:47
@kssenii kssenii force-pushed the s3queue-fix-ordered-mode branch 2 times, most recently from 0074e23 to 3f7020b Compare May 24, 2024 18:22
@kssenii kssenii force-pushed the s3queue-fix-ordered-mode branch from 3f7020b to ab58ff6 Compare May 24, 2024 18:37
@antonio2368 antonio2368 self-assigned this May 27, 2024
@kssenii kssenii marked this pull request as draft May 27, 2024 17:07
@kssenii kssenii marked this pull request as draft May 27, 2024 17:07
@kssenii kssenii marked this pull request as ready for review May 30, 2024 14:35
@kssenii kssenii requested a review from antonio2368 May 31, 2024 11:29
antonio2368
antonio2368 reviewed Jun 3, 2024
View reviewed changes
src/Storages/S3Queue/S3QueueMetadata.cpp
Comment on lines +342 to +344
{
LOG_TEST(log, "Path {} does not exist", zookeeper_failed_path.string());
}
Copy link
Copy Markdown
Member

@antonio2368 antonio2368 Jun 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{
LOG_TEST(log, "Path {} does not exist", zookeeper_failed_path.string());
}
LOG_TEST(log, "Path {} does not exist", zookeeper_failed_path.string());

@kssenii kssenii requested a review from antonio2368 June 3, 2024 15:32
@kssenii
Copy link
Copy Markdown
Member Author

kssenii commented Jun 4, 2024

Integration tests (aarch64) [5/6] — fail: 1

test_checking_s3_blobs_paranoid/test.py::test_query_is_canceled_with_inf_retries -- broken in master

@kssenii kssenii added this pull request to the merge queue Jun 4, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to no response for status checks Jun 4, 2024
@kssenii kssenii added this pull request to the merge queue Jun 4, 2024
@kssenii kssenii removed this pull request from the merge queue due to a manual request Jun 4, 2024
@kssenii kssenii added this pull request to the merge queue Jun 4, 2024
@kssenii kssenii removed this pull request from the merge queue due to a manual request Jun 4, 2024
@kssenii kssenii added this pull request to the merge queue Jun 4, 2024
@kssenii kssenii removed this pull request from the merge queue due to a manual request Jun 4, 2024
@kssenii kssenii force-pushed the s3queue-fix-ordered-mode branch from bb9403f to 1790c85 Compare June 4, 2024 17:42
@kssenii kssenii added this pull request to the merge queue Jun 4, 2024
Merged via the queue into master with commit d044d22 Jun 4, 2024
@kssenii kssenii deleted the s3queue-fix-ordered-mode branch June 4, 2024 21:06
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 4, 2024
@deltarod deltarod mentioned this pull request Jun 4, 2024
Closed
@Felixoid
Copy link
Copy Markdown
Member

Felixoid commented Jun 5, 2024

@kssenii there's clang tidy failure https://github.com/ClickHouse/ClickHouse/actions/runs/9375197268/job/25814470919?pr=64759#step:8:15491

@Felixoid Felixoid mentioned this pull request Jun 5, 2024
Merged
22 tasks
@lgbo-ustc lgbo-ustc mentioned this pull request Jun 5, 2024
Merged
22 tasks
@azat azat mentioned this pull request Jun 5, 2024
Merged
@kssenii kssenii mentioned this pull request Jun 5, 2024
Merged
23 tasks
alexey-milovidov added a commit that referenced this pull request Jun 6, 2024
@alexey-milovidov
Merge pull request #64845 from ClickHouse/follow-up-s3-queue
9ab0693 
Follow-up to #64349
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@antonio2368 antonio2368 antonio2368 approved these changes

Assignees

@antonio2368 antonio2368

Labels

pr-backward-incompatible Pull request with backwards incompatible changes pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kssenii @robot-ch-test-poll3 @Felixoid @antonio2368 @robot-clickhouse