Feat add _etag for object storage by skyoct · Pull Request #65386 · ClickHouse/ClickHouse

skyoct · 2024-06-18T11:18:17Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add _etag virtual column for S3 table engine. Fixes #65312.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

Allow: All Required Checks
Allow: Stateless tests
Allow: Stateful tests
Allow: Integration Tests
Allow: Performance tests
Allow: All Builds
Allow: All NOT Required Checks
Allow: batch 1, 2 for multi-batch jobs
Allow: batch 3, 4, 5, 6 for multi-batch jobs

Exclude: Style check
Exclude: Fast test
Exclude: All with ASAN
Exclude: All with TSAN, MSAN, UBSAN, Coverage
Exclude: All with aarch64, release, debug

Do not test
Upload binaries for special builds
Disable merge-commit
Disable CI cache

robot-ch-test-poll3 · 2024-06-18T14:35:29Z

This is an automated comment for commit 364e973 with description of existing statuses. It's updated for the latest CI running

✅ Click here to open a full report in a separate page

Successful checks

Check name	Description	Status
Docs check	Builds and tests the documentation	✅ success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	✅ success
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	✅ success
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	✅ success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success
Unit tests	Runs the unit tests for different release types	✅ success

skyoct · 2024-06-24T04:02:23Z

@kssenii Hi, How can I rerun CI?

kssenii · 2024-06-24T09:15:43Z

CI fails for a reason.

ClickHouse build check — 34/35 artifact groups are OK

If you open details, you'll see:

clickhouse_common_io.dir/IO/WriteBufferFromS3.cpp.o -c /build/src/IO/WriteBufferFromS3.cpp
Jun 21 02:43:34 /build/src/IO/S3/getObjectInfo.h:18:12: error: redundant string initialization [readability-redundant-string-init,-warnings-as-errors]
Jun 21 02:43:34    18 |     String etag = "";
Jun 21 02:43:34       |            ^~~~~~~~~
Jun 21 02:43:34       |            etag
Jun 21 02:43:34 48866 warnings generated.

kssenii

Could you please add _etag to a test? See https://github.com/ClickHouse/ClickHouse/blob/master/tests/queries/0_stateless/02245_s3_virtual_columns.sql

skyoct · 2024-06-25T01:19:59Z

@kssenii Ok, Thank you very much.

skyoct · 2024-06-25T01:22:37Z

@kssenii The etag should just be a random string, and I have no way of knowing its value. Can I just check if it is empty here?

kssenii · 2024-06-26T11:38:52Z

@kssenii The etag should just be a random string, and I have no way of knowing its value. Can I just check if it is empty here?

Yes, of course.

skyoct · 2024-07-01T09:07:45Z

@kssenii Hi, It seems that I can't find what caused the error from this.

kssenii · 2024-07-01T09:36:57Z

Stateless tests (release) — fail: 1

00992_system_parts_race_condition_zookeeper_long

-- this test is flaky, do not mind this failure. I will mark this failure is ignored. The tests will continue once I fix Cloud fork sync (only for ClickHouse Inc. employees) — tests failed -- will do now.

kssenii · 2024-07-01T09:37:26Z

Docs check — Found errors in docs

Please check this failure it should be related to changes

kssenii · 2024-07-03T09:49:14Z

Stateless tests (release) — fail: 1

00992_system_parts_race_condition_zookeeper_long -- flaky, not related to changes

kssenii · 2024-07-08T15:48:47Z

@skyoct, could you please fix the build?

skyoct · 2024-07-09T02:00:12Z

@kssenii Okay, I will fix it immediately.

kssenii · 2024-07-11T14:10:11Z

Cloud fork sync (only for ClickHouse Inc. employees) — Failed to merge into the sync PR

Fixed it now, @skyoct please don't do merge with master for this PR now, if it is not necessary, because it now breaks the sync each time for this PR.

Could you please also fix style-check?

In /ClickHouse/tests/queries/0_stateless/02270_errors_in_files.sh line 31:
rm "${user_files_path}"/test_02270_1.csv
    ^----------------^ SC2154 (warning): user_files_path is referenced but not assigned (did you mean 'USER_FILES_PATH'?).

kssenii · 2024-07-11T14:11:15Z

tests/queries/0_stateless/02270_errors_in_files.sh

btw why change this?

@kssenii I feel that this test case not passing is quite strange. Want to try again.

skyoct · 2024-07-12T08:53:09Z

@kssenii Is it necessary to rerun the tests after this change?

skyoct · 2024-07-16T06:48:13Z

@kssenii Is it necessary to rerun the tests after this change?

@kssenii Can you help me with this?

…into feat-s3-field

skyoct · 2024-08-07T01:50:49Z

@kssenii Hi, Can you help review?

kssenii · 2024-08-07T10:20:01Z

@kssenii Hi, Can you help review?

Sorry, Cloud For Sync gets stuck in pending state for this PR, I restarted it yesterday via github api, but it does not finish still. I will try again to reset it again via github api, and if it does not help again - I will push an empty commit.

alexey-milovidov · 2024-08-20T06:26:07Z

@kssenii, it is very strange to have ETag in double quotes:

:) SELECT _path, _etag FROM s3('s3://ookla-open-data/parquet/performance/type=*/year=*/quarter=*/*.parquet') LIMIT 10

SELECT
    _path,
    _etag
FROM s3('s3://ookla-open-data/parquet/performance/type=*/year=*/quarter=*/*.parquet')
LIMIT 10

Query id: 992f4385-b9e1-4194-8f3a-55843b093615

    ┌─_path─────────────────────────────────────────────────────────────────────────────────────────────────────────┬─_etag─────────────────────────────────┐
 1. │ ookla-open-data/parquet/performance/type=fixed/year=2019/quarter=1/2019-01-01_performance_fixed_tiles.parquet │ "921e7e414a81f5eb426810e73a3ee108-28" │
 2. │ ookla-open-data/parquet/performance/type=fixed/year=2019/quarter=1/2019-01-01_performance_fixed_tiles.parquet │ "921e7e414a81f5eb426810e73a3ee108-28" │
 3. │ ookla-open-data/parquet/performance/type=fixed/year=2019/quarter=1/2019-01-01_performance_fixed_tiles.parquet │ "921e7e414a81f5eb426810e73a3ee108-28" │
 4. │ ookla-open-data/parquet/performance/type=fixed/year=2019/quarter=1/2019-01-01_performance_fixed_tiles.parquet │ "921e7e414a81f5eb426810e73a3ee108-28" │
 5. │ ookla-open-data/parquet/performance/type=fixed/year=2019/quarter=1/2019-01-01_performance_fixed_tiles.parquet │ "921e7e414a81f5eb426810e73a3ee108-28" │
 6. │ ookla-open-data/parquet/performance/type=fixed/year=2019/quarter=1/2019-01-01_performance_fixed_tiles.parquet │ "921e7e414a81f5eb426810e73a3ee108-28" │
 7. │ ookla-open-data/parquet/performance/type=fixed/year=2019/quarter=1/2019-01-01_performance_fixed_tiles.parquet │ "921e7e414a81f5eb426810e73a3ee108-28" │
 8. │ ookla-open-data/parquet/performance/type=fixed/year=2019/quarter=1/2019-01-01_performance_fixed_tiles.parquet │ "921e7e414a81f5eb426810e73a3ee108-28" │
 9. │ ookla-open-data/parquet/performance/type=fixed/year=2019/quarter=1/2019-01-01_performance_fixed_tiles.parquet │ "921e7e414a81f5eb426810e73a3ee108-28" │
10. │ ookla-open-data/parquet/performance/type=fixed/year=2019/quarter=1/2019-01-01_performance_fixed_tiles.parquet │ "921e7e414a81f5eb426810e73a3ee108-28" │
    └───────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────────────────┘

Unless it already was in double quotes in HTTP headers, it looks like a bug in AWS SDK.

alexey-milovidov · 2024-08-20T06:27:18Z

And I don't see implementation for the URL engine, where it would also be natural.

alexey-milovidov · 2024-08-20T07:40:34Z

@kssenii, _etag does not filter files, and this is contrary to my expectations:

https://pastila.nl/?05873082/36052e9869445d6b588d02dc837b8be7#c8bUlv6de6qe1JNHqp389w==

feat: add etag for object storage

766130b

kssenii self-assigned this Jun 18, 2024

kssenii added the can be tested Allows running workflows for external contributors label Jun 18, 2024

robot-ch-test-poll3 added the pr-improvement Pull request with some product improvements label Jun 18, 2024

skyoct added 2 commits June 19, 2024 14:06

fix get tag

8ca4790

Feat add docs

7523d8b

skyoct marked this pull request as ready for review June 19, 2024 13:26

skyoct added 2 commits June 20, 2024 22:54

ignore docs dict etag

fdfa6ad

ignore etag dict

ff72bbb

fix etag init

531a7e3

kssenii reviewed Jun 24, 2024

View reviewed changes

kssenii and others added 3 commits June 26, 2024 13:42

Merge branch 'master' into feat-s3-field

d6a351b

add s3 tests

15b36c1

fix tests

7cfbd14

remove _last_modified field

f58f6cf

Merge branch 'master' into feat-s3-field

292113e

fix etag init

1505cb2

kssenii reviewed Jul 11, 2024

View reviewed changes

skyoct force-pushed the feat-s3-field branch 2 times, most recently from 79aea8d to 1505cb2 Compare July 12, 2024 03:27

kssenii and others added 3 commits July 26, 2024 14:47

Ping CI

d06b9ca

CI

1c533f7

Merge branch 'feat-s3-field' of https://github.com/skyoct/ClickHouse …

5d082d6

…into feat-s3-field

kssenii enabled auto-merge August 7, 2024 14:19

Ping CI

364e973

kssenii added this pull request to the merge queue Aug 7, 2024

Merged via the queue into ClickHouse:master with commit 315fd54 Aug 7, 2024

robot-ch-test-poll3 added the pr-synced-to-cloud The PR is synced to the cloud repo label Aug 7, 2024

baibaichen mentioned this pull request Aug 8, 2024

[GLUTEN-1632][CH]Daily Update Clickhouse Version (20240808) apache/gluten#6755

Merged

azat mentioned this pull request Mar 31, 2025

Feat Add new virtual column for AWS S3 tagging #77773

Merged

Conversation

skyoct commented Jun 18, 2024 • edited by kssenii Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI Settings (Only check the boxes if you know what you are doing):

Uh oh!

robot-ch-test-poll3 commented Jun 18, 2024 • edited by robot-ch-test-poll1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

skyoct commented Jun 24, 2024

Uh oh!

kssenii commented Jun 24, 2024

Uh oh!

kssenii left a comment

Choose a reason for hiding this comment

Uh oh!

skyoct commented Jun 25, 2024

Uh oh!

skyoct commented Jun 25, 2024

Uh oh!

kssenii commented Jun 26, 2024

Uh oh!

skyoct commented Jul 1, 2024

Uh oh!

kssenii commented Jul 1, 2024

Uh oh!

kssenii commented Jul 1, 2024

Uh oh!

kssenii commented Jul 3, 2024

Uh oh!

kssenii commented Jul 8, 2024

Uh oh!

skyoct commented Jul 9, 2024

Uh oh!

kssenii commented Jul 11, 2024

Uh oh!

kssenii Jul 11, 2024

Choose a reason for hiding this comment

Uh oh!

skyoct Jul 12, 2024

Choose a reason for hiding this comment

Uh oh!

skyoct commented Jul 12, 2024

Uh oh!

skyoct commented Jul 16, 2024

Uh oh!

skyoct commented Aug 7, 2024

Uh oh!

kssenii commented Aug 7, 2024

Uh oh!

alexey-milovidov commented Aug 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Aug 20, 2024

Uh oh!

alexey-milovidov commented Aug 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

skyoct commented Jun 18, 2024 •

edited by kssenii

Loading

robot-ch-test-poll3 commented Jun 18, 2024 •

edited by robot-ch-test-poll1

Loading

alexey-milovidov commented Aug 20, 2024 •

edited

Loading