Conversation
Member
Author
robot-clickhouse-ci-2
added
the
pr-performance
Contributor
|
This is an automated comment for commit bbfbf97 with description of existing statuses. It's updated for the latest CI running
|
al13n321
force-pushed
the
readat
branch
2 times, most recently
from
350ae48 to
bb92b55
Compare
robot-clickhouse-ci-1
added
the
submodule changed
robot-clickhouse
removed
the
submodule changed
Contributor
|
@al13n321 When do you will rewrite parquet reader? |
Member
Author
Contributor
|
I'm reimplementing a parquet reader based on the ClickHouse code. Perhaps an experimental PR can be submitted in June. |
Member
Author
Nice! Then I'm not going to do it, for now. |
Avogar
reviewed
May 31, 2023
src/IO/ReadWriteBufferFromHTTP.cpp
Outdated
Comment on lines
+638
to
+640
| if (!cancelled) | ||
| { | ||
| } |
Avogar
approved these changes
May 31, 2023
baibaichen
added a commit
to Kyligence/gluten
that referenced
this pull request
Jun 3, 2023
zzcclp
pushed a commit
to apache/gluten
that referenced
this pull request
Jun 3, 2023
* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20230603) * Fix Build since ClickHouse/ClickHouse#50400 * Fix Build since ClickHouse/ClickHouse#49539 * Workaround protoc's issue(protocolbuffers/protobuf#9378). --------- Co-authored-by: kyligence-git <[email protected]> Co-authored-by: Chang Chen <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Parquet's parallel random reads from local file are now actually parallel (pread() calls or memcpy from mmap), instead of locking two mutexes for no reason.
In #47964 I refactored things to use SeekableReadBufferFactory in more places, as the abstraction for random-access "files" (local/http/s3). I think that was a mistake. This PR re-refactors everything in the opposite direction: it adds a thread-safe positioned-read method in SeekableReadBuffer and uses it whenever parallel random reading is needed (ParallelReadBuffer and ParquetBlockInputFormat). This turned out better than the factory stuff, I think: less (?) code, fewer boilerplate objects, less useless extra cpu work (like maintaining a pool of ReadBuffers to reuse), less awkward code like
ifs everywhere for buffer vs factory. The factory is removed altogether.Also fixed a confusing error message when getting HTTP 404 error: it used to say "Not a Parquet file" because asArrowFileLoadIntoMemory() was catching exceptions too eagerly.