Closed
Conversation
github-actions
bot
added
parquet
Contributor
Author
|
🤖 |
3 tasks
Contributor
Author
|
🤖: Benchmark completed Details
|
Contributor
|
Nice, no regressions? |
Contributor
Author
Seems like the coalesce batches work is paying off ! |
alamb
force-pushed
the
alamb/cache_filter_result2
branch
from
d236925 to
025d411
Compare
Contributor
Author
|
I have been thinking about memory usage as well as that will be a major factor in if we can cache the predicate results. I annotated the Q22 with code to calculate memory usage of the cached results:
Q22 (some of the best performance gains): // Q22: SELECT "SearchPhrase", MIN("URL"), MIN("Title"), COUNT(*) AS c, COUNT(DISTINCT "UserID") FROM hits WHERE "Title" LIKE '%Google%' AND "URL" NOT LIKE '%.google.%' AND "SearchPhrase" <> '' GROUP BY "SearchPhrase" ORDER BY c DESC LIMIT 10;
Query {
name: "Q22",
filter_columns: vec!["Title", "URL", "SearchPhrase"],
projection_columns: vec!["SearchPhrase", "URL", "Title", "UserID"],
predicates: vec![
ClickBenchPredicate::like_Google(0),
ClickBenchPredicate::nlike_google(1),
ClickBenchPredicate::not_empty(2),
],
expected_row_count: 46,
},for hits_1.parquet, the data sizes are:
(I totally used @XiangpengHao 's https://parquet-viewer.xiangpeng.systems/ for this analysis)
I will try and add some additional debugging / annotation code to see what the peak memory usage was (and if I limit it to 1MB if that will get triggered for any query) |
Contributor
Author
Seems reasonable to me. I need to think about memory handling a bit more carefully now |
Closed
3 tasks
Contributor
Author
|
Superceded by #7850 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
TODO:
coalesce)coalesecekernel (BatchCoalescer) #7761Rationale for this change
I am working on not decoding predicate columns twice when evaluating filters in the reader
In #7513 we prototyped several APIs that we have now started making real (like
BatchCoalescer) so I made a new PR that used those APIs which doesn't have hundreds of comments.I am pleased with how it is looking now, and as before I don't really plan to merge this PR as is, I am using it as a design vehicle
What changes are included in this PR?
Are there any user-facing changes?
Not yet,