Skip to content

Reduce memory usage during prefetches of JSON column in Wide parts#77640

Merged
Avogar merged 2 commits intoClickHouse:masterfrom
Avogar:json-prefetch-memory-usage
Mar 17, 2025
Merged

Reduce memory usage during prefetches of JSON column in Wide parts#77640
Avogar merged 2 commits intoClickHouse:masterfrom
Avogar:json-prefetch-memory-usage

Conversation

@Avogar
Copy link
Copy Markdown
Member

@Avogar Avogar commented Mar 14, 2025

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Reduce memory usage during prefetches of JSON column in Wide parts

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Mar 14, 2025

Workflow [PR], commit [2f390b2]

@clickhouse-gh clickhouse-gh bot added the pr-improvement Pull request with some product improvements label Mar 14, 2025
@Avogar
Copy link
Copy Markdown
Member Author

Avogar commented Mar 14, 2025

For example:
Table 1000000 rows, 1000 subcolumns in each row.
Before:

clickhouse-cloud :) select * from test where y = 1 format Null settings max_parallel_replicas=1

0 rows in set. Elapsed: 8.283 sec. Processed 1.00 million rows, 8.12 MB (120.73 thousand rows/s., 979.72 KB/s.)
Peak memory usage: 69.16 GiB.

Now:

clickhouse-cloud :) select * from test where y = 1 format Null settings max_parallel_replicas=1

0 rows in set. Elapsed: 0.351 sec. Processed 1.00 million rows, 8.11 MB (2.85 million rows/s., 23.15 MB/s.)
Peak memory usage: 1.11 GiB.

@nickitat nickitat self-assigned this Mar 14, 2025
@Avogar Avogar added this pull request to the merge queue Mar 17, 2025
Merged via the queue into ClickHouse:master with commit 635dcdc Mar 17, 2025
120 of 123 checks passed
@Avogar Avogar deleted the json-prefetch-memory-usage branch March 17, 2025 13:37
@robot-clickhouse robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Mar 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-improvement Pull request with some product improvements pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants