Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Reduce memory usage of inserts to JSON by using adaptive write buffer size. A lot of files created by JSON column in wide part contains small amount of data and it doesn't make sense to allocate 1MB buffer for them.

Examples of the changes in memory usage:

Table:

create table test (json JSON) engine=MergeTree order by tuple() settings min_rows_for_wide_part=1, min_bytes_for_wide_part=1;

Before:

avogar-dev :) insert into test from infile 'ghdata_sample.json' format JSONAsObject

Elapsed: 2.064 sec. Processed 5.00 thousand rows, 14.74 MB (2.42 thousand rows/s., 7.14 MB/s.)
Peak memory usage: 6.99 GiB.

avogar-dev :) insert into test select * from s3('s3://clickhouse-public-datasets/gharchive/original/2024-07-01-{0..4}.json.gz', NOSIGN, JSONAsObject)

Elapsed: 33.941 sec. Processed 884.76 thousand rows, 491.26 MB (13.53 thousand rows/s., 7.51 MB/s.)
Peak memory usage: 23.13 GiB.

After:

avogar-dev :) insert into test from infile 'ghdata_sample.json' format JSONAsObject

Elapsed: 1.626 sec. Processed 5.00 thousand rows, 14.74 MB (3.08 thousand rows/s., 9.07 MB/s.)
Peak memory usage: 354.46 MiB.

avogar-dev :) insert into test select * from s3('s3://clickhouse-public-datasets/gharchive/original/2024-07-01-{0..4}.json.gz', NOSIGN, JSONAsObject)

Elapsed: 34.765 sec. Processed 884.76 thousand rows, 491.26 MB (13.52 thousand rows/s., 7.51 MB/s.)
Peak memory usage: 7.65 GiB.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

• Allow: All Required Checks
• Allow: Stateless tests
• Allow: Stateful tests
• Allow: Integration Tests
• Allow: Performance tests
• Allow: All Builds
• Allow: batch 1, 2 for multi-batch jobs
• Allow: batch 3, 4, 5, 6 for multi-batch jobs

• Exclude: Style check
• Exclude: Fast test
• Exclude: All with ASAN
• Exclude: All with TSAN, MSAN, UBSAN, Coverage
• Exclude: All with aarch64, release, debug

• Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
• Exclude: AST fuzzers

• Do not test
• Woolen Wolfdog
• Upload binaries for special builds
• Disable merge-commit
• Disable CI cache