Skip to content

turn on async insert#97590

Merged
CheSema merged 19 commits intomasterfrom
chesema-async-insert-on
Mar 4, 2026
Merged

turn on async insert#97590
CheSema merged 19 commits intomasterfrom
chesema-async-insert-on

Conversation

@CheSema
Copy link
Copy Markdown
Member

@CheSema CheSema commented Feb 21, 2026 •
edited
Loading

With #97562
async and sync inserts use transparent tokens, no reason not to use async inserts.

Changelog category (leave one):

  • Backward Incompatible Change

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

turn on async insert by default. Clickhouse will be batching all small inserts by default now. This setting is set under compatibility. If you set compatibility=<version less than 26.2> then default value would be the previous one, false. You could turn off/on async inserts on the several levels: in the config in users profiles, for the session, for the query, or for the mergetree table.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Feb 21, 2026 •
edited
Loading

Workflow [PR], commit [45ca395]

Summary: ❌

job_name test_name status info comment
Stress test (amd_tsan) failure
Logical error: Unexpected return type from A. Expected B. Got C. Action: (STID: 1611-3d11) FAIL cidb, issue
BuzzHouse (amd_debug) failure
Logical error: 'Inconsistent AST formatting: the query: (STID: 1941-1bfa) FAIL cidb
AST fuzzer (amd_ubsan) error

@clickhouse-gh clickhouse-gh bot added the pr-feature Pull request with new product feature label Feb 21, 2026
@EmeraldShift
Copy link
Copy Markdown
Contributor

EmeraldShift commented Feb 24, 2026

Does this introduce either a wait, or non-immediate read-my-writes behavior, for small inserts? If so, is the workaround to explicitly set async_insert=0?

@CheSema
Copy link
Copy Markdown
Member Author

CheSema commented Feb 24, 2026 •
edited
Loading

Does this introduce either a wait?

Yes it does introduce a wait. You could tune timeouts to speed up or/and set adaptive timeouts.

or non-immediate read-my-writes behavior, for small inserts?

No, by default wait_for_async_insert=1 which means that a client receives OK only when insert has been done on one replica. The data is visible on that replica after success insert.

If so, is the workaround to explicitly set async_insert=0?

You could turn it off with settings in the config file or with setting in the query.
Also on table level settings.

@CheSema CheSema force-pushed the chesema-async-insert-on branch 2 times, most recently from d216a5b to 6793064 Compare February 25, 2026 12:59
@CheSema CheSema force-pushed the chesema-async-insert-on branch from 5b2ed1c to 3af3da5 Compare February 26, 2026 15:21
@CheSema CheSema marked this pull request as ready for review February 27, 2026 11:07
@Michicosun Michicosun self-assigned this Feb 27, 2026
CheSema
CheSema commented Mar 4, 2026
View reviewed changes
src/Interpreters/AsynchronousInsertQueue.cpp

LOG_TRACE(log, "Have {} pending inserts in shard {} with total {} bytes of data for query '{}'",
data->entries.size(), size_t(shard_num), data->size_in_bytes, key.query_str);
LOG_TRACE(log, "Have {} pending inserts in shard {} with total {} bytes of data for the async insert queries '{}'",
Copy link
Copy Markdown
Member Author

@CheSema CheSema Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

key.query_str might contain sensitive info.
we do not need that in the loogs.
What we do need is the query ids of async insert in that batch

CheSema
CheSema commented Mar 4, 2026
View reviewed changes
src/Interpreters/AsynchronousInsertQueue.cpp
table->getInMemoryMetadataPtr(),
query_context,
/* no_destination */false,
insert_context->getSettingsRef()[Setting::insert_allow_materialized_columns]);
Copy link
Copy Markdown
Member Author

@CheSema CheSema Mar 4, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found that sync inserts take into account insert_allow_materialized_columns from context, but async insert took here just false

@CheSema CheSema added this pull request to the merge queue Mar 4, 2026
Merged via the queue into master with commit ed1a949 Mar 4, 2026
144 of 148 checks passed
@CheSema CheSema deleted the chesema-async-insert-on branch March 4, 2026 12:47
@robot-ch-test-poll1 robot-ch-test-poll1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Mar 4, 2026
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR label Mar 4, 2026
@robot-ch-test-poll robot-ch-test-poll mentioned this pull request Mar 4, 2026
Merged
CheSema added a commit that referenced this pull request Mar 4, 2026
@CheSema
Merge pull request #98736 from ClickHouse/cherrypick/26.2/97590
4beb7de 
Cherry pick #97590 to 26.2: turn on async insert
robot-clickhouse added a commit that referenced this pull request Mar 4, 2026
@robot-clickhouse
Backport #97590 to 26.2: turn on async insert
bd62e35
@robot-ch-test-poll4 robot-ch-test-poll4 mentioned this pull request Mar 4, 2026
Merged
@robot-ch-test-poll4 robot-ch-test-poll4 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Mar 4, 2026
clickhouse-gh bot added a commit that referenced this pull request Mar 4, 2026
@clickhouse-gh
Merge pull request #98752 from ClickHouse/backport/26.2/97590
71887ec 
Backport #97590 to 26.2: turn on async insert
@alexey-milovidov alexey-milovidov mentioned this pull request Mar 5, 2026
Merged
@alexey-milovidov
Copy link
Copy Markdown
Member

alexey-milovidov commented Mar 5, 2026

It broke our CI and blocked hundreds of other pull requests: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=98660&sha=ab0f4a79a00958604510c4a5067d4cc703a2b9f6&name_0=PR&name_1=Upgrade%20check%20%28amd_release%29

If you think the judgment of our CI is wrong, fix the CI first.

@CheSema
Copy link
Copy Markdown
Member Author

CheSema commented Mar 5, 2026

#98825

@CheSema CheSema mentioned this pull request Mar 5, 2026
Merged
@CheSema
Copy link
Copy Markdown
Member Author

CheSema commented Mar 6, 2026

Restored with more care to the upgrade check
#98825

@alsugiliazova alsugiliazova mentioned this pull request Mar 24, 2026
Closed
@CheSema CheSema mentioned this pull request Mar 31, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Michicosun Michicosun Michicosun approved these changes

Assignees

@Michicosun Michicosun

Labels

pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore pr-feature Pull request with new product feature pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR pr-synced-to-cloud The PR is synced to the cloud repo v26.2-must-backport

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@CheSema @EmeraldShift @alexey-milovidov @Michicosun @robot-ch-test-poll1 @robot-ch-test-poll4 @robot-ch-test-poll3