Asynchronous inserts mode support#20557
Conversation
robot-clickhouse
added
the
pr-not-for-changelog
This reverts commit ee19492.
| global_context->setAsynchronousInsertQueue(std::make_shared<AsynchronousInsertQueue>( | ||
| settings.async_insert_threads, | ||
| settings.async_insert_max_data_size, | ||
| AsynchronousInsertQueue::Timeout{.busy = settings.async_insert_busy_timeout, .stale = settings.async_insert_stale_timeout})); |
There was a problem hiding this comment.
AFAIU right now async queues will be removed after storages shutdown, and this will loose some newly INSERT'ed data (via async INSERT)
There was a problem hiding this comment.
Do you mean the order of destruction on the whole server shutdown?
There was a problem hiding this comment.
I'll check the exact impact - but doesn't it the same right now? If we shutdown the server in the middle of insertion, then the new part may not get put properly in many ways - if we don't use WAL in first place.
There was a problem hiding this comment.
In the middle - yes, no guarantee.
But if the INSERT query finished that it is better to try to flush these data into tables, however right now it will not happens since storages will be already shutted down.
This reverts commit 14c5eff.
|
@Mergifyio update |
|
Command
|
|
@Mergifyio update |
|
Command
|
Merging #20557 (Asynchronous inserts)
|
Continued in #27537. |
|
Internal documentation ticket: DOCSUP-14941 |
6 participants
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
New asynchronous insert mode allows to accumulate inserted data and store it in a single batch in background. On server-side it controlled by settings
async_insert_threads,async_insert_max_data_sizeandasync_insert_busy_timeout_ms. For client it can be enabled by settingasync_insertforINSERTqueries with data inlined in query or in separate buffer (e.g. forINSERTqueries via HTTP protocol). Ifwait_for_async_insertis true (by default) the client will wait until data will be flushed to table. Implements #18282.TODO: