Skip to content

DELETE FROM table WHERE ...#24755

Closed
myrrc wants to merge 31 commits intoClickHouse:masterfrom
myrrc:feature/lightweight-deletes
Closed

DELETE FROM table WHERE ...#24755
myrrc wants to merge 31 commits intoClickHouse:masterfrom
myrrc:feature/lightweight-deletes

Conversation

@myrrc
Copy link
Copy Markdown
Contributor

@myrrc myrrc commented May 29, 2021

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Lightweight mutations. Unlike ordinary mutations, do not immediately change files, but create a per-part deleted rows mask.

Implementations stages:

  • Parser for DELETE FROM query (+ changes to ClickHouse grammar).
  • Changes for system.mutations table ("is_lightweight" field).
  • Deleted rows mask for parts: support for multiple format versions (e.g. further development like compressed masks).
  • TODO Masks reading pipeline on server [re]start that is consistent with ordinary mutations evaluation.
  • TODO Evaluation of deleted rows predicate in WHERE section for SELECT queries for non-replicated MergeTree tables.
  • TODO Calculating deleted rows mask in executor for ordinary mutations (dispatching).
  • TODO Simple functional tests for DELETE FROM
  • TODO Simple integration tests for DELETE FROM: basic tests, parallel application of lightweight mutations, sequential application of lightweight mutations, intermixing ordinary mutations and lightweight mutations.

@robot-clickhouse robot-clickhouse added doc-alert pr-feature Pull request with new product feature labels May 29, 2021
@alexey-milovidov
Copy link
Copy Markdown
Member

There is no syntax DELETE * FROM TABLE. It is DELETE FROM TABLE. The title was incorrect.

@alexey-milovidov alexey-milovidov changed the title DELETE * FROM table WHERE ... DELETE FROM table WHERE ... Jun 12, 2021
@myrrc
Copy link
Copy Markdown
Contributor Author

myrrc commented Aug 26, 2021

Depends on #28085

@robot-ch-test-poll1 robot-ch-test-poll1 added the submodule changed At least one submodule changed in this PR. label Aug 26, 2021
Copy link
Copy Markdown
Member

@azat azat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@myrrc AFAICS right now it only contains:

  • parser for DELETE (not enabled)
  • new interpreter
  • lots of unrelated changes
  • does not contains any real implementation for DELETE

Am I right?

If so, and if you are going to keep that "lots of unrelated changes", and you really sure that they are worth it, how about moving them into separate PR, to make implementation of DELETE simpler for possible reviewers?

@myrrc
Copy link
Copy Markdown
Contributor Author

myrrc commented Aug 30, 2021

@myrrc AFAICS right now it only contains:

* parser for DELETE (not enabled)

* new interpreter

* lots of unrelated changes

* does not contains any real implementation for DELETE

Am I right?

If so, and if you are going to keep that "lots of unrelated changes", and you really sure that they are worth it, how about moving them into separate PR, to make implementation of DELETE simpler for possible reviewers?

Hi! Currently, yes. But please keep in mind that this is a draft pull request, so by the time the first prototype is ready, all unrelated changes will be either:

  1. Moved out to separate PR's like optional<> semantics for parsing MergeTreePartInfo and DetachedPartInfo #28085 or
  2. Removed from this PR to keep everything clean

@alexey-milovidov
Copy link
Copy Markdown
Member

We can merge syntax part separately. The work was started in #12390.

@robot-ch-test-poll4 robot-ch-test-poll4 removed the submodule changed At least one submodule changed in this PR. label Sep 12, 2021
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Sep 28, 2021

CLA assistant check
All committers have signed the CLA.

@alexey-milovidov
Copy link
Copy Markdown
Member

@myrrc stopped working on this PR, will be continued by @Enmk

@zhangjmruc
Copy link
Copy Markdown
Contributor

@Enmk Do you have any plan for the work on the lightweight DELETE? Will this PR contains UPDATE also?

@weeds085490
Copy link
Copy Markdown
Contributor

@myrrc stopped working on this PR, will be continued by @Enmk

@alexey-milovidov Hi, we are developers from Tencent Cloud Database team. We've implemented light-weight deletion discussed in #19627 using the mutation mechanism. We look forward to contribute this back to the community. Can you assign this task to us? The following is a example of our implementation:

Test SQLs

DROP DATABASE IF EXISTS tencent_test on cluster default_cluster;
CREATE DATABASE tencent_test on cluster default_cluster;

CREATE TABLE tencent_test.t_1 on cluster default_cluster
(
    `order_0` UInt64,
    `ordinary_1` UInt32,
    `ordinary_2` String,
    `ordinary_3` UInt32,
    `p_time` Date,
	    `_version` UInt64
)
ENGINE = ReplicatedReplacingMergeTree(_version)
PARTITION BY toYYYYMM(p_time)
ORDER BY order_0;

CREATE TABLE tencent_test.t_1_all on cluster default_cluster  as tencent_test.t_1 ENGINE = Distributed(default_cluster, tencent_test, t_1,order_0);

CREATE TABLE tencent_test.t_random_1 
(
    `ordinary_1` UInt32,
    `ordinary_2` String,
    `ordinary_3` UInt32
)
ENGINE = GenerateRandom(1, 5, 3);

insert into tencent_test.t_1_all select rowNumberInAllBlocks(),*,NOW(), 1 from tencent_test.t_random_1 limit 100000000;

alter table tencent_test.t_1 delete where order_0 < 100;

Our implementation with light weight delete:
image

The original version without light weight delete :
image

@Enmk
Copy link
Copy Markdown
Contributor

Enmk commented Feb 22, 2022

@alexey-milovidov Hi, we are developers from Tencent Cloud Database team. We've implemented light-weight deletion discussed in #19627 using the mutation mechanism. We look forward to contribute this back to the community. Can you assign this task to us? The following is a example of our implementation:

That is some great news! IMO you should go ahead and submit a PR

@Slach
Copy link
Copy Markdown
Contributor

Slach commented Feb 22, 2022

@weeds085490 please submit PR

@weeds085490
Copy link
Copy Markdown
Contributor

@alexey-milovidov Hi, we are developers from Tencent Cloud Database team. We've implemented light-weight deletion discussed in #19627 using the mutation mechanism. We look forward to contribute this back to the community. Can you assign this task to us? The following is a example of our implementation:

That is some great news! IMO you should go ahead and submit a PR

Ok, since we're on the 21.3 kernel version here's some code cleanup and modification work. We will gradually merge the code into the community

@Enmk
Copy link
Copy Markdown
Contributor

Enmk commented Feb 22, 2022

@weeds085490 do you have any ETA on this?

Or you may even go ahead and push a PR as-is (e.g. into 21.3), and then the core team would be able to validate implementation. And I can help you with porting it atop of master to make things faster.

@weeds085490
Copy link
Copy Markdown
Contributor

@weeds085490 do you have any ETA on this?

Or you may even go ahead and push a PR as-is (e.g. into 21.3), and then the core team would be able to validate implementation. And I can help you with porting it atop of master to make things faster.

OK, we'll do some cleaning work , give our detailed design docs, and submit a PR first...

@dimovnike
Copy link
Copy Markdown

will this be using indexes as select does?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature Pull request with new product feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.