Skip to content

Merging (feature) Use Map data type for system logs tables#23934

Merged
kitaisreal merged 23 commits intoClickHouse:system-query-log-map-type-mergefrom
hexiaoting:system-querylog-map-continue
Jun 28, 2021
Merged

Merging (feature) Use Map data type for system logs tables#23934
kitaisreal merged 23 commits intoClickHouse:system-query-log-map-type-mergefrom
hexiaoting:system-querylog-map-continue

Conversation

@hexiaoting
Copy link
Copy Markdown
Contributor

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Merging #18762

Changelog category (leave one):

  • Backward Incompatible Change

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

  • Backward Incompatible Change:

Use Map data type for system logs tables (system.query_log, system.query_thread_log, system.processes, system.OpenTelemetrySpanLog).
These tables will be auto created with new datatypes, old queries maybe incompatible.
Close #18698

@robot-clickhouse robot-clickhouse added the pr-backward-incompatible Pull request with backwards incompatible changes label May 7, 2021
@hexiaoting hexiaoting changed the title (feature) Use Map data type for system logs tables——continue Merging (feature) Use Map data type for system logs tables May 7, 2021
@hexiaoting
Copy link
Copy Markdown
Contributor Author

Functional stateless tests (address) failed query: 01455_shard_leaf_max_rows_bytes_to_read.sql.
But I run this test case on my server, it's OK. And I don't change any code related to it.
Any advice?

@kitaisreal
Copy link
Copy Markdown
Contributor

@hexiaoting we recently checked and a lot of users rely on our system tables, for example there are tools that are not longer supported by maintainers that rely on some system.tables structure. As additional change we need to keep old columns as virtual storage columns. Check for example system.dictionaries. That way there will be no backward incompatible changes.

@hexiaoting
Copy link
Copy Markdown
Contributor Author

@kitaisreal
So for system tables, I should keep old columns like attributes.names and attributes.values as virtual columns, And also add a corresponding new Column attributes Map(String, String), right?

@kitaisreal
Copy link
Copy Markdown
Contributor

@hexiaoting yes, it will be best solution for us, because if we merge it as is, there will be issues that some tool that cannot be updated is commited to our system tables column names.

@hexiaoting
Copy link
Copy Markdown
Contributor Author

@hexiaoting yes, it will be best solution for us, because if we merge it as is, there will be issues that some tool that cannot be updated is commited to our system tables column names.

OK, I will rewrite the implementation.

@hexiaoting
Copy link
Copy Markdown
Contributor Author

@kitaisreal I have check system.dictionaries and found it has an interface called getVirtuals to set virtual columns.
But system tables like query_log, it is not implemented by basing IStorage, It is using std::make_shared<ASTCreateQuery>() to create, so it has no interface to set virtual columns.

Another way, If I using alias like:
attributes Map(String, String)
attribute.names alias mapKeys(attributes)
attribute.types alias mapValues(attributes)
I think It is redundant and not necessary.

So, any advice?

@hexiaoting
Copy link
Copy Markdown
Contributor Author

hexiaoting commented Jun 1, 2021

@kitaisreal I need your help. Can you give me some advices when you are free, Thank you.

@kitaisreal
Copy link
Copy Markdown
Contributor

@hexiaoting sorry for the delay, I need to investigate this. Maybe it will require some additional refactoring, but we cannot change logs tables in backward incompatible way, please check #23435.

@kitaisreal kitaisreal changed the base branch from master to system-query-log-map-type-merge June 28, 2021 08:26
@kitaisreal kitaisreal merged commit f96383c into ClickHouse:system-query-log-map-type-merge Jun 28, 2021
@kitaisreal kitaisreal mentioned this pull request Jun 28, 2021
azat added a commit to azat/ClickHouse that referenced this pull request Oct 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-backward-incompatible Pull request with backwards incompatible changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use Map data type for system logs tables.

5 participants