Skip to content

Lexer introduce heredoc#26671

Merged
kitaisreal merged 4 commits intoClickHouse:masterfrom
kitaisreal:mysql-dictionaries-support-for-custom-query
Jul 23, 2021
Merged

Lexer introduce heredoc#26671
kitaisreal merged 4 commits intoClickHouse:masterfrom
kitaisreal:mysql-dictionaries-support-for-custom-query

Conversation

@kitaisreal
Copy link
Copy Markdown
Contributor

@kitaisreal kitaisreal commented Jul 21, 2021

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Introduce syntax for here documents. Example SELECT $doc$VALUE$doc$.

@robot-clickhouse robot-clickhouse added doc-alert pr-feature Pull request with new product feature labels Jul 21, 2021
@kitaisreal kitaisreal force-pushed the mysql-dictionaries-support-for-custom-query branch from 06ce4e9 to 0b62434 Compare July 21, 2021 13:35
@kitaisreal kitaisreal force-pushed the mysql-dictionaries-support-for-custom-query branch from 0b62434 to 4998388 Compare July 21, 2021 13:42
@alexey-milovidov alexey-milovidov self-assigned this Jul 21, 2021
return false;
std::string_view here_doc(pos->begin, pos->size());
size_t heredoc_size = here_doc.find('$', 1) + 1;
s = String(pos->begin + heredoc_size, pos->size() - heredoc_size * 2);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add assert pls.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@@ -0,0 +1,4 @@
SELECT $$VALUE$$;
Copy link
Copy Markdown
Member

@alexey-milovidov alexey-milovidov Jul 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need a test with multiline values.
And also with dollar-something sequences in the middle.

$doc$

$do$
$ doc$
$doc $
$doco$

$doc$;

Also a test with multiple heredocs in single query.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also test with cyrillic/chinese characters - both for separators and for values.
A test with non-UTF8 binary data inside doc will be also nice to have.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also add .sh test to check how server reacts to unfinished heredocs.
Both unfinished separator and unfinished content. Zero and non-zero size.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@kitaisreal kitaisreal merged commit f6f8bea into ClickHouse:master Jul 23, 2021
SELECT $doc$VALUE$doc$;
SELECT $doc$'VALUE'$doc$;
SELECT $doc$$do$ $ doc$ $doc $ $doco$$doc$;
SELECT $doc$$do$ $ doc$ $doc $ $doco$$doc$, $doc$$do$ $ doc$ $doc $ $doco$$doc$;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No multiline test?
No test with non-UTF-8?

@sevirov
Copy link
Copy Markdown
Contributor

sevirov commented Jul 23, 2021

Internal documentation ticket: DOCSUP-12136

@alexey-milovidov
Copy link
Copy Markdown
Member

@kitaisreal Still waiting for test with multiline and non-UTF-8 entries.
Also normalizeQuery, normalizedQueryHash functions have to be adjusted.
Please also add a test for these functions.

@olgarev olgarev mentioned this pull request Sep 21, 2021
hazzadous pushed a commit to PostHog/posthog that referenced this pull request Mar 3, 2022
* fix(clickhouse): resolve 21.8 -> 21.9 test issues

This is the minimum change to make
ee/clickhouse/queries/funnels/test/test_funnel_unordered.py::TestFunnelUnorderedStepsBreakdown::test_funnel_aggregate_by_groups_breakdown_group
pass. There may be other tests still failing.

The issue was introduced with the [addition of
heredocs](ClickHouse/ClickHouse#26671) into the
clickhouse syntax. heredocs use `$` as delimiters. We're using `$` for
column identifiers in quite a few places, so now we need to ensure that
we have quoted these before rendering into SQL.

* update other snapshots

* update snapshots

* fix(clickhouse): quote materialized column identifiers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature Pull request with new product feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants