Skip to content

Commit 910dabb

Browse files
morningmanclaude
andcommitted
[fix](nereids) Fix SQL masking and view rewrites after supplementary characters
### Background Several FE features rewrite the text of a statement by cutting the original SQL string at token positions from the parser. Audit masking replaces credential properties and passwords with `*XXX` before the statement reaches `fe.audit.log`, `__internal_schema.audit_log` and the FE log; CREATE / ALTER VIEW and CREATE MATERIALIZED VIEW expand `*` and qualify names in the definition they store; the parser reads `/*+ ... */` hints from comment tokens. The lexer counts its input by Unicode code point, while a Java `String` holds a supplementary character, such as an emoji, as two `char`s. ### What problem does this PR solve? Issue Number: None Related PR: #47595 (introduced the bug in audit masking), #32743 (introduced the bug in view rewriting), #70338 (its review found the masking leak) PR Type: Bugfix Problem Summary: After each supplementary character in a statement, these token positions point one `char` too early, so the text is cut at the wrong place. A credential after or inside such characters stays partly or fully in clear text in the audit log, CREATE VIEW fails with a syntax error, CREATE MATERIALIZED VIEW stores a broken query, and a hint after such a character is silently ignored. This PR converts the positions to `String` offsets, or reads the text through the token, at the five places that cut the original SQL. Tests are 89% of the diff. ### Details - **Change map:** production code +33 -33 in 3 files; tests +240 in 5 files (FE unit tests and one regression suite); generated `.out` +16. (1) `BaseViewInfo.rewriteSql`, shared by audit masking, views and MTMV, converts its positions; the sync-MV builder drops its private copy of it, and `rewriteProjectsToUserDefineAlias` replaces the select list through it. (2) `NereidsParser.getHintMap` reads hints from the token text. (3) `NereidsParser.tryParseExplainPlan` converts the start of the EXPLAIN PLAN body. - **Frequency:** Always, when a character above U+FFFF (an emoji, CJK Extension B) comes before or inside the rewritten part, for example a catalog COMMENT of emoji in front of `'s3.secret_key' = 'secret'`. - **Root cause:** `CaseInsensitiveStream.fromString` keeps ANTLR's code-point stream for any input with a surrogate (before #67452 every input used `CharStreams.fromString`), so `Token.getStartIndex()` and `getStopIndex()` count code points. The five places in the Change map used them as `String` indexes. - **Why existing tests missed it:** Unit tests: the masking (`EncryptSQLTest`, `RepositoryAuditEncryptionTest`), view and MTMV (`CreateViewTest`, `CreateMTMVCommandTest`) and hint tests (`LeanTokenModeTest`) use only BMP text, where both kinds of position coincide; `CaseInsensitiveStreamTest` uses an emoji but checks only the stream's own positions. Regression: `audit/test_external_catalog_credential_masking` and the `ddl_p0/create_view_nereids` suites have no supplementary characters, and the masking suite only checks that the whole credential is absent, which a leaked tail passes. - **Fix:** Convert where the text is cut. `rewriteSql` converts each position with `String.offsetByCodePoints`, walking the sorted positions once so the cost stays linear; `rewriteProjectsToUserDefineAlias` reads expressions through the token's input stream; `getHintMap` reads the hint from the token text, as `removeCommentAndTrimBlank` does. Changing the stream to report `String` positions was rejected: #67452 made code-point positions part of `CaseInsensitiveStream`'s contract, and `CaseInsensitiveStreamTest` pins them. - **Other paths and branches:** Fixed here: the 19 masking visitors of `LogicalPlanBuilderForEncryption` with every caller of `geneEncryptionSQL` (audit log, FE statement log, `_query_plan`) and the streaming insert job; the view, MTMV and sync-MV rewrites. Not affected: the other uses of token positions read text through the token's own stream (such as `getOriginSql`) or only compare positions with each other. branch-3.1, branch-4.0, branch-4.1 and branch-4.2 have the same code. Introduced by #32743 (views), #47595 (masking), #33699 (sync MV) and #33262 (EXPLAIN PLAN); hints since at least #39590. ### Testing - New unit tests fail on master `f1bc8b3690c` and pass with the fix, run locally: `EncryptSQLTest#testEncryptionAfterSupplementaryCharacters`, through `MysqlConnectProcessor` to the audit event (emoji inside a credential, in a catalog COMMENT, in the first of two property clauses, in a user name before `PASSWORD(...)`; master left `secret`, `abc`, `123` and `123456` in clear text); `CreateViewTest#testCreateViewRewritesAfterSupplementaryCharacters` (master: `mismatched input '`e`'`); `CreateMTMVCommandTest#testCreateMTMVRewritesQuerySqlAfterSupplementaryCharacters` (master kept ``SELECT '😀😀' A`e` e`internal`...``); `SupplementaryCharacterOffsetTest` (master dropped the hint, cut the EXPLAIN PLAN body to `AN SELECT 100`, and rewrote the sync MV to `avg_stateg(k)))`). - New regression suite `ddl_p0/create_view_nereids/test_create_view_supplementary_chars`: fails without the fix with the same syntax error and passes with it; the `.out` is generated by the script. Run locally on one FE and one BE, an older master FE build with the three changed classes swapped in. - 27 existing FE UT classes around the change (467 tests) pass. Reverting each part of the fix alone, moving the end of a replaced span by one, or not advancing the position cursor each fails a new test; the off-by-one also fails existing ASCII cases. - By hand on that cluster, without the fix: the secret stayed in `audit_log`, `/* 😀 */ select /*+ SET_VAR(query_timeout=1) */ sleep(3)` ran for 3 s, and an MTMV over `select '😀😀' as e, ...` was created but failed to refresh (`Nereids parse failed`). With the fix: `"s3.secret_key" = "*XXX")`, and the query times out after 1 s. ### Notes for Reviewers - `getHintMap` loses its `sql` parameter, which nothing reads any more; its three callers are in `NereidsParser`. - `rewriteProjectsToUserDefineAlias` replaces the select list through `rewriteSql` instead of `StringUtils.overlay`, so the conversion lives in one method. - Stored definitions are not repaired: an MTMV created by an affected version from such a query keeps its broken definition and has to be recreated. ### Release note Fix passwords and credentials left partly or fully unmasked in the audit log and the FE log when a supplementary character, such as an emoji, comes before or inside them. Also fix CREATE VIEW failing, CREATE MATERIALIZED VIEW storing a broken query, and hints such as SET_VAR being ignored after such a character; materialized views already created with a broken query must be recreated. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: the fix, the tests, and this description - Human review: The author reviewed part of the changes. - Prompts: After the review bot on #70338 reported the masking leak, the author confirmed it is an existing engine defect shared by all masking paths and asked for a separate engine PR that fixes it, followed by a reply on #70338 with the link. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent f1bc8b3 commit 910dabb

9 files changed

Lines changed: 289 additions & 33 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForSyncMv.java‎

Lines changed: 2 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
3434
import org.apache.doris.nereids.trees.plans.Plan;
3535
import org.apache.doris.nereids.trees.plans.commands.CreateMTMVCommand;
36+
import org.apache.doris.nereids.trees.plans.commands.info.BaseViewInfo;
3637
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
3738
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
3839
import org.apache.doris.nereids.util.PlanUtils;
@@ -109,7 +110,7 @@ public LogicalPlan visitQuery(DorisParser.QueryContext ctx) {
109110
for (Expression expr : outputs) {
110111
aggStateFunctionFinder.find(expr, indexInSqlToString);
111112
}
112-
querySql = Optional.of(rewriteSql(originSql, indexInSqlToString));
113+
querySql = Optional.of(BaseViewInfo.rewriteSql(indexInSqlToString, originSql));
113114
} else {
114115
querySql = Optional.of(originSql);
115116
}
@@ -193,18 +194,4 @@ public Expression visitUnboundFunction(UnboundFunction unboundFunction,
193194
return unboundFunction;
194195
}
195196
}
196-
197-
private static String rewriteSql(String querySql,
198-
Map<Pair<Integer, Integer>, String> indexStringSqlMap) {
199-
StringBuilder builder = new StringBuilder();
200-
int beg = 0;
201-
for (Map.Entry<Pair<Integer, Integer>, String> entry : indexStringSqlMap.entrySet()) {
202-
Pair<Integer, Integer> index = entry.getKey();
203-
builder.append(querySql, beg, index.first);
204-
builder.append(entry.getValue());
205-
beg = index.second + 1;
206-
}
207-
builder.append(querySql, beg, querySql.length());
208-
return builder.toString();
209-
}
210197
}

‎fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java‎

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -200,7 +200,10 @@ public static Optional<Pair<ExplainOptions, String>> tryParseExplainPlan(String
200200
return Optional.empty();
201201
}
202202
ExplainOptions explainOptions = new ExplainOptions(explainLevel, showPlanProcess);
203-
return Optional.of(Pair.of(explainOptions, sql.substring(explainPlanBody.getStartIndex())));
203+
// The parser counts a supplementary character such as an emoji as one character, while sql
204+
// counts it as two.
205+
return Optional.of(Pair.of(explainOptions,
206+
sql.substring(sql.offsetByCodePoints(0, explainPlanBody.getStartIndex()))));
204207
} catch (Throwable t) {
205208
return Optional.empty();
206209
}
@@ -349,7 +352,7 @@ private <T> T parse(String sql, @Nullable LogicalPlanBuilder logicalPlanBuilder,
349352
CommonTokenStream tokenStream = parseLeanTokens(sql);
350353
ParserRuleContext tree = toAst(tokenStream, parseFunction);
351354
LogicalPlanBuilder realLogicalPlanBuilder = logicalPlanBuilder == null
352-
? new LogicalPlanBuilder(getHintMap(sql, tokenStream, DorisParser::selectHint))
355+
? new LogicalPlanBuilder(getHintMap(tokenStream, DorisParser::selectHint))
353356
: logicalPlanBuilder;
354357
return (T) realLogicalPlanBuilder.visit(tree);
355358
}
@@ -358,7 +361,7 @@ public LogicalPlan parseForCreateView(String sql) {
358361
CommonTokenStream tokenStream = parseLeanTokens(sql);
359362
ParserRuleContext tree = toAst(tokenStream, DorisParser::singleStatement);
360363
LogicalPlanBuilder realLogicalPlanBuilder = new LogicalPlanBuilderForCreateView(
361-
getHintMap(sql, tokenStream, DorisParser::selectHint));
364+
getHintMap(tokenStream, DorisParser::selectHint));
362365
return (LogicalPlan) realLogicalPlanBuilder.visit(tree);
363366
}
364367

@@ -378,22 +381,24 @@ public Optional<String> parseForSyncMv(String sql) {
378381
CommonTokenStream tokenStream = parseLeanTokens(sql);
379382
ParserRuleContext tree = toAst(tokenStream, DorisParser::singleStatement);
380383
LogicalPlanBuilderForSyncMv logicalPlanBuilderForSyncMv = new LogicalPlanBuilderForSyncMv(
381-
getHintMap(sql, tokenStream, DorisParser::selectHint));
384+
getHintMap(tokenStream, DorisParser::selectHint));
382385
logicalPlanBuilderForSyncMv.visit(tree);
383386
return logicalPlanBuilderForSyncMv.getQuerySql();
384387
}
385388

386389
/** get hint map */
387-
public static Map<Integer, ParserRuleContext> getHintMap(String sql, CommonTokenStream hintTokenStream,
390+
public static Map<Integer, ParserRuleContext> getHintMap(CommonTokenStream hintTokenStream,
388391
Function<DorisParser, ParserRuleContext> parseFunction) {
389392
// parse hint first round
390393
Map<Integer, ParserRuleContext> selectHintMap = Maps.newHashMap();
391394

392395
Iterator<Token> tokenIterator = hintTokenStream.getTokens().iterator();
393396
Token hintToken = tokenIterator.hasNext() ? tokenIterator.next() : null;
394397
while (hintToken != null && hintToken.getType() != DorisLexer.EOF) {
395-
if (hintToken.getChannel() == 2 && sql.charAt(hintToken.getStartIndex() + 2) == '+') {
396-
String hintSql = sql.substring(hintToken.getStartIndex() + 3, hintToken.getStopIndex() + 1);
398+
// Token positions count a supplementary character such as an emoji as one character, so read
399+
// the hint from the token instead of cutting the SQL string.
400+
if (hintToken.getChannel() == 2 && hintToken.getText().startsWith("/*+")) {
401+
String hintSql = hintToken.getText().substring(3);
397402
DorisLexer newHintLexer = new DorisLexer(CaseInsensitiveStream.fromString(hintSql));
398403
CommonTokenStream newHintTokenStream = new CommonTokenStream(newHintLexer);
399404
DorisParser hintParser = new DorisParser(newHintTokenStream);

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java‎

Lines changed: 19 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -81,9 +81,9 @@
8181
import com.google.common.collect.Lists;
8282
import com.google.common.collect.Sets;
8383
import org.antlr.v4.runtime.ParserRuleContext;
84+
import org.antlr.v4.runtime.misc.Interval;
8485
import org.antlr.v4.runtime.tree.ParseTree;
8586
import org.antlr.v4.runtime.tree.RuleNode;
86-
import org.apache.commons.lang3.StringUtils;
8787

8888
import java.util.List;
8989
import java.util.Map;
@@ -128,18 +128,25 @@ protected void analyzeAndFillRewriteSqlMap(String sql, ConnectContext ctx) throw
128128
* Add the full path to the field
129129
*
130130
* @param indexStringSqlMap key is the start and end position of the sql substring that needs to be replaced,
131-
* and value is the new string used for replacement.
131+
* and value is the new string used for replacement. The positions are the token positions the
132+
* parser reports for querySql.
132133
* @param querySql origin query sql
133134
* @return sql rewritten sql
134135
*/
135136
public static String rewriteSql(TreeMap<Pair<Integer, Integer>, String> indexStringSqlMap, String querySql) {
136137
StringBuilder builder = new StringBuilder();
138+
// The parser counts a supplementary character such as an emoji as one character, while querySql
139+
// counts it as two. beg and begTokenPosition are the same place on both scales; the keys ascend,
140+
// so each position is converted by scanning forward from the previous one.
137141
int beg = 0;
142+
int begTokenPosition = 0;
138143
for (Map.Entry<Pair<Integer, Integer>, String> entry : indexStringSqlMap.entrySet()) {
139144
Pair<Integer, Integer> index = entry.getKey();
140-
builder.append(querySql, beg, index.first);
145+
int start = querySql.offsetByCodePoints(beg, index.first - begTokenPosition);
146+
builder.append(querySql, beg, start);
141147
builder.append(entry.getValue());
142-
beg = index.second + 1;
148+
beg = querySql.offsetByCodePoints(start, index.second + 1 - index.first);
149+
begTokenPosition = index.second + 1;
143150
}
144151
builder.append(querySql, beg, querySql.length());
145152
return builder.toString();
@@ -167,10 +174,11 @@ public static String rewriteProjectsToUserDefineAlias(String resSql, List<String
167174
List<NamedExpressionContext> namedExpressionContexts = finder.getNamedExpressionContexts();
168175
StringBuilder replaceWithColsBuilder = new StringBuilder();
169176
for (int i = 0; i < namedExpressionContexts.size(); ++i) {
170-
NamedExpressionContext namedExpressionContext = namedExpressionContexts.get(i);
171-
int start = namedExpressionContext.expression().start.getStartIndex();
172-
int stop = namedExpressionContext.expression().stop.getStopIndex();
173-
replaceWithColsBuilder.append(resSql, start, stop + 1);
177+
ParserRuleContext expression = namedExpressionContexts.get(i).expression();
178+
// Token positions count a supplementary character such as an emoji as one character, so read
179+
// the expression from the parser's input instead of cutting resSql.
180+
replaceWithColsBuilder.append(expression.start.getInputStream().getText(
181+
Interval.of(expression.start.getStartIndex(), expression.stop.getStopIndex())));
174182
replaceWithColsBuilder.append(" AS `");
175183
String escapeBacktick = finalColNames.get(i).replace("`", "``");
176184
replaceWithColsBuilder.append(escapeBacktick);
@@ -179,9 +187,9 @@ public static String rewriteProjectsToUserDefineAlias(String resSql, List<String
179187
replaceWithColsBuilder.append(", ");
180188
}
181189
}
182-
String replaceWithCols = replaceWithColsBuilder.toString();
183-
return StringUtils.overlay(resSql, replaceWithCols, finder.getIndex().first,
184-
finder.getIndex().second + 1);
190+
TreeMap<Pair<Integer, Integer>, String> selectColumns = new TreeMap<>(new Pair.PairComparator<>());
191+
selectColumns.put(finder.getIndex(), replaceWithColsBuilder.toString());
192+
return rewriteSql(selectColumns, resSql);
185193
}
186194

187195
protected void createFinalCols(List<Slot> outputs) throws org.apache.doris.common.AnalysisException {

‎fe/fe-core/src/test/java/org/apache/doris/catalog/CreateViewTest.java‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,26 @@ public void testCreateViewWithDefinedColumnsRewritesAliases() throws Exception {
268268
view.getInlineViewDef());
269269
}
270270

271+
@Test
272+
public void testCreateViewRewritesAfterSupplementaryCharacters() throws Exception {
273+
// The parser counts a supplementary character such as an emoji as one character, while
274+
// the Java string of the view definition counts it as two.
275+
ExceptionChecker.expectThrowsNoException(() -> createView(
276+
"create view test.supplementary_char_view as select '😀😀' as e, k1 from test.tbl1;"));
277+
ExceptionChecker.expectThrowsNoException(() -> createView(
278+
"create view test.supplementary_char_alias_view(c1, c2) as select '😀😀', k1 from test.tbl1;"));
279+
280+
Database db = Env.getCurrentInternalCatalog().getDbOrDdlException("test");
281+
View view = (View) db.getTableOrDdlException("supplementary_char_view");
282+
Assertions.assertEquals(
283+
"select '😀😀' as `e`, `internal`.`test`.`tbl1`.`k1` from `internal`.`test`.`tbl1`",
284+
view.getInlineViewDef());
285+
View aliasView = (View) db.getTableOrDdlException("supplementary_char_alias_view");
286+
Assertions.assertEquals(
287+
"select '😀😀' AS `c1`, `internal`.`test`.`tbl1`.`k1` AS `c2` from `internal`.`test`.`tbl1`",
288+
aliasView.getInlineViewDef());
289+
}
290+
271291
@Test
272292
public void testViewPreservesVarbinary() throws Exception {
273293
createView("create view test.vb_view as select X'AB' as vb;");

‎fe/fe-core/src/test/java/org/apache/doris/nereids/parser/EncryptSQLTest.java‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -505,6 +505,98 @@ public void testAlterUserPasswordMasking() throws Exception {
505505
}
506506
}
507507

508+
@Test
509+
public void testEncryptionAfterSupplementaryCharacters() throws Exception {
510+
ctx.setDatabase("test");
511+
try (MockedConstruction<StmtExecutor> ignored = Mockito.mockConstruction(StmtExecutor.class,
512+
Mockito.withSettings().defaultAnswer(Mockito.CALLS_REAL_METHODS),
513+
(mock, context) -> {
514+
Mockito.doReturn(false).when(mock).isForwardToMaster();
515+
Profile profile = new Profile(false, 0, 0);
516+
Deencapsulation.setField(mock, "profile", profile);
517+
Mockito.doReturn(profile).when(mock).getProfile();
518+
Mockito.doReturn(ctx).when(mock).getContext();
519+
Mockito.doNothing().when(mock).execute();
520+
Deencapsulation.setField(mock, "context", ctx);
521+
if (context.arguments().size() >= 2
522+
&& context.arguments().get(1) instanceof StatementBase) {
523+
Deencapsulation.setField(mock, "parsedStmt",
524+
context.arguments().get(1));
525+
}
526+
if (ctx.getStatementContext() == null) {
527+
ctx.setStatementContext(new StatementContext());
528+
}
529+
})) {
530+
ctx.setEnv(env);
531+
ctx.setCurrentUserIdentity(UserIdentity.ROOT);
532+
// The parser counts a supplementary character such as an emoji as one character, while
533+
// the Java string of the SQL counts it as two.
534+
String sql = "CREATE CATALOG ctl "
535+
+ "PROPERTIES("
536+
+ " \"type\" = \"iceberg\","
537+
+ " \"iceberg.catalog.type\" = \"hadoop\","
538+
+ " \"warehouse\" = \"s3://bucket/dir/key\","
539+
+ " \"s3.access_key\" = \"abc\","
540+
+ " \"s3.secret_key\" = \"😀😀😀😀😀😀😀secret\""
541+
+ ")";
542+
String res = "CREATE CATALOG ctl "
543+
+ "PROPERTIES("
544+
+ " \"type\" = \"iceberg\","
545+
+ " \"iceberg.catalog.type\" = \"hadoop\","
546+
+ " \"warehouse\" = \"s3://bucket/dir/key\","
547+
+ " \"s3.access_key\" = \"*XXX\","
548+
+ " \"s3.secret_key\" = \"*XXX\""
549+
+ ")";
550+
parseAndCheck(sql, res);
551+
552+
sql = "CREATE CATALOG ctl COMMENT \"😀😀😀😀😀😀😀😀\" "
553+
+ "PROPERTIES("
554+
+ " \"type\" = \"iceberg\","
555+
+ " \"iceberg.catalog.type\" = \"hadoop\","
556+
+ " \"warehouse\" = \"s3://bucket/😀/key\","
557+
+ " \"s3.access_key\" = \"abc\","
558+
+ " \"s3.secret_key\" = \"abc\""
559+
+ ")";
560+
res = "CREATE CATALOG ctl COMMENT \"😀😀😀😀😀😀😀😀\" "
561+
+ "PROPERTIES("
562+
+ " \"type\" = \"iceberg\","
563+
+ " \"iceberg.catalog.type\" = \"hadoop\","
564+
+ " \"warehouse\" = \"s3://bucket/😀/key\","
565+
+ " \"s3.access_key\" = \"*XXX\","
566+
+ " \"s3.secret_key\" = \"*XXX\""
567+
+ ")";
568+
parseAndCheck(sql, res);
569+
570+
sql = "CREATE EXTERNAL TABLE broker_tbl("
571+
+ " k1 tinyint) "
572+
+ "ENGINE=broker "
573+
+ "PROPERTIES("
574+
+ " \"broker_name\" = \"hdfs\","
575+
+ " \"path\" = \"hdfs://abc/😀😀😀😀😀😀😀😀/a.txt\""
576+
+ ") "
577+
+ "BROKER PROPERTIES("
578+
+ " \"username\" = \"root\","
579+
+ " \"password\" = \"123\""
580+
+ ")";
581+
res = "CREATE EXTERNAL TABLE broker_tbl("
582+
+ " k1 tinyint) "
583+
+ "ENGINE=broker "
584+
+ "PROPERTIES("
585+
+ " \"broker_name\" = \"hdfs\","
586+
+ " \"path\" = \"hdfs://abc/😀😀😀😀😀😀😀😀/a.txt\""
587+
+ ") "
588+
+ "BROKER PROPERTIES("
589+
+ " \"username\" = \"root\","
590+
+ " \"password\" = \"*XXX\""
591+
+ ")";
592+
parseAndCheck(sql, res);
593+
594+
sql = "SET PASSWORD FOR '😀😀😀😀😀😀😀😀😀' = PASSWORD('123456')";
595+
res = "SET PASSWORD FOR '😀😀😀😀😀😀😀😀😀' = PASSWORD('*XXX')";
596+
parseAndCheck(sql, res);
597+
}
598+
}
599+
508600
private void parseAndCheck(String sql, String expected) throws Exception {
509601
processor.executeQuery(sql);
510602
AuditEvent event = auditEvents.get(auditEvents.size() - 1);
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.nereids.parser;
19+
20+
import org.apache.doris.nereids.properties.SelectHintOrdered;
21+
import org.apache.doris.nereids.trees.plans.Plan;
22+
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
23+
import org.apache.doris.nereids.trees.plans.logical.LogicalSelectHint;
24+
25+
import org.junit.jupiter.api.Assertions;
26+
import org.junit.jupiter.api.Test;
27+
28+
import java.util.List;
29+
30+
/**
31+
* The parser counts a supplementary character such as an emoji as one character, while the Java
32+
* string of the SQL counts it as two. These cases cut the SQL text after such characters.
33+
*/
34+
class SupplementaryCharacterOffsetTest extends ParserTestBase {
35+
private final NereidsParser parser = new NereidsParser();
36+
37+
@Test
38+
void recognizesHintAfterSupplementaryCharacter() {
39+
LogicalPlan plan = parser.parseSingle("/* 😀 */ SELECT /*+ ORDERED */ a FROM t");
40+
List<LogicalSelectHint<Plan>> hintPlans = plan.collectToList(LogicalSelectHint.class::isInstance);
41+
42+
Assertions.assertEquals(1, hintPlans.size());
43+
Assertions.assertTrue(hintPlans.get(0).getHints().stream()
44+
.anyMatch(SelectHintOrdered.class::isInstance));
45+
}
46+
47+
@Test
48+
void extractsExplainPlanBodyAfterSupplementaryCharacters() {
49+
Assertions.assertEquals("SELECT 100",
50+
NereidsParser.tryParseExplainPlan("EXPLAIN /* abc */ PLAN SELECT 100").orElseThrow().second);
51+
Assertions.assertEquals("SELECT 100",
52+
NereidsParser.tryParseExplainPlan("EXPLAIN /* 😀😀😀 */ PLAN SELECT 100").orElseThrow().second);
53+
}
54+
55+
@Test
56+
void rewritesSyncMvAggregateAfterSupplementaryCharacter() {
57+
Assertions.assertEquals("SELECT /* a */ k1, avg_union(avg_state(k2)) FROM t GROUP BY k1",
58+
parser.parseForSyncMv("CREATE MATERIALIZED VIEW mv AS "
59+
+ "SELECT /* a */ k1, avg(k2) FROM t GROUP BY k1").orElseThrow());
60+
Assertions.assertEquals("SELECT /* 😀 */ k1, avg_union(avg_state(k2)) FROM t GROUP BY k1",
61+
parser.parseForSyncMv("CREATE MATERIALIZED VIEW mv AS "
62+
+ "SELECT /* 😀 */ k1, avg(k2) FROM t GROUP BY k1").orElseThrow());
63+
}
64+
}

0 commit comments

Comments
 (0)