Skip to content

[fix](struct) Preserve nested field comments in SHOW CREATE TABLE - #68307

Merged
csun5285 merged 7 commits into
apache:masterfrom
csun5285:fix/struct-type-sql-roundtrip
Oct 20, 2026
Merged

csun5285 merged 7 commits into
apache:masterfrom
csun5285:fix/struct-type-sql-roundtrip

Conversation

@csun5285

@csun5285 csun5285 commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

CREATE TABLE t (id INT, s STRUCT<a:INT COMMENT 'field doc'>)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t;

Before: s is shown as struct<a:int>.

After: s is shown as struct<a:int comment "field doc">.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review at d1294dc070638282ffebb93260133f713d958482 found four blocking issues.

Checkpoint summary:

  • Goal and scope: the simple default-mode and lowercase-name cases work, but supported SQL modes and legal native/external field names remain incorrect. SHOW, CREATE TABLE LIKE, synced DDL, nested ARRAY/MAP recursion, and information_schema descriptor paths were traced.
  • Concurrency, lifecycle, configuration, persistence, and writes: no new lock, shared-state, configuration, row-write, or transaction issue was found. Interactive renderers must use the live SQL mode without making follower-side CCR generation thread-context-dependent.
  • Compatibility: the current patch does not change Thrift or persisted schema and therefore introduces no wire decode break. Rolling BEs can temporarily return old anonymous versus new named STRUCT text; carrying exact display names later should use an optional field with fallback.
  • Tests and results: the changed expected values are statically consistent for identifier-safe/default-mode cases, but coverage misses the two production counterexamples. The external results were derived rather than runner-generated, and the fixed DDL strings are absent from generated output.
  • Observability, performance, and security: no additional concern was found; the rendering work remains linear on metadata paths.

No builds or tests were run because the controlling review prompt prohibits execution in this runner. Author and CI claims were treated as non-independent evidence; the visible checks are style-oriented while functional jobs were skipped. User focus: no additional focus was supplied.

sb.append(type.hideVersionForVersionColumn(true));
// isToSql = true, showNestedComment = true
// SHOW CREATE TABLE and CREATE TABLE LIKE need the nested comment.
sb.append(type.hideVersionForVersionColumn(true, true, false));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Honor NO_BACKSLASH_ESCAPES when rendering comments

Passing false always doubles backslashes. With NO_BACKSLASH_ESCAPES, a stored nested comment containing one backslash is emitted with two; CreateTableLikeCommand then reparses that DDL with the session-aware lexer and stores both literally, so the copied table—and a client replaying SHOW text under that mode—no longer preserves the comment. Please thread the live mode into interactive SHOW/LIKE rendering (while keeping synced CCR output on an explicit canonical mode) and cover this mode in the round-trip test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改

ret += ",";
}
ret += _type_to_string(desc.children[desc.children.size() - 1]);
ret += desc.children[i].columnName + ":" + _type_to_string(desc.children[i]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve and quote struct field identifiers

columnName is the normalized lookup name, and concatenating it raw cannot represent every legal field. For example, Iceberg Metric is reported as metric, while a native field named a-b (declared with backticks) is emitted as struct<a-b:int(11)>, which cannot be parsed back as one field. Please carry the original/display spelling to this boundary, escape it as an SQL identifier, and add mixed-case plus quoted-name coverage.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改

s struct<string,bigint(20),string,string,string>
arr array<struct<bigint(20),int(11),int(11)>>
m map<string,struct<bigint(20),int(11),int(11)>>
s struct<first_pos:string,a:bigint(20),after_a:string,b:string,c2:string>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Regenerate the external expected results

The commit message says the two Iceberg/Spark .out updates were derived from the suite logic because the required environment was unavailable. Repository test rules require regression results to be generated by the owning test scripts, and static tracing cannot prove the refreshed external systems return these exact bytes. Please run both suites in the prescribed environment and commit their generated outputs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已测试

logger.info("SHOW CREATE TABLE struct_nested_comment: ${createStmt}")

// Substring checks, the full statement carries volatile properties that no .out can pin.
assertTrue(createStmt.contains("struct<a:int,b:text comment \"owner''s \\\\path\",c:int>"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Capture the fixed DDL expectations in generated output

These three fragments are deterministic user-visible results, but the new .out contains only the COLUMN_TYPE query. The regression rules require fixed expected results to go through the generated-output workflow instead of Groovy assertions. Please normalize/capture the stable SHOW/LIKE comment text in the suite's generated result; the runtime-derived replay/LIKE equality checks can remain dynamic assertions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改

@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review at 28ace2f71e255511aae981e78dd4df57d2349fbf found no new PR-introduced issue, but the previously raised field-identifier blocker remains unresolved, so I am retaining request changes without duplicating its inline comment.

Checkpoint summary:

  • Goal, correctness, and scope: the patch is focused, preserves nested comments in managed SHOW/CREATE LIKE DDL under the active SQL mode, and adds STRUCT field names to COLUMN_TYPE. However, exact head still sends normalized StructField.getName() rather than the original/display spelling and leaves hyphen, whitespace, and reserved-word field names bare, producing lossy or unparsable type text. This is already covered by the existing inline thread, so no duplicate inline comment was added.
  • Concurrency, lifecycle, and configuration: no new shared state, threads, lock-order edge, lifecycle object, or configuration item was found. The session mode is read once per table, sync rendering short-circuits to the canonical default mode, and CREATE TABLE LIKE parses after releasing the source-table read lock.
  • Compatibility, persistence, and writes: the Thrift shape and persisted catalog/journal formats are unchanged. Mixed-version BEs can temporarily return old anonymous versus new named STRUCT display text, but no request/decode break, transaction change, or data-write path was introduced.
  • Parallel paths and conditions: both managed Env emitters pass the mode consistently; ARRAY/MAP recursion, empty STRUCTs, generated-column restoration, and the sync path were traced. No additional PR-introduced replay or error-handling defect was substantiated.
  • Tests and results: the added JUnit/BE/regression coverage checks nested comments, default and NO_BACKSLASH_ESCAPES rendering, generated SHOW output, recursion, and deterministic metadata ordering. It still does not close the existing mixed-case/general-identifier gap noted above. This was a static-only review: no builds or tests were run because the controlling prompt prohibits them, and author/CI claims were not treated as independent functional validation.
  • Observability and performance: no new logging/metric need was found; the added work is bounded linear metadata formatting with one mode lookup per table.

User focus: no additional focus was supplied.

@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@csun5285

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 27566 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 28ace2f71e255511aae981e78dd4df57d2349fbf, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17714	3861	3775	3775
q2	3052	377	313	313
q3	10777	1339	847	847
q4	4904	480	349	349
q5	7979	814	534	534
q6	183	178	142	142
q7	747	772	587	587
q8	9874	1414	1591	1414
q9	6259	4233	4158	4158
q10	7335	1328	1014	1014
q11	738	272	243	243
q12	699	422	300	300
q13	18739	2602	2027	2027
q14	265	254	235	235
q15	q16	735	719	669	669
q17	1792	1162	1050	1050
q18	6506	5611	5526	5526
q19	1420	1278	1043	1043
q20	494	403	269	269
q21	5378	3010	2776	2776
q22	428	358	295	295
Total cold run time: 106018 ms
Total hot run time: 27566 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4220	4145	4132	4132
q2	704	538	529	529
q3	4459	4842	4307	4307
q4	2213	2296	1433	1433
q5	4295	4174	4391	4174
q6	225	171	133	133
q7	1694	1652	1567	1567
q8	2350	2090	2018	2018
q9	7380	7324	7242	7242
q10	3697	3643	3159	3159
q11	554	425	368	368
q12	746	717	517	517
q13	2527	2801	2143	2143
q14	312	289	279	279
q15	q16	713	737	621	621
q17	7683	7181	6742	6742
q18	11870	11081	11859	11081
q19	1141	1026	1075	1026
q20	2244	2214	1956	1956
q21	5183	4416	4538	4416
q22	535	496	431	431
Total cold run time: 64745 ms
Total hot run time: 58274 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 153435 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 28ace2f71e255511aae981e78dd4df57d2349fbf, data reload: false

query5	4349	615	477	477
query6	431	211	195	195
query7	4817	520	309	309
query8	326	182	159	159
query9	8818	4034	3976	3976
query10	511	309	252	252
query11	5894	3524	3254	3254
query12	149	93	84	84
query13	1250	587	415	415
query14	6506	4439	4267	4267
query14_1	3986	3952	3986	3952
query15	202	204	178	178
query16	3242	480	412	412
query17	1033	676	557	557
query18	2254	465	341	341
query19	207	252	175	175
query20	99	82	84	82
query21	870	145	123	123
query22	14109	13763	13582	13582
query23	14304	13247	12537	12537
query23_1	12476	12561	12550	12550
query24	7370	1131	665	665
query24_1	669	696	683	683
query25	555	423	368	368
query26	1479	324	163	163
query27	2439	519	331	331
query28	4518	2040	1991	1991
query29	1672	689	507	507
query30	390	199	182	182
query31	869	755	623	623
query32	143	89	90	89
query33	539	288	240	240
query34	1155	1136	635	635
query35	715	733	636	636
query36	780	792	692	692
query37	135	102	89	89
query38	1905	1773	1684	1684
query39	682	680	670	670
query39_1	638	660	653	653
query40	208	107	91	91
query41	62	60	57	57
query42	92	94	87	87
query43	333	340	310	310
query44	1348	720	733	720
query45	181	168	164	164
query46	1050	1234	718	718
query47	1491	1468	1422	1422
query48	410	398	300	300
query49	642	388	319	319
query50	940	345	258	258
query51	10576	10671	10287	10287
query52	87	89	72	72
query53	229	247	180	180
query54	245	200	184	184
query55	76	73	67	67
query56	220	203	203	203
query57	1575	1316	1340	1316
query58	277	255	250	250
query59	1995	2031	1856	1856
query60	275	239	222	222
query61	140	147	143	143
query62	403	313	279	279
query63	218	176	182	176
query64	2426	962	769	769
query65	3492	3426	3421	3421
query66	1728	399	293	293
query67	20109	19874	20050	19874
query68	3241	1475	936	936
query69	603	302	263	263
query70	902	845	795	795
query71	312	236	210	210
query72	2753	2456	2245	2245
query73	844	740	442	442
query74	4619	4463	4302	4302
query75	2291	2268	1925	1925
query76	1877	1066	770	770
query77	354	395	302	302
query78	9082	9150	8539	8539
query79	1404	1237	765	765
query80	581	451	370	370
query81	615	317	278	278
query82	603	202	124	124
query83	314	221	191	191
query84	316	137	108	108
query85	826	469	370	370
query86	350	234	223	223
query87	1992	1963	1824	1824
query88	3643	2764	2734	2734
query89	345	282	244	244
query90	1968	177	184	177
query91	165	153	128	128
query92	101	86	90	86
query93	1615	1489	948	948
query94	521	331	276	276
query95	669	451	327	327
query96	1008	810	342	342
query97	2460	2433	2350	2350
query98	161	146	143	143
query99	851	731	611	611
Total cold run time: 244339 ms
Total hot run time: 153435 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of 28ace2f71e255511aae981e78dd4df57d2349fbf: request changes. I found one new P2 in the CCR parse-mode contract and left it inline. The field-name/original-spelling defect is still present on this head, but it is already covered by discussion_r4059151457, so I did not duplicate it.

Critical checkpoint conclusions:

  • Goal and proof: Nested comments now round-trip for interactive SHOW/LIKE in the matching session mode, and normalized struct names are added to COLUMN_TYPE; the end-to-end goal is not complete because CCR can still change backslashes when the target parser mode differs.
  • Scope/focus: The patch is otherwise small and focused. There was no additional user-provided focus; the full ten-file diff was reviewed.
  • Concurrency: No new mutable shared state or lock ordering is introduced. Table metadata is rendered under the existing read lock; the sampled session mode is read-only.
  • Lifecycle/static initialization: No new background lifecycle, ownership cycle, cleanup path, or cross-TU/static-initialization dependency is introduced.
  • Configuration: No new configuration item is added, but the existing dynamic/global sql_mode is precisely the unresolved cross-cluster input called out inline.
  • Compatibility: The Thrift/storage shapes are unchanged and old/new FE-BE combinations remain decodable. The removed internal three-boolean overload has no supported separately versioned consumer found.
  • Parallel paths: SHOW, LIKE, sync serialization, both Env renderers, recursive ARRAY/MAP/STRUCT formatting, FE-to-BE metadata production, and the external CCR consumer were traced. The two Env sites share the same accepted defect.
  • Conditions/errors: Managed/external, sync/interactive, generated-placeholder, nested-comment-present, and old-descriptor branches were checked. Error propagation is unchanged; no new swallowed status or speculative fallback was found.
  • Tests and expected results: The BE/JUnit/regression cases and all modified golden sections were checked statically for registration, ownership, ordering, and expected shapes. They do not cover the differing CCR target mode. Existing output-provenance threads were not duplicated.
  • Observability: This is deterministic metadata rendering; no new metric or logging requirement was identified.
  • Persistence/transactions/data writes: No EditLog, storage, or transaction format changes are introduced. The only persisted boundary affected is the binlog-carried SQL text, whose consumer contract is the new P2.
  • FE-BE variables: No new FE-BE field is added; the existing child descriptor remains wire-compatible. Its original-name loss is the already-open identifier thread.
  • Performance/memory: Work remains bounded metadata-only string construction; no material hot-path, allocation-accounting, or lifetime regression was found.
  • Other: The PR changes user-visible SHOW and information_schema output, so the Release note: None / unchecked behavior-change metadata should also be corrected before merge.

Validation was static-only: the review task prohibits builds and tests, so author/CI execution claims were not independently rerun. Two complete review rounds converged, every candidate was adjudicated or duplicate-fenced, and no unresolved review candidate remains beyond the submitted issue and existing threads.

// session, and reading the mode per column would clone the whole SessionVariable each time.
// Synced DDL travels to another cluster, so pin it to the default mode instead, otherwise
// the master and a replaying follower would ship different text for the same table.
boolean noBackslashEscapes = !getDdlForSync && SqlModeHelper.hasNoBackSlashEscapes();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Pin the CCR parser to this canonical SQL mode

This forces synced DDL to ordinary backslash-escape mode, so one stored backslash is serialized as two. However, CreateTableRecord carries only the SQL, and the current CCR CreateTableOrView -> Spec.Exec path executes it through a pooled db.Exec without setting sql_mode. If the destination global mode includes NO_BACKSLASH_ESCAPES before CCR opens its connection, that session reads both backslashes literally and stores two, so nested comments still diverge across clusters. This is downstream of the earlier interactive-mode thread: generation is now canonical, but the consumer's parse context is not. Please pin ordinary mode and execute CREATE on the same physical connection (or use a mode-independent representation), then cover differing source/target modes.

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 28ace2f71e255511aae981e78dd4df57d2349fbf, data reload: false

query1	0.01	0.01	0.01
query2	0.30	0.05	0.05
query3	0.56	0.13	0.12
query4	1.87	0.13	0.13
query5	0.24	0.22	0.22
query6	1.41	0.93	0.95
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.60	0.33	0.33
query10	0.54	0.55	0.54
query11	0.28	0.14	0.14
query12	0.25	0.15	0.15
query13	0.46	0.48	0.46
query14	0.96	0.95	0.96
query15	0.61	0.58	0.59
query16	0.32	0.33	0.32
query17	1.08	1.06	1.10
query18	0.22	0.21	0.21
query19	2.10	1.98	1.95
query20	0.02	0.01	0.01
query21	16.51	0.21	0.13
query22	4.70	0.06	0.06
query23	17.97	0.30	0.12
query24	13.55	0.44	0.34
query25	0.13	0.05	0.04
query26	0.73	0.20	0.14
query27	0.05	0.04	0.03
query28	3.56	0.74	0.35
query29	12.53	4.01	3.21
query30	0.27	0.15	0.16
query31	2.80	0.57	0.32
query32	3.35	0.60	0.50
query33	3.19	3.16	3.26
query34	15.50	3.91	3.28
query35	3.23	3.20	3.22
query36	0.55	0.44	0.43
query37	0.20	0.07	0.06
query38	0.04	0.03	0.04
query39	0.11	0.03	0.03
query40	0.18	0.14	0.14
query41	0.22	0.03	0.03
query42	0.17	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 111.5 s
Total hot run time: 24.02 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.30% (34386/45068)
Line Coverage 61.19% (385972/630776)
Region Coverage 57.61% (325135/564385)
Branch Coverage 58.41% (148113/253593)

csun5285 and others added 4 commits October 18, 2026 08:44
Two ways a STRUCT loses information on its way back to the user.

SHOW CREATE TABLE dropped the COMMENT of every nested field. Column.toSql
called Type.hideVersionForVersionColumn(true), whose showNestedComment
parameter defaults to false, so the struct branch skipped the comment.
DESCRIBE passes the flag explicitly, so only the DDL emitters lost it.

That is not only a display defect. CreateTableLikeCommand re-parses the
generated statement, so CREATE TABLE LIKE built a table whose nested
comments were gone for good, and binlog/CreateTableRecord ships the same
text to CCR, leaving the downstream schema different from the source.
Backup and restore are unaffected, they serialize the Table object.

information_schema.COLUMNS.COLUMN_TYPE printed struct<int(11),string>:
the STRUCT branch of SchemaColumnsScanner::_type_to_string recursed into
the children but never printed their names, so no client could rebuild the
schema from that text. It now prints name:type per field. The loop also
moves off the "size() - 1" form that only avoided a size_t underflow
because of the surrounding empty check.

Four golden files record a struct COLUMN_TYPE and are updated. The two
under external_table_p0/iceberg were derived from the suites' own schema
evolution sequences and cross-checked against the types and the
element_at() row queries already in those files; they need a real Iceberg
and Spark environment to regenerate.

Fixes DORIS-28314, DORIS-28967.

Co-Authored-By: Claude Opus 5 <[email protected]>
Follow-up on review.

The comment was escaped for the default SQL mode whatever the session was
using. Under NO_BACKSLASH_ESCAPES a back slash was doubled on the way out
and read back as two, so every CREATE TABLE LIKE doubled it again. Read
the mode once per statement, outside the column loop: a synced statement
is built on the journal replay thread, which has no session, and reading
the mode per column would clone the whole SessionVariable each time.
Synced DDL is pinned to the default mode so the master and a replaying
follower ship the same text for one table.

COLUMN_TYPE concatenated the field name next to the separators that give
the text its shape, so a legal name holding one of them, declared with
back quotes, made the text impossible to split back into fields. Such a
name is now back quoted. A name with a hyphen, a space or a keyword
carries no meaning in this text and stays bare.

The two Iceberg golden files are the ones the suites generate, run
against a local iceberg docker stack; they match the values the earlier
commit derived from the suite logic. The fixed SHOW CREATE TABLE text now
goes through the generated output instead of Groovy assertions.

Co-Authored-By: Claude Opus 5 <[email protected]>
@csun5285
csun5285 force-pushed the fix/struct-type-sql-roundtrip branch from ca32519 to a9033f0 Compare October 18, 2026 00:45
### What problem does this PR solve?

Issue Number: N/A
Related PR: apache#68307

Problem Summary: Limit the change to preserving nested STRUCT field comments
in SHOW CREATE TABLE and CREATE TABLE LIKE. Restore the existing COLUMN_TYPE
output and remove its field-name quoting, display-name transport, parser
refactoring and metadata regression changes. Keep SQL-mode-aware comment
escaping and the focused DDL tests.

### Release note

None (scope reduction of an unmerged change).

### Check List (For Author)

- Test: TypeTest passed (21 tests) via run-fe-ut.sh; FE compilation and
  fe-catalog/fe-core Checkstyle passed. Regression suite not rerun locally.
- Behavior changed: Yes, restore existing information_schema metadata output.
- Does this need documentation: No.
@csun5285 csun5285 changed the title [fix](struct) Keep nested field comments and names [fix](struct) Preserve nested field comments in SHOW CREATE TABLE Oct 19, 2026
### What problem does this PR solve?

Related PR: apache#68307

Problem Summary: SHOW CREATE TABLE omits nested STRUCT field comments, so
replaying its DDL or using CREATE TABLE LIKE loses those comments. Enable
the existing nested-comment serializer option and retain its default
escaping behavior. Remove the added session SQL mode plumbing and its
mode-specific tests to keep this fix focused on comment preservation.

### Release note

Preserve nested STRUCT field comments in SHOW CREATE TABLE and CREATE TABLE LIKE.

### Check List (For Author)

- Test: Unit Test — TypeTest (20 tests); Checkstyle passed. Regression suite not rerun locally.
- Behavior changed: Yes — generated DDL includes nested field comments.
- Does this need documentation: No
@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all four changed paths at 4a7cf3bb356e6612c6f3bd742fc23af7242cd66d using the supplied aggregate diff and static analysis. One new P2 finding: the new full SHOW CREATE expectation fails in the supported Cloud p0 pipeline before the replay/LIKE checks.

Critical checkpoint conclusions:

  • Goal and proof: enabling the existing recursive renderer preserves nested comments in managed-table SHOW CREATE and CREATE TABLE LIKE under ordinary SQL mode. Three added JUnit tests cover formatting/escaping and the DESCRIBE flag; the regression includes nested STRUCT, ARRAY, replay and LIKE. Runtime success was not verified by this review.
  • Scope and reuse: the production change is one rendering flag and reuses existing type traversal; the remaining changes are tests/results.
  • Concurrency and locking: no new shared mutation, threads or lock acquisition. SHOW/LIKE use existing table read locks; the added work is local formatting without RPC, IO or journal waits.
  • Lifecycle and initialization: no new resources, ownership, globals or initialization-order dependency.
  • Configuration: no new settings. The existing interactive SQL-mode issue in thread 4059151453 still applies: the new call defaults to backslash escaping while LIKE parses with the session mode. It was independently confirmed and not reposted.
  • Compatibility: no function-signature, persisted schema or wire-format changes; existing nonempty stored comments are read by the current renderer. No new rolling-upgrade requirement identified.
  • Thrift/protobuf: untouched; no field-id or enum changes.
  • Parallel paths: SHOW and LIKE both reach the changed compatible renderer; ARRAY/MAP/STRUCT recursion forwards the comment flag. External non-compatible rendering and type-only DESCRIBE retain their existing behavior. The known CCR SQL-mode concern is covered by thread 4079713811 and was not reposted.
  • Conditional logic and errors: no new branch or error handling; omitted/null comments and explicit empty comments follow the existing presence flag and literal formatter.
  • Test coverage: the additions cover ordinary-mode literals and recursive STRUCT/ARRAY output. They do not add MAP, empty-comment, SQL-mode-switch or persisted-reload round trips. Static tracing found no additional local issue in those shapes; the mode-switch defect is already reported.
  • Expected results: unit expectations and CSV escaping agree with ordinary-mode non-Cloud defaults. The inline finding documents the concrete Cloud mismatch. The earlier generated-output concern in thread 4059151464 has been addressed by adding output, but that new full-row expectation introduces this separate failure. This review cannot attest how the author generated results.
  • Observability: this metadata-formatting change needs no additional logs or metrics.
  • Persistence and failover: no EditLog, replay or transaction processing change; existing comment metadata is reused.
  • Data writes and crash safety: no data-write or atomicity change, and no new leak/deadlock boundary; LIKE uses the existing CREATE execution path.
  • FE/BE propagation: no new transmitted variables.
  • Performance: necessary escaping is linear in comment length on metadata paths; no new catalog scans or query-execution work.
  • Other issues and focus: there was no additional user focus. All candidates and initial risks have accepted, duplicate or evidence-backed dismissed dispositions; no unresolved static-review scope remains. Existing supplied findings are P2, so there are no existing P0/P1 IDs to carry forward.

Validation limits: no builds, tests or runtime/CCR execution were performed, as required by the review contract. Author-reported test success is not treated as executed validation. The downstream CCR compatibility rewriter has a pre-existing limitation with comment-like text inside literals; it is outside this producer change and is not a new inline finding.

Review converged after 2 round(s).

"""

// SHOW CREATE TABLE used to drop every nested field comment.
qt_show_create "SHOW CREATE TABLE struct_nested_comment"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Normalize SHOW properties before recording the expected row

This suite runs in Cloud p0: its configuration selects p0 and excludes neither this suite nor datatype_p0. However, this qt_ call compares the entire DDL with an output containing non-Cloud replication properties. In Cloud mode, Env.addOlapTablePropertyInfo emits file_cache_ttl_seconds instead of replication_allocation/min_load_replica_num, and the framework does no automatic SHOW normalization. The suite therefore fails here before reaching either round-trip check, even when nested comments are correct. Capture the stable column/comment text with a row converter and regenerate the expected output.

### What problem does this PR solve?

Related PR: apache#68307

Problem Summary: Comparing the full SHOW CREATE TABLE statement makes the
nested-comment regression depend on Cloud replication/cache properties,
binlog settings and the default index format. Record the table name and
column definitions for SHOW CREATE, replayed DDL and CREATE TABLE LIKE,
while still replaying the complete generated statement.

### Release note

None

### Check List (For Author)

- Test: Regression test — generated output through run-regression-test.sh;
  verified with default settings and with binlog enabled/index format V2.
- Behavior changed: No production change
- Does this need documentation: No
@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all four changed files at b88a6a8. No new inline findings after duplicate filtering. Static tracing supports nested-comment preservation in ordinary-mode SHOW CREATE TABLE, replay, and CREATE TABLE LIKE.

The existing P2 SQL-mode finding still applies: Column selects the overload that doubles backslashes, while LIKE reparses using the live NO_BACKSLASH_ESCAPES mode. It was independently confirmed and not reposted. The Cloud output concern and fixed-result capture concern are addressed by the column-only converter and three golden blocks. The CCR consumer-mode thread remains prior context; its current external implementation was not revalidated. There are no existing P0/P1 findings to carry forward.

Critical checkpoint conclusions:

  • Goal and proof: the compatible renderer now includes stored nested comments; three unit methods and SHOW/replay/LIKE regression expectations cover the main ordinary-mode behavior. The known SQL-mode limitation above remains.
  • Scope and reuse: one production-line change reuses the established recursive renderer and literal quoting utility; supporting tests are focused.
  • Concurrency and locks: SHOW/LIKE read source metadata under existing table read locks; no shared mutation, lock-order change, external IO, or journal wait is added.
  • Lifecycle: temporary Java strings/builders introduce no new resource ownership, threads, caches, cycles, or static initialization.
  • Configuration: no configuration item is added or changed.
  • Compatibility: generated SQL gains existing COMMENT syntax; symbols, storage formats, and rolling-upgrade protocols are unchanged. SQL-mode compatibility is covered by the known thread.
  • Thrift/protobuf: no definitions, ids, enum values, or required fields change.
  • Parallel paths: SHOW and LIKE share the changed method; STRUCT/ARRAY/MAP recursion forwards comment inclusion. DESCRIBE keeps its explicit inclusion policy; existing external rendering is unaffected.
  • Conditions and errors: absent-comment gating is intentional; no new speculative fallback, swallowed error, or unchecked Status is introduced.
  • Test coverage: unit cases cover missing/present comments, quotes, backslashes and explicit display policy; regression checks nested STRUCT and ARRAY plus replay/LIKE. MAP, mode variants, empty comments and quoted identifiers lack new end-to-end cases; relevant mode/identifier concerns overlap existing context.
  • Changed expectations: all three golden rows, tags, indentation, SQL escaping and framework CSV escaping are statically consistent. The converter runs before both generation and comparison and excludes native/Cloud table properties.
  • Observability: existing tagged expected/actual test diagnostics are sufficient for this metadata-formatting change; no new metrics or logs are needed.
  • Transactions and persistence: comments already reside in schema metadata; no EditLog, replay, failover, visible-version or delete-bitmap implementation changes.
  • Data writes and crash behavior: no row-write or atomicity path changes; LIKE uses the existing CREATE implementation after parsing.
  • FE/BE variables: no new cross-process state or sending path.
  • Performance and memory: extra work is linear in emitted comment bytes and uses request-local Java objects; no new hot-path amplification or retained ownership was found.
  • Other concerns and user focus: no additional focus was supplied. A final edge-case sweep found no distinct introduced issue beyond the existing threads.

Validation limits: static review only; no builds or tests were run. HEAD reports output generation through run-regression-test.sh, but runtime success and generation provenance were not independently verified here. All source scope and risk candidates have explicit dispositions; no unresolved source candidate remains. All three assigned reviewers returned NO_NEW_VALUABLE_FINDINGS in round 1.

Review converged after 1 round(s).

@csun5285

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28791 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b88a6a88f3de1481412245052c7df10ae92a649d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17583	5187	5042	5042
q2	2106	319	268	268
q3	10241	1444	815	815
q4	4675	461	339	339
q5	7432	793	525	525
q6	180	187	152	152
q7	740	795	613	613
q8	9243	1324	1133	1133
q9	5506	4192	4179	4179
q10	6831	1282	1046	1046
q11	436	253	224	224
q12	634	397	274	274
q13	18114	3569	3008	3008
q14	275	271	245	245
q15	q16	746	710	650	650
q17	1195	764	571	571
q18	7555	6192	6146	6146
q19	1120	960	594	594
q20	393	339	227	227
q21	6096	2459	2490	2459
q22	387	336	281	281
Total cold run time: 101488 ms
Total hot run time: 28791 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5990	5856	6184	5856
q2	741	618	553	553
q3	5488	5725	5367	5367
q4	2051	2163	1575	1575
q5	5887	5860	5744	5744
q6	244	189	140	140
q7	2151	1967	1832	1832
q8	2986	2572	2652	2572
q9	7993	7958	7551	7551
q10	3636	3564	3204	3204
q11	577	400	373	373
q12	644	668	518	518
q13	3264	3581	3000	3000
q14	291	279	267	267
q15	q16	678	699	605	605
q17	7806	7049	6886	6886
q18	12922	12160	12892	12160
q19	919	785	777	777
q20	2143	2139	1908	1908
q21	5567	4554	4679	4554
q22	500	449	398	398
Total cold run time: 72478 ms
Total hot run time: 65840 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150204 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b88a6a88f3de1481412245052c7df10ae92a649d, data reload: false

query5	4313	591	439	439
query6	430	208	192	192
query7	4798	446	227	227
query8	322	173	176	173
query9	8795	3100	3117	3100
query10	483	348	249	249
query11	6037	3846	3533	3533
query12	142	87	86	86
query13	1234	433	313	313
query14	6115	3885	3647	3647
query14_1	3366	3367	3361	3361
query15	201	198	178	178
query16	947	410	386	386
query17	859	646	536	536
query18	2404	418	322	322
query19	186	165	130	130
query20	84	83	81	81
query21	208	125	103	103
query22	13107	12943	12745	12745
query23	13562	12867	12433	12433
query23_1	12540	12619	12624	12619
query24	7353	808	468	468
query24_1	461	464	457	457
query25	514	411	331	331
query26	1261	249	132	132
query27	2780	437	269	269
query28	4639	1936	1931	1931
query29	1551	588	434	434
query30	294	223	185	185
query31	1013	930	811	811
query32	149	91	95	91
query33	538	322	245	245
query34	914	857	481	481
query35	738	794	654	654
query36	819	813	717	717
query37	127	103	95	95
query38	1771	1758	1675	1675
query39	804	809	748	748
query39_1	739	720	732	720
query40	227	118	97	97
query41	70	67	69	67
query42	84	84	85	84
query43	344	365	309	309
query44	1295	685	687	685
query45	194	178	167	167
query46	857	962	570	570
query47	3014	3046	2513	2513
query48	297	315	227	227
query49	594	409	304	304
query50	696	273	216	216
query51	10609	10282	10190	10190
query52	82	85	73	73
query53	190	206	153	153
query54	249	203	207	203
query55	72	68	62	62
query56	247	241	218	218
query57	1642	1607	1567	1567
query58	284	279	264	264
query59	2126	2181	1960	1960
query60	298	256	233	233
query61	168	170	171	170
query62	387	335	308	308
query63	192	163	165	163
query64	2845	1045	899	899
query65	3377	3310	3267	3267
query66	1789	481	305	305
query67	20144	19887	19978	19887
query68	3020	1019	619	619
query69	398	288	260	260
query70	828	769	764	764
query71	297	233	212	212
query72	2719	2595	2297	2297
query73	509	522	294	294
query74	4607	4481	4334	4334
query75	2335	2287	1921	1921
query76	2314	1009	599	599
query77	342	386	275	275
query78	9926	9829	9175	9175
query79	967	906	513	513
query80	1284	419	347	347
query81	587	316	270	270
query82	807	131	107	107
query83	295	197	177	177
query84	312	119	105	105
query85	1044	426	377	377
query86	398	238	234	234
query87	2004	1939	1781	1781
query88	3719	2686	2661	2661
query89	337	278	238	238
query90	1885	155	168	155
query91	159	157	139	139
query92	98	86	89	86
query93	1054	997	582	582
query94	626	299	267	267
query95	565	385	294	294
query96	669	490	232	232
query97	2433	2401	2305	2305
query98	165	158	147	147
query99	754	761	643	643
Total cold run time: 236203 ms
Total hot run time: 150204 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b88a6a88f3de1481412245052c7df10ae92a649d, data reload: false

query1	0.01	0.01	0.01
query2	0.08	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.09	0.09
query5	0.23	0.24	0.23
query6	1.18	0.69	0.67
query7	0.04	0.00	0.00
query8	0.05	0.03	0.04
query9	0.40	0.33	0.33
query10	0.61	0.56	0.55
query11	0.20	0.14	0.15
query12	0.18	0.15	0.15
query13	0.48	0.47	0.48
query14	1.02	1.01	1.02
query15	0.63	0.62	0.60
query16	0.35	0.36	0.36
query17	1.16	1.16	1.15
query18	0.20	0.19	0.18
query19	2.13	2.02	2.00
query20	0.02	0.01	0.01
query21	15.46	0.18	0.13
query22	4.96	0.06	0.05
query23	16.27	0.33	0.12
query24	3.11	0.53	0.28
query25	0.14	0.05	0.05
query26	0.84	0.22	0.15
query27	0.05	0.04	0.04
query28	2.90	0.76	0.36
query29	12.49	4.18	3.28
query30	0.28	0.13	0.10
query31	2.76	0.55	0.33
query32	3.22	0.61	0.49
query33	3.19	3.29	3.26
query34	15.39	3.98	3.38
query35	3.34	3.33	3.35
query36	0.54	0.42	0.43
query37	0.08	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.15
query41	0.08	0.03	0.02
query42	0.04	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 96.28 s
Total hot run time: 24.24 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/42) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/98) 🎉
Increment coverage report
Complete coverage report

@csun5285
csun5285 merged commit 886cb3e into apache:master Oct 20, 2026
51 of 52 checks passed
@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

github-actions Bot pushed a commit that referenced this pull request Oct 20, 2026
…8307)

### What problem does this PR solve?

```sql
CREATE TABLE t (id INT, s STRUCT<a:INT COMMENT 'field doc'>)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t;
```

Before: `s` is shown as `struct<a:int>`.

After: `s` is shown as `struct<a:int comment "field doc">`.
yiguolei pushed a commit that referenced this pull request Oct 21, 2026
…TE TABLE #68307 (#69337)

Cherry-picked from #68307

---------

Co-authored-by: Chenyang Sun <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.2.0-merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants