Repository navigation
[fix](struct) Preserve nested field comments in SHOW CREATE TABLE - #68307
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
There was a problem hiding this comment.
Static review at d1294dc070638282ffebb93260133f713d958482 found four blocking issues.
Checkpoint summary:
- Goal and scope: the simple default-mode and lowercase-name cases work, but supported SQL modes and legal native/external field names remain incorrect. SHOW, CREATE TABLE LIKE, synced DDL, nested ARRAY/MAP recursion, and information_schema descriptor paths were traced.
- Concurrency, lifecycle, configuration, persistence, and writes: no new lock, shared-state, configuration, row-write, or transaction issue was found. Interactive renderers must use the live SQL mode without making follower-side CCR generation thread-context-dependent.
- Compatibility: the current patch does not change Thrift or persisted schema and therefore introduces no wire decode break. Rolling BEs can temporarily return old anonymous versus new named STRUCT text; carrying exact display names later should use an optional field with fallback.
- Tests and results: the changed expected values are statically consistent for identifier-safe/default-mode cases, but coverage misses the two production counterexamples. The external results were derived rather than runner-generated, and the fixed DDL strings are absent from generated output.
- Observability, performance, and security: no additional concern was found; the rendering work remains linear on metadata paths.
No builds or tests were run because the controlling review prompt prohibits execution in this runner. Author and CI claims were treated as non-independent evidence; the visible checks are style-oriented while functional jobs were skipped. User focus: no additional focus was supplied.
| sb.append(type.hideVersionForVersionColumn(true)); | ||
| // isToSql = true, showNestedComment = true | ||
| // SHOW CREATE TABLE and CREATE TABLE LIKE need the nested comment. | ||
| sb.append(type.hideVersionForVersionColumn(true, true, false)); |
There was a problem hiding this comment.
[P2] Honor NO_BACKSLASH_ESCAPES when rendering comments
Passing false always doubles backslashes. With NO_BACKSLASH_ESCAPES, a stored nested comment containing one backslash is emitted with two; CreateTableLikeCommand then reparses that DDL with the session-aware lexer and stores both literally, so the copied table—and a client replaying SHOW text under that mode—no longer preserves the comment. Please thread the live mode into interactive SHOW/LIKE rendering (while keeping synced CCR output on an explicit canonical mode) and cover this mode in the round-trip test.
| ret += ","; | ||
| } | ||
| ret += _type_to_string(desc.children[desc.children.size() - 1]); | ||
| ret += desc.children[i].columnName + ":" + _type_to_string(desc.children[i]); |
There was a problem hiding this comment.
[P2] Preserve and quote struct field identifiers
columnName is the normalized lookup name, and concatenating it raw cannot represent every legal field. For example, Iceberg Metric is reported as metric, while a native field named a-b (declared with backticks) is emitted as struct<a-b:int(11)>, which cannot be parsed back as one field. Please carry the original/display spelling to this boundary, escape it as an SQL identifier, and add mixed-case plus quoted-name coverage.
| s struct<string,bigint(20),string,string,string> | ||
| arr array<struct<bigint(20),int(11),int(11)>> | ||
| m map<string,struct<bigint(20),int(11),int(11)>> | ||
| s struct<first_pos:string,a:bigint(20),after_a:string,b:string,c2:string> |
There was a problem hiding this comment.
[P2] Regenerate the external expected results
The commit message says the two Iceberg/Spark .out updates were derived from the suite logic because the required environment was unavailable. Repository test rules require regression results to be generated by the owning test scripts, and static tracing cannot prove the refreshed external systems return these exact bytes. Please run both suites in the prescribed environment and commit their generated outputs.
| logger.info("SHOW CREATE TABLE struct_nested_comment: ${createStmt}") | ||
|
|
||
| // Substring checks, the full statement carries volatile properties that no .out can pin. | ||
| assertTrue(createStmt.contains("struct<a:int,b:text comment \"owner''s \\\\path\",c:int>")) |
There was a problem hiding this comment.
[P2] Capture the fixed DDL expectations in generated output
These three fragments are deterministic user-visible results, but the new .out contains only the COLUMN_TYPE query. The regression rules require fixed expected results to go through the generated-output workflow instead of Groovy assertions. Please normalize/capture the stable SHOW/LIKE comment text in the suite's generated result; the runtime-derived replay/LIKE equality checks can remain dynamic assertions.
|
/review |
There was a problem hiding this comment.
Static review at 28ace2f71e255511aae981e78dd4df57d2349fbf found no new PR-introduced issue, but the previously raised field-identifier blocker remains unresolved, so I am retaining request changes without duplicating its inline comment.
Checkpoint summary:
- Goal, correctness, and scope: the patch is focused, preserves nested comments in managed SHOW/CREATE LIKE DDL under the active SQL mode, and adds STRUCT field names to
COLUMN_TYPE. However, exact head still sends normalizedStructField.getName()rather than the original/display spelling and leaves hyphen, whitespace, and reserved-word field names bare, producing lossy or unparsable type text. This is already covered by the existing inline thread, so no duplicate inline comment was added. - Concurrency, lifecycle, and configuration: no new shared state, threads, lock-order edge, lifecycle object, or configuration item was found. The session mode is read once per table, sync rendering short-circuits to the canonical default mode, and CREATE TABLE LIKE parses after releasing the source-table read lock.
- Compatibility, persistence, and writes: the Thrift shape and persisted catalog/journal formats are unchanged. Mixed-version BEs can temporarily return old anonymous versus new named STRUCT display text, but no request/decode break, transaction change, or data-write path was introduced.
- Parallel paths and conditions: both managed Env emitters pass the mode consistently; ARRAY/MAP recursion, empty STRUCTs, generated-column restoration, and the sync path were traced. No additional PR-introduced replay or error-handling defect was substantiated.
- Tests and results: the added JUnit/BE/regression coverage checks nested comments, default and
NO_BACKSLASH_ESCAPESrendering, generated SHOW output, recursion, and deterministic metadata ordering. It still does not close the existing mixed-case/general-identifier gap noted above. This was a static-only review: no builds or tests were run because the controlling prompt prohibits them, and author/CI claims were not treated as independent functional validation. - Observability and performance: no new logging/metric need was found; the added work is bounded linear metadata formatting with one mode lookup per table.
User focus: no additional focus was supplied.
|
/review |
|
run buildall |
TPC-H: Total hot run time: 27566 ms |
TPC-DS: Total hot run time: 153435 ms |
There was a problem hiding this comment.
Static review of 28ace2f71e255511aae981e78dd4df57d2349fbf: request changes. I found one new P2 in the CCR parse-mode contract and left it inline. The field-name/original-spelling defect is still present on this head, but it is already covered by discussion_r4059151457, so I did not duplicate it.
Critical checkpoint conclusions:
- Goal and proof: Nested comments now round-trip for interactive SHOW/LIKE in the matching session mode, and normalized struct names are added to
COLUMN_TYPE; the end-to-end goal is not complete because CCR can still change backslashes when the target parser mode differs. - Scope/focus: The patch is otherwise small and focused. There was no additional user-provided focus; the full ten-file diff was reviewed.
- Concurrency: No new mutable shared state or lock ordering is introduced. Table metadata is rendered under the existing read lock; the sampled session mode is read-only.
- Lifecycle/static initialization: No new background lifecycle, ownership cycle, cleanup path, or cross-TU/static-initialization dependency is introduced.
- Configuration: No new configuration item is added, but the existing dynamic/global
sql_modeis precisely the unresolved cross-cluster input called out inline. - Compatibility: The Thrift/storage shapes are unchanged and old/new FE-BE combinations remain decodable. The removed internal three-boolean overload has no supported separately versioned consumer found.
- Parallel paths: SHOW, LIKE, sync serialization, both Env renderers, recursive ARRAY/MAP/STRUCT formatting, FE-to-BE metadata production, and the external CCR consumer were traced. The two Env sites share the same accepted defect.
- Conditions/errors: Managed/external, sync/interactive, generated-placeholder, nested-comment-present, and old-descriptor branches were checked. Error propagation is unchanged; no new swallowed status or speculative fallback was found.
- Tests and expected results: The BE/JUnit/regression cases and all modified golden sections were checked statically for registration, ownership, ordering, and expected shapes. They do not cover the differing CCR target mode. Existing output-provenance threads were not duplicated.
- Observability: This is deterministic metadata rendering; no new metric or logging requirement was identified.
- Persistence/transactions/data writes: No EditLog, storage, or transaction format changes are introduced. The only persisted boundary affected is the binlog-carried SQL text, whose consumer contract is the new P2.
- FE-BE variables: No new FE-BE field is added; the existing child descriptor remains wire-compatible. Its original-name loss is the already-open identifier thread.
- Performance/memory: Work remains bounded metadata-only string construction; no material hot-path, allocation-accounting, or lifetime regression was found.
- Other: The PR changes user-visible SHOW and
information_schemaoutput, so theRelease note: None/ unchecked behavior-change metadata should also be corrected before merge.
Validation was static-only: the review task prohibits builds and tests, so author/CI execution claims were not independently rerun. Two complete review rounds converged, every candidate was adjudicated or duplicate-fenced, and no unresolved review candidate remains beyond the submitted issue and existing threads.
| // session, and reading the mode per column would clone the whole SessionVariable each time. | ||
| // Synced DDL travels to another cluster, so pin it to the default mode instead, otherwise | ||
| // the master and a replaying follower would ship different text for the same table. | ||
| boolean noBackslashEscapes = !getDdlForSync && SqlModeHelper.hasNoBackSlashEscapes(); |
There was a problem hiding this comment.
[P2] Pin the CCR parser to this canonical SQL mode
This forces synced DDL to ordinary backslash-escape mode, so one stored backslash is serialized as two. However, CreateTableRecord carries only the SQL, and the current CCR CreateTableOrView -> Spec.Exec path executes it through a pooled db.Exec without setting sql_mode. If the destination global mode includes NO_BACKSLASH_ESCAPES before CCR opens its connection, that session reads both backslashes literally and stores two, so nested comments still diverge across clusters. This is downstream of the earlier interactive-mode thread: generation is now canonical, but the consumer's parse context is not. Please pin ordinary mode and execute CREATE on the same physical connection (or use a mode-independent representation), then cover differing source/target modes.
ClickBench: Total hot run time: 24.02 s |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
28ace2f to
ca32519
Compare
Two ways a STRUCT loses information on its way back to the user. SHOW CREATE TABLE dropped the COMMENT of every nested field. Column.toSql called Type.hideVersionForVersionColumn(true), whose showNestedComment parameter defaults to false, so the struct branch skipped the comment. DESCRIBE passes the flag explicitly, so only the DDL emitters lost it. That is not only a display defect. CreateTableLikeCommand re-parses the generated statement, so CREATE TABLE LIKE built a table whose nested comments were gone for good, and binlog/CreateTableRecord ships the same text to CCR, leaving the downstream schema different from the source. Backup and restore are unaffected, they serialize the Table object. information_schema.COLUMNS.COLUMN_TYPE printed struct<int(11),string>: the STRUCT branch of SchemaColumnsScanner::_type_to_string recursed into the children but never printed their names, so no client could rebuild the schema from that text. It now prints name:type per field. The loop also moves off the "size() - 1" form that only avoided a size_t underflow because of the surrounding empty check. Four golden files record a struct COLUMN_TYPE and are updated. The two under external_table_p0/iceberg were derived from the suites' own schema evolution sequences and cross-checked against the types and the element_at() row queries already in those files; they need a real Iceberg and Spark environment to regenerate. Fixes DORIS-28314, DORIS-28967. Co-Authored-By: Claude Opus 5 <[email protected]>
Follow-up on review. The comment was escaped for the default SQL mode whatever the session was using. Under NO_BACKSLASH_ESCAPES a back slash was doubled on the way out and read back as two, so every CREATE TABLE LIKE doubled it again. Read the mode once per statement, outside the column loop: a synced statement is built on the journal replay thread, which has no session, and reading the mode per column would clone the whole SessionVariable each time. Synced DDL is pinned to the default mode so the master and a replaying follower ship the same text for one table. COLUMN_TYPE concatenated the field name next to the separators that give the text its shape, so a legal name holding one of them, declared with back quotes, made the text impossible to split back into fields. Such a name is now back quoted. A name with a hyphen, a space or a keyword carries no meaning in this text and stays bare. The two Iceberg golden files are the ones the suites generate, run against a local iceberg docker stack; they match the values the earlier commit derived from the suite logic. The fixed SHOW CREATE TABLE text now goes through the generated output instead of Groovy assertions. Co-Authored-By: Claude Opus 5 <[email protected]>
ca32519 to
a9033f0
Compare
### What problem does this PR solve? Issue Number: N/A Related PR: apache#68307 Problem Summary: Limit the change to preserving nested STRUCT field comments in SHOW CREATE TABLE and CREATE TABLE LIKE. Restore the existing COLUMN_TYPE output and remove its field-name quoting, display-name transport, parser refactoring and metadata regression changes. Keep SQL-mode-aware comment escaping and the focused DDL tests. ### Release note None (scope reduction of an unmerged change). ### Check List (For Author) - Test: TypeTest passed (21 tests) via run-fe-ut.sh; FE compilation and fe-catalog/fe-core Checkstyle passed. Regression suite not rerun locally. - Behavior changed: Yes, restore existing information_schema metadata output. - Does this need documentation: No.
### What problem does this PR solve? Related PR: apache#68307 Problem Summary: SHOW CREATE TABLE omits nested STRUCT field comments, so replaying its DDL or using CREATE TABLE LIKE loses those comments. Enable the existing nested-comment serializer option and retain its default escaping behavior. Remove the added session SQL mode plumbing and its mode-specific tests to keep this fix focused on comment preservation. ### Release note Preserve nested STRUCT field comments in SHOW CREATE TABLE and CREATE TABLE LIKE. ### Check List (For Author) - Test: Unit Test — TypeTest (20 tests); Checkstyle passed. Regression suite not rerun locally. - Behavior changed: Yes — generated DDL includes nested field comments. - Does this need documentation: No
|
/review |
There was a problem hiding this comment.
Reviewed all four changed paths at 4a7cf3bb356e6612c6f3bd742fc23af7242cd66d using the supplied aggregate diff and static analysis. One new P2 finding: the new full SHOW CREATE expectation fails in the supported Cloud p0 pipeline before the replay/LIKE checks.
Critical checkpoint conclusions:
- Goal and proof: enabling the existing recursive renderer preserves nested comments in managed-table SHOW CREATE and CREATE TABLE LIKE under ordinary SQL mode. Three added JUnit tests cover formatting/escaping and the DESCRIBE flag; the regression includes nested STRUCT, ARRAY, replay and LIKE. Runtime success was not verified by this review.
- Scope and reuse: the production change is one rendering flag and reuses existing type traversal; the remaining changes are tests/results.
- Concurrency and locking: no new shared mutation, threads or lock acquisition. SHOW/LIKE use existing table read locks; the added work is local formatting without RPC, IO or journal waits.
- Lifecycle and initialization: no new resources, ownership, globals or initialization-order dependency.
- Configuration: no new settings. The existing interactive SQL-mode issue in thread 4059151453 still applies: the new call defaults to backslash escaping while LIKE parses with the session mode. It was independently confirmed and not reposted.
- Compatibility: no function-signature, persisted schema or wire-format changes; existing nonempty stored comments are read by the current renderer. No new rolling-upgrade requirement identified.
- Thrift/protobuf: untouched; no field-id or enum changes.
- Parallel paths: SHOW and LIKE both reach the changed compatible renderer; ARRAY/MAP/STRUCT recursion forwards the comment flag. External non-compatible rendering and type-only DESCRIBE retain their existing behavior. The known CCR SQL-mode concern is covered by thread 4079713811 and was not reposted.
- Conditional logic and errors: no new branch or error handling; omitted/null comments and explicit empty comments follow the existing presence flag and literal formatter.
- Test coverage: the additions cover ordinary-mode literals and recursive STRUCT/ARRAY output. They do not add MAP, empty-comment, SQL-mode-switch or persisted-reload round trips. Static tracing found no additional local issue in those shapes; the mode-switch defect is already reported.
- Expected results: unit expectations and CSV escaping agree with ordinary-mode non-Cloud defaults. The inline finding documents the concrete Cloud mismatch. The earlier generated-output concern in thread 4059151464 has been addressed by adding output, but that new full-row expectation introduces this separate failure. This review cannot attest how the author generated results.
- Observability: this metadata-formatting change needs no additional logs or metrics.
- Persistence and failover: no EditLog, replay or transaction processing change; existing comment metadata is reused.
- Data writes and crash safety: no data-write or atomicity change, and no new leak/deadlock boundary; LIKE uses the existing CREATE execution path.
- FE/BE propagation: no new transmitted variables.
- Performance: necessary escaping is linear in comment length on metadata paths; no new catalog scans or query-execution work.
- Other issues and focus: there was no additional user focus. All candidates and initial risks have accepted, duplicate or evidence-backed dismissed dispositions; no unresolved static-review scope remains. Existing supplied findings are P2, so there are no existing P0/P1 IDs to carry forward.
Validation limits: no builds, tests or runtime/CCR execution were performed, as required by the review contract. Author-reported test success is not treated as executed validation. The downstream CCR compatibility rewriter has a pre-existing limitation with comment-like text inside literals; it is outside this producer change and is not a new inline finding.
Review converged after 2 round(s).
| """ | ||
|
|
||
| // SHOW CREATE TABLE used to drop every nested field comment. | ||
| qt_show_create "SHOW CREATE TABLE struct_nested_comment" |
There was a problem hiding this comment.
[P2] Normalize SHOW properties before recording the expected row
This suite runs in Cloud p0: its configuration selects p0 and excludes neither this suite nor datatype_p0. However, this qt_ call compares the entire DDL with an output containing non-Cloud replication properties. In Cloud mode, Env.addOlapTablePropertyInfo emits file_cache_ttl_seconds instead of replication_allocation/min_load_replica_num, and the framework does no automatic SHOW normalization. The suite therefore fails here before reaching either round-trip check, even when nested comments are correct. Capture the stable column/comment text with a row converter and regenerate the expected output.
### What problem does this PR solve? Related PR: apache#68307 Problem Summary: Comparing the full SHOW CREATE TABLE statement makes the nested-comment regression depend on Cloud replication/cache properties, binlog settings and the default index format. Record the table name and column definitions for SHOW CREATE, replayed DDL and CREATE TABLE LIKE, while still replaying the complete generated statement. ### Release note None ### Check List (For Author) - Test: Regression test — generated output through run-regression-test.sh; verified with default settings and with binlog enabled/index format V2. - Behavior changed: No production change - Does this need documentation: No
|
/review |
There was a problem hiding this comment.
Reviewed all four changed files at b88a6a8. No new inline findings after duplicate filtering. Static tracing supports nested-comment preservation in ordinary-mode SHOW CREATE TABLE, replay, and CREATE TABLE LIKE.
The existing P2 SQL-mode finding still applies: Column selects the overload that doubles backslashes, while LIKE reparses using the live NO_BACKSLASH_ESCAPES mode. It was independently confirmed and not reposted. The Cloud output concern and fixed-result capture concern are addressed by the column-only converter and three golden blocks. The CCR consumer-mode thread remains prior context; its current external implementation was not revalidated. There are no existing P0/P1 findings to carry forward.
Critical checkpoint conclusions:
- Goal and proof: the compatible renderer now includes stored nested comments; three unit methods and SHOW/replay/LIKE regression expectations cover the main ordinary-mode behavior. The known SQL-mode limitation above remains.
- Scope and reuse: one production-line change reuses the established recursive renderer and literal quoting utility; supporting tests are focused.
- Concurrency and locks: SHOW/LIKE read source metadata under existing table read locks; no shared mutation, lock-order change, external IO, or journal wait is added.
- Lifecycle: temporary Java strings/builders introduce no new resource ownership, threads, caches, cycles, or static initialization.
- Configuration: no configuration item is added or changed.
- Compatibility: generated SQL gains existing COMMENT syntax; symbols, storage formats, and rolling-upgrade protocols are unchanged. SQL-mode compatibility is covered by the known thread.
- Thrift/protobuf: no definitions, ids, enum values, or required fields change.
- Parallel paths: SHOW and LIKE share the changed method; STRUCT/ARRAY/MAP recursion forwards comment inclusion. DESCRIBE keeps its explicit inclusion policy; existing external rendering is unaffected.
- Conditions and errors: absent-comment gating is intentional; no new speculative fallback, swallowed error, or unchecked Status is introduced.
- Test coverage: unit cases cover missing/present comments, quotes, backslashes and explicit display policy; regression checks nested STRUCT and ARRAY plus replay/LIKE. MAP, mode variants, empty comments and quoted identifiers lack new end-to-end cases; relevant mode/identifier concerns overlap existing context.
- Changed expectations: all three golden rows, tags, indentation, SQL escaping and framework CSV escaping are statically consistent. The converter runs before both generation and comparison and excludes native/Cloud table properties.
- Observability: existing tagged expected/actual test diagnostics are sufficient for this metadata-formatting change; no new metrics or logs are needed.
- Transactions and persistence: comments already reside in schema metadata; no EditLog, replay, failover, visible-version or delete-bitmap implementation changes.
- Data writes and crash behavior: no row-write or atomicity path changes; LIKE uses the existing CREATE implementation after parsing.
- FE/BE variables: no new cross-process state or sending path.
- Performance and memory: extra work is linear in emitted comment bytes and uses request-local Java objects; no new hot-path amplification or retained ownership was found.
- Other concerns and user focus: no additional focus was supplied. A final edge-case sweep found no distinct introduced issue beyond the existing threads.
Validation limits: static review only; no builds or tests were run. HEAD reports output generation through run-regression-test.sh, but runtime success and generation provenance were not independently verified here. All source scope and risk candidates have explicit dispositions; no unresolved source candidate remains. All three assigned reviewers returned NO_NEW_VALUABLE_FINDINGS in round 1.
Review converged after 1 round(s).
|
run buildall |
TPC-H: Total hot run time: 28791 ms |
TPC-DS: Total hot run time: 150204 ms |
ClickBench: Total hot run time: 24.24 s |
FE UT Coverage ReportIncrement line coverage `` 🎉 |
FE Regression Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
|
PR approved by at least one committer and no changes requested. |
…8307) ### What problem does this PR solve? ```sql CREATE TABLE t (id INT, s STRUCT<a:INT COMMENT 'field doc'>) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1"); SHOW CREATE TABLE t; ``` Before: `s` is shown as `struct<a:int>`. After: `s` is shown as `struct<a:int comment "field doc">`.
…TE TABLE #68307 (#69337) Cherry-picked from #68307 --------- Co-authored-by: Chenyang Sun <[email protected]>
What problem does this PR solve?
Before:
sis shown asstruct<a:int>.After:
sis shown asstruct<a:int comment "field doc">.