Repository navigation
Conversation
…ynamic path in the sparse column
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
`variant_max_subcolumns_count = 0` means "materialize every path as a subcolumn", and nothing lets a
VARIANT column keep every path in its sparse column instead. A changelog needs exactly that for its
AFTER and BEFORE images: they are only read whole, so materializing their paths would double the
subcolumns of the source table for nothing.
This adds a per-column parameter `variant_serialize_type` (`v1` / `v2`). It only changes what a count
of 0 means:
- V1: count 0 materializes every dynamic path, the old meaning. Every column persisted before the parameter
existed is V1 (FE images, ColumnPB and TColumnType without the field).
- V2: count 0 materializes no dynamic path; they all go to the sparse column(s). Typed paths of a
schema template behave as before (materialized unless variant_enable_typed_paths_to_sparse).
- Any count above 0, doc mode and nested group behave the same under both values.
New DDL (CREATE TABLE, ADD COLUMN) creates V2 columns unless the column states
`"variant_serialize_type" = "v1"`.
FE: the catalog and nereids VariantType carry the field as the thrift enum TVariantSerializeType;
ColumnToThrift / ColumnToProtobuf send it as TColumnType.variant_serialize_type (8) and
ColumnPB.variant_serialize_type (36). DESC never shows it; SHOW CREATE TABLE and CREATE TABLE LIKE
state it for every VARIANT column (`"variant_serialize_type" = "v1"` or `"v2"`), so the printed DDL
replays to the same layout whatever the default for new columns is. VariantType equality (catalog and
nereids) and the MODIFY COLUMN check compare it only where it decides the layout (count 0, no doc mode,
no nested group), so restating an old column with a non-zero count as a bare VARIANT keeps working.
BE: TabletColumn keeps the field. The three places that give a count of 0 its meaning now branch on
it: the writer's select_variant_paths (V1 materializes every dynamic path, V2 none) and, in
get_extended_compaction_schema, get_subpaths (only for a count above 0 or V1) and the
all-materialized branch (V1 only). Typed paths, doc mode and nested group are untouched; doc mode and
nested-group columns never reach these places. Compaction of a V2 count-0 column fails with an
InternalError if an input still materializes a dynamic path, which it would otherwise drop. The read
path is unchanged: it already follows what each segment contains.
Upgrade: downtime upgrade only, no rollback. A BE or FE without this change does not know the field:
it treats a V2 count-0 column as V1, so FE and every BE must run the new version together before such
a column is written, and going back to an older version is not supported. DDL printed by this version
states variant_serialize_type, which an older version rejects.
A V2 count-0 column materializes no dynamic path, so it has no inverted index on one either.
Regression cases that use count 0 to get every dynamic path materialized now say so with
`"variant_serialize_type" = "v1"`: 9 suites that pin count 0 (subcolumn indexes for MATCH and
SEARCH, DESC of extracted subcolumns, typed subcolumn checks) and the variant_p2 load, whose
relational benchmark reads materialized subcolumns. Cases whose indexes are on schema-template typed
paths, and cases that only check query results, need nothing. The
fuzzy session variables no longer draw `default_variant_max_subcolumns_count = 0` outside doc mode:
a column created in such a session would lose the subcolumns that cases without a pinned count may
rely on.
### Release note
New VARIANT columns are created with `variant_serialize_type = v2`. For them
`variant_max_subcolumns_count = 0` stores every dynamic path in the sparse column instead of
materializing every one, so such a column has no subcolumn inverted indexes and index-only queries on its paths
(MATCH, SEARCH, BM25 scoring) on dynamic paths fail; typed paths of a schema template are still
materialized. Existing columns keep the old meaning; state
`"variant_serialize_type" = "v1"` on a new column to keep it. SHOW CREATE TABLE states
`variant_serialize_type` for every VARIANT column. Upgrading to this version needs downtime (FE and
all BEs together) and cannot be rolled back.
### Check List (For Author)
- Test: Unit Test, Regression test
- BE UT: SchemaUtil*, TabletSchemaTest, VariantPathBuilderTest, VariantColumnWriterReaderTest,
IndexStorageTestFixture.* and every *Variant* suite: 907 run, 898 passed, 9 skipped (5
pre-existing GTEST_SKIPs, 4 nested-group cases: this build has no nested-group write path), 0
failures. New AllFeVariantParameters/VariantCompactionSchemaMatrixTest: every VARIANT parameter
combination the FE accepts (serialize type x count 0/3/100 x typed paths to sparse x schema
template x sparse shards, doc mode x min rows x doc shards, nested group); for each, the
compaction schema follows the rule, the written and the compacted layouts follow the same rule,
and a real compaction keeps every value. VariantV2CountZeroRejectsMaterializedDynamicInput
covers the InternalError.
- FE UT: TypeTest, ColumnTest, VariantSerializeTypeDdlTest, PropertyAnalyzerTest,
NereidsParserTest, DataTypeTest, ScalarTypeTest, SessionVariablesTest,
VariantLoadParseInjectionTest, PruneNestedColumnTest, BindSinkConnectorVariantTest,
ConnectorComputeVariantPersistenceTest, VariantWritePlanValidatorTest: 0 failures;
NereidsParserTest.testVariantPropertyCombinations parses the same combinations and rejects the
five conflicting ones
- Regression (local coupled cluster): the 27 p0 suites that pin variant count 0: all pass (9 of
them fail without v1). The 9 suites whose random count can be 0, run with the count pinned to
0: all pass without v1. Not run: variant_p2 (needs the 44M-row GitHub
Events dataset), cloud mode, external pipelines, full p0.
- clang-format 16 clean; clang-tidy finds nothing on the changed lines (variant_path_builder.cpp
with clang-analyzer-* off: the 20.1 analyzer crashes in the unchanged rescale_decimal). The new
segment test carries NOLINTNEXTLINE(readability-function-cognitive-complexity) like the other
variant segment tests: GTest macros inflate the case matrix.
- Behavior changed: Yes. New VARIANT columns default to V2, so an explicit
`variant_max_subcolumns_count = 0` on a new column stores every dynamic path in the sparse column.
- Does this need documentation: Yes (no doc PR yet)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
A table with changelog could not hold a VARIANT column: CREATE TABLE, the changelog schema generation
and the three ADD COLUMN paths rejected it.
This lets VARIANT columns into changelog:
- Both changelog images of a VARIANT column, the AFTER column `v` and the BEFORE column
`__DORIS_CHANGELOG_BEFORE_v__`, keep every path in the sparse column (serialize type V2 with
`variant_max_subcolumns_count = 0` and variant_enable_typed_paths_to_sparse, no doc mode, no
nested group), whatever the layout of the source column. Column.generateAfterValueColumn / generateBeforeValueColumn install that layout
through the new VariantType.forChangelog, so CREATE TABLE and ADD COLUMN both get it; the
predefined fields stay, so typed paths keep their declared types. Changelog images are read whole,
so materializing their paths would only double the subcolumns and the compaction work of the
source table.
- A VARIANT column on a table with changelog must be V2 and can not enable nested group: CREATE
TABLE and ADD COLUMN reject it (changelog can only be enabled at CREATE TABLE).
- A changelog read never pushes a Variant sub-path into the scan. VariantSubPathPruning leaves a
statement unchanged when it contains a changelog scan (OlapChangelogTableWrapper), so sub-paths
are taken from the whole value above the scan. Two reasons: BlockReader maps a BEFORE image onto
its AFTER column by whole column, so a pushed sub-path of an AFTER column would show the AFTER
value in an UPDATE_BEFORE row; and a stream plan unions a base-table scan with a changelog scan,
whose union pushdown needs every child to supply the same sub-path slots. Predicates on value
columns already stay above BlockReader for MIN_DELTA / DETAIL scans, so a filter on a sub-path
sees the image of its own row. The cost is that such a statement reads its Variant columns whole,
including from the base table.
Upgrade: as for the serialize type, downtime upgrade only and no rollback: a changelog VARIANT image
relies on the V2 count-0 layout, which an older FE or BE does not know.
### Release note
Tables with changelog support VARIANT columns (serialize type v2, no nested group). Reads through table streams, CHANGES and changelog()
read Variant columns whole and take sub-paths from the whole value.
### Check List (For Author)
- Test: Regression test, Unit Test
- Regression (local coupled cluster): new changelog_p0/test_changelog_variant (raw changelog
with both images and their sub-paths, MIN_DELTA and DETAIL streams with sub-paths and a
sub-path filter, changelog compaction, a stream over history and incremental partitions,
ADD COLUMN); changelog_p0 and table_stream_p0: 42 suites, 0 failures. Not run in cloud mode.
- FE UT: OlapTableChangelogSchemaTest, SchemaChangeHandlerTest (testWithChangelogOpNotSupported
now accepts VARIANT in CREATE TABLE and ADD COLUMN and rejects v1 and nested group),
ExplainTableStreamPlanTest: 0 failures. The regression runs above predate the v2 / nested
group rule and the typed paths of changelog images going to the sparse column.
- Behavior changed: Yes. VARIANT columns are accepted on tables with changelog.
- Does this need documentation: Yes (no doc PR yet)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
After compaction, VariantCompactionUtil::check_path_stats requires every variant path of a DUP_KEYS
tablet to keep its non-null count, since such a tablet never merges rows. A changelog tablet is
DUP_KEYS, but its compaction drops the rows its delete bitmap marks, for example a change that a
higher sequence value overrode. With a VARIANT column in a table with changelog, the check then
failed every compaction of that changelog tablet ("Path stats not match ... input size 3, output
size 2"), so its versions piled up. Before VARIANT was allowed in changelog this could not happen.
check_path_stats now skips changelog tablets, as it skips every non-DUP_KEYS tablet.
Reproduce: a MOW table with changelog, need_historical_value, a sequence column and a VARIANT
column; write a key, rewrite it with a lower sequence value, compact (changelog_p0
test_changelog_compaction with a VARIANT column): the compaction failed with INTERNAL_ERROR; it now
succeeds and the changelog keeps every image.
### Release note
None
### Check List (For Author)
- Test: Regression test
- changelog_p0/test_changelog_compaction and test_cloud_changelog_compaction (cloud mode) with a
VARIANT column, in the next commit: fail before, pass after
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>
… suites with a VARIANT column
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Only changelog_p0/test_changelog_variant covered VARIANT columns in changelog. 22 existing suites
now also give their changelog tables a `jv VARIANT` column, written on every full-row INSERT as
parse_to_variant('{"k": <first value>, "d": {"row": "<row values>"}}'), so an update changes it, and
read with jv['d']['row'] next to the columns they already check:
- changelog_p0: basic (partial update not touching jv, DELETE, sequence column), partial update
of keys only, MOW light delete, compaction, multi segment (jv from a stream load column
mapping), auto partition, TTL, CHANGES syntax, append-only projection, cloud compaction
- table_stream_p0: MIN_DELTA stream, MIN_DELTA delete-before, snapshot and reset streams (sinks
carry jv and every row's jv['k'] must equal its id)
- time_travel_p0: MOW and DUP time travel, including the old jv of a snapshot and after compaction
- mtmv_p0/ivm: basic (jv flows into SELECT * materialized views through incremental refresh),
aggregation, inner join, DUP keys and MTMV changelog
- cloud_p0/table_stream: end to end
The other columns of every existing result are unchanged; only jv columns are appended. Two
assertions count the visible columns or compare SELECT t.* and now account for jv.
changelog_p0/test_changelog_flexible_partial_update is left alone: flexible partial update rejects
tables with a VARIANT column.
### Release note
None
### Check List (For Author)
- Test: Regression test
- all 22 suites pass on a local coupled cluster, and the two cloud suites on a local cloud
cluster; .out files regenerated with -forceGenOut and checked: no existing column changed
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
run buildall |
There was a problem hiding this comment.
Request changes: one P1 data-loss issue and two P2 issues (planning failure and an assertion-enabled cloud BE abort) were verified at commit 58fae14. All 88 changed files were reviewed, including surrounding producers/consumers, tests and expected results. There were no preexisting inline findings and no additional user-provided focus points.
Critical checkpoint conclusions:
- Goal and proof: VARIANT changelog support and V2 zero-count layouts are implemented across FE/BE, but mixed-layout compaction, base/changelog-TVF UNION extraction and assertion-enabled cloud conflict publishing remain incorrect (inline findings). Added tests do not cover those triggers.
- Scope and reuse: production changes are focused on layout metadata, path selection and changelog enablement; shared writer helpers are reused. Regression changes exercise related existing features.
- Concurrency and locking: no new threads, atomics, locks or lock ordering; existing metadata mutation and rowset publication boundaries remain in use. The cloud P2 is triggered by concurrent same-key publishing; no additional race or deadlock found.
- Lifecycle and initialization: the layout flag follows existing schema ownership, and forChangelog creates a separate type. No new cycles, owner-lifetime or cross-translation-unit initialization issue found.
- Configuration: variant_serialize_type is a persisted per-column DDL property; MODIFY rejects changes where count-zero changes layout. No new BE dynamic process setting. Fuzzy session defaults avoid zero outside doc mode.
- Compatibility: absent Gson/thrift/PB fields retain V1 and current root serializers preserve V2. Optionality does not prevent old BEs from producing the mixed layouts in the P1 finding.
- Shared definitions: inspected complete touched definitions on HEAD and live-matching branch-3.1/4.0/4.1/4.2 refs. ColumnPB field 36 and TColumnType field 8 are unused there; additions are optional, enum values agree, defaults preserve V1. No ID collision found.
- Parallel paths: inspected CREATE, ADD/ADD COLUMNS, MODIFY, SHOW CREATE/CREATE LIKE, CTAS, TVF inference, local/cloud schema messages, ordinary/doc/nested-group storage, typed paths, streams, time travel, CTEs and unions. The missed changelog-TVF path and cloud historical-image count invariant are the P2 findings.
- Conditions: layout and image guards have concrete explanations; the inline findings identify incomplete guard/invariant coverage.
- Test breadth: inspected FE parser/property/equality/Gson/DDL tests, BE writer/compaction layout matrices, and changelog/cloud/IVM/stream/time-travel regressions. Missing mixed-upgrade, matching-layout TVF-UNION and concurrent cloud historical-VARIANT publish cases should be added.
- Expected results: all modified output hunks were checked against their changed INSERT/projection and historical-image semantics; no separate mismatch found. Output-generation provenance is unverified.
- Observability: existing compaction error reporting carries column/path context; no new independent logging/metrics gap found. The silent-loss path evades that error and changelog path-stat checking.
- Persistence and failover: the annotated flag follows existing Column image/EditLog and tablet-schema PB mechanisms. No new transaction or journal state machine; schema-version transitions were traced for the P1 finding and publish-time conflict rewrites for the cloud P2.
- Writes and atomicity: existing segment/rowset commit mechanisms remain in use; committed changelog values can nevertheless be lost by the P1 compaction path. The cloud P2 also aborts DEBUG/ASAN BEs during conflict publishing. No additional crash-cleanup or atomicity defect established.
- FE/BE propagation: primary TColumnType/ColumnPB producers and TabletMeta consumers carry the flag; read-only runtime/extracted types do not select root storage layouts.
- Performance and memory: existing allocation/ownership and sorting paths are reused; the whole-plan scan is linear and deliberately disables subpath pruning with changelog scans. No additional substantiated CPU/memory issue found.
- Further recheck: legacy ScalarType, sparse-stat truncation, empty/typed-only inputs, source-image copying, schema caches, cloud index rebuilds and optimizer stage order were revisited; no additional distinct finding or unresolved candidate remains.
Validation was static only: no builds, tests, benchmarks, runtime reproductions, source edits or submodule initialization were performed. Runtime behavior and output provenance remain unverified within that required scope.
Review converged after 3 round(s).
| // Every input path of a V2 count-0 column is in the sparse column(s) or typed. Nothing | ||
| // would carry a materialized dynamic subcolumn into the output, so its data would be lost. | ||
| for (const auto& [path, _] : extended_info.path_to_none_null_values) { | ||
| if (!extended_info.sparse_paths.contains(path) && |
There was a problem hiding this comment.
[P1] Check materialized paths per input before accepting this layout
extended_info.sparse_paths is the union across every input segment, so an older segment materializing a passes this guard whenever another segment stores a sparsely. This is reachable during a rolling upgrade: the new FE sends V2/count-zero changelog columns to an old BE, which ignores the new field and materializes their paths; after upgrading that BE, a light ADD COLUMN and another load produce a higher-schema-version V2 sparse rowset. Compaction selects that V2 schema, but this branch leaves paths_set_info.sparse_path_set empty. _create_sparse_merge_reader therefore skips the older materialized a, while the root-only reader/writer cannot recover it (object roots are empty). Old changelog images silently lose a, and check_path_stats now skips changelog tablets. Populate the sparse transfer set for materialized input leaves, or reject them using per-input materialized metadata rather than membership in the union of sparse paths.
There was a problem hiding this comment.
ok,i will add this check
| // its AFTER column by whole column, so a changelog scan reads Variant columns whole and sub-paths are taken | ||
| // above it. Its plan may union the changelog scan with a base table scan, whose union pushdown needs every | ||
| // child to supply the same sub-path slots, so the whole statement keeps whole-column reads. | ||
| if (plan.anyMatch(node -> node instanceof LogicalOlapScan |
There was a problem hiding this comment.
[P2] Include changelog TVFs in the whole-column guard
changelog(...) remains a LogicalTVFRelation during this rewrite; TableChangelogFunction.getScanNode attaches the wrapper only during physical translation, so this predicate misses it. For a populated changelog-enabled table t (with an inserted object containing a) and nullable v VARIANT<PROPERTIES("variant_max_subcolumns_count"="0","variant_enable_typed_paths_to_sparse"="true")>, SELECT v['a'] FROM (SELECT v FROM t UNION ALL SELECT v FROM changelog("table"="t")) u has matching source/image types and no union casts. Its reduced plan is:
Project(element_at(v_union, 'a'))
UnionAll
OlapScan(t.v)
TVF(changelog.v)
visitLogicalOlapScan generates v.a only for the first child; there is no TVF replacer. visitLogicalUnion then indexes entry.getValue().get(1) although the list contains one slot, failing planning (reversing branches hits the missing-subpath exception). PushProjectThroughUnion rejects ElementAt, so it does not eliminate this shape. Recognize changelog TVF relations here too, or synthesize the missing child expressions, and cover this matching-layout UNION case.
There was a problem hiding this comment.
ok, i will ban this case
| } else if (column.getType().isVariantType() && column.getVariantEnableNestedGroup()) { | ||
| throw new AnalysisException( | ||
| "variant column with nested group can't be created on table with changelog"); | ||
| } else if (column.getType().isVariantType() |
There was a problem hiding this comment.
[P2] Update the cloud Variant-count invariant for BEFORE images
The newly accepted cloud MOW configuration with one VARIANT column and changelog.need_historical_value=true has one source VARIANT but two changelog VARIANT roots (AFTER and BEFORE). CloudMetaMgr::update_tmp_rowset still asserts that those counts are equal at cloud_meta_mgr.cpp:1697-1698. This is reachable in DEBUG/ASAN builds: let two same-key UPSERTs flush against the same visible snapshot, then publish one before the other. The second publish's BaseTablet::update_delete_bitmap creates a transient group writer to rebuild the historical image, emits replacement rows, and reaches CloudTablet::save_delete_bitmap with nonzero writer rows; that calls update_tmp_rowset and aborts the BE on DCHECK_EQ(1, 2). RELEASE builds compile out this assertion, but assertion-enabled cloud builds cannot handle this supported write schedule. Make the cloud invariant account for the BEFORE columns and cover concurrent publish-time rewrites with a historical VARIANT column.
There was a problem hiding this comment.
ok, i will change this check
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
… images in cloud changelog
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Review findings on the VARIANT serialize type and changelog support:
1. Compaction of a V2 count-0 column rejects an input that materializes a dynamic path, because it
would lose that path. The check tested every stats path against the union of the inputs' sparse
paths, so a segment materializing `a` passed whenever another segment kept `a` in its sparse
column, and the materialized `a` was dropped. The inputs' materialized paths are now collected
on their own (VariantExtendedInfo.materialized_paths) and any non-typed one fails the compaction.
2. CloudMetaMgr::update_tmp_rowset asserted that a rowset and its changelog rowset have as many
VARIANT columns. With changelog.need_historical_value the changelog also keeps a BEFORE image of
every VARIANT column, so an assertion-enabled cloud BE aborted when a publish-time conflict
rewrite reached it. The assertion now adds the BEFORE images.
A third finding, a UNION of a base table and its changelog() TVF failing to plan when a VARIANT
sub-path is read above it, did not reproduce: the sub-path projection is pushed into both union
children and the result is correct. changelog_p0/test_changelog_variant now covers that query.
### Release note
None
### Check List (For Author)
- Test: Unit Test, Regression test
- BE UT IndexStorageTestFixture.VariantV2CountZeroRejectsMaterializedDynamicInput now writes a
V1 rowset materializing `a` and a V2 rowset keeping `a` sparse; it fails with the old
union check and passes now. VariantCompactionSchemaMatrixTest: 64 passed, 4 nested-group
cases skipped (no nested-group write path in this build).
- Regression: changelog_p0/test_changelog_variant (with the new UNION query) and
test_changelog_compaction pass on a local coupled cluster.
- The cloud assertion is not reproduced locally (it needs a concurrent publish conflict in a
cloud cluster); the fix is checked by reading the code.
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>
TPC-H: Total hot run time: 28820 ms |
TPC-DS: Total hot run time: 151312 ms |
ClickBench: Total hot run time: 25.12 s |
FE UT Coverage ReportIncrement line coverage |
…mode too
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
The fuzzy session variables (CI p0, nonConcurrent, cloud_p0, p1) set
default_variant_max_subcolumns_count = 0 whenever they pick doc mode. A case that then turns doc mode
off, through `set default_variant_enable_doc_mode = false` or a column's
`"variant_enable_doc_mode" = "false"`, creates a V2 column with count 0, which materializes no
dynamic path, so MATCH and SEARCH on its subcolumn indexes fail. P0 hit it in
inverted_index_p0/test_bm25_score_variant ("match_phrase not support execute_match") and
variant_p0/test_variant_search_historical_rowset_index ("SEARCH index unavailable for Variant field
'v.missing'").
The fuzzer no longer zeroes the count in doc mode: a doc mode column takes count 0 anyway
(LogicalPlanBuilder), so the session default is only seen by columns without doc mode, and it now
always draws 1..9.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- SessionVariablesTest.testFuzzyVariantMaxSubcolumnsCountIsNeverZero (was
...IsZeroOnlyInDocMode): 24 run, 0 failures; fe-core checkstyle clean
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
/review |
|
run buildall |
|
/review |
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
TPC-H: Total hot run time: 28953 ms |
TPC-DS: Total hot run time: 151491 ms |
ClickBench: Total hot run time: 25.41 s |
Brings in test_match_keyword_score (apache#69790), which the next commit pins to a positive subcolumn count. No conflicts; ColumnPB 36 and TColumn 31 stay unique after the merge. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…est_match_keyword_score ### What problem does this PR solve? Issue Number: None Related PR: apache#69812, apache#69790 Problem Summary: test_match_keyword_score (apache#69790) creates `v VARIANT NULL`, so its subcolumn count comes from the session default, which the fuzzy session variables can set to 0. With this PR a new column with count 0 keeps every dynamic path in the sparse column, so `s` has no subcolumn index, MATCH still finds the rows, and score() returns 0 instead of 0.693147. Cloud P0 build 1089519 failed this way on `V2_variant_text_path`. The column now states a count of 2048, like the other suites that need indexed subcolumns. The expected output is unchanged. ### Release note None ### Check List (For Author) - Test: Regression test. With `SET GLOBAL default_variant_max_subcolumns_count = 0` the suite fails on `V2_variant_text_path` without this change and passes with it; it also passes with the default count. FE UT on the merged code: 485 tests, 0 failures. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
/review |
|
run buildall |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
FE UT Coverage ReportIncrement line coverage |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
TPC-H: Total hot run time: 28954 ms |
TPC-DS: Total hot run time: 151345 ms |
ClickBench: Total hot run time: 24.53 s |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
…hold and the shard count ### What problem does this PR solve? Issue Number: None Related PR: apache#69812 Problem Summary: The previous commit renamed `variant_doc_materialization_min_rows` to `variant_subcolumn_extraction_min_rows` and `variant_doc_hash_shard_count` to `variant_sparse_shard_count`, and kept the old names as aliases. Both old names belong to doc mode, whose 4.1 columns already follow the new semantics on the assumption that doc mode is not in use there, so they are not accepted any more: CREATE TABLE, ALTER TABLE and CAST reject them with the error for unknown VARIANT properties ("only support for ..."). `variant_sparse_hash_shard_count`, the shard count that released versions print without doc mode, stays an alias of `variant_sparse_shard_count`, so their SHOW CREATE TABLE output still parses. The regression suites that set the doc mode names set the new ones now, and test_variant_tuning_properties checks that the old ones are refused. ### Release note `variant_doc_materialization_min_rows` and `variant_doc_hash_shard_count` are no longer accepted; use `variant_subcolumn_extraction_min_rows` and `variant_sparse_shard_count`. The SHOW CREATE TABLE output of a doc mode table from 4.1 has to be renamed before it runs. ### Check List (For Author) - Test: Unit Test, Regression test - FE UT: NereidsParserTest, PropertyAnalyzerTest and CreateTableCommandTest, 129 tests, 0 failures. - Regression: the 8 suites whose VARIANT properties changed, where test_variant_tuning_properties is regenerated by running it and its two refusals fail with "only support for ..."; and check_before_quit, which recreates every table from its SHOW CREATE TABLE and compares. - Behavior changed: Yes. The two doc mode property names are refused. - Does this need documentation: Yes, the documentation uses the new names. Co-Authored-By: Claude Opus 5.5 <[email protected]>
… let MODIFY COLUMN change the row threshold ### What problem does this PR solve? Issue Number: None Related PR: apache#69812, apache#61100 Problem Summary: Two choices of the earlier commits went further than renaming the properties: - `variant_sparse_shard_count` was written into both stored shard fields, `variantSparseHashShardCount` and `variantDocShardCount`, and MODIFY COLUMN compared the field of the column's mode instead of both fields. The property only has a new name: with doc mode the value belongs to the doc shard field and without it to the sparse shard field, as each mode had its own property before. The parser now fills that field only and leaves the other one 0, which is what every released version stores, so `Column.checkSchemaChangeAllowed` compares the two fields again as master does. The BE and SHOW CREATE TABLE already read the field of the mode. - MODIFY COLUMN refused any change of `variant_subcolumn_extraction_min_rows`, while apache#61100 lets it change `variant_doc_materialization_min_rows`. It can change again, with and without doc mode, except on a column without doc mode created before this change (a V1 column): its BE ignores the threshold, because its count-0 compaction branch cannot extract paths that only the sparse column holds, so a value stored there would do nothing. test_variant_modify_doc_materialization_min_rows has its steps from master again (an ALTER lowers the threshold and the next compaction extracts the old paths), and test_variant_tuning_properties raises the threshold with an ALTER again. The BE comment and the UT description of a compaction below the threshold name the ALTER again; the BE code does not change. ### Release note MODIFY COLUMN can change `variant_subcolumn_extraction_min_rows`, but not on a column without doc mode created before the upgrade. ### Check List (For Author) - Test: Unit Test, Regression test - FE UT: the 11 classes that the VARIANT properties touch (PropertyAnalyzerTest, VariantSerializeTypeDdlTest, TypeTest, ColumnTest, SchemaChangeHandlerTest, DataTypeTest, NereidsParserTest, VariantLoadParseInjectionTest, CreateTableCommandTest, ShowCreateTableCommandTest, SessionVariablesTest), 286 tests, 0 failures. - Regression: test_variant_tuning_properties and test_variant_modify_doc_materialization_min_rows, regenerated by running them (the latter extracts the same subcolumns as on master); every enabled suite of variant_p0 (187), where only outfile_csv_variant_doc_value fails because it needs S3 credentials; six suites outside variant_p0 that create VARIANT tables; check_before_quit, which recreates every table from its SHOW CREATE TABLE and compares. - Behavior changed: Yes. MODIFY COLUMN can change the row threshold again. - Does this need documentation: Yes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
… typed paths to sparse internal ### What problem does this PR solve? Issue Number: None Related PR: apache#69812 Problem Summary: - Doc mode columns created before this change (V1, doc mode shipped in 4.1.0 and forced their count to 0) keep their layout: they still extract every path once a file reaches the row threshold. The load shredder (finish_doc) and the compaction schema (choose_doc_compaction_subpaths) pass the column's serialize type instead of V2, so select_variant_paths and get_subpaths give a count of 0 its V1 meaning, and every doc bucket writer extracts all paths of its bucket. The catalog and Nereids VariantType.equals compare the serialize type of a doc column with a count of 0 again. A new doc column keeps the new meaning: 0 extracts no dynamic path. - `variant_enable_typed_paths_to_sparse` is internal now. The property only takes `false`, which released versions print in SHOW CREATE TABLE for every column without doc mode, so that DDL still runs; `true` is refused ("variant_enable_typed_paths_to_sparse = true is not supported"). A new column takes the value from the session variable `default_variant_enable_typed_paths_to_sparse`, which only tests set, and a column of an older version keeps its value. SHOW CREATE TABLE and DESC do not print it. A restated definition cannot state it, so MODIFY COLUMN no longer compares it, and SchemaChangeHandler carries the stored value over as it does the serialize type. The regression suites that created columns with `true` set the session variable around those CREATE TABLE statements instead. - Without doc mode only a V2 column has a row threshold: older versions cleared it and MODIFY COLUMN refuses one on a V1 column. The load shredder (finish_ordinary) and the compaction schema (choose_sparse_compaction_subpaths) therefore compare the rows with it without testing the serialize type. The BE unit tests no longer give a V1 column without doc mode a threshold, a state that cannot occur. - Doc mode never moves typed paths to the sparse encoding, so finish_doc asserts that instead of testing the flag. - The doc statistics loop is inlined into aggregate_variant_extended_info, its only caller, and choose_compaction_subpaths is renamed choose_sparse_compaction_subpaths, the counterpart of choose_doc_compaction_subpaths. ### Release note Doc mode columns created by 4.1 keep extracting every path. `variant_enable_typed_paths_to_sparse` cannot be set to `true` any more and is no longer printed by SHOW CREATE TABLE; columns that have it keep it. ### Check List (For Author) - Test: Unit Test, Regression test - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1018 run, 1009 passed, 9 skipped, 0 failed. The 48 cases fewer than before are the V1 columns without doc mode that had a threshold. Two VariantShredderTest cases gave the default V1 ordinary layout a threshold and now set it for the doc layout only. - FE UT: the 11 classes that the VARIANT properties touch, 286 tests, and VariantSerializeTypeDdlTest with the new testModifyColumnKeepsTypedPathsToSparse, 4 tests; 0 failures. - Regression: the 11 suites whose output printed the property, regenerated by running them; each .out only loses the property, and test_all_prdefine_type_to_sparse keeps its subcolumns, so the session variable reproduces the layout. test_paimon_catalog_variant.out needs a Paimon environment and was edited by hand to the type text test_adbc_type_semantic now prints. Every enabled suite of variant_p0 (187), where only outfile_csv_variant_doc_value fails because it needs S3 credentials; 20 suites outside variant_p0 that use the property or create VARIANT tables; check_before_quit, which recreates every table from its SHOW CREATE TABLE and compares. Not run: the two variant_p1 suites and test_index_disk_usage_tvf, which only write `false`. - clang-format, clang-tidy on the changed lines, checkstyle and check-build-hygiene.sh are clean. Inlining the doc statistics loop gives aggregate_variant_extended_info a cognitive complexity of 53, so it carries a NOLINT with the reason. - Behavior changed: Yes. See the release note. - Does this need documentation: Yes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…f the shard count ### What problem does this PR solve? Issue Number: None Related PR: apache#69812 Problem Summary: The earlier commits named the shard count of the sparse encoding `variant_sparse_shard_count` and kept `variant_sparse_hash_shard_count`, the name that released versions use without doc mode, as an alias. The property keeps its released name instead: `variant_sparse_hash_shard_count` sets the shard count in both modes (the field of the column's mode, as before), so the DDL of released versions needs no alias and the alias code goes (renameDeprecatedVariantProperties and its "names of the same property" error). The doc mode name `variant_doc_hash_shard_count` stays refused. The session variable is `default_variant_sparse_hash_shard_count` again, as on master, and also sets the shard count of doc mode. SHOW CREATE TABLE and DESC print `variant_sparse_hash_shard_count`. Two smaller changes in the doc mode compaction code: - VariantShredderOptions takes the dynamic paths a compaction chose as a `const PathSet*` instead of a std::function that tests the same set; variant_shredder.h includes tablet_schema.h for PathSet. - Shorter comments on choose_doc_compaction_subpaths, choose_sparse_compaction_subpaths and the load branch of finish_doc. ### Release note The shard count of the sparse encoding keeps its name `variant_sparse_hash_shard_count` and now also applies with doc mode. ### Check List (For Author) - Test: Unit Test, Regression test - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1018 run, 1009 passed, 9 skipped, 0 failed. - FE UT: the 11 classes that the VARIANT properties touch, 287 tests, 0 failures. - Regression: the 12 suites whose output prints the shard count, regenerated by running them; each .out only changes the printed name, except test_variant_tuning_properties, which drops the table it created with the old alias. test_paimon_catalog_variant.out needs a Paimon environment and was edited by hand to the type text test_adbc_type_semantic now prints. Every enabled suite of variant_p0 (187), where only outfile_csv_variant_doc_value fails because it needs S3 credentials; 20 suites outside variant_p0 that create VARIANT tables or set the session variables; check_before_quit, which recreates every table from its SHOW CREATE TABLE and compares. - clang-format, clang-tidy on the changed lines, checkstyle and check-build-hygiene.sh are clean. - Behavior changed: Yes. See the release note. - Does this need documentation: Yes, the documentation uses `variant_sparse_hash_shard_count`. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…t extract every path ### What problem does this PR solve? Issue Number: None Related PR: apache#69812 Problem Summary: The serialize type of a VARIANT column (V1 for columns created before the upgrade, V2 for new ones) is meant to decide one thing: what a subcolumn count of 0 means, every path under V1 and no dynamic path under V2. MODIFY COLUMN also used it to refuse `variant_subcolumn_extraction_min_rows` on every V1 column without doc mode, so a table of an older version, whose count is 2048 by default, could not take a row threshold after the upgrade. Only a count of 0 needs that refusal. A segment below the threshold leaves its dynamic paths in the sparse column. A compaction of a V1 column with a count of 0 then chooses every path to extract, which drops them from the sparse column, but only builds the subcolumns that its inputs extracted, so the paths that only the sparse column held would be lost. With a count above 0, V1 and V2 behave the same. - Column.checkSchemaChangeAllowed refuses a non-zero threshold only on a V1 column without doc mode whose count is 0, with a message that says so. - A load below the threshold uses the V2 meaning of a count of 0 again (extract no dynamic path), so a V1 column with a count above 0 does not extract every path there. - Step 4 of the extended compaction schema ends with an explicit branch for a V2 column with a count of 0 and no typed path, which asserts that the compaction chose no path to extract: such a path would be dropped from the sparse column with no subcolumn to hold it. ### Release note MODIFY COLUMN can set `variant_subcolumn_extraction_min_rows` on a VARIANT column without doc mode that was created before the upgrade, unless its `variant_max_subcolumns_count` is 0. ### Check List (For Author) - Test: Unit Test, Manual test - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1050 run, 1041 passed, 9 skipped, 0 failed. The compaction matrix runs V1 columns with a count above 0 with thresholds again, expecting the paths of V2. - FE UT: ColumnTest (20 tests) and VariantSerializeTypeDdlTest (4 tests), 0 failures; ColumnTest covers a V1 column with a count of 2048 taking a threshold and one with a count of 0 refusing it. - Manual test on a local cluster: a new column with a count of 0 and no typed path, with and without a row threshold, through two loads and a full compaction: no subcolumn extracted, every value read back, no assertion. No regression suite was run for this commit. - clang-format, clang-tidy on the changed lines, checkstyle and check-build-hygiene.sh are clean. - Behavior changed: Yes. See the release note. - Does this need documentation: No. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…to sparse ### What problem does this PR solve? Issue Number: None Related PR: apache#69812 Problem Summary: The row threshold `variant_subcolumn_extraction_min_rows` now applies with and without doc mode, but fuzzy mode (`use_fuzzy_session_variable`, on in the p0, p1, nonConcurrent and external pipelines) still randomized `default_variant_subcolumn_extraction_min_rows` with doc mode only. `default_variant_enable_typed_paths_to_sparse` is an internal switch for tests now, yet no fuzzy run ever set it to true: SessionVariable.initFuzzyModeVariables only forced it to false with doc mode. - initFuzzyModeVariables randomizes the row threshold in both modes, and `default_variant_enable_typed_paths_to_sparse` without doc mode. Doc mode keeps it false; the parser forces it to false there anyway. - The parser's "only support for ..." error no longer lists `variant_enable_typed_paths_to_sparse`, which is internal now. The property still takes `false` from the DDL of released versions. - Suites that pin every other fuzzed VARIANT variable now pin these two where their output depends on which paths a data file extracts. They were found by running every suite that defines a VARIANT column (variant_p0 and 106 others) with the global defaults set to a threshold of 19 and typed paths to sparse, then with each one alone, against a run with the defaults: - 18 suites pin the threshold to 0: the type text they print shows it, they check MATCH, SEARCH or scores with `enable_match_without_inverted_index = false` or need an index on every path, or a debug point checks the layout. - 3 suites pin typed paths to sparse to false: test_variant_v2_snii_index (count 0, so typed paths to sparse leave no indexed path) and both test_array_contains_with_inverted_index (a typed array path left sparse keeps an empty array that its subcolumn drops). ### Release note None ### Check List (For Author) - Test: Regression test, Unit Test - With the global defaults set to a threshold of 19 and typed paths to sparse, before the pins: variant_p0 187 suites, 17 failed; 106 other suites, 12 failed. After the pins, the 21 pinned suites pass with those settings and with the defaults. - Suites that fail with the defaults too, for environment reasons: outfile_csv_variant_doc_value (S3 credentials), test_get_tables_schema (arrow flight), test_default_expr and test_json_load (time zone), nereids_insert_random, test_bitmap_filter and test_bitmap_filter_nereids (databases of other suites), test_ivm_agg_expr_over_agg_1 (enable_feature_changelog). - FE UT: NereidsParserTest, 95 tests, 0 failures. - checkstyle is clean. - Behavior changed: No (fuzzy tests and an error message only). - Does this need documentation: No. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…t extract every path ### What problem does this PR solve? Issue Number: None Related PR: apache#69812 Problem Summary: The serialize type of a VARIANT column (V1 for columns created before the upgrade, V2 for new ones) is meant to decide one thing: what a subcolumn count of 0 means, every path under V1 and no dynamic path under V2. MODIFY COLUMN also used it to refuse `variant_subcolumn_extraction_min_rows` on every V1 column without doc mode, so a table of an older version, whose count is 2048 by default, could not take a row threshold after the upgrade. Only a count of 0 needs that refusal. A segment below the threshold leaves its dynamic paths in the sparse column. A compaction of a V1 column with a count of 0 then chooses every path to extract, which drops them from the sparse column, but only builds the subcolumns that its inputs extracted, so the paths that only the sparse column held would be lost. With a count above 0, V1 and V2 behave the same. - Column.checkSchemaChangeAllowed refuses a non-zero threshold only on a V1 column without doc mode whose count is 0, with a message that says so. - select_variant_paths takes the rows of the data file and the shredder options and applies the row threshold itself: below it no dynamic path is extracted, whatever the serialize type. The load used to pass a count of 0 for that, which under V1 extracts every path, so a V1 column with a count above 0 would have extracted every path below the threshold. Both callers, the load without doc mode and the load with doc mode, pass the options; the selection is unchanged otherwise. - Step 4 of the extended compaction schema ends with an explicit branch for a V2 column with a count of 0 and no typed path, which asserts that the compaction chose no path to extract: such a path would be dropped from the sparse column with no subcolumn to hold it. ### Release note MODIFY COLUMN can set `variant_subcolumn_extraction_min_rows` on a VARIANT column without doc mode that was created before the upgrade, unless its `variant_max_subcolumns_count` is 0. ### Check List (For Author) - Test: Unit Test, Manual test - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1050 run, 1041 passed, 9 skipped, 0 failed. The compaction matrix runs V1 columns with a count above 0 with thresholds again, expecting the paths of V2. The selection test covers a data file below and at the threshold under V1 and V2. - FE UT: ColumnTest (20 tests) and VariantSerializeTypeDdlTest (4 tests), 0 failures; ColumnTest covers a V1 column with a count of 2048 taking a threshold and one with a count of 0 refusing it. - Manual test on a local cluster: a new column with a count of 0 and no typed path, with and without a row threshold, through two loads and a full compaction: no subcolumn extracted, every value read back, no assertion. Again after the selection change, with a column of count 2 and a threshold of 3 (a 2-row load extracts nothing, a 4-row load its top 2, the compaction of both the top 2 of all rows) and a doc mode column of count 1. No regression suite was run for this commit. - clang-format, clang-tidy on the changed lines, checkstyle and check-build-hygiene.sh are clean. - Behavior changed: Yes. See the release note. - Does this need documentation: No. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…to sparse ### What problem does this PR solve? Issue Number: None Related PR: apache#69812 Problem Summary: The row threshold `variant_subcolumn_extraction_min_rows` now applies with and without doc mode, but fuzzy mode (`use_fuzzy_session_variable`, on in the p0, p1, nonConcurrent and external pipelines) still randomized `default_variant_subcolumn_extraction_min_rows` with doc mode only. `default_variant_enable_typed_paths_to_sparse` is an internal switch for tests now, yet no fuzzy run ever set it to true: SessionVariable.initFuzzyModeVariables only forced it to false with doc mode. - initFuzzyModeVariables randomizes the row threshold in both modes, and `default_variant_enable_typed_paths_to_sparse` without doc mode. Doc mode keeps it false; the parser forces it to false there anyway. - The parser's "only support for ..." error no longer lists `variant_enable_typed_paths_to_sparse`, which is internal now. The property still takes `false` from the DDL of released versions. - Suites that pin every other fuzzed VARIANT variable now pin these two where their output depends on which paths a data file extracts. They were found by running every suite that defines a VARIANT column (variant_p0 and 106 others) with the global defaults set to a threshold of 19 and typed paths to sparse, then with each one alone, against a run with the defaults: - 18 suites pin the threshold to 0: the type text they print shows it, they check MATCH, SEARCH or scores with `enable_match_without_inverted_index = false` or need an index on every path, or a debug point checks the layout. - 3 suites pin typed paths to sparse to false: test_variant_v2_snii_index (count 0, so typed paths to sparse leave no indexed path) and both test_array_contains_with_inverted_index (a typed array path left sparse keeps an empty array that its subcolumn drops). ### Release note None ### Check List (For Author) - Test: Regression test, Unit Test - With the global defaults set to a threshold of 19 and typed paths to sparse, before the pins: variant_p0 187 suites, 17 failed; 106 other suites, 12 failed. After the pins, the 21 pinned suites pass with those settings and with the defaults. - Suites that fail with the defaults too, for environment reasons: outfile_csv_variant_doc_value (S3 credentials), test_get_tables_schema (arrow flight), test_default_expr and test_json_load (time zone), nereids_insert_random, test_bitmap_filter and test_bitmap_filter_nereids (databases of other suites), test_ivm_agg_expr_over_agg_1 (enable_feature_changelog). - FE UT: NereidsParserTest, 95 tests, 0 failures. - checkstyle is clean. - Behavior changed: No (fuzzy tests and an error message only). - Does this need documentation: No. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…nt and fail reads of a mismatched layout ### What problem does this PR solve? Issue Number: None Related PR: apache#69812 Problem Summary: A review of this PR found two problems and a stale comment. - CTAS makes every VARIANT column it creates V2, but it kept the count of the query's type. The type of a VARIANT expression such as `parse_to_variant` (the default VARIANT type of the planner) has a count of 0 under V1, which extracts every path, so `CREATE TABLE t AS SELECT parse_to_variant(s) AS v ...` created a column with a count of 0 under V2, which extracts no dynamic path: every path went to the sparse column, while 4.1 extracts them all. The same happened when CTAS copies a column of an older version with a count of 0, or a VARIANT column of a file table function. Such a type now becomes V2 with the session's default count (`default_variant_max_subcolumns_count`), as a column declared without the count would; a count of 0 that a new column states is kept, and so is the count of a nested group column. - The reader checks that a V2 column with a count of 0 holds no dynamic subcolumn. A BE of an older version, which does not know the serialize type, writes such a column with every path extracted, for example when an FE is upgraded before every BE. The check was a DORIS_CHECK, so opening such a segment crashed the BE, and compaction, which retries the tablet, crashed it again. The check now returns an InternalError that names the segment, so the query or the compaction fails instead. It stays an error: compaction of such a column folds none of those subcolumns into the sparse column and would drop them. - A comment in LogicalPlanBuilder still said that the property check reads deprecated names; that code went away with the rename. ### Release note None ### Check List (For Author) - Test: Unit Test, Regression test, Manual test - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1050 run, 1041 passed, 9 skipped, 0 failed. reader_rejects_dynamic_subcolumn_of_v2_count_zero_column now expects the InternalError with the path and the segment file instead of a crash. - FE UT: VariantSerializeTypeDdlTest, 6 tests, 0 failures; it covers CTAS of an old column with a count of 0, of a VARIANT expression, and of a new column with a count of 0. - Regression: test_variant_count_zero_keeps_paths_sparse covers CTAS of `parse_to_variant(...)`; its .out was regenerated by the framework and only gains the two new tags. Every suite of variant_p0 (187), where only outfile_csv_variant_doc_value fails because it needs S3 credentials, and 6 suites outside it that run CTAS or views over VARIANT columns. - Manual test: the reproduction (CTAS of `parse_to_variant('{"a": 1, "b": 2}')`) gave a count of 0 and no subcolumn before, a count of 2048 and the subcolumns v.a and v.b after. - clang-format, clang-tidy on the changed lines, checkstyle and check-build-hygiene.sh are clean. - Behavior changed: Yes. CTAS of a VARIANT expression, or of an older column with a count of 0, keeps extracting the paths, as in 4.1. - Does this need documentation: No. Co-Authored-By: Claude Opus 5.5 <[email protected]>
### What problem does this PR solve? Issue Number: None Related PR: apache#69812 Problem Summary: The previous commit gave a column that CTAS creates from a type with a count of 0 under V1 (a column of an older version, the type of a VARIANT expression such as `parse_to_variant`, or a column of a file table function) the session's default subcolumn count, so that it kept extracting every path as in 4.1. A column that CTAS creates is a new column, and on a new column a count of 0 means that no dynamic path is extracted. CTAS keeps the count of the query's type again, as before that commit: such a column is V2 with a count of 0 and keeps its dynamic paths in the sparse column. Casting the expression to `VARIANT` in the query gives the column the session's default count instead, because the target type of the cast is parsed like a declared column. The reader check and the comment fix of the previous commit stay. ### Release note CTAS of a VARIANT expression such as `parse_to_variant(...)`, or of a VARIANT column of an older version with `variant_max_subcolumns_count = 0`, creates a column that extracts no dynamic path, while 4.1 extracts every path; cast the expression to `VARIANT` to get the default subcolumn count. ### Check List (For Author) - Test: Unit Test, Regression test, Manual test - FE UT: VariantSerializeTypeDdlTest, 6 tests, 0 failures; CTAS of an older column with a count of 0, of a VARIANT expression and of a new column with a count of 0 all give V2 with a count of 0. - Regression: test_variant_count_zero_keeps_paths_sparse, its .out regenerated by the framework (the CTAS of `parse_to_variant(...)` prints a count of 0 and no subcolumn, and reads back the same values); test_variant_tuning_properties and the 6 suites outside variant_p0 that run CTAS or views over VARIANT columns. 8 suites, 0 failed. - Manual test: CTAS of `parse_to_variant('{"a": 1, "b": 2}')` gives a count of 0 and no subcolumn; with `CAST(... AS VARIANT)` a count of 2048 and the subcolumns v.a and v.b. - checkstyle is clean. - Behavior changed: Yes. See the release note. - Does this need documentation: Yes, the CTAS rule goes with the VARIANT documentation. Co-Authored-By: Claude Opus 5.5 <[email protected]>
What problem does this PR solve?
variant_max_subcolumns_count = 0means. On a new column (V2), 0 means no dynamic subcolumns and every dynamic path stays in the sparse column. A column created before this field (V1) keeps the old meaning, where 0 materializes every dynamic path.doc: apache/doris-website#4578
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)