Skip to content

[feature](changelog) Support VARIANT columns in changelog - #69812

Open
csun5285 wants to merge 15 commits into
apache:masterfrom
csun5285:variant-serialize-type
Open

csun5285 wants to merge 15 commits into
apache:masterfrom
csun5285:variant-serialize-type

Conversation

@csun5285

@csun5285 csun5285 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

  • Add a hidden field to the VARIANT type, the serialize type. Users cannot set it and DESC and SHOW CREATE TABLE do not print it.
    • It changes what variant_max_subcolumns_count = 0 means. On a new column (V2), 0 means no dynamic subcolumns and every dynamic path stays in the sparse column. A column created before this field (V1) keeps the old meaning, where 0 materializes every dynamic path.
    • Tables with changelog now support VARIANT columns. Their changelog images are always stored with this new count 0 layout, so no dynamic path of an image is materialized.

doc: apache/doris-website#4578

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

csun5285 and others added 2 commits October 29, 2026 06:00
…ynamic path in the sparse column

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

`variant_max_subcolumns_count = 0` means "materialize every path as a subcolumn", and nothing lets a
VARIANT column keep every path in its sparse column instead. A changelog needs exactly that for its
AFTER and BEFORE images: they are only read whole, so materializing their paths would double the
subcolumns of the source table for nothing.

This adds a per-column parameter `variant_serialize_type` (`v1` / `v2`). It only changes what a count
of 0 means:
- V1: count 0 materializes every dynamic path, the old meaning. Every column persisted before the parameter
  existed is V1 (FE images, ColumnPB and TColumnType without the field).
- V2: count 0 materializes no dynamic path; they all go to the sparse column(s). Typed paths of a
  schema template behave as before (materialized unless variant_enable_typed_paths_to_sparse).
- Any count above 0, doc mode and nested group behave the same under both values.

New DDL (CREATE TABLE, ADD COLUMN) creates V2 columns unless the column states
`"variant_serialize_type" = "v1"`.

FE: the catalog and nereids VariantType carry the field as the thrift enum TVariantSerializeType;
ColumnToThrift / ColumnToProtobuf send it as TColumnType.variant_serialize_type (8) and
ColumnPB.variant_serialize_type (36). DESC never shows it; SHOW CREATE TABLE and CREATE TABLE LIKE
state it for every VARIANT column (`"variant_serialize_type" = "v1"` or `"v2"`), so the printed DDL
replays to the same layout whatever the default for new columns is. VariantType equality (catalog and
nereids) and the MODIFY COLUMN check compare it only where it decides the layout (count 0, no doc mode,
no nested group), so restating an old column with a non-zero count as a bare VARIANT keeps working.

BE: TabletColumn keeps the field. The three places that give a count of 0 its meaning now branch on
it: the writer's select_variant_paths (V1 materializes every dynamic path, V2 none) and, in
get_extended_compaction_schema, get_subpaths (only for a count above 0 or V1) and the
all-materialized branch (V1 only). Typed paths, doc mode and nested group are untouched; doc mode and
nested-group columns never reach these places. Compaction of a V2 count-0 column fails with an
InternalError if an input still materializes a dynamic path, which it would otherwise drop. The read
path is unchanged: it already follows what each segment contains.

Upgrade: downtime upgrade only, no rollback. A BE or FE without this change does not know the field:
it treats a V2 count-0 column as V1, so FE and every BE must run the new version together before such
a column is written, and going back to an older version is not supported. DDL printed by this version
states variant_serialize_type, which an older version rejects.

A V2 count-0 column materializes no dynamic path, so it has no inverted index on one either.
Regression cases that use count 0 to get every dynamic path materialized now say so with
`"variant_serialize_type" = "v1"`: 9 suites that pin count 0 (subcolumn indexes for MATCH and
SEARCH, DESC of extracted subcolumns, typed subcolumn checks) and the variant_p2 load, whose
relational benchmark reads materialized subcolumns. Cases whose indexes are on schema-template typed
paths, and cases that only check query results, need nothing. The
fuzzy session variables no longer draw `default_variant_max_subcolumns_count = 0` outside doc mode:
a column created in such a session would lose the subcolumns that cases without a pinned count may
rely on.

### Release note

New VARIANT columns are created with `variant_serialize_type = v2`. For them
`variant_max_subcolumns_count = 0` stores every dynamic path in the sparse column instead of
materializing every one, so such a column has no subcolumn inverted indexes and index-only queries on its paths
(MATCH, SEARCH, BM25 scoring) on dynamic paths fail; typed paths of a schema template are still
materialized. Existing columns keep the old meaning; state
`"variant_serialize_type" = "v1"` on a new column to keep it. SHOW CREATE TABLE states
`variant_serialize_type` for every VARIANT column. Upgrading to this version needs downtime (FE and
all BEs together) and cannot be rolled back.

### Check List (For Author)

- Test: Unit Test, Regression test
    - BE UT: SchemaUtil*, TabletSchemaTest, VariantPathBuilderTest, VariantColumnWriterReaderTest,
      IndexStorageTestFixture.* and every *Variant* suite: 907 run, 898 passed, 9 skipped (5
      pre-existing GTEST_SKIPs, 4 nested-group cases: this build has no nested-group write path), 0
      failures. New AllFeVariantParameters/VariantCompactionSchemaMatrixTest: every VARIANT parameter
      combination the FE accepts (serialize type x count 0/3/100 x typed paths to sparse x schema
      template x sparse shards, doc mode x min rows x doc shards, nested group); for each, the
      compaction schema follows the rule, the written and the compacted layouts follow the same rule,
      and a real compaction keeps every value. VariantV2CountZeroRejectsMaterializedDynamicInput
      covers the InternalError.
    - FE UT: TypeTest, ColumnTest, VariantSerializeTypeDdlTest, PropertyAnalyzerTest,
      NereidsParserTest, DataTypeTest, ScalarTypeTest, SessionVariablesTest,
      VariantLoadParseInjectionTest, PruneNestedColumnTest, BindSinkConnectorVariantTest,
      ConnectorComputeVariantPersistenceTest, VariantWritePlanValidatorTest: 0 failures;
      NereidsParserTest.testVariantPropertyCombinations parses the same combinations and rejects the
      five conflicting ones
    - Regression (local coupled cluster): the 27 p0 suites that pin variant count 0: all pass (9 of
      them fail without v1). The 9 suites whose random count can be 0, run with the count pinned to
      0: all pass without v1. Not run: variant_p2 (needs the 44M-row GitHub
      Events dataset), cloud mode, external pipelines, full p0.
    - clang-format 16 clean; clang-tidy finds nothing on the changed lines (variant_path_builder.cpp
      with clang-analyzer-* off: the 20.1 analyzer crashes in the unchanged rescale_decimal). The new
      segment test carries NOLINTNEXTLINE(readability-function-cognitive-complexity) like the other
      variant segment tests: GTest macros inflate the case matrix.
- Behavior changed: Yes. New VARIANT columns default to V2, so an explicit
  `variant_max_subcolumns_count = 0` on a new column stores every dynamic path in the sparse column.
- Does this need documentation: Yes (no doc PR yet)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

A table with changelog could not hold a VARIANT column: CREATE TABLE, the changelog schema generation
and the three ADD COLUMN paths rejected it.

This lets VARIANT columns into changelog:
- Both changelog images of a VARIANT column, the AFTER column `v` and the BEFORE column
  `__DORIS_CHANGELOG_BEFORE_v__`, keep every path in the sparse column (serialize type V2 with
  `variant_max_subcolumns_count = 0` and variant_enable_typed_paths_to_sparse, no doc mode, no
  nested group), whatever the layout of the source column. Column.generateAfterValueColumn / generateBeforeValueColumn install that layout
  through the new VariantType.forChangelog, so CREATE TABLE and ADD COLUMN both get it; the
  predefined fields stay, so typed paths keep their declared types. Changelog images are read whole,
  so materializing their paths would only double the subcolumns and the compaction work of the
  source table.
- A VARIANT column on a table with changelog must be V2 and can not enable nested group: CREATE
  TABLE and ADD COLUMN reject it (changelog can only be enabled at CREATE TABLE).
- A changelog read never pushes a Variant sub-path into the scan. VariantSubPathPruning leaves a
  statement unchanged when it contains a changelog scan (OlapChangelogTableWrapper), so sub-paths
  are taken from the whole value above the scan. Two reasons: BlockReader maps a BEFORE image onto
  its AFTER column by whole column, so a pushed sub-path of an AFTER column would show the AFTER
  value in an UPDATE_BEFORE row; and a stream plan unions a base-table scan with a changelog scan,
  whose union pushdown needs every child to supply the same sub-path slots. Predicates on value
  columns already stay above BlockReader for MIN_DELTA / DETAIL scans, so a filter on a sub-path
  sees the image of its own row. The cost is that such a statement reads its Variant columns whole,
  including from the base table.

Upgrade: as for the serialize type, downtime upgrade only and no rollback: a changelog VARIANT image
relies on the V2 count-0 layout, which an older FE or BE does not know.

### Release note

Tables with changelog support VARIANT columns (serialize type v2, no nested group). Reads through table streams, CHANGES and changelog()
read Variant columns whole and take sub-paths from the whole value.

### Check List (For Author)

- Test: Regression test, Unit Test
    - Regression (local coupled cluster): new changelog_p0/test_changelog_variant (raw changelog
      with both images and their sub-paths, MIN_DELTA and DETAIL streams with sub-paths and a
      sub-path filter, changelog compaction, a stream over history and incremental partitions,
      ADD COLUMN); changelog_p0 and table_stream_p0: 42 suites, 0 failures. Not run in cloud mode.
    - FE UT: OlapTableChangelogSchemaTest, SchemaChangeHandlerTest (testWithChangelogOpNotSupported
      now accepts VARIANT in CREATE TABLE and ADD COLUMN and rejects v1 and nested group),
      ExplainTableStreamPlanTest: 0 failures. The regression runs above predate the v2 / nested
      group rule and the typed paths of changelog images going to the sparse column.
- Behavior changed: Yes. VARIANT columns are accepted on tables with changelog.
- Does this need documentation: Yes (no doc PR yet)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@csun5285
csun5285 requested a review from yiguolei as a code owner October 29, 2026 06:25
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285

Copy link
Copy Markdown
Contributor Author

/review

csun5285 and others added 2 commits October 29, 2026 06:59
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

After compaction, VariantCompactionUtil::check_path_stats requires every variant path of a DUP_KEYS
tablet to keep its non-null count, since such a tablet never merges rows. A changelog tablet is
DUP_KEYS, but its compaction drops the rows its delete bitmap marks, for example a change that a
higher sequence value overrode. With a VARIANT column in a table with changelog, the check then
failed every compaction of that changelog tablet ("Path stats not match ... input size 3, output
size 2"), so its versions piled up. Before VARIANT was allowed in changelog this could not happen.

check_path_stats now skips changelog tablets, as it skips every non-DUP_KEYS tablet.

Reproduce: a MOW table with changelog, need_historical_value, a sequence column and a VARIANT
column; write a key, rewrite it with a lower sequence value, compact (changelog_p0
test_changelog_compaction with a VARIANT column): the compaction failed with INTERNAL_ERROR; it now
succeeds and the changelog keeps every image.

### Release note

None

### Check List (For Author)

- Test: Regression test
    - changelog_p0/test_changelog_compaction and test_cloud_changelog_compaction (cloud mode) with a
      VARIANT column, in the next commit: fail before, pass after
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… suites with a VARIANT column

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Only changelog_p0/test_changelog_variant covered VARIANT columns in changelog. 22 existing suites
now also give their changelog tables a `jv VARIANT` column, written on every full-row INSERT as
parse_to_variant('{"k": <first value>, "d": {"row": "<row values>"}}'), so an update changes it, and
read with jv['d']['row'] next to the columns they already check:
- changelog_p0: basic (partial update not touching jv, DELETE, sequence column), partial update
  of keys only, MOW light delete, compaction, multi segment (jv from a stream load column
  mapping), auto partition, TTL, CHANGES syntax, append-only projection, cloud compaction
- table_stream_p0: MIN_DELTA stream, MIN_DELTA delete-before, snapshot and reset streams (sinks
  carry jv and every row's jv['k'] must equal its id)
- time_travel_p0: MOW and DUP time travel, including the old jv of a snapshot and after compaction
- mtmv_p0/ivm: basic (jv flows into SELECT * materialized views through incremental refresh),
  aggregation, inner join, DUP keys and MTMV changelog
- cloud_p0/table_stream: end to end
The other columns of every existing result are unchanged; only jv columns are appended. Two
assertions count the visible columns or compare SELECT t.* and now account for jv.
changelog_p0/test_changelog_flexible_partial_update is left alone: flexible partial update rejects
tables with a VARIANT column.

### Release note

None

### Check List (For Author)

- Test: Regression test
    - all 22 suites pass on a local coupled cluster, and the two cloud suites on a local cloud
      cluster; .out files regenerated with -forceGenOut and checked: no existing column changed
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/40829143390

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@csun5285

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: one P1 data-loss issue and two P2 issues (planning failure and an assertion-enabled cloud BE abort) were verified at commit 58fae14. All 88 changed files were reviewed, including surrounding producers/consumers, tests and expected results. There were no preexisting inline findings and no additional user-provided focus points.

Critical checkpoint conclusions:

  1. Goal and proof: VARIANT changelog support and V2 zero-count layouts are implemented across FE/BE, but mixed-layout compaction, base/changelog-TVF UNION extraction and assertion-enabled cloud conflict publishing remain incorrect (inline findings). Added tests do not cover those triggers.
  2. Scope and reuse: production changes are focused on layout metadata, path selection and changelog enablement; shared writer helpers are reused. Regression changes exercise related existing features.
  3. Concurrency and locking: no new threads, atomics, locks or lock ordering; existing metadata mutation and rowset publication boundaries remain in use. The cloud P2 is triggered by concurrent same-key publishing; no additional race or deadlock found.
  4. Lifecycle and initialization: the layout flag follows existing schema ownership, and forChangelog creates a separate type. No new cycles, owner-lifetime or cross-translation-unit initialization issue found.
  5. Configuration: variant_serialize_type is a persisted per-column DDL property; MODIFY rejects changes where count-zero changes layout. No new BE dynamic process setting. Fuzzy session defaults avoid zero outside doc mode.
  6. Compatibility: absent Gson/thrift/PB fields retain V1 and current root serializers preserve V2. Optionality does not prevent old BEs from producing the mixed layouts in the P1 finding.
  7. Shared definitions: inspected complete touched definitions on HEAD and live-matching branch-3.1/4.0/4.1/4.2 refs. ColumnPB field 36 and TColumnType field 8 are unused there; additions are optional, enum values agree, defaults preserve V1. No ID collision found.
  8. Parallel paths: inspected CREATE, ADD/ADD COLUMNS, MODIFY, SHOW CREATE/CREATE LIKE, CTAS, TVF inference, local/cloud schema messages, ordinary/doc/nested-group storage, typed paths, streams, time travel, CTEs and unions. The missed changelog-TVF path and cloud historical-image count invariant are the P2 findings.
  9. Conditions: layout and image guards have concrete explanations; the inline findings identify incomplete guard/invariant coverage.
  10. Test breadth: inspected FE parser/property/equality/Gson/DDL tests, BE writer/compaction layout matrices, and changelog/cloud/IVM/stream/time-travel regressions. Missing mixed-upgrade, matching-layout TVF-UNION and concurrent cloud historical-VARIANT publish cases should be added.
  11. Expected results: all modified output hunks were checked against their changed INSERT/projection and historical-image semantics; no separate mismatch found. Output-generation provenance is unverified.
  12. Observability: existing compaction error reporting carries column/path context; no new independent logging/metrics gap found. The silent-loss path evades that error and changelog path-stat checking.
  13. Persistence and failover: the annotated flag follows existing Column image/EditLog and tablet-schema PB mechanisms. No new transaction or journal state machine; schema-version transitions were traced for the P1 finding and publish-time conflict rewrites for the cloud P2.
  14. Writes and atomicity: existing segment/rowset commit mechanisms remain in use; committed changelog values can nevertheless be lost by the P1 compaction path. The cloud P2 also aborts DEBUG/ASAN BEs during conflict publishing. No additional crash-cleanup or atomicity defect established.
  15. FE/BE propagation: primary TColumnType/ColumnPB producers and TabletMeta consumers carry the flag; read-only runtime/extracted types do not select root storage layouts.
  16. Performance and memory: existing allocation/ownership and sorting paths are reused; the whole-plan scan is linear and deliberately disables subpath pruning with changelog scans. No additional substantiated CPU/memory issue found.
  17. Further recheck: legacy ScalarType, sparse-stat truncation, empty/typed-only inputs, source-image copying, schema caches, cloud index rebuilds and optimizer stage order were revisited; no additional distinct finding or unresolved candidate remains.

Validation was static only: no builds, tests, benchmarks, runtime reproductions, source edits or submodule initialization were performed. Runtime behavior and output provenance remain unverified within that required scope.

Review converged after 3 round(s).

Comment thread be/src/exec/common/variant_util.cpp Outdated
// Every input path of a V2 count-0 column is in the sparse column(s) or typed. Nothing
// would carry a materialized dynamic subcolumn into the output, so its data would be lost.
for (const auto& [path, _] : extended_info.path_to_none_null_values) {
if (!extended_info.sparse_paths.contains(path) &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Check materialized paths per input before accepting this layout

extended_info.sparse_paths is the union across every input segment, so an older segment materializing a passes this guard whenever another segment stores a sparsely. This is reachable during a rolling upgrade: the new FE sends V2/count-zero changelog columns to an old BE, which ignores the new field and materializes their paths; after upgrading that BE, a light ADD COLUMN and another load produce a higher-schema-version V2 sparse rowset. Compaction selects that V2 schema, but this branch leaves paths_set_info.sparse_path_set empty. _create_sparse_merge_reader therefore skips the older materialized a, while the root-only reader/writer cannot recover it (object roots are empty). Old changelog images silently lose a, and check_path_stats now skips changelog tablets. Populate the sparse transfer set for materialized input leaves, or reject them using per-input materialized metadata rather than membership in the union of sparse paths.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok,i will add this check

// its AFTER column by whole column, so a changelog scan reads Variant columns whole and sub-paths are taken
// above it. Its plan may union the changelog scan with a base table scan, whose union pushdown needs every
// child to supply the same sub-path slots, so the whole statement keeps whole-column reads.
if (plan.anyMatch(node -> node instanceof LogicalOlapScan

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Include changelog TVFs in the whole-column guard

changelog(...) remains a LogicalTVFRelation during this rewrite; TableChangelogFunction.getScanNode attaches the wrapper only during physical translation, so this predicate misses it. For a populated changelog-enabled table t (with an inserted object containing a) and nullable v VARIANT<PROPERTIES("variant_max_subcolumns_count"="0","variant_enable_typed_paths_to_sparse"="true")>, SELECT v['a'] FROM (SELECT v FROM t UNION ALL SELECT v FROM changelog("table"="t")) u has matching source/image types and no union casts. Its reduced plan is:

Project(element_at(v_union, 'a'))
  UnionAll
    OlapScan(t.v)
    TVF(changelog.v)

visitLogicalOlapScan generates v.a only for the first child; there is no TVF replacer. visitLogicalUnion then indexes entry.getValue().get(1) although the list contains one slot, failing planning (reversing branches hits the missing-subpath exception). PushProjectThroughUnion rejects ElementAt, so it does not eliminate this shape. Recognize changelog TVF relations here too, or synthesize the missing child expressions, and cover this matching-layout UNION case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, i will ban this case

} else if (column.getType().isVariantType() && column.getVariantEnableNestedGroup()) {
throw new AnalysisException(
"variant column with nested group can't be created on table with changelog");
} else if (column.getType().isVariantType()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Update the cloud Variant-count invariant for BEFORE images

The newly accepted cloud MOW configuration with one VARIANT column and changelog.need_historical_value=true has one source VARIANT but two changelog VARIANT roots (AFTER and BEFORE). CloudMetaMgr::update_tmp_rowset still asserts that those counts are equal at cloud_meta_mgr.cpp:1697-1698. This is reachable in DEBUG/ASAN builds: let two same-key UPSERTs flush against the same visible snapshot, then publish one before the other. The second publish's BaseTablet::update_delete_bitmap creates a transient group writer to rebuild the historical image, emits replacement rows, and reaches CloudTablet::save_delete_bitmap with nonzero writer rows; that calls update_tmp_rowset and aborts the BE on DCHECK_EQ(1, 2). RELEASE builds compile out this assertion, but assertion-enabled cloud builds cannot handle this supported write schedule. Make the cloud invariant account for the BEFORE columns and cover concurrent publish-time rewrites with a historical VARIANT column.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, i will change this check

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.65% (2101/2741)
Line Coverage 66.04% (38812/58772)
Region Coverage 53.80% (36446/67743)
Branch Coverage 57.22% (11858/20722)

… images in cloud changelog

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Review findings on the VARIANT serialize type and changelog support:

1. Compaction of a V2 count-0 column rejects an input that materializes a dynamic path, because it
   would lose that path. The check tested every stats path against the union of the inputs' sparse
   paths, so a segment materializing `a` passed whenever another segment kept `a` in its sparse
   column, and the materialized `a` was dropped. The inputs' materialized paths are now collected
   on their own (VariantExtendedInfo.materialized_paths) and any non-typed one fails the compaction.

2. CloudMetaMgr::update_tmp_rowset asserted that a rowset and its changelog rowset have as many
   VARIANT columns. With changelog.need_historical_value the changelog also keeps a BEFORE image of
   every VARIANT column, so an assertion-enabled cloud BE aborted when a publish-time conflict
   rewrite reached it. The assertion now adds the BEFORE images.

A third finding, a UNION of a base table and its changelog() TVF failing to plan when a VARIANT
sub-path is read above it, did not reproduce: the sub-path projection is pushed into both union
children and the result is correct. changelog_p0/test_changelog_variant now covers that query.

### Release note

None

### Check List (For Author)

- Test: Unit Test, Regression test
    - BE UT IndexStorageTestFixture.VariantV2CountZeroRejectsMaterializedDynamicInput now writes a
      V1 rowset materializing `a` and a V2 rowset keeping `a` sparse; it fails with the old
      union check and passes now. VariantCompactionSchemaMatrixTest: 64 passed, 4 nested-group
      cases skipped (no nested-group write path in this build).
    - Regression: changelog_p0/test_changelog_variant (with the new UNION query) and
      test_changelog_compaction pass on a local coupled cluster.
    - The cloud assertion is not reproduced locally (it needs a concurrent publish conflict in a
      cloud cluster); the fix is checked by reading the code.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28820 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 58fae14bf3eedbe87f3aa31b74f36a2b5a834431, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17618	5235	5057	5057
q2	2129	309	247	247
q3	10196	1391	835	835
q4	4676	480	336	336
q5	7431	800	530	530
q6	179	183	150	150
q7	760	810	604	604
q8	9352	1251	1126	1126
q9	5657	4208	4207	4207
q10	6887	1299	1033	1033
q11	431	249	233	233
q12	638	392	271	271
q13	18132	3548	3041	3041
q14	275	271	246	246
q15	q16	761	727	655	655
q17	1214	774	585	585
q18	6908	6204	6115	6115
q19	1088	977	614	614
q20	396	332	223	223
q21	5052	2517	2429	2429
q22	383	332	283	283
Total cold run time: 100163 ms
Total hot run time: 28820 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5947	5865	5868	5865
q2	724	619	573	573
q3	5565	5713	5369	5369
q4	2075	2133	1564	1564
q5	5850	5843	5635	5635
q6	238	192	144	144
q7	2224	2054	1799	1799
q8	2842	2459	2494	2459
q9	7554	7481	7480	7480
q10	3640	3564	3128	3128
q11	582	413	376	376
q12	672	664	499	499
q13	3237	3619	3029	3029
q14	283	301	269	269
q15	q16	666	668	639	639
q17	7815	7005	6854	6854
q18	13015	12228	12900	12228
q19	910	793	794	793
q20	2143	2138	1915	1915
q21	5698	4648	4642	4642
q22	502	454	427	427
Total cold run time: 72182 ms
Total hot run time: 65687 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151312 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 58fae14bf3eedbe87f3aa31b74f36a2b5a834431, data reload: false

query5	4315	594	441	441
query6	431	210	193	193
query7	4799	451	232	232
query8	330	163	154	154
query9	8783	3216	3202	3202
query10	478	312	262	262
query11	6039	3834	3583	3583
query12	145	91	88	88
query13	1243	430	332	332
query14	6155	3948	3630	3630
query14_1	3405	3427	3407	3407
query15	202	197	181	181
query16	935	419	425	419
query17	857	655	536	536
query18	2430	423	315	315
query19	223	181	137	137
query20	85	87	85	85
query21	209	128	113	113
query22	12967	13098	12757	12757
query23	13495	12725	12525	12525
query23_1	12601	12653	12550	12550
query24	7365	826	491	491
query24_1	495	485	498	485
query25	535	429	371	371
query26	1248	259	181	181
query27	2726	434	265	265
query28	4622	2064	2062	2062
query29	1550	590	430	430
query30	294	216	183	183
query31	995	917	803	803
query32	144	90	86	86
query33	510	291	232	232
query34	904	867	500	500
query35	729	744	631	631
query36	829	818	728	728
query37	123	98	80	80
query38	1782	1713	1650	1650
query39	786	816	741	741
query39_1	727	733	729	729
query40	218	114	92	92
query41	65	64	62	62
query42	89	83	81	81
query43	340	364	309	309
query44	1318	705	707	705
query45	174	177	164	164
query46	831	961	564	564
query47	2833	3003	2700	2700
query48	313	313	221	221
query49	564	397	289	289
query50	687	272	201	201
query51	10407	10558	10499	10499
query52	77	80	69	69
query53	185	214	150	150
query54	250	202	187	187
query55	80	67	63	63
query56	219	208	213	208
query57	1644	1608	1567	1567
query58	280	265	260	260
query59	2219	2213	2002	2002
query60	281	245	228	228
query61	148	146	148	146
query62	381	344	281	281
query63	191	162	157	157
query64	2797	936	767	767
query65	3320	3266	3243	3243
query66	1786	416	301	301
query67	20244	20439	19914	19914
query68	3081	1016	625	625
query69	398	302	246	246
query70	839	769	769	769
query71	306	234	205	205
query72	2740	2305	2409	2305
query73	515	528	302	302
query74	4623	4509	4359	4359
query75	2328	2289	1914	1914
query76	2331	1006	589	589
query77	361	378	278	278
query78	9897	9701	9166	9166
query79	983	864	551	551
query80	606	437	363	363
query81	530	318	274	274
query82	636	132	104	104
query83	280	197	193	193
query84	306	125	102	102
query85	799	417	371	371
query86	350	234	226	226
query87	1987	1947	1807	1807
query88	3753	2770	2760	2760
query89	327	282	242	242
query90	1915	168	159	159
query91	157	150	136	136
query92	101	91	90	90
query93	1007	1000	576	576
query94	449	295	282	282
query95	587	382	312	312
query96	689	481	238	238
query97	2445	2415	2336	2336
query98	160	156	157	156
query99	765	759	667	667
Total cold run time: 234425 ms
Total hot run time: 151312 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 58fae14bf3eedbe87f3aa31b74f36a2b5a834431, data reload: false

query1	0.01	0.01	0.01
query2	0.13	0.09	0.08
query3	0.33	0.13	0.13
query4	1.60	0.14	0.13
query5	0.26	0.25	0.24
query6	1.15	0.65	0.64
query7	0.04	0.01	0.00
query8	0.09	0.08	0.07
query9	0.43	0.40	0.35
query10	0.57	0.60	0.57
query11	0.32	0.18	0.17
query12	0.31	0.18	0.19
query13	0.51	0.52	0.54
query14	0.93	0.94	0.92
query15	0.66	0.58	0.58
query16	0.35	0.34	0.36
query17	1.03	1.01	1.03
query18	0.28	0.28	0.27
query19	1.94	1.81	1.87
query20	0.02	0.01	0.02
query21	15.45	0.33	0.31
query22	4.94	0.13	0.13
query23	15.88	0.48	0.29
query24	2.02	0.58	0.40
query25	0.15	0.11	0.10
query26	0.77	0.27	0.21
query27	0.11	0.10	0.10
query28	3.48	0.77	0.38
query29	12.61	4.27	3.36
query30	0.35	0.16	0.14
query31	2.76	0.58	0.33
query32	3.23	0.59	0.48
query33	3.01	3.01	3.01
query34	15.87	4.02	3.44
query35	3.39	3.38	3.37
query36	0.58	0.50	0.50
query37	0.11	0.10	0.09
query38	0.07	0.07	0.07
query39	0.05	0.06	0.06
query40	0.20	0.17	0.16
query41	0.11	0.06	0.05
query42	0.06	0.05	0.06
query43	0.07	0.06	0.06
Total cold run time: 96.23 s
Total hot run time: 25.12 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 79.49% (31/39) 🎉
Increment coverage report
Complete coverage report

…mode too

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

The fuzzy session variables (CI p0, nonConcurrent, cloud_p0, p1) set
default_variant_max_subcolumns_count = 0 whenever they pick doc mode. A case that then turns doc mode
off, through `set default_variant_enable_doc_mode = false` or a column's
`"variant_enable_doc_mode" = "false"`, creates a V2 column with count 0, which materializes no
dynamic path, so MATCH and SEARCH on its subcolumn indexes fail. P0 hit it in
inverted_index_p0/test_bm25_score_variant ("match_phrase not support execute_match") and
variant_p0/test_variant_search_historical_rowset_index ("SEARCH index unavailable for Variant field
'v.missing'").

The fuzzer no longer zeroes the count in doc mode: a doc mode column takes count 0 anyway
(LogicalPlanBuilder), so the session default is only seen by columns without doc mode, and it now
always draws 1..9.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - SessionVariablesTest.testFuzzyVariantMaxSubcolumnsCountIsNeverZero (was
      ...IsZeroOnlyInDocMode): 24 run, 0 failures; fe-core checkstyle clean
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@csun5285

Copy link
Copy Markdown
Contributor Author

run buildall

@csun5285

csun5285 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/review

@csun5285

csun5285 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.65% (2101/2741)
Line Coverage 65.99% (38782/58772)
Region Coverage 53.79% (36439/67743)
Branch Coverage 57.22% (11857/20722)

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41556964845

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28953 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 008d6965263d80485e262bb13a29b4c1d2ac8e1c, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17401	5152	5048	5048
q2	2129	316	281	281
q3	10124	1407	832	832
q4	4682	474	344	344
q5	7448	815	535	535
q6	178	182	151	151
q7	757	798	611	611
q8	9282	1356	1068	1068
q9	5568	4230	4369	4230
q10	6893	1296	1040	1040
q11	445	250	227	227
q12	639	408	287	287
q13	18104	3760	3052	3052
q14	275	276	251	251
q15	q16	727	714	672	672
q17	1222	767	584	584
q18	7027	6186	6118	6118
q19	1121	974	595	595
q20	390	344	223	223
q21	5981	2534	2517	2517
q22	382	316	287	287
Total cold run time: 100775 ms
Total hot run time: 28953 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5937	5920	5931	5920
q2	736	633	602	602
q3	5629	5692	5215	5215
q4	2241	2165	1665	1665
q5	5794	5700	5809	5700
q6	239	191	149	149
q7	2262	1998	1808	1808
q8	2928	2544	2495	2495
q9	7524	7437	7497	7437
q10	3632	3588	3151	3151
q11	576	411	377	377
q12	677	665	515	515
q13	3389	3570	3103	3103
q14	287	300	263	263
q15	q16	663	679	618	618
q17	7859	7032	7011	7011
q18	13072	12202	12867	12202
q19	965	837	851	837
q20	2153	2160	1901	1901
q21	5707	4631	4734	4631
q22	511	449	432	432
Total cold run time: 72781 ms
Total hot run time: 66032 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151491 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 008d6965263d80485e262bb13a29b4c1d2ac8e1c, data reload: false

query5	4315	612	461	461
query6	442	202	188	188
query7	4797	450	230	230
query8	323	165	150	150
query9	8784	3194	3210	3194
query10	479	311	252	252
query11	6032	3873	3581	3581
query12	144	93	87	87
query13	1259	466	321	321
query14	6139	3913	3645	3645
query14_1	3434	3466	3431	3431
query15	215	207	191	191
query16	935	467	427	427
query17	889	699	550	550
query18	2432	429	335	335
query19	202	171	141	141
query20	85	87	86	86
query21	214	129	108	108
query22	12982	12968	12918	12918
query23	13583	12901	12660	12660
query23_1	12582	12556	12535	12535
query24	7350	892	455	455
query24_1	478	483	495	483
query25	523	395	337	337
query26	1268	251	136	136
query27	2788	441	268	268
query28	4617	2047	2031	2031
query29	1543	585	440	440
query30	287	221	188	188
query31	1021	926	812	812
query32	156	89	87	87
query33	529	280	236	236
query34	926	875	504	504
query35	732	764	651	651
query36	818	816	723	723
query37	120	96	83	83
query38	1811	1720	1651	1651
query39	811	784	742	742
query39_1	734	726	736	726
query40	214	127	97	97
query41	66	66	62	62
query42	85	85	79	79
query43	345	362	311	311
query44	1363	714	707	707
query45	190	176	160	160
query46	841	964	574	574
query47	2766	3026	2484	2484
query48	300	317	231	231
query49	569	407	292	292
query50	688	278	207	207
query51	10498	10602	10352	10352
query52	81	81	77	77
query53	189	218	160	160
query54	223	201	181	181
query55	72	66	64	64
query56	234	209	198	198
query57	1660	1610	1532	1532
query58	281	268	248	248
query59	2189	2195	2021	2021
query60	281	275	227	227
query61	147	152	151	151
query62	392	336	283	283
query63	192	157	160	157
query64	2740	931	758	758
query65	3355	3272	3277	3272
query66	1784	414	293	293
query67	20323	20129	20100	20100
query68	2988	1035	627	627
query69	393	282	250	250
query70	873	766	781	766
query71	286	232	213	213
query72	3031	2657	2307	2307
query73	529	553	301	301
query74	4655	4530	4381	4381
query75	2309	2320	1916	1916
query76	2290	1003	602	602
query77	373	381	273	273
query78	9746	9767	9177	9177
query79	907	882	542	542
query80	471	425	382	382
query81	514	323	270	270
query82	205	134	105	105
query83	194	200	184	184
query84	288	129	103	103
query85	769	423	366	366
query86	303	245	229	229
query87	2012	1975	1821	1821
query88	3825	2824	2791	2791
query89	316	283	244	244
query90	2067	171	175	171
query91	166	153	131	131
query92	101	87	89	87
query93	1005	1021	576	576
query94	452	298	282	282
query95	576	412	299	299
query96	689	517	241	241
query97	2451	2434	2300	2300
query98	161	149	149	149
query99	763	767	652	652
Total cold run time: 234255 ms
Total hot run time: 151491 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 008d6965263d80485e262bb13a29b4c1d2ac8e1c, data reload: false

query1	0.01	0.01	0.00
query2	0.14	0.08	0.09
query3	0.34	0.13	0.13
query4	1.60	0.13	0.14
query5	0.28	0.27	0.26
query6	1.16	0.66	0.65
query7	0.03	0.01	0.00
query8	0.09	0.07	0.07
query9	0.47	0.38	0.38
query10	0.61	0.61	0.61
query11	0.32	0.18	0.19
query12	0.32	0.19	0.19
query13	0.52	0.55	0.53
query14	0.95	0.95	0.96
query15	0.67	0.59	0.59
query16	0.36	0.37	0.37
query17	1.03	1.04	1.03
query18	0.30	0.29	0.28
query19	1.99	1.87	1.91
query20	0.02	0.01	0.02
query21	15.41	0.36	0.30
query22	4.91	0.13	0.13
query23	15.86	0.47	0.29
query24	1.99	0.55	0.38
query25	0.14	0.11	0.10
query26	0.77	0.28	0.21
query27	0.09	0.09	0.10
query28	3.55	0.88	0.40
query29	12.60	4.38	3.37
query30	0.34	0.16	0.14
query31	2.76	0.62	0.34
query32	3.22	0.62	0.49
query33	2.97	2.96	3.01
query34	15.91	4.12	3.44
query35	3.36	3.42	3.40
query36	0.60	0.52	0.49
query37	0.12	0.10	0.10
query38	0.07	0.06	0.07
query39	0.07	0.06	0.06
query40	0.19	0.17	0.17
query41	0.11	0.06	0.06
query42	0.07	0.06	0.06
query43	0.06	0.06	0.06
Total cold run time: 96.38 s
Total hot run time: 25.41 s

csun5285 and others added 2 commits November 3, 2026 03:11
Brings in test_match_keyword_score (apache#69790), which the next commit pins to a positive
subcolumn count. No conflicts; ColumnPB 36 and TColumn 31 stay unique after the merge.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…est_match_keyword_score

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812, apache#69790

Problem Summary:

test_match_keyword_score (apache#69790) creates `v VARIANT NULL`, so its subcolumn count comes from
the session default, which the fuzzy session variables can set to 0. With this PR a new column with
count 0 keeps every dynamic path in the sparse column, so `s` has no subcolumn index, MATCH still
finds the rows, and score() returns 0 instead of 0.693147. Cloud P0 build 1089519 failed this way on
`V2_variant_text_path`. The column now states a count of 2048, like the other suites that need
indexed subcolumns. The expected output is unchanged.

### Release note

None

### Check List (For Author)

- Test: Regression test. With `SET GLOBAL default_variant_max_subcolumns_count = 0` the suite fails
  on `V2_variant_text_path` without this change and passes with it; it also passes with the default
  count. FE UT on the merged code: 485 tests, 0 failures.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@csun5285

csun5285 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/review

@csun5285

csun5285 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41581383650

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@csun5285

csun5285 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 64.71% (22/34) 🎉
Increment coverage report
Complete coverage report

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41587717620

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@csun5285

csun5285 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28954 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24cb6a2d4a519fbc7b5375eedf3f812dd774ddec, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17607	5240	5048	5048
q2	2142	316	259	259
q3	10247	1406	866	866
q4	4680	476	347	347
q5	7443	820	533	533
q6	183	192	153	153
q7	763	795	604	604
q8	9263	1335	1119	1119
q9	5579	4226	4211	4211
q10	6843	1286	1019	1019
q11	442	245	230	230
q12	637	403	283	283
q13	18121	3690	3046	3046
q14	282	275	252	252
q15	q16	733	730	674	674
q17	1211	759	589	589
q18	7075	6188	6119	6119
q19	1107	988	631	631
q20	398	338	229	229
q21	6135	2520	2477	2477
q22	378	330	265	265
Total cold run time: 101269 ms
Total hot run time: 28954 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6002	5998	6123	5998
q2	725	620	613	613
q3	5540	5801	5379	5379
q4	2008	2201	1562	1562
q5	5790	5866	5500	5500
q6	241	191	150	150
q7	2266	2053	1793	1793
q8	2942	2648	2679	2648
q9	7967	7598	7472	7472
q10	3645	3573	3126	3126
q11	584	405	373	373
q12	670	670	515	515
q13	3325	3540	3023	3023
q14	280	286	265	265
q15	q16	660	693	621	621
q17	7696	6957	6894	6894
q18	12955	12207	12882	12207
q19	933	802	815	802
q20	2134	2126	1932	1932
q21	5636	4546	4688	4546
q22	497	458	417	417
Total cold run time: 72496 ms
Total hot run time: 65836 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151345 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24cb6a2d4a519fbc7b5375eedf3f812dd774ddec, data reload: false

query5	4319	597	443	443
query6	428	212	198	198
query7	4821	457	234	234
query8	324	182	153	153
query9	8763	3193	3181	3181
query10	458	302	267	267
query11	6036	3836	3569	3569
query12	146	92	88	88
query13	1280	440	331	331
query14	6148	3893	3658	3658
query14_1	3385	3398	3402	3398
query15	207	194	192	192
query16	932	431	422	422
query17	862	677	534	534
query18	2419	421	310	310
query19	202	167	139	139
query20	86	86	86	86
query21	213	127	110	110
query22	13028	12905	12758	12758
query23	13565	13059	12489	12489
query23_1	12486	12563	12543	12543
query24	7397	809	463	463
query24_1	491	473	476	473
query25	514	395	339	339
query26	1249	256	134	134
query27	2790	438	280	280
query28	4618	2027	2039	2027
query29	1559	585	430	430
query30	308	219	182	182
query31	1010	922	811	811
query32	149	87	87	87
query33	514	317	224	224
query34	907	861	501	501
query35	732	771	645	645
query36	809	843	748	748
query37	115	96	79	79
query38	1789	1755	1649	1649
query39	782	799	747	747
query39_1	729	740	743	740
query40	219	116	96	96
query41	66	64	61	61
query42	84	84	81	81
query43	345	363	314	314
query44	1320	698	711	698
query45	183	183	162	162
query46	830	968	564	564
query47	2953	2955	2564	2564
query48	303	315	220	220
query49	583	406	295	295
query50	710	288	213	213
query51	10560	10775	10452	10452
query52	80	83	70	70
query53	185	207	149	149
query54	238	188	171	171
query55	73	71	69	69
query56	238	206	198	198
query57	1632	1596	1540	1540
query58	290	255	256	255
query59	2151	2163	2012	2012
query60	299	239	219	219
query61	150	165	170	165
query62	386	355	287	287
query63	199	163	161	161
query64	2904	1067	921	921
query65	3355	3274	3299	3274
query66	1803	444	311	311
query67	20104	20512	20051	20051
query68	3078	1029	619	619
query69	391	304	268	268
query70	916	785	779	779
query71	295	238	210	210
query72	2729	2565	2312	2312
query73	522	539	295	295
query74	4638	4479	4376	4376
query75	2321	2311	1912	1912
query76	2322	1029	585	585
query77	341	373	265	265
query78	9790	9790	9235	9235
query79	1018	870	543	543
query80	1398	431	337	337
query81	602	324	276	276
query82	612	133	107	107
query83	279	196	184	184
query84	317	122	104	104
query85	1077	437	367	367
query86	394	247	224	224
query87	1989	1944	1802	1802
query88	3793	2806	2777	2777
query89	327	287	244	244
query90	1838	175	169	169
query91	166	144	126	126
query92	96	90	78	78
query93	1072	1000	594	594
query94	642	338	279	279
query95	588	383	315	315
query96	680	499	239	239
query97	2433	2440	2311	2311
query98	165	152	153	152
query99	754	766	649	649
Total cold run time: 236159 ms
Total hot run time: 151345 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.53 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24cb6a2d4a519fbc7b5375eedf3f812dd774ddec, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.06
query3	0.26	0.09	0.09
query4	1.62	0.11	0.10
query5	0.27	0.25	0.24
query6	1.18	0.67	0.69
query7	0.05	0.01	0.01
query8	0.05	0.04	0.04
query9	0.39	0.36	0.36
query10	0.58	0.58	0.60
query11	0.20	0.16	0.15
query12	0.18	0.15	0.15
query13	0.49	0.50	0.48
query14	1.03	1.01	1.01
query15	0.61	0.61	0.61
query16	0.34	0.35	0.36
query17	1.18	1.17	1.13
query18	0.21	0.19	0.18
query19	2.14	2.04	1.92
query20	0.02	0.01	0.02
query21	15.48	0.19	0.13
query22	4.86	0.05	0.06
query23	16.28	0.33	0.13
query24	2.91	0.53	0.50
query25	0.09	0.07	0.06
query26	1.54	0.22	0.15
query27	0.04	0.05	0.05
query28	4.02	0.82	0.37
query29	12.54	4.11	3.27
query30	0.28	0.13	0.10
query31	2.77	0.55	0.34
query32	3.23	0.60	0.50
query33	3.27	3.24	3.17
query34	15.44	4.01	3.37
query35	3.33	3.35	3.41
query36	0.63	0.51	0.50
query37	0.10	0.07	0.06
query38	0.05	0.05	0.04
query39	0.04	0.03	0.03
query40	0.16	0.16	0.15
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 98.12 s
Total hot run time: 24.53 s

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41607507679

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

csun5285 added a commit to csun5285/doris that referenced this pull request Nov 9, 2026
…hold and the shard count

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

The previous commit renamed `variant_doc_materialization_min_rows` to
`variant_subcolumn_extraction_min_rows` and `variant_doc_hash_shard_count` to
`variant_sparse_shard_count`, and kept the old names as aliases. Both old names belong to doc mode,
whose 4.1 columns already follow the new semantics on the assumption that doc mode is not in use
there, so they are not accepted any more: CREATE TABLE, ALTER TABLE and CAST reject them with the
error for unknown VARIANT properties ("only support for ..."). `variant_sparse_hash_shard_count`,
the shard count that released versions print without doc mode, stays an alias of
`variant_sparse_shard_count`, so their SHOW CREATE TABLE output still parses.

The regression suites that set the doc mode names set the new ones now, and
test_variant_tuning_properties checks that the old ones are refused.

### Release note

`variant_doc_materialization_min_rows` and `variant_doc_hash_shard_count` are no longer accepted; use
`variant_subcolumn_extraction_min_rows` and `variant_sparse_shard_count`. The SHOW CREATE TABLE
output of a doc mode table from 4.1 has to be renamed before it runs.

### Check List (For Author)

- Test: Unit Test, Regression test
    - FE UT: NereidsParserTest, PropertyAnalyzerTest and CreateTableCommandTest, 129 tests,
      0 failures.
    - Regression: the 8 suites whose VARIANT properties changed, where test_variant_tuning_properties
      is regenerated by running it and its two refusals fail with "only support for ..."; and
      check_before_quit, which recreates every table from its SHOW CREATE TABLE and compares.
- Behavior changed: Yes. The two doc mode property names are refused.
- Does this need documentation: Yes, the documentation uses the new names.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 9, 2026
… let MODIFY COLUMN change the row threshold

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812, apache#61100

Problem Summary:

Two choices of the earlier commits went further than renaming the properties:

- `variant_sparse_shard_count` was written into both stored shard fields, `variantSparseHashShardCount`
  and `variantDocShardCount`, and MODIFY COLUMN compared the field of the column's mode instead of
  both fields. The property only has a new name: with doc mode the value belongs to the doc shard
  field and without it to the sparse shard field, as each mode had its own property before. The
  parser now fills that field only and leaves the other one 0, which is what every released version
  stores, so `Column.checkSchemaChangeAllowed` compares the two fields again as master does. The BE
  and SHOW CREATE TABLE already read the field of the mode.
- MODIFY COLUMN refused any change of `variant_subcolumn_extraction_min_rows`, while apache#61100 lets it
  change `variant_doc_materialization_min_rows`. It can change again, with and without doc mode,
  except on a column without doc mode created before this change (a V1 column): its BE ignores the
  threshold, because its count-0 compaction branch cannot extract paths that only the sparse column
  holds, so a value stored there would do nothing. test_variant_modify_doc_materialization_min_rows
  has its steps from master again (an ALTER lowers the threshold and the next compaction extracts the
  old paths), and test_variant_tuning_properties raises the threshold with an ALTER again. The BE
  comment and the UT description of a compaction below the threshold name the ALTER again; the BE
  code does not change.

### Release note

MODIFY COLUMN can change `variant_subcolumn_extraction_min_rows`, but not on a column without doc
mode created before the upgrade.

### Check List (For Author)

- Test: Unit Test, Regression test
    - FE UT: the 11 classes that the VARIANT properties touch (PropertyAnalyzerTest,
      VariantSerializeTypeDdlTest, TypeTest, ColumnTest, SchemaChangeHandlerTest, DataTypeTest,
      NereidsParserTest, VariantLoadParseInjectionTest, CreateTableCommandTest,
      ShowCreateTableCommandTest, SessionVariablesTest), 286 tests, 0 failures.
    - Regression: test_variant_tuning_properties and test_variant_modify_doc_materialization_min_rows,
      regenerated by running them (the latter extracts the same subcolumns as on master); every
      enabled suite of variant_p0 (187), where only outfile_csv_variant_doc_value fails because it
      needs S3 credentials; six suites outside variant_p0 that create VARIANT tables;
      check_before_quit, which recreates every table from its SHOW CREATE TABLE and compares.
- Behavior changed: Yes. MODIFY COLUMN can change the row threshold again.
- Does this need documentation: Yes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 9, 2026
… typed paths to sparse internal

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

- Doc mode columns created before this change (V1, doc mode shipped in 4.1.0 and forced their count
  to 0) keep their layout: they still extract every path once a file reaches the row threshold. The
  load shredder (finish_doc) and the compaction schema (choose_doc_compaction_subpaths) pass the
  column's serialize type instead of V2, so select_variant_paths and get_subpaths give a count of 0
  its V1 meaning, and every doc bucket writer extracts all paths of its bucket. The catalog and
  Nereids VariantType.equals compare the serialize type of a doc column with a count of 0 again. A
  new doc column keeps the new meaning: 0 extracts no dynamic path.
- `variant_enable_typed_paths_to_sparse` is internal now. The property only takes `false`, which
  released versions print in SHOW CREATE TABLE for every column without doc mode, so that DDL still
  runs; `true` is refused ("variant_enable_typed_paths_to_sparse = true is not supported"). A new
  column takes the value from the session variable `default_variant_enable_typed_paths_to_sparse`,
  which only tests set, and a column of an older version keeps its value. SHOW CREATE TABLE and
  DESC do not print it. A restated definition cannot state it, so MODIFY COLUMN no longer compares
  it, and SchemaChangeHandler carries the stored value over as it does the serialize type. The
  regression suites that created columns with `true` set the session variable around those CREATE
  TABLE statements instead.
- Without doc mode only a V2 column has a row threshold: older versions cleared it and MODIFY COLUMN
  refuses one on a V1 column. The load shredder (finish_ordinary) and the compaction schema
  (choose_sparse_compaction_subpaths) therefore compare the rows with it without testing the
  serialize type. The BE unit tests no longer give a V1 column without doc mode a threshold, a state
  that cannot occur.
- Doc mode never moves typed paths to the sparse encoding, so finish_doc asserts that instead of
  testing the flag.
- The doc statistics loop is inlined into aggregate_variant_extended_info, its only caller, and
  choose_compaction_subpaths is renamed choose_sparse_compaction_subpaths, the counterpart of
  choose_doc_compaction_subpaths.

### Release note

Doc mode columns created by 4.1 keep extracting every path. `variant_enable_typed_paths_to_sparse`
cannot be set to `true` any more and is no longer printed by SHOW CREATE TABLE; columns that have it
keep it.

### Check List (For Author)

- Test: Unit Test, Regression test
    - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1018 run, 1009 passed,
      9 skipped, 0 failed. The 48 cases fewer than before are the V1 columns without doc mode that
      had a threshold. Two VariantShredderTest cases gave the default V1 ordinary layout a threshold
      and now set it for the doc layout only.
    - FE UT: the 11 classes that the VARIANT properties touch, 286 tests, and VariantSerializeTypeDdlTest
      with the new testModifyColumnKeepsTypedPathsToSparse, 4 tests; 0 failures.
    - Regression: the 11 suites whose output printed the property, regenerated by running them; each
      .out only loses the property, and test_all_prdefine_type_to_sparse keeps its subcolumns, so
      the session variable reproduces the layout. test_paimon_catalog_variant.out needs a Paimon
      environment and was edited by hand to the type text test_adbc_type_semantic now prints. Every
      enabled suite of variant_p0 (187), where only outfile_csv_variant_doc_value fails because it
      needs S3 credentials; 20 suites outside variant_p0 that use the property or create VARIANT
      tables; check_before_quit, which recreates every table from its SHOW CREATE TABLE and
      compares. Not run: the two variant_p1 suites and test_index_disk_usage_tvf, which only write
      `false`.
    - clang-format, clang-tidy on the changed lines, checkstyle and check-build-hygiene.sh are clean.
      Inlining the doc statistics loop gives aggregate_variant_extended_info a cognitive complexity
      of 53, so it carries a NOLINT with the reason.
- Behavior changed: Yes. See the release note.
- Does this need documentation: Yes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 9, 2026
…f the shard count

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

The earlier commits named the shard count of the sparse encoding `variant_sparse_shard_count` and
kept `variant_sparse_hash_shard_count`, the name that released versions use without doc mode, as an
alias. The property keeps its released name instead: `variant_sparse_hash_shard_count` sets the
shard count in both modes (the field of the column's mode, as before), so the DDL of released
versions needs no alias and the alias code goes (renameDeprecatedVariantProperties and its "names of
the same property" error). The doc mode name `variant_doc_hash_shard_count` stays refused. The
session variable is `default_variant_sparse_hash_shard_count` again, as on master, and also sets the
shard count of doc mode. SHOW CREATE TABLE and DESC print `variant_sparse_hash_shard_count`.

Two smaller changes in the doc mode compaction code:

- VariantShredderOptions takes the dynamic paths a compaction chose as a `const PathSet*` instead of
  a std::function that tests the same set; variant_shredder.h includes tablet_schema.h for PathSet.
- Shorter comments on choose_doc_compaction_subpaths, choose_sparse_compaction_subpaths and the
  load branch of finish_doc.

### Release note

The shard count of the sparse encoding keeps its name `variant_sparse_hash_shard_count` and now also
applies with doc mode.

### Check List (For Author)

- Test: Unit Test, Regression test
    - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1018 run, 1009 passed,
      9 skipped, 0 failed.
    - FE UT: the 11 classes that the VARIANT properties touch, 287 tests, 0 failures.
    - Regression: the 12 suites whose output prints the shard count, regenerated by running them;
      each .out only changes the printed name, except test_variant_tuning_properties, which drops
      the table it created with the old alias. test_paimon_catalog_variant.out needs a Paimon
      environment and was edited by hand to the type text test_adbc_type_semantic now prints. Every
      enabled suite of variant_p0 (187), where only outfile_csv_variant_doc_value fails because it
      needs S3 credentials; 20 suites outside variant_p0 that create VARIANT tables or set the
      session variables; check_before_quit, which recreates every table from its SHOW CREATE TABLE
      and compares.
    - clang-format, clang-tidy on the changed lines, checkstyle and check-build-hygiene.sh are clean.
- Behavior changed: Yes. See the release note.
- Does this need documentation: Yes, the documentation uses `variant_sparse_hash_shard_count`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 10, 2026
…t extract every path

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

The serialize type of a VARIANT column (V1 for columns created before the upgrade, V2 for new ones)
is meant to decide one thing: what a subcolumn count of 0 means, every path under V1 and no dynamic
path under V2. MODIFY COLUMN also used it to refuse `variant_subcolumn_extraction_min_rows` on every
V1 column without doc mode, so a table of an older version, whose count is 2048 by default, could
not take a row threshold after the upgrade.

Only a count of 0 needs that refusal. A segment below the threshold leaves its dynamic paths in the
sparse column. A compaction of a V1 column with a count of 0 then chooses every path to extract,
which drops them from the sparse column, but only builds the subcolumns that its inputs extracted, so
the paths that only the sparse column held would be lost. With a count above 0, V1 and V2 behave the
same.

- Column.checkSchemaChangeAllowed refuses a non-zero threshold only on a V1 column without doc mode
  whose count is 0, with a message that says so.
- A load below the threshold uses the V2 meaning of a count of 0 again (extract no dynamic path),
  so a V1 column with a count above 0 does not extract every path there.
- Step 4 of the extended compaction schema ends with an explicit branch for a V2 column with a count
  of 0 and no typed path, which asserts that the compaction chose no path to extract: such a path
  would be dropped from the sparse column with no subcolumn to hold it.

### Release note

MODIFY COLUMN can set `variant_subcolumn_extraction_min_rows` on a VARIANT column without doc mode
that was created before the upgrade, unless its `variant_max_subcolumns_count` is 0.

### Check List (For Author)

- Test: Unit Test, Manual test
    - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1050 run, 1041 passed,
      9 skipped, 0 failed. The compaction matrix runs V1 columns with a count above 0 with
      thresholds again, expecting the paths of V2.
    - FE UT: ColumnTest (20 tests) and VariantSerializeTypeDdlTest (4 tests), 0 failures; ColumnTest
      covers a V1 column with a count of 2048 taking a threshold and one with a count of 0 refusing it.
    - Manual test on a local cluster: a new column with a count of 0 and no typed path, with and
      without a row threshold, through two loads and a full compaction: no subcolumn extracted, every
      value read back, no assertion. No regression suite was run for this commit.
    - clang-format, clang-tidy on the changed lines, checkstyle and check-build-hygiene.sh are clean.
- Behavior changed: Yes. See the release note.
- Does this need documentation: No.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 10, 2026
…to sparse

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

The row threshold `variant_subcolumn_extraction_min_rows` now applies with and without doc mode,
but fuzzy mode (`use_fuzzy_session_variable`, on in the p0, p1, nonConcurrent and external
pipelines) still randomized `default_variant_subcolumn_extraction_min_rows` with doc mode only.
`default_variant_enable_typed_paths_to_sparse` is an internal switch for tests now, yet no fuzzy run
ever set it to true: SessionVariable.initFuzzyModeVariables only forced it to false with doc mode.

- initFuzzyModeVariables randomizes the row threshold in both modes, and
  `default_variant_enable_typed_paths_to_sparse` without doc mode. Doc mode keeps it false; the
  parser forces it to false there anyway.
- The parser's "only support for ..." error no longer lists `variant_enable_typed_paths_to_sparse`,
  which is internal now. The property still takes `false` from the DDL of released versions.
- Suites that pin every other fuzzed VARIANT variable now pin these two where their output depends
  on which paths a data file extracts. They were found by running every suite that defines a VARIANT
  column (variant_p0 and 106 others) with the global defaults set to a threshold of 19 and typed
  paths to sparse, then with each one alone, against a run with the defaults:
  - 18 suites pin the threshold to 0: the type text they print shows it, they check MATCH, SEARCH or
    scores with `enable_match_without_inverted_index = false` or need an index on every path, or a
    debug point checks the layout.
  - 3 suites pin typed paths to sparse to false: test_variant_v2_snii_index (count 0, so typed paths
    to sparse leave no indexed path) and both test_array_contains_with_inverted_index (a typed array
    path left sparse keeps an empty array that its subcolumn drops).

### Release note

None

### Check List (For Author)

- Test: Regression test, Unit Test
    - With the global defaults set to a threshold of 19 and typed paths to sparse, before the pins:
      variant_p0 187 suites, 17 failed; 106 other suites, 12 failed. After the pins, the 21 pinned
      suites pass with those settings and with the defaults.
    - Suites that fail with the defaults too, for environment reasons: outfile_csv_variant_doc_value
      (S3 credentials), test_get_tables_schema (arrow flight), test_default_expr and test_json_load
      (time zone), nereids_insert_random, test_bitmap_filter and test_bitmap_filter_nereids
      (databases of other suites), test_ivm_agg_expr_over_agg_1 (enable_feature_changelog).
    - FE UT: NereidsParserTest, 95 tests, 0 failures.
    - checkstyle is clean.
- Behavior changed: No (fuzzy tests and an error message only).
- Does this need documentation: No.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 10, 2026
…t extract every path

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

The serialize type of a VARIANT column (V1 for columns created before the upgrade, V2 for new ones)
is meant to decide one thing: what a subcolumn count of 0 means, every path under V1 and no dynamic
path under V2. MODIFY COLUMN also used it to refuse `variant_subcolumn_extraction_min_rows` on every
V1 column without doc mode, so a table of an older version, whose count is 2048 by default, could
not take a row threshold after the upgrade.

Only a count of 0 needs that refusal. A segment below the threshold leaves its dynamic paths in the
sparse column. A compaction of a V1 column with a count of 0 then chooses every path to extract,
which drops them from the sparse column, but only builds the subcolumns that its inputs extracted, so
the paths that only the sparse column held would be lost. With a count above 0, V1 and V2 behave the
same.

- Column.checkSchemaChangeAllowed refuses a non-zero threshold only on a V1 column without doc mode
  whose count is 0, with a message that says so.
- select_variant_paths takes the rows of the data file and the shredder options and applies the
  row threshold itself: below it no dynamic path is extracted, whatever the serialize type. The
  load used to pass a count of 0 for that, which under V1 extracts every path, so a V1 column with
  a count above 0 would have extracted every path below the threshold. Both callers, the load
  without doc mode and the load with doc mode, pass the options; the selection is unchanged
  otherwise.
- Step 4 of the extended compaction schema ends with an explicit branch for a V2 column with a count
  of 0 and no typed path, which asserts that the compaction chose no path to extract: such a path
  would be dropped from the sparse column with no subcolumn to hold it.

### Release note

MODIFY COLUMN can set `variant_subcolumn_extraction_min_rows` on a VARIANT column without doc mode
that was created before the upgrade, unless its `variant_max_subcolumns_count` is 0.

### Check List (For Author)

- Test: Unit Test, Manual test
    - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1050 run, 1041 passed,
      9 skipped, 0 failed. The
      compaction matrix runs V1 columns with a count above 0 with thresholds again, expecting the
      paths of V2. The selection test covers a data file below and at the threshold under V1 and V2.
    - FE UT: ColumnTest (20 tests) and VariantSerializeTypeDdlTest (4 tests), 0 failures; ColumnTest
      covers a V1 column with a count of 2048 taking a threshold and one with a count of 0 refusing it.
    - Manual test on a local cluster: a new column with a count of 0 and no typed path, with and
      without a row threshold, through two loads and a full compaction: no subcolumn extracted, every
      value read back, no assertion. Again after the selection change, with a column of count 2 and
      a threshold of 3 (a 2-row load extracts nothing, a 4-row load its top 2, the compaction of both
      the top 2 of all rows) and a doc mode column of count 1. No regression suite was run for this
      commit.
    - clang-format, clang-tidy on the changed lines, checkstyle and check-build-hygiene.sh are clean.
- Behavior changed: Yes. See the release note.
- Does this need documentation: No.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 10, 2026
…to sparse

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

The row threshold `variant_subcolumn_extraction_min_rows` now applies with and without doc mode,
but fuzzy mode (`use_fuzzy_session_variable`, on in the p0, p1, nonConcurrent and external
pipelines) still randomized `default_variant_subcolumn_extraction_min_rows` with doc mode only.
`default_variant_enable_typed_paths_to_sparse` is an internal switch for tests now, yet no fuzzy run
ever set it to true: SessionVariable.initFuzzyModeVariables only forced it to false with doc mode.

- initFuzzyModeVariables randomizes the row threshold in both modes, and
  `default_variant_enable_typed_paths_to_sparse` without doc mode. Doc mode keeps it false; the
  parser forces it to false there anyway.
- The parser's "only support for ..." error no longer lists `variant_enable_typed_paths_to_sparse`,
  which is internal now. The property still takes `false` from the DDL of released versions.
- Suites that pin every other fuzzed VARIANT variable now pin these two where their output depends
  on which paths a data file extracts. They were found by running every suite that defines a VARIANT
  column (variant_p0 and 106 others) with the global defaults set to a threshold of 19 and typed
  paths to sparse, then with each one alone, against a run with the defaults:
  - 18 suites pin the threshold to 0: the type text they print shows it, they check MATCH, SEARCH or
    scores with `enable_match_without_inverted_index = false` or need an index on every path, or a
    debug point checks the layout.
  - 3 suites pin typed paths to sparse to false: test_variant_v2_snii_index (count 0, so typed paths
    to sparse leave no indexed path) and both test_array_contains_with_inverted_index (a typed array
    path left sparse keeps an empty array that its subcolumn drops).

### Release note

None

### Check List (For Author)

- Test: Regression test, Unit Test
    - With the global defaults set to a threshold of 19 and typed paths to sparse, before the pins:
      variant_p0 187 suites, 17 failed; 106 other suites, 12 failed. After the pins, the 21 pinned
      suites pass with those settings and with the defaults.
    - Suites that fail with the defaults too, for environment reasons: outfile_csv_variant_doc_value
      (S3 credentials), test_get_tables_schema (arrow flight), test_default_expr and test_json_load
      (time zone), nereids_insert_random, test_bitmap_filter and test_bitmap_filter_nereids
      (databases of other suites), test_ivm_agg_expr_over_agg_1 (enable_feature_changelog).
    - FE UT: NereidsParserTest, 95 tests, 0 failures.
    - checkstyle is clean.
- Behavior changed: No (fuzzy tests and an error message only).
- Does this need documentation: No.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 10, 2026
…nt and fail reads of a mismatched layout

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

A review of this PR found two problems and a stale comment.

- CTAS makes every VARIANT column it creates V2, but it kept the count of the query's type. The type of
  a VARIANT expression such as `parse_to_variant` (the default VARIANT type of the planner) has a count
  of 0 under V1, which extracts every path, so `CREATE TABLE t AS SELECT parse_to_variant(s) AS v ...`
  created a column with a count of 0 under V2, which extracts no dynamic path: every path went to the
  sparse column, while 4.1 extracts them all. The same happened when CTAS copies a column of an older
  version with a count of 0, or a VARIANT column of a file table function. Such a type now becomes V2
  with the session's default count (`default_variant_max_subcolumns_count`), as a column declared
  without the count would; a count of 0 that a new column states is kept, and so is the count of a
  nested group column.
- The reader checks that a V2 column with a count of 0 holds no dynamic subcolumn. A BE of an older
  version, which does not know the serialize type, writes such a column with every path extracted, for
  example when an FE is upgraded before every BE. The check was a DORIS_CHECK, so opening such a segment
  crashed the BE, and compaction, which retries the tablet, crashed it again. The check now returns an
  InternalError that names the segment, so the query or the compaction fails instead. It stays an
  error: compaction of such a column folds none of those subcolumns into the sparse column and would
  drop them.
- A comment in LogicalPlanBuilder still said that the property check reads deprecated names; that code
  went away with the rename.

### Release note

None

### Check List (For Author)

- Test: Unit Test, Regression test, Manual test
    - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1050 run, 1041 passed,
      9 skipped, 0 failed. reader_rejects_dynamic_subcolumn_of_v2_count_zero_column now expects the
      InternalError with the path and the segment file instead of a crash.
    - FE UT: VariantSerializeTypeDdlTest, 6 tests, 0 failures; it covers CTAS of an old column with a
      count of 0, of a VARIANT expression, and of a new column with a count of 0.
    - Regression: test_variant_count_zero_keeps_paths_sparse covers CTAS of `parse_to_variant(...)`; its
      .out was regenerated by the framework and only gains the two new tags. Every suite of variant_p0
      (187), where only outfile_csv_variant_doc_value fails because it needs S3 credentials, and 6 suites
      outside it that run CTAS or views over VARIANT columns.
    - Manual test: the reproduction (CTAS of `parse_to_variant('{"a": 1, "b": 2}')`) gave a count of 0
      and no subcolumn before, a count of 2048 and the subcolumns v.a and v.b after.
    - clang-format, clang-tidy on the changed lines, checkstyle and check-build-hygiene.sh are clean.
- Behavior changed: Yes. CTAS of a VARIANT expression, or of an older column with a count of 0, keeps
  extracting the paths, as in 4.1.
- Does this need documentation: No.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 10, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

The previous commit gave a column that CTAS creates from a type with a count of 0 under V1 (a column of
an older version, the type of a VARIANT expression such as `parse_to_variant`, or a column of a file
table function) the session's default subcolumn count, so that it kept extracting every path as in
4.1. A column that CTAS creates is a new column, and on a new column a count of 0 means that no dynamic
path is extracted. CTAS keeps the count of the query's type again, as before that commit: such a
column is V2 with a count of 0 and keeps its dynamic paths in the sparse column. Casting the
expression to `VARIANT` in the query gives the column the session's default count instead, because
the target type of the cast is parsed like a declared column.

The reader check and the comment fix of the previous commit stay.

### Release note

CTAS of a VARIANT expression such as `parse_to_variant(...)`, or of a VARIANT column of an older
version with `variant_max_subcolumns_count = 0`, creates a column that extracts no dynamic path,
while 4.1 extracts every path; cast the expression to `VARIANT` to get the default subcolumn count.

### Check List (For Author)

- Test: Unit Test, Regression test, Manual test
    - FE UT: VariantSerializeTypeDdlTest, 6 tests, 0 failures; CTAS of an older column with a count
      of 0, of a VARIANT expression and of a new column with a count of 0 all give V2 with a count of 0.
    - Regression: test_variant_count_zero_keeps_paths_sparse, its .out regenerated by the framework (the
      CTAS of `parse_to_variant(...)` prints a count of 0 and no subcolumn, and reads back the same
      values); test_variant_tuning_properties and the 6 suites outside variant_p0 that run CTAS or
      views over VARIANT columns. 8 suites, 0 failed.
    - Manual test: CTAS of `parse_to_variant('{"a": 1, "b": 2}')` gives a count of 0 and no subcolumn;
      with `CAST(... AS VARIANT)` a count of 2048 and the subcolumns v.a and v.b.
    - checkstyle is clean.
- Behavior changed: Yes. See the release note.
- Does this need documentation: Yes, the CTAS rule goes with the VARIANT documentation.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants