Repository navigation
[feature](variant) add variant doc mode - #59183
Merged
Merged
Conversation
csun5285
requested review from
dataroaring,
gavinchou,
yiguolei and
zclllyybb
as code owners
December 19, 2025 05:28
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
csun5285
force-pushed
the
test_variant
branch
from
December 19, 2025 07:23
b307644 to
7d6f45a
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
csun5285
force-pushed
the
test_variant
branch
from
December 22, 2025 09:05
7d6f45a to
a1b24a3
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run p0 10 |
TPC-H: Total hot run time: 36375 ms |
TPC-DS: Total hot run time: 180018 ms |
ClickBench: Total hot run time: 27.17 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
csun5285
force-pushed
the
test_variant
branch
from
December 23, 2025 04:04
a1b24a3 to
943db93
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 35042 ms |
TPC-DS: Total hot run time: 178720 ms |
ClickBench: Total hot run time: 28.28 s |
eldenmoon
reviewed
Dec 23, 2025
Contributor
Author
|
run p0 10 |
csun5285
force-pushed
the
test_variant
branch
from
December 26, 2025 11:02
943db93 to
3cb0f62
Compare
fix ut fix fix case fix rebase case fix
csun5285
force-pushed
the
test_variant
branch
from
January 21, 2026 03:39
623100a to
3009664
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 31732 ms |
TPC-DS: Total hot run time: 174411 ms |
ClickBench: Total hot run time: 27.99 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
FE Regression Coverage ReportIncrement line coverage |
gavinchou
approved these changes
Jan 21, 2026
zclllyybb
approved these changes
Jan 21, 2026
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Mar 12, 2026
**DOC encoding**: materialize all subcolumns, and additionally keep the original JSON as a “stored field” so queries can quickly return the entire JSON document. doc: apache/doris-website#3253
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Mar 14, 2026
**DOC encoding**: materialize all subcolumns, and additionally keep the original JSON as a “stored field” so queries can quickly return the entire JSON document. doc: apache/doris-website#3253
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Mar 16, 2026
**DOC encoding**: materialize all subcolumns, and additionally keep the original JSON as a “stored field” so queries can quickly return the entire JSON document. doc: apache/doris-website#3253
eldenmoon
added a commit
that referenced
this pull request
Mar 16, 2026
yiguolei
pushed a commit
that referenced
this pull request
Jun 17, 2026
…its queries (#64533) ### What problem does this PR solve? In `regression_test_variant_doc_mode_p2`, `load.groovy` (which creates and loads the `github_events` table) is tagged `nonConcurrent,p2`, while the 76 `sql/*.sql` query cases in the same directory fall back to the path-derived default group `variant_doc_mode_p2` (no `nonConcurrent`). `nonConcurrent` is a **separate pipeline**, so the load step and the queries get routed to different runs/clusters. In the P2 run the load never executes, and every query fails instantly with: ``` Table [github_events] does not exist in database [regression_test_variant_doc_mode_p2] ``` This produced **66 cascade failures in a single branch-4.1 P2 build** (e.g. build #35 / 198091), all from the same root cause. ### Root cause The `nonConcurrent` tag was applied only to `load.groovy`, not to the dependent queries. The correct nonConcurrent sibling (`variant_github_events_nonConcurrent_p2`) encodes it in the **directory name** so the whole suite (load + queries) routes together; `variant_doc_mode_p2` instead split the data-prep away from its queries. The load only sets **session variables** and does no global-state mutation (no `set global` / FE-BE config / debug points), so it does not actually need `nonConcurrent` isolation. ### Fix Drop `nonConcurrent` so `load.groovy` shares the `p2` group/pipeline with its queries. `load.groovy` sorts before `sql/`, runs first, and `github_events` is present when the queries execute. ```diff -suite("test_doc_value_p2", "nonConcurrent,p2"){ +suite("test_doc_value_p2", "p2"){ ``` > Note: the identical structure exists on master (`#59183`); a follow-up there is advisable. ### Release note None 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Nov 12, 2026
…and make nested group DDL replayable ### What problem does this PR solve? Issue Number: None Related PR: apache#59183 Problem Summary: Follow-up to the first commit of this PR, which makes CREATE TABLE LIKE copy the VARIANT types of the source and the `default_variant_*` session variables test-only. 1. The parser fills the VARIANT properties a DDL leaves out from the `default_variant_*` session variables, and doc mode or nested group then clears the subcolumn properties. So a default could switch on a mode next to written properties that exclude it, and the written properties were lost: ```sql SET default_variant_enable_doc_mode = true; CREATE TABLE t (id INT, v VARIANT<PROPERTIES("variant_max_subcolumns_count" = "3")>) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES("replication_num" = "1"); SHOW CREATE TABLE t; -- before: v variant<PROPERTIES ("variant_enable_doc_mode" = "true",...)>, the count 3 is gone, although -- writing variant_enable_doc_mode next to the count fails with "variant_max_subcolumns_count and -- variant_enable_doc_mode cannot be set together" -- after: v variant<PROPERTIES ("variant_max_subcolumns_count" = "3",...)> ``` `default_variant_enable_nested_group = true` likewise turned a written count above 0 or a written doc mode into a nested group column. A default now applies only where the DDL could write it: written subcolumn properties keep doc mode off, a written doc mode or a written count above 0 keeps nested group off. The doc mode suites that wrote a subcolumn count and took doc mode from the session now write `variant_enable_doc_mode`, so they still test doc mode columns. 2. SHOW CREATE TABLE printed a nested group column (master only, added by apache#64680) with the subcolumn properties nested group clears, including `variant_max_sparse_column_statistics_size = 0`, which does not parse. CREATE TABLE LIKE of such a table failed with "variant_max_sparse_column_statistics_size must between 1 and 50000", and the printed DDL could not be replayed. A nested group VARIANT type now prints only `variant_enable_nested_group`, as a doc mode type prints only the doc mode properties. 3. PropertyAnalyzerTest called the statistics size checker of the session variable with debug points off, which the first commit made fail. ### Release note None. Nested group VARIANT columns are not in a release yet, and the first commit already makes the `default_variant_*` session variables test-only. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - FE UT: NereidsParserTest (session defaults next to written properties), TypeTest (nested group text), CreateTableLikeTest (adds a count 0 column and a nested group column), PropertyAnalyzerTest - Regression test: variant_p0/test_variant_session_defaults_written_properties (new), variant_p0/test_variant_create_table_like_session_defaults (adds the two columns), and the doc mode suites changed here - Manual test: the DDL above on a local cluster - Behavior changed: Yes. A session default no longer overrides written VARIANT properties; SHOW CREATE TABLE of a nested group VARIANT column prints only variant_enable_nested_group. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Nov 12, 2026
… fix CREATE TABLE LIKE of VARIANT ### What problem does this PR solve? Issue Number: None Related PR: apache#59183 Problem Summary: CREATE TABLE LIKE prints the source table as DDL text (the SHOW CREATE TABLE text) and parses it again. The text of a VARIANT column omits `variant_enable_doc_mode` and `variant_enable_nested_group` when they are false, and the parser fills every property missing from a DDL from the `default_variant_*` session variables. So the copy depended on the session that ran the statement: ```sql CREATE TABLE src (id INT, v VARIANT<'a.b':INT, PROPERTIES("variant_max_subcolumns_count"="3")>) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES("replication_num"="1"); SET default_variant_enable_doc_mode = true; CREATE TABLE dst LIKE src; SHOW CREATE TABLE dst; -- before: v variant<MATCH_NAME 'a.b':int,PROPERTIES ("variant_enable_doc_mode" = "true", -- "variant_doc_materialization_min_rows" = "0","variant_doc_hash_shard_count" = "64")> -- (doc mode, the subcolumn count 3 is gone) -- after: the SET fails; with the default session v is as in src ``` `default_variant_enable_nested_group = true` turned an ordinary or a doc mode column into a nested group column the same way. The fuzzy regression runs set `default_variant_enable_doc_mode` at random per connection, so CREATE TABLE LIKE of a VARIANT table was random there too. Separately, SHOW CREATE TABLE printed a nested group VARIANT column (master only, apache#64680) with the subcolumn properties nested group clears, including `variant_max_sparse_column_statistics_size = 0`, which does not parse: CREATE TABLE LIKE of such a table always failed with "variant_max_sparse_column_statistics_size must between 1 and 50000". The `default_variant_*` session variables exist for the fuzzy tests only. Fix: 1. The parser reads the `default_variant_*` session variables only when the FE runs with `enable_debug_points = true`, which every regression pipeline and docker test cluster sets. Otherwise the properties a DDL leaves out take the built-in values. It does not use the global values either: a value set with SET GLOBAL on an older version still loads from an image. So a DDL text, such as the one CREATE TABLE LIKE prints and parses, creates the same column in every session. 2. The variables are invisible in SHOW VARIABLES, and SET / SET GLOBAL / SET_VAR of them fails unless debug points are enabled. Users state VARIANT properties in the column definition. 3. A nested group VARIANT type prints only `variant_enable_nested_group`, as a doc mode type prints only the doc mode properties. With debug points enabled, CREATE TABLE LIKE still follows the session; a test that copies a VARIANT table sets the variables first, as the existing suites do. ### Release note The `default_variant_*` session variables (`default_variant_max_subcolumns_count`, `default_variant_enable_typed_paths_to_sparse`, `default_variant_max_sparse_column_statistics_size`, `default_variant_sparse_hash_shard_count`, `default_variant_enable_doc_mode`, `default_variant_doc_materialization_min_rows`, `default_variant_doc_hash_shard_count`, `default_variant_enable_nested_group`) can no longer be set and are not shown by SHOW VARIABLES; set the properties in the VARIANT column definition instead. A value set with SET GLOBAL before the upgrade no longer takes effect. CREATE TABLE LIKE copies the VARIANT properties of the source table regardless of the session. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - FE UT: NereidsParserTest (session values used only with debug points), CreateTableLikeTest (LIKE under doc mode and nested group session defaults), VariableMgrTest (SET fails, hidden, forwarded values, replay), TypeTest (nested group text), PropertyAnalyzerTest - Regression test: variant_p0/test_variant_create_table_like - Manual test: the DDL above on a local cluster; with enable_debug_points = false, SET, SET GLOBAL and SET_VAR of default_variant_* fail, SHOW VARIABLES does not list them, and with use_fuzzy_session_variable = true the fuzzed session values do not reach new VARIANT columns - Behavior changed: Yes. SET of the default_variant_* session variables fails and their values are ignored unless enable_debug_points is on; SHOW CREATE TABLE of a nested group VARIANT column prints only variant_enable_nested_group. - Does this need documentation: Yes, VARIANT.md lists the session variables (doc PR to follow) Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Nov 12, 2026
… fix CREATE TABLE LIKE of VARIANT ### What problem does this PR solve? Issue Number: None Related PR: apache#59183 Problem Summary: CREATE TABLE LIKE prints the source table as DDL text (the SHOW CREATE TABLE text) and parses it again. The text of a VARIANT column omits `variant_enable_doc_mode` and `variant_enable_nested_group` when they are false, and the parser fills every property missing from a DDL from the `default_variant_*` session variables. So the copy depended on the session that ran the statement: ```sql CREATE TABLE src (id INT, v VARIANT<'a.b':INT, PROPERTIES("variant_max_subcolumns_count"="3")>) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES("replication_num"="1"); SET default_variant_enable_doc_mode = true; CREATE TABLE dst LIKE src; SHOW CREATE TABLE dst; -- before: v variant<MATCH_NAME 'a.b':int,PROPERTIES ("variant_enable_doc_mode" = "true", -- "variant_doc_materialization_min_rows" = "0","variant_doc_hash_shard_count" = "64")> -- (doc mode, the subcolumn count 3 is gone) -- after: the SET fails; with the default session v is as in src ``` `default_variant_enable_nested_group = true` turned an ordinary or a doc mode column into a nested group column the same way. The fuzzy regression runs set `default_variant_enable_doc_mode` at random per connection, so CREATE TABLE LIKE of a VARIANT table was random there too. Separately, SHOW CREATE TABLE printed a nested group VARIANT column (master only, apache#64680) with the subcolumn properties nested group clears, including `variant_max_sparse_column_statistics_size = 0`, which does not parse: CREATE TABLE LIKE of such a table always failed with "variant_max_sparse_column_statistics_size must between 1 and 50000". The `default_variant_*` session variables exist for the fuzzy tests only. Fix: 1. The parser reads the `default_variant_*` session variables only when the FE runs with `enable_debug_points = true`, which every regression pipeline and docker test cluster sets. Otherwise, and without a session, the properties a DDL leaves out take the built-in values (a parse without a session used its own constants before, such as subcolumn count 0 and nested group on). It does not use the global values either: a value set with SET GLOBAL on an older version still loads from an image. So a DDL text, such as the one CREATE TABLE LIKE prints and parses, creates the same column in every session. 2. The variables are invisible in SHOW VARIABLES, and SET / SET GLOBAL / SET_VAR of them fails unless debug points are enabled. Users state VARIANT properties in the column definition. 3. A nested group VARIANT type prints only `variant_enable_nested_group`, as a doc mode type prints only the doc mode properties. With debug points enabled, CREATE TABLE LIKE still follows the session; a test that copies a VARIANT table sets the variables first, as the existing suites do. ### Release note The `default_variant_*` session variables (`default_variant_max_subcolumns_count`, `default_variant_enable_typed_paths_to_sparse`, `default_variant_max_sparse_column_statistics_size`, `default_variant_sparse_hash_shard_count`, `default_variant_enable_doc_mode`, `default_variant_doc_materialization_min_rows`, `default_variant_doc_hash_shard_count`, `default_variant_enable_nested_group`) can no longer be set and are not shown by SHOW VARIABLES; set the properties in the VARIANT column definition instead. A value set with SET GLOBAL before the upgrade no longer takes effect. CREATE TABLE LIKE copies the VARIANT properties of the source table regardless of the session. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - FE UT: NereidsParserTest (session values used only with debug points), CreateTableLikeTest (LIKE under doc mode and nested group session defaults), VariableMgrTest (SET fails, hidden, forwarded values, replay), TypeTest (nested group text, defaults without a session), PropertyAnalyzerTest - Regression test: variant_p0/test_variant_create_table_like - Manual test: the DDL above on a local cluster; with enable_debug_points = false, SET, SET GLOBAL and SET_VAR of default_variant_* fail, SHOW VARIABLES does not list them, and with use_fuzzy_session_variable = true the fuzzed session values do not reach new VARIANT columns - Behavior changed: Yes. SET of the default_variant_* session variables fails and their values are ignored unless enable_debug_points is on; SHOW CREATE TABLE of a nested group VARIANT column prints only variant_enable_nested_group. - Does this need documentation: Yes, VARIANT.md lists the session variables (doc PR to follow) Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
DOC encoding: materialize all subcolumns, and additionally keep the original JSON as a “stored field” so queries can quickly return the entire JSON document.
doc: apache/doris-website#3253
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)