Skip to content

[refactor](be) Encapsulate cached constant columns - #66614

Closed
Mryange wants to merge 2 commits into
apache:masterfrom
Mryange:refactor-constant-column-wrapper
Closed

Mryange wants to merge 2 commits into
apache:masterfrom
Mryange:refactor-constant-column-wrapper

Conversation

@Mryange

@Mryange Mryange commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Cached constant expressions could expose either a ColumnConst or a single-row data column, forcing callers to inspect the physical column type. This change makes ColumnPtrWrapper store a private, single-row data column, unwraps ColumnConst without copying, and creates ColumnConst only when broadcasting the cached value. It also removes redundant constant evaluation from VMatchPredicate::open().

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange

Mryange commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: the production refactor is sound, but the test-contract migration is incomplete and normal BE-UT configurations cannot pass as submitted.

Critical checkpoint conclusions:

  • Goal and proof: Centralizing the cached single-row data column removes caller-side physical-shape checks and preserves production behavior, but the changed/new tests do not prove a passing cross-build contract; two blocking test failures are called out inline.
  • Scope and clarity: The change is conceptually focused. The accessor migration covers all repository consumers; the related VMatchPredicate removal was verified as a no-op for valid plans because MATCH retains a nonconstant slot child.
  • Concurrency: The wrapper is immutable after construction and existing shared pointers are populated during expression open. No new thread, lock, atomic, dependency-order, or deadlock behavior was introduced; valid scan constant-false paths still signal readiness as before.
  • Lifecycle, memory, and nullability: The wrapper owns the unwrapped ColumnPtr; FunctionContext clones share that ownership, so cached StringRef values remain live. Nullable and array/map/struct row-zero semantics and COW ownership are preserved, with no new untracked allocation or reference cycle.
  • Configuration and compatibility: No configuration, FE-BE variable, serialization, storage-format, or rolling-upgrade contract changes. The internal source API migration is complete across the repository.
  • Parallel paths and special checks: File, mock, OLAP, generic scan, scalar-function, collection-IN, constant rebroadcast, and ANN paths were traced. The one-row check is a valid production invariant, but its direct test/mock callers and build-mode-specific failure behavior were not fully migrated.
  • Test coverage and results: Existing scan-normalization cases now fail during setup, and the new unconditional death test fails under NDEBUG. The visible macOS BE check is compile-only with MAKE_TEST=OFF, so it does not cover either issue. No local builds or tests were run because this review runner prohibits builds.
  • Observability: No new runtime path requires logs, metrics, or tracing.
  • Transactions, persistence, and writes: Not applicable; the PR changes neither transaction state nor data-write/persistence behavior.
  • Performance: Constant unwrapping remains no-copy and is performed once at wrapper construction; no hot-path regression or redundant allocation was found.
  • Additional user focus: none was provided. The complete PR was reviewed without further narrowing.

Review completion: converged in Round 2 with two inline findings and no unresolved candidate.

} else {
_column_ptr = std::move(column);
}
DORIS_CHECK_EQ(_column_ptr->size(), 1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Update the existing two-row mock-constant tests

This invariant is reached immediately by the existing non-disabled ScanNormalizePredicate.test_eval_const_conjuncts3 and test_eval_const_conjuncts4: lines 130 and 154 pass two-row non-const columns through MockFnCall::set_const_expr_col(), which constructs this wrapper outside any death/exception assertion. The normal BE-UT CMake glob includes exec/*.cpp, so those tests now abort or throw before reaching their remaining code. (The green macOS check only builds BE with MAKE_TEST=OFF.) Please update/remove those malformed fixtures, or assert the new failure at construction, together with this invariant.


TEST(ColumnPtrWrapperTest, RejectsNonConstMultiRowColumn) {
auto column = ColumnHelper::create_column<DataTypeInt64>({7, 8});
EXPECT_DEATH(static_cast<void>(ColumnPtrWrapper(column)), "");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Assert the release-mode exception path here

This only behaves as a death assertion while DCHECKs are enabled. With NDEBUG, DORIS_CHECK_EQ uses the release comparison macro and doris_check_fail() throws doris::Exception; GoogleTest does not count leaving a death-test statement by exception as death. The existing be/test/common/check_test.cpp handles this contract by capturing the exception under NDEBUG and using EXPECT_DEATH only under DCHECK_IS_ON(). Please mirror that split (or use a build-mode-neutral helper), otherwise release BE-UT fails this new test.

@Mryange

Mryange commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28896 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 82d95715c456989a9987e5b383c96bf9ee31d509, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17780	4017	4017	4017
q2	2010	330	202	202
q3	10289	1432	819	819
q4	4684	470	347	347
q5	7523	844	548	548
q6	187	182	140	140
q7	758	800	598	598
q8	9325	1541	1681	1541
q9	5347	4139	4054	4054
q10	6718	1656	1367	1367
q11	522	354	320	320
q12	750	566	449	449
q13	18072	3284	2704	2704
q14	262	260	253	253
q15	q16	739	729	659	659
q17	952	1006	1025	1006
q18	6625	5605	5587	5587
q19	1159	1334	1055	1055
q20	795	704	600	600
q21	5504	2557	2327	2327
q22	424	353	303	303
Total cold run time: 100425 ms
Total hot run time: 28896 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4281	4209	4181	4181
q2	270	327	212	212
q3	4538	4935	4365	4365
q4	2150	2254	1404	1404
q5	4205	4104	4176	4104
q6	231	176	126	126
q7	1695	1572	1424	1424
q8	2624	2245	2140	2140
q9	7443	7488	7565	7488
q10	4344	4307	3976	3976
q11	563	391	373	373
q12	726	718	513	513
q13	3172	3477	2837	2837
q14	301	305	280	280
q15	q16	706	722	631	631
q17	1341	1273	1277	1273
q18	12227	11034	11876	11034
q19	1190	1146	1203	1146
q20	2215	2217	1928	1928
q21	5624	4829	4845	4829
q22	544	486	417	417
Total cold run time: 60390 ms
Total hot run time: 54681 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 158807 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 82d95715c456989a9987e5b383c96bf9ee31d509, data reload: false

query5	4313	587	444	444
query6	472	222	201	201
query7	4874	577	348	348
query8	321	160	146	146
query9	8755	3997	4037	3997
query10	475	351	293	293
query11	5834	2229	1970	1970
query12	147	100	93	93
query13	1285	597	460	460
query14	6099	4261	3949	3949
query14_1	3778	3783	3762	3762
query15	203	194	185	185
query16	987	519	475	475
query17	912	717	570	570
query18	2436	482	337	337
query19	206	187	152	152
query20	105	104	103	103
query21	234	160	135	135
query22	12984	13046	13518	13046
query23	16219	15469	14997	14997
query23_1	15176	14727	14796	14727
query24	7466	1687	1236	1236
query24_1	1233	1229	1233	1229
query25	531	425	341	341
query26	1295	356	201	201
query27	2635	633	383	383
query28	4534	2019	2024	2019
query29	1063	596	457	457
query30	338	258	220	220
query31	1161	1096	1035	1035
query32	101	60	56	56
query33	511	305	257	257
query34	1199	1147	615	615
query35	714	739	634	634
query36	767	741	703	703
query37	162	101	84	84
query38	1821	1776	1663	1663
query39	821	822	786	786
query39_1	777	778	826	778
query40	246	160	148	148
query41	62	60	59	59
query42	88	89	89	89
query43	313	316	283	283
query44	1456	758	764	758
query45	176	167	163	163
query46	1058	1180	680	680
query47	1549	1549	1487	1487
query48	409	422	286	286
query49	578	408	304	304
query50	1059	421	341	341
query51	10424	10418	10455	10418
query52	89	86	76	76
query53	281	287	200	200
query54	295	247	229	229
query55	75	73	67	67
query56	328	309	314	309
query57	1020	1008	951	951
query58	310	270	268	268
query59	1556	1640	1426	1426
query60	324	282	261	261
query61	177	173	173	173
query62	396	333	267	267
query63	240	195	197	195
query64	2994	1067	830	830
query65	3861	3803	3780	3780
query66	1822	461	351	351
query67	20513	20295	20129	20129
query68	3712	1500	1038	1038
query69	412	285	260	260
query70	840	752	810	752
query71	381	357	328	328
query72	3045	2714	2353	2353
query73	818	858	448	448
query74	4618	4484	4294	4294
query75	2364	2374	1965	1965
query76	2407	1159	748	748
query77	335	377	265	265
query78	11280	11174	10605	10605
query79	1198	1243	779	779
query80	594	537	453	453
query81	457	327	285	285
query82	285	174	139	139
query83	314	325	301	301
query84	306	173	129	129
query85	918	634	530	530
query86	275	234	223	223
query87	1986	1961	1843	1843
query88	3669	2775	2754	2754
query89	361	325	280	280
query90	2140	199	188	188
query91	200	188	165	165
query92	73	61	56	56
query93	1563	1557	947	947
query94	554	334	315	315
query95	770	499	580	499
query96	1047	850	333	333
query97	2462	2493	2384	2384
query98	197	182	184	182
query99	733	721	622	622
Total cold run time: 245122 ms
Total hot run time: 158807 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.84 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 82d95715c456989a9987e5b383c96bf9ee31d509, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.25	0.13	0.13
query4	1.61	0.14	0.13
query5	0.24	0.22	0.22
query6	1.16	0.81	0.81
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.38	0.32	0.30
query10	0.55	0.57	0.57
query11	0.18	0.14	0.13
query12	0.18	0.13	0.14
query13	0.44	0.46	0.46
query14	1.03	0.98	0.96
query15	0.61	0.58	0.59
query16	0.39	0.33	0.33
query17	1.12	1.08	1.09
query18	0.20	0.20	0.19
query19	2.05	1.91	1.91
query20	0.02	0.02	0.01
query21	15.43	0.17	0.14
query22	5.01	0.06	0.05
query23	16.15	0.30	0.12
query24	2.91	0.42	0.33
query25	0.12	0.05	0.04
query26	0.71	0.20	0.16
query27	0.04	0.03	0.04
query28	3.53	0.80	0.34
query29	12.47	4.02	3.19
query30	0.27	0.15	0.14
query31	2.77	0.56	0.31
query32	3.22	0.59	0.50
query33	3.15	3.25	3.22
query34	15.57	3.90	3.26
query35	3.23	3.22	3.24
query36	0.55	0.45	0.41
query37	0.08	0.08	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.14	0.15
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.21 s
Total hot run time: 23.84 s

@Mryange

Mryange commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review opinion: no new blocking findings at 82d95715c456989a9987e5b383c96bf9ee31d509. The two earlier inline concerns are addressed at this head, and the change looks ready from a code-review perspective.

Critical checkpoint conclusions:

  • Goal and proof: The change consistently stores cached constants as owned, single-row physical data columns, unwraps ColumnConst without copying, and recreates ColumnConst only when broadcasting. Production get_const_col() executes for one row and validates size and declared type before caching. The added wrapper tests cover direct data, const unwrapping, and build-mode-specific invariant failure; the surviving scan tests cover both const-wrapped and direct one-row Boolean constants.
  • Scope and clarity: The refactor is focused. Repository-wide searches found no stale public-field access or missed ColumnPtrWrapper producer/consumer, and the mechanical accessor changes preserve the existing row-zero semantics.
  • Data correctness, nullability, and lifecycle: ColumnConst delegates Field, StringRef, null, and complex-value reads to nested row zero, so eager unwrapping preserves scalar, nullable, array/map/struct, JSON, regex, date/time, IN-set, and ANN behavior. The wrapper owns the nested column, and cloned FunctionContexts retain shared ownership; retained buffers remain live with no COW mutation, dangling reference, or ownership cycle.
  • Concurrency: Constant wrappers are populated at fragment open before thread-local cloning and are read-only afterward. Scan false/NULL normalization still sets EOS and wakes the scan dependency before scanner preparation. No new thread, lock, atomic, lock-order, blocking, or deadlock behavior is introduced.
  • Error handling: Null and one-row wrapper preconditions fail loudly. Production expression size/type mismatches are returned as Status before wrapper construction, and existing status/exception boundaries remain intact.
  • Parallel paths and special checks: File, OLAP, Mock, and shared scan normalization paths, function-state consumers, constant rebroadcast, MATCH index/slow execution, and ANN TopN/range extraction were traced. Valid MATCH retains a slot child, so the removed root constant-cache call was a no-op; child constants and analyzer state are still initialized for original and cloned contexts. ANN keeps an owning nested Float32 pointer and preserves outer-null, empty, inner-null, array-shape, element-type, and dimension checks.
  • Configuration and compatibility: No configuration, serialization, storage-format, FE-BE variable, external symbol, transaction, persistence, data-write, or rolling-upgrade contract changes apply.
  • Test coverage and results: The two earlier test-contract findings are fixed at this head. Live BE UT, COMPILE, Clang Formatter, and relevant regression checks are passing. No local build or test was run because this review runner explicitly prohibits builds.
  • Observability and performance: No new runtime path needs logs or metrics. Normalization is no-copy, removes repeated const-shape dispatch, and adds no hot-loop allocation or obvious CPU/memory regression.
  • Additional user focus: No additional focus was provided; the complete PR was reviewed without narrowing.

Review completion: converged in Round 1. The main risk scan, two normal full-review partitions, and a separate risk-focused review completed; all returned no new valuable findings, every risk item has a concrete conclusion, and the final changed-file/unresolved-candidate sweep is clean. No new inline comments are needed.

@Mryange Mryange closed this Oct 27, 2026
Mryange added a commit that referenced this pull request Oct 29, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: #66614

Problem Summary:

Root cause: expression constness does not guarantee a physical
`ColumnConst`. Cached constant expressions can return either a const
wrapper or a single-row data column, while scan consumers sometimes cast
the result directly to `ColumnConst`.

Store an owned single-row data column in `ConstantExprResult`, unwrap
`ColumnConst` without copying, and expose read-only accessors. Migrate
scan, scalar-function, and ANN consumers, and recreate `ColumnConst`
when broadcasting the cached value. Preserve current function
initialization and NULL handling.

### Release note

Handle constant expressions that return ordinary single-row columns
consistently in scan predicate processing.

### Check List (For Author)

- Test:
- ASAN BE UT build passed. Ran 276 tests across 16 affected suites: 275
passed and the existing `MathFunctionTest.random_test` was skipped.
Coverage includes zero-copy normalization, nullable values and
ownership, arrays with nested NULLs, invalid multi-row input,
broadcasting, scan predicates, function constant arguments, and ANN
extraction.
- clang-format 16, `git diff --check`, and BE build hygiene checks
passed.
- clang-tidy checked all 43 modified C++ files. New diagnostics were
fixed and rechecked; existing complexity diagnostics remain on unchanged
lines in column and LIKE tests.
- Behavior changed: Yes. Scan consumers accept cached single-row
constants regardless of their original physical representation;
malformed multi-row cache inputs fail at construction.
- Does this need documentation: No. No SQL syntax, configuration, or
serialization changes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants