Skip to content

[fix](function) Reject a non-constant seed in array_shuffle - #68438

Merged
csun5285 merged 7 commits into
apache:masterfrom
csun5285:fix/array-shuffle-row-seed
Oct 9, 2026
Merged

csun5285 merged 7 commits into
apache:masterfrom
csun5285:fix/array-shuffle-row-seed

Conversation

@csun5285

@csun5285 csun5285 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #44901

Problem Summary:

  1. array_shuffle now rejects a non-constant seed, which was silently ignored for every row but the first one of each block.
  2. The seed now uses all 64 bits (std::mt19937_64), so any BIGINT seed works, including a negative one, which failed with an out-of-range error since [fix](compile) Fix MacOS compile failed #44901.
  3. Every row is now shuffled on its own, so a constant array no longer gets the same order on every row, and blocks without a seed no longer repeat each other.

doc: apache/doris-website#4176

Release note

array_shuffle / shuffle rejects a non-constant seed, uses all 64 bits of the seed (so the order for a given seed changes), and shuffles every row on its own.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. See the three points above.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:
`array_shuffle(arr, seed)` only read the seed of the first row in a block
and built one `std::mt19937` from it. Every later row kept drawing from the
same random stream, so its own seed was ignored and its result depended on
how many rows came before it in the block.

Reproduce (all rows in one block):

    CREATE TABLE t (k INT, a ARRAY<INT>, s BIGINT) DUPLICATE KEY(k)
    DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
    INSERT INTO t VALUES (1,[1,2,3,4,5],1),(2,[1,2,3,4,5],2),(3,[1,2,3,4,5],1);
    SELECT k, array_shuffle(a, s) FROM t ORDER BY k;

Before: k=2 gave [1,3,2,5,4] while array_shuffle([1,2,3,4,5], 2) gives
[4,1,3,5,2], and k=3 (seed 1) differed from k=1 (seed 1). Even a constant
seed gave a different result on every row.

Fix: when a seed is given, re-seed the generator with the row's own seed
before shuffling that row, for both column and constant seeds. The result
of a row now depends only on its array and seed, and matches the constant
call. Without a seed the function still uses one time-based stream per
block, as before.

The `array_shuffle` cases in test_array_functions were commented out
because "the result depends on the row position in the column"; they are
stable now and enabled again.

### Release note

Fix `array_shuffle(arr, seed)` ignoring the per-row seed. With a seed, a
row's result now depends only on its array and seed, so rows with the same
array and seed (including a constant seed) get the same permutation.

### Check List (For Author)

- Test: Regression test / Unit Test
    - BE UT function_array_shuffle_test (fails before the fix, passes after)
    - Regression test_array_shuffle_seed (new), test_array_functions
      (re-enabled shuffle cases), test_size_function
- Behavior changed: Yes. With a constant seed, all rows with the same array
  now get the same permutation instead of different ones per row.
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@csun5285

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 27825 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b94dfb4cde94ff95d1028e905233dc0929a1e71, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17613	3835	4056	3835
q2	2593	351	288	288
q3	10035	1407	838	838
q4	4688	477	350	350
q5	7471	854	539	539
q6	181	168	136	136
q7	748	790	602	602
q8	9295	1411	1517	1411
q9	5486	4187	4173	4173
q10	6941	1339	1023	1023
q11	468	270	255	255
q12	649	428	297	297
q13	18059	2628	2018	2018
q14	267	253	233	233
q15	q16	738	717	657	657
q17	1725	1148	1032	1032
q18	6526	5583	5514	5514
q19	1323	1259	984	984
q20	568	394	262	262
q21	5842	3503	3073	3073
q22	452	369	305	305
Total cold run time: 101668 ms
Total hot run time: 27825 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4754	4580	4455	4455
q2	868	583	559	559
q3	4883	5154	4565	4565
q4	2199	2311	1459	1459
q5	4589	4415	4677	4415
q6	229	177	127	127
q7	1779	1701	1486	1486
q8	2343	2027	2045	2027
q9	7322	7232	7203	7203
q10	3709	3596	3063	3063
q11	508	366	339	339
q12	707	700	500	500
q13	2296	2588	2000	2000
q14	275	274	249	249
q15	q16	664	684	601	601
q17	7309	6712	6651	6651
q18	11920	10999	11754	10999
q19	1072	981	987	981
q20	2194	2199	1895	1895
q21	4997	4099	4304	4099
q22	500	443	405	405
Total cold run time: 65117 ms
Total hot run time: 58078 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: two issues are documented inline.

Checkpoint conclusions:

  • Goal and data correctness: the per-row reseed design fixes the original block-order dependence for ordinary vector and constant seeds, but nullable execution can still fail rows whose propagated result must be NULL (inline P1).
  • Scope and compatibility: the change is focused; FE/BE signatures and the shuffle alias are unchanged, and the one-argument time-seeded stream remains unchanged. There is no storage-format, protocol, configuration, or rolling-upgrade surface.
  • Concurrency, lifecycle, and memory: generators and callbacks are stack-local per execution, with no shared mutable state, lock ordering, static-initialization, or persistent ownership concern. New includes are confined to the .cpp.
  • Null/error handling and parallel paths: default nullable wrapping is the blocking correctness gap; both function spellings share the same implementation and tests.
  • Tests and expected results: the new BE test is discovered by the existing recursive test target and supplies a direct one-block oracle. Added/re-enabled regression results are ordered or order-invariant and internally consistent across aliases, constants, vector seeds, NULLs, empty arrays, lengths, and element types. They do not cover the combined-null failure or the no-op reseed cost identified inline.
  • Performance: empty and singleton seeded arrays initialize the full mt19937 state per row despite requiring no shuffle (inline P2).
  • Persistence, transactions, writes, and observability: not applicable to this scalar-function change; no additional logging or metrics are needed.
  • User focus: no additional focus was provided.

Validation was static only, as required by the review runner; I did not run builds or tests and do not treat author/CI claims as independent execution evidence.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 27234 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b94dfb4cde94ff95d1028e905233dc0929a1e71, data reload: false

------ Round 1 ----------------------------------
============================================
q1	18270	3842	3758	3758
q2	2630	355	300	300
q3	10340	1390	755	755
q4	4697	476	355	355
q5	7867	815	547	547
q6	183	174	142	142
q7	748	769	601	601
q8	9393	1414	1496	1414
q9	6194	4134	4154	4134
q10	7433	1308	1013	1013
q11	489	273	236	236
q12	730	417	292	292
q13	18223	2587	2065	2065
q14	257	256	233	233
q15	q16	738	718	652	652
q17	1736	1113	1045	1045
q18	6511	5567	5530	5530
q19	1202	1178	969	969
q20	482	399	264	264
q21	5444	2919	2640	2640
q22	420	350	289	289
Total cold run time: 103987 ms
Total hot run time: 27234 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4190	4037	4018	4018
q2	721	551	513	513
q3	4430	4833	4313	4313
q4	2192	2286	1447	1447
q5	4157	4116	4077	4077
q6	223	169	125	125
q7	1721	1576	1646	1576
q8	2242	1971	1974	1971
q9	7326	7234	7123	7123
q10	3679	3643	3147	3147
q11	571	395	378	378
q12	712	715	523	523
q13	2367	2821	2131	2131
q14	294	303	264	264
q15	q16	732	707	643	643
q17	7824	6835	6837	6835
q18	11972	11045	11747	11045
q19	1144	1034	1024	1024
q20	2242	2231	1961	1961
q21	5270	4359	4486	4359
q22	544	481	410	410
Total cold run time: 64553 ms
Total hot run time: 57883 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 153556 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6b94dfb4cde94ff95d1028e905233dc0929a1e71, data reload: false

query5	4312	591	452	452
query6	440	216	192	192
query7	4927	547	295	295
query8	453	173	163	163
query9	8833	3927	3902	3902
query10	446	326	250	250
query11	5800	3545	3222	3222
query12	150	88	85	85
query13	1244	607	419	419
query14	6509	4539	4202	4202
query14_1	3971	3932	3913	3913
query15	205	204	180	180
query16	3268	474	431	431
query17	912	670	533	533
query18	2227	452	336	336
query19	218	181	146	146
query20	82	81	79	79
query21	312	135	113	113
query22	13397	13949	13754	13754
query23	14352	13387	12932	12932
query23_1	12682	12492	12525	12492
query24	7278	1109	671	671
query24_1	708	653	713	653
query25	571	428	379	379
query26	1220	318	165	165
query27	2785	581	340	340
query28	4565	1972	1959	1959
query29	1626	712	527	527
query30	295	210	181	181
query31	964	734	632	632
query32	152	82	85	82
query33	507	301	235	235
query34	1226	1088	609	609
query35	718	746	621	621
query36	822	798	688	688
query37	139	99	83	83
query38	1823	1763	1673	1673
query39	684	658	653	653
query39_1	647	625	652	625
query40	214	123	92	92
query41	63	60	58	58
query42	92	84	87	84
query43	335	342	297	297
query44	1396	712	701	701
query45	181	169	158	158
query46	1095	1144	762	762
query47	1488	1459	1383	1383
query48	392	385	296	296
query49	633	407	281	281
query50	942	331	249	249
query51	10527	10487	10466	10466
query52	86	94	75	75
query53	237	250	173	173
query54	253	197	176	176
query55	80	72	68	68
query56	255	214	237	214
query57	1587	1435	1263	1263
query58	287	250	251	250
query59	1974	2067	1847	1847
query60	287	238	217	217
query61	149	173	145	145
query62	429	312	271	271
query63	210	177	176	176
query64	2471	943	849	849
query65	3475	3425	3396	3396
query66	1724	417	293	293
query67	20261	20028	20461	20028
query68	3242	1584	961	961
query69	402	299	255	255
query70	863	782	806	782
query71	301	228	206	206
query72	2692	2528	2197	2197
query73	855	733	435	435
query74	4605	4508	4275	4275
query75	2305	2277	1923	1923
query76	1881	1054	741	741
query77	351	391	293	293
query78	9073	9228	8523	8523
query79	1431	1244	787	787
query80	971	457	365	365
query81	628	324	274	274
query82	649	156	123	123
query83	282	226	195	195
query84	361	139	112	112
query85	1176	509	381	381
query86	436	231	229	229
query87	1984	1987	1825	1825
query88	3620	2705	2687	2687
query89	381	283	250	250
query90	1802	184	169	169
query91	169	159	124	124
query92	92	87	83	83
query93	1542	1510	879	879
query94	658	338	303	303
query95	662	370	332	332
query96	1068	802	348	348
query97	2414	2423	2340	2340
query98	160	158	142	142
query99	787	738	626	626
Total cold run time: 240595 ms
Total hot run time: 153556 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.01 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6b94dfb4cde94ff95d1028e905233dc0929a1e71, data reload: false

query1	0.01	0.01	0.00
query2	0.26	0.05	0.05
query3	0.41	0.14	0.14
query4	1.72	0.14	0.14
query5	0.24	0.23	0.22
query6	1.15	0.93	0.93
query7	0.04	0.01	0.01
query8	0.05	0.04	0.03
query9	0.50	0.33	0.33
query10	0.53	0.54	0.54
query11	0.27	0.15	0.15
query12	0.23	0.15	0.14
query13	0.46	0.47	0.46
query14	0.96	0.94	0.94
query15	0.61	0.59	0.57
query16	0.30	0.33	0.32
query17	1.09	1.09	1.09
query18	0.21	0.19	0.21
query19	2.09	1.86	1.97
query20	0.02	0.01	0.02
query21	15.66	0.19	0.14
query22	4.81	0.05	0.06
query23	16.68	0.30	0.12
query24	6.91	0.41	0.36
query25	0.14	0.05	0.04
query26	0.77	0.20	0.14
query27	0.04	0.03	0.03
query28	3.54	0.82	0.39
query29	12.52	4.05	3.21
query30	0.28	0.14	0.16
query31	2.80	0.54	0.31
query32	3.36	0.60	0.48
query33	3.12	3.19	3.26
query34	15.62	3.91	3.28
query35	3.23	3.22	3.22
query36	0.57	0.44	0.44
query37	0.16	0.07	0.07
query38	0.06	0.04	0.04
query39	0.12	0.03	0.03
query40	0.18	0.15	0.14
query41	0.15	0.03	0.02
query42	0.11	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 102.02 s
Total hot run time: 24.01 s

### What problem does this PR solve?

Issue Number: None

Related PR: apache#44901

Problem Summary:
With the per-row seed fix, every row's seed goes through
`cast_set<uint32_t>`, which throws INTERNAL_ERROR for a negative seed or
one above 4294967295:

    SELECT array_shuffle([1, 2, 3, 4, 5], -1);
    -- [INTERNAL_ERROR]value -1 cast  to type j out of range [0,4294967295]

Default NULL handling runs the function on every physical row, so this
also fails rows whose result is NULL: a NULL array next to seed -1 in the
same block, or `NULLIF(s, -1)`, which keeps -1 under the NULL. The range
check came from apache#44901 (a macOS build fix in 4.0); 2.1/3.0/3.1 accepted
these seeds.

Rows with 0 or 1 element also re-seeded the generator, which fills the
624-word mt19937 state, although shuffling them draws no random numbers.

Fix:
- Like ClickHouse's arrayShuffle, use the whole BIGINT as a 64-bit seed
  with std::mt19937_64 (the engine uniform() already seeds per row). Any
  BIGINT is a valid seed, seeds with the same low 32 bits (such as -1 and
  4294967295) no longer give the same result, and no row can fail on its
  seed. The query above now returns [3, 1, 4, 2, 5].
- Skip arrays with fewer than 2 elements before seeding the generator.
- The no-seed path uses the same engine; it is still one time-based
  stream per block.
- The UT helper now stops when execute fails, instead of reading the
  missing result column and aborting the whole test binary.

### Release note

array_shuffle(arr, seed) accepts any BIGINT seed again, including
negative values. The permutation for a given seed differs from earlier
versions.

### Check List (For Author)

- Test: Unit Test / Regression test
    - BE UT function_array_shuffle_test: 4/4 pass; the new any_bigint_seed
      case fails on the previous code with the INTERNAL_ERROR above
    - Regression test_array_shuffle_seed (new cases: negative seeds, the
      BIGINT limits, a NULL array with seed -1, NULLIF(s, -1)),
      test_array_functions, test_size_function, test_array_function_doc
      (.out regenerated, only seeded shuffle results changed),
      test_array_functions_by_literal, test_array_zip_array_enumerate_uniq
- Behavior changed: Yes. Negative seeds and seeds above 4294967295 work
  instead of failing, and the permutation for a given seed differs from
  earlier versions.
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review result for exact head 61608979f6e81905b0fce7f173241fa0ea2eca71: requesting changes for one compatibility issue, detailed inline.

Critical checkpoints:

  • Goal and correctness: the per-row seed bug is otherwise fixed. Seeded rows are isolated from preceding rows and block boundaries; constant and column seeds, nullable inputs, both aliases, negative seeds, and BIGINT extremes follow the intended local call chain.
  • Scope and compatibility: replacing mt19937 with mt19937_64 changes established outputs even for a first/single row and also changes the unrelated unseeded stream. Doris documents a supplied seed as producing reproducible results, but this PR provides no old implementation selected during rolling upgrades. This is the blocking point.
  • Concurrency and lifecycle: generators are stack-local to each execution call; no shared mutable state, lock ordering, static initialization, or special lifetime issue is introduced.
  • Error, NULL, and parallel paths: default constant/NULL wrappers keep physical row and seed indexes aligned. The prior NULL-payload and empty/singleton concerns were treated as duplicate fences; every BIGINT is now accepted and the short-array path skips generator setup. array_shuffle and shuffle share the same BE implementation and FE BIGINT coercion.
  • Tests and results: the new BE test is discovered by the recursive UT glob and uses the public function wrapper. Regression coverage includes per-row and constant seeds, aliases, nullable combinations, NULLIF, short arrays, block-versus-single-row behavior, and BIGINT limits; changed golden blocks map to their queries. No build or test was run in this review environment, per the review contract, so this is static evidence rather than independent execution.
  • Configuration, persistence, data writes, FE/BE protocol, observability: not applicable; the PR adds no config, persisted or serialized state, transaction/write path, transmitted variable, or operational signal.
  • Performance: per-row state initialization is inherent in the chosen fix for nontrivial seeded rows, and the new < 2 guard removes the avoidable trivial-row cost. No additional distinct performance issue survived review.
  • User focus: no additional user-provided focus was supplied.

Review completeness: the authoritative eight-file diff received a main scan, two normal full-review passes per round, a separate risk-focused pass per round, candidate adjudication, duplicate fencing, and a final changed-file sweep. Round 2 converged with NO_NEW_VALUABLE_FINDINGS; M-001 below is the only accepted new issue.

@csun5285

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 27989 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 61608979f6e81905b0fce7f173241fa0ea2eca71, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17966	3883	3972	3883
q2	2224	380	339	339
q3	10005	1454	819	819
q4	4687	477	346	346
q5	7480	812	555	555
q6	182	169	143	143
q7	750	782	590	590
q8	9315	1582	1619	1582
q9	5512	4189	4223	4189
q10	6831	1333	1032	1032
q11	429	275	241	241
q12	638	412	289	289
q13	18028	2605	1999	1999
q14	261	262	252	252
q15	q16	733	715	666	666
q17	1661	1128	939	939
q18	6430	5591	5546	5546
q19	1145	1262	956	956
q20	471	396	263	263
q21	5273	3186	3047	3047
q22	442	369	313	313
Total cold run time: 100463 ms
Total hot run time: 27989 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4664	4728	4378	4378
q2	700	570	563	563
q3	4805	5227	4585	4585
q4	2212	2315	1439	1439
q5	4571	4340	4586	4340
q6	227	175	127	127
q7	1817	1693	1518	1518
q8	2409	2207	1978	1978
q9	7288	7275	7168	7168
q10	3668	3623	3152	3152
q11	512	373	341	341
q12	699	705	502	502
q13	2268	2585	2013	2013
q14	264	269	240	240
q15	q16	661	675	604	604
q17	7334	6720	6633	6633
q18	11860	11045	11744	11045
q19	1084	1018	1039	1018
q20	2198	2188	1943	1943
q21	4945	4164	4275	4164
q22	535	470	408	408
Total cold run time: 64721 ms
Total hot run time: 58159 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 152903 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 61608979f6e81905b0fce7f173241fa0ea2eca71, data reload: false

query5	4329	598	469	469
query6	437	214	189	189
query7	4961	537	321	321
query8	320	178	164	164
query9	8793	3998	4020	3998
query10	460	312	254	254
query11	5995	3541	3246	3246
query12	149	96	89	89
query13	1259	582	445	445
query14	6527	4470	4223	4223
query14_1	3973	3982	3942	3942
query15	196	193	175	175
query16	967	439	414	414
query17	881	657	532	532
query18	2466	453	333	333
query19	195	175	136	136
query20	80	79	77	77
query21	218	130	113	113
query22	13007	13024	12794	12794
query23	13784	12990	12317	12317
query23_1	12470	12429	12443	12429
query24	7225	1232	696	696
query24_1	730	715	739	715
query25	547	413	359	359
query26	1273	312	171	171
query27	2667	590	337	337
query28	4571	2060	2042	2042
query29	1629	730	537	537
query30	304	228	185	185
query31	897	760	654	654
query32	142	106	95	95
query33	535	318	257	257
query34	1218	1146	635	635
query35	725	754	647	647
query36	802	805	746	746
query37	153	106	93	93
query38	1830	1755	1689	1689
query39	715	712	653	653
query39_1	663	662	665	662
query40	230	127	103	103
query41	74	70	71	70
query42	102	101	97	97
query43	339	348	297	297
query44	1377	741	742	741
query45	186	186	168	168
query46	1129	1214	767	767
query47	1510	1496	1385	1385
query48	437	396	311	311
query49	591	394	279	279
query50	936	364	254	254
query51	10477	10305	10404	10305
query52	85	86	75	75
query53	248	255	185	185
query54	260	204	189	189
query55	86	87	72	72
query56	244	230	204	204
query57	1376	1366	1389	1366
query58	278	258	251	251
query59	1983	2051	1860	1860
query60	275	243	225	225
query61	150	147	146	146
query62	388	319	272	272
query63	209	177	175	175
query64	2758	981	862	862
query65	3462	3429	3395	3395
query66	1800	420	300	300
query67	19946	19864	19875	19864
query68	3268	1541	938	938
query69	411	317	257	257
query70	891	848	829	829
query71	295	234	208	208
query72	2628	2588	2290	2290
query73	877	818	468	468
query74	4682	4579	4379	4379
query75	2329	2358	1978	1978
query76	2326	1158	772	772
query77	364	395	294	294
query78	9113	8986	8513	8513
query79	1204	1166	773	773
query80	509	461	367	367
query81	536	322	283	283
query82	270	159	128	128
query83	218	226	206	206
query84	302	140	110	110
query85	802	466	388	388
query86	271	246	231	231
query87	1972	1981	1822	1822
query88	3624	2800	2772	2772
query89	330	281	251	251
query90	2113	178	176	176
query91	169	158	128	128
query92	103	91	89	89
query93	1436	1383	892	892
query94	498	349	315	315
query95	661	455	344	344
query96	1008	773	330	330
query97	2421	2396	2299	2299
query98	155	151	143	143
query99	715	729	618	618
Total cold run time: 235129 ms
Total hot run time: 152903 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 61608979f6e81905b0fce7f173241fa0ea2eca71, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.25	0.14	0.13
query4	1.61	0.13	0.14
query5	0.25	0.22	0.21
query6	1.17	0.93	0.91
query7	0.04	0.01	0.00
query8	0.06	0.03	0.04
query9	0.39	0.34	0.34
query10	0.56	0.55	0.59
query11	0.19	0.15	0.15
query12	0.19	0.14	0.15
query13	0.47	0.48	0.47
query14	0.96	0.94	0.94
query15	0.62	0.59	0.58
query16	0.31	0.34	0.34
query17	1.09	1.14	1.09
query18	0.22	0.20	0.21
query19	1.97	1.92	1.97
query20	0.02	0.02	0.01
query21	15.48	0.21	0.13
query22	4.88	0.06	0.05
query23	16.14	0.29	0.12
query24	2.89	0.40	0.31
query25	0.13	0.05	0.04
query26	0.72	0.20	0.15
query27	0.04	0.04	0.03
query28	3.54	0.79	0.35
query29	12.50	4.06	3.22
query30	0.28	0.15	0.14
query31	2.77	0.56	0.31
query32	3.23	0.60	0.49
query33	3.27	3.16	3.26
query34	15.81	3.90	3.28
query35	3.28	3.23	3.25
query36	0.56	0.43	0.40
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.16	0.14
query41	0.09	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.52 s
Total hot run time: 23.96 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.35% (34399/45053)
Line Coverage 61.24% (386263/630765)
Region Coverage 57.55% (324916/564573)
Branch Coverage 58.44% (148272/253716)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#44901

Problem Summary:

array_shuffle(arr[, seed]) builds one random generator per block, and every row of the
block draws from it. The earlier commits on this branch re-seeded the generator for every
row, which gave each row its own seed but costs about 1 us per row, because seeding
std::mt19937_64 fills 312 state words. Other engines with a seeded array shuffle
(ClickHouse arrayShuffle, Spark shuffle, Velox, the DataFusion Spark-compatible shuffle) do
not take a per-row seed: the seed must be a constant, and each block draws from one random
sequence. This commit switches to that model and fixes the related problems:

1. A non-constant seed was accepted but silently ignored, because the generator was seeded
   from the seed of the first row of the block. The FE now rejects it in
   ArrayShuffle.checkLegalityBeforeTypeCoercion ("The seed of array_shuffle must be a
   constant"), and the BE rejects it in open(), so a BE that gets such a plan from an older
   FE during a rolling upgrade fails instead of ignoring the seeds.
2. Since apache#44901, a negative seed or a seed larger than 4294967295 failed with an
   out-of-range INTERNAL_ERROR from cast_set<uint32_t>. The seed now keeps its low 32 bits
   with static_cast<uint32_t>, which is the 3.x behavior. Results for seeds in
   [0, 4294967295] do not change.
3. A constant array got the same order on every row, because the BE computed the call once
   for all-constant arguments, and with enable_fold_constant_by_be = true the FE folded the
   call into one literal. The BE now returns false from
   use_default_implementation_for_constants(), and ArrayShuffle extends UniqueFunction like
   random() and uuid(), so it is never folded into a constant or merged with another call.
   Before: 100 rows of a constant array gave 1 order; after: 100 orders.
4. Without a seed, each block was seeded with time(nullptr), so blocks processed in the same
   second repeated the same orders: 100000 rows of a 20-element array gave only 26560
   different orders. Each block is now seeded from std::random_device, like rand(), and the
   same query gives 100000 orders.

A single call with a constant seed, such as array_shuffle([1, 2, 3, 4], 0), still gives the
same result as before.

### Release note

array_shuffle / shuffle: a non-constant seed now reports an error instead of being silently
ignored; a negative seed or a seed larger than 4294967295 is accepted again, and only its
low 32 bits are used; a constant array is shuffled on each row instead of getting the same
order on every row; without a seed, rows processed in the same second no longer repeat the
same orders.

### Check List (For Author)

- Test: Unit Test / Regression test
    - BE unit test: function_array_shuffle_test, 6 cases passed
    - FE unit test: UniqueFunctionTest, 18 cases passed
    - Regression test: test_array_shuffle_seed passed; test_array_functions,
      test_size_function and test_array_function_doc passed and their .out files are the
      same as on master
- Behavior changed: Yes. A non-constant seed reports an error, a negative or large seed no
  longer reports an error, and a constant array gets a different order on each row.
- Does this need documentation: Yes, an apache/doris-website PR will follow

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
csun5285 and others added 2 commits September 28, 2026 14:14
### What problem does this PR solve?

Issue Number: None

Related PR: apache#44901

Problem Summary:

array_shuffle kept only the low 32 bits of its BIGINT seed for std::mt19937, so seeds with the
same low 32 bits, such as -1 and 4294967295, gave the same order. ClickHouse arrayShuffle gives
the full 64-bit seed to pcg64_fast. This commit switches the generator to std::mt19937_64 and
uses all 64 bits of the seed, so -1 and 4294967295 now give different orders. Without a seed,
each block is seeded with 64 bits from std::random_device.

Because the generator changed, the order for a given seed is different from before, so the
seeded results of test_size_function, test_array_function_doc and test_array_shuffle_seed are
regenerated. Unit tests that expect two orders to differ now use a 20-element array, because a
5-element array has only 120 orders and two seeds can give the same one by chance.

This commit also fixes the comments that said all rows share one random sequence. Each block
starts its own sequence from the seed, so a per-row seed would be ignored, which is why a
non-constant seed is rejected.

### Release note

array_shuffle / shuffle uses all 64 bits of the seed, so the order for a given seed differs from
earlier versions, and seeds with the same low 32 bits no longer give the same order.

### Check List (For Author)

- Test: Unit Test / Regression test
    - BE unit test: function_array_shuffle_test, 6 cases passed
    - Regression test: test_array_shuffle_seed, test_array_functions, test_size_function and
      test_array_function_doc passed; test_array_functions keeps its .out file
- Behavior changed: Yes. The order for a given seed changes, and seeds with the same low 32 bits
  give different orders.
- Does this need documentation: Yes, an apache/doris-website PR will follow

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
### What problem does this PR solve?

Issue Number: None

Related PR: apache#44901

Problem Summary:

An earlier commit on this branch read the constant seed with unpack_if_const, which takes the
one-value column inside the ColumnConst instead of expanding it to a full column. Both give the
same seed, and expanding the constant only copies one BIGINT per row of the block, which is
small next to shuffling the arrays. Go back to convert_to_full_column_if_const, as the original
code does, so the change to this function stays small, and drop the two includes that were only
added for unpack_if_const.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - BE unit test: function_array_shuffle_test, 6 cases passed
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@csun5285 csun5285 changed the title [fix](be) Use each row's seed in array_shuffle [fix](function) Reject a non-constant seed in array_shuffle Sep 28, 2026
@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: {"type":"error","status":400,"error":{"type":"invalid_request_error","message":"The 'gpt-6-sol' model is not supported when using Codex with a ChatGPT account."}}
Workflow run: https://github.com/apache/doris/actions/runs/36388314263

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@shuke987

Copy link
Copy Markdown
Contributor

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: one P1 data-correctness issue is inline. The new per-row behavior for a constant-array shuffle can make external file scans run the same volatile predicate twice and discard rows. The existing threads already cover rolling-upgrade output, NULL seeds, and short arrays; this review does not repeat them. The user supplied no additional focus points.

Critical checkpoints (static review):

  • Goal and proof: FE and BE now reject nonconstant seeds, use the full BIGINT seed, and vary constant-array and unseeded results by row/block as intended. The added BE, FE, and SQL tests cover these direct behaviors. They do not cover a shuffle predicate on an external file scan, where the accepted issue occurs.
  • Scope and parallel paths: The implementation is focused on the function and its FE identity. shuffle shares the BE registration and FE binding. BE's separate determinism check used by file-scan localization was not updated for either name.
  • Concurrency and lifecycle: The generator is local to execution; no new shared state or lock order appears. The FunctionContext constant check runs at open, and the generator has block lifetime. No new long-lived ownership or static initialization issue was found.
  • Configuration and compatibility: No configuration or FE/BE protocol variable was added. Seeded output deliberately changes; the rolling-upgrade concern is already in an inline thread.
  • Conditions and tests: FE constantness and BE constant-column validation align for the reviewed literal, expression, and slot cases. The nullable and zero-row wrappers avoid a separate established failure. Expected-error tests use the suite pattern; changed goldens correspond to the source queries by static inspection. Exact permutations and test execution were not independently verified.
  • Observability, persistence, and writes: No new persistent state, transaction, data-write path, or distributed operation is involved. Existing error status is adequate for the seed rejection; no new log or metric requirement was established.
  • Performance and remaining issues: The new random-device call is per block; no separate material regression was substantiated. The short-array cost is already covered by an existing thread. The file-scan double-evaluation issue is the sole new blocking finding.

This was a static review only. The review environment prohibits builds and tests, so no runtime validation is claimed.

@csun5285

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 27796 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f79d2b0609745c9972b4f50fccf5f58599d02d3a, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17610	3860	3844	3844
q2	2175	367	320	320
q3	10078	1439	780	780
q4	4685	478	349	349
q5	7463	811	573	573
q6	182	170	138	138
q7	730	838	604	604
q8	9302	1580	1575	1575
q9	5481	4233	4175	4175
q10	6840	1398	995	995
q11	434	274	239	239
q12	635	416	299	299
q13	18062	2620	2003	2003
q14	263	264	240	240
q15	q16	745	721	670	670
q17	1741	1179	952	952
q18	6510	5601	5571	5571
q19	1360	1249	986	986
q20	485	397	258	258
q21	5849	3405	2912	2912
q22	470	359	313	313
Total cold run time: 101100 ms
Total hot run time: 27796 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4629	4709	4422	4422
q2	726	587	546	546
q3	4841	5124	4605	4605
q4	2256	2355	1480	1480
q5	4599	4440	4585	4440
q6	227	174	131	131
q7	1818	1721	1478	1478
q8	2377	2006	2065	2006
q9	7294	7289	7279	7279
q10	3675	3589	3069	3069
q11	523	365	338	338
q12	704	707	496	496
q13	2269	2579	1994	1994
q14	270	279	244	244
q15	q16	664	681	593	593
q17	7530	6716	6646	6646
q18	11859	11066	11774	11066
q19	1170	1014	1035	1014
q20	2175	2194	1902	1902
q21	4946	4130	4285	4130
q22	518	452	387	387
Total cold run time: 65070 ms
Total hot run time: 58266 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 152414 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f79d2b0609745c9972b4f50fccf5f58599d02d3a, data reload: false

query5	4343	601	478	478
query6	417	203	191	191
query7	4809	560	298	298
query8	327	186	183	183
query9	8849	3997	3982	3982
query10	466	300	246	246
query11	5696	3557	3230	3230
query12	151	93	89	89
query13	1262	558	412	412
query14	6559	4461	4255	4255
query14_1	3961	3927	3908	3908
query15	206	200	183	183
query16	1007	447	411	411
query17	919	682	549	549
query18	2432	499	328	328
query19	192	172	131	131
query20	80	80	78	78
query21	218	128	109	109
query22	12998	12896	12782	12782
query23	14068	13021	12344	12344
query23_1	12478	12745	12546	12546
query24	7242	1193	639	639
query24_1	688	712	764	712
query25	542	410	362	362
query26	1257	296	158	158
query27	2738	551	323	323
query28	4596	1985	1986	1985
query29	1551	676	493	493
query30	297	220	186	186
query31	886	746	654	654
query32	142	99	87	87
query33	504	303	244	244
query34	1159	1100	627	627
query35	722	743	620	620
query36	806	796	743	743
query37	135	101	91	91
query38	1809	1748	1720	1720
query39	679	705	653	653
query39_1	658	654	667	654
query40	229	120	97	97
query41	65	65	62	62
query42	94	92	90	90
query43	334	357	293	293
query44	1364	726	716	716
query45	192	174	164	164
query46	1067	1200	725	725
query47	1490	1486	1410	1410
query48	404	409	295	295
query49	591	440	297	297
query50	988	359	254	254
query51	10842	10684	10504	10504
query52	89	90	78	78
query53	245	258	187	187
query54	265	216	203	203
query55	85	74	76	74
query56	241	228	221	221
query57	1445	1447	1376	1376
query58	285	271	263	263
query59	1978	2080	1876	1876
query60	296	254	238	238
query61	174	178	170	170
query62	397	325	269	269
query63	215	180	173	173
query64	2955	1121	945	945
query65	3469	3380	3413	3380
query66	1846	416	296	296
query67	19987	19676	19922	19676
query68	3190	1605	906	906
query69	401	303	262	262
query70	913	818	848	818
query71	291	216	211	211
query72	2637	2480	2227	2227
query73	806	764	431	431
query74	4607	4465	4270	4270
query75	2299	2279	1935	1935
query76	2320	1133	754	754
query77	349	393	294	294
query78	9025	9013	8497	8497
query79	1386	1178	758	758
query80	1230	466	359	359
query81	598	324	281	281
query82	686	161	120	120
query83	306	225	198	198
query84	309	145	113	113
query85	903	458	381	381
query86	387	231	234	231
query87	2002	1963	1828	1828
query88	3598	2714	2713	2713
query89	362	285	246	246
query90	1877	185	168	168
query91	172	157	129	129
query92	97	92	90	90
query93	1476	1377	840	840
query94	677	322	305	305
query95	665	450	336	336
query96	1043	817	342	342
query97	2441	2409	2319	2319
query98	161	156	147	147
query99	738	737	636	636
Total cold run time: 237170 ms
Total hot run time: 152414 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f79d2b0609745c9972b4f50fccf5f58599d02d3a, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.27	0.13	0.13
query4	1.60	0.14	0.14
query5	0.25	0.23	0.21
query6	1.15	0.95	0.91
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.39	0.34	0.33
query10	0.54	0.54	0.56
query11	0.20	0.15	0.14
query12	0.19	0.15	0.15
query13	0.45	0.46	0.46
query14	0.95	0.95	0.92
query15	0.60	0.58	0.59
query16	0.30	0.31	0.31
query17	1.10	1.12	1.11
query18	0.22	0.20	0.20
query19	2.03	1.88	1.87
query20	0.02	0.01	0.01
query21	15.49	0.20	0.13
query22	4.95	0.05	0.06
query23	16.15	0.31	0.12
query24	2.98	0.40	0.31
query25	0.10	0.05	0.04
query26	0.73	0.21	0.14
query27	0.04	0.03	0.05
query28	3.51	0.74	0.34
query29	12.49	4.10	3.22
query30	0.28	0.16	0.15
query31	2.78	0.55	0.30
query32	3.22	0.59	0.49
query33	3.12	3.24	3.22
query34	15.85	3.91	3.30
query35	3.23	3.19	3.25
query36	0.54	0.43	0.42
query37	0.09	0.07	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.44 s
Total hot run time: 23.92 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.33% (30044/46700)
Line Coverage 48.98% (313644/640357)
Region Coverage 44.45% (252884/568856)
Branch Coverage 46.05% (117790/255781)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.36% (34512/45194)
Line Coverage 61.30% (388014/632976)
Region Coverage 57.63% (326493/566520)
Branch Coverage 58.50% (149145/254962)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 87.50% (7/8) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 62.50% (5/8) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.36% (34512/45194)
Line Coverage 61.30% (388014/632976)
Region Coverage 57.63% (326493/566520)
Branch Coverage 58.50% (149145/254962)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.56% (7/1252) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.36% (34512/45194)
Line Coverage 61.30% (388023/632976)
Region Coverage 57.60% (326335/566520)
Branch Coverage 58.49% (149139/254962)

@yujun777 yujun777 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

yiguolei pushed a commit to apache/doris-website that referenced this pull request Oct 9, 2026
Update ARRAY_SHUFFLE docs, for apache/doris#68438:

- seed must be a constant; a column or another non-constant seed reports
an error
- all 64 bits of the seed are used, so any BIGINT works, a negative one
too
- each row is shuffled on its own, a constant array too, and each batch
starts a random sequence from the seed

## Versions 

- [x] dev
- [x] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)

## Languages

- [ ] Chinese
- [ ] English

## Docs Checklist

- [ ] Checked by AI
- [ ] Test Cases Built
- [ ] Updated required version and language counterparts, or explained
why not
- [ ] If only one language changed, confirmed whether source/translation
counterparts need sync

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
@csun5285
csun5285 merged commit a53b182 into apache:master Oct 9, 2026
42 of 43 checks passed
@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

github-actions Bot pushed a commit that referenced this pull request Oct 9, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: #44901

Problem Summary:

1. `array_shuffle` now rejects a non-constant seed, which was silently
ignored for every row but the first one of each block.
2. The seed now uses all 64 bits (`std::mt19937_64`), so any BIGINT seed
works, including a negative one, which failed with an out-of-range error
since #44901.
3. Every row is now shuffled on its own, so a constant array no longer
gets the same order on every row, and blocks without a seed no longer
repeat each other.

doc: apache/doris-website#4176

### Release note

`array_shuffle` / `shuffle` rejects a non-constant seed, uses all 64
bits of the seed (so the order for a given seed changes), and shuffles
every row on its own.


### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [x] Yes. See the three points above.

- Does this need documentation?
    - [ ] No.
    - [x] Yes. <!-- apache/doris-website PR link to be added -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 10, 2026
#68438 (#68832)

Cherry-picked from #68438

Co-authored-by: Chenyang Sun <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 10, 2026
#68438 (#68832)

Cherry-picked from #68438

Co-authored-by: Chenyang Sun <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
yiguolei pushed a commit that referenced this pull request Oct 10, 2026
…#68672)

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Condition cache can incorrectly reuse false ranges for expressions whose
results vary across executions, and for virtual-column predicates whose
definitions are omitted from the digest. For example, running `abs(k +
1) + abs(k + 1) = 0` before `abs(k - 9999) + abs(k - 9999) = 0` can
incorrectly omit `k = 9999`.

Share expression eligibility with query cache, inspect scan predicates,
virtual columns and runtime-filter probes (including implicit casts),
and pass scan eligibility to BE before computing the cache digest.
Recognize scalar and ARRAY/MAP/STRUCT TIME-to-date conversions while
preserving supported NULL folding. VirtualSlotRef returns the
zero-digest sentinel to disable reuse when its definition is absent from
the inherited digest.

Add per-query ConditionCacheLookup counters for hits and misses.
Regression coverage identifies individual execution profiles, checks
that unsafe scans perform zero lookups, and retains positive checks for
deterministic cache hits.

Rebased onto master `a53b182f810be2b3731c7dfdf6f40ff5a0ad63bd`. Preserve
master's #68438 shuffle identity and constant-seed validation while
allowing NULL-only folding, and retain master's grouped non-blocking
runtime-filter test alongside cache-eligibility coverage. All five
original commits are retained.

### Release note

Prevent incorrect cached results for volatile predicates,
query-date-dependent TIME casts, and virtual-column predicates. Preserve
supported NULL folding for TIME casts and shuffle calls. Expose
ConditionCacheLookup in OLAP scan profiles.

Rolling upgrade: upgrade BE before FE. The optional plan flag defaults
false, so a new BE disables condition cache for old FE plans; an old BE
cannot enforce the new eligibility flag.

### Check List (For Author)

- Test:
- [x] Unit Test: 39 targeted FE tests passed after rebase using
`FE_UT_PARALLEL=48 bash run-fe-ut.sh --run
'org.apache.doris.nereids.util.CacheExpressionTest,org.apache.doris.planner.ConditionCacheEligibilityTest,org.apache.doris.nereids.glue.translator.RuntimeFilterTranslatorBucketPruneTest,org.apache.doris.nereids.trees.expressions.functions.scalar.UniqueFunctionTest'`
(8 + 6 + 7 + 18).
- [x] Manual test: FE Checkstyle passed with zero violations; repository
check-format.sh with clang-format 16.0.6 and BE build hygiene passed.
Original patch comparison confirms identical added/removed lines for 23
files; ArrayShuffle adapts to the equivalent functionality now in
master.
- [ ] Regression test: BE tests and regression suites were not rerun
locally for this rebase; existing coverage is retained and run buildall
will verify integration.
- Behavior changed:
- [x] Yes. Unsafe scans bypass condition cache; deterministic scans
retain eligibility. No additional behavior change introduced by rebase.
- Does this need documentation?
- [x] No. Correctness fix using existing cache switches; upgrade
considerations are stated above.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.5-merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants