Skip to content

[fix](be) Open children of Iceberg nested struct expressions - #69832

Merged
Gabriel39 merged 2 commits into
apache:masterfrom
hubgeter:fix/iceberg-equality-delete-expr-open
Nov 4, 2026
Merged

Gabriel39 merged 2 commits into
apache:masterfrom
hubgeter:fix/iceberg-equality-delete-expr-open

Conversation

@hubgeter

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: #69782

Related PR: #69711

Problem Summary:
NestedStructFieldExpr::open() reuses the prepare guard and skips opening its child. Use the normal open lifecycle, completing the nested traversal fix after #69711.

Release note

Fix initialization of nested struct expressions in FileScannerV2 Iceberg equality deletes.

Check List (For Author)

  • Test
    • Unit Test: added coverage for cast defaults, nullable ancestors, and cloned contexts.
  • Behavior changed:
    • Yes. Open nested expression children before execution.
  • Does this need documentation?
    • No.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: apache#69782

Related PR: apache#69711

Problem Summary: NestedStructFieldExpr reuses the prepare-state guard in
open(), so it returns without opening its child or marking itself ready.
Use the normal expression open lifecycle, completing the nested traversal
path alongside the AncestorNullDefaultExpr fix in apache#69711.

Add a focused unit test for struct traversal readiness, converted initial
defaults, independent ancestor NULL masks, and cloned contexts.

### Release note

Fix initialization of nested struct traversal expressions used by
FileScannerV2 Iceberg equality deletes.

### Check List (For Author)

- Test: clang-format v16, build hygiene, and git diff --check. Unit test added.
- Behavior changed: Yes. Nested struct expressions open their children
  before evaluation.
- Does this need documentation: No. Internal expression lifecycle repair.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter

Copy link
Copy Markdown
Contributor Author

run buildall

@hubgeter

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/40842294191

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28392 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ffdbc16fb405df37d0f657049190282a26640fc, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17588	5234	5037	5037
q2	2166	304	272	272
q3	10230	1455	831	831
q4	4678	468	341	341
q5	7451	811	532	532
q6	184	184	156	156
q7	731	803	602	602
q8	9248	1318	1049	1049
q9	5457	4184	4204	4184
q10	6803	1274	1035	1035
q11	436	245	223	223
q12	634	404	284	284
q13	18108	3666	2992	2992
q14	279	274	255	255
q15	q16	746	704	677	677
q17	1224	746	577	577
q18	7030	6196	6089	6089
q19	1107	954	600	600
q20	393	346	222	222
q21	4650	2173	2270	2173
q22	354	296	261	261
Total cold run time: 99497 ms
Total hot run time: 28392 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5625	5577	5568	5568
q2	718	559	542	542
q3	5186	5320	4952	4952
q4	1972	2151	1560	1560
q5	5246	5161	5500	5161
q6	263	211	158	158
q7	2395	2064	1920	1920
q8	3010	2717	2856	2717
q9	7935	8032	8125	8032
q10	3775	3669	3220	3220
q11	615	437	421	421
q12	718	674	519	519
q13	3577	3995	3278	3278
q14	319	319	283	283
q15	q16	686	713	656	656
q17	8270	7579	7250	7250
q18	12961	12158	12937	12158
q19	910	799	813	799
q20	2158	2117	1915	1915
q21	5649	4566	4647	4566
q22	478	441	421	421
Total cold run time: 72466 ms
Total hot run time: 66096 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151453 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ffdbc16fb405df37d0f657049190282a26640fc, data reload: false

query5	4302	611	460	460
query6	424	207	189	189
query7	4821	472	243	243
query8	326	177	155	155
query9	8781	3210	3207	3207
query10	472	316	259	259
query11	6012	3857	3573	3573
query12	144	96	87	87
query13	1248	447	343	343
query14	6118	3962	3669	3669
query14_1	3450	3431	3425	3425
query15	205	203	176	176
query16	935	428	404	404
query17	840	662	527	527
query18	2417	429	308	308
query19	191	164	130	130
query20	87	86	80	80
query21	201	128	105	105
query22	12888	12919	12931	12919
query23	13484	12842	12436	12436
query23_1	12482	12737	12513	12513
query24	7374	870	481	481
query24_1	478	485	472	472
query25	523	416	354	354
query26	1278	275	140	140
query27	2761	459	273	273
query28	4643	2069	2071	2069
query29	1533	613	478	478
query30	304	224	192	192
query31	1006	926	812	812
query32	158	98	101	98
query33	542	316	273	273
query34	1020	918	529	529
query35	744	761	650	650
query36	810	803	718	718
query37	129	101	83	83
query38	1786	1724	1666	1666
query39	795	800	760	760
query39_1	733	721	743	721
query40	224	124	103	103
query41	72	67	67	67
query42	88	90	87	87
query43	348	373	325	325
query44	1399	713	718	713
query45	180	181	176	176
query46	903	1027	605	605
query47	2865	2985	2686	2686
query48	307	327	231	231
query49	590	425	319	319
query50	717	291	212	212
query51	10326	10399	10306	10306
query52	90	88	75	75
query53	204	227	158	158
query54	257	218	188	188
query55	76	72	69	69
query56	239	262	212	212
query57	1645	1621	1604	1604
query58	300	270	267	267
query59	2115	2152	1990	1990
query60	309	267	241	241
query61	171	166	161	161
query62	400	345	290	290
query63	202	169	170	169
query64	2863	1075	781	781
query65	3327	3281	3244	3244
query66	1768	435	327	327
query67	20103	20037	19850	19850
query68	2985	1027	620	620
query69	389	289	244	244
query70	863	768	782	768
query71	305	234	218	218
query72	2729	2525	2077	2077
query73	533	552	301	301
query74	4624	4525	4351	4351
query75	2459	2347	1972	1972
query76	2306	1117	642	642
query77	353	399	292	292
query78	9828	9764	9214	9214
query79	1041	935	541	541
query80	1145	460	340	340
query81	588	323	285	285
query82	791	136	108	108
query83	306	205	197	197
query84	311	115	106	106
query85	890	455	383	383
query86	373	228	230	228
query87	1975	1936	1819	1819
query88	3841	2778	2773	2773
query89	358	304	258	258
query90	1821	173	178	173
query91	168	157	133	133
query92	102	88	91	88
query93	1043	1018	579	579
query94	578	307	264	264
query95	604	415	318	318
query96	688	503	244	244
query97	2484	2434	2306	2306
query98	170	153	150	150
query99	782	759	644	644
Total cold run time: 235695 ms
Total hot run time: 151453 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ffdbc16fb405df37d0f657049190282a26640fc, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.25	0.09	0.10
query4	1.61	0.10	0.11
query5	0.24	0.25	0.25
query6	1.17	0.68	0.66
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.41	0.36	0.36
query10	0.60	0.60	0.59
query11	0.20	0.15	0.14
query12	0.19	0.16	0.15
query13	0.49	0.49	0.48
query14	1.02	1.02	1.01
query15	0.62	0.61	0.62
query16	0.35	0.34	0.32
query17	1.17	1.18	1.16
query18	0.20	0.18	0.18
query19	2.09	1.95	2.01
query20	0.02	0.01	0.01
query21	15.44	0.20	0.13
query22	4.89	0.06	0.06
query23	16.24	0.32	0.12
query24	3.01	0.52	0.52
query25	0.09	0.07	0.05
query26	1.44	0.20	0.15
query27	0.04	0.05	0.05
query28	3.98	0.77	0.35
query29	12.54	4.17	3.23
query30	0.28	0.13	0.11
query31	2.77	0.57	0.31
query32	3.22	0.59	0.50
query33	3.20	3.20	3.30
query34	15.31	3.97	3.43
query35	3.31	3.35	3.34
query36	0.55	0.45	0.43
query37	0.09	0.07	0.07
query38	0.06	0.05	0.04
query39	0.04	0.03	0.02
query40	0.18	0.15	0.14
query41	0.08	0.03	0.03
query42	0.03	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.66 s
Total hot run time: 24.51 s

@hubgeter
hubgeter marked this pull request as ready for review November 2, 2026 06:40
@hubgeter

hubgeter commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

/review

Gabriel39
Gabriel39 previously approved these changes Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41408586882

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

### What problem does this PR solve?

Issue Number: apache#69782

Related PR: apache#69832

Problem Summary: The nested equality-delete expression test calls get_int()
on ColumnNullable, which throws instead of reading the BIGINT value. Reuse
the existing nullable BIGINT assertion helper to check NULL, NULL, and 7
for both the original and cloned expression contexts.

### Release note

None

### Check List (For Author)

- Test: clang-format v16 and git diff --check; existing unit test updated.
- Behavior changed: No. Test assertion only.
- Does this need documentation: No.
@hubgeter

hubgeter commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hubgeter

hubgeter commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41601994051

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 67.25% (32084/47707)
Line Coverage 52.10% (337563/647900)
Region Coverage 47.65% (274548/576161)
Branch Coverage 49.51% (129497/261582)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28604 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e06e81742abf72492afcbb727785c4c7653bf1ab, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17624	5122	5067	5067
q2	2143	299	259	259
q3	10248	1401	818	818
q4	4682	464	331	331
q5	7431	798	529	529
q6	192	186	150	150
q7	762	791	603	603
q8	9262	1323	1047	1047
q9	5555	4220	4190	4190
q10	6816	1285	1028	1028
q11	427	247	225	225
q12	635	405	294	294
q13	18083	3765	3054	3054
q14	275	274	253	253
q15	q16	738	709	654	654
q17	1208	758	591	591
q18	6961	6183	6163	6163
q19	1238	973	593	593
q20	401	342	227	227
q21	5573	2277	2352	2277
q22	350	305	251	251
Total cold run time: 100604 ms
Total hot run time: 28604 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5632	5630	5642	5630
q2	676	586	522	522
q3	5185	5375	4992	4992
q4	2006	2141	1577	1577
q5	5280	5139	5778	5139
q6	253	199	159	159
q7	2318	2116	1910	1910
q8	3111	2929	2690	2690
q9	8136	8009	8000	8000
q10	3751	3637	3248	3248
q11	622	470	410	410
q12	679	696	523	523
q13	3593	4096	3309	3309
q14	305	319	288	288
q15	q16	695	713	664	664
q17	8470	7647	7090	7090
q18	12928	12140	12984	12140
q19	911	789	801	789
q20	2150	2150	1919	1919
q21	5847	4900	5034	4900
q22	500	475	406	406
Total cold run time: 73048 ms
Total hot run time: 66305 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150978 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e06e81742abf72492afcbb727785c4c7653bf1ab, data reload: false

query5	4313	584	460	460
query6	431	207	192	192
query7	4787	459	225	225
query8	313	164	149	149
query9	8793	3151	3183	3151
query10	476	304	253	253
query11	6067	3863	3559	3559
query12	148	94	87	87
query13	1249	464	345	345
query14	6143	3926	3706	3706
query14_1	3394	3420	3384	3384
query15	211	203	201	201
query16	940	457	383	383
query17	824	627	496	496
query18	2412	413	299	299
query19	187	160	126	126
query20	81	83	79	79
query21	211	124	104	104
query22	13031	12949	12709	12709
query23	13500	12855	12461	12461
query23_1	12500	12521	12513	12513
query24	7354	854	470	470
query24_1	473	470	478	470
query25	509	390	338	338
query26	1258	259	144	144
query27	2768	439	266	266
query28	4658	2022	2054	2022
query29	1550	592	459	459
query30	304	226	187	187
query31	991	915	821	821
query32	141	96	89	89
query33	512	291	227	227
query34	1001	861	510	510
query35	726	739	656	656
query36	808	806	729	729
query37	124	99	85	85
query38	1781	1731	1678	1678
query39	804	796	761	761
query39_1	727	730	748	730
query40	220	116	97	97
query41	73	70	71	70
query42	91	86	81	81
query43	351	362	313	313
query44	1390	708	716	708
query45	184	180	165	165
query46	846	970	574	574
query47	2485	2882	2748	2748
query48	304	319	226	226
query49	586	407	309	309
query50	685	275	223	223
query51	10559	10462	10209	10209
query52	80	86	72	72
query53	185	213	154	154
query54	250	208	191	191
query55	76	70	67	67
query56	255	245	232	232
query57	1657	1625	1610	1610
query58	295	272	266	266
query59	2167	2214	1962	1962
query60	299	258	238	238
query61	168	165	160	160
query62	394	334	282	282
query63	196	159	170	159
query64	2857	1054	823	823
query65	3344	3290	3281	3281
query66	1779	418	303	303
query67	20243	20291	20012	20012
query68	3173	1008	622	622
query69	384	292	254	254
query70	849	772	758	758
query71	287	228	209	209
query72	2878	2645	2274	2274
query73	511	532	295	295
query74	4636	4501	4368	4368
query75	2362	2282	1920	1920
query76	2349	1050	611	611
query77	342	386	272	272
query78	9746	9668	9113	9113
query79	1023	882	529	529
query80	1218	436	334	334
query81	590	310	265	265
query82	583	133	104	104
query83	305	194	181	181
query84	313	127	105	105
query85	838	422	374	374
query86	403	245	227	227
query87	1981	1938	1818	1818
query88	3824	2772	2739	2739
query89	337	285	244	244
query90	1914	169	165	165
query91	159	147	123	123
query92	101	89	85	85
query93	1075	1016	591	591
query94	623	300	272	272
query95	555	392	310	310
query96	695	500	245	245
query97	2451	2428	2315	2315
query98	157	152	156	152
query99	772	757	650	650
Total cold run time: 235647 ms
Total hot run time: 150978 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e06e81742abf72492afcbb727785c4c7653bf1ab, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.10	0.10
query5	0.24	0.25	0.24
query6	1.17	0.69	0.67
query7	0.04	0.01	0.01
query8	0.06	0.04	0.03
query9	0.41	0.35	0.36
query10	0.56	0.58	0.59
query11	0.20	0.14	0.14
query12	0.18	0.15	0.15
query13	0.52	0.51	0.50
query14	1.03	1.01	1.01
query15	0.63	0.60	0.62
query16	0.36	0.35	0.36
query17	1.18	1.15	1.12
query18	0.21	0.20	0.20
query19	2.02	2.01	1.98
query20	0.02	0.02	0.02
query21	15.42	0.20	0.13
query22	4.81	0.06	0.05
query23	16.30	0.31	0.12
query24	2.89	0.69	0.30
query25	0.12	0.05	0.06
query26	1.28	0.23	0.15
query27	0.05	0.03	0.03
query28	3.42	0.77	0.38
query29	12.49	4.15	3.24
query30	0.29	0.14	0.10
query31	2.78	0.58	0.34
query32	3.22	0.60	0.51
query33	3.18	3.17	3.18
query34	15.34	3.96	3.37
query35	3.36	3.34	3.32
query36	0.55	0.44	0.45
query37	0.09	0.07	0.06
query38	0.04	0.05	0.04
query39	0.04	0.03	0.02
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.81 s
Total hot run time: 24.24 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.76% (35940/46220)
Line Coverage 63.12% (404467/640833)
Region Coverage 59.73% (342563/573474)
Branch Coverage 60.89% (158701/260642)

@hubgeter

hubgeter commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69832
commit: e06e81742abf72492afcbb727785c4c7653bf1ab
base: fc1b82283cbbab0ef4e23a8d9249dc3c83d71411
reviewed_at: 2026-11-04T10:59+08:00
reviewer: hubgeter
model: gpt-6.1-sol
effort: max
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • be/test/format_v2/table/iceberg_reader_test.cpp:4581 — Static source review only; no build, unit/regression test, or cluster run was performed.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@Gabriel39
Gabriel39 merged commit 58d7b81 into apache:master Nov 4, 2026
37 of 38 checks passed
github-actions Bot pushed a commit that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Issue Number: #69782

Related PR: #69711

Problem Summary:
`NestedStructFieldExpr::open()` reuses the prepare guard and skips
opening its child. Use the normal open lifecycle, completing the nested
traversal fix after #69711.

### Release note

Fix initialization of nested struct expressions in FileScannerV2 Iceberg
equality deletes.

### Check List (For Author)

- Test
- [x] Unit Test: added coverage for cast defaults, nullable ancestors,
and cloned contexts.
- Behavior changed:
    - [x] Yes. Open nested expression children before execution.
- Does this need documentation?
    - [x] No.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants