Skip to content

[test](fe) Stabilize scan termination and retry profile assertions - #69894

Merged
CalvinKirs merged 1 commit into
apache:masterfrom
924060929:codex/stabilize-fe-ut-lifecycle
Nov 2, 2026
Merged

CalvinKirs merged 1 commit into
apache:masterfrom
924060929:codex/stabilize-fe-ut-lifecycle

Conversation

@924060929

@924060929 924060929 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: N/A

Problem Summary:

Two FE unit tests depend on asynchronous work without owning its complete lifecycle:

  • SplitProducerTest.readThatOutlivesItsStatementLoadsNothingIntoTheClosedScope waits for the source-close signal and immediately checks the producer completion count. SplitProducer.finish invokes the completion callback after closing the source, so the assertion can run before that callback. Wait for the assignment termination future, which completes after both cleanup and the callback.
  • StmtExecutorTest.testSuccessfulRetryRendersMergedProfileAndActualRows registers its profile with the global background ProfileManager while also writing and reading the profile synchronously. Use a real manager whose daemon is not started, scoped to this test, so automatic spill, memory release and archival cannot interfere with the temporary storage lifecycle. Preserve the real persistence path and all merged-profile and actual-row assertions, and include the readback content in the failing assertion.

The tests originated in #69718 and #68658 respectively. These test-only changes are independent of the Coordinator refactoring in #69830. The exact CI interleaving was not reproduced locally; both original tests also passed local repeated execution. No production behavior changes.

Release note

None

Check List (For Author)

  • Test: On this independent master-based branch, ./run-fe-ut.sh --run 'SplitProducerTest,StmtExecutorTest' passed all 81 cases (26 + 55). Before extraction, both target cases also passed 50 repetitions each (179 executions across both classes); FE Checkstyle passed with 0 violations.
  • Behavior changed: No
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Problem Summary: FE UT reported a split producer completion assertion before its completion callback finished and a retry profile assertion after persistence. Wait for the scan assignment termination future before checking completion. Isolate the retry profile test with a real ProfileManager whose daemon is not started, so background spill and memory release cannot race the test's synchronous persistence and readback. Keep all merged-profile and actual-row checks.

### Release note

None

### Check List (For Author)

- Test: Both affected classes passed with each target test repeated 50 times, 179 executions total; FE Checkstyle passed. The original CI interleaving did not reproduce in the local baseline stress run. No production code changed.
- Behavior changed: No
- Does this need documentation: No
@924060929
924060929 requested a review from CalvinKirs as a code owner October 30, 2026 10:04
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR 69894 at head 384e24a found no new issues. The change is limited to two FE tests. In SplitProducerTest, the release latch precedes a bounded termination-future wait, which follows the producer's source-close and completion path and removes an assertion race around late connector reads. In StmtExecutorTest, the fresh unstarted ProfileManager is used inside a scoped static mock; all exercised profile and query-manager calls are synchronous on the test thread, while the nested finally unregisters both attempts and removes the profile before the mock closes. The storage checks remain synchronous and validate both merged in-memory and serialized output.

Critical checkpoints: the task goal is achieved by making cleanup and profile-storage assertions deterministic; the test edits are focused; the split cancellation path and profile retry path were traced through their upstream/downstream lifecycle and locking boundaries; no new configuration, protocol, persistence, or production concurrency path is introduced; no parallel path or error-handling defect was found; and the existing global/resource cleanup is preserved. The user focus file specified no additional focus.

Validation was static only as required by the review contract; no builds or tests were executed. Existing inline review threads and blocking IDs are empty. Three reviewers completed the required round-1 coverage and returned NO_NEW_VALUABLE_FINDINGS, with all initial risk items dismissed using concrete control-flow evidence. Review converged in one round.

Review converged after 1 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28832 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 384e24a30a4ed958cf24b53e183c1724535ff291, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17616	5192	5046	5046
q2	2232	316	258	258
q3	10165	1383	816	816
q4	4692	461	337	337
q5	7440	789	532	532
q6	184	181	151	151
q7	760	806	608	608
q8	9252	1363	1074	1074
q9	5472	4193	4201	4193
q10	6828	1305	1053	1053
q11	434	243	230	230
q12	644	398	285	285
q13	18126	3971	3047	3047
q14	272	274	244	244
q15	q16	762	710	649	649
q17	1238	773	602	602
q18	6988	6150	6126	6126
q19	1112	966	629	629
q20	403	336	232	232
q21	5636	2442	2519	2442
q22	380	330	278	278
Total cold run time: 100636 ms
Total hot run time: 28832 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5953	5909	6187	5909
q2	743	607	601	601
q3	5633	5819	5301	5301
q4	2050	2165	1570	1570
q5	5754	5854	5596	5596
q6	250	199	148	148
q7	2254	1986	1795	1795
q8	2980	2628	2739	2628
q9	7894	7478	7469	7469
q10	3654	3552	3195	3195
q11	598	406	374	374
q12	678	682	498	498
q13	3318	3641	3013	3013
q14	287	284	271	271
q15	q16	663	674	617	617
q17	7763	7178	6865	6865
q18	12913	12183	12841	12183
q19	907	855	798	798
q20	2138	2143	1908	1908
q21	5698	4619	4708	4619
q22	488	461	418	418
Total cold run time: 72616 ms
Total hot run time: 65776 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151643 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 384e24a30a4ed958cf24b53e183c1724535ff291, data reload: false

query5	4310	595	462	462
query6	455	214	211	211
query7	4813	442	231	231
query8	313	173	155	155
query9	8818	3229	3231	3229
query10	466	316	272	272
query11	6000	3836	3561	3561
query12	155	98	90	90
query13	1252	463	326	326
query14	6122	3945	3614	3614
query14_1	3445	3364	3472	3364
query15	213	202	184	184
query16	956	445	436	436
query17	1101	667	549	549
query18	2427	422	318	318
query19	216	170	137	137
query20	108	85	84	84
query21	217	131	107	107
query22	12847	13021	12696	12696
query23	13536	13038	12500	12500
query23_1	12484	12658	12656	12656
query24	7341	859	482	482
query24_1	502	500	492	492
query25	534	427	366	366
query26	1253	262	142	142
query27	2754	447	271	271
query28	4623	2040	2053	2040
query29	1588	610	473	473
query30	298	228	184	184
query31	1013	923	822	822
query32	160	102	94	94
query33	527	315	248	248
query34	945	863	522	522
query35	748	767	635	635
query36	800	816	745	745
query37	133	101	85	85
query38	1777	1709	1685	1685
query39	801	795	745	745
query39_1	725	746	738	738
query40	228	123	99	99
query41	75	68	68	68
query42	90	87	92	87
query43	353	381	307	307
query44	1447	690	712	690
query45	174	174	167	167
query46	847	978	572	572
query47	2805	2815	2808	2808
query48	303	322	220	220
query49	577	400	294	294
query50	716	279	216	216
query51	10748	10562	10376	10376
query52	78	82	68	68
query53	197	211	158	158
query54	240	194	176	176
query55	73	68	63	63
query56	229	211	232	211
query57	1668	1638	1566	1566
query58	282	265	252	252
query59	2067	2126	1973	1973
query60	281	241	224	224
query61	153	149	143	143
query62	387	328	303	303
query63	192	165	164	164
query64	2739	945	762	762
query65	3368	3265	3245	3245
query66	1778	420	302	302
query67	20272	20241	20088	20088
query68	2919	1024	639	639
query69	392	291	258	258
query70	835	795	792	792
query71	298	243	212	212
query72	2697	2566	2279	2279
query73	539	546	302	302
query74	4597	4491	4293	4293
query75	2336	2294	1914	1914
query76	2319	1014	618	618
query77	367	366	279	279
query78	9752	9735	9198	9198
query79	1033	885	560	560
query80	1260	430	370	370
query81	593	322	264	264
query82	714	143	107	107
query83	290	197	177	177
query84	265	124	102	102
query85	877	419	358	358
query86	393	238	233	233
query87	1999	1938	1814	1814
query88	3936	2814	2782	2782
query89	340	281	240	240
query90	1949	172	169	169
query91	163	155	123	123
query92	100	80	81	80
query93	1152	1033	590	590
query94	638	309	270	270
query95	582	396	305	305
query96	687	499	248	248
query97	2455	2411	2338	2338
query98	173	149	154	149
query99	774	763	643	643
Total cold run time: 236180 ms
Total hot run time: 151643 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 384e24a30a4ed958cf24b53e183c1724535ff291, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.05	0.05
query3	0.26	0.08	0.08
query4	1.61	0.10	0.10
query5	0.23	0.24	0.25
query6	1.16	0.68	0.69
query7	0.04	0.01	0.01
query8	0.05	0.04	0.03
query9	0.40	0.35	0.35
query10	0.57	0.57	0.58
query11	0.20	0.13	0.14
query12	0.17	0.14	0.15
query13	0.49	0.47	0.49
query14	1.02	1.01	1.00
query15	0.63	0.61	0.60
query16	0.31	0.35	0.35
query17	1.13	1.17	1.11
query18	0.19	0.19	0.19
query19	2.05	2.04	2.07
query20	0.02	0.01	0.01
query21	15.54	0.22	0.13
query22	4.81	0.06	0.05
query23	16.27	0.32	0.12
query24	3.12	0.54	0.40
query25	0.11	0.05	0.07
query26	0.85	0.22	0.16
query27	0.05	0.05	0.03
query28	2.93	0.78	0.38
query29	12.49	4.23	3.24
query30	0.29	0.13	0.10
query31	2.76	0.58	0.31
query32	3.23	0.61	0.50
query33	3.22	3.29	3.24
query34	15.31	4.10	3.39
query35	3.34	3.33	3.33
query36	0.53	0.44	0.42
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.05	0.04	0.03
Total cold run time: 96 s
Total hot run time: 24.35 s

@924060929

Copy link
Copy Markdown
Contributor Author

run cloud_p0

@924060929

Copy link
Copy Markdown
Contributor Author

run check_coverage

@CalvinKirs
CalvinKirs merged commit 25968c9 into apache:master Nov 2, 2026
43 of 44 checks passed
@924060929
924060929 deleted the codex/stabilize-fe-ut-lifecycle branch November 2, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants