Skip to content

[improvement](stream) Collect changelog load rows in both storage modes - #69896

Open
yujun777 wants to merge 7 commits into
apache:masterfrom
yujun777:ivm-changelog-load-rows
Open

yujun777 wants to merge 7 commits into
apache:masterfrom
yujun777:ivm-changelog-load-rows

Conversation

@yujun777

@yujun777 yujun777 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Background

Changelog indexes retain physical change records for table streams. Their row-count estimates come from periodic tablet statistics, independently of base-table load statistics. Updates and deletes can produce changelog records whose counts differ from the base-table load rows.

What problem does this PR solve?

Issue Number: None
Related PR: #69850 (persists consumption baselines with offsets and estimates stream ranges)
PR Type: Enhancement

Problem Summary:

Changelog row counts can remain stale or unknown between tablet reports, and cloud commit responses previously carried no partition/index-level changelog delta. Successful visible commits now supplement these estimates in both storage modes, while valid periodic reports calibrate retained stock. Most of the diff tests publication, retries, incomplete reports and recovery; no new configuration is required after upgrading.

Details

  • Change map: production code +313/-41, tests +1108/-1, and generated test output +148/-0; no documentation or generated protocol source. Production changes have three parts: final published rowset counting and partition replica normalization in SN; optional MS response aggregation and version-based deduplication in cloud; valid periodic stock publication, locking and restored-replica polling. Tests cover those paths and physical counts.
  • Before and after: Counts previously waited for periodic reports. Commits now add physical changelog rows: the regression's append/update/delete sequence reports 3, 5 and 6. Zero deltas preserve unknown counts; positive partial estimates do not imply a complete report. Periodic reports can lower retained counts after TTL cleanup, and restored cloud replicas fetch valid statistics before resuming persisted polling backoff.
  • Configs and defaults: None. No new runtime log statements are added.
  • Impact on existing users: Existing SQL and data visibility are unchanged. Optional protobuf fields preserve compatibility when either side lacks the new statistics. Counts remain estimates: periodic reports and deltas can overlap or lag, and failed publication can miss an immediate delta until the next report.

Testing

  • 122 focused FE tests passed locally across runs, covering index counts, publication, retries and restored-cloud polling. The recovery scheduling test failed before the fix and passed afterwards.

  • One ASAN BE test passed with real concurrent MoW publication and retry; five ASAN MS RPC tests passed for ordinary, lazy and subtransaction commits, including failures and base/changelog separation.

  • Shared-nothing changelog_p0: 27 suites passed before the final logging cleanup and cloud test additions; the immediate-count suite passed again after its cloud-compatible FORCE cleanup change.

  • Local ASAN cloud cluster (1 FE, 1 BE, MS, Recycler and FDB): immediate-count and subtransaction suites passed with periodic publication paused; forcing lazy commits passed both suites against the same golden files, with existing MS logs confirming commit_txn_eventually. The full changelog_p0 directory passed 24/28 suites. Four existing suites failed on row contents: test_changelog_before_column_name, test_cloud_changelog_publish_conflict, test_changelog_multi_segment, and test_changelog_basic; all four failed again in an isolated batch, and their expected results were not changed.

  • With periodic reports paused, the extended immediate-count suite passed in SN and cloud: two successive loads and stream consumptions grow retained changelog counts from 2 to 3 to 4; consumption leaves current counts unchanged, and each consumed interval becomes empty. The new golden sections were generated by the runner.

Notes for Reviewers

  • Current index counts are independent of each stream's consumption baseline. This PR updates C1; [fix](stream) Estimate stream rows from consumption baselines #69850 snapshots B0/C0 together with the consumed offset and uses stock fallback for nonempty ranges whose deltas are stale.
  • Counts are physical records; UPDATE records are not multiplied by two. Commit statistics are separate from base updatedRows.
  • Tablet identity checks prevent delayed responses from updating replaced data. DROP STATS preserves the estimate; data replacement resets it. Complete periodic reports retire covered deduplication versions.

Release note

Successful loads supplement changelog row-count estimates before the next periodic tablet report in both storage modes. Periodic statistics continue to calibrate retained counts.

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: No
  • Pre-submission status: Partially verified: the four cloud row-content failures listed above remain; the new row-count suites pass in ordinary, lazy and subtransaction paths.

AI Disclosure

  • Tool: Codex; gpt-6.1-sol/xhigh for independent reviews
  • Generated: Code, tests, and this description
  • Human review: not stated

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@yujun777

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete four-file diff at 1cb6c79. Two P2 findings need correction: normal grouped changelog loads never report their attached changelog rowsets, and the new Java import violates the mandatory FE Checkstyle gate. Both are posted inline. There were no existing inline findings to carry forward and no additional user-provided focus points.

The review converged in 2 rounds: all three independent reviewers returned NO_NEW_VALUABLE_FINDINGS in the latest round, every changed file and assigned mechanism was covered, and all candidates and initial risks were verified, deduplicated or dismissed with source evidence.

Critical checkpoint conclusions:

  1. Goal and proof: separate base/changelog load counters are the intended behavior. The FE accumulator works for directly supplied values, but the BE loop sees base transaction rowsets while the changelog resides in attach_changelog; therefore ordinary loads do not populate the new counter. The added test bypasses this producer and does not prove the complete behavior.
  2. Scope and reuse: the four-file change is focused and reuses the existing tablet delta report and statistics serialization.
  3. Concurrency and locking: the publish parent owns the report map and joins its workers; existing changelog-first tablet locks are unchanged. All accesses and lazy initialization of the new map take the TableStatsMeta monitor. No added IO, journal waits, or lock-order cycle was found. A repair must read the final attachment count after workers finish because MoW publication can append changelog rows.
  4. Lifecycle and initialization: TabletTxnInfo retains attachment ownership through publication and recovery. The new map belongs to the statistics record, survives ordinary analyze updates, and is discarded with that record. No new static initialization or reference cycle is introduced.
  5. Configuration: no configuration is added.
  6. Compatibility: the optional Gson field is initialized lazily when absent, following existing concurrent-map serialization patterns. The existing report field is reused; no new protocol shape or toolchain/library dependency is introduced. Older producers do not supply changelog history.
  7. Thrift/protobuf IDs: no definition is changed, so no field-ID or required-field compatibility change applies.
  8. Parallel paths: normal, retry, asynchronous and cloud reporting paths were traced. Existing replica averaging/missing-report approximations remain. Cloud/legacy events still report base table scalars and do not populate the new index counter; this is a feature limitation, with no separate current consumer failure established.
  9. Conditional logic: comments explain why changelog rows differ from base rows, but the BE condition tests the wrong transaction representation (first inline finding). Zero additions intentionally leave the map unchanged. Missing-tablet/checkpoint concerns were checked against actual index rebuild and replay registration, without another reachable regression.
  10. Test coverage: the new deterministic JUnit case covers additive map behavior, zero additions and base isolation. It adds no publish/report integration, routing, replay or serialization coverage. Existing attachment recovery tests were inspected. No builds, tests, Checkstyle execution, benchmarks or runtime validation were performed under the static-only contract.
  11. Expected results: the added direct-map assertions match their inputs; no regression output artifact is changed.
  12. Observability: existing tablet/transaction identifiers and report logging remain. No separate logging or metrics defect was substantiated.
  13. Persistence and failover: transaction delta maps already persist with VISIBLE state and use the same statistics application on replay; image/CREATE/TRUNCATE paths register changelog tablet metadata. The new map participates in statistics JSON/image persistence. No additional replay divergence was verified.
  14. Writes and atomicity: the patch changes statistics reporting and estimates, without changing rowset visibility, version selection, delete bitmap publication or committed data writes. No new data-loss or memory-safety issue was identified by source inspection.
  15. FE/BE passing: the existing callback and aggregation chain accepts tablet-keyed counts, but the attached changelog count is missing at its producer (first finding). No other new variable needs propagation.
  16. Performance: the new FE classification adds one inverted-index lookup per reported tablet on changelog tables and bounded synchronized map work. No additional current-head performance regression was substantiated.
  17. Other issues: the import-order finding is supported by error-severity CustomImportOrder, the Maven validate binding and the FE Checkstyle workflow. A suspected stream-wrapper null dereference was dismissed because the sole current consumer cannot reach that fallback with the wrapper's empty local partition map.

Coverage limits: this is a static review. The lifetime accumulator alone is not an exact retained-row or per-stream unconsumed-row count after TTL, statistics recreation or upgrade. The stream-estimation consumer described in dependent PR #69850 is outside this diff, so no end-to-end stream-plan improvement or failure is claimed here. There are no unresolved candidates or unread assigned source paths.

Review converged after 2 round(s).

auto tablet_id = rowset_meta_ptr->tablet_id();
if (_publish_version_req.base_tablet_ids.contains(tablet_id)) {
if (_publish_version_req.base_tablet_ids.contains(tablet_id)
|| tablet->is_changelog_tablet()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Report the attached changelog rowset for grouped loads

Normal changelog loads commit only the base rowset into TxnManager: GroupRowsetBuilder::commit_txn stores the companion in txn_info_it->second->attach_changelog.rowset, and ChangelogRowsetBuilder::commit_txn only marks it committed. get_txn_related_tablets therefore exposes the base tablet to this loop, so the new is_changelog_tablet() branch never inserts the companion tablet ID or its rows. Those loads leave the new FE changelog counter at zero. Report the attached rowset under its own tablet ID after publish workers finish (MoW publication can append changelog segments), and cover the emitted report map in a publish-path test; the added direct-accumulator test cannot detect this omission.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5bb280e. You are right that the changelog rowset is attached to the base tablet's transaction info rather than committed as a tablet of its own, so the is_changelog_tablet() branch never saw it; it is gone now. The publish path collects the attached rowset of each base tablet and reports it under its own tablet id after token->wait(), so a rowset that a merge-on-write publication extends with changelog segments is counted with its final row count.

import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Table;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.TabletMeta;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Place TabletMeta after Tablet to pass FE Checkstyle

The new import precedes Tablet, but fe/check/checkstyle/checkstyle.xml enables alphabetical CustomImportOrder at error severity. fe/pom.xml binds the failing check to validate, and the FE Checkstyle workflow also runs it, with no applicable suppression. Move TabletMeta below Tablet so the required validation gate accepts this change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5bb280e: TabletMeta now follows Tablet in the import block.

@yujun777
yujun777 force-pushed the ivm-changelog-load-rows branch from 1cb6c79 to 5bb280e Compare October 30, 2026 13:15
@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

1 similar comment
@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@yujun777

Copy link
Copy Markdown
Contributor Author

/review

@yujun777 yujun777 changed the title [improvement](ivm) Record the rows a load writes to a changelog index [improvement](stream) Record the rows a load writes to a changelog index Oct 30, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete four-file diff at exact PR head 5bb280e against base 374d87d. The intended goal is to carry the rows written to the attached changelog index into FE statistics so changelog scans can use a useful row estimate. The accumulator and its direct unit test cover the storage side, but two end-to-end paths still lose that information.

Critical checkpoint conclusions:

  1. Goal and proof: the FE accumulator correctly keeps changelog rows separate from base rows for a direct tablet-record event, but a partial BE publish retry drops already-published tablets from the final report, and cloud commits never populate the accumulator. The added unit test does not exercise either producer path.
  2. Scope and reuse: the changes are focused and reuse the existing tablet delta report, transaction publication, and TableStatsMeta persistence mechanisms.
  3. Concurrency and locking: BE publication tasks run concurrently and are joined by token->wait() before the rowset count is read; existing changelog-first tablet lock order is unchanged. TableStatsMeta additions and reads are synchronized. The retry lifecycle still loses a successful tablet's report.
  4. Lifecycle: TxnManager removes a successful tablet's txn entry during publish. The worker clears its delta map before retrying, so a later successful retry cannot rediscover that tablet's attached changelog rowset.
  5. Configuration: no configuration is added.
  6. Compatibility: the new Gson field is optional and lazily initialized, with no thrift/protobuf or wire-shape change.
  7. Parallel paths: local BE tablet-record reporting is updated, but the cloud commit path still reports only table-level base-index rows. Asynchronous tablet-stat refresh may eventually fill the gap, but the immediate post-load estimate remains zero.
  8. Conditional logic and data correctness: changelog rows are intentionally excluded from base-table updatedRows and partition base-index accounting. The two findings are statistics undercounting issues; rowset visibility and delete-bitmap publication are otherwise unchanged.
  9. Persistence and failover: the existing transaction delta map is persisted with visible transaction state and replayed through AnalysisManager; the new per-index map participates in TableStatsMeta JSON persistence. No separate replay or locking defect was verified.
  10. Testing and observability: the direct JUnit test covers additive accumulation, zero additions, and base-index isolation. No publish-path, retry, cloud, replay, or serialization integration test was added. Under the static-only contract, no build, test, Checkstyle, benchmark, or runtime validation was run. Existing identifiers and report logging remain sufficient for these paths.
  11. Performance: the new local classification adds one inverted-index lookup per reported tablet and bounded synchronized map work; no additional current-head performance regression was substantiated.

Existing inline threads 4422920699 and 4422920705 were verified as fixed on this head and are not reposted. There is no additional user-provided focus.

The review reached the round limit after the required final sweep. All changed files and main producer/consumer paths were statically covered and the two candidates below were independently verified, but the latest delegated sweeps were interrupted by the temporary model-capacity error, so the contract's all-subagent NO_NEW_VALUABLE_FINDINGS convergence condition was not evidenced.

Review did not converge after 3 rounds. This automated review cannot pass the review check.

The final static sweep covered all four changed files and verified both findings, but the required latest-round delegated NO_NEW_VALUABLE_FINDINGS evidence was unavailable because both round-2 child sweeps hit the temporary model-capacity error. No unresolved code candidate remains; convergence is false solely because the delegated convergence evidence could not be obtained before the three-round cap.

// derive them from the rows it loaded into the base tablets -- an update writes change
// rows without adding a base row, and a delete adds none to the base but at least one
// to the changelog -- and it keeps the count to estimate how many rows a read of the
// changelog returns.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve attached changelog rows across publish retries

If one tablet publishes successfully and a sibling tablet fails, this vector is local to the failed execute attempt. TxnManager::publish_txn removes the successful tablet's txn info, PublishVersionWorkerPool clears table_id_to_tablet_id_to_num_delta_rows before retrying, and PublishVersionDaemon discards reports from tasks with any error tablet. The final successful retry therefore contains only the previously failed tablet, so the already-published tablet's changelog rows are never added to TableStatsMeta. Preserve successful tablets' changelog counts across retries or report them from the final aggregation so a partial publish cannot permanently undercount the changelog index.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retry path here has the same guarantee as the base index delta rows, and I kept it that way rather than diverging:

  • Within one execute() the retry loop is inside the function and the collected changelog rowsets are declared before it, so a retry of the same attempt keeps the rowset of a tablet that already published (the rowset object stays valid, and its row count is the final one).
  • Across attempts the report is discarded as a whole: PublishVersionDaemon.calculateTaskUpdateRows skips a task that reports any error tablet, for the base delta rows as well as for these. So a tablet that published in a failed attempt contributes its delta with the next report, which is the periodic tablet report and includes the changelog index. That is the existing behaviour of the base index counter, and changing it means aggregating per-tablet reports across attempts on the frontend, which I would rather do for both counters in one change than only for the changelog one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 9fb10c4. The FE now applies changelog deltas directly to the partition's MaterializedIndex row-count estimate; the TableStatsMeta lifetime accumulator is gone. A later tablet-stat report refreshes the physical count, rather than repairing a cumulative counter. My earlier reply conflated those two mechanisms.

Cross-attempt publish reporting can still omit a tablet that succeeded in an earlier failed attempt, just as for base deltas. This remains a documented limitation; preserving deltas across attempts should cover both paths in a follow-up.

@@ -1525,6 +1535,7 @@ public void replayUpdateRowsRecord(UpdateRowsEvent event) {
+ "rows for all replica {}, tablets count {}",
olapTable.getName(), tableUpdateRows, replicaNum, rowsForAllReplica, tabletRows.size());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Feed cloud changelog writes into the per-index counter

The new counter is updated only for local tabletRecords. Cloud commits call updateUpdatedRows(updatedRows, ...) with a table-level value from meta-service calc_table_stats, and that helper sums only base_tablet_ids, so attached changelog rowsets are never added. Immediately after a cloud load, before the asynchronous tablet-stat refresh reports the changelog index, getRowCountWithDeltaRows therefore returns zero for a changelog scan even though rows are visible. Extend the cloud commit/statistics path to report changelog rows per index (or carry them in the existing response) before relying on this accumulator.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and it is recorded as a known limitation in the description: a cloud commit reports rows per table from the meta-service calc_table_stats, which sums only base_tablet_ids, so there is no per-index value to route. Until that path carries the changelog rows, a cloud stream read keeps falling back to the row count of the table, which is the behaviour before this PR — safe, just loose. Feeding the cloud counter needs the meta-service side to sum the attached changelog rowsets as well; I would keep it in a follow-up rather than growing this one into the cloud txn path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cloud limitation remains explicit in the updated description. Cloud commits do not provide per-index changelog deltas, so this PR only updates shared-nothing publish reporting. Cloud changelog counts still come from asynchronous tablet statistics.

In 9fb10c4 the lifetime accumulator and its zero-valued fallback were removed; local deltas now update the partition index read by P/Q. My earlier description of the cloud fallback as safe was too strong: stale changelog/base counts do not guarantee an upper bound. Extending the cloud commit response remains a separate change.

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@yujun777

Copy link
Copy Markdown
Contributor Author

/review

@yujun777
yujun777 force-pushed the ivm-changelog-load-rows branch from 5bb280e to fecb190 Compare October 30, 2026 14:17
@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@yujun777

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28398 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fecb190d2b353d80368c0d37610cddbf6ef1569d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17589	5128	5032	5032
q2	2132	312	260	260
q3	10230	1382	816	816
q4	4682	459	335	335
q5	7438	797	515	515
q6	181	179	153	153
q7	772	814	599	599
q8	9253	1312	1116	1116
q9	5500	4210	4150	4150
q10	6810	1279	1020	1020
q11	433	258	228	228
q12	635	398	286	286
q13	18067	3629	2971	2971
q14	272	270	245	245
q15	q16	709	703	648	648
q17	1359	745	587	587
q18	6954	6172	6110	6110
q19	1099	959	643	643
q20	385	342	232	232
q21	5004	2189	2303	2189
q22	348	305	263	263
Total cold run time: 99852 ms
Total hot run time: 28398 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5607	5592	5620	5592
q2	709	571	535	535
q3	5191	5372	4954	4954
q4	1996	2173	1562	1562
q5	5255	5167	5572	5167
q6	259	200	157	157
q7	2331	2022	1799	1799
q8	3048	2698	2759	2698
q9	7941	8024	7972	7972
q10	3720	3928	3272	3272
q11	604	441	413	413
q12	720	707	539	539
q13	3525	3917	3309	3309
q14	338	326	284	284
q15	q16	724	736	663	663
q17	8341	7081	6877	6877
q18	13024	12149	12883	12149
q19	910	778	785	778
q20	2180	2143	1928	1928
q21	5750	4783	4786	4783
q22	495	456	406	406
Total cold run time: 72668 ms
Total hot run time: 65837 ms

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete four-file diff at exact PR head fecb190 against base 374d87d.

The PR's goal is to carry rows written to attached changelog rowsets into the FE per-index statistics accumulator. The grouped-load producer fix is present, and the direct accumulator test is correct, but three lifecycle paths still make the new count incorrect:

  1. TableStatsMeta(OlapTable) creates a zero changelog baseline. After DROP STATS (or another record recreation), existing changelog rowsets are still readable while the physical changelog index count is unknown. StatsCalculator then falls back to getRowCountWithDeltaRows, whose new branch returns only post-creation indexUpdatedRows, underestimating the visible changelog until a later tablet report.
  2. Partial TRUNCATE PARTITION replaces changelog tablets but retains the table statistics record. The asynchronous dropStats path invalidates column/partition statistics without clearing indexUpdatedRows, so the branch continues returning rows from the removed partition.
  3. A load whose partition is force-dropped before visibility can still carry old tablet delta rows. DatabaseTransactionMgr removes the missing partition from commit info, but still forwards the transaction delta map; after the old tablet metadata is removed, isChangelogTablet returns false and this code routes those changelog rows into the base updatedRows counter permanently.

Critical checkpoint conclusions:

  • Goal and proof: the normal attached-rowset path now reports the final row count after token->wait(), and FE routes direct tablet records by index. The new unit test covers additive accumulation and base-index isolation, but it does not cover record recreation, partition replacement, missing metadata, or the publish/report integration.
  • Scope and reuse: the four-file change is focused and reuses the existing tablet delta map and TableStatsMeta JSON persistence. The three findings above are in the new counter's lifecycle and consumer boundary.
  • Concurrency and locking: BE publication workers are joined before the attached rowset metadata is read; existing changelog-first tablet lock order is unchanged. The FE counter methods synchronize on TableStatsMeta, and no new deadlock or blocking I/O was found. Existing publish-retry aggregation remains a known P2 thread and is not duplicated here.
  • Lifecycle and initialization: the new accumulator has no partial-truncate reset and no pre-existing-row baseline, and the missing-metadata branch silently chooses the wrong index. No static initialization or ownership cycle was introduced.
  • Configuration, Thrift/protobuf, ABI, and protocol compatibility: no configuration, generated definition, field ID, wire shape, or toolchain change is present.
  • Parallel paths: local tablet-record reporting is updated. Cloud table-level reporting still omits changelog rows, and publish retries can lose successful siblings; both are already reported by existing inline P2 comments 4424460818 and 4424460813 and are not reposted. The grouped-load omission and import-order comments are fixed on this head.
  • Error handling and data correctness: rowset visibility, version selection, delete bitmaps, and committed data writes are unchanged. The new failures affect statistics/cardinality accounting; unknown tablet metadata must not be silently treated as a base tablet.
  • Persistence/failover: the optional Gson field participates in normal serialization, but stale or zero lifecycle state is what causes the findings above. Transaction delta maps are still applied through the existing visible/replay path.
  • Testing and observability: only the added deterministic JUnit behavior is represented in the diff; no publish-path, replay, truncate, serialization, or cloud integration test was added. Under the static-only contract, I did not run builds, tests, Checkstyle, benchmarks, or runtime validation. Existing transaction/tablet identifiers and logging are sufficient for the reviewed paths.
  • Performance: the new classification performs one inverted-index lookup per reported tablet and bounded synchronized map work; no separate current-head performance regression was substantiated.
  • User focus: no additional user-provided focus points were supplied.

The review converged after 2 rounds. All three assigned round-2 rechecks returned NO_NEW_VALUABLE_FINDINGS; all changed files and assigned mechanisms were covered, and every candidate was verified, deduplicated, or dismissed. The review remains static-only, so runtime ordering of asynchronous reports is a stated coverage limit. Existing P2 threads 4424460813 and 4424460818 remain applicable to this head but are not duplicated. No existing P0/P1 inline finding applies, so existing_blocking_comment_ids is empty.

Review converged after 2 round(s).

public synchronized long getRowCountWithDeltaRows(OlapTable table, long indexId) {
if (isChangelogIndex(table, indexId)) {
// The changelog index has no collected row count: the rows a load wrote to it are the
// only count this record keeps, and they are added rather than taken as a lower bound

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the existing changelog baseline when recreating table statistics. After DROP STATS (or another removal/recreation of TableStatsMeta), new TableStatsMeta(OlapTable) initializes no changelog row count. While the changelog tablet's FE row count is still unknown, StatsCalculator falls back to this branch, which returns only indexUpdatedRows from loads after the record was created. Existing changelog rowsets therefore remain visible but are estimated as zero/partial until a separate tablet report supplies a count. Preserve an existing changelog baseline or keep the fallback unknown instead of converting a missing baseline to zero.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9fb10c4 by removing the TableStatsMeta changelog accumulator and its row-count fallback. Published deltas now update the partition's changelog MaterializedIndex directly, preserving its existing row-count estimate independently of DROP STATS or statistics-record recreation. An unavailable index count starts from the reported load delta as a best-effort estimate; it is not claimed to contain unknown historical rows.

The new AnalysisManagerTest case verifies that removing/recreating TableStatsMeta leaves the public changelog row-count lookup unchanged.

return;
}
indexUpdatedRows().merge(indexId, rows, Long::sum);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Reset changelog deltas when a partition is truncated. Partial TRUNCATE PARTITION replaces the partition and its changelog tablets but keeps the table's TableStatsMeta; the subsequent dropStats invalidates column/partition statistics without clearing indexUpdatedRows. A later changelog row-count fallback therefore returns rows from the deleted partition as if they were still present. Clear or partition-scope this accumulator as part of the truncate/replay lifecycle so the estimate cannot retain removed rows.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9fb10c4. Changelog load deltas now update each partition's own MaterializedIndex, rather than an index-wide lifetime accumulator in TableStatsMeta. Partial TRUNCATE replaces the affected partition/index and therefore discards its old estimate while retaining the other partitions' counts.

The new test replaces one partition, verifies that the old rows disappear from the public row-count lookups, and checks that late reports for the old partition do not repopulate the replacement.

for (Entry<Long, Long> entry : tabletRows.entrySet()) {
if (changelogIndexId > 0 && isChangelogTablet(entry.getKey(), changelogIndexId)) {
// The change rows of the load, which are neither the rows loaded into the
// base index nor a lower bound of the table's rows: see TableStatsMeta.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Do not route unknown tablet IDs into the base counter. When a load's partition is force-dropped before its transaction becomes visible, DatabaseTransactionMgr removes the missing partition from the commit info but still forwards the transaction's tablet delta map. The old changelog tablet metadata has already been removed, so isChangelogTablet returns false and this branch adds its rows to updatedRows. That permanently inflates the base estimate and leaves the changelog counter missing. Carry the index identity with the event or discard/handle missing tablet metadata explicitly instead of treating it as a base tablet.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9fb10c4. For changelog tables, tablet metadata and the owning partition are resolved before classifying any reported tablet. Missing metadata or a removed partition causes that entry to be skipped, rather than falling through into base updatedRows.

AnalysisManagerTest covers both deleted tablet metadata and a replaced partition whose old tablet metadata is still present, and verifies that neither case inflates the base counter.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151672 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fecb190d2b353d80368c0d37610cddbf6ef1569d, data reload: false

query5	4308	581	462	462
query6	420	201	194	194
query7	4809	447	227	227
query8	323	161	165	161
query9	8806	3216	3220	3216
query10	472	311	260	260
query11	6028	3875	3580	3580
query12	148	91	86	86
query13	1240	440	336	336
query14	6090	3917	3653	3653
query14_1	3392	3414	3405	3405
query15	200	195	180	180
query16	922	419	395	395
query17	841	628	501	501
query18	2420	411	307	307
query19	190	165	130	130
query20	83	85	79	79
query21	204	126	107	107
query22	12982	12999	12873	12873
query23	13431	12848	12335	12335
query23_1	12538	12469	12456	12456
query24	7367	864	463	463
query24_1	476	474	475	474
query25	515	390	338	338
query26	1245	267	142	142
query27	2768	445	273	273
query28	4659	2039	2009	2009
query29	1586	607	439	439
query30	293	213	182	182
query31	1002	929	818	818
query32	133	90	96	90
query33	526	294	233	233
query34	955	865	520	520
query35	723	755	638	638
query36	850	853	737	737
query37	125	96	79	79
query38	1775	1719	1701	1701
query39	832	818	759	759
query39_1	738	722	754	722
query40	214	114	94	94
query41	67	64	63	63
query42	91	85	85	85
query43	363	367	318	318
query44	1377	729	740	729
query45	179	178	171	171
query46	852	988	585	585
query47	2668	2947	2889	2889
query48	306	316	217	217
query49	564	398	291	291
query50	683	286	213	213
query51	10563	10558	10321	10321
query52	81	79	69	69
query53	193	209	150	150
query54	253	193	175	175
query55	68	67	62	62
query56	239	224	197	197
query57	1638	1642	1571	1571
query58	280	266	256	256
query59	2146	2144	1987	1987
query60	293	256	241	241
query61	171	170	167	167
query62	383	346	285	285
query63	192	157	172	157
query64	2866	1048	931	931
query65	3344	3292	3269	3269
query66	1776	435	319	319
query67	20186	20420	20123	20123
query68	3145	1035	617	617
query69	408	286	252	252
query70	883	772	809	772
query71	301	245	229	229
query72	2819	2716	2288	2288
query73	518	531	292	292
query74	4560	4516	4360	4360
query75	2326	2313	1927	1927
query76	2315	1031	628	628
query77	350	374	264	264
query78	9799	9701	9208	9208
query79	1039	879	536	536
query80	719	441	343	343
query81	541	317	271	271
query82	578	133	100	100
query83	269	197	175	175
query84	317	123	100	100
query85	899	419	375	375
query86	353	268	241	241
query87	2003	1947	1814	1814
query88	3839	2886	2868	2868
query89	331	281	238	238
query90	1802	176	178	176
query91	160	154	132	132
query92	101	85	89	85
query93	1016	1025	580	580
query94	466	328	275	275
query95	582	333	305	305
query96	694	498	238	238
query97	2442	2446	2313	2313
query98	172	153	151	151
query99	760	761	641	641
Total cold run time: 234761 ms
Total hot run time: 151672 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fecb190d2b353d80368c0d37610cddbf6ef1569d, data reload: false

query1	0.00	0.00	0.01
query2	0.09	0.06	0.05
query3	0.26	0.08	0.08
query4	1.61	0.13	0.12
query5	0.24	0.25	0.25
query6	1.17	0.69	0.71
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.41	0.34	0.36
query10	0.63	0.57	0.59
query11	0.19	0.15	0.14
query12	0.20	0.16	0.15
query13	0.50	0.52	0.50
query14	1.03	1.03	1.03
query15	0.63	0.61	0.62
query16	0.33	0.35	0.37
query17	1.22	1.15	1.11
query18	0.21	0.19	0.19
query19	2.13	2.01	2.05
query20	0.02	0.01	0.01
query21	15.42	0.20	0.13
query22	4.83	0.05	0.05
query23	16.27	0.34	0.13
query24	3.09	0.61	0.30
query25	0.12	0.05	0.07
query26	0.90	0.22	0.15
query27	0.04	0.04	0.05
query28	2.86	0.75	0.37
query29	12.52	4.14	3.27
query30	0.28	0.16	0.11
query31	2.77	0.55	0.34
query32	3.22	0.59	0.52
query33	3.18	3.11	3.18
query34	15.31	3.99	3.42
query35	3.34	3.34	3.34
query36	0.58	0.45	0.43
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.22 s
Total hot run time: 24.36 s

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR 69896 at head fecb190 converged after three bounded rounds. Four new P2 findings are inline: the synthetic changelog wrapper returns a zero row estimate before the new TableStatsMeta fallback; user-injected base stats bypass the changelog accumulator; TTL compaction leaves the persisted accumulator stale; and per-partition replica allocations are divided by the table default. Existing review context was independently revalidated: stale/unknown tablet routing (r4425241009), partial-truncate reset (r4425241002), recreated-statistics baseline (r4425240994), and the cloud limitation (r4424460818) still apply and were not reposted; the retry concern (r4424460813) is shared pre-existing base-delta behavior; the checkstyle import issue (r4422920705) is fixed. No additional user focus was supplied. BE publication, FE routing/locking, lifecycle, serialization, and test coverage were inspected; no new issue was found beyond the inline set. This was a static-only review: builds and tests were not executed. All anchors were verified against the authoritative diff, and all final reviewers returned NO_NEW_VALUABLE_FINDINGS.

Review converged after 3 round(s).

if (isChangelogIndex(table, indexId)) {
// The changelog index has no collected row count: the rows a load wrote to it are the
// only count this record keeps, and they are added rather than taken as a lower bound
// of the table's rows.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Route changelog scans through the new row-count fallback. StatsCalculator calls TableStatsMeta.getRowCountWithDeltaRows only when getRowCountForIndex(..., true) returns -1, but OlapChangelogTableWrapper inherits OlapTable.getRowCountForIndex without overriding it and its inherited idToPartition is empty, so that method returns 0. Every real changelog scan therefore bypasses this branch and is planned as an empty table despite indexUpdatedRows. Delegate the lookup to the origin table or override the wrapper so the accumulated changelog count is reachable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wrapper lookup issue belongs to the consumer change in #69850, which adds row-count delegation and stream-specific estimates. It is not fixed by this PR alone.

In 9fb10c4 this PR no longer introduces a TableStatsMeta fallback: published deltas update the partition changelog MaterializedIndex directly, so the existing partition-index lookup used by P/Q reads the updated estimate. The new test exercises both public partition and whole-index row-count lookups. The wrapper work remains scoped to #69850.

* while this call runs are not part of the snapshot, whichever state reads them accumulates them in
* {@link #updatedRows} and reports them as delta rows.
*/
public synchronized long getRowCountWithDeltaRows(OlapTable table, long indexId) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep the changelog accumulator reachable with injected stats. When userInjected is true, StatsCalculator returns getRowCount(selectedIndexId) without the fallback. ALTER STATS normally injects only the base index, so a changelog scan receives UNKNOWN and ignores the rows accumulated by this new field. Apply the changelog branch before the userInjected shortcut or otherwise supply a changelog-index value.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The StatsCalculator shortcut for injected statistics is a consumer-side issue to address with explicit wrapper-estimate priority in #69850; this PR does not change that shortcut.

In 9fb10c4 the separate TableStatsMeta accumulator was removed. Changelog deltas now update the partition MaterializedIndex row count, which the P/Q metadata lookup reads independently of injected base statistics. This supplies the source count, while wrapper estimate selection remains in the consumer PR.

*/
public synchronized void addIndexUpdatedRows(long indexId, long rows) {
if (rows <= 0) {
return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Account for changelog TTL expiration in this accumulator. Each load is merged permanently into indexUpdatedRows, but BE changelog TTL compaction removes expired rowsets and no FE path subtracts or invalidates those rows. During the interval when the changelog index row count is still unknown, getRowCountWithDeltaRows reports expired rows as readable, overestimating the scan. Invalidate this fallback on TTL-driven removal or tie it to a current reported baseline.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9fb10c4 by removing the lifetime accumulator. Loads now adjust the partition's changelog MaterializedIndex estimate, and the existing periodic tablet-stat path replaces that estimate with the retained row count, including TTL removal. The estimate can still be stale until a fresh report arrives; it no longer permanently accumulates expired rows in a separate persisted map.

The new test replaces the estimate with a lower reported retained count and verifies that the next load is added to that refreshed baseline.

@@ -1525,6 +1535,7 @@ public void replayUpdateRowsRecord(UpdateRowsEvent event) {
+ "rows for all replica {}, tablets count {}",
olapTable.getName(), tableUpdateRows, replicaNum, rowsForAllReplica, tabletRows.size());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Use the partition's replica allocation for changelog deltas. This code divides every changelog tablet's aggregate by the table default replica count, but MODIFY PARTITION allows a changelog partition to use a different allocation. With default 3 replicas and a one-replica partition, nine emitted rows are recorded as 3, so update/delete-only loads permanently undercount the changelog estimate. Divide per tablet/partition (or carry the producer cardinality) instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9fb10c4. Each changelog tablet's aggregate delta is divided by its owning partition's replica allocation before updating that partition's index estimate. It is no longer divided by the table default.

The new test uses a table default of three replicas with one- and three-replica partitions: reported totals of 9 and 18 add 9 and 6 rows respectively, while changelog rows remain excluded from the base counter.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 65.00% (13/20) 🎉
Increment coverage report
Complete coverage report

yujun777 added a commit to yujun777/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69850, apache#69896

Problem Summary: Stream P/Q snapshots read partition index row counts, but
changelog load deltas were stored in a separate lifetime counter in table
statistics. That counter was not read by the partition lookup, lost its
baseline after DROP STATS, and retained removed rows after TTL or partial
TRUNCATE.

Apply published changelog deltas directly to the partition index row-count
estimate. Periodic tablet statistics replace the estimate, and partition
replacement naturally discards it. Normalize each changelog tablet's rows
using its partition replica allocation and ignore late reports for removed
tablets or partitions so changelog rows cannot enter the base counter.
Remove the unused lifetime counter and cover the partition lookup and
statistics/partition lifecycle in AnalysisManagerTest.

Cloud commits and partial publish retries remain subject to the existing
asynchronous statistics limitations. A report that already includes the
load may temporarily cause an overestimate until the next report.

### Release note

Shared-nothing loads update partition changelog row-count estimates before
periodic tablet statistics arrive.

### Check List (For Author)

- Test: Unit Test
    - AnalysisManagerTest: 28 passed
    - TableStatsMetaTest: 15 passed
    - FE Checkstyle validation passed
- Behavior changed: Yes, publish deltas update partition changelog estimates
- Does this need documentation: No
@yujun777

yujun777 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@yujun777

yujun777 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 9fb10c4c0e147d4b9b079b4e57961ab39d03bdaf. One new P2 finding is inline: unconditional companion rows change the semantics of a base-only report consumed by older FEs during supported BE-before-FE upgrades. No existing P0/P1 finding remains to carry forward.

The reviewed flow is grouped load -> attached changelog rowset -> publish workers/MoW append -> token wait -> tablet delta report -> transaction visibility/replay -> partition MaterializedIndex -> public row-count lookup.

Critical checkpoint conclusions:

  1. Goal and evidence: the same-head shared-nothing path supplies final changelog counts to partition estimates. The added FE test's assertions cover both public getters; full transport/P/Q integration is not tested here.
  2. Scope and reuse: the production change is focused and reuses existing reporting and partition index state.
  3. Concurrency and locking: master visibility and follower/checkpoint transaction replay hold table write locks, excluding TabletStatMgr's read-locked count replacement. The coordinator reads rowset counts after worker completion. No new lock order or heavy locked operation was found.
  4. Lifecycle/static initialization: shared ownership retains attached rowsets through reporting; DROP STATS preserves index counts and partition replacement discards the affected baseline. Checkpoint loading/replay rebuilds tablet metadata. No new static initializer, cycle or leak was found.
  5. Configuration: none added.
  6. Compatibility: existing storage fields remain usable, but reused report semantics are incompatible with older FE readers; see the new P2.
  7. Thrift/protobuf IDs: no definitions changed, so cross-branch field-ID comparison is not applicable.
  8. Parallel paths: cloud commits and async/cross-attempt publication still depend on later tablet statistics for omitted changelog deltas. Existing threads already cover cloud and retries; they are not reposted. Wrapper delegation and injected-stat priority remain consumer work discussed in #69850.
  9. Conditions: attachment-null, missing-tablet and removed-partition branches have concrete cases and nearby rationale; partition-specific replica allocation is used for changelog rows.
  10. Test coverage: inspected positive and negative FE assertions cover mixed one-/three-replica partitions, missing metadata, DROP STATS, baseline replacement and partial TRUNCATE with late reports. BE emitted-map/MoW integration, mixed-version and end-to-end consumer tests remain absent.
  11. Test results: no result artifacts changed. This was static inspection only: no builds, tests, benchmarks or runtime validation were executed.
  12. Observability: existing publish and delta-report logs provide transaction/tablet/count context; no additional critical gap was found.
  13. Transaction/persistence/failover: visible transaction state carries the delta map and replays through the same path; MaterializedIndex.rowCount already persists. No new journal record is required. Counts remain approximate across independent statistics samples and failover.
  14. Data writes/atomicity/crashes: rowset visibility, versions and delete-bitmap transitions are unchanged; the new state is an estimate. Known failed-attempt omissions and stale/double-counted baselines do not establish an exact count or upper bound.
  15. FE/BE propagation: same-head transport reaches the correct index and keeps changelog rows out of base/partition update counters; mixed-version propagation is the inline defect.
  16. Performance/memory: additional work is linear in reported tablets with one metadata lookup and a shared pointer per companion; no material new retention or performance defect was found.
  17. Further sweep: all three changed files, upstream/downstream paths, candidate dispositions and risk items were rechecked. No other verified new finding or unresolved static candidate remains.

Existing review context: attached-rowset reporting and import ordering are fixed. The revised partition-index design addresses the reported replica-allocation, missing-tablet, DROP STATS, TRUNCATE and lifetime-TTL accumulator issues. Retry/cloud omissions remain acknowledged limitations; wrapper/injected-stat concerns remain consumer follow-ups. All existing findings are P2, so existing_blocking_comment_ids is empty. There was no additional user-provided focus.

Static review converged in 2 rounds. All assigned reviewers completed the final pass without additional valuable findings. Coverage limits are the unexecuted integration/upgrade behavior and absent tests described above; all changed paths and static risk questions were covered.

Review converged after 2 round(s).

}
for (const RowsetSharedPtr& changelog_rowset : attached_changelog_rowsets) {
const RowsetMetaSharedPtr& rowset_meta = changelog_rowset->rowset_meta();
tablet_id_to_num_delta_rows.insert({rowset_meta->tablet_id(), rowset_meta->num_rows()});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Negotiate changelog deltas before extending the base-only report

The supported rolling-upgrade order upgrades BE before FE. During that window, a pre-PR FE's AnalysisManager.replayUpdateRowsRecord sums every entry in this existing map into base updatedRows, without classifying the new changelog tablet IDs. An insert producing 10 base rows and 10 changelog rows on each of three replicas therefore records 60 / 3 = 20 base delta rows instead of 10, inflating the base statistics counter and any row-preserving estimate that uses its delta. The new FE classifier cannot protect the old master, and PublishVersionTask.toThrift() sends no capability indicating that the receiver understands companion rows. Gate this reporting on a negotiated capability, or carry changelog deltas in a separate optional field that old FEs ignore, and cover the old-reader/new-writer case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changelog has not been officially released yet, and there are no changelog tablets in production. Therefore, the rolling-upgrade compatibility scenario described here does not arise. Compatibility between unreleased changelog implementations is outside the scope of this PR.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28271 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9fb10c4c0e147d4b9b079b4e57961ab39d03bdaf, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17595	5083	5051	5051
q2	2117	304	261	261
q3	10265	1355	811	811
q4	4686	461	335	335
q5	7446	789	520	520
q6	180	184	151	151
q7	747	808	597	597
q8	9247	1271	1042	1042
q9	5484	4185	4139	4139
q10	6816	1292	1016	1016
q11	446	243	227	227
q12	629	391	271	271
q13	18118	3632	3016	3016
q14	272	270	245	245
q15	q16	742	710	694	694
q17	1173	735	563	563
q18	7001	6181	6108	6108
q19	1104	950	607	607
q20	381	331	230	230
q21	5383	2141	2283	2141
q22	347	295	246	246
Total cold run time: 100179 ms
Total hot run time: 28271 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5591	5608	5598	5598
q2	676	573	544	544
q3	5198	5337	4973	4973
q4	1985	2138	1544	1544
q5	5257	5120	5509	5120
q6	259	191	144	144
q7	2322	2087	1871	1871
q8	3005	2692	2698	2692
q9	8131	8052	7964	7964
q10	3977	3635	3256	3256
q11	621	452	417	417
q12	668	681	527	527
q13	3461	3920	3243	3243
q14	331	319	292	292
q15	q16	699	699	645	645
q17	8014	7014	6845	6845
q18	13005	12122	12918	12122
q19	877	760	767	760
q20	2125	2159	1922	1922
q21	5554	4661	4678	4661
q22	493	459	395	395
Total cold run time: 72249 ms
Total hot run time: 65535 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150930 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9fb10c4c0e147d4b9b079b4e57961ab39d03bdaf, data reload: false

query5	4296	603	453	453
query6	423	220	191	191
query7	4810	443	231	231
query8	330	179	153	153
query9	8780	3196	3238	3196
query10	471	313	267	267
query11	6015	3858	3553	3553
query12	143	100	90	90
query13	1239	429	333	333
query14	6147	3926	3681	3681
query14_1	3460	3519	3509	3509
query15	217	212	189	189
query16	937	443	422	422
query17	895	660	551	551
query18	2419	419	311	311
query19	205	172	135	135
query20	88	85	85	85
query21	210	127	109	109
query22	12932	13010	12757	12757
query23	13358	12830	12404	12404
query23_1	12615	12520	12477	12477
query24	7347	829	478	478
query24_1	487	487	496	487
query25	535	416	371	371
query26	1270	261	137	137
query27	2758	439	275	275
query28	4634	2061	2056	2056
query29	1591	597	465	465
query30	314	213	195	195
query31	1037	943	829	829
query32	142	96	91	91
query33	522	298	246	246
query34	881	840	530	530
query35	735	753	661	661
query36	821	829	715	715
query37	122	102	82	82
query38	1766	1713	1654	1654
query39	802	794	752	752
query39_1	738	729	732	729
query40	222	129	98	98
query41	93	86	70	70
query42	91	84	82	82
query43	347	367	312	312
query44	1339	723	730	723
query45	180	192	166	166
query46	839	961	576	576
query47	2933	2969	2867	2867
query48	340	303	229	229
query49	581	388	299	299
query50	683	273	212	212
query51	10304	10185	10392	10185
query52	83	79	68	68
query53	186	208	164	164
query54	233	202	174	174
query55	69	70	62	62
query56	240	220	197	197
query57	1667	1598	1597	1597
query58	288	260	255	255
query59	2116	2129	2021	2021
query60	284	243	224	224
query61	152	140	145	140
query62	386	328	281	281
query63	192	160	163	160
query64	2725	980	796	796
query65	3325	3240	3277	3240
query66	1793	420	346	346
query67	20112	20296	20011	20011
query68	2937	1019	615	615
query69	376	297	238	238
query70	857	810	770	770
query71	296	232	205	205
query72	2768	2663	1857	1857
query73	532	525	299	299
query74	4618	4521	4354	4354
query75	2350	2301	1924	1924
query76	2329	1023	599	599
query77	351	378	267	267
query78	9865	9769	9147	9147
query79	1007	922	529	529
query80	742	424	343	343
query81	563	329	272	272
query82	576	133	110	110
query83	281	196	198	196
query84	328	132	100	100
query85	911	432	373	373
query86	341	242	232	232
query87	1997	1919	1846	1846
query88	3827	2810	2801	2801
query89	334	283	248	248
query90	1785	174	174	174
query91	164	154	130	130
query92	102	83	92	83
query93	1004	1008	579	579
query94	472	315	264	264
query95	569	402	308	308
query96	683	489	243	243
query97	2445	2424	2289	2289
query98	166	152	147	147
query99	775	764	658	658
Total cold run time: 234446 ms
Total hot run time: 150930 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9fb10c4c0e147d4b9b079b4e57961ab39d03bdaf, data reload: false

query1	0.00	0.00	0.01
query2	0.09	0.05	0.05
query3	0.27	0.09	0.09
query4	1.61	0.10	0.10
query5	0.24	0.25	0.24
query6	1.17	0.69	0.67
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.41	0.35	0.35
query10	0.57	0.57	0.58
query11	0.20	0.14	0.15
query12	0.18	0.14	0.15
query13	0.51	0.50	0.49
query14	1.02	1.02	1.01
query15	0.64	0.60	0.63
query16	0.35	0.35	0.35
query17	1.17	1.12	1.12
query18	0.21	0.19	0.19
query19	2.11	2.02	1.94
query20	0.03	0.01	0.02
query21	15.51	0.19	0.14
query22	4.85	0.05	0.05
query23	16.27	0.32	0.13
query24	3.11	0.53	0.34
query25	0.10	0.05	0.05
query26	0.86	0.22	0.15
query27	0.05	0.07	0.04
query28	3.82	0.83	0.38
query29	12.48	4.10	3.29
query30	0.28	0.13	0.12
query31	2.77	0.56	0.33
query32	3.24	0.59	0.51
query33	3.16	3.26	3.18
query34	15.24	3.98	3.38
query35	3.36	3.31	3.32
query36	0.54	0.44	0.43
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.18	0.15	0.17
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 97.06 s
Total hot run time: 24.3 s

yujun777 added a commit to yujun777/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69896

Problem Summary: When partition analysis is enabled, changelog companion
IDs remain unmatched in the base-tablet accounting loop. Each remaining
ID is probed against every partition while the transaction holds the table
write lock.

Collect accepted base entries separately when partition analysis is enabled
and pass only that map to partition accounting. Preserve the original
transaction report for replay. Extend the existing lifecycle test to enable
partition analysis, check base partition counts, and verify that an older
unrelated partition is not probed after the loaded base tablet is matched.
Also clarify that attached rowsets are retained for one publish attempt,
not across publish retries.

### Release note

Avoid unnecessary partition scans when loading changelog tables with
partition analysis enabled.

### Check List (For Author)

- Test: Unit Test
    - AnalysisManagerTest: 28 passed
    - FE Checkstyle validation passed
    - C++ format check passed
- Behavior changed: Yes, partition accounting excludes changelog tablet IDs
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 67.26% (32089/47708)
Line Coverage 52.14% (337820/647936)
Region Coverage 47.67% (274659/576167)
Branch Coverage 49.52% (129546/261596)

@yujun777

yujun777 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69896
commit: df4dea79118b3af97bf92cc1200b522859da0b12
base: 374d87d81de557a8b002cb94a5816b7d8d650c12
reviewed_at: 2026-11-04T11:32+08:00
reviewer: yujun777
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • Static review only; no build, test, or cluster execution was performed during this run.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69850

Problem Summary:

A table stream estimates the change rows a read returns from the row count of the partition's
changelog index: a read of the range between the stream offset and the end of the read returns at
most the change rows the changelog still holds. The count the frontend holds comes from the tablet
reports, which a backend sends every tablet_stat_update_interval_second (60s by default), so a read
which follows a load within that interval sees the count from before the load. The estimate falls
back to the row count of the table, which is safe but loose: it answers with the whole table where
the answer is a few change rows.

The frontend cannot derive the change rows from the rows it loaded into the base index: an update
writes change rows without adding a base row, and a delete adds none to the base but at least one to
the changelog. The backend knows them, so it now reports them.

- The publish task reports the delta rows of the changelog tablets as well, not only of the base
  tablets: the report already visits the rowsets of every tablet a transaction wrote, and a
  changelog tablet is one of them.
- AnalysisManager.replayUpdateRowsRecord routes those tablets to the table record by the index of
  each tablet (from the tablet inverted index, which the followers have as well, so a replay
  reproduces the routing without a new journal entry), and TableStatsMeta accumulates them per
  index (Gson field "iur"). The base index rows are counted as before: the change rows are neither
  added to them nor mistaken for them.
- TableStatsMeta.getRowCountWithDeltaRows returns the accumulated rows for a changelog index: an
  analysis never collects its row count, so the rows of the loads are the count there.

A stream read consumes this through TableStatsMeta.getRowCountWithDeltaRows, which is where the
estimate of PR apache#69850 reads its counters from; until that lands, the recorded rows are visible in
the table statistics only.

### Release note

None

### Check List (For Author)

- Test: Unit test
    - Unit test: TableStatsMetaTest, a new case (the rows a load writes to the changelog index are
      accumulated per index and reported by getRowCountWithDeltaRows, while the base index rows are
      not affected). TableStatsMetaTest 16/16 and AnalysisManagerTest 27/27 pass with the change.
    - No regression exercises the reporting window: it needs a load followed by a read within the
      tablet reporting interval, which the suites cannot time reliably. Every load now reports the
      change rows as well, so the existing suites which read the row counts of a loaded table cover
      the base index side of the change.
- Behavior changed: No user visible change; a load additionally reports the rows it wrote to the
  changelog index.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69850, apache#69896

Problem Summary: Stream P/Q snapshots read partition index row counts, but
changelog load deltas were stored in a separate lifetime counter in table
statistics. That counter was not read by the partition lookup, lost its
baseline after DROP STATS, and retained removed rows after TTL or partial
TRUNCATE.

Apply published changelog deltas directly to the partition index row-count
estimate. Periodic tablet statistics replace the estimate, and partition
replacement naturally discards it. Normalize each changelog tablet's rows
using its partition replica allocation and ignore late reports for removed
tablets or partitions so changelog rows cannot enter the base counter.
Remove the unused lifetime counter and cover the partition lookup and
statistics/partition lifecycle in AnalysisManagerTest.

Cloud commits and partial publish retries remain subject to the existing
asynchronous statistics limitations. A report that already includes the
load may temporarily cause an overestimate until the next report.

### Release note

Shared-nothing loads update partition changelog row-count estimates before
periodic tablet statistics arrive.

### Check List (For Author)

- Test: Unit Test
    - AnalysisManagerTest: 28 passed
    - TableStatsMetaTest: 15 passed
    - FE Checkstyle validation passed
- Behavior changed: Yes, publish deltas update partition changelog estimates
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69896

Problem Summary: When partition analysis is enabled, changelog companion
IDs remain unmatched in the base-tablet accounting loop. Each remaining
ID is probed against every partition while the transaction holds the table
write lock.

Collect accepted base entries separately when partition analysis is enabled
and pass only that map to partition accounting. Preserve the original
transaction report for replay. Extend the existing lifecycle test to enable
partition analysis, check base partition counts, and verify that an older
unrelated partition is not probed after the loaded base tablet is matched.
Also clarify that attached rowsets are retained for one publish attempt,
not across publish retries.

### Release note

Avoid unnecessary partition scans when loading changelog tables with
partition analysis enabled.

### Check List (For Author)

- Test: Unit Test
    - AnalysisManagerTest: 28 passed
    - FE Checkstyle validation passed
    - C++ format check passed
- Behavior changed: Yes, partition accounting excludes changelog tablet IDs
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69850

Problem Summary:

Changelog index row counts can remain stale or unknown until periodic tablet statistics arrive. Base-table load rows cannot describe changelog writes: updates and deletes have their own physical changelog records. Cloud commit responses previously carried no partition/index-level changelog delta, and default cloud replica zeros could be marked as fully reported counts.

This PR completes physical changelog row-count collection in shared-nothing and cloud deployments:

- Read attached changelog rowsets after publish workers finish, including segments appended by merge-on-write publication. Normalize shared-nothing deltas by each partition's replica allocation and filter removed or replaced tablets.
- Return optional partition/index-level changelog deltas from ordinary, lazy and subtransaction cloud commits, with tablet identities and the final partition version. Only completed visible commits supply additive statistics. Cloud rows are counted once without replica division and remain separate from base updatedRows.
- Preserve unknown counts on zero deltas and establish positive partial estimates without setting the complete-report flag. Apply duplicate and out-of-order cloud responses once using compact version ranges; complete periodic reports retire superseded versions.
- Overwrite retained counts from valid periodic tablet statistics so TTL can lower the count. Preserve partial-report validity and serialize count publication with commit updates under the table metadata lock. Replacing tablet data resets its local estimate; DROP STATS preserves it.

Counts are physical records; UPDATE records are not multiplied by two. Periodic reports and commit deltas can temporarily overlap or lag. Failed publish attempts can miss immediate deltas, which later tablet reports calibrate; these estimates are not exact historical counts or strict upper bounds.

### Release note

Successful loads update partition changelog row-count estimates in both shared-nothing and cloud deployments before the next periodic tablet report.

### Check List (For Author)

- Test: Unit Test / Regression test
    - 121 focused FE tests covering analysis statistics, index counts, periodic reports, cloud commit callbacks and retries passed across focused runs.
    - ASAN BE publish test passed with real concurrent MoW writes, publish-time changelog segment growth, final row-count reporting and publish retry.
    - 5 ASAN meta-service RPC tests passed for ordinary, lazy and subtransaction commits, failures/retries, final temporary rowset metadata and base/changelog separation.
    - Shared-nothing changelog_p0 regression directory: 27 suites passed. The new suite pauses periodic statistics publication and checks immediate counts, stream results, DROP STATS and TRUNCATE.
    - FE build (excluding unavailable OBS dependencies), ASAN BE build, Checkstyle, clang-format, clang-tidy, build hygiene and cross-release protobuf checks passed.
    - Cloud end-to-end cluster regression was not run; cloud coverage consists of the RPC and FE tests above.
- Behavior changed: Yes, successful loads immediately supplement partition changelog row-count estimates in both storage modes, while periodic statistics continue to calibrate retained stock.
- Does this need documentation: No
@yujun777
yujun777 force-pushed the ivm-changelog-load-rows branch from df4dea7 to 740771a Compare November 13, 2026 07:45
@yujun777 yujun777 changed the title [improvement](stream) Record the rows a load writes to a changelog index [improvement](stream) Collect changelog load rows in both storage modes Nov 13, 2026
@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69896

Problem Summary: Cloud replica row-count validity is transient, while checkpoints retain the last fetch time and adaptive polling interval. After FE recovery an idle tablet at the infinite interval was never fetched again, leaving base and changelog index counts incomplete. Fetch replicas without a valid report before applying the persisted backoff. A failed fetch remains eligible for retry; a successful report restores completeness and resumes normal backoff.

### Release note

Cloud tablet row counts become available after FE recovery even for idle tablets that reached the maximum statistics polling interval.

### Check List (For Author)

- Test: Unit Test
    - The new mode-2 recovery regression failed before the fix (no fetch request) and passed after it.
    - All 5 TabletStatMgrTest cases passed, including positive restored base/changelog counts, infinite/unexpired intervals, retry after a failed fetch, successful revalidation and resumed backoff.
    - Java incremental compilation and Checkstyle passed.
- Behavior changed: Yes, replicas without a valid row-count report are fetched before polling backoff applies.
- Does this need documentation: No
@yujun777

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69896
commit: e8ec7a37192a39f3ec326b9e41fc4616d7ca384d
base: ccfd4378d424292a363a96142ad795b8c9ace21e
reviewed_at: 2026-11-13T16:30+08:00
reviewer: yujun777
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

None.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

### Background

Changelog row-count estimates combine commit deltas with periodic tablet statistics. Cloud commits have ordinary, lazy and subtransaction paths.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69896 (follow-up verification)
PR Type: Test

Problem Summary:

Verify cloud commit counts independently of periodic publication and remove five unnecessary runtime log statements. Make the immediate-count suite's stream cleanup work in cloud mode.

### Details

- **Scenarios:** Multi-partition and multi-tablet INSERT SELECT subtransactions, commit, rollback and a subsequent commit; ordinary and forced-lazy immediate counts on a local ASAN cloud cluster.
- **What existing tests miss:** RPC and FE unit tests did not exercise the full cloud SQL-to-MS-to-FE count path, and the immediate suite's cleanup lacked the FORCE required in cloud mode.
- **Fails when broken:** With periodic publication paused, SHOW DATA must match physical changelog rows: 4 after the first subtransaction commit, 4 after rollback and 5 after the next commit.

### Testing

- Cloud immediate-count and subtransaction suites passed in ordinary and forced-lazy runs. Existing MS logs confirmed commit_txn_eventually; the fuzzy configuration was restored to zero.
- Cloud changelog_p0: 24/28 suites passed. Four existing row-content suites failed again in an isolated batch: test_changelog_before_column_name, test_cloud_changelog_publish_conflict, test_changelog_multi_segment and test_changelog_basic. Their golden files were not changed.
- The immediate-count suite also passed in shared-nothing mode after the FORCE cleanup change.
- FE and ASAN BE/MS builds, Java Checkstyle, C++ formatting and targeted MS clang-tidy passed after logging cleanup.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Partially verified: the four cloud row-content suites listed above remain failing; the new count coverage passes.

### AI Disclosure

- Tool: Codex (model not stated)
- Generated: Logging cleanup and cloud regression coverage.
- Human review: Not stated.
@yujun777

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69896
commit: a22b63e8cae0e60ca5874f230bc6e0848607429b
base: ccfd4378d424292a363a96142ad795b8c9ace21e
reviewed_at: 2026-11-13T18:03+08:00
reviewer: yujun777
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • Validation remains partial: cloud changelog_p0 passed 24/28. test_changelog_before_column_name, test_cloud_changelog_publish_conflict, test_changelog_multi_segment, and test_changelog_basic failed again in isolation on OP/before-image contents; their goldens are unchanged. Static tracing found no PR-specific causal edit. No merge-base cluster comparison was run, and the exact key_detail history-miss cause remains unproved.
  • This review did not build or run tests. It inspected supplied validation logs, including passing ordinary cloud count/subtransaction suites and the same goldens with lazy forcing enabled; existing MS logs confirm the actual eventual-commit route for ordinary commits.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

### Background

Visible loads supplement changelog index counts before periodic tablet statistics. Stream consumption advances its own offset independently of those current counts.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69850 (stream consumption baselines)
PR Type: Test

Problem Summary:

Cover successive loads and stream consumptions while periodic row-count publication is paused. Current physical counts must grow after each load and remain unchanged by consumption; each successful consumption must empty its interval.

### Details

- **Scenarios:** Extend the existing SN/cloud immediate-count suite with two load/consume cycles. Generated golden sections show retained counts 2, 3 and 4, the two new rows in the sink, and empty intervals after consumption.
- **What existing tests miss:** The existing paused-report test performed only SELECT from the stream, without advancing its offset between writes.
- **Fails when broken:** A missing immediate delta changes the recorded index count; failed offset advancement leaves a nonempty interval or repeats rows in the sink.

### Testing

The extended suite passed on local shared-nothing and ASAN cloud clusters against the same runner-generated golden file. No production code changed in this commit. Four previously reported cloud row-content failures elsewhere remain outside this verification.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex
- Generated: Regression additions, generated golden output, and this description
- Human review: not stated
yujun777 added a commit to yujun777/doris that referenced this pull request Nov 13, 2026
### Background

Table streams read changelog ranges or reconstruct snapshots at a consumption offset. Optimizer statistics originate from base and changelog indexes and can lag data visibility. Each stream needs its own consumption-bound baseline independently of current index statistics.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#62058 (introduced the OLAP wrapper), apache#69896 (supplements current changelog counts after visible loads)
PR Type: Bugfix

Problem Summary:

Stream wrappers can size large reads as one row or use retained changelog stock as an incremental range. Estimate each read from its selected partitions, actual scan type and consumption-bound counts, with lifecycle checks and ordinary statistics fallback. Store the consumed offset, base count and changelog count in one partition state, prepare that state before transaction persistence, and publish the same tuple only after success. Shared-nothing transactions and cloud MS offsets persist the tuple through their existing storage paths.

### Details

- **Change map:** production +790/-40, tests +1266/-23, generated golden output +44/-0, no repository documentation changes. Production changes cover stateless estimate formulas and wrapper/statistics plumbing; one persisted partition consumption state and source lifecycle binding; existing local transaction and cloud creation/commit integration; and nine optional protobuf fields with atomic MS offset storage/readback. Tests cover estimates, lifecycle/concurrency, transaction replay/abort and cloud/MS storage boundaries.
- **Frequency:** Affected wrapper reads can collapse to one row; range quality depends on available statistics. Nonempty offsets with stale, zero or negative deltas require a stock fallback instead of proving zero rows.
- **Root cause:** Wrappers inherited empty partition maps, while retained index counts do not describe a consumed range. Separate or post-commit baseline sampling can lose the association between the offset and its sampled counts.
- **Why existing tests missed it:** Existing tests checked metadata delegation, SQL results and offset advancement without planner count assertions or durable offset/count tuples. Cloud transaction fixtures used unregistered streams and did not cover nonempty production capture/publication, report changes during RPCs, or the fair-lock cycle reproduced during review.
- **Fix:** Persist offset/B0/C0 together in PartitionStreamState and pending OlapTableStreamUpdate. Sample before persistence/RPC; publish unchanged samples after success, leave failed/aborted updates unpublished, and retain counts for unchanged consumed offsets. Use positive changelog/base deltas, then stock/table fallback; only offsets prove an empty range. Apply actual scan-type expansion once and preserve reset/snapshot branch scales. Source sampling and cloud publication use nonblocking metadata locks; publication takes no source lock. FE/data lifecycle bindings invalidate stale estimates without discarding persisted numbers.
- **Other paths and branches:** Covers SN/cloud creation, consumption, image/replay, MS latest/versioned offset storage and existing read states. First initial-snapshot consumption can transition to consumed at the same numeric offset. Column statistics and BE/MS current-count collection remain separate; apache#69896 updates current C1 without changing stream baselines. Stream/IVM storage is unreleased, so no old-format JSON migration is added. All new protocol fields are optional with unused IDs; stream messages are absent from branch-3.1/4.0/4.1/4.2.
- **Behavior change:** Stream scan estimates use the consumed range and selected partitions. Offset and sampled counts are durable together; unknown or invalid baselines use existing fallback rules. Counts remain approximate, and completely unknown inputs retain ordinary OLAP handling.

### Testing

- Local FE build and Checkstyle passed. 144 distinct focused FE tests passed across eleven classes, including actual TransactionState serialization/replay and abort, cloud registered-source commit/create hooks, unchanged offsets, first full consumption at an unchanged numeric TSO, and a nonempty range between reports.
- ASAN Cloud build passed. All 42 MetaServiceTableStreamTest cases passed, covering offset/count storage and readback, restart, failed/CAS-conflicting commits, idempotence, unchanged consumed offsets and latest/versioned storage. Clang-format v16 and changed-line clang-tidy passed.
- All 17 local SN stream regression suites passed, including planner estimates and ordered query results. The three prior concurrency reproductions failed before their fix and passed afterwards. The ASAN BE build passed earlier; no BE execution source changes are included.
- Cloud FE paths used mocked MS RPCs and real in-memory MS transaction tests; this PR's new baseline protocol was not exercised on a live cloud cluster. The companion apache#69896 paused-report regression passed in SN and cloud with two successive load/consume cycles, against runner-generated output.

### Notes for Reviewers

C0 is retained physical changelog stock, not the number of consumed rows. Initial full reads use base scan estimates. Replay preserves stored counts but clears their transient FE/source binding until a later successful consumption establishes one. Periodic/current-count updates never write stream baselines; report/delta overlap and lag remain approximation limits.

### Release note

Table stream scans retain useful row estimates for incremental and snapshot reads, and persist sampled consumption counts together with their offsets.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex; gpt-6.1-sol/xhigh for independent reviews
- Generated: Code, tests, and this description
- Human review: not stated
@yujun777

Copy link
Copy Markdown
Contributor Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69896
commit: b352751e678f173ebe8140c245029708ee2963f7
base: ccfd4378d424292a363a96142ad795b8c9ace21e
reviewed_at: 2026-11-13T19:48+08:00
reviewer: yujun777
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • Validation remains partial: cloud changelog_p0 passed 24/28. test_changelog_before_column_name, test_cloud_changelog_publish_conflict, test_changelog_multi_segment, and test_changelog_basic failed again in isolation on OP/before-image contents; their goldens are unchanged. Current full review re-confirmed the inherited causal boundaries without substantiating a PR-specific defect. No merge-base cluster comparison or exact key_detail history-miss cause was established.
  • regression-test/suites/changelog_p0/test_changelog_immediate_row_count.groovy:102-128 now checks two successive load/consume rounds with periodic publication paused: current changelog stock grows 2→3→4 and consumption leaves it unchanged, each consumed range becomes empty, and the sink receives exactly the two new rows. The extended suite passed once in SN and once in cloud.
  • This review did not build or run tests. The new runs used existing binaries reporting version 1f556; that version string is not proof that every final production file was built, and no exact b352751 binary rebuild or combined-companion-PR runtime result is claimed. The prior recovery fix retains its separate inherited FE red/green evidence.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

yujun777 added a commit to yujun777/doris that referenced this pull request Nov 13, 2026
### Background

Stream updates and scan wrappers keep partition offsets together with consumption counts or read metadata. Their parallel maps share partition IDs, but key sets differ for first consumption, empty partitions, cloud UNKNOWN states and optional scan statistics.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69896 (current changelog row counts)
PR Type: Refactor

Problem Summary:

Store each partition's related update or scan fields in one map instead of separate maps. Preserve absent previous offsets, ranges, baselines, fixed versions and estimates, including their distinct projected key sets.

### Details

- **Change map:** production +204/-183 and tests +118/-17. No protocol, generated output or repository documentation changes. Three state containers and their consumers are grouped by partition; tests check distinct field memberships, persistence, merge and identity.
- **Moves vs. logic changes:** Replace two update maps with a map of previous offset and next complete state. Replace three maps in each wrapper with one map of partition metadata. Adapt consumption and publication to the grouped representation; keep existing merge, CAS, estimate and identity behavior.
- **Behavior change:** No SQL or estimation behavior changes. Explicit row counts remain excluded from wrapper equality/hash; offset-only and version-only partitions are excluded from estimated totals.
- **Compatibility:** The unreleased stream update JSON uses a single updates field. No migration for intermediate unreleased formats. FE/MS and FE/BE protocols are unchanged.
- **Performance impact:** One stored map per class. Preserve direct O(1) partition lookups and cached presence flags. Existing map interfaces use lazy views rather than copying complete maps on every call. No benchmark claim.
- **Rollback:** Revert this refactor before release; intermediate unreleased JSON is not migrated.

### Testing

199 focused FE tests across thirteen classes passed, including new mixed previous-offset merge/serialization/CAS tests, empty/UNKNOWN read memberships, disjoint scan metadata fields and estimate-free identity. Local FE build and Checkstyle passed. All 17 existing SN stream regression suites passed against unchanged golden files. No native source or protocol changes in this commit.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex; gpt-6.1-sol/xhigh for independent reviews
- Generated: Refactor, tests and this description
- Human review: not stated
yujun777 added a commit to yujun777/doris that referenced this pull request Nov 13, 2026
### Background

Stream updates and scan wrappers carry per-partition offsets, row-count baselines or read metadata. Their optional fields have different memberships, so one partition state must preserve which fields exist.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69896 (current changelog counts)
PR Type: Refactor

Problem Summary:

Use named update values and one state map at construction boundaries. Remove split-map constructor overloads, use compact persisted names, and express initial/restored consumption states through explicit factories. Preserve read memberships, offset CAS and estimate-free wrapper identity.

### Details

- **Change map:** Production +143/-120, tests +174/-128; no protocol or generated output changes. Three state APIs and their construction/consumption callers; tests adapt fixtures, check compact JSON and request an early BE tablet report before polling FE counts.
- **Moves vs. logic changes:** Replace map-value Pair with immutable PartitionUpdate(previousOffset,nextState), and serialize it under u/p/n. Wrapper/update constructors accept one named-state map. PartitionStreamState has one complete internal constructor, an immutable initial seed and an explicit restored-fields factory; existing transitions preserve epoch/count semantics.
- **Behavior change:** SQL behavior and read estimates are unchanged. The unreleased update JSON uses shorter names. The regression waiter requests an SN BE tablet report before its existing FE statistics condition.
- **Compatibility:** No migration for intermediate unreleased stream JSON, as the feature is not externally released. All Java construction callers are adapted; FE/BE and FE/MS protocols are unchanged.
- **Performance impact:** Retain expected-size allocation in extraction, update construction/commit copying and scan-state assembly. Immutable initial states share an empty seed; map views avoid full copies for existing read interfaces. No benchmark claim.
- **Rollback:** Revert this refactor before release; intermediate unreleased JSON is not migrated.

### Testing

Local FE build, Checkstyle and all test-source compilation passed. 199 focused FE tests across thirteen classes and 17 existing SN stream regression suites passed against unchanged golden files. Tests cover JSON short fields plus type discriminator, merge/roundtrip/CAS, initial and replay states, wrapper membership/identity, and planner/transaction/cloud creation and commit callers.

### Release note

None

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex; gpt-6.1-sol/xhigh for independent reviews
- Generated: Refactor, tests and this description
- Human review: not stated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants