Repository navigation
Commit 6d2b951
committed
[fix](ivm) Estimate a stream read's row count from what the read answers with
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
A scan that reads a table stream is planned over a lightweight wrapper of the base table:
OlapTableWrapper for the base-table reads (stream@reset, and the base part of a snapshot read),
OlapChangelogTableWrapper for the changelog part. The wrapper delegates partition lookups but holds
no partitions of its own, while OlapTable derives a row count by iterating that map. Every scan over
a wrapper was therefore estimated at zero rows, which StatsCalculator.computeOlapScan raises to one:
a read of a four-billion-row table looked like a one-row scan, and the optimizer could pick the real
large side as the broadcast build, exhausting the task memory limit of an incremental view
maintenance refresh.
In cloud mode NormalizeOlapTableStreamScan wraps the base table for every read once the cloud read
states are installed, so both stream@reset() and stream@snapshot() were estimated at one row.
Fix, part one -- ask the table the wrapper reads:
- OlapTableWrapper delegates getRowCountForIndex, getRowCountForPartitionIndex, getPartitionNum and
getPartitionItemOrAnalysisException to the table it wraps. A reset read, and a snapshot read whose
offsets already reached the end, see exactly that table's content; a snapshot read of an older
offset is bounded by it. With the partition count in place, a partition-pruned scan takes the
statistics calculator's per-partition path, which looks up the selected partitions' items to
tighten the partition-key min/max.
- StatsCalculator.getOlapTableRowCount asks a wrapper which records its own estimate before it reads
the statistics record: that record describes the base index of the table, and for a changelog read
it has no entry at all, which would leave the read at the one-row floor.
Fix, part two -- record what a stream read answers with:
An incremental read answers with the change rows written after the stream's offset, and a snapshot
read with the image at that offset. Neither is the row count of the table, and the frontend cannot
derive either from the metadata: the changelog index of a partition holds the change rows of every
offset range.
- OlapTableStream records, next to each partition's offset, a PartitionStreamState: the row count of
the partition's changelog index (P) and of its base index (Q) at the moment the offset advanced
(Gson field "ps", written by initializeLocalOffsets and by unprotectedUpdateStreamUpdate; in cloud
mode TableStreamManager.recordPartitionStreamStates records it after the commit RPC succeeds).
- A changelog read is estimated with max(P_now - P_stored, P_now when the counters are unusable): a
read of an empty range is recognized by the TSOs, which do not lag behind like the row counts, and
everything else takes a value which bounds the read rather than the one-row floor.
- A snapshot read is estimated with max(Q, current base rows), or base rows plus the change rows of
the range for a unique or aggregating table, whose image also holds the rows deleted after the
offset.
Every rule errs on the large side: over-estimating only makes the optimizer plan the read
differently, while under-estimating lets a large read become the build side of a broadcast join.
The estimate relies on the tablet row counts the frontend holds, so it is a bound once the backends
have reported them (a stream records its counters from those reports, and the regression waits for
them); a load followed by a refresh before the report arrives still falls back to a value of the
right order of magnitude rather than to one row.
### Release note
Fix stream reads (including incremental view maintenance refreshes) estimating every scan at one
row, which could make the optimizer broadcast a very large build side and exhaust memory during a
refresh. An incremental read of a stream is now estimated with the change rows after its offset, and
a snapshot read with the image at that offset.
### Check List (For Author)
- Test: Unit test / Regression test / Manual test
- Unit test: OlapTableWrapperTest (row count delegation; the changelog wrapper falls back to the
changelog row count; a recorded estimate overrides the wrapped table; partition item lookup
delegation).
- Unit test: PartitionStreamStateTest (the estimate rules: an empty range, a normal delta, no
baseline, counters moving backwards, unreported counts, an unknown range, both snapshot rules).
- Unit test: StreamScanRowCountTest (TestWithFeService: a changelog table, a stream over it and a
row count injected as ALTER TABLE ... SET STATS does; the whole planner runs and the stream scan
is estimated at the injected row count for both a snapshot and a reset read, inside a join).
- Regression test: changelog_p0/test_stream_scan_row_count, which runs in both the shared-nothing
and the cloud P0 pipelines: reset and snapshot estimates, then consuming the stream and checking
that the incremental read is planned as an empty relation when nothing was written and at three
rows after three more are loaded.
- Manual test: the reproduction below, verified through query profiles on a local cloud cluster,
and the regression suite run against a local shared-nothing cluster.
- Behavior changed: Yes. Scans over a stream read are estimated from what the read answers with
instead of one row; an incremental read is no longer unknown to the optimizer.
- Does this need documentation: No1 parent 2a4191c commit 6d2b951
13 files changed
Lines changed: 1035 additions & 19 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris
- catalog
- stream
- cloud/transaction
- nereids
- rules/rewrite
- stats
- test/java/org/apache/doris
- catalog
- stream
- nereids/stats
- regression-test/suites/changelog_p0
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
38 | 51 | | |
39 | 52 | | |
40 | 53 | | |
| |||
50 | 63 | | |
51 | 64 | | |
52 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
53 | 71 | | |
54 | | - | |
| 72 | + | |
55 | 73 | | |
56 | 74 | | |
57 | 75 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
46 | 52 | | |
47 | 53 | | |
48 | 54 | | |
49 | | - | |
| 55 | + | |
| 56 | + | |
50 | 57 | | |
51 | 58 | | |
52 | 59 | | |
53 | 60 | | |
54 | 61 | | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
55 | 70 | | |
56 | 71 | | |
57 | 72 | | |
58 | 73 | | |
59 | 74 | | |
60 | 75 | | |
61 | 76 | | |
| 77 | + | |
62 | 78 | | |
63 | 79 | | |
64 | 80 | | |
| |||
67 | 83 | | |
68 | 84 | | |
69 | 85 | | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
70 | 91 | | |
71 | 92 | | |
72 | 93 | | |
73 | 94 | | |
74 | 95 | | |
75 | 96 | | |
76 | 97 | | |
| 98 | + | |
77 | 99 | | |
78 | 100 | | |
79 | 101 | | |
| |||
149 | 171 | | |
150 | 172 | | |
151 | 173 | | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
152 | 179 | | |
153 | 180 | | |
154 | 181 | | |
| |||
190 | 217 | | |
191 | 218 | | |
192 | 219 | | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
193 | 271 | | |
194 | 272 | | |
195 | 273 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| 42 | + | |
41 | 43 | | |
42 | 44 | | |
43 | 45 | | |
44 | 46 | | |
45 | 47 | | |
| 48 | + | |
46 | 49 | | |
| 50 | + | |
47 | 51 | | |
48 | 52 | | |
49 | 53 | | |
| |||
56 | 60 | | |
57 | 61 | | |
58 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
59 | 70 | | |
60 | 71 | | |
61 | 72 | | |
| |||
128 | 139 | | |
129 | 140 | | |
130 | 141 | | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
131 | 194 | | |
132 | 195 | | |
133 | 196 | | |
| |||
147 | 210 | | |
148 | 211 | | |
149 | 212 | | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
150 | 217 | | |
151 | 218 | | |
152 | 219 | | |
| |||
318 | 385 | | |
319 | 386 | | |
320 | 387 | | |
| 388 | + | |
321 | 389 | | |
322 | 390 | | |
323 | 391 | | |
| |||
341 | 409 | | |
342 | 410 | | |
343 | 411 | | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
344 | 417 | | |
345 | 418 | | |
346 | 419 | | |
| |||
353 | 426 | | |
354 | 427 | | |
355 | 428 | | |
| 429 | + | |
356 | 430 | | |
357 | 431 | | |
358 | 432 | | |
0 commit comments