You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0ff6ed4
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](fluss) Refuse a union read whose log tails will not fit before reading them
BE holds the keys of every log tail it has touched for the whole life of
the scan: the lake splits of one bucket arrive spread among the splits of
every other bucket, and each of them needs its own tail to suppress with.
`fluss.union_read.max_tail_rows` bounds one bucket's tail, so a table with
many partitions and many buckets can retain `partitions * buckets` tails
at once while every individual one stays far below its own ceiling. That
is a query that fails on memory although its input could be streamed.
Two changes, both in planning, neither of which needs a byte of the tail:
- `fluss.union_read.max_total_tail_rows` bounds the sum. It defaults to
ten times the per-bucket ceiling, and a value below the per-bucket one
is refused at CREATE rather than silently taken as a smaller
per-bucket limit, which would make every later error quote a number
nobody configured.
- The per-bucket ceiling is now checked here too. The offsets fluss
reported while planning already say how many records each tail holds
-- `stop - lakeEnd` -- and nothing was reading them. It was left to
the readers, so the answer came back only after one of them had read
two million rows.
Over either ceiling, `auto` gives up the lake half exactly as a truncated
tail already does, and the fluss-only read that replaces it caches no tail
at all; `required` fails with the number that tripped it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-fluss/src/main/java/org/apache/doris/connector/fluss/FlussCatalogProperties.java
+46-1Lines changed: 46 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -92,6 +92,29 @@ public final class FlussCatalogProperties {
92
92
/** Default {@link #UNION_READ_MAX_TAIL_ROWS}: two million rows per bucket. */
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-fluss/src/main/java/org/apache/doris/connector/fluss/FlussScanPlanProvider.java
+63Lines changed: 63 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -187,6 +187,9 @@ public class FlussScanPlanProvider implements ConnectorScanPlanProvider {
187
187
/** {@link #degradedReason} when the log no longer holds the tail the lake snapshot stops before. */
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-fluss/src/test/java/org/apache/doris/connector/fluss/FlussCatalogPropertiesTest.java
+40Lines changed: 40 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -151,12 +151,52 @@ public void theTailCeilingDefaultsToTwoMillionRowsAndMustBePositive() {
0 commit comments