You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 44e3ae2
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](binlog) Track committed TSO and fence uncertain commits for bounded incremental reads (#67820)
### What problem does this PR solve?
Issue Number: None
Related PR: #67181, #67594
Problem Summary:
Bounded cloud `@incr` reads need a safe end timestamp. Waiting by
transaction ID can include transactions outside the requested time
window, while one slow transaction can hold a global committed prefix
and block unrelated tables. FE failover also needs to prevent an old
master or an uncertain commit RPC from later committing a TSO that FE
has already treated as finished.
This PR introduces a durable committed TSO and keeps the active
commit-attempt set in the FE master:
- Allocate and register a commit TSO immediately before the first Meta
Service commit RPC, under the TSO allocator lock. Retries after a fenced
response replace the registration with a fresh TSO above the returned
fence.
- On an explicit success, already-visible result or already-aborted
result, remove the transaction registration. On another explicit
failure, remove only the exact attempted TSO.
- On `KV_TXN_MAYBE_COMMITTED`, an RPC timeout, connection loss or
another uncertain result, synchronously advance the per-instance Meta
Service fence to the attempted TSO before removing it. If fencing fails,
deactivate the TSO service so reads and allocations cannot use an unsafe
prefix.
- Disable transport-level retries for commits carrying a positive TSO.
`MS_TOO_BUSY` remains retryable because Meta Service explicitly rejects
it before processing. Non-TSO transactions keep the existing retry
behavior.
- Derive the committed TSO from the current allocated TSO and the oldest
registered attempt, and persist it together with the allocation-window
end in one BDB journal record. The default persistence window is reduced
from five seconds to one second.
- Remove the startup transaction scan, recovery RPC, recovery state, and
periodic `checkTransactions()`. A new master persists a fresh window and
advances the same global fence before enabling TSO service, so commits
using an older TSO are rejected without waiting for PREPARED
transactions.
- A successful lazy commit can be released from FE memory. Incremental
scans ask Meta Service to wait for pending transactions on the selected
partitions before fixing their visible versions, preserving readability
without a global recovery list.
For a strongly consistent cloud read with an explicit end on every
incremental relation, FE applies these rules:
| Requested end | Behavior |
| --- | --- |
| After the current TSO physical time | Return `5100 /
ERR_INCR_WINDOW_NOT_READY` immediately. |
| At or before the durable committed TSO physical time | Admit the
window directly. |
| Between committed and current TSO physical times | Wait only for the
captured FE registrations whose TSO is within the window and whose
tables are queried. Return `5101 / ERR_INCR_VISIBLE_WAIT_TIMEOUT` when
the wait expires. |
After admission, version resolution waits only for pending transactions
on the selected partitions. Unrelated tables do not participate in
either wait.
`information_schema.tso_status` exposes `COMMITTED_TSO` and
`COMMITTED_TSO_PHYSICAL_TIME`; the latter is epoch milliseconds and is
the maximum globally admitted end timestamp. The existing current TSO
and allocation-window columns remain available.
The Meta Service fence is one monotonic, non-versioned current-state key
in the instance transaction namespace. Commit reads it in the same FDB
transaction as the metadata commit and rejects `commit_tso <= fence`
with `TXN_COMMIT_TSO_EXPIRED`, returning the effective fence. The check
runs only when both the FE request flag and the mutable Meta Service
configuration are enabled. Cloud snapshot, clone and rollback do not
restore this leadership/commit fence; a restored or cloned instance
publishes its current fence when its FE master initializes.
Upgrade Meta Service before FE. A missing or unavailable fence RPC keeps
a new FE TSO service uninitialized. The new Meta Service error code uses
`actual_code` while old clients receive the compatible fallback code.
#### Connector error handling
| Code | Name | Meaning | Connector action |
| --- | --- | --- | --- |
| 5100 | `ERR_INCR_WINDOW_NOT_READY` | The requested end is after
current TSO, or the FE master/TSO generation changed while establishing
the window. | Retry the same end timestamp after `retryAfterMs`; do not
advance the incremental offset. Route a retry through the current master
when the reason is `TSO_MASTER_CHANGED`. |
| 5101 | `ERR_INCR_VISIBLE_WAIT_TIMEOUT` | Relevant commit attempts did
not finish within `change_visible_timeout_ms`. | Retry the same end
timestamp; do not advance the incremental offset. |
MySQL returns SQLSTATE `HY000`; the message contains `reason`, requested
end, current TSO, committed TSO, their physical times, timeout and retry
delay. Arrow Flight SQL returns `UNAVAILABLE` only for these two errors
and includes `doris-error-code` and `doris-error-name` metadata. Other
Flight failures retain the original `INTERNAL` wrapping.
### Release note
Improve bounded, strongly consistent cloud incremental reads with a
durable committed TSO, table-scoped waits and per-instance commit TSO
fencing. Expose the committed TSO through
`information_schema.tso_status`, distinguish an unavailable window
(5100) from a visibility wait timeout (5101), and remove FE
startup/periodic transaction recovery scans. Upgrade Meta Service before
FE.
### Check List (For Author)
- Test
- [x] FE unit tests for TSO service/tracker, commit-result handling,
Meta Service retry behavior, incremental version waits, MySQL error
propagation and Arrow Flight SQL error mapping.
- [x] Cloud ASAN unit tests for fence monotonicity and stale-commit
rejection, fence-key encoding, and response-code compatibility.
- [x] Regression tests `test_committed_tso` and
`test_binlog_changes_syntax` passed earlier in this PR with outputs
generated by the standard runner.
- [x] `./build.sh -j32` and `./build.sh --cloud -j32`.
- [x] clang-format 16, check-format, Cloud clang-tidy and build-hygiene
checks.
- Behavior changed: Yes. Bounded cloud reads use committed
TSO/table-scoped waits; uncertain TSO commit attempts are fenced before
release; FE no longer scans transactions during startup or periodically.
- Does this need documentation: Yes. Connector-facing error handling is
documented above.
0 commit comments