Skip to content

Commit bb628eb

Browse files
morningmanclaude
andauthored
[fix](arrow-flight) Harden split source error path to avoid BE crash on external table scan (#64797)
### What problem does this PR solve? Issue Number: ref #62259 (partial — robustness layer only, does not close the issue) Related PR: N/A Problem Summary: When an Arrow Flight SQL query scans an external table (e.g. Iceberg) in batch split mode, the BE fetches file splits from the FE via the `fetchSplitBatch` thrift RPC *during* the scan. If that fetch fails — most notably when the split source has already been released — the error path could crash the BE (SIGSEGV in `arrow::flight::internal::TransportStatus::FromStatus`) instead of failing the query gracefully (see #62259). This PR is the **robustness layer** for that issue: it ensures any `fetchSplitBatch` failure makes the query fail gracefully rather than crashing the BE. It does **not** fix the underlying split source lifecycle problem (the source being released after `GetFlightInfo` but before `DoGet` on the Arrow Flight two-phase path), which is tracked separately in the issue. Changes: 1. **BE `split_source_connector`** — guard `result.status.error_msgs[0]` with an `empty()` check to avoid an out-of-bounds vector read when the FE returns a non-OK status without an error message. 2. **BE `to_arrow_status`** — truncate the error message handed to Arrow/gRPC to a length well below 8192. The message is carried in the gRPC trailer (an HTTP2 header) and may be percent-encoded, so an oversized one can break the response or crash the flight transport status conversion. The 8192 limit was already documented in a comment in this function but was never enforced. The full message is still logged on the BE. 3. **FE `fetchSplitBatch`** — when the split source has been released, return a structured `TStatus(NOT_FOUND)` with a message instead of throwing a bare `TException`. The BE then receives a well-formed, non-empty error through the normal result path instead of a thrift transport exception. ### Release note Fix a BE crash (SIGSEGV) that could happen on the error path of Arrow Flight SQL queries against external tables when fetching split batches from the FE fails. ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [x] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [x] Other reason: This is defensive hardening of an error path (an out-of-bounds guard, a message-length cap, and returning a structured error status instead of throwing). It only triggers when `fetchSplitBatch` fails; existing tests cover the success path, and the crash depends on Arrow/gRPC transport internals that are hard to reproduce deterministically in CI. - Behavior changed: - [x] Yes. On `fetchSplitBatch` failure the query now fails with a clear error message instead of (potentially) crashing the BE, and the FE no longer throws a bare `TException` for a released split source (it returns a `NOT_FOUND` status). - Does this need documentation? - [x] No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 896bf29 commit bb628eb

3 files changed

Lines changed: 23 additions & 3 deletions

File tree

‎be/src/exec/scan/split_source_connector.cpp‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,9 @@ Status RemoteSplitSourceConnector::get_next(bool* has_next, TFileRangeDesc* rang
5959
coord->fetchSplitBatch(result, request);
6060
if (result.__isset.status && result.status.status_code != TStatusCode::OK) {
6161
return Status::IOError<false>("Failed to get batch of split source: {}",
62-
result.status.error_msgs[0]);
62+
result.status.error_msgs.empty()
63+
? "unknown error"
64+
: result.status.error_msgs[0]);
6365
}
6466
} catch (std::exception& e) {
6567
return Status::IOError<false>("Failed to get batch of split source: {}", e.what());

‎be/src/format/arrow/arrow_utils.cpp‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,17 @@ arrow::Status to_arrow_status(const Status& status) {
3838
// The length of exception msg returned to the ADBC Client cannot larger than 8192,
3939
// otherwise ADBC Client will receive:
4040
// `INTERNAL: http2 exception Header size exceeded max allowed size (8192)`.
41-
return arrow::Status::Invalid(status.to_string_no_stack());
41+
// The message is carried in the gRPC trailer (an HTTP2 header) and may be
42+
// percent-encoded (which can expand its size), so an oversized message can break the
43+
// response or even crash the flight transport status conversion. Truncate it well below
44+
// 8192 to leave headroom; the full message is already logged above.
45+
constexpr size_t kMaxArrowStatusMsgLen = 1024;
46+
std::string msg = status.to_string_no_stack();
47+
if (msg.size() > kMaxArrowStatusMsgLen) {
48+
msg.resize(kMaxArrowStatusMsgLen);
49+
msg += "... (truncated)";
50+
}
51+
return arrow::Status::Invalid(msg);
4252
}
4353
}
4454

‎fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1046,7 +1046,15 @@ public TFetchSplitBatchResult fetchSplitBatch(TFetchSplitBatchRequest request) t
10461046
SplitSource splitSource =
10471047
Env.getCurrentEnv().getSplitSourceManager().getSplitSource(request.getSplitSourceId());
10481048
if (splitSource == null) {
1049-
throw new TException("Split source " + request.getSplitSourceId() + " is released");
1049+
// Return a structured error status instead of throwing a bare TException, so the BE
1050+
// receives a well-formed error (with a non-empty message) through the normal result
1051+
// path. Throwing here surfaces as a thrift transport exception on the BE side and,
1052+
// on the Arrow Flight path, may feed an empty/invalid error string into the gRPC
1053+
// status conversion. See https://github.com/apache/doris/issues/62259
1054+
LOG.warn("split source {} is released", request.getSplitSourceId());
1055+
result.status = new TStatus(TStatusCode.NOT_FOUND);
1056+
result.status.addToErrorMsgs("Split source " + request.getSplitSourceId() + " is released");
1057+
return result;
10501058
}
10511059
try {
10521060
List<TScanRangeLocations> locations = splitSource.getNextBatch(request.getMaxNumSplits());

0 commit comments

Comments
 (0)