[Bug] Arrow Flight SQL + Iceberg: SplitSource released before DoGet (fetchSplitBatch crash) #62259

Search before asking

• I had searched in the issues and found no similar issues.

Version

4.1.0-rc01 (built from source, tag apache-iceberg-4.1.0-rc01, ARM64)

What's Wrong

Arrow Flight SQL queries against Iceberg external catalog tables crash the BE with SIGSEGV. MySQL protocol works perfectly for the same queries.

Root cause: FE releases the Iceberg SplitSource after GetFlightInfo returns, before BE's DoGet calls fetchSplitBatch. Arrow Flight's 2-phase execution (GetFlightInfo → DoGet) doesn't align with the SplitSource lifecycle. MySQL protocol is unaffected because plan + execute share one coordinator lifecycle.

Reproduction

1. Set up Iceberg REST catalog (Polaris) with a Silver SCD2 table (~775K rows, ~48 columns)
2. Query via Arrow Flight SQL (ADBC Python driver):

import adbc_driver_flightsql.dbapi as flight_sql
conn = flight_sql.connect("grpc://doris-fe:8040", db_kwargs={"username": "root", "password": ""})
cursor = conn.cursor()
cursor.execute("SELECT * FROM silver.cashify_mart.orders LIMIT 100")
table = cursor.fetch_arrow_table()  # → BE SIGSEGV

3. Same query via MySQL protocol: works fine (2.2s)
4. SELECT 1 via Arrow Flight: works fine (internal table, no SplitSource)

FE Error Log

Caused by: org.apache.thrift.TException: Split source 0 is released
    at org.apache.doris.service.FrontendServiceImpl.fetchSplitBatch(FrontendServiceImpl.java:1009)

RuntimeLogger ERROR (thrift-server-pool-2) [ProcessFunction.process():47] Internal error processing fetchSplitBatch
java.lang.reflect.UndeclaredThrowableException
    at org.apache.doris.thrift.FrontendService$Processor$fetchSplitBatch.getResult(FrontendService.java:5668)
Caused by: org.apache.thrift.TException: Split source 0 is released
    at org.apache.doris.service.FrontendServiceImpl.fetchSplitBatch(FrontendServiceImpl.java:1009)
    at org.apache.doris.nereids.NereidsPlanner.splitFragments(NereidsPlanner.java:592)

BE Crash Stack Trace

 0# doris::signal::FailureSignalHandler(int, siginfo_t*, void*)
 4# std::__cxx11::basic_string constructor at basic_string.h:707
 5# arrow::flight::internal::TransportStatus::FromStatus(arrow::Status const&)
 6# arrow::flight::transport::grpc::ToGrpcStatus(arrow::Status const&, grpc::ServerContext*)
 7# GrpcServiceHandler::DoGet(grpc::ServerContext*, Ticket const*, ServerWriter<FlightData>*)

The SIGSEGV is a secondary bug: TransportStatus::FromStatus crashes on a null/invalid error string when converting the fetchSplitBatch IO_ERROR to a gRPC status. Even without the crash, the query would fail with "Split source 0 is released".

Analysis

The batch split source mechanism (PR #35107, #46398) creates an async SplitSource in FE during query planning. BE fetches splits via Thrift RPC fetchSplitBatch. The lifecycle issue:

1. GetFlightInfo → FE plans query, creates SplitSource, returns FlightInfo
2. Coordinator.close() is called after GetFlightInfo returns → releases SplitSource
3. DoGet → BE starts scan, calls fetchSplitBatch → "Split source 0 is released"

In MySQL protocol, plan + execute happen in one request, so the coordinator (and SplitSource) stays alive throughout. In Arrow Flight, the 2-phase execution means the coordinator is closed between phases.

Workaround

Setting extremely high batch mode threshold forces the old getSplits() path (all splits materialized during planning, before GetFlightInfo returns):

SET GLOBAL num_partitions_in_batch_mode = 999999999;
SET GLOBAL num_files_in_batch_mode = 999999999;

This bypasses the async SplitSource entirely. Verified working on 4.1.0-rc01 with 2 BEs.

Suggested Fix

Option A (minimal, safe for RC patch): Detect Arrow Flight protocol and force non-batch mode:

if (connectContext.getProtocolType() == ProtocolType.ARROW_FLIGHT) {
    sessionVariable.setNumPartitionsInBatchMode(0);
}

Option B (proper): Defer Coordinator.close() / SplitSource cleanup until DoGet completes. Tie the coordinator lifecycle to the query ID, not the GetFlightInfo request.

Environment

• Doris: 4.1.0-rc01 (built from source, ARM64/Graviton4)
• Iceberg: REST catalog (Apache Polaris 1.3.0), Iceberg 1.10.1
• Client: adbc-driver-flightsql 1.4.0 (Python)
• K8s: EKS, 1 FE + 2 BEs

cc @AshinGau (PR #35107 batch split source) @xinyiZzz (Arrow Flight SQL maintainer)