[Bug] format=arrow Stream Load: BE SIGSEGV instead of a rejected load when an Arrow column's type does not match the target column (BOOLEAN and DECIMAL serdes dereference an unchecked dynamic_cast)

Search before asking

I had searched in the issues and found no similar issues.

Version

FE doris-4.1.0-39053bed, BE doris-4.1.0-6a680508 — our builds from apache/doris master as of 2026-10-20 with local
patches that do not touch the Arrow reader or the serdes. Cloud mode (compute groups), aarch64 (Graviton), Ubuntu 22.04,
Kubernetes. enable_arrow_input_validation = true (the default).
The lines involved are byte-identical on apache/doris master today (2026-10-29):
be/src/core/data_type_serde/data_type_number_serde.cpp 957–966 and 968–974 (the HALF_FLOAT contrast),
be/src/core/data_type_serde/data_type_decimal_serde.cpp 921–923 and 948–951.

What's Wrong?

A Stream Load with format: arrow whose IPC stream carries a column of an Arrow type the target column's serde does not
expect crashes the BE (SIGSEGV) instead of rejecting the load. Two serdes observed in production, same shape:

1. BOOLEAN target, Int32 Arrow array — DataTypeNumberSerDe<TYPE_BOOLEAN>::read_column_from_arrow
(data_type_number_serde.cpp):

    if constexpr (T == TYPE_BOOLEAN) {
        const auto* concrete_array = dynamic_cast<const arrow::BooleanArray*>(arrow_array);   // 958: nullptr for an Int32Array
        if (config::enable_arrow_input_validation) {
            check_arrow_boolean_buffer(*concrete_array);                                        // 960: dereferences nullptr → SIGSEGV
        }
        for (int64_t bool_i = start; bool_i != end; ++bool_i) {
            col_data.emplace_back(concrete_array->Value(bool_i));                                // 963: same, with validation off
        }
        return Status::OK();
    }

Stack (BE container stdout, 2026-10-21 23:13:41 UTC; the faulting read is array.data() at offset 8 of a null object):

 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at ../src/common/signal_handler.h:418
 1# 0x0000FFFFAA3A7838 in linux-vdso.so.1
 2# doris::check_arrow_validity_bitmap(arrow::Array const&) at ../src/core/data_type_serde/arrow_validation.h:125
 3# doris::DataTypeNumberSerDe<(doris::PrimitiveType)2>::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const at ./be/build_Release/../src/core/data_type_serde/data_type_number_serde.cpp:960
 4# doris::DataTypeNullableSerDe::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const at ./be/src/core/data_type_serde/data_type_nullable_serde.cpp:425
 5# doris::ArrowStreamReader::_do_get_next_block(doris::Block*, unsigned long*, bool*) at ./be/build_Release/../src/format/arrow/arrow_stream_reader.cpp:117
 6# doris::GenericReader::get_next_block(doris::Block*, unsigned long*, bool*) at ../src/format/generic_reader.h:115
 7# doris::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/file_scanner.cpp:586
 8# doris::FileScanner::_get_block_impl(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/file_scanner.cpp:515
 9# doris::Scanner::get_block(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/scanner.cpp:192
10# doris::Scanner::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*) in /opt/apache-doris/be/lib/doris_be
11# doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) at ./be/src/exec/scan/scanner_scheduler.cpp:193
12# doris::ScannerScheduler::execute_scan_task(std::shared_ptr<doris::ScannerContext> const&, std::shared_ptr<doris::ScanTask> const&) at ./be/src/exec/scan/scanner_scheduler.cpp:101
13# doris::ThreadPoolSimplifiedScanScheduler::_run_context(std::shared_ptr<doris::ScannerContext>) at ./be/src/exec/scan/simplified_scan_scheduler.cpp:140
14# std::_Function_handler<void (), doris::ThreadPoolSimplifiedScanScheduler::schedule_scan_task(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>, std::unique_lock<std::mutex>&)::$_0>::_M_invoke(std::_Any_data const&)
15# doris::ThreadPool::dispatch_thread() at ./be/src/util/threadpool.cpp:628
16# doris::Thread::supervise_thread(void*) at ./be/src/util/thread.cpp:461
17# 0x0000FFFFA8DA0398 in /lib/aarch64-linux-gnu/libc.so.6
18# 0x0000FFFFA8E09E9C in /lib/aarch64-linux-gnu/libc.so.6

Frame 4 shows the nullable wrapper had already validated the array (data_type_nullable_serde.cpp:425), so the input was a
well-formed Arrow column — just not a BooleanArray (the client typed a MySQL tinyint(1) column as int32).

2. DECIMAL target, Utf8 Arrow array — DataTypeDecimalSerDe<TYPE_DECIMAL64>::read_column_from_arrow
(data_type_decimal_serde.cpp), target column DECIMAL(10,2) (decimalv3), the client sent the value as a utf8 string:

    } else if constexpr (T == TYPE_DECIMAL32 || T == TYPE_DECIMAL64 || T == TYPE_DECIMAL128I) {
        const auto* concrete_array = dynamic_cast<const arrow::DecimalArray*>(arrow_array);   // 949: nullptr for a StringArray
        for (auto value_i = start; value_i < end; ++value_i) {
            const auto* value = concrete_array->Value(value_i);                               // 951: SIGSEGV

(The TYPE_DECIMALV2 branch at 921–923 has the same unchecked cast.) Stack (2026-10-25 19:19:06 UTC):

 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at ../src/common/signal_handler.h:418
 1# 0x0000FFFFBD9E2838 in linux-vdso.so.1
 2# doris::DataTypeDecimalSerDe<(doris::PrimitiveType)29>::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const at ./be/src/core/data_type_serde/data_type_decimal_serde.cpp:951
 3# doris::DataTypeNullableSerDe::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const at ./be/src/core/data_type_serde/data_type_nullable_serde.cpp:425
 4# doris::ArrowStreamReader::_do_get_next_block(doris::Block*, unsigned long*, bool*) at ./be/build_Release/../src/format/arrow/arrow_stream_reader.cpp:117
 5# doris::GenericReader::get_next_block(doris::Block*, unsigned long*, bool*) at ../src/format/generic_reader.h:115
 6# doris::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/file_scanner.cpp:586
 7# doris::FileScanner::_get_block_impl(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/file_scanner.cpp:515
 8# doris::Scanner::get_block(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/scanner.cpp:192
 9# doris::Scanner::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*) in /opt/apache-doris/be/lib/doris_be
10# doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) at ./be/src/exec/scan/scanner_scheduler.cpp:193
11# doris::ScannerScheduler::execute_scan_task(std::shared_ptr<doris::ScannerContext> const&, std::shared_ptr<doris::ScanTask> const&) at ./be/src/exec/scan/scanner_scheduler.cpp:101
12# doris::ThreadPoolSimplifiedScanScheduler::_run_context(std::shared_ptr<doris::ScannerContext>) at ./be/src/exec/scan/simplified_scan_scheduler.cpp:140
13# std::_Function_handler<void (), doris::ThreadPoolSimplifiedScanScheduler::schedule_scan_task(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>, std::unique_lock<std::mutex>&)::$_0>::_M_invoke(std::_Any_data const&)
14# doris::ThreadPool::dispatch_thread() at ./be/src/util/threadpool.cpp:628
15# doris::Thread::supervise_thread(void*) at ./be/src/util/thread.cpp:461
16# 0x0000FFFFBC3E0398 in /lib/aarch64-linux-gnu/libc.so.6
17# 0x0000FFFFBC449E9C in /lib/aarch64-linux-gnu/libc.so.6

Blast radius. The client sees a reset connection, not a load error, so it retries the same batch on the next BE: in
our case one bad batch crashed all three BEs of the compute group in turn within three minutes (identical stack on each),
and every later commit of that table re-armed the loop until the Arrow path was switched off client-side.

Contrast inside the same function. The HALF_FLOAT branch directly below the BOOLEAN branch does it right
(data_type_number_serde.cpp 969–974):

            const auto* concrete_array = dynamic_cast<const arrow::HalfFloatArray*>(arrow_array);
            if (concrete_array == nullptr) {
                return Status::InvalidArgument("Expected Arrow HalfFloatArray, got {}",
                                               arrow_array->type()->name());
            }

and a width mismatch (a float32 array into a DOUBLE column) is already rejected cleanly by the validation layer
([E33] Invalid Arrow float: data buffer too small: 32 bytes available, 64 required). A type mismatch should fail the
same way.

What You Expected?

The Stream Load fails with an error that names the column and the Arrow type it received (an InvalidArgument of the
HALF_FLOAT shape), the HTTP response carries Status: Fail with that message, and the BE stays up.

How to Reproduce?

Only on a scratch cluster — this kills the BE process. One BE, default config (enable_arrow_input_validation=true;
the crash also happens with it off, one line later).

CREATE DATABASE IF NOT EXISTS repro;
CREATE TABLE repro.t_bool (k INT, b BOOLEAN NULL)
  DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
CREATE TABLE repro.t_dec (k INT, v DECIMAL(10,2) NULL)
  DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");

# make_streams.py — pyarrow IPC streams (pip install pyarrow)
import pyarrow as pa, pyarrow.ipc as ipc
from decimal import Decimal

def write(path, table):
    with pa.OSFile(path, "wb") as f, ipc.new_stream(f, table.schema) as w:
        w.write_table(table)

write("bool_crash.arrow", pa.table({"k": pa.array([1], pa.int32()), "b": pa.array([1], pa.int32())}))          # int32 → BOOLEAN: crash
write("bool_ok.arrow",    pa.table({"k": pa.array([1], pa.int32()), "b": pa.array([True], pa.bool_())}))      # control: loads
write("dec_crash.arrow",  pa.table({"k": pa.array([1], pa.int32()), "v": pa.array(["1.50"], pa.utf8())}))     # utf8 → DECIMAL(10,2): crash
write("dec_ok.arrow",     pa.table({"k": pa.array([1], pa.int32()),
                                    "v": pa.array([Decimal("1.50")], pa.decimal128(10, 2))}))                 # control: loads

FE=http://<fe-host>:8030
# controls first — both return "Status": "Success"
curl --location-trusted -u root: -H "format: arrow" -H "columns: k,b" -T bool_ok.arrow $FE/api/repro/t_bool/_stream_load
curl --location-trusted -u root: -H "format: arrow" -H "columns: k,v" -T dec_ok.arrow  $FE/api/repro/t_dec/_stream_load
# the crashes — each returns "Empty reply from server" / connection reset, be.out prints the stack above, the BE exits
curl --location-trusted -u root: -H "format: arrow" -H "columns: k,b" -T bool_crash.arrow $FE/api/repro/t_bool/_stream_load
curl --location-trusted -u root: -H "format: arrow" -H "columns: k,v" -T dec_crash.arrow  $FE/api/repro/t_dec/_stream_load

Expected on both crash cases: a rejected load. Actual: BE SIGSEGV at data_type_number_serde.cpp:960 /
data_type_decimal_serde.cpp:951 (frames above), process exit, the transaction left in PREPARE until the stream-load
timeout.

Anything Else?

Proposed fix (minimal, the HALF_FLOAT pattern): in every read_column_from_arrow that dynamic_casts the array —
BOOLEAN, DECIMALV2, DECIMAL32/64/128I, and an audit of the other serdes — check the result and return
Status::InvalidArgument("Expected Arrow {}Array for column type {}, got {}", ...). Accepting Int8/Int16/Int32/UInt8
arrays for BOOLEAN (a cast, since MySQL tinyint(1) arrives that way from most Arrow writers) would be a reasonable
follow-up, but the reject alone closes the crash.
A BE unit test for each: an arrow::Int32Array into a BOOLEAN column and an arrow::StringArray into a DECIMAL64
column both expect InvalidArgument, with enable_arrow_input_validation on and off.
Our fork does not yet carry the fix; the two incidents were stopped client-side (the writer now types BOOLEAN targets as
bool and DECIMAL targets as decimal128(p,s)). We are willing to submit the PR.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Breakwater-GitHub-Analysis-Slot: slot_381a558cb6cd

This is a source-confirmed error-handling defect in Arrow ingestion and merits high-priority BE/load triage. A valid Arrow array with an unsupported target-type pairing reaches an unchecked cast and can terminate the BE instead of failing the load. The reported retry-driven compute-group outage increases the operational impact. The issue is currently open, with no labels or prior comments at review time.

Verified source evidence

I checked the configured local checkout, then fetched the relevant upstream files at the reported BE commit, 6a680508899258b93f78a15407a5ef93fb56fe9f, and current master, 2a4191c1dc94e789e1725d6cb345f46c31a17272. The BOOLEAN and DECIMAL Arrow-reading functions are unchanged between those two upstream revisions. This is source inspection; I did not run the crashing reproducer or inspect the locally patched production binary.

ArrowStreamReader checks column names and passes each array to the destination serde without establishing type compatibility. The nullable wrapper checks the read range and validity bitmap before delegating. Those checks do not establish that the array matches the destination type or fully validate every value buffer; reaching the nested serde alone is not proof of full Arrow validity.
BOOLEAN conversion casts to BooleanArray without checking the result. An Int32Array produces a null pointer: validation enabled dereferences it through check_arrow_boolean_buffer; validation disabled reaches Value() for the supplied nonempty batch. Disabling validation is therefore not a workaround. The adjacent HALF_FLOAT branch already demonstrates checked rejection.
DECIMAL conversion has the same unchecked cast in DECIMALV2 and DECIMAL32/64/128I. DECIMALV2 also accesses cast type metadata before reading values, so rejection must precede that access. DECIMAL256 has an equivalent unchecked Decimal256Array cast and belongs in the fix/audit scope.

The source mechanism matches the supplied stacks. There is no architecture- or cloud-mode condition in these failing branches, although behavior on other deployments has not been independently reproduced. The three-BE outage and transaction timeout remain reporter-observed facts, not independently verified results.

Recommended next steps

Proceed with the offered PR: return InvalidArgument immediately after failed array casts, before validation, metadata access, or value reads, regardless of enable_arrow_input_validation. Keep integer-to-BOOLEAN coercion as a separately specified behavior change. Preserve existing supported conversions.
Add the column name and destination type at ArrowStreamReader, which has that context; serde messages alone cannot identify the table column. Preserve the error code while adding context and include the received Arrow type. The reader already propagates returned errors. The ordinary Stream Load failure path attempts rollback, and response serialization produces Status: Fail and a message. Verify the complete HTTP/transaction path with an integration test.
Add direct serde tests for both reported mismatches with validation on/off, nullable and nonnullable targets, and successful correctly typed controls. Extend coverage to DECIMALV2/32/128I/256, empty and all-null arrays, and nested mismatches. Integration tests should assert an informative failed load, no committed rows, and successful subsequent loading on the same BE.
Audit remaining Arrow casts against the target revision. MAP and STRUCT also contain unchecked incoming-array casts. These are source-level audit findings, not additional reproduced incidents. ARRAY already checks its supported list casts in this revision. Buffer-size validation alone is not a general type-compatibility check, so the float-width rejection should not be treated as proof that all other mismatches are handled.

Remaining evidence and immediate mitigation

The supplied DDL, generators, commands, versions, and stacks are sufficient to start the fix; a query profile is unnecessary for this deterministic type-dispatch failure. To close verification, record the PyArrow version and scratch-cluster results on unmodified upstream at the pinned commit, then attach the post-fix responses and BE-survival results. Production logs/core details are only needed if that reproduction diverges. Affected release branches and backport targets still require separate verification; the 4.1.0 build string alone does not establish release-wide coverage.

Continue emitting Arrow bool and decimal128(p,s) matching these target columns, as in the supplied controls. Quarantine a failing batch instead of repeatedly sending the unchanged payload to other BEs; retain the client-side Arrow fallback until a tested fix is deployed.

Breakwater-GitHub-Analysis-Slot: slot_381a558cb6cd

This is a source-confirmed error-handling defect in Arrow ingestion and merits high-priority BE/load triage. A valid Arrow array with an unsupported target-type pairing reaches an unchecked cast and can terminate the BE instead of failing the load. The reported retry-driven compute-group outage increases the operational impact. The issue is currently open, with no labels or prior comments at review time.

Verified source evidence

I checked the configured local checkout, then fetched the relevant upstream files at the reported BE commit, 6a680508899258b93f78a15407a5ef93fb56fe9f, and current master, 2a4191c1dc94e789e1725d6cb345f46c31a17272. The BOOLEAN and DECIMAL Arrow-reading functions are unchanged between those two upstream revisions. This is source inspection; I did not run the crashing reproducer or inspect the locally patched production binary.

ArrowStreamReader checks column names and passes each array to the destination serde without establishing type compatibility. The nullable wrapper checks the read range and validity bitmap before delegating. Those checks do not establish that the array matches the destination type or fully validate every value buffer; reaching the nested serde alone is not proof of full Arrow validity.
BOOLEAN conversion casts to BooleanArray without checking the result. An Int32Array produces a null pointer: validation enabled dereferences it through check_arrow_boolean_buffer; validation disabled reaches Value() for the supplied nonempty batch. Disabling validation is therefore not a workaround. The adjacent HALF_FLOAT branch already demonstrates checked rejection.
DECIMAL conversion has the same unchecked cast in DECIMALV2 and DECIMAL32/64/128I. DECIMALV2 also accesses cast type metadata before reading values, so rejection must precede that access. DECIMAL256 has an equivalent unchecked Decimal256Array cast and belongs in the fix/audit scope.

The source mechanism matches the supplied stacks. There is no architecture- or cloud-mode condition in these failing branches, although behavior on other deployments has not been independently reproduced. The three-BE outage and transaction timeout remain reporter-observed facts, not independently verified results.

Recommended next steps

Proceed with the offered PR: return InvalidArgument immediately after failed array casts, before validation, metadata access, or value reads, regardless of enable_arrow_input_validation. Keep integer-to-BOOLEAN coercion as a separately specified behavior change. Preserve existing supported conversions.
Add the column name and destination type at ArrowStreamReader, which has that context; serde messages alone cannot identify the table column. Preserve the error code while adding context and include the received Arrow type. The reader already propagates returned errors. The ordinary Stream Load failure path attempts rollback, and response serialization produces Status: Fail and a message. Verify the complete HTTP/transaction path with an integration test.
Add direct serde tests for both reported mismatches with validation on/off, nullable and nonnullable targets, and successful correctly typed controls. Extend coverage to DECIMALV2/32/128I/256, empty and all-null arrays, and nested mismatches. Integration tests should assert an informative failed load, no committed rows, and successful subsequent loading on the same BE.
Audit remaining Arrow casts against the target revision. MAP and STRUCT also contain unchecked incoming-array casts. These are source-level audit findings, not additional reproduced incidents. ARRAY already checks its supported list casts in this revision. Buffer-size validation alone is not a general type-compatibility check, so the float-width rejection should not be treated as proof that all other mismatches are handled.

Remaining evidence and immediate mitigation

The supplied DDL, generators, commands, versions, and stacks are sufficient to start the fix; a query profile is unnecessary for this deterministic type-dispatch failure. To close verification, record the PyArrow version and scratch-cluster results on unmodified upstream at the pinned commit, then attach the post-fix responses and BE-survival results. Production logs/core details are only needed if that reproduction diverges. Affected release branches and backport targets still require separate verification; the 4.1.0 build string alone does not establish release-wide coverage.

Continue emitting Arrow bool and decimal128(p,s) matching these target columns, as in the supplied controls. Quarantine a failing batch instead of repeatedly sending the unchanged payload to other BEs; retain the client-side Arrow fallback until a tested fix is deployed.

[fix](be) Reject incompatible Arrow arrays without crashing Stream Lo…

Navigation Menu

Repository navigation

[Bug] format=arrow Stream Load: BE SIGSEGV instead of a rejected load when an Arrow column's type does not match the target column (BOOLEAN and DECIMAL serdes dereference an unchecked dynamic_cast) #69858

Search before asking

Version

What's Wrong?

What You Expected?

How to Reproduce?

Anything Else?

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug] format=arrow Stream Load: BE SIGSEGV instead of a rejected load when an Arrow column's type does not match the target column (BOOLEAN and DECIMAL serdes dereference an unchecked dynamic_cast) #69858

Description

Search before asking

Version

What's Wrong?

What You Expected?

How to Reproduce?

Anything Else?

Are you willing to submit PR?

Code of Conduct

Activity

Doris-Breakwater commented on Oct 29, 2026

Doris-Breakwater commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions