Skip to content

Commit 7865646

Browse files
authored
[improvement](be) Support Arrow loads and schema inference in FileScannerV2 (apache#69668)
### What problem does this PR solve? Arrow Stream Load still selects FileScanner V1, and CSV/TEXT/JSON schema discovery still constructs legacy readers even when the subsequent scan uses V2. Add a native Arrow IPC FileReader to the V2 load path. It decodes record batches incrementally, supports concatenated streams and compressed IPC, uses tracked allocations, and retains load expressions, defaults, strict-mode checks, and WHERE filtering. Source timestamp units and timezones are preserved for Doris SerDe rounding, including nested values. Stream Load plans explicitly set `enable_file_scanner_v2=true` in their query options because these plans construct options independently of query sessions. Arrow binding preserves positional STRUCT children, DATE64 range validation, and unsigned IPv4 address bits. Before checked recursive casts, ancestor validity masks hide NULL-struct payloads and NULL-list/map ranges are compacted. Ordinary numeric overflow checks remain enabled. A load-specific schema converter preserves native VARIANT carriers and extensions recursively, including SQL NULLs, and binds AGG_STATE through its actual serialization type. Fixed-length aggregate states use the serialization column's byte width; invalid binary lengths fail before SerDe materialization. Recursive NULL masking skips unchanged children and leaves incompatible container kinds to explicit cast errors. Nullable fixed-size-list inputs are viewed through 64-bit variable offsets before compaction, so NULL parents drop complete child ranges without synthesizing NULLs in non-nullable STRUCT fields or native VARIANT carriers. DECIMALV2 retains decimal128 wire scales for its existing scale-nine rescaling and truncation rules; other decimal types retain checked conversions. Keep the existing empty-file schema RPC diagnostics, including empty LZOP data and header-name versus inferred-column-number errors. Route CSV, TEXT, and JSON schema discovery through V2 readers and the shared schema entry point, alongside the existing native Parquet/ORC readers. Preserve header handling, nullable string inference, JSON root/path options, and independent sampling of HTTP Stream Load data. Also avoid consuming a rooted JSON array's ondemand cursor before jsonpaths materialization. ### Release note Arrow Stream Load and CSV/TEXT/JSON schema discovery now use FileScanner V2 readers. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test - ASAN BE build and 221 focused unit tests passed, including scanner routing, native schema inference, Arrow IPC loading, and existing reader coverage. Focused cases cover positional STRUCT names, DATE64 range checks, unsigned IPv4, NULL container payloads, numeric overflow, native VARIANT storage/extensions, aggregate-state widths, incompatible container input, empty LZOP, NULL fixed-size lists with non-nullable descendants, and DECIMALV2 rescaling with DECIMALV3 rejection controls. - Six regression suites passed against a locally built FE/ASAN BE: Arrow Stream Load, V2 HTTP schema discovery, and existing JSON, compressed CSV, CSV-with-names, and CSV-with-names-and-types suites. Results were compared with expected output. - clang-format 16, changed-line clang-tidy, and build hygiene checks passed. - FE Checkstyle and the FE production build passed after adding the explicit Stream Load query option. - Behavior changed: - [x] Yes. Arrow loads and text/JSON schema inference use native V2 readers. - Does this need documentation? - [x] No. Existing load interfaces and options are retained. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent f878b28 commit 7865646

22 files changed

Lines changed: 2141 additions & 96 deletions

‎be/src/exec/operator/file_scan_operator.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,7 @@ bool FileScanLocalState::should_use_file_scanner_v2(const TQueryOptions& query_o
181181
case TFileFormatType::FORMAT_PROTO:
182182
case TFileFormatType::FORMAT_TEXT:
183183
case TFileFormatType::FORMAT_WAL:
184+
case TFileFormatType::FORMAT_ARROW:
184185
return true;
185186
default:
186187
return false;

‎be/src/exec/scan/file_scanner_v2.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -360,7 +360,10 @@ bool FileScannerV2::is_supported(const TFileScanRangeParams& params, const TFile
360360
} else if (format_type == TFileFormatType::FORMAT_ES_HTTP) {
361361
return table_format_name(range) == "es";
362362
} else if (format_type == TFileFormatType::FORMAT_ARROW) {
363-
return is_supported_arrow_table_format(range);
363+
// A plain Arrow range is an IPC load pipe; query Arrow ranges retain their connector.
364+
return is_supported_arrow_table_format(range) ||
365+
(table_format_name(range) == "NotSet" && params.__isset.src_tuple_id &&
366+
params.__isset.file_type && params.file_type == TFileType::FILE_STREAM);
364367
} else if (format_type == TFileFormatType::FORMAT_JNI) {
365368
return is_supported_jni_table_format(range);
366369
} else if (is_wal_format(format_type)) {

‎be/src/format_v2/arrow/arrow_stream_reader.cpp‎

Lines changed: 527 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#pragma once
19+
20+
#include <memory>
21+
#include <optional>
22+
#include <vector>
23+
24+
#include "format_v2/file_reader.h"
25+
#include "runtime/runtime_profile.h"
26+
27+
namespace arrow {
28+
class Array;
29+
class DataType;
30+
class MemoryPool;
31+
class RecordBatch;
32+
namespace ipc {
33+
class RecordBatchStreamReader;
34+
}
35+
} // namespace arrow
36+
37+
namespace doris {
38+
class RuntimeState;
39+
class SlotDescriptor;
40+
} // namespace doris
41+
42+
namespace doris::format::arrow_stream {
43+
44+
class ArrowInputStream;
45+
46+
// IPC decoding binds the load planner's typed source slots, including expression inputs.
47+
// These describe the pipe's input contract; destination casts, defaults, and row rejection
48+
// belong to LoadTableReader.
49+
class ArrowStreamReader final : public FileReader {
50+
public:
51+
ArrowStreamReader(std::shared_ptr<io::FileSystemProperties>& properties,
52+
std::unique_ptr<io::FileDescription>& file,
53+
std::shared_ptr<io::IOContext> io_ctx, RuntimeProfile* profile,
54+
std::vector<SlotDescriptor*> file_slots, std::optional<TUniqueId> load_id);
55+
~ArrowStreamReader() override;
56+
57+
Status init(RuntimeState* state) override;
58+
Status get_schema(std::vector<ColumnDefinition>* schema) const override;
59+
std::unique_ptr<TableColumnMapper> create_column_mapper(
60+
TableColumnMapperOptions options) const override;
61+
Status open(std::shared_ptr<FileScanRequest> request) override;
62+
Status get_block(Block* block, size_t* rows, bool* eof) override;
63+
Status close() override;
64+
65+
private:
66+
void _init_profile() override;
67+
Status _next_batch();
68+
Status _open_next_stream();
69+
Status _prepare_arrays();
70+
71+
RuntimeState* _state = nullptr;
72+
const std::vector<SlotDescriptor*> _file_slots;
73+
const std::optional<TUniqueId> _load_id;
74+
std::unique_ptr<arrow::MemoryPool> _arrow_pool;
75+
std::vector<std::shared_ptr<arrow::DataType>> _arrow_types;
76+
std::vector<int> _arrow_positions;
77+
std::vector<std::shared_ptr<arrow::Array>> _arrays;
78+
std::shared_ptr<ArrowInputStream> _input;
79+
std::shared_ptr<arrow::ipc::RecordBatchStreamReader> _ipc_reader;
80+
std::shared_ptr<arrow::RecordBatch> _batch;
81+
int64_t _batch_offset = 0;
82+
RuntimeProfile::Counter* _total_time = nullptr;
83+
RuntimeProfile::Counter* _read_time = nullptr;
84+
RuntimeProfile::Counter* _materialize_time = nullptr;
85+
};
86+
87+
} // namespace doris::format::arrow_stream

‎be/src/format_v2/delimited_text/csv_reader.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
#include "format_v2/delimited_text/csv_reader.h"
1919

2020
#include <cstring>
21+
#include <limits>
2122
#include <utility>
2223

2324
#include "core/assert_cast.h"
@@ -120,7 +121,9 @@ Status CsvReader::_init_format_state() {
120121
_empty_field_as_null = text_params.empty_field_as_null;
121122
}
122123
if (_scan_params->file_attributes.hive_open_csv) {
123-
size_t fields = _source_file_slot_descs.size();
124+
// A probe precedes slot planning; a zero OpenCSV field limit would hide every column.
125+
size_t fields =
126+
_schema_only ? std::numeric_limits<size_t>::max() : _source_file_slot_descs.size();
124127
for (int index : _scan_params->column_idxs) {
125128
fields = std::max(fields, static_cast<size_t>(index) + 1);
126129
}

‎be/src/format_v2/delimited_text/delimited_text_reader.cpp‎

Lines changed: 47 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@
4343
#include "runtime/runtime_state.h"
4444
#include "util/decompressor.h"
4545
#include "util/string_util.h"
46+
#include "util/utf8_check.h"
4647

4748
namespace doris::format {
4849
namespace {
@@ -242,7 +243,7 @@ Status DelimitedTextReader::init(RuntimeState* state) {
242243
// each physical file slot maps to a field ordinal in the text row. The local id exposed in the
243244
// file schema is therefore the text-field ordinal, not the slot vector position.
244245
_source_column_idxs.clear();
245-
if (_scan_params->__isset.column_idxs && !_scan_params->column_idxs.empty()) {
246+
if (!_schema_only && _scan_params->__isset.column_idxs && !_scan_params->column_idxs.empty()) {
246247
if (_scan_params->column_idxs.size() != _source_file_slot_descs.size()) {
247248
return Status::InvalidArgument(
248249
"{} v2 reader column_idxs size {} does not match file slot size {}",
@@ -281,6 +282,49 @@ Status DelimitedTextReader::init(RuntimeState* state) {
281282
return Status::OK();
282283
}
283284

285+
Status DelimitedTextReader::infer_schema(RuntimeState* state,
286+
std::vector<ColumnDefinition>* schema) {
287+
_schema_only = true;
288+
RETURN_IF_ERROR(init(state));
289+
if (_file_description->range_start_offset != 0) {
290+
return Status::InvalidArgument("Schema inference requires a zero start offset");
291+
}
292+
if (_scan_params->file_type == TFileType::FILE_BROKER) {
293+
return Status::NotSupported("Delimited schema inference does not support broker load");
294+
}
295+
RETURN_IF_ERROR(_open_file());
296+
RETURN_IF_ERROR(_create_decompressor());
297+
RETURN_IF_ERROR(_create_line_reader());
298+
// A header is the schema input, not a data record to skip. For example,
299+
// csv_with_names must return "id,name" rather than the first row's values.
300+
_skip_lines = 0;
301+
Slice line;
302+
bool eof = false;
303+
RETURN_IF_ERROR(_read_next_line(&line, &eof));
304+
const auto header = to_lower(_scan_params->file_attributes.header_type);
305+
const bool has_names =
306+
header == BeConsts::CSV_WITH_NAMES || header == BeConsts::CSV_WITH_NAMES_AND_TYPES;
307+
if (eof || line.empty()) {
308+
// Empty compressed files reach this same check after decompression. Preserve the
309+
// schema RPC's existing diagnostics for callers such as the empty-LZOP TVF case.
310+
return Status::InternalError("The first line is empty, can not parse column {}",
311+
has_names ? "names" : "numbers");
312+
}
313+
// TEXT scans allow raw bytes, but schema RPCs must retain their UTF-8 validation contract
314+
// (for example, a header containing 0xFF must fail unless validation is explicitly disabled).
315+
if (!validate_utf8(*_scan_params, line.data, line.size)) {
316+
return Status::InternalError("Only support csv data in utf8 codec");
317+
}
318+
_split_line(line);
319+
schema->clear();
320+
for (size_t i = 0; i < _split_values.size(); ++i) {
321+
const auto name = has_names ? _split_values[i].to_string() : fmt::format("c{}", i + 1);
322+
schema->push_back(synthetic_file_child(
323+
name, make_nullable(std::make_shared<DataTypeString>()), cast_set<int32_t>(i)));
324+
}
325+
return Status::OK();
326+
}
327+
284328
Status DelimitedTextReader::get_schema(std::vector<ColumnDefinition>* file_schema) const {
285329
SCOPED_TIMER(_text_profile.total_time);
286330
if (file_schema == nullptr) {
@@ -556,8 +600,8 @@ Status DelimitedTextReader::_open_file() {
556600
// Stream load/http_stream data lives in NewLoadStreamMgr rather than a filesystem. The
557601
// generic FileFactory path only supports real file systems, so FILE_STREAM must use the
558602
// same pipe-reader lookup as the old CSV reader.
559-
RETURN_IF_ERROR(FileFactory::create_pipe_reader(*_stream_load_id, &_file_reader,
560-
_runtime_state, /*need_schema=*/false));
603+
RETURN_IF_ERROR(FileFactory::create_pipe_reader(
604+
*_stream_load_id, &_file_reader, _runtime_state, /*need_schema=*/_schema_only));
561605
} else {
562606
auto reader_options = FileFactory::get_reader_options(_runtime_state->query_options(),
563607
*_file_description);

‎be/src/format_v2/delimited_text/delimited_text_reader.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,8 @@ class DelimitedTextReader : public FileReader {
4848
~DelimitedTextReader() override;
4949

5050
Status init(RuntimeState* state) override;
51+
// Discover the first record without FE slots or skipping a configured header.
52+
Status infer_schema(RuntimeState* state, std::vector<ColumnDefinition>* schema);
5153
Status get_schema(std::vector<ColumnDefinition>* file_schema) const override;
5254
std::unique_ptr<TableColumnMapper> create_column_mapper(
5355
TableColumnMapperOptions options) const override;
@@ -135,6 +137,7 @@ class DelimitedTextReader : public FileReader {
135137
const uint8_t* _remove_bom(const uint8_t* ptr, size_t* size);
136138
void _init_profile() override;
137139

140+
bool _schema_only = false;
138141
const TFileScanRangeParams* _scan_params = nullptr;
139142
std::vector<SlotDescriptor*> _source_file_slot_descs;
140143
std::vector<int32_t> _source_column_idxs;

‎be/src/format_v2/json/json_reader.cpp‎

Lines changed: 82 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
#include "format_v2/json/json_reader.h"
1919

2020
#include <rapidjson/document.h>
21+
#include <rapidjson/error/en.h>
2122

2223
#include <algorithm>
2324
#include <cstring>
@@ -249,6 +250,83 @@ Status JsonReader::init(RuntimeState* state) {
249250
return Status::OK();
250251
}
251252

253+
Status JsonReader::infer_schema(RuntimeState* state, std::vector<ColumnDefinition>* schema) {
254+
RETURN_IF_ERROR(init(state));
255+
_reader_range = _json_range();
256+
RETURN_IF_ERROR(_open_file_reader(/*need_schema=*/true));
257+
RETURN_IF_ERROR(_create_decompressor());
258+
if (_read_json_by_line) {
259+
RETURN_IF_ERROR(_create_line_reader());
260+
}
261+
RETURN_IF_ERROR(_parse_jsonpath_and_json_root());
262+
if (_reader_range.size <= 0 && _scan_params->file_type != TFileType::FILE_STREAM) {
263+
_reader_range.size = cast_set<int64_t>(_physical_file_reader->size()) - _current_offset;
264+
}
265+
size_t size = 0;
266+
bool eof = false;
267+
RETURN_IF_ERROR(_read_one_document(&size, &eof));
268+
if (eof || size == 0) {
269+
return Status::EndOfFile("Empty file");
270+
}
271+
// Schema probing retains the RPC's DOM semantics (including json_root before outer-array
272+
// selection); normal scans continue to use the streaming v2 materializer.
273+
rapidjson::Document document;
274+
const bool parse_error = _num_as_string
275+
? document.Parse<rapidjson::kParseNumbersAsStringsFlag>(
276+
_document_view.data(), size)
277+
.HasParseError()
278+
: document.Parse(_document_view.data(), size).HasParseError();
279+
if (parse_error) {
280+
return Status::DataQualityError(
281+
"Parse json data for JsonDoc failed. code: {}, error info: {}",
282+
document.GetParseError(), rapidjson::GetParseError_En(document.GetParseError()));
283+
}
284+
rapidjson::Value* value = &document;
285+
if (!_parsed_json_root.empty()) {
286+
value = JsonFunctions::get_json_object_from_parsed_json(_parsed_json_root, &document,
287+
document.GetAllocator());
288+
if (value == nullptr) {
289+
return Status::DataQualityError("JSON Root not found");
290+
}
291+
}
292+
if (value->IsArray() && !_strip_outer_array) {
293+
return Status::DataQualityError(
294+
"JSON data is array-object, `strip_outer_array` must be TRUE.");
295+
}
296+
if (!value->IsArray() && _strip_outer_array) {
297+
return Status::DataQualityError(
298+
"JSON data is not an array-object, `strip_outer_array` must be FALSE.");
299+
}
300+
if (value->IsArray()) {
301+
if (value->Empty()) {
302+
return Status::InternalError("Empty first json line");
303+
}
304+
value = &(*value)[0];
305+
}
306+
if (!value->IsObject()) {
307+
return Status::DataQualityError("JSON data is not an object");
308+
}
309+
schema->clear();
310+
const auto append = [&](const std::string& name) {
311+
schema->push_back(synthetic_file_child(name,
312+
make_nullable(std::make_shared<DataTypeString>()),
313+
cast_set<int32_t>(schema->size())));
314+
};
315+
if (!_parsed_jsonpaths.empty()) {
316+
for (const auto& path : _parsed_jsonpaths) {
317+
if (path.empty()) {
318+
return Status::InvalidArgument("Invalid jsonpaths");
319+
}
320+
append(path.back().key);
321+
}
322+
} else {
323+
for (const auto& member : value->GetObject()) {
324+
append(std::string(member.name.GetString(), member.name.GetStringLength()));
325+
}
326+
}
327+
return Status::OK();
328+
}
329+
252330
Status JsonReader::get_schema(std::vector<ColumnDefinition>* file_schema) const {
253331
SCOPED_TIMER(_total_time);
254332
if (file_schema == nullptr) {
@@ -444,7 +522,7 @@ TFileRangeDesc JsonReader::_json_range() const {
444522
return range;
445523
}
446524

447-
Status JsonReader::_open_file_reader() {
525+
Status JsonReader::_open_file_reader(bool need_schema) {
448526
_current_offset = _reader_range.start_offset;
449527
if (_current_offset != 0) {
450528
// Include the whole delimiter when the split starts inside it, so skipping the first
@@ -456,7 +534,7 @@ Status JsonReader::_open_file_reader() {
456534
return Status::InvalidArgument("JSON v2 stream reader requires load id");
457535
}
458536
RETURN_IF_ERROR(FileFactory::create_pipe_reader(*_stream_load_id, &_physical_file_reader,
459-
_runtime_state, /*need_schema=*/false));
537+
_runtime_state, need_schema));
460538
} else {
461539
_file_description->mtime =
462540
_reader_range.__isset.modification_time ? _reader_range.modification_time : 0;
@@ -683,10 +761,8 @@ Status JsonReader::_extract_json_value(size_t size, bool* eof, bool* is_empty_ro
683761
return Status::DataQualityError(
684762
"JSON data is not an array-object, `strip_outer_array` must be FALSE.");
685763
}
686-
if (!_parsed_jsonpaths.empty() && _strip_outer_array &&
687-
_json_value.count_elements().value() == 0) {
688-
*is_empty_row = true;
689-
}
764+
// A rooted ondemand array has a single cursor. Pre-counting {"rows":[...]} here
765+
// consumes it before jsonpaths materialization; the append path detects empty arrays.
690766
return Status::OK();
691767
}
692768

‎be/src/format_v2/json/json_reader.h‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ class JsonReader final : public FileReader {
9797

9898
// Initializes scan attributes and builds the synthetic schema from FE slots.
9999
Status init(RuntimeState* state) override;
100+
Status infer_schema(RuntimeState* state, std::vector<ColumnDefinition>* schema);
100101
Status get_schema(std::vector<ColumnDefinition>* file_schema) const override;
101102
std::unique_ptr<TableColumnMapper> create_column_mapper(
102103
TableColumnMapperOptions options) const override;
@@ -129,7 +130,7 @@ class JsonReader final : public FileReader {
129130
std::vector<RequestedColumn>* columns) const;
130131
// Reconciles TableReader's split/range descriptor with FileReader's concrete file description.
131132
TFileRangeDesc _json_range() const;
132-
Status _open_file_reader();
133+
Status _open_file_reader(bool need_schema = false);
133134
Status _create_decompressor();
134135
Status _create_line_reader();
135136
Status _parse_jsonpath_and_json_root();

0 commit comments

Comments
 (0)