You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7865646
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](be) Support Arrow loads and schema inference in FileScannerV2 (apache#69668)
### What problem does this PR solve?
Arrow Stream Load still selects FileScanner V1, and CSV/TEXT/JSON schema
discovery still constructs legacy readers even when the subsequent scan
uses V2.
Add a native Arrow IPC FileReader to the V2 load path. It decodes record
batches incrementally, supports concatenated streams and compressed IPC,
uses tracked allocations, and retains load expressions, defaults,
strict-mode checks, and WHERE filtering. Source timestamp units and
timezones are preserved for Doris SerDe rounding, including nested
values.
Stream Load plans explicitly set `enable_file_scanner_v2=true` in their
query options because these plans construct options independently of
query sessions.
Arrow binding preserves positional STRUCT children, DATE64 range
validation, and unsigned IPv4 address bits. Before checked recursive
casts, ancestor validity masks hide NULL-struct payloads and
NULL-list/map ranges are compacted. Ordinary numeric overflow checks
remain enabled.
A load-specific schema converter preserves native VARIANT carriers and
extensions recursively, including SQL NULLs, and binds AGG_STATE through
its actual serialization type. Fixed-length aggregate states use the
serialization column's byte width; invalid binary lengths fail before
SerDe materialization. Recursive NULL masking skips unchanged children
and leaves incompatible container kinds to explicit cast errors.
Nullable fixed-size-list inputs are viewed through 64-bit variable
offsets before compaction, so NULL parents drop complete child ranges
without synthesizing NULLs in non-nullable STRUCT fields or native
VARIANT carriers. DECIMALV2 retains decimal128 wire scales for its
existing scale-nine rescaling and truncation rules; other decimal types
retain checked conversions.
Keep the existing empty-file schema RPC diagnostics, including empty
LZOP data and header-name versus inferred-column-number errors.
Route CSV, TEXT, and JSON schema discovery through V2 readers and the
shared schema entry point, alongside the existing native Parquet/ORC
readers. Preserve header handling, nullable string inference, JSON
root/path options, and independent sampling of HTTP Stream Load data.
Also avoid consuming a rooted JSON array's ondemand cursor before
jsonpaths materialization.
### Release note
Arrow Stream Load and CSV/TEXT/JSON schema discovery now use FileScanner
V2 readers.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test
- ASAN BE build and 221 focused unit tests passed, including scanner
routing, native schema inference, Arrow IPC loading, and existing reader
coverage. Focused cases cover positional STRUCT names, DATE64 range
checks, unsigned IPv4, NULL container payloads, numeric overflow, native
VARIANT storage/extensions, aggregate-state widths, incompatible
container input, empty LZOP, NULL fixed-size lists with non-nullable
descendants, and DECIMALV2 rescaling with DECIMALV3 rejection controls.
- Six regression suites passed against a locally built FE/ASAN BE: Arrow
Stream Load, V2 HTTP schema discovery, and existing JSON, compressed
CSV, CSV-with-names, and CSV-with-names-and-types suites. Results were
compared with expected output.
- clang-format 16, changed-line clang-tidy, and build hygiene checks
passed.
- FE Checkstyle and the FE production build passed after adding the
explicit Stream Load query option.
- Behavior changed:
- [x] Yes. Arrow loads and text/JSON schema inference use native V2
readers.
- Does this need documentation?
- [x] No. Existing load interfaces and options are retained.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
0 commit comments