You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Physical plan proto silently drops CSV/JSON scan options (newline_delimited, terminator, compression) #24180
The physical plan protobuf encoding for CSV and JSON scans silently drops several
scan options. They are not represented in CsvScanExecNode / JsonScanExecNode
(or in the shared FileScanExecConf), so encoding succeeds without error and the
decoded plan reads the data differently from the original:
Option
Encoded?
Decodes as
JsonSource::newline_delimited
no
always true (NDJSON)
CsvSource::terminator
no
None (default \n)
FileScanConfig::file_compression_type (CSV, JSON)
no
UNCOMPRESSED
For newline_delimited and file_compression_type this is a correctness problem,
not just a lost setting: a JSON-array file decoded as NDJSON, or a gzipped file
decoded as uncompressed, produces wrong results or a parse error on the remote
node rather than a serialization failure at the sender.
This is pre-existing behavior, not a regression. It was surfaced while migrating
these sources onto the per-plan proto hooks (#23494), where it is now recorded in
doc comments on CsvSource::try_from_proto and JsonSource::try_from_proto
(#24177, #24178).
To Reproduce
Round-trip a DataSourceExec through PhysicalPlanNode encode/decode with any of:
Add the missing fields to the proto messages (optional bool newline_delimited
on JsonScanExecNode, optional string terminator on CsvScanExecNode, and a
compression field for CSV/JSON) and encode/decode them. Making them optional
keeps the wire format backward compatible: an absent field decodes to today's
default.
At minimum, return an error from try_to_proto when a non-default value is set,
so the loss is loud instead of silent.
(1) is preferable — the options are cheap to encode and (2) turns a working local
plan into a hard failure the moment it is serialized.
Activity
buraksenn commentedon Aug 8, 2026
take
alamb commentedon Aug 8, 2026
Is this a newly found bug or a new introduced bug (regression)?
I am going through the issues for the 55 release and seeing what, if anything, we should wait for
55.0.0(Jul / Aug 2026) #22393adriangb commentedon Aug 8, 2026
This is a newly found bug that exists in 54 (and likely way before that too)
fix(proto): preserve CSV/JSON scan options on roundtrip (apache#24233)