Skip to content

fix: truncated_bytes() returns 0 on empty input (#2887) - #2889

Merged
lemire merged 1 commit into
masterfrom
fix-truncated-bytes-empty-stream
Sep 29, 2026
Merged

lemire merged 1 commit into
masterfrom
fix-truncated-bytes-empty-stream

Conversation

@lemire

@lemire lemire commented Sep 29, 2026

Copy link
Copy Markdown
Member

Fixes #2887.

Bug

When the input is empty, stage 1 returns EMPTY at json_structural_indexer.h:197 before it writes the index sentinels. truncated_bytes() subtracts those sentinels anyway, so on a reused parser an empty stream reports whatever the previous stream left behind. That can be a small stale count, or 4294967295 if the previous stream stopped mid-batch on an error. This matches the Arrow report ("only after repeated parser operations"). DOM parse_many has the same problem.

Only zero-length input is affected: whitespace-only input still runs the final stage-1 pass, which writes the sentinels. I checked 1, 200 and 5000 bytes of whitespace, with batch sizes 64 and 1 MB.

Fix

if (len == 0) { return 0; } in both truncated_bytes() implementations. This also covers [] in comma_delimited_array mode, BOM-only input, and a default-constructed On-Demand stream (null parser). truncated_bytes() is not on the parsing path, so throughput is unaffected.

Docs

  • parse_many.md had the "only call truncated_bytes() after iterating…" paragraph twice. The duplicate is removed.
  • In both guides, the conditions under which the value is meaningful are now a list. The text also says that outside those conditions the value is arbitrary (it can wrap), and that empty or whitespace-only input returns 0.
  • comma_delimited_array is now listed with json_sequence and comma_delimited as a format where the value is meaningless (it is rewritten to comma_delimited). The header comments are updated to match.

Tests

truncated_empty_stream_reused_parser in both the DOM and On-Demand suites. Each run first parses a stream with a truncated tail (expects 5), then an empty input in every format (expects 0). Without the fix, both tests fail with returned '5' (expected '0').

Stage 1 returns EMPTY on zero-length input before it writes the index
sentinels that truncated_bytes() subtracts, so on a reused parser an
empty stream reported the previous stream's value (possibly wrapped to
4294967295). Guard len == 0 in both the DOM and On-Demand streams; this
also covers "[]" in comma_delimited_array mode and BOM-only input.

Docs: state the empty-input behavior, list comma_delimited_array among
the formats where the value is meaningless, turn the conditions into a
list, and remove the duplicated paragraph in parse_many.md.

Fixes #2887
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

truncated_bytes() returns arbitrary values for empty streams

1 participant