You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 85be500
Browse filesBrowse the repository at this point in the historyBrowse files
fix: document_stream::truncated_bytes() returns 0 on empty input (#2887) (#2889)
Stage 1 returns EMPTY on zero-length input before it writes the index
sentinels that truncated_bytes() subtracts, so on a reused parser an
empty stream reported the previous stream's value (possibly wrapped to
4294967295). Guard len == 0 in both the DOM and On-Demand streams; this
also covers "[]" in comma_delimited_array mode and BOM-only input.
Docs: state the empty-input behavior, list comma_delimited_array among
the formats where the value is meaningless, turn the conditions into a
list, and remove the duplicated paragraph in parse_many.md.
Fixes#2887
Copy file name to clipboardExpand all lines: doc/iterate_many.md
+7-1Lines changed: 7 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -385,7 +385,13 @@ This will print:
385
385
39 bytes
386
386
```
387
387
388
-
Importantly, you should only call `truncated_bytes()` after iterating through all of the documents since the stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet. Further, it is only applicable if the format is `whitespace_delimited` or `newline_delimited`. In `json_sequence` and `comma_delimited` mode, the value is meaningless. Importantly, it assumes no document reported an error. Iteration stops at the first failed document, which can leave the bookkeeping from a mid-stream batch.
388
+
The value returned by `truncated_bytes()` is only meaningful when all of the following hold:
389
+
390
+
- You have iterated through all of the documents. The stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet.
391
+
- The format is `whitespace_delimited` or `newline_delimited`. In `json_sequence`, `comma_delimited` and `comma_delimited_array` mode, the value is meaningless, even when every document was parsed.
392
+
- No document reported an error. Iteration stops at the first failed document, which can leave the bookkeeping from a mid-stream batch.
393
+
394
+
Outside these conditions, the value is not merely imprecise: it is arbitrary, and it may exceed `size_in_bytes()` or wrap around to a huge value such as `4294967295`. An empty input (zero bytes) or an input made only of white space contains no document, and `truncated_bytes()` returns zero for it.
389
395
390
396
If you need to detect a truncated tail outside those conditions, track it yourself from the last document that parsed successfully, using `current_index()` and `source()` on the iterator.
Copy file name to clipboardExpand all lines: doc/parse_many.md
+6-2Lines changed: 6 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -319,9 +319,13 @@ Consider the following example where a truncated document (`{"key":"intentionall
319
319
```
320
320
321
321
322
-
Importantly, you should only call `truncated_bytes()` after iterating through all of the documents since the stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet.
322
+
The value returned by `truncated_bytes()` is only meaningful when all of the following hold:
323
323
324
-
Importantly, you should only call `truncated_bytes()` after iterating through all of the documents since the stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet. Further, it is only applicable if the format is `whitespace_delimited` or `newline_delimited`. In `json_sequence` and `comma_delimited` mode, the value is meaningless. Importantly, it assumes no document reported an error. Iteration stops at the first failed document, which can leave the bookkeeping from a mid-stream batch.
324
+
- You have iterated through all of the documents. The stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet.
325
+
- The format is `whitespace_delimited` or `newline_delimited`. In `json_sequence`, `comma_delimited` and `comma_delimited_array` mode, the value is meaningless, even when every document was parsed.
326
+
- No document reported an error. Iteration stops at the first failed document, which can leave the bookkeeping from a mid-stream batch.
327
+
328
+
Outside these conditions, the value is not merely imprecise: it is arbitrary, and it may exceed `size_in_bytes()` or wrap around to a huge value such as `4294967295`. An empty input (zero bytes) or an input made only of white space contains no document, and `truncated_bytes()` returns zero for it.
325
329
326
330
327
331
If you need to detect a truncated tail outside those conditions, track it yourself from the last document that parsed successfully, using `current_index()` and `source()` on the iterator.
0 commit comments