Skip to content

Commit 85be500

Browse files
authored
fix: document_stream::truncated_bytes() returns 0 on empty input (#2887) (#2889)
Stage 1 returns EMPTY on zero-length input before it writes the index sentinels that truncated_bytes() subtracts, so on a reused parser an empty stream reported the previous stream's value (possibly wrapped to 4294967295). Guard len == 0 in both the DOM and On-Demand streams; this also covers "[]" in comma_delimited_array mode and BOM-only input. Docs: state the empty-input behavior, list comma_delimited_array among the formats where the value is meaningless, turn the conditions into a list, and remove the duplicated paragraph in parse_many.md. Fixes #2887
1 parent 8c512a3 commit 85be500

8 files changed

Lines changed: 96 additions & 7 deletions

File tree

‎doc/iterate_many.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -385,7 +385,13 @@ This will print:
385385
39 bytes
386386
```
387387
388-
Importantly, you should only call `truncated_bytes()` after iterating through all of the documents since the stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet. Further, it is only applicable if the format is `whitespace_delimited` or `newline_delimited`. In `json_sequence` and `comma_delimited` mode, the value is meaningless. Importantly, it assumes no document reported an error. Iteration stops at the first failed document, which can leave the bookkeeping from a mid-stream batch.
388+
The value returned by `truncated_bytes()` is only meaningful when all of the following hold:
389+
390+
- You have iterated through all of the documents. The stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet.
391+
- The format is `whitespace_delimited` or `newline_delimited`. In `json_sequence`, `comma_delimited` and `comma_delimited_array` mode, the value is meaningless, even when every document was parsed.
392+
- No document reported an error. Iteration stops at the first failed document, which can leave the bookkeeping from a mid-stream batch.
393+
394+
Outside these conditions, the value is not merely imprecise: it is arbitrary, and it may exceed `size_in_bytes()` or wrap around to a huge value such as `4294967295`. An empty input (zero bytes) or an input made only of white space contains no document, and `truncated_bytes()` returns zero for it.
389395
390396
If you need to detect a truncated tail outside those conditions, track it yourself from the last document that parsed successfully, using `current_index()` and `source()` on the iterator.
391397

‎doc/parse_many.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -319,9 +319,13 @@ Consider the following example where a truncated document (`{"key":"intentionall
319319
```
320320
321321
322-
Importantly, you should only call `truncated_bytes()` after iterating through all of the documents since the stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet.
322+
The value returned by `truncated_bytes()` is only meaningful when all of the following hold:
323323
324-
Importantly, you should only call `truncated_bytes()` after iterating through all of the documents since the stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet. Further, it is only applicable if the format is `whitespace_delimited` or `newline_delimited`. In `json_sequence` and `comma_delimited` mode, the value is meaningless. Importantly, it assumes no document reported an error. Iteration stops at the first failed document, which can leave the bookkeeping from a mid-stream batch.
324+
- You have iterated through all of the documents. The stream cannot tell whether there are truncated documents at the very end when it may not have accessed that part of the data yet.
325+
- The format is `whitespace_delimited` or `newline_delimited`. In `json_sequence`, `comma_delimited` and `comma_delimited_array` mode, the value is meaningless, even when every document was parsed.
326+
- No document reported an error. Iteration stops at the first failed document, which can leave the bookkeeping from a mid-stream batch.
327+
328+
Outside these conditions, the value is not merely imprecise: it is arbitrary, and it may exceed `size_in_bytes()` or wrap around to a huge value such as `4294967295`. An empty input (zero bytes) or an input made only of white space contains no document, and `truncated_bytes()` returns zero for it.
325329
326330
327331
If you need to detect a truncated tail outside those conditions, track it yourself from the last document that parsed successfully, using `current_index()` and `source()` on the iterator.

‎include/simdjson/dom/document_stream-inl.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -277,6 +277,9 @@ inline size_t document_stream::size_in_bytes() const noexcept {
277277
}
278278

279279
inline size_t document_stream::truncated_bytes() const noexcept {
280+
// Stage 1 returns EMPTY on zero-length input before it writes the index
281+
// sentinels read below, so they would still hold a previous stream's values.
282+
if (len == 0) { return 0; }
280283
if(error == CAPACITY) { return len - batch_start; }
281284
return parser->implementation->structural_indexes[parser->implementation->n_structural_indexes] - parser->implementation->structural_indexes[parser->implementation->n_structural_indexes + 1];
282285
}

‎include/simdjson/dom/document_stream.h‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -114,16 +114,20 @@ class document_stream {
114114
* Check it only when all of the following hold:
115115
*
116116
* - the format is whitespace_delimited or newline_delimited. In
117-
* json_sequence and comma_delimited mode the stage-1 filter rewrites the
118-
* structural index in place and the bookkeeping is lost, so the value is
119-
* meaningless even for a stream that parsed completely;
117+
* json_sequence, comma_delimited and comma_delimited_array mode the
118+
* stage-1 filter rewrites the structural index in place and the
119+
* bookkeeping is lost, so the value is meaningless even for a stream
120+
* that parsed completely;
120121
* - you iterated all the way to the end of the stream;
121122
* - no document reported an error. Iteration stops at the first failed
122123
* document, which can leave the bookkeeping from a mid-stream batch.
123124
*
124125
* If you need to know about a truncated tail outside those conditions, track
125126
* it yourself from the last successful document (see iterator::current_index()
126127
* and iterator::source()).
128+
*
129+
* An empty input (zero bytes) or an input made only of white space contains
130+
* no document: truncated_bytes() returns zero.
127131
*/
128132
inline size_t truncated_bytes() const noexcept;
129133
/**

‎include/simdjson/generic/ondemand/document_stream-inl.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,9 @@ inline size_t document_stream::size_in_bytes() const noexcept {
138138
}
139139

140140
inline size_t document_stream::truncated_bytes() const noexcept {
141+
// Stage 1 returns EMPTY on zero-length input before it writes the index
142+
// sentinels read below, so they would still hold a previous stream's values.
143+
if (len == 0) { return 0; }
141144
if(error == CAPACITY) { return len - batch_start; }
142145
return parser->implementation->structural_indexes[parser->implementation->n_structural_indexes] - parser->implementation->structural_indexes[parser->implementation->n_structural_indexes + 1];
143146
}

‎include/simdjson/generic/ondemand/document_stream.h‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,14 +118,19 @@ class document_stream {
118118
*
119119
* IMPORTANT: this value is only meaningful under the conditions below.
120120
*
121-
* - the format is whitespace_delimited or newline_delimited;
121+
* - the format is whitespace_delimited or newline_delimited. In
122+
* json_sequence, comma_delimited and comma_delimited_array mode the value
123+
* is meaningless even for a stream that parsed completely;
122124
* - you iterated all the way to the end of the stream;
123125
* - no document reported an error. Iteration stops at the first failed
124126
* document, which can leave the bookkeeping from a mid-stream batch.
125127
*
126128
* If you need to know about a truncated tail outside those conditions, track
127129
* it yourself from the last successful document (see iterator::current_index()
128130
* and iterator::source()).
131+
*
132+
* An empty input (zero bytes) or an input made only of white space contains
133+
* no document: truncated_bytes() returns zero.
129134
*/
130135
inline size_t truncated_bytes() const noexcept;
131136

‎tests/dom/document_stream_tests.cpp‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -591,6 +591,37 @@ namespace document_stream_tests {
591591
}
592592

593593

594+
// Issue 2887: an empty input must report zero truncated bytes, even when
595+
// the parser was previously used on a stream with a truncated tail.
596+
bool truncated_empty_stream_reused_parser() {
597+
std::cout << "Running " << __func__ << std::endl;
598+
simdjson::dom::parser parser;
599+
auto truncated_json = R"({"a":1} {"b":2} {"c":)"_padded;
600+
const std::pair<simdjson::padded_string, simdjson::stream_format> empty_inputs[] = {
601+
{""_padded, simdjson::stream_format::whitespace_delimited},
602+
{""_padded, simdjson::stream_format::newline_delimited},
603+
{""_padded, simdjson::stream_format::json_sequence},
604+
{""_padded, simdjson::stream_format::comma_delimited},
605+
{"[]"_padded, simdjson::stream_format::comma_delimited_array},
606+
{"\xEF\xBB\xBF"_padded, simdjson::stream_format::whitespace_delimited},
607+
};
608+
for (const auto &input : empty_inputs) {
609+
simdjson::dom::document_stream stream;
610+
ASSERT_SUCCESS( parser.parse_many(truncated_json).get(stream) );
611+
size_t count = 0;
612+
for (auto doc : stream) { ASSERT_SUCCESS( doc.error() ); count++; }
613+
ASSERT_EQUAL( count, 2 );
614+
ASSERT_EQUAL( stream.truncated_bytes(), 5 );
615+
616+
ASSERT_SUCCESS( parser.parse_many(input.first, simdjson::dom::DEFAULT_BATCH_SIZE, input.second).get(stream) );
617+
count = 0;
618+
for (auto doc : stream) { (void)doc; count++; }
619+
ASSERT_EQUAL( count, 0 );
620+
ASSERT_EQUAL( stream.truncated_bytes(), 0 );
621+
}
622+
return true;
623+
}
624+
594625
bool truncated_window() {
595626
std::cout << "Running " << __func__ << std::endl;
596627
// The last JSON document is
@@ -2310,6 +2341,7 @@ namespace document_stream_tests {
23102341
test_crazy_leading_spaces() &&
23112342
simple_example() &&
23122343
truncated_window() &&
2344+
truncated_empty_stream_reused_parser() &&
23132345
truncated_window_unclosed_string_in_object() &&
23142346
truncated_window_unclosed_string() &&
23152347
issue1307() &&

‎tests/ondemand/ondemand_document_stream_tests.cpp‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -530,6 +530,37 @@ namespace document_stream_tests {
530530
TEST_SUCCEED();
531531
}
532532

533+
// Issue 2887: an empty input must report zero truncated bytes, even when
534+
// the parser was previously used on a stream with a truncated tail.
535+
bool truncated_empty_stream_reused_parser() {
536+
TEST_START();
537+
ondemand::parser parser;
538+
auto truncated_json = R"({"a":1} {"b":2} {"c":)"_padded;
539+
const std::pair<padded_string, stream_format> empty_inputs[] = {
540+
{""_padded, stream_format::whitespace_delimited},
541+
{""_padded, stream_format::newline_delimited},
542+
{""_padded, stream_format::json_sequence},
543+
{""_padded, stream_format::comma_delimited},
544+
{"[]"_padded, stream_format::comma_delimited_array},
545+
{"\xEF\xBB\xBF"_padded, stream_format::whitespace_delimited},
546+
};
547+
for (const auto &input : empty_inputs) {
548+
ondemand::document_stream stream;
549+
ASSERT_SUCCESS(parser.iterate_many(truncated_json).get(stream));
550+
size_t counter{0};
551+
for (auto doc : stream) { ASSERT_SUCCESS(doc.error()); counter++; }
552+
ASSERT_EQUAL(counter, 2);
553+
ASSERT_EQUAL(stream.truncated_bytes(), 5);
554+
555+
ASSERT_SUCCESS(parser.iterate_many(input.first, ondemand::DEFAULT_BATCH_SIZE, input.second).get(stream));
556+
counter = 0;
557+
for (auto doc : stream) { (void)doc; counter++; }
558+
ASSERT_EQUAL(counter, 0);
559+
ASSERT_EQUAL(stream.truncated_bytes(), 0);
560+
}
561+
TEST_SUCCEED();
562+
}
563+
533564
bool truncated_complete_docs() {
534565
TEST_START();
535566
auto json = R"([1,2,3] {"1":1,"2":3,"4":4} [1,2] )"_padded;
@@ -2581,6 +2612,7 @@ bool run() {
25812612
source_test() &&
25822613
truncated() &&
25832614
truncated_complete_docs() &&
2615+
truncated_empty_stream_reused_parser() &&
25842616
truncated_unclosed_string() &&
25852617
small_window() &&
25862618
large_window() &&

0 commit comments

Comments
 (0)