Skip to content

Commit 07dbba8

Browse files
Gabriel39Your Name
authored andcommitted
[fix](be) Preserve JSON and WAL rows with empty file projections (#70154)
### What problem does this PR solve? Issue Number: close #69926 Problem Summary: Hive JSON queries that only need partition columns can return zero rows even when the selected files contain records. For example, `SELECT * ... WHERE dt = 'p1'` returns data while `SELECT COUNT(*) ... WHERE dt = 'p1'` returns zero. The V2 JSON reader derived its row count from the output block, which has no columns after physical field pruning. Writer-backed reader tests reproduce the same row-count loss for empty WAL projections. Track accepted JSON records independently of materialized columns, including batch progress, array expansion, error rollback, and ignore-malformed NULL rows. Preserve JSON-path record selection for empty projections, and let the existing TableReader materialize partition values using the returned row count. For WAL, save the deserialized source row count before materialization transfers column ownership, then pass it through the existing filter pipeline. This preserves the row count when no file columns are requested. The empty-projection reproduction is at the reader interface; production replay reachability of an empty projection remains unverified. Add ten JSON BE tests covering empty file schemas and projections, arrays, JSON roots/paths, malformed input, batch boundaries, EOF, and actual TableReader partition materialization. Add eight writer-backed WAL cases covering empty, single-column, and reordered full projections across multiple blocks, with and without schema discovery, plus repeated EOF reads. Add a WAL SQL regression that forces two original async-load transactions to abort, then verifies row counts, NULLs, aggregates, filters, and actual values after replay. Add a Hive SQL regression using the existing partitioned JSON fixture for COUNT, partition projections, grouping, and file-column filters with external COUNT pushdown enabled and disabled. ### Release note Fix incorrect zero counts and missing rows when querying only partition columns of Hive JSON tables. Preserve logical row counts for empty WAL reader projections. ### Check List (For Author) - Test - [x] Unit Test: 166 ASAN tests passed, including JSON reader, TableReader, WAL V2 reader, WAL reader/writer, and CSV/Text empty-projection controls. Before the JSON fix, nine of ten new tests failed; before the WAL fix, both empty-projection cases failed while six projection controls passed. - [x] WAL SQL regression: `test_group_commit_wal_row_count` passed twice on isolated Doris 4.1.5 with scanner V2 enabled. Generated the `.out` file with `run-regression-test.sh`, then reran for result comparison. Execution on master is pending CI. - [x] Hive SQL regression: added `test_hive_json_partition_only`; Groovy compilation passed. Full execution against the external Hive fixture is pending CI. - Behavior changed: - [x] Yes. Empty JSON and WAL file projections preserve logical row counts. Hive JSON partition-only queries produce the correct rows and aggregates. - Does this need documentation? - [x] No.
1 parent cbe3752 commit 07dbba8

8 files changed

Lines changed: 517 additions & 13 deletions

File tree

‎be/src/format_v2/json/json_reader.cpp‎

Lines changed: 41 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -384,22 +384,22 @@ Status JsonReader::get_block(Block* file_block, size_t* rows, bool* eof) {
384384
return Status::InternalError("JSON v2 reader is not open");
385385
}
386386

387-
const auto batch_size = _runtime_state->batch_size();
387+
const auto batch_size = cast_set<size_t>(_runtime_state->batch_size());
388388
const auto max_block_bytes = _runtime_state->preferred_block_size_bytes();
389389
*rows = 0;
390390
*eof = false;
391+
_logical_rows = 0;
391392
_missing_fields.clear();
392393

393-
while (file_block->rows() < batch_size && !_reader_eof &&
394-
file_block->bytes() < max_block_bytes) {
394+
while (_logical_rows < batch_size && !_reader_eof && file_block->bytes() < max_block_bytes) {
395395
if (_read_json_by_line && _skip_first_line) {
396396
RETURN_IF_ERROR(_line_reader->skip_split_prefix(
397397
_reader_range.start_offset, _line_delimiter, &_reader_eof, _io_ctx.get()));
398398
_skip_first_line = false;
399399
continue;
400400
}
401401

402-
const size_t original_rows = file_block->rows();
402+
const size_t original_rows = _logical_rows;
403403
size_t size = 0;
404404
bool is_empty_row = false;
405405
Status st = Status::OK();
@@ -428,12 +428,13 @@ Status JsonReader::get_block(Block* file_block, size_t* rows, bool* eof) {
428428
}
429429
// An ignored or empty JSON object can produce no row. Avoid spinning forever on a document
430430
// that was consumed but produced no materialized value.
431-
if (!is_empty_row && file_block->rows() == original_rows) {
431+
if (!is_empty_row && _logical_rows == original_rows) {
432432
break;
433433
}
434434
}
435435

436-
*rows = file_block->rows();
436+
*rows = _logical_rows;
437+
DCHECK(file_block->columns() == 0 || file_block->rows() == *rows);
437438
for (auto& [_, mask] : _missing_fields) {
438439
mask.resize(*rows, 0);
439440
}
@@ -794,7 +795,7 @@ Status JsonReader::_append_simple_json_rows(Block* block, bool* is_empty_row, bo
794795
_array_iter = _array.begin();
795796
while (_array_iter != _array.end()) {
796797
simdjson::ondemand::object object_value = (*_array_iter).get_object();
797-
const size_t row_start = block->rows();
798+
const size_t row_start = _logical_rows;
798799
const auto status = _set_column_values_from_object(&object_value, block, &valid);
799800
if (!status.ok()) {
800801
if (!_request->reject_invalid_rows) {
@@ -825,12 +826,12 @@ Status JsonReader::_append_flat_array_jsonpath_rows(Block* block, bool* is_empty
825826
DORIS_CHECK(block != nullptr);
826827
DORIS_CHECK(is_empty_row != nullptr);
827828
DORIS_CHECK(eof != nullptr);
828-
const size_t original_rows = block->rows();
829+
const size_t original_rows = _logical_rows;
829830
bool valid = true;
830831
_array = _json_value.get_array();
831832
_array_iter = _array.begin();
832833
while (_array_iter != _array.end()) {
833-
const size_t row_start = block->rows();
834+
const size_t row_start = _logical_rows;
834835
simdjson::ondemand::object object_value = (*_array_iter).get_object();
835836
if (!_parsed_from_json_root && !_parsed_json_root.empty()) {
836837
// Reject roots per element for loads, preserving earlier siblings and their presence
@@ -869,7 +870,7 @@ Status JsonReader::_append_flat_array_jsonpath_rows(Block* block, bool* is_empty
869870
}
870871
++_array_iter;
871872
}
872-
*is_empty_row = block->rows() == original_rows;
873+
*is_empty_row = _logical_rows == original_rows;
873874
return Status::OK();
874875
}
875876

@@ -893,7 +894,7 @@ Status JsonReader::_set_column_values_from_object(simdjson::ondemand::object* ob
893894
DORIS_CHECK(block != nullptr);
894895
DORIS_CHECK(valid != nullptr);
895896
std::vector<bool> seen_columns(block->columns(), false);
896-
const size_t cur_row_count = block->rows();
897+
const size_t cur_row_count = _logical_rows;
897898
bool has_valid_value = false;
898899
size_t key_index = 0;
899900

@@ -944,21 +945,46 @@ Status JsonReader::_set_column_values_from_object(simdjson::ondemand::object* ob
944945
}
945946
}
946947
*valid = true;
948+
++_logical_rows;
947949
if (!has_valid_value) {
948950
return Status::OK();
949951
}
950952
return Status::OK();
951953
}
952954

955+
Status JsonReader::_has_matching_jsonpath(simdjson::ondemand::object* object_value,
956+
bool* matches) const {
957+
*matches = false;
958+
for (const auto& path : _parsed_jsonpaths) {
959+
// A root path selects the current document without a field lookup.
960+
if (JsonFunctions::is_root_path(path)) {
961+
*matches = true;
962+
continue;
963+
}
964+
simdjson::ondemand::value value;
965+
const auto status = JsonFunctions::extract_from_object(*object_value, path, &value);
966+
if (!status.ok() && !status.is<ErrorCode::NOT_FOUND>()) {
967+
return status;
968+
}
969+
*matches = *matches || status.ok();
970+
}
971+
return Status::OK();
972+
}
973+
953974
Status JsonReader::_write_columns_by_jsonpath(simdjson::ondemand::object* object_value,
954975
Block* block, bool* valid) {
955976
DORIS_CHECK(object_value != nullptr);
956977
DORIS_CHECK(block != nullptr);
957978
DORIS_CHECK(valid != nullptr);
958979
bool has_valid_value = false;
959-
const size_t cur_row_count = block->rows();
980+
const size_t cur_row_count = _logical_rows;
960981
_cached_string_values.clear();
961982

983+
// Even without projected fields, jsonpaths still decides whether the object contributes a row.
984+
if (_requested_columns.empty()) {
985+
RETURN_IF_ERROR(_has_matching_jsonpath(object_value, &has_valid_value));
986+
}
987+
962988
for (size_t i = 0; i < _requested_columns.size(); ++i) {
963989
const auto& requested = _requested_columns[i];
964990
auto* column_ptr = block->get_by_position(i).column->assert_mutable().get();
@@ -1015,6 +1041,7 @@ Status JsonReader::_write_columns_by_jsonpath(simdjson::ondemand::object* object
10151041
return Status::OK();
10161042
}
10171043
*valid = true;
1044+
++_logical_rows;
10181045
return Status::OK();
10191046
}
10201047

@@ -1217,6 +1244,7 @@ Status JsonReader::_append_null_for_malformed_json(Block* block) {
12171244
assert_cast<ColumnNullable*>(column.get())->insert_default();
12181245
column_with_type.column = std::move(column);
12191246
}
1247+
++_logical_rows;
12201248
return Status::OK();
12211249
}
12221250

@@ -1228,6 +1256,7 @@ Status JsonReader::_handle_json_error(const Status& status, Block* block, size_t
12281256
// block to the row count before this document before either surfacing the error or appending
12291257
// the ignore-malformed null row.
12301258
_truncate_block_to_rows(block, original_rows);
1259+
_logical_rows = original_rows;
12311260
for (auto& [_, mask] : _missing_fields) {
12321261
mask.resize(original_rows, 0);
12331262
}

‎be/src/format_v2/json/json_reader.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,7 @@ class JsonReader final : public FileReader {
148148
Status _append_nested_jsonpath_row(Block* block, bool* is_empty_row, bool* eof);
149149
Status _set_column_values_from_object(simdjson::ondemand::object* object_value, Block* block,
150150
bool* valid);
151+
Status _has_matching_jsonpath(simdjson::ondemand::object* object_value, bool* matches) const;
151152
Status _write_columns_by_jsonpath(simdjson::ondemand::object* object_value, Block* block,
152153
bool* valid);
153154
template <bool use_string_cache>
@@ -191,6 +192,8 @@ class JsonReader final : public FileReader {
191192
std::unique_ptr<Decompressor> _decompressor;
192193
std::unique_ptr<NewPlainTextLineReader> _line_reader;
193194
int64_t _current_offset = 0;
195+
// Partition-only scans have no file columns, so Block::rows() cannot count their records.
196+
size_t _logical_rows = 0;
194197
bool _reader_eof = false;
195198
bool _skip_first_line = false;
196199
bool _single_document_read = false;

‎be/src/format_v2/wal/wal_reader.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,8 +195,11 @@ Status WalReader::get_block(Block* file_block, size_t* rows, bool* eof) {
195195
return Status::Corruption("WAL block has {} columns but header declares {}",
196196
source_block.columns(), _column_ids.size());
197197
}
198+
// Empty projections still contribute every WAL row. Capture the count before materialization
199+
// moves source columns into the output block and can leave the source with no owned columns.
200+
const size_t source_rows = source_block.rows();
198201
RETURN_IF_ERROR(_materialize_requested_columns(&source_block, file_block));
199-
*rows = file_block->rows();
202+
*rows = source_rows;
200203
_record_scan_rows(cast_set<int64_t>(*rows));
201204
RETURN_IF_ERROR(
202205
apply_materialized_reader_filters(_request.get(), _io_ctx.get(), file_block, rows));

‎be/test/format_v2/json/json_reader_test.cpp‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@
4343
#include "exprs/vexpr_context.h"
4444
#include "format/json/new_json_reader.h"
4545
#include "format_v2/column_data.h"
46+
#include "format_v2/table_reader.h"
4647
#include "io/io_common.h"
4748
#include "runtime/descriptors.h"
4849
#include "runtime/runtime_profile.h"
@@ -856,4 +857,165 @@ TEST(JsonReaderTest, AppliesDeleteAndNormalConjunctsWithPredicateFilterAccountin
856857
EXPECT_EQ(io_ctx->predicate_filtered_rows, 1);
857858
}
858859

860+
// Partition-only scans have no file-local output columns, but still need one logical row per
861+
// accepted JSON record so TableReader can materialize the partition constants.
862+
TEST(JsonReaderTest, EmptyFileProjectionPreservesRows) {
863+
ObjectPool pool;
864+
const auto slots = build_slots(&pool);
865+
for (const auto& source_slots : {slots, std::vector<SlotDescriptor*> {}}) {
866+
auto result = read_once("empty_projection.jsonl", "{\"id\":1}\n\n{}\n{\"id\":3}\n",
867+
json_scan_params(), source_slots, {}, true, true);
868+
ASSERT_TRUE(result.status.ok()) << result.status;
869+
EXPECT_EQ(result.block.columns(), 0);
870+
EXPECT_EQ(result.rows, 3);
871+
EXPECT_FALSE(result.eof);
872+
EXPECT_TRUE(result.second_status.ok());
873+
EXPECT_EQ(result.second_rows, 0);
874+
EXPECT_TRUE(result.second_eof);
875+
}
876+
}
877+
878+
TEST(JsonReaderTest, EmptyFileProjectionPreservesOuterArrayRows) {
879+
auto result = read_once("empty_projection_array.json", R"([{"id":1},{},{"id":3}])",
880+
json_scan_params(false, true), {}, {}, true, true);
881+
ASSERT_TRUE(result.status.ok()) << result.status;
882+
EXPECT_EQ(result.rows, 3);
883+
EXPECT_TRUE(result.second_status.ok());
884+
EXPECT_TRUE(result.second_eof);
885+
}
886+
887+
TEST(JsonReaderTest, EmptyFileProjectionPreservesJsonRootRows) {
888+
auto result = read_once("empty_projection_root.jsonl",
889+
"{\"payload\":{\"id\":1}}\n{\"payload\":{\"id\":2}}\n",
890+
json_scan_params(true, false, "", "$.payload"), {}, {}, true, true);
891+
ASSERT_TRUE(result.status.ok()) << result.status;
892+
EXPECT_EQ(result.rows, 2);
893+
EXPECT_TRUE(result.second_status.ok());
894+
EXPECT_TRUE(result.second_eof);
895+
}
896+
897+
TEST(JsonReaderTest, EmptyFileProjectionHonorsJsonPaths) {
898+
ObjectPool pool;
899+
const auto slots = build_slots(&pool);
900+
auto result = read_once("empty_projection_jsonpaths.jsonl",
901+
"{\"payload\":{\"id\":1}}\n{\"other\":2}\n{\"payload\":{\"id\":3}}\n",
902+
json_scan_params(true, false, R"(["$.payload.id","$.payload.user"])"),
903+
slots, {}, true, true);
904+
ASSERT_TRUE(result.status.ok()) << result.status;
905+
EXPECT_EQ(result.rows, 2);
906+
EXPECT_TRUE(result.second_status.ok());
907+
EXPECT_TRUE(result.second_eof);
908+
}
909+
910+
TEST(JsonReaderTest, EmptyFileProjectionPreservesRootJsonPathRows) {
911+
auto result = read_once("empty_projection_root_path.jsonl", "{\"id\":1}\n{}\n",
912+
json_scan_params(true, false, R"(["$."])"), {}, {}, true, true);
913+
ASSERT_TRUE(result.status.ok()) << result.status;
914+
EXPECT_EQ(result.rows, 2);
915+
EXPECT_TRUE(result.second_eof);
916+
}
917+
918+
TEST(JsonReaderTest, EmptyFileProjectionPreservesRootedArrayJsonPathRows) {
919+
auto result = read_once("empty_projection_rooted_array.json",
920+
R"({"rows":[{"payload":{"id":1}},{"other":2},{"payload":{"id":3}}]})",
921+
json_scan_params(false, true, R"(["$.payload.id"])", "$.rows"), {}, {},
922+
true, true);
923+
ASSERT_TRUE(result.status.ok()) << result.status;
924+
EXPECT_EQ(result.rows, 2);
925+
EXPECT_TRUE(result.second_status.ok());
926+
EXPECT_TRUE(result.second_eof);
927+
}
928+
929+
TEST(JsonReaderTest, EmptyFileProjectionPreservesMalformedNullRows) {
930+
auto result = read_once("empty_projection_malformed.jsonl", "not-json\n{\"id\":1}\n",
931+
json_scan_params(true, false, "", "", true), {}, {}, true, true);
932+
ASSERT_TRUE(result.status.ok()) << result.status;
933+
EXPECT_EQ(result.rows, 2);
934+
EXPECT_TRUE(result.second_status.ok());
935+
EXPECT_TRUE(result.second_eof);
936+
937+
auto strict = read_once("empty_projection_strict.jsonl", "not-json\n", json_scan_params(), {},
938+
{}, false, true);
939+
EXPECT_FALSE(strict.status.ok());
940+
}
941+
942+
TEST(JsonReaderTest, EmptyFileProjectionPreservesEmptyInput) {
943+
for (const std::string content : {"", "\n", "[]\n"}) {
944+
auto result = read_once("empty_projection_empty.jsonl", content,
945+
json_scan_params(true, true), {}, {}, false, true);
946+
ASSERT_TRUE(result.status.ok()) << result.status;
947+
EXPECT_EQ(result.rows, 0);
948+
EXPECT_TRUE(result.eof);
949+
}
950+
}
951+
952+
TEST(JsonReaderTest, EmptyFileProjectionHonorsBatchSize) {
953+
const auto path = write_json_file("empty_projection_batches.jsonl",
954+
"{\"id\":1}\n{\"id\":2}\n{\"id\":3}\n"
955+
"{\"id\":4}\n{\"id\":5}\n");
956+
auto properties = std::make_shared<io::FileSystemProperties>();
957+
properties->system_type = TFileType::FILE_LOCAL;
958+
auto desc = file_description(path.string());
959+
auto params = json_scan_params();
960+
RuntimeProfile profile("json_empty_projection_batches");
961+
MockRuntimeState state;
962+
state._batch_size = 2;
963+
JsonReader reader(properties, desc, std::make_shared<io::IOContext>(), &profile, &params,
964+
file_range(path), {});
965+
ASSERT_TRUE(reader.init(&state).ok());
966+
ASSERT_TRUE(reader.open(std::make_shared<FileScanRequest>()).ok());
967+
for (const size_t expected_rows : {2, 2, 1, 0}) {
968+
Block block;
969+
size_t rows = 0;
970+
bool eof = false;
971+
ASSERT_TRUE(reader.get_block(&block, &rows, &eof).ok());
972+
EXPECT_EQ(block.columns(), 0);
973+
EXPECT_EQ(rows, expected_rows);
974+
EXPECT_EQ(eof, expected_rows == 0);
975+
}
976+
}
977+
978+
TEST(JsonReaderTest, EmptyFileProjectionMaterializesPartitionRows) {
979+
const auto path =
980+
write_json_file("partition_only.jsonl", "{\"id\":1}\n{\"id\":2}\n{\"id\":3}\n");
981+
auto params = json_scan_params();
982+
const std::vector<SlotDescriptor*> file_slots;
983+
ColumnDefinition partition;
984+
partition.identifier = Field::create_field<TYPE_STRING>("dt");
985+
partition.local_id = 0;
986+
partition.name = "dt";
987+
partition.type = std::make_shared<DataTypeString>();
988+
partition.is_partition_key = true;
989+
RuntimeProfile profile("json_partition_only");
990+
MockRuntimeState state;
991+
state._batch_size = 2;
992+
TableReader reader;
993+
ASSERT_TRUE(reader.init({.projected_columns = {partition},
994+
.conjuncts = {},
995+
.format = FileFormat::JSON,
996+
.scan_params = &params,
997+
.io_ctx = std::make_shared<io::IOContext>(),
998+
.runtime_state = &state,
999+
.scanner_profile = &profile,
1000+
.file_slot_descs = &file_slots})
1001+
.ok());
1002+
SplitReadOptions split;
1003+
split.current_range = file_range(path);
1004+
split.current_split_format = FileFormat::JSON;
1005+
split.partition_values.emplace("dt", Field::create_field<TYPE_STRING>("p1"));
1006+
ASSERT_TRUE(reader.prepare_split(split).ok());
1007+
bool eos = false;
1008+
for (const size_t expected_rows : {2, 1, 0}) {
1009+
Block block({{partition.type->create_column(), partition.type, partition.name}});
1010+
ASSERT_TRUE(reader.get_block(&block, &eos).ok());
1011+
ASSERT_EQ(block.rows(), expected_rows);
1012+
EXPECT_EQ(eos, expected_rows == 0);
1013+
const auto column = block.get_by_position(0).column->convert_to_full_column_if_const();
1014+
for (size_t row = 0; row < expected_rows; ++row) {
1015+
EXPECT_EQ(column->get_data_at(row).to_string(), "p1");
1016+
}
1017+
}
1018+
EXPECT_TRUE(reader.close().ok());
1019+
}
1020+
8591021
} // namespace doris::format::json

0 commit comments

Comments
 (0)