Skip to content

Commit 4bde8d0

Browse files
authored
[fix](be) Preserve JSON and WAL rows with empty file projections (apache#70154)
### What problem does this PR solve? Issue Number: close apache#69926 Problem Summary: Hive JSON queries that only need partition columns can return zero rows even when the selected files contain records. For example, `SELECT * ... WHERE dt = 'p1'` returns data while `SELECT COUNT(*) ... WHERE dt = 'p1'` returns zero. The V2 JSON reader derived its row count from the output block, which has no columns after physical field pruning. Writer-backed reader tests reproduce the same row-count loss for empty WAL projections. Track accepted JSON records independently of materialized columns, including batch progress, array expansion, error rollback, and ignore-malformed NULL rows. Preserve JSON-path record selection for empty projections, and let the existing TableReader materialize partition values using the returned row count. For WAL, save the deserialized source row count before materialization transfers column ownership, then pass it through the existing filter pipeline. This preserves the row count when no file columns are requested. The empty-projection reproduction is at the reader interface; production replay reachability of an empty projection remains unverified. Add ten JSON BE tests covering empty file schemas and projections, arrays, JSON roots/paths, malformed input, batch boundaries, EOF, and actual TableReader partition materialization. Add eight writer-backed WAL cases covering empty, single-column, and reordered full projections across multiple blocks, with and without schema discovery, plus repeated EOF reads. Add a WAL SQL regression that forces two original async-load transactions to abort, then verifies row counts, NULLs, aggregates, filters, and actual values after replay. Add a Hive SQL regression using the existing partitioned JSON fixture for COUNT, partition projections, grouping, and file-column filters with external COUNT pushdown enabled and disabled. ### Release note Fix incorrect zero counts and missing rows when querying only partition columns of Hive JSON tables. Preserve logical row counts for empty WAL reader projections. ### Check List (For Author) - Test - [x] Unit Test: 166 ASAN tests passed, including JSON reader, TableReader, WAL V2 reader, WAL reader/writer, and CSV/Text empty-projection controls. Before the JSON fix, nine of ten new tests failed; before the WAL fix, both empty-projection cases failed while six projection controls passed. - [x] WAL SQL regression: `test_group_commit_wal_row_count` passed twice on isolated Doris 4.1.5 with scanner V2 enabled. Generated the `.out` file with `run-regression-test.sh`, then reran for result comparison. Execution on master is pending CI. - [x] Hive SQL regression: added `test_hive_json_partition_only`; Groovy compilation passed. Full execution against the external Hive fixture is pending CI. - Behavior changed: - [x] Yes. Empty JSON and WAL file projections preserve logical row counts. Hive JSON partition-only queries produce the correct rows and aggregates. - Does this need documentation? - [x] No.
1 parent 75ab5ba commit 4bde8d0

8 files changed

Lines changed: 517 additions & 13 deletions

File tree

‎be/src/format_v2/json/json_reader.cpp‎

Lines changed: 41 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -385,22 +385,22 @@ Status JsonReader::get_block(Block* file_block, size_t* rows, bool* eof) {
385385
return Status::InternalError("JSON v2 reader is not open");
386386
}
387387

388-
const auto batch_size = _runtime_state->batch_size();
388+
const auto batch_size = cast_set<size_t>(_runtime_state->batch_size());
389389
const auto max_block_bytes = _runtime_state->preferred_block_size_bytes();
390390
*rows = 0;
391391
*eof = false;
392+
_logical_rows = 0;
392393
_missing_fields.clear();
393394

394-
while (file_block->rows() < batch_size && !_reader_eof &&
395-
file_block->bytes() < max_block_bytes) {
395+
while (_logical_rows < batch_size && !_reader_eof && file_block->bytes() < max_block_bytes) {
396396
if (_read_json_by_line && _skip_first_line) {
397397
RETURN_IF_ERROR(_line_reader->skip_split_prefix(
398398
_reader_range.start_offset, _line_delimiter, &_reader_eof, _io_ctx.get()));
399399
_skip_first_line = false;
400400
continue;
401401
}
402402

403-
const size_t original_rows = file_block->rows();
403+
const size_t original_rows = _logical_rows;
404404
size_t size = 0;
405405
bool is_empty_row = false;
406406
Status st = Status::OK();
@@ -429,12 +429,13 @@ Status JsonReader::get_block(Block* file_block, size_t* rows, bool* eof) {
429429
}
430430
// An ignored or empty JSON object can produce no row. Avoid spinning forever on a document
431431
// that was consumed but produced no materialized value.
432-
if (!is_empty_row && file_block->rows() == original_rows) {
432+
if (!is_empty_row && _logical_rows == original_rows) {
433433
break;
434434
}
435435
}
436436

437-
*rows = file_block->rows();
437+
*rows = _logical_rows;
438+
DCHECK(file_block->columns() == 0 || file_block->rows() == *rows);
438439
for (auto& [_, mask] : _missing_fields) {
439440
mask.resize(*rows, 0);
440441
}
@@ -810,7 +811,7 @@ Status JsonReader::_append_simple_json_rows(Block* block, bool* is_empty_row, bo
810811
_array_iter = _array.begin();
811812
while (_array_iter != _array.end()) {
812813
simdjson::ondemand::object object_value = (*_array_iter).get_object();
813-
const size_t row_start = block->rows();
814+
const size_t row_start = _logical_rows;
814815
const auto status = _set_column_values_from_object(&object_value, block, &valid);
815816
if (!status.ok()) {
816817
if (!_request->reject_invalid_rows) {
@@ -841,12 +842,12 @@ Status JsonReader::_append_flat_array_jsonpath_rows(Block* block, bool* is_empty
841842
DORIS_CHECK(block != nullptr);
842843
DORIS_CHECK(is_empty_row != nullptr);
843844
DORIS_CHECK(eof != nullptr);
844-
const size_t original_rows = block->rows();
845+
const size_t original_rows = _logical_rows;
845846
bool valid = true;
846847
_array = _json_value.get_array();
847848
_array_iter = _array.begin();
848849
while (_array_iter != _array.end()) {
849-
const size_t row_start = block->rows();
850+
const size_t row_start = _logical_rows;
850851
simdjson::ondemand::object object_value = (*_array_iter).get_object();
851852
if (!_parsed_from_json_root && !_parsed_json_root.empty()) {
852853
// Reject roots per element for loads, preserving earlier siblings and their presence
@@ -885,7 +886,7 @@ Status JsonReader::_append_flat_array_jsonpath_rows(Block* block, bool* is_empty
885886
}
886887
++_array_iter;
887888
}
888-
*is_empty_row = block->rows() == original_rows;
889+
*is_empty_row = _logical_rows == original_rows;
889890
return Status::OK();
890891
}
891892

@@ -909,7 +910,7 @@ Status JsonReader::_set_column_values_from_object(simdjson::ondemand::object* ob
909910
DORIS_CHECK(block != nullptr);
910911
DORIS_CHECK(valid != nullptr);
911912
std::vector<bool> seen_columns(block->columns(), false);
912-
const size_t cur_row_count = block->rows();
913+
const size_t cur_row_count = _logical_rows;
913914
bool has_valid_value = false;
914915
size_t key_index = 0;
915916

@@ -960,21 +961,46 @@ Status JsonReader::_set_column_values_from_object(simdjson::ondemand::object* ob
960961
}
961962
}
962963
*valid = true;
964+
++_logical_rows;
963965
if (!has_valid_value) {
964966
return Status::OK();
965967
}
966968
return Status::OK();
967969
}
968970

971+
Status JsonReader::_has_matching_jsonpath(simdjson::ondemand::object* object_value,
972+
bool* matches) const {
973+
*matches = false;
974+
for (const auto& path : _parsed_jsonpaths) {
975+
// A root path selects the current document without a field lookup.
976+
if (JsonFunctions::is_root_path(path)) {
977+
*matches = true;
978+
continue;
979+
}
980+
simdjson::ondemand::value value;
981+
const auto status = JsonFunctions::extract_from_object(*object_value, path, &value);
982+
if (!status.ok() && !status.is<ErrorCode::NOT_FOUND>()) {
983+
return status;
984+
}
985+
*matches = *matches || status.ok();
986+
}
987+
return Status::OK();
988+
}
989+
969990
Status JsonReader::_write_columns_by_jsonpath(simdjson::ondemand::object* object_value,
970991
Block* block, bool* valid) {
971992
DORIS_CHECK(object_value != nullptr);
972993
DORIS_CHECK(block != nullptr);
973994
DORIS_CHECK(valid != nullptr);
974995
bool has_valid_value = false;
975-
const size_t cur_row_count = block->rows();
996+
const size_t cur_row_count = _logical_rows;
976997
_cached_string_values.clear();
977998

999+
// Even without projected fields, jsonpaths still decides whether the object contributes a row.
1000+
if (_requested_columns.empty()) {
1001+
RETURN_IF_ERROR(_has_matching_jsonpath(object_value, &has_valid_value));
1002+
}
1003+
9781004
for (size_t i = 0; i < _requested_columns.size(); ++i) {
9791005
const auto& requested = _requested_columns[i];
9801006
auto* column_ptr = block->get_by_position(i).column->assert_mutable().get();
@@ -1031,6 +1057,7 @@ Status JsonReader::_write_columns_by_jsonpath(simdjson::ondemand::object* object
10311057
return Status::OK();
10321058
}
10331059
*valid = true;
1060+
++_logical_rows;
10341061
return Status::OK();
10351062
}
10361063

@@ -1233,6 +1260,7 @@ Status JsonReader::_append_null_for_malformed_json(Block* block) {
12331260
assert_cast<ColumnNullable*>(column.get())->insert_default();
12341261
column_with_type.column = std::move(column);
12351262
}
1263+
++_logical_rows;
12361264
return Status::OK();
12371265
}
12381266

@@ -1244,6 +1272,7 @@ Status JsonReader::_handle_json_error(const Status& status, Block* block, size_t
12441272
// block to the row count before this document before either surfacing the error or appending
12451273
// the ignore-malformed null row.
12461274
_truncate_block_to_rows(block, original_rows);
1275+
_logical_rows = original_rows;
12471276
for (auto& [_, mask] : _missing_fields) {
12481277
mask.resize(original_rows, 0);
12491278
}

‎be/src/format_v2/json/json_reader.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,7 @@ class JsonReader final : public FileReader {
148148
Status _append_nested_jsonpath_row(Block* block, bool* is_empty_row, bool* eof);
149149
Status _set_column_values_from_object(simdjson::ondemand::object* object_value, Block* block,
150150
bool* valid);
151+
Status _has_matching_jsonpath(simdjson::ondemand::object* object_value, bool* matches) const;
151152
Status _write_columns_by_jsonpath(simdjson::ondemand::object* object_value, Block* block,
152153
bool* valid);
153154
template <bool use_string_cache>
@@ -191,6 +192,8 @@ class JsonReader final : public FileReader {
191192
std::unique_ptr<Decompressor> _decompressor;
192193
std::unique_ptr<NewPlainTextLineReader> _line_reader;
193194
int64_t _current_offset = 0;
195+
// Partition-only scans have no file columns, so Block::rows() cannot count their records.
196+
size_t _logical_rows = 0;
194197
bool _reader_eof = false;
195198
bool _skip_first_line = false;
196199
bool _single_document_read = false;

‎be/src/format_v2/wal/wal_reader.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,8 +195,11 @@ Status WalReader::get_block(Block* file_block, size_t* rows, bool* eof) {
195195
return Status::Corruption("WAL block has {} columns but header declares {}",
196196
source_block.columns(), _column_ids.size());
197197
}
198+
// Empty projections still contribute every WAL row. Capture the count before materialization
199+
// moves source columns into the output block and can leave the source with no owned columns.
200+
const size_t source_rows = source_block.rows();
198201
RETURN_IF_ERROR(_materialize_requested_columns(&source_block, file_block));
199-
*rows = file_block->rows();
202+
*rows = source_rows;
200203
_record_scan_rows(cast_set<int64_t>(*rows));
201204
RETURN_IF_ERROR(
202205
apply_materialized_reader_filters(_request.get(), _io_ctx.get(), file_block, rows));

‎be/test/format_v2/json/json_reader_test.cpp‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@
4646
#include "exprs/vexpr.h"
4747
#include "exprs/vexpr_context.h"
4848
#include "format_v2/column_data.h"
49+
#include "format_v2/table_reader.h"
4950
#include "io/io_common.h"
5051
#include "runtime/descriptors.h"
5152
#include "runtime/exec_env.h"
@@ -1031,4 +1032,165 @@ TEST(JsonReaderTest, AppliesDeleteAndNormalConjunctsWithPredicateFilterAccountin
10311032
EXPECT_EQ(io_ctx->predicate_filtered_rows, 1);
10321033
}
10331034

1035+
// Partition-only scans have no file-local output columns, but still need one logical row per
1036+
// accepted JSON record so TableReader can materialize the partition constants.
1037+
TEST(JsonReaderTest, EmptyFileProjectionPreservesRows) {
1038+
ObjectPool pool;
1039+
const auto slots = build_slots(&pool);
1040+
for (const auto& source_slots : {slots, std::vector<SlotDescriptor*> {}}) {
1041+
auto result = read_once("empty_projection.jsonl", "{\"id\":1}\n\n{}\n{\"id\":3}\n",
1042+
json_scan_params(), source_slots, {}, true, true);
1043+
ASSERT_TRUE(result.status.ok()) << result.status;
1044+
EXPECT_EQ(result.block.columns(), 0);
1045+
EXPECT_EQ(result.rows, 3);
1046+
EXPECT_FALSE(result.eof);
1047+
EXPECT_TRUE(result.second_status.ok());
1048+
EXPECT_EQ(result.second_rows, 0);
1049+
EXPECT_TRUE(result.second_eof);
1050+
}
1051+
}
1052+
1053+
TEST(JsonReaderTest, EmptyFileProjectionPreservesOuterArrayRows) {
1054+
auto result = read_once("empty_projection_array.json", R"([{"id":1},{},{"id":3}])",
1055+
json_scan_params(false, true), {}, {}, true, true);
1056+
ASSERT_TRUE(result.status.ok()) << result.status;
1057+
EXPECT_EQ(result.rows, 3);
1058+
EXPECT_TRUE(result.second_status.ok());
1059+
EXPECT_TRUE(result.second_eof);
1060+
}
1061+
1062+
TEST(JsonReaderTest, EmptyFileProjectionPreservesJsonRootRows) {
1063+
auto result = read_once("empty_projection_root.jsonl",
1064+
"{\"payload\":{\"id\":1}}\n{\"payload\":{\"id\":2}}\n",
1065+
json_scan_params(true, false, "", "$.payload"), {}, {}, true, true);
1066+
ASSERT_TRUE(result.status.ok()) << result.status;
1067+
EXPECT_EQ(result.rows, 2);
1068+
EXPECT_TRUE(result.second_status.ok());
1069+
EXPECT_TRUE(result.second_eof);
1070+
}
1071+
1072+
TEST(JsonReaderTest, EmptyFileProjectionHonorsJsonPaths) {
1073+
ObjectPool pool;
1074+
const auto slots = build_slots(&pool);
1075+
auto result = read_once("empty_projection_jsonpaths.jsonl",
1076+
"{\"payload\":{\"id\":1}}\n{\"other\":2}\n{\"payload\":{\"id\":3}}\n",
1077+
json_scan_params(true, false, R"(["$.payload.id","$.payload.user"])"),
1078+
slots, {}, true, true);
1079+
ASSERT_TRUE(result.status.ok()) << result.status;
1080+
EXPECT_EQ(result.rows, 2);
1081+
EXPECT_TRUE(result.second_status.ok());
1082+
EXPECT_TRUE(result.second_eof);
1083+
}
1084+
1085+
TEST(JsonReaderTest, EmptyFileProjectionPreservesRootJsonPathRows) {
1086+
auto result = read_once("empty_projection_root_path.jsonl", "{\"id\":1}\n{}\n",
1087+
json_scan_params(true, false, R"(["$."])"), {}, {}, true, true);
1088+
ASSERT_TRUE(result.status.ok()) << result.status;
1089+
EXPECT_EQ(result.rows, 2);
1090+
EXPECT_TRUE(result.second_eof);
1091+
}
1092+
1093+
TEST(JsonReaderTest, EmptyFileProjectionPreservesRootedArrayJsonPathRows) {
1094+
auto result = read_once("empty_projection_rooted_array.json",
1095+
R"({"rows":[{"payload":{"id":1}},{"other":2},{"payload":{"id":3}}]})",
1096+
json_scan_params(false, true, R"(["$.payload.id"])", "$.rows"), {}, {},
1097+
true, true);
1098+
ASSERT_TRUE(result.status.ok()) << result.status;
1099+
EXPECT_EQ(result.rows, 2);
1100+
EXPECT_TRUE(result.second_status.ok());
1101+
EXPECT_TRUE(result.second_eof);
1102+
}
1103+
1104+
TEST(JsonReaderTest, EmptyFileProjectionPreservesMalformedNullRows) {
1105+
auto result = read_once("empty_projection_malformed.jsonl", "not-json\n{\"id\":1}\n",
1106+
json_scan_params(true, false, "", "", true), {}, {}, true, true);
1107+
ASSERT_TRUE(result.status.ok()) << result.status;
1108+
EXPECT_EQ(result.rows, 2);
1109+
EXPECT_TRUE(result.second_status.ok());
1110+
EXPECT_TRUE(result.second_eof);
1111+
1112+
auto strict = read_once("empty_projection_strict.jsonl", "not-json\n", json_scan_params(), {},
1113+
{}, false, true);
1114+
EXPECT_FALSE(strict.status.ok());
1115+
}
1116+
1117+
TEST(JsonReaderTest, EmptyFileProjectionPreservesEmptyInput) {
1118+
for (const std::string content : {"", "\n", "[]\n"}) {
1119+
auto result = read_once("empty_projection_empty.jsonl", content,
1120+
json_scan_params(true, true), {}, {}, false, true);
1121+
ASSERT_TRUE(result.status.ok()) << result.status;
1122+
EXPECT_EQ(result.rows, 0);
1123+
EXPECT_TRUE(result.eof);
1124+
}
1125+
}
1126+
1127+
TEST(JsonReaderTest, EmptyFileProjectionHonorsBatchSize) {
1128+
const auto path = write_json_file("empty_projection_batches.jsonl",
1129+
"{\"id\":1}\n{\"id\":2}\n{\"id\":3}\n"
1130+
"{\"id\":4}\n{\"id\":5}\n");
1131+
auto properties = std::make_shared<io::FileSystemProperties>();
1132+
properties->system_type = TFileType::FILE_LOCAL;
1133+
auto desc = file_description(path.string());
1134+
auto params = json_scan_params();
1135+
RuntimeProfile profile("json_empty_projection_batches");
1136+
MockRuntimeState state;
1137+
state._batch_size = 2;
1138+
JsonReader reader(properties, desc, std::make_shared<io::IOContext>(), &profile, &params,
1139+
file_range(path), {});
1140+
ASSERT_TRUE(reader.init(&state).ok());
1141+
ASSERT_TRUE(reader.open(std::make_shared<FileScanRequest>()).ok());
1142+
for (const size_t expected_rows : {2, 2, 1, 0}) {
1143+
Block block;
1144+
size_t rows = 0;
1145+
bool eof = false;
1146+
ASSERT_TRUE(reader.get_block(&block, &rows, &eof).ok());
1147+
EXPECT_EQ(block.columns(), 0);
1148+
EXPECT_EQ(rows, expected_rows);
1149+
EXPECT_EQ(eof, expected_rows == 0);
1150+
}
1151+
}
1152+
1153+
TEST(JsonReaderTest, EmptyFileProjectionMaterializesPartitionRows) {
1154+
const auto path =
1155+
write_json_file("partition_only.jsonl", "{\"id\":1}\n{\"id\":2}\n{\"id\":3}\n");
1156+
auto params = json_scan_params();
1157+
const std::vector<SlotDescriptor*> file_slots;
1158+
ColumnDefinition partition;
1159+
partition.identifier = Field::create_field<TYPE_STRING>("dt");
1160+
partition.local_id = 0;
1161+
partition.name = "dt";
1162+
partition.type = std::make_shared<DataTypeString>();
1163+
partition.is_partition_key = true;
1164+
RuntimeProfile profile("json_partition_only");
1165+
MockRuntimeState state;
1166+
state._batch_size = 2;
1167+
TableReader reader;
1168+
ASSERT_TRUE(reader.init({.projected_columns = {partition},
1169+
.conjuncts = {},
1170+
.format = FileFormat::JSON,
1171+
.scan_params = &params,
1172+
.io_ctx = std::make_shared<io::IOContext>(),
1173+
.runtime_state = &state,
1174+
.scanner_profile = &profile,
1175+
.file_slot_descs = &file_slots})
1176+
.ok());
1177+
SplitReadOptions split;
1178+
split.current_range = file_range(path);
1179+
split.current_split_format = FileFormat::JSON;
1180+
split.partition_values.emplace("dt", Field::create_field<TYPE_STRING>("p1"));
1181+
ASSERT_TRUE(reader.prepare_split(split).ok());
1182+
bool eos = false;
1183+
for (const size_t expected_rows : {2, 1, 0}) {
1184+
Block block({{partition.type->create_column(), partition.type, partition.name}});
1185+
ASSERT_TRUE(reader.get_block(&block, &eos).ok());
1186+
ASSERT_EQ(block.rows(), expected_rows);
1187+
EXPECT_EQ(eos, expected_rows == 0);
1188+
const auto column = block.get_by_position(0).column->convert_to_full_column_if_const();
1189+
for (size_t row = 0; row < expected_rows; ++row) {
1190+
EXPECT_EQ(column->get_data_at(row).to_string(), "p1");
1191+
}
1192+
}
1193+
EXPECT_TRUE(reader.close().ok());
1194+
}
1195+
10341196
} // namespace doris::format::json

0 commit comments

Comments
 (0)