[Bug] 对 Hive textfile + JsonSerDe 分区表，count(*) + 分区列过滤 恒为 0，而 select * 有数据 #69926

Activity

1. Doris-Breakwater commented on Nov 1, 2026

   

   Doris-Breakwater

   on Nov 1, 2026

   Collaborator

   More actions

   The attachment points to a likely query-correctness defect in the V2 JSON reader when no physical file columns are projected. The evidence is strong enough to focus investigation on row-count preservation in that path, but the exact runtime projection and a reproducer/A-B result are still needed to close the diagnosis. Suggested triage: BE external scan / Hive JSON / wrong results.

   What is established by 1111.xlsx:

   • Both queries report doris-4.1.5-release-917b20be838, Is Cached: No, and UseScannerV2: true.
   • Both plans select partition=1/9, with 15 splits and totalFileSize=489359939. Both explicitly show pushdown agg=NONE. This capture therefore does not support blaming the external COUNT pushdown optimization.
   • The count profile reads 522.64 MB, opens 15 files, and reports ScanRows=0, RowsProduced=0, and FinalizeBlockTime=0ns. EmptyFileNum, NotFoundFileNum, and RuntimeFilterPartitionPrunedRangeNum are all zero. Bytes are being read, but no rows reach the aggregate.
   • The SELECT * ... LIMIT 10 profile reports 4,064 scan rows and returns 10 rows through the same V2 JSON reader. It proves that matching rows exist; a limited query does not establish the full partition count or independently verify the approximately 1.148 million rows mentioned in the screenshot.

   I checked the source at the exact reported commit, 917b20be83878e98f94e3ebf7efccc80346e3e50. The following mechanism matches the observed difference:

   1. The count plan projects a constant above the partition filter. Unlike SELECT *, it need not materialize a business column from the JSON file. FileScannerV2::_build_projected_columns separates partition columns from _file_slot_descs.
   2. The column mapper supplies partition values as constants and adds physical read columns only for mappings with a file-local column ID. Thus a partition-only projection can produce an empty file block. See column_mapper.cpp.
   3. JsonReader::get_block derives its returned row count from file_block->rows(). With no requested file columns, parsing an object does not append any column values; an empty block has zero rows. The reader also records ScanRows before applying its filters, so the observed zero is consistent with losing the row count during materialization, rather than normal rows subsequently being rejected by the partition predicate.
   4. TableReader::get_block skips finalize_chunk() whenever current_rows == 0. Consequently the partition constants never receive rows, matching FinalizeBlockTime=0ns and the final zero count.

   This is a code-supported diagnosis, not a claim that I reproduced the query against your data. In particular, the plan's operativeCols field alone is not a dump of the physical reader's requested columns.

   The most useful next checks are:

   1. In one session, keep the partition files unchanged and compare SELECT etl_date ... LIMIT 10 with SELECT id, etl_date ... LIMIT 10, using the same partition predicate and a real file column such as id. Keep V2 enabled and external COUNT pushdown disabled. If only the second query returns rows, that directly supports the empty-file-projection diagnosis. Please attach both plans and plain-text profiles.

   2. Test the original count with these session-only settings, recording the previous values first:

      SET enable_profile = true;
      SET enable_count_push_down_for_external_table = false;
      SET enable_file_scanner_v2 = false;
      -- Run the original COUNT(*) and EXPLAIN for the same partition.

      Confirm UseScannerV2: false in the new profile and pushdown agg=NONE in the plan, and compare the result with Hive over the same files. Restore the previous settings afterward. This is an unverified workaround candidate: the V1 JSON reader maintains read_rows independently, and the V1 scanner uses it to fill partition columns on the non-COUNT-pushdown path. Disabling COUNT pushdown alone does not address the V2 mechanism above.

   3. Supply a sanitized full Hive DDL (including SerDe properties), Doris DESC, a few representative JSON records, and Hive's exact count. If the comparison is inconclusive, capture the scan's required slots/file-column request and query-ID-correlated BE logs; also confirm all participating FE/BE builds match the reported version.

   For maintainers, the fix direction is to preserve the number of valid JSON records even with zero physical output columns, or provide a correct fallback. Regression coverage should include partition-only versus file-column projections, COUNT and scanner-version combinations, empty input/objects, split boundaries, and existing malformed-JSON semantics. COUNT(id) is not a generally equivalent replacement for COUNT(*) because it excludes NULLs. No permanent fix or fixed release has been verified in this investigation.

   Breakwater-GitHub-Analysis-Slot: slot_0f7168a6ada3
2. Gabriel39

   added 2 commits that reference this issue on Nov 4, 2026

   [fix](be) Preserve WAL row counts for empty file projections

   ...

   This issue will close once commit 3f6f6f3 is merged into the 'master' branch.

   3f6f6f3

   [test](regression) Cover row counts after WAL replay

   ...

   This issue will close once commit 8009387 is merged into the 'master' branch.

   8009387
3. Gabriel39 commented on Nov 4, 2026

   

   Gabriel39

   on Nov 4, 2026

   Last edited by Gabriel39

   Contributor

   More actions

   Hi @xwj924195077 , thanks for this bug report. This is a bug introduced by FileScannerV2. You can bypass this code path by enable_file_scanner_v2 = false and we will fix it in #70154
4. Gabriel39

   closed this as completedin #70154on Nov 4, 2026
5. github-actions

   added a commit that references this issue on Nov 4, 2026

   [fix](be) Preserve JSON and WAL rows with empty file projections (#70154

   ...

   This issue will close once commit 07dbba8 is merged into the 'master' branch.

   07dbba8
6. qiuyanjun888

   added a commit that references this issue on Nov 4, 2026

   [fix](be) Preserve JSON and WAL rows with empty file projections (apa…

   ...

   This issue will close once commit 4bde8d0 is merged into the 'master' branch.

   4bde8d0