Repository navigation
[Bug] Query Hive catalog fails with NullPointerException after enabling enable_binary_search_filtering_partitions #64800
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
3.1
maybe 4.0 and 4.1
What's Wrong?
After enabling the session variable enable_binary_search_filtering_partitions, queries against Hive external catalogs fail with a NullPointerException.
Error stack trace:
java.lang.NullPointerException: null value in entry: dt=2026-06-22=null
at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33)
at com.google.common.collect.RegularImmutableMap.fromEntryArrayCheckingBucketOverflow(RegularImmutableMap.java:132)
at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:98)
at com.google.common.collect.RegularImmutableMap.fromEntries(RegularImmutableMap.java:81)
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:729)
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:702)
at org.apache.doris.nereids.trees.plans.logical.LogicalFileScan$SelectedPartitions.<init>(LogicalFileScan.java:173)
at org.apache.doris.nereids.rules.rewrite.PruneFileScanPartition.pruneExternalPartitions(PruneFileScanPartition.java:107)
at org.apache.doris.nereids.rules.rewrite.PruneFileScanPartition.lambda$build$1(PruneFileScanPartition.java:64)
...
Root Cause:
In PruneFileScanPartition.java, the SortedPartitionRanges retrieved at partition pruning time differs from the partition list captured at query start — extra partitions (with null values) appear, causing ImmutableMap.copyOf() to throw NullPointerException when constructing SelectedPartitions.
What You Expected?
No NPE.
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
zclllyybb commentedon Jun 24, 2026
Breakwater-GitHub-Analysis-Slot: slot_688d925f92cd
This content is generated by AI for reference only.
Initial triage: this looks like a real FE planner bug in Hive external partition pruning, not a BE scan failure.
What I checked:
upstream/master, and also onbranch-3.1,branch-4.0, andbranch-4.1,PruneFileScanPartition.pruneExternalPartitions()usesscan.getSelectedPartitions().selectedPartitionsas the selected partition map.enable_binary_search_filtering_partitionsis enabled, the same rule also callsexternalTable.getSortedPartitionRanges(scan).HMSExternalTable.getSortedPartitionRanges()obtains ranges fromHivePartitionValues; it does not derive the ranges from the scan's captured selected-partition map.PartitionPruner.binarySearchFiltering()can therefore return partition names from the sorted-range source.PruneFileScanPartitionthen doesselectedPartitionItems.put(name, nameToPartitionItem.get(name)). If that name is absent from the scan's selected map, the value inserted isnull.LogicalFileScan.SelectedPartitionsthen callsImmutableMap.copyOf(selectedPartitions), which exactly matches the reported exception:null value in entry: dt=2026-06-22=null.Assessment:
The reporter's root-cause direction is plausible and matches the current code path. The binary-search pruning path mixes two partition sources: the scan's captured selected-partition map and Hive's current sorted partition ranges. For Hive,
loadSnapshot()returns an empty MVCC snapshot, so the planner does not pin a Hive partition-list version. If HMS/cache state changes between scan construction and partition pruning, the binary-search result can contain a partition name that is not present in the scan map, causing the NPE.A practical workaround should be to disable
enable_binary_search_filtering_partitions, because the sequential pruning path iterates only the scan's selected-partition map and should not produce missing-name/null-value entries.Suggested maintainer next steps:
scan.getSelectedPartitions().selectedPartitions, for example by buildingSortedPartitionRangesfrom that map for this pruning pass, or otherwise enforce a coherent snapshot/map before constructingSelectedPartitions.nameToPartitionItem.get(name)intoselectedPartitionItemswithout verifying the value is present; a missing name indicates a partition-source mismatch and should be handled by rebuilding/falling back rather than producing a null map value.Information still needed from the reporter to confirm the exact blast radius:
dt=2026-06-22was added, dropped, or refreshed in HMS while the failing query was being planned.[Fix](nereids) Freeze sortedPartitionRanges in SelectedPartitions to …
[Fix](nereids) Freeze sortedPartitionRanges in SelectedPartitions to …
924060929 commentedon Jul 24, 2026
fixed by #65659
[Fix](nereids) Freeze sortedPartitionRanges in SelectedPartitions to …
[Fix](nereids) Freeze sortedPartitionRanges in SelectedPartitions to …