Repository navigation
[fix](be) Fail the scan when a deletion vector file is missing - #69578
Merged
morningman merged 1 commit intoOct 26, 2026
Merged
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#69571 Problem Summary: When a Paimon split's deletion vector file cannot be found, both native readers report NOT_FOUND when they open it (V1 PaimonParquetReader / PaimonOrcReader::_init_deletion_vector, V2 TableReader::_parse_delete_predicates). FileScanner and FileScannerV2 treat NOT_FOUND as a data file that a stale listing still names, and with ignore_not_found_file_in_external_table (default true) they skip the whole range. The data file is still there, so its surviving rows are dropped together with the deleted ones, and the query returns a successful but incomplete result. Paimon's own reader, the JNI reader and the Rust reader fail instead. A deletion vector goes missing when Paimon expires the snapshot that wrote it while the data file stays live under a newer one, or when it is looked up under a partition name that is never found (apache#69571). A deletion vector belongs to the snapshot that planned its data file, so it is never a stale listing. DeletionVectorReader, which opens every deletion vector for V1 Paimon, V1 Iceberg and the V2 table reader, now reports a missing file as INTERNAL_ERROR naming its path, so the query fails. A missing data file still reports NOT_FOUND and is skipped as before. Iceberg ranges already fail on any missing file (apache#68142); only the error code of a missing Iceberg deletion vector changes. ### Release note A Paimon query now fails when a deletion vector file of the snapshot is missing, instead of silently skipping the data file it applies to. ### Check List (For Author) - Test: Unit Test - be/test/format/table/paimon_reader_test.cpp: V1 Parquet and ORC readers report INTERNAL_ERROR with the path for a missing deletion vector file - be/test/format_v2/table/paimon_reader_test.cpp: the V2 reader fails the split for a missing deletion vector file, and a missing data file with an existing deletion vector still reports NOT_FOUND - Behavior changed: Yes. A missing Paimon deletion vector file fails the query instead of skipping its data file. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
No actionable regression found in the three changed files at d278820. Static review supports the fix: a missing deletion vector becomes a query error at both open and read boundaries, preventing the scanners from silently discarding the surviving rows of its data file. There are no existing inline findings to carry forward and no additional user-provided focus points.
Execution paths checked: V1 FileScanner -> GenericReader after-init hook -> Paimon ORC/Parquet DV loader -> DeletionVectorReader; V2 FileScannerV2 -> native Paimon child -> TableReader split preparation -> the same DV reader. The shared Iceberg helper was also traced. Both scanner versions skip NOT_FOUND only; the converted INTERNAL_ERROR is propagated.
Critical checkpoint conclusions:
- Goal and tests: both DV I/O boundaries are covered by the conversion. Four added unit cases assert V1 ORC/Parquet missing-DV errors, V2 missing-DV failure, and V2 missing-data NOT_FOUND after a valid DV. These tests were inspected, not executed.
- Scope and reuse: one private helper at the shared DV layer is a focused change and avoids duplicating connector policy.
- Concurrency and locking: no thread, lock, atomic, or shared state is added. Existing per-key cache factory locking and scanner task context are unchanged; the adapter adds no lock-order dependency.
- Lifecycle and static initialization: Status owns its message; formatting consumes it before destruction. Failed factories return their status before DV pointer use. Existing reader/context cleanup and cache ownership remain intact, with no new static initializer or ownership cycle.
- Configuration: no new configuration. The existing ignore-not-found option still governs missing data-file statuses.
- Compatibility: no public interface, storage format, or wire protocol change. The DV error-code change is intentional. Iceberg already fails on missing snapshot files, and its existing helper test asserts failure/path rather than NOT_FOUND. No new glibc API or build/link dependency is introduced.
- Thrift/protobuf ids: not applicable; no shared definitions changed.
- Parallel paths: the shared adapter covers native V1 ORC/Parquet, V2 table readers, and Iceberg. JNI/Rust dispatch is unchanged. Both eager and deferred I/O errors reach the adapter.
- Conditions and comments: the exact NOT_FOUND branch has a concrete F/D1 explanation. Other statuses, stop/EOF behavior, and short-read errors retain their existing treatment. ORC's text-based missing-file conversion runs before the Paimon after-init hook and cannot reclassify this DV error.
- Test coverage: the new test file is included by recursive CMake collection. Existing DV parsing, Iceberg missing-file, and scanner status-gate tests supplement the new cases. Local open-time errors are covered; deferred read errors, remote/cache paths, full scanner option on/off integration, and V2 ORC remain test gaps.
- Test-result artifacts: no golden outputs changed. The new assertions agree with the inspected control flow; runtime pass/fail was not independently verified.
- Observability: the new diagnostic identifies the deletion-vector path and preserves the underlying message. Existing timers/counters remain; no added per-row logging or new metric is needed for this classification change.
- Transactions and persistence: no commit, version visibility, delete bitmap, EditLog, replay, or failover protocol is modified. Returning the read error protects snapshot result correctness.
- Data writes and atomicity: production changes only read-error classification; no write or recovery protocol changes. Existing resource cleanup still applies on failure.
- FE/BE variables: none added or changed; no additional forwarding path is required.
- Performance and memory: success adds a status check outside row loops. Formatting occurs only on missing files; there is no new file-content copy, allocation proportional to data, or remote I/O.
- Additional recheck: cache misses recover internally before final errors reach the adapter; decoded/byte-cache hits can use available DV contents. Existing partition pruning and trusted metadata COUNT paths need no DV bytes. If both data and DV are absent, the first attempted open determines skip versus failure, as disclosed in the PR; no new silent-data-loss path was found.
Limits: this was a static-only review. No builds, tests, cluster runs, source changes, or final ELF ABI audits were performed. The test gaps above are validation limits, with no substantiated unresolved code defect. All changed files and both initial risk questions were checked and disposed of with evidence.
Review converged in 2 rounds: both independent reviewers returned NO_NEW_VALUABLE_FINDINGS on the unchanged final payload. New inline findings: 0. Existing confirmed P0/P1 findings: 0.
Review converged after 2 round(s).
Contributor
TPC-H: Total hot run time: 28205 ms |
Contributor
TPC-DS: Total hot run time: 150135 ms |
Contributor
ClickBench: Total hot run time: 24.31 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run cloud_p0 |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Gabriel39
approved these changes
Oct 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #69571, #68142
Problem Summary:
In short. When a Paimon split's deletion vector file is missing, the native readers report
NOT_FOUND, and both file scanners take that for a missing data file and skip the whole range. The query succeeds and silently drops every surviving row of the data file. This PR reports a missing deletion vector as an error, so the query fails instead. A missing data file is still skipped as before.Background. A Paimon split for a table with deletion vectors names a data file and the deletion vector that marks its deleted rows. The native readers open the deletion vector before they read a row: V1 in
PaimonParquetReader/PaimonOrcReader::_init_deletion_vector, V2 inTableReader::_parse_delete_predicates, both throughDeletionVectorReader.FileScannerandFileScannerV2skip a range whose reader reportsNOT_FOUNDwhileignore_not_found_file_in_external_tableis on (the default). That setting exists for file listings that are cached by the FE and may name a file that has since been removed. #68142 already exempted Iceberg ranges, whose files all belong to a committed snapshot.The problem, and what it cost. A deletion vector is never a stale listing: it belongs to the snapshot that planned its data file. It can still be missing, for example when Paimon expires the snapshot that wrote it while the data file stays live under a newer one, or when it is looked up under a partition name that is never found (#69571). The data file is still there, but the scanners skip the whole range, so its surviving rows are dropped together with the deleted ones. The BE logs
Skip missing filewith the path of the data file, which exists. Paimon's own reader, the JNI reader and the Rust reader fail on the same split instead.How this PR fixes it.
DeletionVectorReader, which opens every deletion vector for V1 Paimon, V1 Iceberg and the V2 table reader, now reports a missing file asINTERNAL_ERRORwith the path of the deletion vector, both when it opens the file and when it reads it. The scanners only skipNOT_FOUND, so the query fails. A missing data file still reportsNOT_FOUNDand is skipped as before, with or without a deletion vector. For Iceberg, which already failed on any missing file (#68142), only the error code of a missing deletion vector changes.One boundary: if the data file is gone as well, the result depends on which file the reader reaches first. FileScannerV2 parses the deletion vector while it prepares the split, before it opens the data file, so such a split now fails; in the local run below V1 Parquet failed too, while V1 ORC, which opens the data file first, still skipped it.
Results. A local cluster with a
file://Paimon catalog, two append tables (Parquet and ORC,deletion-vectors.enabled = true), five rows each, the first row deleted through a deletion vector the way a SparkDELETEpersists it:2,3,4,52,3,4,5Skip missing file: path=<data file>in be.INFO[INTERNAL_ERROR]Deletion vector file is missing: path .../index/index-...-0, failed to get file size ...: No such file or directoryRelease note
A Paimon query now fails when a deletion vector file of its snapshot is missing, instead of silently skipping the data file it applies to.
Check List (For Author)
be/test/format/table/paimon_reader_test.cpp(new): the V1 Parquet and ORC readers reportINTERNAL_ERRORwith the path for a missing deletion vector file. These restore the V1 deletion vector tests of [fix](be) Release deletion vector rows on read errors #65351 that went away withpaimon_cpp_reader_test.cppin [chore](paimon) remove paimon-cpp dependency #67385.be/test/format_v2/table/paimon_reader_test.cpp: the V2 reader fails the split for a missing deletion vector file, and a missing data file with an existing deletion vector still reportsNOT_FOUND.PaimonReaderTest, the V2 Iceberg deletion vector tests andFileScanRangeUtilsTestpass (51 tests).doris_be.Check List (For Reviewer who merge this PR)