Skip to content

branch-4.2: [fix](cloud) Skip global packed index probe when opening files of committed rowsets #69980 - #70235

Merged
yiguolei merged 1 commit into
branch-4.2from
auto-pick-69980-branch-4.2
Nov 6, 2026
Merged

yiguolei merged 1 commit into
branch-4.2from
auto-pick-69980-branch-4.2

Conversation

@github-actions

@github-actions github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Cherry-picked from #69980

…mitted rowsets (#69980)

### What problem does this PR solve?

Issue Number: None

Related PR: #64995

Problem Summary:

A cloud BE logs a burst of WARN lines with a full stack trace like

meet error status: [NOT_FOUND]File not found in global packed index:
data/<tablet>/<rowset>_0.idx

while compaction reads its input rowsets. In one instance a 4-hour BE
log had 42k
such lines over 21k paths in 158 tablets, yet every compaction
succeeded.

Reproduce: write a rowset whose segment file is smaller than
small_file_threshold_bytes (so it is packed) but whose inverted index
file is
larger (so it is uploaded as a regular object), then compact or query
it. Each
open of the regular .idx file prints the WARN with a stack, twice per
compaction
(eligibility check and the merge itself).

Root cause: #64995 made PackedFileSystem::open_file_impl fall back to
PackedFileManager::get_packed_slice_location when a path is not in
_index_map,
so that the file cache preload can read back a segment of the rowset
being
written, whose rowset meta does not exist yet. The fallback was added
unconditionally, so it also runs for a PackedFileSystem built from a
rowset
meta. That index map already holds every packed slice of the rowset, so
a miss
means a regular file, and the probe can only fail. The failure is a
Status::NotFound, and NOT_FOUND records a stack trace and logs a WARN as
soon as
it is constructed, before the caller falls back to opening the regular
file.

Fix:
- Only consult the manager when the index map is not initialized, i.e.
on the
write path, the same split exists_impl and file_size_impl already make.
- Return NOT_FOUND without a stack trace from get_packed_slice_location:
on the
  write path a miss is also expected, for files too large to be packed.

After the fix, reading committed rowsets no longer takes the global
index lock
or logs anything for regular files; the write-path preload still finds
packed
slices through the manager.

### Release note

Fix noisy "File not found in global packed index" WARN logs with stack
traces
when reading rowsets that mix packed and regular files in cloud mode.
@github-actions
github-actions Bot requested a review from yiguolei as a code owner November 5, 2026 02:05
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

Copy link
Copy Markdown
Contributor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.96% (30479/46923)
Line Coverage 49.90% (323409/648150)
Region Coverage 45.64% (260533/570795)
Branch Coverage 47.38% (123466/260607)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.36% (34801/45575)
Line Coverage 61.31% (394049/642748)
Region Coverage 58.00% (331554/571647)
Branch Coverage 59.07% (153747/260291)

@yiguolei
yiguolei merged commit 6612220 into branch-4.2 Nov 6, 2026
35 of 36 checks passed
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants