You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a7c0a78
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](cloud) bind the packed slice location lifetime to its writer (#67347)
Problem Summary:
A small file is handed to `PackedFileManager` when the segment is
flushed, but its slice
location is read again only when the rowset is closed —
`wait_upload_done()` from
`finish_close()`, and `get_packed_slice_location()` when the rowset meta
is built. The gap
between the two is the duration of the whole load, not the lifetime of
the packed file.
Both reads went through `_global_slice_locations`, which
`cleanup_expired_data()` recycles
purely on the age of the small file (`uploaded_file_retention_seconds`,
1800s by default).
Any load running longer than that lost the mapping and failed with:
```
[INTERNAL_ERROR]File not found in global index: data/<tablet_id>/<segment>_0.idx
```
The packed file itself had been uploaded successfully — only the
in-memory mapping was gone.
We hit this in production with a broker load that ran for 38 minutes.
Two related points:
`_uploaded_packed_files` is recycled on the same config, so a waiter
could also fail with
`Packed file not found for path: ...`; and this is not specific to
inverted index files,
since segments smaller than `small_file_threshold_bytes` take the same
async-close /
late-wait path.
### What is changed and how it works?
A slice location should live as long as whoever may still read it, not
until a wall clock
deadline.
1. `append_small_file()` returns the location as a `std::shared_ptr` and
`PackedFileWriter`
keeps it for as long as it lives. `wait_upload_done()` and the writer's
`get_packed_slice_location()` take that reference, and
`CloudRowsetWriter` asks the writer
instead of the manager, so neither can fail because the index was
recycled. The terminal
upload state (UPLOADED / FAILED) is recorded on the slices, so a waiter
does not need the
`PackedFileContext` either.
2. The index itself gets the same property.
`PackedFileSystem::open_file_impl()` falls back
to `_global_slice_locations` for a file whose location has not reached
the rowset meta
yet, which is every read during the load — `SegmentIndexFileCacheLoader`
does exactly that
when it preloads a segment's index and footer ranges into file cache
right after close.
Since a location is now a `shared_ptr`, the cleanup can tell whether an
entry is still
needed: entries some writer or packed file context still holds are
skipped, and the age
check applies only to those the index alone holds. The last owner going
away drops the
reference, so an entry outlives everything that could still read it.
3. `PackedFileContext::slice_locations` is keyed by the small file path,
so one path written
twice into the same packed file left only the second handle in there and
the first never
reached a terminal state. Every appended handle is now kept alongside
that map and
notified from it. No caller does this today —
`CloudTablet::create_transient_rowset_writer()` sets `allow_packed_file
= false`, so the
MOW publish retry that writes one segment path twice never produces
packed files — but
the interface allowed it and failed quietly.
Behavior note: an index entry is no longer guaranteed to be dropped
after
`uploaded_file_retention_seconds`. It now stays for as long as a writer
or a packed file
context holds the slice, so a load producing N packed small files keeps
N entries resident
until it finishes. That is the point of the fix — those entries were
being dropped while
still in use — but it does change how much the map holds during a long
load.
Behavior for short loads is unchanged.
### Release note
Fix `File not found in global index` failures for loads that run longer
than
`uploaded_file_retention_seconds` when packed file is enabled.
0 commit comments