Skip to content

Commit a7c0a78

Browse files
authored
[fix](cloud) bind the packed slice location lifetime to its writer (#67347)
Problem Summary: A small file is handed to `PackedFileManager` when the segment is flushed, but its slice location is read again only when the rowset is closed — `wait_upload_done()` from `finish_close()`, and `get_packed_slice_location()` when the rowset meta is built. The gap between the two is the duration of the whole load, not the lifetime of the packed file. Both reads went through `_global_slice_locations`, which `cleanup_expired_data()` recycles purely on the age of the small file (`uploaded_file_retention_seconds`, 1800s by default). Any load running longer than that lost the mapping and failed with: ``` [INTERNAL_ERROR]File not found in global index: data/<tablet_id>/<segment>_0.idx ``` The packed file itself had been uploaded successfully — only the in-memory mapping was gone. We hit this in production with a broker load that ran for 38 minutes. Two related points: `_uploaded_packed_files` is recycled on the same config, so a waiter could also fail with `Packed file not found for path: ...`; and this is not specific to inverted index files, since segments smaller than `small_file_threshold_bytes` take the same async-close / late-wait path. ### What is changed and how it works? A slice location should live as long as whoever may still read it, not until a wall clock deadline. 1. `append_small_file()` returns the location as a `std::shared_ptr` and `PackedFileWriter` keeps it for as long as it lives. `wait_upload_done()` and the writer's `get_packed_slice_location()` take that reference, and `CloudRowsetWriter` asks the writer instead of the manager, so neither can fail because the index was recycled. The terminal upload state (UPLOADED / FAILED) is recorded on the slices, so a waiter does not need the `PackedFileContext` either. 2. The index itself gets the same property. `PackedFileSystem::open_file_impl()` falls back to `_global_slice_locations` for a file whose location has not reached the rowset meta yet, which is every read during the load — `SegmentIndexFileCacheLoader` does exactly that when it preloads a segment's index and footer ranges into file cache right after close. Since a location is now a `shared_ptr`, the cleanup can tell whether an entry is still needed: entries some writer or packed file context still holds are skipped, and the age check applies only to those the index alone holds. The last owner going away drops the reference, so an entry outlives everything that could still read it. 3. `PackedFileContext::slice_locations` is keyed by the small file path, so one path written twice into the same packed file left only the second handle in there and the first never reached a terminal state. Every appended handle is now kept alongside that map and notified from it. No caller does this today — `CloudTablet::create_transient_rowset_writer()` sets `allow_packed_file = false`, so the MOW publish retry that writes one segment path twice never produces packed files — but the interface allowed it and failed quietly. Behavior note: an index entry is no longer guaranteed to be dropped after `uploaded_file_retention_seconds`. It now stays for as long as a writer or a packed file context holds the slice, so a load producing N packed small files keeps N entries resident until it finishes. That is the point of the fix — those entries were being dropped while still in use — but it does change how much the map holds during a long load. Behavior for short loads is unchanged. ### Release note Fix `File not found in global index` failures for loads that run longer than `uploaded_file_retention_seconds` when packed file is enabled.
1 parent 17ac3d9 commit a7c0a78

6 files changed

Lines changed: 415 additions & 123 deletions

File tree

‎be/src/cloud/cloud_rowset_writer.cpp‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -223,10 +223,11 @@ Status CloudRowsetWriter::_collect_packed_slice_location(io::FileWriter* file_wr
223223
return Status::OK();
224224
}
225225

226-
// Get packed slice location directly from PackedFileManager
226+
// Ask the writer, which holds a reference to its own slice location. Looking it up by
227+
// path in PackedFileManager would race with the retention based cleanup of the index.
227228
io::PackedSliceLocation index;
228229
RETURN_IF_ERROR(
229-
io::PackedFileManager::instance()->get_packed_slice_location(file_path, &index));
230+
static_cast<io::PackedFileWriter*>(file_writer)->get_packed_slice_location(&index));
230231
if (index.packed_file_path.empty()) {
231232
return Status::OK(); // File not in packed file, skip
232233
}

‎be/src/io/fs/packed_file_manager.cpp‎

Lines changed: 63 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -337,7 +337,9 @@ Status PackedFileManager::ensure_file_system(const std::string& resource_id,
337337
}
338338

339339
Status PackedFileManager::append_small_file(const std::string& path, const Slice& data,
340-
const PackedAppendContext& info) {
340+
const PackedAppendContext& info,
341+
PackedSliceHandlePtr* handle) {
342+
*handle = nullptr;
341343
// Check if file is too large to be merged
342344
if (data.get_size() > config::small_file_threshold_bytes) {
343345
return Status::OK(); // Skip merging for large files
@@ -387,7 +389,9 @@ Status PackedFileManager::append_small_file(const std::string& path, const Slice
387389
location.resource_id = info.resource_id;
388390
location.txn_id = info.txn_id;
389391

390-
active_state->slice_locations[path] = location;
392+
auto slice_handle = std::make_shared<PackedSliceHandle>(std::move(location));
393+
active_state->slice_locations[path] = slice_handle;
394+
active_state->appended_slices.push_back(slice_handle);
391395
active_state->current_offset += data.get_size();
392396
active_state->total_size += data.get_size();
393397

@@ -409,9 +413,10 @@ Status PackedFileManager::append_small_file(const std::string& path, const Slice
409413
// Update global index
410414
{
411415
std::lock_guard<std::mutex> global_lock(_global_index_mutex);
412-
_global_slice_locations[path] = location;
416+
_global_slice_locations[path] = slice_handle;
413417
}
414418

419+
*handle = std::move(slice_handle);
415420
return Status::OK();
416421
}
417422

@@ -431,17 +436,20 @@ Status PackedFileManager::wait_for_packed_file_upload(PackedFileContext* packed_
431436
return Status::OK();
432437
}
433438

434-
Status PackedFileManager::wait_upload_done(const std::string& path) {
435-
std::string packed_file_path;
436-
{
437-
std::lock_guard<std::mutex> global_lock(_global_index_mutex);
438-
auto it = _global_slice_locations.find(path);
439-
if (it == _global_slice_locations.end()) {
440-
return Status::InternalError("File not found in global index: " + path);
441-
}
442-
packed_file_path = it->second.packed_file_path;
439+
Status PackedFileManager::wait_upload_done(const PackedSliceHandlePtr& handle) {
440+
if (handle == nullptr) {
441+
return Status::InternalError("Missing packed slice handle");
443442
}
444443

444+
// The upload already finished, so there is no need to look up the PackedFileContext,
445+
// which is only kept for a limited time after the upload.
446+
auto upload_state = handle->upload_state();
447+
if (upload_state == PackedSliceUploadState::UPLOADED) {
448+
return Status::OK();
449+
}
450+
const bool upload_failed = upload_state == PackedSliceUploadState::FAILED;
451+
const std::string& packed_file_path = handle->packed_file_path();
452+
445453
// Find the packed file in uploaded files first - if already uploaded, no need to wait
446454
std::shared_ptr<PackedFileContext> managed_packed_file;
447455
std::shared_ptr<PackedFileContext> failed_packed_file;
@@ -498,8 +506,12 @@ Status PackedFileManager::wait_upload_done(const std::string& path) {
498506
}
499507

500508
if (!packed_file_ptr) {
509+
if (upload_failed) {
510+
// The context carrying the original error has already been recycled
511+
return Status::InternalError("Packed file upload failed: " + packed_file_path);
512+
}
501513
// Packed file not found in any location, this is unexpected
502-
return Status::InternalError("Packed file not found for path: " + path);
514+
return Status::InternalError("Packed file not found: " + packed_file_path);
503515
}
504516

505517
Status wait_status = wait_for_packed_file_upload(packed_file_ptr);
@@ -515,10 +527,19 @@ Status PackedFileManager::get_packed_slice_location(const std::string& path,
515527
return Status::NotFound("File not found in global packed index: {}", path);
516528
}
517529

518-
*location = it->second;
530+
*location = it->second->location();
519531
return Status::OK();
520532
}
521533

534+
void PackedFileManager::mark_slices_upload_result(const PackedFileContext& packed_file,
535+
PackedSliceUploadState state) {
536+
// Not slice_locations: a path written twice into this packed file is only in there once,
537+
// and the writer of the shadowed slice is waiting for the result too
538+
for (const auto& handle : packed_file.appended_slices) {
539+
handle->set_upload_result(state, packed_file.total_size);
540+
}
541+
}
542+
522543
void PackedFileManager::start_background_manager() {
523544
if (_background_thread) {
524545
return; // Already started
@@ -733,18 +754,11 @@ void PackedFileManager::process_uploading_packed_files() {
733754
slices_stream << "; ";
734755
}
735756
first_slice = false;
736-
slices_stream << small_file_path << "(txn=" << index.txn_id
737-
<< ", offset=" << index.offset << ", size=" << index.size << ")";
738-
739-
// Update packed_file_size in global index
740-
{
741-
std::lock_guard<std::mutex> global_lock(_global_index_mutex);
742-
auto it = _global_slice_locations.find(small_file_path);
743-
if (it != _global_slice_locations.end()) {
744-
it->second.packed_file_size = packed_file->total_size;
745-
}
746-
}
757+
auto location = index->location();
758+
slices_stream << small_file_path << "(txn=" << location.txn_id
759+
<< ", offset=" << location.offset << ", size=" << location.size << ")";
747760
}
761+
mark_slices_upload_result(*packed_file, PackedSliceUploadState::UPLOADED);
748762
LOG(INFO) << "Packed file " << packed_file->packed_file_path
749763
<< " uploaded; slices=" << packed_file->slice_locations.size()
750764
<< ", total_bytes=" << packed_file->total_size << ", slice_detail=["
@@ -770,6 +784,7 @@ void PackedFileManager::process_uploading_packed_files() {
770784
const Status& status) {
771785
LOG(WARNING) << "Failed to upload packed file: " << packed_file->packed_file_path
772786
<< ", error: " << status.to_string();
787+
mark_slices_upload_result(*packed_file, PackedSliceUploadState::FAILED);
773788
{
774789
std::lock_guard<std::mutex> upload_lock(packed_file->upload_mutex);
775790
packed_file->state = PackedFileState::FAILED;
@@ -797,19 +812,20 @@ void PackedFileManager::process_uploading_packed_files() {
797812
packed_file_info.set_resource_id(packed_file->resource_id);
798813

799814
for (const auto& [small_file_path, index] : packed_file->slice_locations) {
815+
auto location = index->location();
800816
auto* small_file = packed_file_info.add_slices();
801817
small_file->set_path(small_file_path);
802-
small_file->set_offset(index.offset);
803-
small_file->set_size(index.size);
818+
small_file->set_offset(location.offset);
819+
small_file->set_size(location.size);
804820
small_file->set_deleted(false);
805-
if (index.tablet_id != 0) {
806-
small_file->set_tablet_id(index.tablet_id);
821+
if (location.tablet_id != 0) {
822+
small_file->set_tablet_id(location.tablet_id);
807823
}
808-
if (!index.rowset_id.empty()) {
809-
small_file->set_rowset_id(index.rowset_id);
824+
if (!location.rowset_id.empty()) {
825+
small_file->set_rowset_id(location.rowset_id);
810826
}
811-
if (index.txn_id != 0) {
812-
small_file->set_txn_id(index.txn_id);
827+
if (location.txn_id != 0) {
828+
small_file->set_txn_id(location.txn_id);
813829
}
814830
}
815831

@@ -842,8 +858,9 @@ void PackedFileManager::process_uploading_packed_files() {
842858
oss << ", ";
843859
}
844860
first = false;
845-
oss << "[" << small_file_path << ", offset=" << index.offset
846-
<< ", size=" << index.size << "]";
861+
auto location = index->location();
862+
oss << "[" << small_file_path << ", offset=" << location.offset
863+
<< ", size=" << location.size << "]";
847864
}
848865
VLOG_DEBUG << oss.str();
849866
} else {
@@ -940,9 +957,17 @@ void PackedFileManager::cleanup_expired_data() {
940957
std::lock_guard<std::mutex> global_lock(_global_index_mutex);
941958
auto it = _global_slice_locations.begin();
942959
while (it != _global_slice_locations.end()) {
943-
const auto& index = it->second;
944-
if (index.create_time > 0 &&
945-
current_time - index.create_time > config::uploaded_file_retention_seconds) {
960+
// A writer or a packed file context still holds this slice, so whoever holds it
961+
// may still read the entry back. Age says nothing about that: a load can run for
962+
// much longer than the retention time. Only entries the index alone holds are
963+
// stale, and the reference is dropped for us when the last owner goes away.
964+
if (it->second.use_count() > 1) {
965+
++it;
966+
continue;
967+
}
968+
const auto create_time = it->second->create_time();
969+
if (create_time > 0 &&
970+
current_time - create_time > config::uploaded_file_retention_seconds) {
946971
it = _global_slice_locations.erase(it);
947972
} else {
948973
++it;

‎be/src/io/fs/packed_file_manager.h‎

Lines changed: 69 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,53 @@ struct PackedSliceLocation {
5353
int64_t packed_file_size = -1; // Total size of the packed file, -1 means not set
5454
};
5555

56+
// Upload state of the packed file a slice belongs to
57+
enum class PackedSliceUploadState : uint8_t {
58+
PENDING = 0,
59+
UPLOADED,
60+
FAILED,
61+
};
62+
63+
// A slice of a packed file, shared by PackedFileManager and the PackedFileWriter that
64+
// produced it. The writer holds its handle for as long as it lives, so it can wait for the
65+
// upload and read the location back at the end of the load, whatever the manager recycled
66+
// from its by-path index in the meantime.
67+
class PackedSliceHandle {
68+
public:
69+
explicit PackedSliceHandle(PackedSliceLocation location) : _location(std::move(location)) {}
70+
71+
const std::string& packed_file_path() const { return _location.packed_file_path; }
72+
73+
int64_t create_time() const { return _location.create_time; }
74+
75+
PackedSliceUploadState upload_state() const {
76+
return _upload_state.load(std::memory_order_acquire);
77+
}
78+
79+
// Called once the packed file this slice belongs to reaches a terminal state
80+
void set_upload_result(PackedSliceUploadState state, int64_t packed_file_size) {
81+
if (state == PackedSliceUploadState::UPLOADED) {
82+
_packed_file_size.store(packed_file_size, std::memory_order_relaxed);
83+
}
84+
_upload_state.store(state, std::memory_order_release);
85+
}
86+
87+
PackedSliceLocation location() const {
88+
PackedSliceLocation location = _location;
89+
if (upload_state() == PackedSliceUploadState::UPLOADED) {
90+
location.packed_file_size = _packed_file_size.load(std::memory_order_relaxed);
91+
}
92+
return location;
93+
}
94+
95+
private:
96+
const PackedSliceLocation _location; // Immutable once the slice has been appended
97+
std::atomic<int64_t> _packed_file_size {-1};
98+
std::atomic<PackedSliceUploadState> _upload_state {PackedSliceUploadState::PENDING};
99+
};
100+
101+
using PackedSliceHandlePtr = std::shared_ptr<PackedSliceHandle>;
102+
56103
struct PackedAppendContext {
57104
std::string resource_id;
58105
int64_t tablet_id = 0;
@@ -73,14 +120,17 @@ class PackedFileManager {
73120
// Initialize manager state; file system will be resolved lazily
74121
Status init();
75122

76-
// Write a small file to the current packed file
123+
// Write a small file to the current packed file. On success `handle` receives a handle
124+
// to the new slice, or nullptr if `data` was too large to be packed.
77125
Status append_small_file(const std::string& path, const Slice& data,
78-
const PackedAppendContext& info);
126+
const PackedAppendContext& info, PackedSliceHandlePtr* handle);
79127

80-
// Block until the small file's packed file is uploaded to S3
81-
Status wait_upload_done(const std::string& path);
128+
// Block until the packed file holding `handle` is uploaded to S3
129+
Status wait_upload_done(const PackedSliceHandlePtr& handle);
82130

83-
// Get packed file index information for a small file
131+
// Look a slice location up by small file path, for readers that have no handle to the
132+
// slice. The entry lives as long as anything else holds the slice, so this only fails
133+
// for a file whose writer and packed file context are both long gone.
84134
Status get_packed_slice_location(const std::string& path, PackedSliceLocation* location);
85135

86136
// Start the background management thread
@@ -121,6 +171,10 @@ class PackedFileManager {
121171
// Clean up expired data
122172
void cleanup_expired_data();
123173

174+
// Record the terminal upload state of `packed_file` on the slices it contains
175+
void mark_slices_upload_result(const PackedFileContext& packed_file,
176+
PackedSliceUploadState state);
177+
124178
// Internal structure to track packed file state
125179
enum class PackedFileState {
126180
INIT, // Initial state, no files written yet
@@ -134,7 +188,11 @@ class PackedFileManager {
134188
struct PackedFileContext {
135189
std::string packed_file_path;
136190
std::unique_ptr<FileWriter> writer;
137-
std::unordered_map<std::string, PackedSliceLocation> slice_locations;
191+
std::unordered_map<std::string, PackedSliceHandlePtr> slice_locations;
192+
// Every slice appended to this packed file. `slice_locations` is keyed by path, so
193+
// writing one path twice into the same packed file only leaves the last handle
194+
// there, while the upload result still has to reach both.
195+
std::vector<PackedSliceHandlePtr> appended_slices;
138196
int64_t current_offset = 0;
139197
int64_t total_size = 0;
140198
int64_t create_time;
@@ -180,8 +238,11 @@ class PackedFileManager {
180238
std::unordered_map<std::string, std::shared_ptr<PackedFileContext>> _uploaded_packed_files;
181239
std::mutex _packed_files_mutex;
182240

183-
// Global index mapping small file path to packed file index
184-
std::unordered_map<std::string, PackedSliceLocation> _global_slice_locations;
241+
// Global index mapping small file path to packed file index, for readers that have no
242+
// handle to the slice, such as PackedFileSystem::open_file_impl() reading a segment back
243+
// before its rowset meta exists. An entry is only recycled once nothing else holds the
244+
// slice, so it outlives every writer and packed file context that could still read it.
245+
std::unordered_map<std::string, PackedSliceHandlePtr> _global_slice_locations;
185246
std::mutex _global_index_mutex;
186247

187248
#ifdef BE_TEST

‎be/src/io/fs/packed_file_writer.cpp‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -176,8 +176,8 @@ Status PackedFileWriter::_close_sync() {
176176
Status PackedFileWriter::_wait_packed_upload() {
177177
DCHECK(!_is_direct_write);
178178
// Only wait if we have data that was sent to packed manager
179-
if (_bytes_appended > 0 && _packed_file_manager != nullptr) {
180-
return _packed_file_manager->wait_upload_done(_file_path);
179+
if (_bytes_appended > 0 && _packed_file_manager != nullptr && _packed_slice_handle != nullptr) {
180+
return _packed_file_manager->wait_upload_done(_packed_slice_handle);
181181
}
182182
return Status::OK();
183183
}
@@ -217,19 +217,29 @@ Status PackedFileWriter::_send_to_packed_manager() {
217217
}
218218

219219
Slice data_slice(_buffer.data(), _buffer.size());
220-
RETURN_IF_ERROR(_packed_file_manager->append_small_file(_file_path, data_slice, _append_info));
220+
RETURN_IF_ERROR(_packed_file_manager->append_small_file(_file_path, data_slice, _append_info,
221+
&_packed_slice_handle));
222+
if (_packed_slice_handle == nullptr) {
223+
// append_small_file() skips data larger than `small_file_threshold_bytes` without
224+
// writing it anywhere. appendv() switches to direct write before the buffer can
225+
// reach that size, unless the threshold was lowered in between. Report the failure
226+
// instead of closing successfully on a file that does not exist.
227+
return Status::InternalError(
228+
"Packed file did not accept the buffered data for {}, size={}, threshold={}",
229+
_file_path, data_slice.get_size(), config::small_file_threshold_bytes);
230+
}
221231
_release_buffer();
222232
return Status::OK();
223233
}
224234

225235
Status PackedFileWriter::get_packed_slice_location(PackedSliceLocation* location) const {
226236
DCHECK(_state == State::CLOSED)
227237
<< " file_path: " << _file_path << " bytes_appended: " << _bytes_appended;
228-
if (_is_direct_write) {
238+
if (_is_direct_write || _packed_slice_handle == nullptr) {
229239
*location = PackedSliceLocation {};
230240
return Status::OK();
231241
}
232-
RETURN_IF_ERROR(_packed_file_manager->get_packed_slice_location(_file_path, location));
242+
*location = _packed_slice_handle->location();
233243
LOG(INFO) << "get_packed_slice_location: " << _file_path
234244
<< " packed_path: " << location->packed_file_path << " " << location->offset << " "
235245
<< location->size;

‎be/src/io/fs/packed_file_writer.h‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -89,9 +89,10 @@ class PackedFileWriter final : public FileWriter {
8989
std::string _buffer;
9090
size_t _bytes_appended = 0;
9191
State _state = State::OPENED;
92-
bool _is_direct_write = false; // Whether to use direct write mode
93-
PackedFileManager* _packed_file_manager = nullptr; // Packed file manager instance
94-
mutable PackedSliceLocation _packed_slice_location; // Packed slice info (mutable for lazy init)
92+
bool _is_direct_write = false; // Whether to use direct write mode
93+
PackedFileManager* _packed_file_manager = nullptr; // Packed file manager instance
94+
// Handle to this file's slice, shared with PackedFileManager
95+
PackedSliceHandlePtr _packed_slice_handle;
9596
PackedAppendContext _append_info;
9697
std::optional<std::chrono::steady_clock::time_point> _first_append_timestamp;
9798
bool _close_latency_recorded = false;

0 commit comments

Comments
 (0)