Skip to content

Commit 72a587f

Browse files
authored
[fix](be) Use file-backed staging for SNII ANN indexes (apache#66856)
### What problem does this PR solve? Issue Number: None Related PR: apache#66052 Problem Summary: SNII staged every serialized Faiss ANN sub-file in a growing in-memory vector. A large HNSW or IVF index therefore retained a second complete copy of the serialized index and could exhaust BE memory during index construction. A 64 MiB staging reproduction increased allocated heap by 67,388,976 bytes before this change. Fault-injection reproductions also showed that final buffered append or fsync failures could escape `AnnIndexColumnWriter::finish()` as `CLuceneError`; the IVF data path additionally leaked its raw output owner. This change reuses the self-cleaning `StagedBlobFile` abstraction for ANN sub-files and keeps only the fixed CLucene output buffer resident. It also gives both `ann.faiss` and `ann.ivfdata` explicit RAII ownership, closes them in a catchable context, and converts staging failures to `Status`. The successful Faiss sub-file bytes, SNII container layout, persisted index format, and query behavior are unchanged. Existing SNII indexes remain readable after upgrade. ### Release note Reduce peak BE memory while building SNII ANN indexes by staging serialized Faiss files on temporary storage instead of retaining a complete in-memory copy.
1 parent 6133049 commit 72a587f

20 files changed

Lines changed: 1081 additions & 199 deletions

‎be/src/storage/index/ann/ann_index_writer.cpp‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,17 @@ Status AnnIndexColumnWriter::_build_and_save(Int64 min_train_rows, Int64 effecti
171171
// full-segment build buffer is released before saving the index.
172172
PODArray<float> empty_buffered_vectors;
173173
_buffered_vectors.swap(empty_buffered_vectors);
174-
return _vector_index->save(_dir.get());
174+
Status status = _vector_index->save(_dir.get());
175+
if (!status.ok()) {
176+
// A failed save keeps whatever it managed to write, and under SNII that is
177+
// an ANN-sized staging file plus its descriptor. Nothing downstream
178+
// unwinds it: the segment flush returns before clear() and
179+
// close_inverted_index(), and ADD INDEX holds every producer until the
180+
// whole rowset has been closed. Drop BOTH owners of the staging area here
181+
// -- the index file writer's, and this writer's own.
182+
_index_file_writer->discard_ann_staging_directory(_index_meta);
183+
_dir.reset();
184+
}
185+
return status;
175186
}
176187
} // namespace doris::segment_v2

‎be/src/storage/index/ann/faiss_ann_index.cpp‎

Lines changed: 109 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -219,17 +219,41 @@ FaissVectorIndex::~FaissVectorIndex() {
219219
}
220220
}
221221

222+
namespace {
223+
224+
Status close_index_output(std::unique_ptr<lucene::store::IndexOutput>& output,
225+
const char* description) {
226+
DCHECK(output != nullptr);
227+
try {
228+
output->close();
229+
} catch (const CLuceneError& e) {
230+
output.reset();
231+
return Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>("Failed to close {}: {}",
232+
description, e.what());
233+
}
234+
output.reset();
235+
return Status::OK();
236+
}
237+
238+
} // namespace
239+
222240
struct FaissIndexWriter : faiss::IOWriter {
223241
public:
224-
FaissIndexWriter() = default;
225-
FaissIndexWriter(lucene::store::IndexOutput* output) : _output(output) {}
242+
explicit FaissIndexWriter(lucene::store::IndexOutput* output) : _output(output) {}
226243
~FaissIndexWriter() override {
227244
if (_output != nullptr) {
228-
_output->close();
229-
delete _output;
245+
try {
246+
_output->close();
247+
} catch (const CLuceneError& e) {
248+
// Serialization already failed, so this is cleanup only. A
249+
// destructor must not turn an I/O error into std::terminate.
250+
LOG(WARNING) << "Failed to close vector index output during cleanup: " << e.what();
251+
}
230252
}
231253
}
232254

255+
Status close() { return close_index_output(_output, "vector index output"); }
256+
233257
size_t operator()(const void* ptr, size_t size, size_t nitems) override {
234258
size_t bytes = size * nitems;
235259
if (bytes > 0) {
@@ -252,7 +276,7 @@ struct FaissIndexWriter : faiss::IOWriter {
252276
return nitems;
253277
};
254278

255-
lucene::store::IndexOutput* _output = nullptr;
279+
std::unique_ptr<lucene::store::IndexOutput> _output;
256280
};
257281

258282
struct FaissIndexReader : faiss::IOReader {
@@ -953,84 +977,88 @@ doris::Status FaissVectorIndex::range_search(const float* query_vec, const float
953977
doris::Status FaissVectorIndex::save(lucene::store::Directory* dir) {
954978
auto start_time = std::chrono::high_resolution_clock::now();
955979

956-
if (_index_type == AnnIndexType::IVF_ON_DISK) {
957-
// IVF_ON_DISK: write ivf data to a separate file, then write index metadata.
958-
//
959-
// Why do we replace invlists here in save() instead of at build() time?
960-
// During build/train/add, IndexIVF needs a writable ArrayInvertedLists to
961-
// receive vectors via add_entries(). PreadInvertedLists inherits from
962-
// ReadOnlyInvertedLists and does not support writes. The original
963-
// OnDiskInvertedLists does support writes but requires mmap on a real file,
964-
// which is unavailable at build time (Directory is only passed to save()).
965-
// So the standard faiss pattern is: build in-memory with ArrayInvertedLists,
966-
// then convert to on-disk format at serialization time. The replace_invlists
967-
// call in Step 2 is purely a serialization format switch (to emit "ilod"
968-
// fourcc instead of "ilar"), not a runtime data structure change.
969-
//
970-
// Step 1: The in-memory index has ArrayInvertedLists. Write them to ann.ivfdata
971-
// by converting to OnDiskInvertedLists format:
972-
// For each list: [codes: capacity*code_size][ids: capacity*sizeof(idx_t)]
973-
auto* ivf = dynamic_cast<faiss::IndexIVF*>(_index.get());
974-
DCHECK(ivf != nullptr);
975-
auto* ails = dynamic_cast<faiss::ArrayInvertedLists*>(ivf->invlists);
976-
DCHECK(ails != nullptr);
977-
978-
const size_t nlist = ails->nlist;
979-
const size_t code_size = ails->code_size;
980-
981-
// Build OnDiskOneList metadata and write data to ann.ivfdata
982-
std::vector<faiss::OnDiskOneList> lists(nlist);
983-
lucene::store::IndexOutput* ivfdata_output = dir->createOutput(faiss_ivfdata_file_name);
984-
size_t offset = 0;
985-
for (size_t i = 0; i < nlist; i++) {
986-
size_t list_size = ails->list_size(i);
987-
lists[i].size = list_size;
988-
lists[i].capacity = list_size;
989-
lists[i].offset = offset;
990-
991-
if (list_size > 0) {
992-
// Write codes
993-
const uint8_t* codes = ails->get_codes(i);
994-
size_t codes_bytes = list_size * code_size;
995-
ivfdata_output->writeBytes(codes, cast_set<Int32>(codes_bytes));
996-
997-
// Write ids
998-
const faiss::idx_t* ids = ails->get_ids(i);
999-
size_t ids_bytes = list_size * sizeof(faiss::idx_t);
1000-
ivfdata_output->writeBytes(reinterpret_cast<const uint8_t*>(ids),
1001-
cast_set<Int32>(ids_bytes));
1002-
}
980+
try {
981+
if (_index_type == AnnIndexType::IVF_ON_DISK) {
982+
// IVF_ON_DISK: write ivf data to a separate file, then write index metadata.
983+
//
984+
// Why do we replace invlists here in save() instead of at build() time?
985+
// During build/train/add, IndexIVF needs a writable ArrayInvertedLists to
986+
// receive vectors via add_entries(). PreadInvertedLists inherits from
987+
// ReadOnlyInvertedLists and does not support writes. The original
988+
// OnDiskInvertedLists does support writes but requires mmap on a real file,
989+
// which is unavailable at build time (Directory is only passed to save()).
990+
// So the standard faiss pattern is: build in-memory with ArrayInvertedLists,
991+
// then convert to on-disk format at serialization time. The replace_invlists
992+
// call in Step 2 is purely a serialization format switch (to emit "ilod"
993+
// fourcc instead of "ilar"), not a runtime data structure change.
994+
//
995+
// Step 1: The in-memory index has ArrayInvertedLists. Write them to ann.ivfdata
996+
// by converting to OnDiskInvertedLists format:
997+
// For each list: [codes: capacity*code_size][ids: capacity*sizeof(idx_t)]
998+
auto* ivf = dynamic_cast<faiss::IndexIVF*>(_index.get());
999+
DCHECK(ivf != nullptr);
1000+
auto* ails = dynamic_cast<faiss::ArrayInvertedLists*>(ivf->invlists);
1001+
DCHECK(ails != nullptr);
1002+
1003+
const size_t nlist = ails->nlist;
1004+
const size_t code_size = ails->code_size;
1005+
1006+
// Build OnDiskOneList metadata and write data to ann.ivfdata
1007+
std::vector<faiss::OnDiskOneList> lists(nlist);
1008+
auto ivfdata_output = std::unique_ptr<lucene::store::IndexOutput>(
1009+
dir->createOutput(faiss_ivfdata_file_name));
1010+
size_t offset = 0;
1011+
for (size_t i = 0; i < nlist; i++) {
1012+
size_t list_size = ails->list_size(i);
1013+
lists[i].size = list_size;
1014+
lists[i].capacity = list_size;
1015+
lists[i].offset = offset;
1016+
1017+
if (list_size > 0) {
1018+
// Write codes
1019+
const uint8_t* codes = ails->get_codes(i);
1020+
size_t codes_bytes = list_size * code_size;
1021+
ivfdata_output->writeBytes(codes, cast_set<Int32>(codes_bytes));
1022+
1023+
// Write ids
1024+
const faiss::idx_t* ids = ails->get_ids(i);
1025+
size_t ids_bytes = list_size * sizeof(faiss::idx_t);
1026+
ivfdata_output->writeBytes(reinterpret_cast<const uint8_t*>(ids),
1027+
cast_set<Int32>(ids_bytes));
1028+
}
10031029

1004-
offset += list_size * (code_size + sizeof(faiss::idx_t));
1030+
offset += list_size * (code_size + sizeof(faiss::idx_t));
1031+
}
1032+
size_t totsize = offset;
1033+
RETURN_IF_ERROR(close_index_output(ivfdata_output, "IVF data output"));
1034+
1035+
// Step 2: Replace ArrayInvertedLists with OnDiskInvertedLists so that
1036+
// write_index serializes in "ilod" format (metadata only).
1037+
auto* od = new faiss::OnDiskInvertedLists();
1038+
od->nlist = nlist;
1039+
od->code_size = code_size;
1040+
od->lists = std::move(lists);
1041+
od->totsize = totsize;
1042+
od->ptr = nullptr;
1043+
od->read_only = true;
1044+
// filename is not used during load (we use separate ivfdata file),
1045+
// but write it for format completeness.
1046+
od->filename = faiss_ivfdata_file_name;
1047+
ivf->replace_invlists(od, true);
1048+
1049+
// Step 3: Write index metadata to ann.faiss (includes "ilod" fourcc)
1050+
FaissIndexWriter writer(dir->createOutput(faiss_index_fila_name));
1051+
RETURN_IF_CATCH_EXCEPTION(faiss::write_index(_index.get(), &writer));
1052+
RETURN_IF_ERROR(writer.close());
1053+
} else {
1054+
// HNSW / IVF: write the full index to ann.faiss
1055+
FaissIndexWriter writer(dir->createOutput(faiss_index_fila_name));
1056+
RETURN_IF_CATCH_EXCEPTION(faiss::write_index(_index.get(), &writer));
1057+
RETURN_IF_ERROR(writer.close());
10051058
}
1006-
size_t totsize = offset;
1007-
ivfdata_output->close();
1008-
delete ivfdata_output;
1009-
1010-
// Step 2: Replace ArrayInvertedLists with OnDiskInvertedLists so that
1011-
// write_index serializes in "ilod" format (metadata only).
1012-
auto* od = new faiss::OnDiskInvertedLists();
1013-
od->nlist = nlist;
1014-
od->code_size = code_size;
1015-
od->lists = std::move(lists);
1016-
od->totsize = totsize;
1017-
od->ptr = nullptr;
1018-
od->read_only = true;
1019-
// filename is not used during load (we use separate ivfdata file),
1020-
// but write it for format completeness.
1021-
od->filename = faiss_ivfdata_file_name;
1022-
ivf->replace_invlists(od, true);
1023-
1024-
// Step 3: Write index metadata to ann.faiss (includes "ilod" fourcc)
1025-
lucene::store::IndexOutput* idx_output = dir->createOutput(faiss_index_fila_name);
1026-
auto writer = std::make_unique<FaissIndexWriter>(idx_output);
1027-
faiss::write_index(_index.get(), writer.get());
1028-
1029-
} else {
1030-
// HNSW / IVF: write the full index to ann.faiss
1031-
lucene::store::IndexOutput* idx_output = dir->createOutput(faiss_index_fila_name);
1032-
auto writer = std::make_unique<FaissIndexWriter>(idx_output);
1033-
faiss::write_index(_index.get(), writer.get());
1059+
} catch (const CLuceneError& e) {
1060+
return Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
1061+
"Failed to save vector index: {}", e.what());
10341062
}
10351063

10361064
auto end_time = std::chrono::high_resolution_clock::now();

‎be/src/storage/index/index_file_writer.cpp‎

Lines changed: 43 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -134,8 +134,8 @@ Status IndexFileWriter::_insert_directory_into_map(int64_t index_id,
134134

135135
Result<std::shared_ptr<DorisFSDirectory>> IndexFileWriter::open(const TabletIndex* index_meta) {
136136
// No index under SNII writes through a CLucene filesystem directory: text
137-
// postings go through the SPIMI writer, and an ANN index stages into memory
138-
// (see open_ann_directory) so that nothing has to be cleaned off disk.
137+
// postings go through the SPIMI writer, and an ANN index uses self-cleaning
138+
// per-file staging (see open_ann_directory).
139139
if (_storage_format == InvertedIndexStorageFormatPB::SNII) {
140140
return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_NOT_SUPPORTED>(
141141
"SNII format does not open CLucene filesystem directories"));
@@ -186,6 +186,37 @@ Result<std::shared_ptr<lucene::store::Directory>> IndexFileWriter::_open_snii_an
186186
return dir;
187187
}
188188

189+
void IndexFileWriter::discard_ann_staging_directory(const TabletIndex* index_meta) {
190+
// Only SNII stages an ANN index somewhere disposable. Branching here rather
191+
// than in the caller keeps the format knowledge on the side that owns it,
192+
// exactly as open_ann_directory() does.
193+
if (_storage_format != InvertedIndexStorageFormatPB::SNII) {
194+
return;
195+
}
196+
DCHECK(index_meta != nullptr);
197+
const auto key = std::make_pair(index_meta->index_id(), index_meta->get_index_suffix());
198+
_indices_dirs.erase(key);
199+
_snii_blob_dir_metas.erase(key);
200+
}
201+
202+
void IndexFileWriter::abandon_snii_staging() {
203+
if (_storage_format != InvertedIndexStorageFormatPB::SNII) {
204+
return;
205+
}
206+
// Empty each directory before dropping the map. Releasing only this writer's
207+
// reference would free nothing while a producer is still alive holding the
208+
// same directory through its own _dir -- which is exactly the state a segment
209+
// that failed before clear() is in. Emptying makes the abort independent of
210+
// who else is still holding on.
211+
for (const auto& [key, dir] : _indices_dirs) {
212+
if (std::strcmp(dir->getObjectName(),
213+
snii_doris::SniiBlobStagingDirectory::getClassName()) == 0) {
214+
static_cast<snii_doris::SniiBlobStagingDirectory*>(dir.get())->discard_staged_files();
215+
}
216+
}
217+
_release_snii_blob_directories();
218+
}
219+
189220
Status IndexFileWriter::_seal_snii_blob_directories() {
190221
DORIS_CHECK(_storage_format == InvertedIndexStorageFormatPB::SNII);
191222
for (const auto& [key, dir] : _indices_dirs) {
@@ -202,25 +233,26 @@ Status IndexFileWriter::_seal_snii_blob_directories() {
202233
DORIS_CHECK(std::strcmp(dir->getObjectName(),
203234
snii_doris::SniiBlobStagingDirectory::getClassName()) == 0);
204235

205-
// Nothing here can throw: the staged bytes are plain buffers, and each
206-
// source keeps its own alive, so finish() may pull them after this
207-
// directory is gone.
236+
// The sources TAKE the staged files, so they own them alone from here on:
237+
// finish() may pull the bytes after this directory is gone, and it can
238+
// unlink each sub-file as soon as it has copied it instead of waiting for
239+
// whoever else happens to still hold the directory.
208240
auto* staging = static_cast<snii_doris::SniiBlobStagingDirectory*>(dir.get());
209241
// All cold: a faiss index is read at QUERY time, never at container open,
210242
// so nothing here belongs in the hot area the text metadata groups share.
211243
RETURN_IF_ERROR(add_snii_blob_index(meta_it->second.get(),
212244
doris::snii::format::LogicalIndexKind::kAnn,
213-
staging->blob_sources(), {}));
245+
staging->take_blob_sources(), {}));
214246
}
215247
return Status::OK();
216248
}
217249

218250
void IndexFileWriter::_release_snii_blob_directories() {
219-
// Dropping the map is the whole release: a staging directory holds its bytes
220-
// in memory and owns no file, so there is nothing on disk to remove and --
221-
// unlike DorisFSDirectory::deleteDirectory() -- no throwing call to make from
222-
// a Status-returning close path. Any buffer a registered blob source still
223-
// needs stays alive through that source until finish() has pulled it.
251+
// Dropping the map is all this has to do: sealing already handed the staged
252+
// files to the blob sources, which unlink them as finish() copies them, and
253+
// an unsealed directory unlinks its own on the way out. So -- unlike
254+
// DorisFSDirectory::deleteDirectory() -- there is no throwing cleanup call to
255+
// make from this Status-returning close path.
224256
_indices_dirs.clear();
225257
_snii_blob_dir_metas.clear();
226258
}

‎be/src/storage/index/index_file_writer.h‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,20 @@ class IndexFileWriter {
9191
// need the DorisFSDirectory subclass.
9292
Result<std::shared_ptr<lucene::store::Directory>> open_ann_directory(
9393
const TabletIndex* index_meta);
94+
// SNII only: drops the staging directory of one ANN index whose serialization
95+
// failed. Its sub-files unlink themselves once their last owner is gone, and
96+
// the producer releases its own reference alongside this call -- without that
97+
// a failed save keeps an ANN-sized file and its descriptor on the temp
98+
// filesystem until this writer is destroyed, which for a rowset build is not
99+
// until every other segment has been written. V1/V2 keep their directory: its
100+
// files ARE the index output, and begin_close() is what removes them.
101+
void discard_ann_staging_directory(const TabletIndex* index_meta);
102+
// SNII only: drops the staging of EVERY index on this writer because the
103+
// segment they belong to is being abandoned. Layered above the per-index
104+
// discard, not a duplicate of it: that one fires the instant one ANN
105+
// serialization fails, while this one covers a segment that failed AFTER its
106+
// indexes staged successfully, when nothing else will ever seal them.
107+
void abandon_snii_staging();
94108
// Write-path facts for one SNII index flush.
95109
struct SniiAddIndexOptions {
96110
// This flush serves a stream/broker load (DataWriteType::TYPE_DIRECT):

0 commit comments

Comments
 (0)