Skip to content

Commit 9846834

Browse files
authored
feat: add synchronous index prewarm C and C++ APIs (#94)
C and C++ callers currently cannot invoke the Rust SDK's index prewarm operation before serving queries. Add `lance_dataset_prewarm_index(dataset, index_name)` and `Dataset::prewarm_index(index_name)` as synchronous bindings to the existing SDK method. The operation uses one dataset snapshot and its existing session, prewarms all segments of the named logical index, and preserves the dataset version. It uses the established FFI error/panic handling and rejects NULL, empty, and invalid UTF-8 arguments. The README and public header explain shared-session reuse, eviction, partial failures, snapshot scope, and the distinction between index prewarming and result-row reads. No Lance Rust or dependency changes are required. Tests cover invalid inputs, missing indexes, multi-segment B-tree and IVF-Flat indexes, historical snapshots after index removal, repeated calls, and dataset/session handle lifetimes. An independent cold query records partition loads; queries from a newly opened handle sharing the prewarmed session return the expected nearest neighbors with zero partition loads. C and C++ runtime tests exercise the exported symbol and wrapper. Validation: - `cargo fmt -- --check` - `cargo check --all-targets` - `cargo clippy --all-targets -- -D warnings` - `cargo test` — 479 passed, 0 failed (3 separately executed tests ignored by default) - `cargo test --test compile_and_run_test -- --ignored --test-threads=1` — 3 passed, 0 failed Implements the upstream binding portion of M1 in apache/doris#68692. The Doris dependency update and synchronous SQL integration remain separate follow-up work after this merges.
1 parent cd63420 commit 9846834

7 files changed

Lines changed: 300 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -362,6 +362,53 @@ auto ds = lance::Dataset::open_with_session(session, "data.lance");
362362
auto stats = session.cache_stats();
363363
```
364364

365+
### Prewarm an index synchronously
366+
367+
Prewarm a logical index before serving queries to move cold index reads out of
368+
foreground query execution. The call blocks until the Rust SDK's
369+
`Dataset::prewarm_index` finishes for all segments with that name in the dataset
370+
handle's snapshot. It does not create an index or change the dataset version.
371+
372+
```c
373+
LanceSession* session = lance_session_new(64 * 1024 * 1024, 16 * 1024 * 1024);
374+
if (!session) return -1;
375+
LanceDataset* ds = lance_dataset_open_with_session("data.lance", NULL, 0, session);
376+
if (!ds) {
377+
lance_session_close(session);
378+
return -1;
379+
}
380+
/* Assumes an index named embedding_idx already exists. */
381+
int32_t status = lance_dataset_prewarm_index(ds, "embedding_idx");
382+
if (status != 0) {
383+
const char* message = lance_last_error_message();
384+
fprintf(stderr, "%s\n", message);
385+
lance_free_string(message);
386+
}
387+
lance_dataset_close(ds);
388+
/* Keep session alive and reuse it when opening subsequent query datasets. */
389+
lance_session_close(session);
390+
```
391+
392+
```cpp
393+
lance::Session session(64 * 1024 * 1024, 16 * 1024 * 1024);
394+
auto ds = lance::Dataset::open_with_session(session, "data.lance");
395+
ds.prewarm_index("embedding_idx"); // Blocks; throws lance::Error on failure.
396+
```
397+
398+
The destination is the dataset's process-local session index cache. Reusing the
399+
same session across handles preserves warmed entries after a dataset closes.
400+
Independent sessions and other processes do not share that cache. Entries remain
401+
evictable: success guarantees completion, not that the entire index fits in the
402+
cache or remains resident. Replacing an index creates new segments that must be
403+
warmed separately; historical handles still use their own snapshots. A failure
404+
can leave partially warmed entries in the cache; retrying is safe.
405+
406+
This binding uses the SDK's index-specific prewarm behavior without additional
407+
options. Tests cover multi-segment IVF-Flat and B-tree indexes; other index types
408+
and formats follow the underlying SDK's support and error behavior. It does not
409+
prewarm ordinary data columns or eliminate result-row reads, and it adds no
410+
persistent cache, automatic refresh, or background task management.
411+
365412
### Open at a specific version
366413

367414
`lance_dataset_open` takes a `version` argument — `0` means the latest, any

‎include/lance/lance.h‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2051,6 +2051,26 @@ int32_t lance_dataset_commit_index_segments(
20512051
size_t segment_count
20522052
);
20532053

2054+
/**
2055+
* Synchronously prewarm a logical index in the dataset's session index cache.
2056+
*
2057+
* Delegates to the Rust SDK's prewarm_index for all segments with index_name
2058+
* in the handle's current snapshot. Does not change the dataset version.
2059+
* Index types/formats follow the SDK's prewarm support and error behavior.
2060+
*
2061+
* Reuse the same LanceSession when opening subsequent query datasets to reuse
2062+
* warmed entries across handles. Cache entries remain evictable; success does
2063+
* not guarantee full or permanent residency. Ordinary data columns are not
2064+
* prewarmed. An error may leave partially warmed entries; retrying is safe.
2065+
*
2066+
* dataset must be valid and index_name must be a non-empty, NUL-terminated
2067+
* UTF-8 string. Both must remain valid until this blocking call returns.
2068+
* Returns 0 on success, -1 on error (see lance_last_error_code/message).
2069+
* NULL/empty/invalid UTF-8 arguments yield LANCE_ERR_INVALID_ARGUMENT;
2070+
* an index absent from this snapshot yields LANCE_ERR_NOT_FOUND.
2071+
*/
2072+
int32_t lance_dataset_prewarm_index(const LanceDataset* dataset, const char* index_name);
2073+
20542074
/** Drop an index by name. Returns -1 (NOT_FOUND) if no such index. */
20552075
int32_t lance_dataset_drop_index(LanceDataset* dataset, const char* name);
20562076

‎include/lance/lance.hpp‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1017,6 +1017,14 @@ class Dataset {
10171017
check_error();
10181018
}
10191019

1020+
/// Synchronously prewarm all segments of an index in this snapshot.
1021+
/// Uses the dataset's session index cache; entries remain evictable.
1022+
/// Throws lance::Error on failure, including an absent or empty index name.
1023+
void prewarm_index(const std::string& index_name) const {
1024+
if (lance_dataset_prewarm_index(handle_.get(), index_name.c_str()) != 0)
1025+
check_error();
1026+
}
1027+
10201028
/// Number of user indexes (excludes system indexes).
10211029
uint64_t index_count() const {
10221030
uint64_t n = lance_dataset_index_count(handle_.get());

‎src/index.rs‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -269,6 +269,38 @@ unsafe fn dataset_index_segments_inner(
269269
Ok(0)
270270
}
271271

272+
/// Synchronously prewarm all segments of a logical index in this snapshot.
273+
///
274+
/// Uses the dataset's session index cache. Success does not pin cache entries.
275+
#[unsafe(no_mangle)]
276+
pub unsafe extern "C" fn lance_dataset_prewarm_index(
277+
dataset: *const LanceDataset,
278+
index_name: *const c_char,
279+
) -> i32 {
280+
ffi_try!(unsafe { prewarm_index_inner(dataset, index_name) }, neg)
281+
}
282+
283+
unsafe fn prewarm_index_inner(
284+
dataset: *const LanceDataset,
285+
index_name: *const c_char,
286+
) -> Result<i32> {
287+
if dataset.is_null() {
288+
return Err(lance_core::Error::invalid_input_source(
289+
"dataset must not be NULL".into(),
290+
));
291+
}
292+
let index_name = unsafe { helpers::parse_c_string(index_name)? }
293+
.filter(|name| !name.is_empty())
294+
.ok_or_else(|| {
295+
lance_core::Error::invalid_input_source("index_name must not be NULL or empty".into())
296+
})?;
297+
// Keep one snapshot and its shared session alive without holding the
298+
// dataset lock across potentially long-running index IO.
299+
let snapshot = unsafe { &*dataset }.snapshot();
300+
block_on(snapshot.prewarm_index(index_name))?;
301+
Ok(0)
302+
}
303+
272304
/// Drop an index by name.
273305
#[unsafe(no_mangle)]
274306
pub unsafe extern "C" fn lance_dataset_drop_index(

‎tests/c_api_test.rs‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17804,3 +17804,179 @@ fn test_scanner_nearest_batch_null_arguments() {
1780417804
lance_dataset_close(ds);
1780517805
}
1780617806
}
17807+
17808+
#[test]
17809+
fn test_prewarm_index_invalid_arguments_and_missing_index() {
17810+
let (_tmp, uri) = create_test_dataset();
17811+
unsafe {
17812+
let ds = lance_dataset_open(c_str(&uri).as_ptr(), ptr::null(), 0);
17813+
assert!(!ds.is_null());
17814+
let name = c_str("missing_idx");
17815+
assert_eq!(lance_dataset_prewarm_index(ptr::null(), name.as_ptr()), -1);
17816+
assert_eq!(lance_last_error_code(), LanceErrorCode::InvalidArgument);
17817+
for name in [ptr::null(), c"".as_ptr(), c"\xff".as_ptr()] {
17818+
assert_eq!(lance_dataset_prewarm_index(ds, name), -1);
17819+
assert_eq!(lance_last_error_code(), LanceErrorCode::InvalidArgument);
17820+
}
17821+
assert_eq!(lance_dataset_prewarm_index(ds, name.as_ptr()), -1);
17822+
assert_eq!(lance_last_error_code(), LanceErrorCode::NotFound);
17823+
let message = lance_last_error_message();
17824+
assert!(!message.is_null());
17825+
assert!(
17826+
std::ffi::CStr::from_ptr(message)
17827+
.to_string_lossy()
17828+
.contains("missing_idx")
17829+
);
17830+
lance_free_string(message);
17831+
assert_eq!(lance_dataset_count_rows(ds), 5);
17832+
lance_dataset_close(ds);
17833+
}
17834+
}
17835+
17836+
#[test]
17837+
fn test_prewarm_index_scalar_segments_and_snapshot() {
17838+
let (_tmp, uri, _) = create_scalar_segment_fixture(lance_index::IndexType::BTree, false);
17839+
unsafe {
17840+
let ds = lance_dataset_open(c_str(&uri).as_ptr(), ptr::null(), 0);
17841+
assert!(!ds.is_null());
17842+
assert_eq!(lance_dataset_index_count(ds), 2);
17843+
let version = lance_dataset_version(ds);
17844+
let historical = lance_dataset_open(c_str(&uri).as_ptr(), ptr::null(), version);
17845+
assert!(!historical.is_null());
17846+
assert_eq!(lance_dataset_prewarm_index(ds, c"key_idx".as_ptr()), 0);
17847+
assert_eq!(lance_dataset_version(ds), version);
17848+
assert_eq!(lance_dataset_drop_index(ds, c"key_idx".as_ptr()), 0);
17849+
assert_eq!(lance_dataset_prewarm_index(ds, c"key_idx".as_ptr()), -1);
17850+
assert_eq!(lance_last_error_code(), LanceErrorCode::NotFound);
17851+
assert_eq!(
17852+
lance_dataset_prewarm_index(historical, c"key_idx".as_ptr()),
17853+
0
17854+
);
17855+
assert_eq!(lance_last_error_code(), LanceErrorCode::Ok);
17856+
assert_eq!(lance_dataset_version(historical), version);
17857+
lance_dataset_close(historical);
17858+
lance_dataset_close(ds);
17859+
}
17860+
}
17861+
17862+
#[test]
17863+
fn test_prewarm_index_vector_segments_reuse_shared_session() {
17864+
use lance::index::{DatasetIndexExt, vector::VectorIndexParams};
17865+
let (_tmp, uri) = create_multi_fragment_vector_dataset(2, 64, 8, false);
17866+
lance_c::runtime::block_on(async {
17867+
let mut ds = Dataset::open(&uri).await.unwrap();
17868+
let params = VectorIndexParams::ivf_flat(2, lance_linalg::distance::MetricType::L2);
17869+
let mut segments = Vec::new();
17870+
for fragment in ds.get_fragments() {
17871+
segments.push(
17872+
ds.create_index_builder(&["embedding"], lance_index::IndexType::Vector, &params)
17873+
.name("vector_idx".into())
17874+
.fragments(vec![fragment.id() as u32])
17875+
.execute_uncommitted()
17876+
.await
17877+
.unwrap(),
17878+
);
17879+
}
17880+
ds.commit_existing_index_segments("vector_idx", "embedding", segments)
17881+
.await
17882+
.unwrap();
17883+
});
17884+
17885+
fn query(ds: *mut LanceDataset, row: i32) -> CapturedScanStatistics {
17886+
unsafe {
17887+
let scanner = lance_scanner_new(ds, ptr::null(), ptr::null());
17888+
assert!(!scanner.is_null());
17889+
let query: Vec<f32> = (0..8).map(|i| row as f32 + i as f32 / 8.0).collect();
17890+
assert_eq!(
17891+
lance_scanner_nearest(
17892+
scanner,
17893+
c"embedding".as_ptr(),
17894+
query.as_ptr().cast(),
17895+
8,
17896+
LanceDataType::Float32 as i32,
17897+
1
17898+
),
17899+
0
17900+
);
17901+
assert_eq!(lance_scanner_set_nprobes(scanner, 2), 0);
17902+
let mut captured = CapturedScanStatistics::default();
17903+
assert_eq!(
17904+
lance_scanner_set_statistics_callback(
17905+
scanner,
17906+
Some(capture_scan_statistics),
17907+
(&mut captured as *mut CapturedScanStatistics).cast()
17908+
),
17909+
0
17910+
);
17911+
let batches = scan_all_rows_from_scanner(scanner);
17912+
assert_eq!(batches.iter().map(|b| b.num_rows()).sum::<usize>(), 1);
17913+
assert_eq!(
17914+
batches[0]
17915+
.column_by_name("id")
17916+
.unwrap()
17917+
.as_any()
17918+
.downcast_ref::<Int32Array>()
17919+
.unwrap()
17920+
.value(0),
17921+
row
17922+
);
17923+
assert_eq!(captured.calls, 1);
17924+
lance_scanner_close(scanner);
17925+
captured
17926+
}
17927+
}
17928+
17929+
// Build the index with a different session, so only prewarm can populate
17930+
// the cache used by the first query on each subsequently opened handle.
17931+
unsafe {
17932+
let cold = lance_dataset_open(c_str(&uri).as_ptr(), ptr::null(), 0);
17933+
assert!(!cold.is_null());
17934+
assert!(query(cold, 5).index_partitions_loaded > 0);
17935+
lance_dataset_close(cold);
17936+
17937+
let session = lance_session_new(64 * 1024 * 1024, 16 * 1024 * 1024);
17938+
assert!(!session.is_null());
17939+
let ds = lance_dataset_open_with_session(c_str(&uri).as_ptr(), ptr::null(), 0, session);
17940+
assert!(!ds.is_null());
17941+
assert_eq!(
17942+
lance_dataset_index_segment_count(ds, c"vector_idx".as_ptr()),
17943+
2
17944+
);
17945+
let version = lance_dataset_version(ds);
17946+
assert_eq!(lance_dataset_prewarm_index(ds, c"vector_idx".as_ptr()), 0);
17947+
assert_eq!(lance_dataset_version(ds), version);
17948+
let mut warmed = LanceSessionCacheStats::default();
17949+
assert_eq!(lance_session_get_cache_stats(session, &mut warmed), 0);
17950+
assert!(warmed.index_cache_entries > 0);
17951+
assert!(warmed.index_cache_size_bytes > 0);
17952+
lance_dataset_close(ds);
17953+
17954+
let reopened =
17955+
lance_dataset_open_with_session(c_str(&uri).as_ptr(), ptr::null(), version, session);
17956+
assert!(!reopened.is_null());
17957+
let mut query_stats = Vec::new();
17958+
// Query both segments and all IVF partitions, rather than a single
17959+
// vector that could pass even if prewarm skipped one segment.
17960+
for row in [5, 100] {
17961+
query_stats.push(query(reopened, row));
17962+
}
17963+
let mut queried = LanceSessionCacheStats::default();
17964+
assert_eq!(lance_session_get_cache_stats(session, &mut queried), 0);
17965+
assert!(queried.index_cache_hits > warmed.index_cache_hits);
17966+
for stats in query_stats {
17967+
assert_eq!(
17968+
stats.index_partitions_loaded, 0,
17969+
"prewarmed partitions should not be loaded again"
17970+
);
17971+
}
17972+
// Repeating the operation is safe, and the dataset owns the session
17973+
// even after its C session handle is released.
17974+
lance_session_close(session);
17975+
assert_eq!(
17976+
lance_dataset_prewarm_index(reopened, c"vector_idx".as_ptr()),
17977+
0
17978+
);
17979+
assert_eq!(lance_dataset_version(reopened), version);
17980+
lance_dataset_close(reopened);
17981+
}
17982+
}

‎tests/cpp/test_c_api.c‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1216,6 +1216,11 @@ static void test_commit_index_segments(const char *uri) {
12161216
"commit must bump the dataset version exactly once");
12171217
ASSERT(lance_dataset_index_segment_count(ds, "c_distributed_idx") == 2,
12181218
"committed index must have two segments");
1219+
ASSERT(lance_dataset_prewarm_index(ds, "c_distributed_idx") == 0,
1220+
"prewarm both index segments");
1221+
ASSERT(lance_dataset_prewarm_index(ds, "c_distributed_idx") == 0,
1222+
"repeat prewarm");
1223+
12191224
uint8_t committed_uuids[32] = {0};
12201225
uint64_t committed_count = 0;
12211226
ASSERT(lance_dataset_index_segments(ds, "c_distributed_idx",

‎tests/cpp/test_cpp_api.cpp‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -528,6 +528,18 @@ static void test_index_lifecycle(const std::string& uri) {
528528
assert(json.find("id_idx") != std::string::npos);
529529
printf("listed: %s... ", json.c_str());
530530

531+
const auto& readonly_ds = ds;
532+
readonly_ds.prewarm_index("id_idx");
533+
readonly_ds.prewarm_index("id_idx");
534+
bool missing_index = false;
535+
try {
536+
readonly_ds.prewarm_index("missing_idx");
537+
} catch (const lance::Error& error) {
538+
assert(error.code == LANCE_ERR_NOT_FOUND);
539+
missing_index = true;
540+
}
541+
assert(missing_index);
542+
531543
ds.drop_index("id_idx");
532544
assert(ds.index_count() == 0);
533545

0 commit comments

Comments
 (0)