feat(index): write zone map seeds into data file footers during append#7427
Conversation
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
| //! Index seed writers — compact per-fragment summaries embedded in data files. | ||
| //! | ||
| //! A seed writer observes column values as they are written to a data file, | ||
| //! accumulates compact statistics in memory, and serializes them to a byte |
There was a problem hiding this comment.
At the moment we assume the entire seed can comfortably fit in memory but in theory we could split it into multiple global buffers in the future if we needed to.
f93e5cc to
338199a
Compare
| // Number of rows per zone. Optional for backwards compatibility: absent on | ||
| // datasets written before this field was added. When absent, no seed writer | ||
| // is created for the index. | ||
| optional uint64 rows_per_zone = 1; |
There was a problem hiding this comment.
We will need a vote for this change.
There was a problem hiding this comment.
Hmm, I guess I'll start the discussion.
There was a problem hiding this comment.
Since we are going to touch this file. I'm thinking we need a better name for it.
There was a problem hiding this comment.
🤷 I'm open to suggestions. It's basically the .proto for all index details that need to be in the lance.table package for backwards compatibility (newer indexes go in the lance.index.pb package and go in index.proto)
| // Defaults to true for variable-length column types (strings, binary) and | ||
| // fixed-width types wider than 8 bytes when not explicitly set by the user. | ||
| optional bool use_seeds = 2; | ||
| } |
There was a problem hiding this comment.
It looks like this data file seeds mechanism can be expanded to basically compute indexes at write time. Furthermore, it feels deliberately designed to support more than just ZoneMap in the future. Do we foresee every supported in type having a use_seeds config in the *IndexDetails?
There was a problem hiding this comment.
Possibly. I think seeds make a lot of sense for small zonal indexes like zone map and bloom filter. I think it's a bit more questionable for something like btree or bitmap as the "seed" would probably be similar in size to the data (so we aren't really saving I/O but maybe some compute).
It's also not strictly necessary to record it in the details. I could just look at the data files and see if they have seeds and, if so, use them. However, that felt like a bit of an expensive test. I figured placing this flag in the details would make it easy to do a super cheap check based on the manifest before we start look in any file headers.
Or are you suggesting we promote this to a general index concept so we don't have to repeat ourselves in each index? I think that could be fine too but I'd maybe want the idea to be a bit less experimental first.
wjones127
left a comment
There was a problem hiding this comment.
question: do you think index seeds will only be used for zone maps? Or could you see them be used for other things like FTS or BTREE?
| /// Schema metadata key prefix for all seed buffers: `"lance.seed.<column_name>"`. | ||
| pub const SEED_META_KEY_PREFIX: &str = "lance.seed."; |
There was a problem hiding this comment.
question: why column name? Isn't this supposed to be a column offset?
I remember elsewhere we were saying the field_ids nor field names should be hardcoded anywhere in the data files.
There was a problem hiding this comment.
Hmm, fair point, it could be an issue if the column was renamed after the write and before the index update for example.
Maybe, but it will likely be different for each. For example, for btree, we could store a sorted copy of the column, but this wouldn't actually reduce I/O at all at index build time. It just moves the sort CPU cost from index time to write time and it isn't clear that's a win for anything. Also, it would require caching the entire column in memory during the write. Alternatively, we could capture some kind of n-tiles sketch, which we could use to inform a later indexer so that it knows the expected range of data and can setup a partitioned sort-merge (e.g. I think Spark does this when doing a sort and it requires reading the column twice so this could skip that first read). I'm not sure about FTS, tokenizing the column is pretty expensive, and I don't think people will want to pay that cost at write time. I do think it will be useful for something like a bloom filter index. |
338199a to
6da4175
Compare
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds configurable zone-map seed metadata and APIs, embeds serialized seeds in V2 data files during writes, reconstructs incremental indexes from harvested seeds, preserves fallback updates, and provides benchmarks across several data types. ChangesZone-map seed indexing
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant DatasetWriter
participant V2FileWriter
participant AppendOptimizer
participant ZoneMapPlugin
DatasetWriter->>V2FileWriter: write seed buffer and metadata
AppendOptimizer->>V2FileWriter: read fragment seed data
V2FileWriter-->>AppendOptimizer: return harvested seeds
AppendOptimizer->>ZoneMapPlugin: update index from seeds
ZoneMapPlugin-->>AppendOptimizer: return updated zone-map index
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 19
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
rust/lance-index/src/scalar/zonemap.rs (1)
457-507: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick winUpdate the remaining direct callers for the new parameter.
This signature change leaves calls at Lines 1676, 3339, 3374, 3401, and 3473 with too few arguments, so this test module no longer compiles.
Proposed fix
- ZoneMapIndex::load(store, None, &LanceCache::no_cache()) + ZoneMapIndex::load(store, None, &LanceCache::no_cache(), false) - ZoneMapIndex::try_from_serialized(..., Some(modern_null_rows)) + ZoneMapIndex::try_from_serialized(..., Some(modern_null_rows), false)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-index/src/scalar/zonemap.rs` around lines 457 - 507, Update every remaining direct call to the changed scalar index loading or construction API in the test module, especially the callers near the referenced locations, to pass the new required parameter consistently with the updated signature. Use the surrounding test setup and existing production call sites to determine the correct argument value, ensuring all affected tests compile.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@protos/index_old.proto`:
- Around line 32-38: Clarify the use_seeds documentation to distinguish absent
on-disk values, which represent disabled seeds for old datasets and are treated
as false by create_seed_writer, from creation-time defaults applied by index
creation logic. Document that newly created indexes may explicitly set the field
to true for variable-length or wider fixed-width types, while present false
disables seeds.
In `@python/python/benchmarks/zone_map_seeds.py`:
- Around line 71-117: Public benchmark helpers gen_int_batches, gen_fsb_batches,
gen_large_binary_batches, and run_scenario lack required API documentation.
Either rename them with a leading underscore and update all call sites, or add
docstrings containing usage examples; for run_scenario, also cross-reference
IndexConfig and Dataset.optimize.optimize_indices().
- Around line 118-119: zone_map_seeds should not recursively delete
deterministic, caller-owned paths under --tmpdir. Update the benchmark setup to
create a unique run directory beneath the user-supplied temporary directory,
pass that directory to the scenarios, and remove or limit the shutil.rmtree call
in the dataset setup so only the benchmark-owned run directory can be deleted.
- Around line 102-124: Update run_scenario to declare a dict[str, float] return
type, and annotate the timings variable as dict[str, float] so the
phase-duration mapping is consistently typed.
In `@rust/lance-index/src/scalar.rs`:
- Line 52: The public seed module lacks a linked usage example demonstrating its
main workflow. Update the module documentation for `seed` to include a doc link
and concise example using `IndexSeedWriter` with `FragmentSeed`, showing how to
construct and use them.
In `@rust/lance-index/src/scalar/registry.rs`:
- Around line 282-328: Document the public hooks create_seed_writer and
update_from_seeds with a concise implementation example in the surrounding trait
documentation. Show how create_seed_writer returns an IndexSeedWriter that
observes writes and serializes seed data, and how update_from_seeds consumes
FragmentSeed values to update the reference index or returns None for scan
fallback.
In `@rust/lance-index/src/scalar/seed.rs`:
- Around line 17-59: Add runnable Rust documentation examples for the exported
SEED_META_KEY_PREFIX, IndexSeedWriter, and FragmentSeed contract. Demonstrate
generating the schema metadata key, implementing or invoking a writer through
observe_batch and finish, and later harvesting a FragmentSeed while inspecting
metadata_value and bytes; ensure the examples compile as doctests and use
suitable minimal test data or mocks.
In `@rust/lance-index/src/scalar/zonemap.rs`:
- Around line 142-146: Add linked Rust doc examples for the public zone-map seed
APIs: document rows_per_zone on ZoneMapIndex, ZoneMapSeedWriter, and
ZoneMapSeedWriter::new with examples demonstrating configuration and writer
usage, including the related APIs around the seed writer implementation.
- Around line 1106-1125: Update test_zone_map_seeds_used_during_update to match
default_use_seeds: replace the default Int32 zone map with a default-enabled
type such as LargeBinary, or configure the index parameters to explicitly enable
seeds, while preserving the seed-write assertion.
- Around line 1106-1125: FixedSizeList seed support is inconsistent because
training rejects the type, the writer hook returns None, and default_use_seeds
excludes it. Update the relevant training validation, writer hook, and
default_use_seeds logic to accept FixedSizeList and enable seeds where
appropriate, while preserving existing nested-type handling for unsupported
types.
- Around line 1370-1476: Persist each zone’s actual length in the seed IPC:
update seed batch construction and deserialization around seed_batch_from_zones
and deserialize_seed to write/read a UInt64 zone_length column and use it for
ZoneBound.length instead of rows_per_zone. Update the round-trip test to cover a
partial final zone and assert its restored length matches the actual row count.
- Around line 1275-1294: In create_seed_writer, stop swallowing
ZoneMapSeedWriter::new construction errors: replace the match that converts Err
into Ok(None) with error propagation using ?. Keep Ok(None) only for nested
data, missing rows_per_zone, or disabled seeds, and return the successfully
constructed writer wrapped in Some.
In `@rust/lance/src/dataset/write.rs`:
- Around line 1383-1390: Move the inline imports from
create_seed_writers—DatasetIndexExt, IndexDetails, fetch_index_details, and
IndexMetadata—to the file-level use declarations, removing them from the
function body while preserving all existing references.
- Around line 1406-1424: Add debug or warning logs before each early continue in
the seed-writer setup loop, especially for failures from schema field-path
lookup, missing field metadata, fetch_index_details, and details.get_plugin.
Include the relevant index or field context and the underlying error where
available, while preserving the existing skip behavior.
- Around line 742-746: Update the cleanup call in the seed-flush error path of
the write flow to pass the required target-base argument: invoke
cleanup_data_fragments with cleanup_bases.as_deref() between base_dir and
fragments, matching the loop-error cleanup path and ensuring routed target-base
files are removed.
- Around line 4616-4675: The zone-map seed test currently uses Int32, which does
not enable seeds by default. In the test around gen_batch and the WriteParams
append, either change the val column to a seed-eligible wider type or set
use_seeds: Some(true) in WriteParams, keeping the metadata assertion valid.
- Around line 4590-4592: The test function `test_multi_base_target_all_bases` is
missing its closing brace; add `}` immediately after the `assert_eq!(file_bases,
vec![None, Some(1), Some(2)])` assertion and before the `#[tokio::test]`
declaration for `test_zone_map_seeds_used_during_update`.
In `@rust/lance/src/index/append.rs`:
- Around line 244-246: Replace the bare unwrap in the metadata parsing logic
with an explicit let-else or equivalent pattern that returns Ok(None) when
split(':') yields no value, then parse the extracted segment as u32 while
preserving the existing fallback behavior.
- Around line 206-231: Update the fragment seed lookup to inspect all entries in
`fragment.files` and select the data file containing the indexed column’s seed
metadata instead of always using `files.first()`. Preserve the fallback when no
suitable file is found, and add debug or warning logs for scheduler open and
FileReader initialization failures so I/O errors are observable. Use the
surrounding seed-reading logic and its indexed-column symbols to identify the
required metadata.
---
Outside diff comments:
In `@rust/lance-index/src/scalar/zonemap.rs`:
- Around line 457-507: Update every remaining direct call to the changed scalar
index loading or construction API in the test module, especially the callers
near the referenced locations, to pass the new required parameter consistently
with the updated signature. Use the surrounding test setup and existing
production call sites to determine the correct argument value, ensuring all
affected tests compile.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: e9f56559-ec00-4b8e-a642-f1e286dff8d4
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (10)
protos/index_old.protopython/python/benchmarks/zone_map_seeds.pyrust/lance-index/Cargo.tomlrust/lance-index/src/scalar.rsrust/lance-index/src/scalar/registry.rsrust/lance-index/src/scalar/seed.rsrust/lance-index/src/scalar/zonemap.rsrust/lance/src/dataset/fragment/write.rsrust/lance/src/dataset/write.rsrust/lance/src/index/append.rs
| def gen_int_batches(num_rows: int, batch_size: int): | ||
| rng = np.random.default_rng() | ||
| for start in range(0, num_rows, batch_size): | ||
| n = min(batch_size, num_rows - start) | ||
| yield pa.table({"value": pa.array(rng.integers(0, 2**31, n, dtype=np.int64))}) | ||
|
|
||
|
|
||
| def gen_fsb_batches(num_rows: int, blob_bytes: int, batch_size: int): | ||
| for start in range(0, num_rows, batch_size): | ||
| n = min(batch_size, num_rows - start) | ||
| raw = os.urandom(n * blob_bytes) | ||
| arr = pa.FixedSizeBinaryArray.from_buffers( | ||
| pa.binary(blob_bytes), n, [None, pa.py_buffer(raw)] | ||
| ) | ||
| yield pa.table({"value": arr}) | ||
|
|
||
|
|
||
| def gen_large_binary_batches(num_rows: int, blob_bytes: int, batch_size: int): | ||
| for start in range(0, num_rows, batch_size): | ||
| n = min(batch_size, num_rows - start) | ||
| yield pa.table({"value": pa.array( | ||
| [os.urandom(blob_bytes) for _ in range(n)], | ||
| type=pa.large_binary(), | ||
| )}) | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Core benchmark runner | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| def run_scenario( | ||
| dataset_path: Path, | ||
| schema: pa.Schema, | ||
| seed_rows: int, | ||
| seed_batch_size: int, | ||
| gen_seed_batches, # () -> Iterator[pa.Table] for the initial write | ||
| gen_bulk_batches, # () -> Iterator[pa.Table] for the appended data | ||
| use_seeds: bool, | ||
| ) -> dict: | ||
| """ | ||
| Run one full scenario and return per-phase timings. | ||
|
|
||
| Seeds are only engaged by optimize_indices() when there is at least one | ||
| previously indexed fragment. This scenario ensures that by writing | ||
| seed_rows of data and building the index before any bulk ingest. | ||
| """ |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Document the externally importable benchmark helpers.
gen_*_batches and run_scenario are public names, but they lack API examples and cross-references. Add them (including links to IndexConfig and Dataset.optimize.optimize_indices() for run_scenario) or make purely local helpers private.
As per coding guidelines: “All public APIs must have documentation with examples, and documentation should link to relevant structs and methods.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/python/benchmarks/zone_map_seeds.py` around lines 71 - 117, Public
benchmark helpers gen_int_batches, gen_fsb_batches, gen_large_binary_batches,
and run_scenario lack required API documentation. Either rename them with a
leading underscore and update all call sites, or add docstrings containing usage
examples; for run_scenario, also cross-reference IndexConfig and
Dataset.optimize.optimize_indices().
Source: Coding guidelines
| def run_scenario( | ||
| dataset_path: Path, | ||
| schema: pa.Schema, | ||
| seed_rows: int, | ||
| seed_batch_size: int, | ||
| gen_seed_batches, # () -> Iterator[pa.Table] for the initial write | ||
| gen_bulk_batches, # () -> Iterator[pa.Table] for the appended data | ||
| use_seeds: bool, | ||
| ) -> dict: | ||
| """ | ||
| Run one full scenario and return per-phase timings. | ||
|
|
||
| Seeds are only engaged by optimize_indices() when there is at least one | ||
| previously indexed fragment. This scenario ensures that by writing | ||
| seed_rows of data and building the index before any bulk ingest. | ||
| """ | ||
| if dataset_path.exists(): | ||
| shutil.rmtree(dataset_path) | ||
|
|
||
| index_config = IndexConfig( | ||
| index_type="ZONEMAP", parameters={"use_seeds": use_seeds} | ||
| ) | ||
| timings = {} |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="python/python/benchmarks/zone_map_seeds.py"
echo "== outline =="
ast-grep outline "$file" --view expanded || true
echo
echo "== relevant lines =="
nl -ba "$file" | sed -n '1,220p'
echo
echo "== search for similar return annotations in benchmarks =="
rg -n "-> dict(:|\\[|$)|timings\s*[:=]" python/python/benchmarks -g '*.py' || trueRepository: lance-format/lance
Length of output: 846
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="python/python/benchmarks/zone_map_seeds.py"
echo "== lines 102-130 =="
sed -n '102,130p' "$file" | cat -n
echo
echo "== timing/mapping annotations in file =="
rg -n "\bdict\b|dict\[|timings\s*[:=]|->\s*dict" "$file"
echo
echo "== nearby function signatures =="
sed -n '150,220p' "$file" | cat -nRepository: lance-format/lance
Length of output: 4481
Parameterize the timing mapping type. run_scenario() should return dict[str, float], and timings should be annotated the same way to match the phase-duration mapping.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/python/benchmarks/zone_map_seeds.py` around lines 102 - 124, Update
run_scenario to declare a dict[str, float] return type, and annotate the timings
variable as dict[str, float] so the phase-duration mapping is consistently
typed.
Source: Coding guidelines
| pub mod registry; | ||
| #[cfg(feature = "geo")] | ||
| pub mod rtree; | ||
| pub mod seed; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add a linked usage example for the public seed module.
The exported module has descriptive docs but no example showing the IndexSeedWriter/FragmentSeed workflow.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rust/lance-index/src/scalar.rs` at line 52, The public seed module lacks a
linked usage example demonstrating its main workflow. Update the module
documentation for `seed` to include a doc link and concise example using
`IndexSeedWriter` with `FragmentSeed`, showing how to construct and use them.
Source: Coding guidelines
| for index in indices.iter() { | ||
| if index.fields.len() != 1 { | ||
| continue; | ||
| } | ||
| let field_id = index.fields[0]; | ||
| let Ok(field_path) = dataset.schema().field_path(field_id) else { | ||
| continue; | ||
| }; | ||
| let Some(data_type) = dataset.schema().field(&field_path).map(|f| f.data_type()) else { | ||
| continue; | ||
| }; | ||
|
|
||
| let Ok(index_details) = fetch_index_details(dataset, &field_path, index).await else { | ||
| continue; | ||
| }; | ||
| let details = IndexDetails(index_details.clone()); | ||
| let Ok(plugin) = details.get_plugin() else { | ||
| continue; | ||
| }; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Consider logging when seed-writer setup is skipped.
The let Ok(..) else { continue } / let Some(..) else { continue } guards silently drop errors from field_path, fetch_index_details, and get_plugin. A failed fetch_index_details here silently disables seeds for that index (later falling back to a full scan), which will be hard to diagnose. A debug!/warn! on the skip would preserve observability.
As per coding guidelines: "Log warnings for silent no-op skipped operations."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rust/lance/src/dataset/write.rs` around lines 1406 - 1424, Add debug or
warning logs before each early continue in the seed-writer setup loop,
especially for failures from schema field-path lookup, missing field metadata,
fetch_index_details, and details.get_plugin. Include the relevant index or field
context and the underlying error where available, while preserving the existing
skip behavior.
Source: Coding guidelines
| for fragment in fragments { | ||
| let Some(data_file) = fragment.files.first() else { | ||
| return Ok(None); | ||
| }; | ||
|
|
||
| let path = dataset | ||
| .base | ||
| .clone() | ||
| .join(crate::dataset::DATA_DIR) | ||
| .join(data_file.path.as_str()); | ||
| let Ok(file_scheduler) = scheduler.open_file(&path, &CachedFileSize::unknown()).await | ||
| else { | ||
| return Ok(None); | ||
| }; | ||
|
|
||
| let Ok(reader) = lance_file::reader::FileReader::try_open( | ||
| file_scheduler, | ||
| None, | ||
| Default::default(), | ||
| &dataset.metadata_cache.file_metadata_cache(&path), | ||
| FileReaderOptions::default(), | ||
| ) | ||
| .await | ||
| else { | ||
| return Ok(None); | ||
| }; |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Where is seed schema metadata attached during writes, and to which file?
rg -nP -C3 'add_schema_metadata|schema_metadata_key|flush_seed_writers' rust/lance/src/dataset/write.rs
# Do v2 fragments ever carry >1 data file for a single indexed scalar column?
rg -nP -C3 'files\.first\(\)|files\[0\]|fn files' rust/lance/src/index/append.rsRepository: lance-format/lance
Length of output: 2406
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== append.rs around harvest loop =="
sed -n '180,290p' rust/lance/src/index/append.rs
echo
echo "== fragment/file layout definitions =="
rg -n -C3 'struct Fragment|enum Fragment|files:|pub files|data files|FileFragment|FragmentFile' rust/lance/src -g '!target'
echo
echo "== docs/comments mentioning file order or multiple files per fragment =="
rg -n -C3 'first\(\)|files\[0\]|multiple data files|blob-bearing|v2\.2|seed' rust/lance/src rust/lance-core/src -g '!target'
echo
echo "== logging in the harvest/fallback path =="
rg -n -C3 'debug!|warn!|info!|trace!' rust/lance/src/index/append.rsRepository: lance-format/lance
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== rust/lance/src/index/append.rs (harvest loop and surrounding helpers) =="
sed -n '180,270p' rust/lance/src/index/append.rs
echo
echo "== rust/lance/src/dataset/fragment.rs: file ordering and metadata helpers =="
sed -n '1350,1455p' rust/lance/src/dataset/fragment.rs
echo
echo "== occurrences of fragment.files.first() / files[0] in codebase =="
rg -n -C2 'fragment\.files\.first\(\)|metadata\.files\[0\]|files\[0\]' rust/lance/src -g '!target' | head -n 120
echo
echo "== harvest/fallback logging in append.rs =="
rg -n -C2 'debug!|warn!|info!|trace!' rust/lance/src/index/append.rs | head -n 80Repository: lance-format/lance
Length of output: 16606
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== try_harvest_seeds callers and handling =="
rg -n -C4 'try_harvest_seeds\(' rust/lance/src/index/append.rs rust/lance/src -g '!target'
echo
echo "== seed writer plumbing in write.rs =="
rg -n -C4 'seed_writers|flush_seed_writers|schema_metadata_key|schema_metadata_value|add_global_buffer' rust/lance/src/dataset/write.rs
echo
echo "== fragment/file semantics: how many files per fragment and what they represent =="
rg -n -C3 'files.len\(\)|files: vec!\[|split.*file|multiple data files|column.*file|field ids are distinct between data files' rust/lance/src/dataset/fragment.rs rust/lance/src/dataset/write.rs rust/lance/src/dataset/transaction.rs | head -n 120Repository: lance-format/lance
Length of output: 17653
Don’t assume the seed is in fragment.files.first(). Fragments can have multiple data files with different field coverage, and the seed metadata is written per file, so this can miss the indexed column on multi-file fragments and force a full scan. If the Ok(None) fallback stays, add a debug/warn on open/read failures so I/O problems aren’t silent.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rust/lance/src/index/append.rs` around lines 206 - 231, Update the fragment
seed lookup to inspect all entries in `fragment.files` and select the data file
containing the indexed column’s seed metadata instead of always using
`files.first()`. Preserve the fallback when no suitable file is found, and add
debug or warning logs for scheduler open and FileReader initialization failures
so I/O errors are observable. Use the surrounding seed-reading logic and its
indexed-column symbols to identify the required metadata.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
rust/lance/src/index/append.rs (1)
211-218: 🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy liftResolve seed files through their
base_id.This always opens
dataset.base/data/<path>using the primary store. Appended V2 files may be routed to a target base, so harvesting those seeds fails and unnecessarily falls back to a full column scan. Resolve the store and path fromdata_file.base_id(including non-dataset-root bases) before opening the file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/index/append.rs` around lines 211 - 218, Update the seed-file handling around scheduler.open_file to resolve both the target store and file path from data_file.base_id, rather than always joining dataset.base with DATA_DIR and data_file.path. Support base IDs that refer to non-dataset-root bases, then open the resolved path with the corresponding scheduler before retaining the existing Ok(None) fallback.rust/lance/src/dataset/write.rs (1)
601-611: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winRestrict or document
do_write_fragments.
rust/lance/src/dataset.rsexposesdataset::writepublicly, so this helper is reachable as API surface even though it’s only used internally. Make itpub(crate)if callers shouldn’t use it; otherwise add API docs forseed_writersand the write behavior.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/dataset/write.rs` around lines 601 - 611, Restrict do_write_fragments to pub(crate) because it is an internal helper rather than intended public API. Keep its existing behavior and signature otherwise unchanged, including the seed_writers parameter.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@rust/lance/src/dataset/write.rs`:
- Around line 601-611: Restrict do_write_fragments to pub(crate) because it is
an internal helper rather than intended public API. Keep its existing behavior
and signature otherwise unchanged, including the seed_writers parameter.
In `@rust/lance/src/index/append.rs`:
- Around line 211-218: Update the seed-file handling around scheduler.open_file
to resolve both the target store and file path from data_file.base_id, rather
than always joining dataset.base with DATA_DIR and data_file.path. Support base
IDs that refer to non-dataset-root bases, then open the resolved path with the
corresponding scheduler before retaining the existing Ok(None) fallback.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 492506d9-32f4-46f2-891e-85462448bae5
📒 Files selected for processing (5)
protos/index_old.protopython/python/benchmarks/zone_map_seeds.pyrust/lance-index/src/scalar/zonemap.rsrust/lance/src/dataset/write.rsrust/lance/src/index/append.rs
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (7)
rust/lance-index/src/scalar/zonemap.rs (3)
1414-1420: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winMove production imports to module scope.
The
arrow_ipcandCursorimports are insidedeserialize_seed; repository guidelines require imports at the top of the file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-index/src/scalar/zonemap.rs` around lines 1414 - 1420, Move the arrow_ipc::reader::FileReader and std::io::Cursor imports from inside deserialize_seed to the module-level import section, then keep deserialize_seed unchanged otherwise.Source: Coding guidelines
3505-3509: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winMove test imports to module scope.
The new tests add
usedeclarations inside function bodies; move these imports to the test module’s import block.Also applies to: 3565-3569, 3602-3605
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-index/src/scalar/zonemap.rs` around lines 3505 - 3509, Move the test-only imports from the function bodies of test_zone_map_seed_writer_round_trip and the other newly added tests into the enclosing test module’s import block. Consolidate the referenced IndexSeedWriter, ZoneMapSeedWriter, ArrayRef, and Int32Array imports at module scope, removing the duplicate local use declarations.Source: Coding guidelines
1469-1483: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winReject invalid persisted bounds before rebuilding the index.
i as u64 * rows_per_zonecan overflow, andzone_length_col.value(i) as usizecan truncate or accept lengths larger thanrows_per_zone. A malformed seed can therefore produce incorrect zone bounds. Use checked arithmetic, fallible conversion, and validate0 < zone_length <= rows_per_zonewith contextual errors.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-index/src/scalar/zonemap.rs` around lines 1469 - 1483, The zone reconstruction loop creating ZoneMapStatistics must reject malformed persisted bounds before pushing them. In the loop around zone_start and zone_length_col.value, use checked multiplication for the zone start, a fallible conversion for the persisted length, and validate that length is greater than zero and no larger than rows_per_zone; return contextual errors for overflow, conversion failure, or invalid lengths.Source: Coding guidelines
rust/lance/src/dataset/write.rs (4)
1407-1408: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winPreallocate the seed-writer vector.
The index count is already known, so initialize this with
Vec::with_capacity(indices.len())to avoid repeated reallocations.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/dataset/write.rs` around lines 1407 - 1408, Update the seed-writer vector initialization in the dataset index-writing flow to use Vec::with_capacity(indices.len()), leveraging the count loaded by dataset.load_indices() and preserving the existing writer type and behavior.Source: Coding guidelines
601-611: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winDocument or narrow the public
do_write_fragmentsAPI.This public function now exposes
seed_writers, but has no rustdoc or example explaining ownership, observation, and flush behavior. If it is crate-internal, preferpub(crate); otherwise document the contract with a compiling example.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/dataset/write.rs` around lines 601 - 611, Narrow do_write_fragments to pub(crate) if it is not intended for external callers; otherwise add rustdoc documenting seed_writers ownership, observation, and flush behavior, including a compiling example. Preserve the existing write behavior and signature semantics.Source: Coding guidelines
4684-4708: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftMake the integration test prove seed harvesting, not only fallback correctness.
The test checks that metadata exists and that the optimized index is correct, but a broken seed-update path would still pass because optimization can fall back to scanning the data. Add a seam, metric, or failure-injection assertion that confirms the harvested-seed path was actually selected.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/dataset/write.rs` around lines 4684 - 4708, Strengthen the integration test around Dataset::optimize_indices so it verifies seed harvesting was selected rather than merely validating the resulting ZoneMap. Add an available seam, metric, or failure-injection assertion before or during optimization, and assert that the seed-update path is exercised while preserving the existing index correctness checks.
4599-4611: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winMove test imports to module scope.
The integration test declares all imports inside the function body; move them to the test module’s import block.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/dataset/write.rs` around lines 4599 - 4611, Move the imports currently declared inside test_zone_map_seeds_used_during_update to the surrounding test module’s import block, removing the redundant function-local use declarations while preserving the test’s behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@rust/lance-index/src/scalar/zonemap.rs`:
- Around line 1414-1420: Move the arrow_ipc::reader::FileReader and
std::io::Cursor imports from inside deserialize_seed to the module-level import
section, then keep deserialize_seed unchanged otherwise.
- Around line 3505-3509: Move the test-only imports from the function bodies of
test_zone_map_seed_writer_round_trip and the other newly added tests into the
enclosing test module’s import block. Consolidate the referenced
IndexSeedWriter, ZoneMapSeedWriter, ArrayRef, and Int32Array imports at module
scope, removing the duplicate local use declarations.
- Around line 1469-1483: The zone reconstruction loop creating ZoneMapStatistics
must reject malformed persisted bounds before pushing them. In the loop around
zone_start and zone_length_col.value, use checked multiplication for the zone
start, a fallible conversion for the persisted length, and validate that length
is greater than zero and no larger than rows_per_zone; return contextual errors
for overflow, conversion failure, or invalid lengths.
In `@rust/lance/src/dataset/write.rs`:
- Around line 1407-1408: Update the seed-writer vector initialization in the
dataset index-writing flow to use Vec::with_capacity(indices.len()), leveraging
the count loaded by dataset.load_indices() and preserving the existing writer
type and behavior.
- Around line 601-611: Narrow do_write_fragments to pub(crate) if it is not
intended for external callers; otherwise add rustdoc documenting seed_writers
ownership, observation, and flush behavior, including a compiling example.
Preserve the existing write behavior and signature semantics.
- Around line 4684-4708: Strengthen the integration test around
Dataset::optimize_indices so it verifies seed harvesting was selected rather
than merely validating the resulting ZoneMap. Add an available seam, metric, or
failure-injection assertion before or during optimization, and assert that the
seed-update path is exercised while preserving the existing index correctness
checks.
- Around line 4599-4611: Move the imports currently declared inside
test_zone_map_seeds_used_during_update to the surrounding test module’s import
block, removing the redundant function-local use declarations while preserving
the test’s behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 61c7983d-7901-4781-aec9-f1c8da4c50bb
⛔ Files ignored due to path filters (2)
java/lance-jni/Cargo.lockis excluded by!**/*.lockpython/Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (2)
rust/lance-index/src/scalar/zonemap.rsrust/lance/src/dataset/write.rs
wjones127
left a comment
There was a problem hiding this comment.
I have a minor question but it otherwise looks good.
| flush_seed_writers(w.as_mut(), &mut seed_writers).await?; | ||
| let (num_rows, data_file) = w.finish().await?; |
There was a problem hiding this comment.
question: Below, if there's any error from flush_seed_writers, we drop the writer and call cleanup_data_fragments. But we don't here. Any particular reason?
Introduces "index seeds" — compact per-fragment zone map statistics embedded as global buffers in data file footers at write time. During index update the seeds are harvested directly from the file footer, skipping the full column scan that would otherwise be required. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> refactor(index): remove seed_writers from WriteParams; use &mut self on IndexSeedWriter Move seed writer creation out of WriteParams (a config struct shouldn't hold mutable stateful objects) into write_fragments_internal, passing them as an explicit parameter to do_write_fragments. Switch IndexSeedWriter trait methods to &mut self, eliminating the need for Arc + Mutex. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> refactor(index): make seed writer creation a generic ScalarIndexPlugin capability Add `create_seed_writer` to `ScalarIndexPlugin` with a default `Ok(None)` implementation so any index type can opt in. Implement it on `ZoneMapIndexPlugin` by reading `rows_per_zone` from the index file metadata — no full index load required. Replace the hardcoded `create_zone_map_seed_writers` in the write path with a generic `create_seed_writers` that iterates all single-field indices, resolves the plugin via `IndexDetails::get_plugin`, and delegates to `plugin.create_seed_writer`. Also move `ZONEMAP_SEED_META_KEY_PREFIX` to `seed.rs` as the shared `SEED_META_KEY_PREFIX`. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> refactor(index): store rows_per_zone in ZoneMapIndexDetails; remove I/O from create_seed_writer Add `optional uint64 rows_per_zone` to the `ZoneMapIndexDetails` proto and populate it at all four CreatedIndex emission sites. This lets `ZoneMapIndexPlugin::create_seed_writer` read rows_per_zone directly from the decoded proto with no file I/O. Old datasets that predate the field fall back to ROWS_PER_ZONE_DEFAULT via the optional/None path. Also remove the `index_store` parameter from `ScalarIndexPlugin::create_seed_writer` and document that the method must not perform I/O — all needed parameters must come from `index_details`. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> fix(index): skip seed writer when rows_per_zone absent from ZoneMapIndexDetails Old datasets that predate the rows_per_zone proto field should not fall back to a default value — the seed buffer would silently use the wrong zone size. Return Ok(None) instead so no seed writer is created. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> fix(index): propagate errors from create_seed_writer instead of silently ignoring them Co-Authored-By: Claude Sonnet 4.6 <[email protected]> refactor(index): make seed-based index update a generic ScalarIndexPlugin capability Add `update_from_seeds` to `ScalarIndexPlugin` (default `Ok(None)`) so any index type can opt into seed-based incremental updates without coupling the update path to a specific index type. Add `metadata_value` to `FragmentSeed` so plugins can validate seed compatibility (e.g. confirming `rows_per_zone`). Implement `ZoneMapIndexPlugin::update_from_seeds` with `rows_per_zone` validation and rename `try_harvest_zonemap_seeds` → `try_harvest_seeds` in `append.rs`. Remove the hardcoded `IndexType::ZoneMap` branch; the merge path now tries seeds generically before falling back to a full column scan. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> feat(index): add use_seeds parameter to zone map index Add `use_seeds: bool` to `ZoneMapIndexDetails` proto and propagate it through the zone map plugin stack. The field defaults to `true` for variable-length (string, binary) and wide primitive types (≥ 8 bytes) where skipping a column scan has the most impact, and `false` for narrow fixed-width types (Int32, Float32, …) that are fast enough to scan directly. `ScalarIndexPlugin` gains a `might_use_seeds` hook (default `false`) that lets the update path in `append.rs` skip opening data files entirely when the index configuration is known to never write seeds. `ZoneMapIndexPlugin` implements the hook by reading `use_seeds` from the proto details. `create_seed_writer` now also respects `use_seeds`, so no seed buffer is embedded in the data file for indexes where seeds are disabled. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> fix(index): raise use_seeds threshold to > 8 bytes for fixed-width types 8-byte primitives (Int64, Float64, Timestamp, …) scan fast enough that seed overhead is not worthwhile. `default_use_seeds` now returns true only for variable-length types (strings, binary) and fixed-width types strictly wider than 8 bytes (Decimal128, Decimal256, FixedSizeBinary(n > 8)). Co-Authored-By: Claude Sonnet 4.6 <[email protected]> fix(index): rebase fixup — borrow selected_old_indices as slice `build_per_segment_filters` and `open_and_merge_segments` take `&[&IndexMetadata]` after an upstream signature change in main; pass `&selected_old_indices` to match. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> bench(index): add zone map seeds vs. no-seeds Python benchmark Compares index-update time with and without use_seeds across three data types (int64, FixedSizeBinary 4 KiB, LargeBinary 20 KiB). The benchmark also documents the correct workflow for exercising seeds: an initial write + index build must precede the bulk ingest so that optimize_indices() has a non-empty segment to merge against. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
52fa537 to
888732c
Compare
|
Hi @westonpace , quick question regarding the seed. do we plan to support pruning using seed? Seeds could potentially provide temporary pruning for newly appended, not-yet-indexed fragments. |
This PR introduces the concept of "index seeds". Indexes can opt-in to planting seeds during ingestion. The seeds are placed into the data files as global buffers. Later, when updating the index to include these data files, we can harvest the seeds instead of scanning the data itself.
This is primarily intended to avoid a potentially expensive data scan to update the index. As an example this PR adds index seeds for wide (binary, fixed-size-list, string) columns when creating a zone map index. Now we calculate the min/max/nulls during ingestion, when the data is already present and flowing through the system. Then, when we go to update the index, all we are doing is reading back those counts and adding them to the index (instead of scanning the large column all over again).
Summary by CodeRabbit
rows-per-zoneand seed usage (use-seeds) with backward-compatible behavior when omitted.