test(python): add data overlay benchmark suite#7544
Conversation
1156dd2 to
687507f
Compare
687507f to
92734fa
Compare
d1e3be6 to
d1fa12b
Compare
92734fa to
9a09276
Compare
91b3987 to
ecfb29c
Compare
6d52845 to
968d024
Compare
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
40a4a5c to
c3837ce
Compare
📝 WalkthroughWalkthroughAdds shared utilities and datagen support for creating overlay datasets, then introduces benchmarks measuring manifest growth and overlay read performance across layer counts, coverage patterns, formats, workloads, and embedding widths. ChangesOverlay benchmark suite
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Benchmark
participant OverlayUtilities
participant LanceDataset
participant ReadWorkload
Benchmark->>OverlayUtilities: create base dataset and overlays
OverlayUtilities->>LanceDataset: commit DataOverlay layers
Benchmark->>LanceDataset: verify overlay visibility
Benchmark->>ReadWorkload: run take or scan measurements
ReadWorkload->>LanceDataset: read dataset and collect I/O stats
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Benchmark resultsRebased onto latest Setup: 1M rows, local disk (NVMe), Manifest growth (
|
| overlays | 1% contiguous | 1% stride | 10% stride |
|---|---|---|---|
| 1 | 16.4 KB/ea | 40.2 KB/ea | 253 KB/ea |
| 4 | 10.3 KB/ea | 25.2 KB/ea | 158 KB/ea |
| 16 | 8.8 KB/ea | 21.4 KB/ea | 134 KB/ea |
| 64 | 8.4 KB/ea | 20.5 KB/ea | 128 KB/ea |
(64 layers @ 10% stride → ~8.2 MB manifest total.)
Write cost (test_overlay_write.py)
Updating 1% of rows via an overlay vs. the existing rewrite-based mechanisms. narrow = id + val; wide = id + val + 64-d float32 payload:
| approach | narrow: read / persisted | wide: read / persisted |
|---|---|---|
| update | 6.2 MB / 100 KB | 8.7 MB / 2.66 MB |
| merge_insert | 2.4 MB / 101 KB | 2.4 MB / 2.66 MB |
| merge_insert (indexed) | 12.7 MB / 100 KB | 15.3 MB / 2.66 MB |
| overlay | 143 B / 105 KB | 143 B / 105 KB |
| full_rewrite | 6.2 MB / 6.17 MB | 262 MB / 262 MB |
The rewrite-based approaches delete-and-reinsert whole rows, so their cost scales with row width. An overlay only ever touches the changed column: read bytes stay ~0 and persisted bytes stay constant (~105 KB, dominated by the coverage-bitmap + manifest overhead) regardless of how wide the row is — the advantage over update/merge_insert grows directly with row width, and vs. full_rewrite it's already ~60-2500x smaller at this width.
Read scaling (test_overlay_read.py)
1M-row int32 scan/take, strided coverage, v2.0 format:
| overlays | take (mean) | scan (mean) |
|---|---|---|
| 0 | ~0.25 ms | ~1.7 ms |
| 4 | ~0.7 ms | ~7-8 ms |
| 16 | ~1.2 ms | ~40 ms |
- take stays roughly flat (rank pushdown means only the requested rows are touched).
- scan degrades close to linearly with overlay-layer count (~24x at 16 layers) — this is the existing
route_overlayscost profile (offset-major bitmap probing), not something this PR changes. It's the strongest argument for overlay compaction. - Wide embeddings (3072-d): scan is ~600-870 ms regardless of overlay count, since base I/O volume dominates over overlay-routing cost at that width.
cc for the demo this week: the standout numbers are the overlay write cost (143 B read, constant persisted bytes independent of row width) and the scan-cost-vs-layer-count curve as the motivating case for compaction.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Adds a Python benchmark suite for data overlay files in python/ci_benchmarks/, covering three areas: - manifest size growth vs. number of overlays and coverage, - bytes written updating 1% of a column via an overlay vs. update / merge_insert / full rewrite, across narrow and wide rows, - take and scan time + cold read IO as overlay layers, coverage fraction, fragmentation, and data type vary. Shared dataset/overlay construction and measurement helpers live in ci_benchmarks/overlays.py and build fixtures through the public lance.LanceOperation.DataOverlay binding. The write benchmark verifies each approach actually performs the update and splits persisted bytes into data vs. metadata, since the rewrite mechanisms are delete-and- reinsert (whole rows, all columns) while an overlay writes only the changed column. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The overlay read benchmark's take/scan scaling sweep only covered a 4-byte int32 value column, where the base read is trivial and per-layer merge cost dominates. ML overlays typically target wide vector columns, so add test_overlay_read_wide: the same overlays x coverage-pattern grid on a 3072-d float32 embedding (12 KiB/row). Thread an embedding_dim through the overlay helpers so the value width is configurable. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
… API `DataOverlayFile` takes a single `offsets` param, not `shared_offsets`, and the release-build opt-in env var is `LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES` (not `LANCE_ENABLE_DATA_OVERLAY_FILES`). Both diverged from the benchmark helper after PR #7540 landed the real Python bindings; commit()'s returned handle skipped flag validation so this only surfaced on a fresh dataset open.
Self-review pass on the overlay benchmark suite:
- Remove the unused `base_path` parameter of `commit_overlay_layers` (the
body derives its data dir from the dataset URI) and the dead `dir_size`
helper; both had no effect and misled the reader.
- Rename `value_type` -> `_value_type` to match the module's private helpers.
- Document the `{u64::MAX - version}.manifest` naming in `manifest_size`.
- Guard the read and manifest benchmark fixtures: assert a covered cell reads
back the overlaid value and that overlays grow the manifest, so a
silently-broken overlay can't let the benchmark pass while measuring plain
base reads.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
c3837ce to
8204329
Compare
…t read benchmark - README: remove the per-benchmark descriptions; keep it to general suite advice. - Manifest/read benchmarks: report via record_property only, no printing. - Drop test_overlay_read_dtype: its int32 and embedding read costs are already covered by test_overlay_read_scaling (int32) and test_overlay_read_wide (embedding) across the full overlay-layer sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The write-cost comparison (overlay vs. update / merge_insert / full rewrite) is a useful one-off measurement but not something we need to keep re-running in CI for regressions. Remove it, along with the helpers it exclusively used (file_sizes, written_breakdown, proc_io_counters, and the make_base_dataset payload column). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The datagen binding could only size generated batches by bytes. The underlying lance-datagen builder already supports an exact row count (into_reader_rows), so expose it: rand_batches gains a rows_per_batch option, mutually exclusive with batch_size_bytes. This lets callers request an exact number of rows per batch rather than approximating via a byte budget. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the hand-rolled base-dataset generator in make_base_dataset with lance._datagen.rand_batches (row-count mode), matching the other ci_benchmarks suites. One batch per fragment; num_rows must be a multiple of rows_per_file. Overlay values keep their own generator so each layer stays distinct from the base (the read-visibility guard depends on that). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
There was a problem hiding this comment.
Actionable comments posted: 2
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
🟡 Other comments (4)
python/src/datagen.rs-32-53 (1)
32-53: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winInclude sizing inputs in both error paths.
The exceptions omit the requested row/byte sizes and batch count, making invalid configuration failures difficult to diagnose. Include parameter names and values in the conflicting-options and reader-construction errors.
As per coding guidelines,
**/*.{rs,py,java}requires full error context, including “variable names, values, sizes, and types.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/src/datagen.rs` around lines 32 - 53, Update the validation error in the rows_in_batch/bytes_in_batch conflict check and the reader-construction error inside the match on rows_in_batch to include the relevant parameter names and values: rows_in_batch, bytes_in_batch, and batch_count, including resolved defaults where applicable. Preserve the existing exception types and generation behavior while providing full sizing context in both messages.Source: Coding guidelines
python/python/ci_benchmarks/benchmarks/test_overlay_read.py-36-38 (1)
36-38: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winParameterize the return type.
Use
list[int]instead of barelistso the indices passed tods.takeretain their element type.-def _take_indices(num_rows: int) -> list: +def _take_indices(num_rows: int) -> list[int]:As per coding guidelines,
python/**/*.pyrequires parameterized type hints and prohibits bare generics.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/python/ci_benchmarks/benchmarks/test_overlay_read.py` around lines 36 - 38, Update the _take_indices function’s return annotation from the bare list generic to list[int], preserving the existing sorted random-index behavior and ensuring the indices passed to ds.take retain their element type.Source: Coding guidelines
python/python/ci_benchmarks/overlays.py-70-74 (1)
70-74: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winValidate dimensions before the modulo operation.
rows_per_file=0raisesZeroDivisionError, andnum_rows=0proceeds until later helpers assume a first fragment exists. Reject non-positive dimensions with a contextualValueErrorbefore calculating divisibility.Proposed fix
def make_base_dataset( ... ) -> lance.LanceDataset: + if ( + not isinstance(num_rows, int) + or isinstance(num_rows, bool) + or not isinstance(rows_per_file, int) + or isinstance(rows_per_file, bool) + or num_rows <= 0 + or rows_per_file <= 0 + ): + raise ValueError( + "num_rows and rows_per_file must be positive integers; " + f"got num_rows={num_rows!r} ({type(num_rows).__name__}), " + f"rows_per_file={rows_per_file!r} ({type(rows_per_file).__name__})" + ) if num_rows % rows_per_file:As per coding guidelines,
**/*.{rs,py,java}requires validating API-boundary inputs with descriptive errors.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/python/ci_benchmarks/overlays.py` around lines 70 - 74, Validate num_rows and rows_per_file as positive values before the divisibility check in the surrounding overlay setup. Raise a contextual ValueError for either non-positive dimension, then retain the existing modulo-based multiple validation for valid inputs.Source: Coding guidelines
python/python/ci_benchmarks/overlays.py-95-100 (1)
95-100: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winReject invalid coverage and preserve the stride pattern.
max(1, ...)silently turns zero or negative coverage into one cell. Also, whencount > num_rows / 2,stepbecomes1, so"stride"degenerates into a leading contiguous run rather than distributing coverage across the fragment.Proposed fix
+import math + def coverage_offsets(num_rows: int, fraction: float, pattern: str) -> List[int]: ... - count = max(1, int(round(num_rows * fraction))) + if num_rows <= 0 or not math.isfinite(fraction) or not 0 < fraction <= 1: + raise ValueError( + f"expected num_rows > 0 and 0 < fraction <= 1; " + f"got num_rows={num_rows!r}, fraction={fraction!r}" + ) + count = int(round(num_rows * fraction)) + if count == 0: + raise ValueError( + f"fraction={fraction!r} selects zero rows for num_rows={num_rows!r}" + ) if pattern == "contiguous": return list(range(count)) if pattern == "stride": - step = max(1, num_rows // count) - return list(range(0, num_rows, step))[:count] + return [(index * num_rows) // count for index in range(count)]As per coding guidelines,
**/*.{rs,py,java}requires invalid inputs to be rejected and never silently adjusted.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/python/ci_benchmarks/overlays.py` around lines 95 - 100, Update the coverage/count calculation in the overlay selection logic to reject zero or negative coverage instead of clamping it to one, raising the module’s appropriate invalid-input exception. Revise the “stride” branch so selected rows remain distributed across the full range when count exceeds half of num_rows, rather than degenerating into a leading contiguous run; preserve existing contiguous behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/src/datagen.rs`:
- Around line 19-25: Add a synchronized Rustdoc example to the public function
annotated with #[pyfunction], using its current schema, batch_count,
bytes_in_batch, and rows_in_batch signature, and link relevant API types such as
RecordBatchReader. Ensure the example reflects the documented mutually exclusive
batch-size options and default behavior without changing the implementation.
- Around line 32-55: Add focused coverage to test_rand_batches or nearby datagen
tests for rows_per_batch, asserting each generated batch has the requested row
count. Also add a case providing both batch_size_bytes and rows_per_batch, and
assert it raises ValueError matching the mutual-exclusion validation in the rand
batch-generation function.
---
Other comments:
In `@python/python/ci_benchmarks/benchmarks/test_overlay_read.py`:
- Around line 36-38: Update the _take_indices function’s return annotation from
the bare list generic to list[int], preserving the existing sorted random-index
behavior and ensuring the indices passed to ds.take retain their element type.
In `@python/python/ci_benchmarks/overlays.py`:
- Around line 70-74: Validate num_rows and rows_per_file as positive values
before the divisibility check in the surrounding overlay setup. Raise a
contextual ValueError for either non-positive dimension, then retain the
existing modulo-based multiple validation for valid inputs.
- Around line 95-100: Update the coverage/count calculation in the overlay
selection logic to reject zero or negative coverage instead of clamping it to
one, raising the module’s appropriate invalid-input exception. Revise the
“stride” branch so selected rows remain distributed across the full range when
count exceeds half of num_rows, rather than degenerating into a leading
contiguous run; preserve existing contiguous behavior.
In `@python/src/datagen.rs`:
- Around line 32-53: Update the validation error in the
rows_in_batch/bytes_in_batch conflict check and the reader-construction error
inside the match on rows_in_batch to include the relevant parameter names and
values: rows_in_batch, bytes_in_batch, and batch_count, including resolved
defaults where applicable. Preserve the existing exception types and generation
behavior while providing full sizing context in both messages.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: 4fd0fc46-890e-4f84-95e5-3dd2715a977c
📒 Files selected for processing (5)
python/python/ci_benchmarks/benchmarks/test_overlay_manifest.pypython/python/ci_benchmarks/benchmarks/test_overlay_read.pypython/python/ci_benchmarks/overlays.pypython/python/lance/_datagen.pypython/src/datagen.rs
| /// Generate `batch_count` batches of random data for `schema`. | ||
| /// | ||
| /// Batch size is set either by `rows_in_batch` (exact rows per batch) or | ||
| /// `bytes_in_batch` (approximate bytes per batch); the two are mutually | ||
| /// exclusive. When neither is given the byte-based default is used. | ||
| #[pyfunction] | ||
| #[pyo3(signature=(schema, batch_count=None, bytes_in_batch=None))] | ||
| #[pyo3(signature=(schema, batch_count=None, bytes_in_batch=None, rows_in_batch=None))] |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add a synchronized usage example and API links.
This public API documents behavior but provides neither an example nor links to relevant types such as RecordBatchReader.
As per coding guidelines, **/*.{rs,md,rst} requires: “Document all public APIs with examples and links to relevant structs and methods; keep examples synchronized with actual signatures.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/src/datagen.rs` around lines 19 - 25, Add a synchronized Rustdoc
example to the public function annotated with #[pyfunction], using its current
schema, batch_count, bytes_in_batch, and rows_in_batch signature, and link
relevant API types such as RecordBatchReader. Ensure the example reflects the
documented mutually exclusive batch-size options and default behavior without
changing the implementation.
Source: Coding guidelines
| if rows_in_batch.is_some() && bytes_in_batch.is_some() { | ||
| return Err(pyo3::exceptions::PyValueError::new_err( | ||
| "rows_in_batch and bytes_in_batch are mutually exclusive", | ||
| )); | ||
| } | ||
| let builder = lance_datagen::rand(&schema.0); | ||
| let batch_count = BatchCount::from(batch_count.unwrap_or(DEFAULT_BATCH_COUNT)); | ||
| let reader: Box<dyn RecordBatchReader> = match rows_in_batch { | ||
| Some(rows) => Box::new(builder.into_reader_rows(RowCount::from(rows), batch_count)), | ||
| None => Box::new( | ||
| builder | ||
| .into_reader_bytes( | ||
| ByteCount::from(bytes_in_batch.unwrap_or(DEFAULT_BATCH_SIZE_BYTES)), | ||
| batch_count, | ||
| lance_datagen::RoundingBehavior::RoundUp, | ||
| ) | ||
| .map_err(|e| { | ||
| pyo3::exceptions::PyValueError::new_err(format!( | ||
| "Failed to generate batches: {}", | ||
| e | ||
| )) | ||
| })?, | ||
| ), | ||
| }; |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
fd -t f -i 'test*datagen*' python
rg -n -C2 'rand_batches|rows_per_batch|rows_in_batch|bytes_in_batch' pythonRepository: lance-format/lance
Length of output: 10858
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## python/src/datagen.rs"
sed -n '1,80p' python/src/datagen.rs
echo
echo "## python/python/tests/test_datagen.py"
cat -n python/python/tests/test_datagen.py
echo
echo "## python/python/lance/_datagen.py"
cat -n python/python/lance/_datagen.py
echo
echo "## benchmark row-mode call"
sed -n '65,85p' python/python/ci_benchmarks/overlays.py
sed -n '1,35p' python/python/benchmarks/test_bulk_write.pyRepository: lance-format/lance
Length of output: 7004
Add focused tests for the row-sizing contract.
test_rand_batches() only exercises byte sizing and batch count assertions; it does not test rows_per_batch row sizes or the mutually exclusive sizing validation. Add datagen tests that assert row mode keeps the expected rows per batch and raises ValueError when both batch_size_bytes and rows_per_batch are provided.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/src/datagen.rs` around lines 32 - 55, Add focused coverage to
test_rand_batches or nearby datagen tests for rows_per_batch, asserting each
generated batch has the requested row count. Also add a case providing both
batch_size_bytes and rows_per_batch, and assert it raises ValueError matching
the mutual-exclusion validation in the rand batch-generation function.
Source: Coding guidelines
Adds a Python benchmark suite for data overlay files in
python/ci_benchmarks/, stacked on #7540 (which exposes theDataOverlaycommit binding these benchmarks use to build fixtures).The benchmarks generate their own base datasets and overlays into a temp dir, so no pre-generation step is needed. Three areas:
test_overlay_manifest.py— manifest growth vs. number of overlays and coverage size/pattern.test_overlay_write.py— bytes written updating 1% of a column via an overlay vs.update/merge_insert/ full rewrite.test_overlay_read.py— take and scan wall time (pytest-benchmark) plus cold read IO (io_stats_incremental) as overlay layers, coverage fraction, fragmentation (contiguous vs. strided), and data type (int32 vs. fixed-size-list embedding) vary, across v2.0 and v2.1.Shared dataset/overlay construction and measurement helpers (
/proc/self/io, on-disk size, manifest size) live inci_benchmarks/overlays.py.Notes:
update-via-overlay path does not exist yet, so the write benchmark's overlay arm is hand-rolled and writes fresh values without reading the base column first; the headline metric is bytes written, latency secondary.🤖 Generated with Claude Code