Skip to content

test(python): add data overlay benchmark suite#7544

Open
wjones127 wants to merge 8 commits into
mainfrom
will/overlay-benchmarks
Open

test(python): add data overlay benchmark suite#7544
wjones127 wants to merge 8 commits into
mainfrom
will/overlay-benchmarks

Conversation

@wjones127

Copy link
Copy Markdown
Contributor

Adds a Python benchmark suite for data overlay files in python/ci_benchmarks/, stacked on #7540 (which exposes the DataOverlay commit binding these benchmarks use to build fixtures).

The benchmarks generate their own base datasets and overlays into a temp dir, so no pre-generation step is needed. Three areas:

  • test_overlay_manifest.py — manifest growth vs. number of overlays and coverage size/pattern.
  • test_overlay_write.py — bytes written updating 1% of a column via an overlay vs. update / merge_insert / full rewrite.
  • test_overlay_read.py — take and scan wall time (pytest-benchmark) plus cold read IO (io_stats_incremental) as overlay layers, coverage fraction, fragmentation (contiguous vs. strided), and data type (int32 vs. fixed-size-list embedding) vary, across v2.0 and v2.1.

Shared dataset/overlay construction and measurement helpers (/proc/self/io, on-disk size, manifest size) live in ci_benchmarks/overlays.py.

Notes:

🤖 Generated with Claude Code

@github-actions github-actions Bot added A-python Python bindings chore labels Jun 30, 2026
@wjones127
wjones127 force-pushed the will/overlay-benchmarks branch 2 times, most recently from 1156dd2 to 687507f Compare June 30, 2026 20:26
@wjones127
wjones127 force-pushed the will/overlay-benchmarks branch from 687507f to 92734fa Compare June 30, 2026 21:25
@wjones127
wjones127 force-pushed the will/overlay-python-commit branch from d1e3be6 to d1fa12b Compare June 30, 2026 23:12
@wjones127
wjones127 force-pushed the will/overlay-benchmarks branch from 92734fa to 9a09276 Compare June 30, 2026 23:12
@wjones127
wjones127 force-pushed the will/overlay-python-commit branch 3 times, most recently from 91b3987 to ecfb29c Compare July 8, 2026 22:13
@wjones127
wjones127 force-pushed the will/overlay-python-commit branch from 6d52845 to 968d024 Compare July 15, 2026 19:52
Base automatically changed from will/overlay-python-commit to main July 15, 2026 23:01
@github-actions

Copy link
Copy Markdown
Contributor

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

  • Start a vote following the Lance community voting process.
    Format specification modifications need 3 binding +1 votes (excluding the
    proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
  • Once the vote passes, link the completed vote in this PR. It should not be
    merged until the vote is linked.

@wjones127
wjones127 force-pushed the will/overlay-benchmarks branch from 40a4a5c to c3837ce Compare July 21, 2026 00:24
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Adds shared utilities and datagen support for creating overlay datasets, then introduces benchmarks measuring manifest growth and overlay read performance across layer counts, coverage patterns, formats, workloads, and embedding widths.

Changes

Overlay benchmark suite

Layer / File(s) Summary
Row-based datagen support
python/src/datagen.rs, python/python/lance/_datagen.py
rand_batches supports row-based or byte-based batch sizing, rejects both modes together, and preserves feature validation.
Overlay dataset construction
python/python/ci_benchmarks/overlays.py
Adds synthetic dataset generation, contiguous and stride coverage, overlay commits, typed values, and manifest-size measurement.
Manifest and read benchmarks
python/python/ci_benchmarks/benchmarks/test_overlay_manifest.py, python/python/ci_benchmarks/benchmarks/test_overlay_read.py
Adds parameterized benchmarks for manifest growth, cold I/O, warm reads, overlay visibility, and wide embedding values.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Benchmark
  participant OverlayUtilities
  participant LanceDataset
  participant ReadWorkload
  Benchmark->>OverlayUtilities: create base dataset and overlays
  OverlayUtilities->>LanceDataset: commit DataOverlay layers
  Benchmark->>LanceDataset: verify overlay visibility
  Benchmark->>ReadWorkload: run take or scan measurements
  ReadWorkload->>LanceDataset: read dataset and collect I/O stats
Loading

Possibly related PRs

Suggested labels: enhancement, A-python

Suggested reviewers: westonpace

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding a Python data overlay benchmark suite.
Description check ✅ Passed The description is directly related to the benchmark suite and matches the changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch will/overlay-benchmarks

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@wjones127

Copy link
Copy Markdown
Contributor Author

Benchmark results

Rebased onto latest main and stacked on #7549 (now itself rebased onto current main — was 28 commits behind and conflicting). Ran the full suite below after fixing two API drifts the benchmark helpers had picked up (DataOverlayFile(offsets=...) not shared_offsets=, and the release-build opt-in env var is LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES). All 65 tests pass.

Setup: 1M rows, local disk (NVMe), release-no-lto build (full LTO release hit a reproducible rustc/LLVM crash on the build machine, unrelated to this PR).

Manifest growth (test_overlay_manifest.py)

Bytes added to the manifest per overlay layer — roaring compresses contiguous coverage noticeably better than strided:

overlays 1% contiguous 1% stride 10% stride
1 16.4 KB/ea 40.2 KB/ea 253 KB/ea
4 10.3 KB/ea 25.2 KB/ea 158 KB/ea
16 8.8 KB/ea 21.4 KB/ea 134 KB/ea
64 8.4 KB/ea 20.5 KB/ea 128 KB/ea

(64 layers @ 10% stride → ~8.2 MB manifest total.)

Write cost (test_overlay_write.py)

Updating 1% of rows via an overlay vs. the existing rewrite-based mechanisms. narrow = id + val; wide = id + val + 64-d float32 payload:

approach narrow: read / persisted wide: read / persisted
update 6.2 MB / 100 KB 8.7 MB / 2.66 MB
merge_insert 2.4 MB / 101 KB 2.4 MB / 2.66 MB
merge_insert (indexed) 12.7 MB / 100 KB 15.3 MB / 2.66 MB
overlay 143 B / 105 KB 143 B / 105 KB
full_rewrite 6.2 MB / 6.17 MB 262 MB / 262 MB

The rewrite-based approaches delete-and-reinsert whole rows, so their cost scales with row width. An overlay only ever touches the changed column: read bytes stay ~0 and persisted bytes stay constant (~105 KB, dominated by the coverage-bitmap + manifest overhead) regardless of how wide the row is — the advantage over update/merge_insert grows directly with row width, and vs. full_rewrite it's already ~60-2500x smaller at this width.

Read scaling (test_overlay_read.py)

1M-row int32 scan/take, strided coverage, v2.0 format:

overlays take (mean) scan (mean)
0 ~0.25 ms ~1.7 ms
4 ~0.7 ms ~7-8 ms
16 ~1.2 ms ~40 ms
  • take stays roughly flat (rank pushdown means only the requested rows are touched).
  • scan degrades close to linearly with overlay-layer count (~24x at 16 layers) — this is the existing route_overlays cost profile (offset-major bitmap probing), not something this PR changes. It's the strongest argument for overlay compaction.
  • Wide embeddings (3072-d): scan is ~600-870 ms regardless of overlay count, since base I/O volume dominates over overlay-routing cost at that width.

cc for the demo this week: the standout numbers are the overlay write cost (143 B read, constant persisted bytes independent of row width) and the scan-cost-vs-layer-count curve as the motivating case for compaction.

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

wjones127 and others added 4 commits July 23, 2026 09:42
Adds a Python benchmark suite for data overlay files in
python/ci_benchmarks/, covering three areas:

- manifest size growth vs. number of overlays and coverage,
- bytes written updating 1% of a column via an overlay vs. update /
  merge_insert / full rewrite, across narrow and wide rows,
- take and scan time + cold read IO as overlay layers, coverage
  fraction, fragmentation, and data type vary.

Shared dataset/overlay construction and measurement helpers live in
ci_benchmarks/overlays.py and build fixtures through the public
lance.LanceOperation.DataOverlay binding. The write benchmark verifies
each approach actually performs the update and splits persisted bytes
into data vs. metadata, since the rewrite mechanisms are delete-and-
reinsert (whole rows, all columns) while an overlay writes only the
changed column.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The overlay read benchmark's take/scan scaling sweep only covered a 4-byte
int32 value column, where the base read is trivial and per-layer merge cost
dominates. ML overlays typically target wide vector columns, so add
test_overlay_read_wide: the same overlays x coverage-pattern grid on a 3072-d
float32 embedding (12 KiB/row). Thread an embedding_dim through the overlay
helpers so the value width is configurable.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
… API

`DataOverlayFile` takes a single `offsets` param, not `shared_offsets`, and
the release-build opt-in env var is `LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES`
(not `LANCE_ENABLE_DATA_OVERLAY_FILES`). Both diverged from the benchmark
helper after PR #7540 landed the real Python bindings; commit()'s returned
handle skipped flag validation so this only surfaced on a fresh dataset open.
Self-review pass on the overlay benchmark suite:

- Remove the unused `base_path` parameter of `commit_overlay_layers` (the
  body derives its data dir from the dataset URI) and the dead `dir_size`
  helper; both had no effect and misled the reader.
- Rename `value_type` -> `_value_type` to match the module's private helpers.
- Document the `{u64::MAX - version}.manifest` naming in `manifest_size`.
- Guard the read and manifest benchmark fixtures: assert a covered cell reads
  back the overlaid value and that overlays grow the manifest, so a
  silently-broken overlay can't let the benchmark pass while measuring plain
  base reads.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@wjones127
wjones127 force-pushed the will/overlay-benchmarks branch from c3837ce to 8204329 Compare July 23, 2026 16:53
Comment thread python/python/ci_benchmarks/README.md Outdated
Comment thread python/python/ci_benchmarks/benchmarks/test_overlay_manifest.py Outdated
Comment thread python/python/ci_benchmarks/benchmarks/test_overlay_read.py Outdated
Comment thread python/python/ci_benchmarks/benchmarks/test_overlay_read.py Outdated
Comment thread python/python/ci_benchmarks/benchmarks/test_overlay_write.py Outdated
Comment thread python/python/ci_benchmarks/overlays.py
wjones127 and others added 4 commits July 23, 2026 10:18
…t read benchmark

- README: remove the per-benchmark descriptions; keep it to general suite advice.
- Manifest/read benchmarks: report via record_property only, no printing.
- Drop test_overlay_read_dtype: its int32 and embedding read costs are already
  covered by test_overlay_read_scaling (int32) and test_overlay_read_wide
  (embedding) across the full overlay-layer sweep.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The write-cost comparison (overlay vs. update / merge_insert / full rewrite) is
a useful one-off measurement but not something we need to keep re-running in CI
for regressions. Remove it, along with the helpers it exclusively used
(file_sizes, written_breakdown, proc_io_counters, and the make_base_dataset
payload column).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The datagen binding could only size generated batches by bytes. The underlying
lance-datagen builder already supports an exact row count (into_reader_rows), so
expose it: rand_batches gains a rows_per_batch option, mutually exclusive with
batch_size_bytes. This lets callers request an exact number of rows per batch
rather than approximating via a byte budget.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the hand-rolled base-dataset generator in make_base_dataset with
lance._datagen.rand_batches (row-count mode), matching the other ci_benchmarks
suites. One batch per fragment; num_rows must be a multiple of rows_per_file.
Overlay values keep their own generator so each layer stays distinct from the
base (the read-visibility guard depends on that).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@wjones127
wjones127 marked this pull request as ready for review July 23, 2026 20:04
@wjones127
wjones127 requested a review from westonpace July 23, 2026 20:04

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (4)
python/src/datagen.rs-32-53 (1)

32-53: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Include sizing inputs in both error paths.

The exceptions omit the requested row/byte sizes and batch count, making invalid configuration failures difficult to diagnose. Include parameter names and values in the conflicting-options and reader-construction errors.

As per coding guidelines, **/*.{rs,py,java} requires full error context, including “variable names, values, sizes, and types.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/src/datagen.rs` around lines 32 - 53, Update the validation error in
the rows_in_batch/bytes_in_batch conflict check and the reader-construction
error inside the match on rows_in_batch to include the relevant parameter names
and values: rows_in_batch, bytes_in_batch, and batch_count, including resolved
defaults where applicable. Preserve the existing exception types and generation
behavior while providing full sizing context in both messages.

Source: Coding guidelines

python/python/ci_benchmarks/benchmarks/test_overlay_read.py-36-38 (1)

36-38: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Parameterize the return type.

Use list[int] instead of bare list so the indices passed to ds.take retain their element type.

-def _take_indices(num_rows: int) -> list:
+def _take_indices(num_rows: int) -> list[int]:

As per coding guidelines, python/**/*.py requires parameterized type hints and prohibits bare generics.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/python/ci_benchmarks/benchmarks/test_overlay_read.py` around lines 36
- 38, Update the _take_indices function’s return annotation from the bare list
generic to list[int], preserving the existing sorted random-index behavior and
ensuring the indices passed to ds.take retain their element type.

Source: Coding guidelines

python/python/ci_benchmarks/overlays.py-70-74 (1)

70-74: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Validate dimensions before the modulo operation.

rows_per_file=0 raises ZeroDivisionError, and num_rows=0 proceeds until later helpers assume a first fragment exists. Reject non-positive dimensions with a contextual ValueError before calculating divisibility.

Proposed fix
 def make_base_dataset(
 ...
 ) -> lance.LanceDataset:
+    if (
+        not isinstance(num_rows, int)
+        or isinstance(num_rows, bool)
+        or not isinstance(rows_per_file, int)
+        or isinstance(rows_per_file, bool)
+        or num_rows <= 0
+        or rows_per_file <= 0
+    ):
+        raise ValueError(
+            "num_rows and rows_per_file must be positive integers; "
+            f"got num_rows={num_rows!r} ({type(num_rows).__name__}), "
+            f"rows_per_file={rows_per_file!r} ({type(rows_per_file).__name__})"
+        )
     if num_rows % rows_per_file:

As per coding guidelines, **/*.{rs,py,java} requires validating API-boundary inputs with descriptive errors.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/python/ci_benchmarks/overlays.py` around lines 70 - 74, Validate
num_rows and rows_per_file as positive values before the divisibility check in
the surrounding overlay setup. Raise a contextual ValueError for either
non-positive dimension, then retain the existing modulo-based multiple
validation for valid inputs.

Source: Coding guidelines

python/python/ci_benchmarks/overlays.py-95-100 (1)

95-100: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Reject invalid coverage and preserve the stride pattern.

max(1, ...) silently turns zero or negative coverage into one cell. Also, when count > num_rows / 2, step becomes 1, so "stride" degenerates into a leading contiguous run rather than distributing coverage across the fragment.

Proposed fix
+import math
+
 def coverage_offsets(num_rows: int, fraction: float, pattern: str) -> List[int]:
 ...
-    count = max(1, int(round(num_rows * fraction)))
+    if num_rows <= 0 or not math.isfinite(fraction) or not 0 < fraction <= 1:
+        raise ValueError(
+            f"expected num_rows > 0 and 0 < fraction <= 1; "
+            f"got num_rows={num_rows!r}, fraction={fraction!r}"
+        )
+    count = int(round(num_rows * fraction))
+    if count == 0:
+        raise ValueError(
+            f"fraction={fraction!r} selects zero rows for num_rows={num_rows!r}"
+        )
     if pattern == "contiguous":
         return list(range(count))
     if pattern == "stride":
-        step = max(1, num_rows // count)
-        return list(range(0, num_rows, step))[:count]
+        return [(index * num_rows) // count for index in range(count)]

As per coding guidelines, **/*.{rs,py,java} requires invalid inputs to be rejected and never silently adjusted.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/python/ci_benchmarks/overlays.py` around lines 95 - 100, Update the
coverage/count calculation in the overlay selection logic to reject zero or
negative coverage instead of clamping it to one, raising the module’s
appropriate invalid-input exception. Revise the “stride” branch so selected rows
remain distributed across the full range when count exceeds half of num_rows,
rather than degenerating into a leading contiguous run; preserve existing
contiguous behavior.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/src/datagen.rs`:
- Around line 19-25: Add a synchronized Rustdoc example to the public function
annotated with #[pyfunction], using its current schema, batch_count,
bytes_in_batch, and rows_in_batch signature, and link relevant API types such as
RecordBatchReader. Ensure the example reflects the documented mutually exclusive
batch-size options and default behavior without changing the implementation.
- Around line 32-55: Add focused coverage to test_rand_batches or nearby datagen
tests for rows_per_batch, asserting each generated batch has the requested row
count. Also add a case providing both batch_size_bytes and rows_per_batch, and
assert it raises ValueError matching the mutual-exclusion validation in the rand
batch-generation function.

---

Other comments:
In `@python/python/ci_benchmarks/benchmarks/test_overlay_read.py`:
- Around line 36-38: Update the _take_indices function’s return annotation from
the bare list generic to list[int], preserving the existing sorted random-index
behavior and ensuring the indices passed to ds.take retain their element type.

In `@python/python/ci_benchmarks/overlays.py`:
- Around line 70-74: Validate num_rows and rows_per_file as positive values
before the divisibility check in the surrounding overlay setup. Raise a
contextual ValueError for either non-positive dimension, then retain the
existing modulo-based multiple validation for valid inputs.
- Around line 95-100: Update the coverage/count calculation in the overlay
selection logic to reject zero or negative coverage instead of clamping it to
one, raising the module’s appropriate invalid-input exception. Revise the
“stride” branch so selected rows remain distributed across the full range when
count exceeds half of num_rows, rather than degenerating into a leading
contiguous run; preserve existing contiguous behavior.

In `@python/src/datagen.rs`:
- Around line 32-53: Update the validation error in the
rows_in_batch/bytes_in_batch conflict check and the reader-construction error
inside the match on rows_in_batch to include the relevant parameter names and
values: rows_in_batch, bytes_in_batch, and batch_count, including resolved
defaults where applicable. Preserve the existing exception types and generation
behavior while providing full sizing context in both messages.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 4fd0fc46-890e-4f84-95e5-3dd2715a977c

📥 Commits

Reviewing files that changed from the base of the PR and between d8030a3 and 1be2bec.

📒 Files selected for processing (5)
  • python/python/ci_benchmarks/benchmarks/test_overlay_manifest.py
  • python/python/ci_benchmarks/benchmarks/test_overlay_read.py
  • python/python/ci_benchmarks/overlays.py
  • python/python/lance/_datagen.py
  • python/src/datagen.rs

Comment thread python/src/datagen.rs
Comment on lines +19 to +25
/// Generate `batch_count` batches of random data for `schema`.
///
/// Batch size is set either by `rows_in_batch` (exact rows per batch) or
/// `bytes_in_batch` (approximate bytes per batch); the two are mutually
/// exclusive. When neither is given the byte-based default is used.
#[pyfunction]
#[pyo3(signature=(schema, batch_count=None, bytes_in_batch=None))]
#[pyo3(signature=(schema, batch_count=None, bytes_in_batch=None, rows_in_batch=None))]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Add a synchronized usage example and API links.

This public API documents behavior but provides neither an example nor links to relevant types such as RecordBatchReader.

As per coding guidelines, **/*.{rs,md,rst} requires: “Document all public APIs with examples and links to relevant structs and methods; keep examples synchronized with actual signatures.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/src/datagen.rs` around lines 19 - 25, Add a synchronized Rustdoc
example to the public function annotated with #[pyfunction], using its current
schema, batch_count, bytes_in_batch, and rows_in_batch signature, and link
relevant API types such as RecordBatchReader. Ensure the example reflects the
documented mutually exclusive batch-size options and default behavior without
changing the implementation.

Source: Coding guidelines

Comment thread python/src/datagen.rs
Comment on lines +32 to +55
if rows_in_batch.is_some() && bytes_in_batch.is_some() {
return Err(pyo3::exceptions::PyValueError::new_err(
"rows_in_batch and bytes_in_batch are mutually exclusive",
));
}
let builder = lance_datagen::rand(&schema.0);
let batch_count = BatchCount::from(batch_count.unwrap_or(DEFAULT_BATCH_COUNT));
let reader: Box<dyn RecordBatchReader> = match rows_in_batch {
Some(rows) => Box::new(builder.into_reader_rows(RowCount::from(rows), batch_count)),
None => Box::new(
builder
.into_reader_bytes(
ByteCount::from(bytes_in_batch.unwrap_or(DEFAULT_BATCH_SIZE_BYTES)),
batch_count,
lance_datagen::RoundingBehavior::RoundUp,
)
.map_err(|e| {
pyo3::exceptions::PyValueError::new_err(format!(
"Failed to generate batches: {}",
e
))
})?,
),
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
fd -t f -i 'test*datagen*' python
rg -n -C2 'rand_batches|rows_per_batch|rows_in_batch|bytes_in_batch' python

Repository: lance-format/lance

Length of output: 10858


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "## python/src/datagen.rs"
sed -n '1,80p' python/src/datagen.rs

echo
echo "## python/python/tests/test_datagen.py"
cat -n python/python/tests/test_datagen.py

echo
echo "## python/python/lance/_datagen.py"
cat -n python/python/lance/_datagen.py

echo
echo "## benchmark row-mode call"
sed -n '65,85p' python/python/ci_benchmarks/overlays.py
sed -n '1,35p' python/python/benchmarks/test_bulk_write.py

Repository: lance-format/lance

Length of output: 7004


Add focused tests for the row-sizing contract.

test_rand_batches() only exercises byte sizing and batch count assertions; it does not test rows_per_batch row sizes or the mutually exclusive sizing validation. Add datagen tests that assert row mode keeps the expected rows per batch and raises ValueError when both batch_size_bytes and rows_per_batch are provided.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/src/datagen.rs` around lines 32 - 55, Add focused coverage to
test_rand_batches or nearby datagen tests for rows_per_batch, asserting each
generated batch has the requested row count. Also add a case providing both
batch_size_bytes and rows_per_batch, and assert it raises ValueError matching
the mutual-exclusion validation in the rand batch-generation function.

Source: Coding guidelines

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings chore

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant