Repository navigation
Incorrect physical_rows recorded when migrating from old versions of tables with deletions #1531
Description
v0.8.0: We started recording the number of physical rows, but implementation used count_rows() to compute this.
v0.7.5 is the version right before that.
How to reproduce
Install pylance 0.7.5
python -m venv .venv
source .venv/bin/activate
pip install pylance==0.7.5
pythonWrite a table, and delete some rows
import lance
import pyarrow as pa
tab = pa.table({'x': range(100)})
dataset = lance.write_dataset(tab, 'test')
dataset.delete("x >= 10 and x < 20")
dataset.count_rows()90
Now, install pylance 0.8.0:
pip install pylance==0.8.0
pythonCount rows at first is correct. However, once we write to the table, it becomes incorrect:
import lance
import pyarrow as pa
dataset = lance.dataset('test')
dataset.count_rows() # 90
tab = pa.table({'x': range(5)})
dataset = lance.write_dataset(tab, 'test', mode='append')
dataset.count_rows() # 85
dataset.to_table().num_rows # 95
Activity
wjones127 commentedon Nov 7, 2023
If I rerun the repro using the latest version of Lance instead of
v0.8.0we will also experience data loss:This seems to be because we pass the dataset fragments and not the new to-be-committed fragments here:
https://github.com/lancedb/lance/blob/dc83011078a4e966d5eae908af3ab982f4638e48/rust/lance/src/io/commit.rs#L170
fix: add versioning and bypass broken row counts (#1534)
perf(dataset): sum manifest row counts in `Dataset::count_rows(None)` (…