Repository navigation
head, slice and search_first on a declared-sorted Parquet file or directory skip the first matching rows #325
Description
Part of the launch EPIC #321 (gate G1, P0 for tables with a declared order). This is the "head across files" case that #281 flagged under "Needs your attention" without an issue, plus a single-file variant found in the launch audit.
Problem
On a Parquet source with a declared order, head, slice and search_first can return rows that are not the first matching rows. The declaration is correct; the rows are wrong. It happens at one partition too, so it is not the partitioning order loss of #311.
import os, pyarrow as pa, pyarrow.parquet as pq
from ltseq import LTSeq
pq.write_table(pa.table({"x": list(range(100_000))}), "sorted.parquet", row_group_size=4096)
t = LTSeq.read_parquet("sorted.parquet").assume_sorted("x")
t.filter(lambda r: r.x >= 10).head(3) # x = [4096, 4097, 4098], expected [10, 11, 12]
t.search_first(lambda r: r.x >= 10) # x = [4096], expected [10]
os.makedirs("d", exist_ok=True)
for i in range(4):
pq.write_table(pa.table({"x": list(range(i * 25_000, (i + 1) * 25_000))}), f"d/part-{i:05d}.parquet")
LTSeq.read_parquet("d").assume_sorted("x").head(3) # x = [75000, 75001, 75002], expected [0, 1, 2]Verified on main (ffd3247) with target_partitions=1 and at the default layout. search_first is advertised as an ordered search, and "pre-sorted Parquet" is the path assume_sorted exists for.
Cause (from reading DataFusion 55's source; no patched build was tested)
When a limit has no required ordering, DataFusion 55's Parquet opener prunes by limit (prune_by_limit, datafusion-datasource-parquet 55, opener/mod.rs around line 1356): it keeps only row groups whose statistics show every row matches the predicate, which skips the partially matching first row group. On a directory, the plan scans only part-00003 with limit=5, because the listing table takes files for a limit in listing order.
Putting .sort(key) before the limit gives the limit a required ordering, which turns this pruning off; that workaround returned the correct rows in every run. The root-order work in #311 (a sort with fetch beneath every limit on a table with a declared order) would likely close this too. That is an inference to verify, not a reason to close this issue early.
Acceptance criteria
- On a correctly sorted Parquet file with three or more row groups, and on a directory of three or more files, under
assume_sorted, with a predicate that partially matches the first row group:filter(...).head(n),slice(0, n),search_first, directoryhead(n)andto_arrow()return the first rows in file order, at everyengine_layout(partitions 1, 2 and 2×CPU; batch sizes 1, 7 and default). - Without a declared order, the reference page (Reference, support matrix and known-limitations pages for the installed API #327) states which rows
head(n)returns on a multi-file source.
Activity
iceboundrock commentedon Oct 22, 2026
Checked on PR #337 (
7a7b12d), which requires the declared order at the root of every read and putshead/slice/search_first's limit above it. With the body's data, read through my own script, the first rows in file order come back attarget_partitions1, 2 and the default (8 CPUs):filter(x >= 10).head(3)[10, 11, 12]search_first(x >= 10)[10]slice(0, 3)[0, 1, 2]head(3)[0, 1, 2]That fits the body's reading: the limit now has a required ordering, as with the
.sort(key)workaround. I did not run the acceptance'sengine_layoutmatrix (batch sizes 1 and 7, 2×CPU partitions) orto_arrow()on the directory, so this stays open for its own tests.iceboundrock commentedon Oct 22, 2026
Status after the final audit of PR #337 (head
2a7971f, not merged; audit report). The acceptance criteria hold in an independent check at every engine layout, but no test is committed, so this issue stays open.main(2075c03)assume_sorted("x")filter(x >= 10).head(3),slice(0, 3),search_first(x >= 10),head(3),to_arrow()filter.head,search_first)This confirms the body's reading: the limit now has a required ordering, which turns off Parquet's limit pruning.
Remaining:
engine_layout, passing five repetitions. It could live beside [High] 游标与写路径的顺序保证:多分区 execute_stream 批次乱序 #148'sassume_sortedtests, which use the same inputs.head(n)returns on a multi-file source with no declared order, is documentation for Reference, support matrix and known-limitations pages for the installed API #327.iceboundrock commentedon Oct 22, 2026
PR #337 merged on 2026-10-22 as
03d37c6.head,sliceandsearch_firstnow put their limit above the declared order. On declared-sorted Parquet they therefore return the first rows in file order. The pre-merge audit found 0 failures in 90 runs over a file and a directory (status).This issue stays open for a committed test of those criteria under
engine_layout, passing five repetitions. It can share #148'sassume_sortedinputs. Which rowshead(n)returns on a multi-file source with no declared order is documentation for #327.refactor: keep KeepFileOrder private and say why #325's inputs are th…
iceboundrock commentedon Oct 23, 2026
Status: #337 did not fully fix this; PR #343 fixes the rest and adds the tests. The status comments of 2026-10-22 rest on inputs that one partition reads: 10,000 rows, under the 1 MiB from which DataFusion splits a file. With two or more partitions the bug still reproduces on
main(6fe3bd1). With the body's shape (row groups of 4,096 rows) in a 2.1 MiB file,filter(date >= 10).head(3)returns[4096, 4097, 4098]andsearch_first(date >= 10)returns[4096]at 2 and 16 partitions, so at the default layout too. A directory of files with several row groups each fails the same way. The body's own 100,000-row example (0.53 MiB) passes only because DataFusion does not split it.Cause. On one partition the root sort that #337 adds becomes a limit that keeps the scan's
preserve_order. On several partitions it becomes aSortPreservingMergeExecwith a fetch, and DataFusion 55'sLimitPushdownpushes that fetch into the scan withpreserve_ordercleared, which lets the Parquet opener prune row groups by the limit. apache/datafusion#26434 fixes this in DataFusion 56; there is no 55.x backport. PR #343 sets the flag again with a physical optimizer rule until the upgrade, and adds criterion 1's tests at every engine layout and at 100,000 rows. They pass in five consecutive runs, and 9 of the 22 fail onmain.Criterion 2 (which rows
head(n)returns without a declared order) is posted on #327, per the comments above: #327 (comment). The removal of the rule at the DataFusion 56 upgrade is recorded on #273: #273 (comment).