Async hybrid query with .offset() repeats and drops rows across pages #4027

LanceDB version

main (0.38.0-beta.3), reproduced against the published 0.37.1 wheel as well

What happened?

Paginating an async hybrid query with .offset() returns rows that were already on the previous page, and silently drops rows that belong on the current one.

full   = await table.query().nearest_to(vec).nearest_to_text("dog").limit(4).with_row_id().to_arrow()
page1  = await table.query().nearest_to(vec).nearest_to_text("dog").limit(2).with_row_id().to_arrow()
page2  = await table.query().nearest_to(vec).nearest_to_text("dog").offset(2).limit(2).with_row_id().to_arrow()

full["_rowid"].to_pylist()    # [3, 0, 2, 1]
page1["_rowid"].to_pylist()   # [0, 3]
page2["_rowid"].to_pylist()   # [3, 1]   <- rowid 3 repeats from page 1, rowid 2 never appears

Expected page2 to equal full[2:], i.e. [2, 1].

.offset() on a hybrid query pushes the offset down into both sub-queries (HybridQuery::offset → inner_vec + inner_fts), so each of the vector and FTS sides independently skips its own first N rows before the two are fused and reranked. AsyncHybridQuery.to_batches then calls _combine_hybrid_results(..., limit=self._inner.get_limit()) without passing offset=, so the combined table is sliced from position 0 and the sub-query limits are never raised to limit + offset. The window ends up taken from the wrong place on both sides.

The sync path had exactly this bug and it was fixed in #3769. The async twin was not updated — and this was known at the time: the author of #3765 explicitly scoped it out, noting "the async path ... applies offset independently to each sub-query before combining/reranking ... That's a separate, Rust-level concern and is out of scope for this issue". As far as I can tell no follow-up issue was ever filed, so I'm filing one.

Only the async hybrid path is affected. Sync hybrid is correct since #3769, and plain vector/FTS queries are unaffected.

Are there known steps to reproduce?

Yes — build any table with both a vector column and an FTS index, then compare limit(4) against limit(2) and offset(2).limit(2) as above. It reproduces deterministically; a regression test covering it is in the linked PR.