Run DataFusion's own functions in einfold's library, not through the FFI - #132

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01M96xSez6PZwTmfaGh2eJgS

🪭 Claude (einfold)

This sets einfold for Python's version to `0.2.0rc1`. Once it merges and
`python-v0.2.0rc1` is tagged on it, the release candidate goes to PyPI.
Users then get it with `pip install --upgrade --pre
"einfold[datafusion]"`, and the plain install keeps 0.1.0. It's for
xarray-sql's acceptance testing. xarray-sql is an XQL Systems project
that queries Xarray datasets with SQL.

It holds the first part of M3, from [spike
S31](docs/spikes/s31-ddx-gradient-steps/README.md):
- #129 reads ddx's fill as a domain's default. ddx is an XQL Systems
project for automatic differentiation of SQL queries.
- #131 finds products whose factors read three or more relations.
- #132 runs DataFusion's own functions in einfold's library rather than
through the FFI.

On xarray-sql's demos, from a local build of these changes on an
i7-8700, in medians of 3 runs:

| | 0.1.0 | 0.2.0rc1 | DataFusion | DuckDB |
|---|---:|---:|---:|---:|
| attention gradient, 1,024 tokens, window | 2.30 s | 0.90 s | 4.03 s |
5.42 s |
| attention gradient, 1,024 tokens, `GROUP BY` | 1.89 s | 0.59 s | 4.67
s | 7.60 s |
| GPT training step, nanoGPT's CPU size | 4.03 s | 2.61 s | 12.05 s | |

That's 4.5× and 7.9× DataFusion on attention's gradient, and 4.6× on the
GPT step, past M3's gate of 4× (design v20).

Two of the gate's items aren't met yet, and stay open for M3:
- A flat chain of three matrix products is faster than both hosts at 64
and 96, but ties DataFusion at 32 × 32. Choosing the contraction order
would fix that.
- einfold changes none of TPC-H's 22 plans, but from Python each query
pays the FFI's fixed cost: 1.06× DataFusion's time at scale factor 1.

## Decisions to review

- **A release candidate now,** before M3 is done, so that xarray-sql
tests the gradient work while the rest of M3 continues: the contraction
order, the FFI's per-query cost (#81, #58), and the fused backward pass,
which waits on xqlsystems/ddx#243.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01M96xSez6PZwTmfaGh2eJgS

Co-authored-by: Claude Opus 5.5 <[email protected]>

🪤 Claude (fuzz adversary)

The simulation, einfold's random differential test against plain
DataFusion, now generates the three shapes that #129 and #131 taught
detection to find. Each run before this one reached none of them.
Nothing changes for users. These are the generators for the soak of
0.2.0rc1.

## Decisions to review

- **Fills with other constants**
([`fill`](https://github.com/xqlsystems/einfold/blob/8ec90e5/crates/einfold-datafusion/tests/simulation/queries.rs#L430)).
A fold from a domain, or joined back to one, now reads its sum through
`COALESCE`, ddx's `CASE`, and the `IS NOT NULL` form DataFusion writes
for `COALESCE`. ddx is an XQL Systems project for automatic
differentiation of SQL queries. The constant is `0.0`, `-0.0`, `1.0`,
`-2.5`, NaN or `-inf`. Some fills sit inside an expression or inside
another fill. The look-alikes are:
  - the bare sum;
  - the sum filled with a column;
  - two fills of one sum.

The two fills test the sum with `IS NULL` and with `IS NOT NULL`. If
both tested it the same way, DataFusion would compute the shared test in
a projection below, and no fill would reach detection. I first wrote
them alike, and a planted bug went unseen.
- **Finite values in some cases**
([`data.rs`](https://github.com/xqlsystems/einfold/blob/8ec90e5/crates/einfold-datafusion/tests/simulation/data.rs#L203)).
With values that are all finite or NULL, in 30% of cases, the dense
algorithm can take operands with missing rows, which is #129's purpose.
A NaN or an infinity anywhere kept it from doing so before. The tally
counts these cases, and a run of 10,000 cases or more asserts that one
happened (4 in 2,000 here).
- **Products over three relations**
([`three_relations`](https://github.com/xqlsystems/einfold/blob/8ec90e5/crates/einfold-datafusion/tests/simulation/queries.rs#L598)).
The third relation is `d` as `m`. It joins `l`, `r` or both, and the
factor on its side reads it, or neither factor does. The join order is
drawn, or the joins are written as a comma join. Factors use `*`, `-`,
`%`, or (with inexact values) `/`, and groups may be `m`'s keys. The
look-alikes are both factors reading `m`, and `m` joining both sides
with neither factor reading it. `d` and the operands get NULL keys more
often
([here](https://github.com/xqlsystems/einfold/blob/8ec90e5/crates/einfold-datafusion/tests/simulation.rs#L271)),
since a NULL key in the regrouped join must join nothing.
- **Float `/` and `%` in factors**
([`factor`](https://github.com/xqlsystems/einfold/blob/8ec90e5/crates/einfold-datafusion/tests/simulation/queries.rs#L114)).
With exact values these are `v / v`, `1.5 % v` and `v / 2.0`, all exact.
With inexact values it is `1.0 / v`, whose magnitude the bound's twin
tables still give.
- **Integer `/` and `%` that fail only on dropped rows**
([here](https://github.com/xqlsystems/einfold/blob/8ec90e5/crates/einfold-datafusion/tests/simulation/queries.rs#L131)).
The divisor is 0 exactly where the join key is NULL. The existing
fallible factor casts `v` to `BIGINT`, and that cast already keeps the
factor above the join, so no factor before this one tested whether
integer division moves.

## How it is tested

The invariants are unchanged: the same rows as the host, bit for bit
with exact values; the same errors; and the rest listed at the top of
`simulation.rs`. To check that the new generators reach the new code, I
planted bugs in it. Each ran for 2,000 cases from seed 777000.

| Planted bug | Caught |
|---|---|
| `fill_default` accepts two fills of different constants | 6 `[rows]`
and `[kernels]` failures |
| Integer `%` counts as movable | 10 `[error]` failures |
| The regrouped join makes NULL equal NULL | 3 in 500 cases forced to
three relations with einfold's kernels; about 1 in 10,000 in the normal
mix |
| `join_back.rs` keeps a `-0.0` default | not seen |

Bug 4 can't change a result. With a `-0.0` default, the dense algorithm
doesn't take missing rows, so the node runs the host's plan
(`host_plans=1`). The filter is a second guard that no output shows.

On main, 2,000 cases from seed 777000 pass.

## Not in this pull request

- A soak of 0.2.0rc1 with these generators, dispatched in CI once this
is up. Findings will go to issues.
- An invariant for #132: that relinking DataFusion's functions in the
Python bridge changes no function's answer, under session settings such
as a time zone. It belongs in Python tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01SCMAk87qC9VKCk22P7HDH1

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>

…141)

🪤 Claude (fuzz adversary)

A new Python test checks that installing einfold in a datafusion-python
session doesn't change what DataFusion's built-in functions return.
datafusion-python is DataFusion's Python bindings. #132 made einfold run
its own copy of each built-in instead of the session's. The two copies
come from different DataFusion releases (55.2 in einfold, 55.0 in
datafusion-python), and einfold's copy runs where the session's
configuration may not reach it. The test found one function that does
differ: #136. Nothing changes for users.

## Decisions to review

- **Calls generated from the session's own catalog**
([`_calls`](https://github.com/xqlsystems/einfold/blob/5dabbbf/python/tests/test_builtins.py#L130)).
For every deterministic scalar, aggregate and window function in
`information_schema.routines`, it makes one call per signature in
`information_schema.parameters` that sample columns can fill. There are
columns of 31 types, and text of four kinds, so each function meets
input it accepts and input it rejects. Functions with variadic
signatures, which the schema doesn't list, get a menu of 13 argument
lists. That's how it reaches `to_unixtime`. The alternative was a
hand-picked list, which would miss the next function to differ.
- **Errors compare too**
([`_answer`](https://github.com/xqlsystems/einfold/blob/5dabbbf/python/tests/test_builtins.py#L161)),
apart from the `FFI error:` prefix that an error crossing the FFI
(foreign function interface) gains.
- **A floor on what answers**
([here](https://github.com/xqlsystems/einfold/blob/5dabbbf/python/tests/test_builtins.py#L210)).
At least 120 functions must answer, so that a change to the sample
columns can't make the test pass by calling nothing. 127 do.
- **#136 is pinned, not skipped.** `_KNOWN` leaves out `to_unixtime`
under `+05:00`. `test_to_unixtime_ignores_the_sessions_time_zone` is a
strict expected failure, so it fails once #136 is fixed, which is the
signal to remove both. A Rust pin test for the upstream gap, #137, can't
be written: a session that never leaves its library never becomes
foreign, and `FFI_SessionRef` is private to datafusion-ffi.

## How it is tested

The invariant: for each call, a session with einfold gives the same
column types and values (every NaN alike), or the same error, as one
without, with the session's default time zone and with `+05:00`. That's
about 1,190 calls a time zone, in 14 s for both.

To check that it reaches einfold's copies, I logged the bridge's
relinking temporarily: it replaced 84 scalar functions and never fell
back to the session's copies. A planted bug, `sqrt` relinked to `cbrt`
(same types, other values), fails the test. Without the `_KNOWN` entry,
the `+05:00` case fails on `to_unixtime` only. The other 24 Python tests
pass.

## Not in this pull request

- The fix for #136, which is the maintainer's to choose, and the
upstream draft #137.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01SCMAk87qC9VKCk22P7HDH1

Co-authored-by: Claude Opus 5.5 <[email protected]>

🪭 Claude (einfold)

Sets einfold for Python's version to `0.2.0`. Once this merges, the tag
`python-v0.2.0` publishes it to PyPI, and `pip install --upgrade
"einfold[datafusion]"` gets it without `--pre`. It's the same code as
0.2.0rc2, which xarray-sql acceptance-tested. xarray-sql is an XQL
Systems project that queries Xarray datasets with SQL. The maintainer
approved the release.

Since 0.1.0:
- #129, #131 and #143 add folds with a fill as the domain's default,
regrouped operands, and chains of three products in the cheaper order,
under a float guard. A chain of matrix products is 3× DataFusion at `n =
32`, 22× at 64, and 450× at 128.
- #132 and #140 run DataFusion's default functions in einfold's library,
with the session's configuration. `to_unixtime` reads text in the
session's time zone (#136).
- #141 checks that einfold changes no built-in function's answer from
Python. #134 and #142 extend the simulation to the new shapes.

Known issues, for later releases:
- #145 and #146: a `DataFrame` that reads an in-memory table fails once
its `SessionContext` is dropped. The cause is in datafusion-python; the
workaround is to keep the context alive.
- The Python bridge's fixed cost per query (#81, #58): a chain whose
last factor is a vector is 0.8× DataFusion at `n = 32`, and TPC-H is
1.06× DataFusion's time.

## Decisions to review

None.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01M96xSez6PZwTmfaGh2eJgS

Co-authored-by: Claude Opus 5.5 <[email protected]>