Run folds with batch dimensions densely, and small folds on one thread - #86

🪭 Claude (einfold): This is #91's commit, moved onto `main` so that it
can merge ahead of the stack it was part of. #91 is closed in its favor.

Answers #79's second ask, and opens its third, as design v18.

**The goal.** For models trained in SQL, such as the GPT in the ML demos
of xarray-sql, an XQL Systems project for querying xarray datasets with
SQL (its `benchmarks/ml/02_nanogpt.py`), memory limits model size as
much as time does. Principle 9, "never more memory than the host", says
how einfold's operators keep to it. They reserve what they read, and
what their kernels allocate, from the host's memory pool, before
allocating it. Where the pool can't give it, they run the host's own
plan, whose operators can spill to disk. Every benchmark measures peak
memory next to time.

**What changes in the design.**

- **Section 4** adds principle 9.
- **Section 10.2** gains a bullet on memory: what `EinFoldExec` does
(#89, #90), and what comes next. The positional fold should stream its
larger operand in row blocks, so that its peak stays below the host's
even without a pool to enforce it. That needs each dimension's extent
before reading, and #80's dictionaries answer it: the dictionary's
length, when it is the whole axis. Exact statistics or a first pass over
the keys also give it.
- **Section 15** reports peak memory next to time, which
`python/benchmarks/memory.py` (#85) does for the Python extension.
- **Section 14** opens #79's third ask, dense chains of folds: handing
the dense form from one einfold operator to the next, so the joined
intermediates never exist as rows. It lists what is open: the form
handed on, how a host operator between two folds reads it, and when a
chain is worth keeping dense.
- **`lessons.md`** records what running einfold under the GPT taught
(#77, #79, #81):
  - profile through the real bridge;
  - report statistics, and cache them;
  - plan a fallback only if it runs;
  - batch dimensions are common.

**Where the stack stands on #79**, at its size (128 channels, context
64, batch 12, one layer), through datafusion-python on an Intel i7-8700:

| | DataFusion alone | einfold, before #86 | einfold, with #86–#90 |
|---|---:|---:|---:|
| the loss | 108 s, 8.9 GB peak | 42 s, 11.8 GB | 13.6 s, 6.9 GB |
| the gradients | 36.5 s, 9.8 GB | | 4.2 s, 1.8 GB |

The peak is above the loaded tables. So the GPT already keeps to the
principle at this size. Streaming in row blocks would make that hold by
construction, not only under a pool.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01M96xSez6PZwTmfaGh2eJgS



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Documentation**
* Updated the design guidance to cover host-managed memory use, fallback
behavior, peak-memory benchmarks, and current limitations around
whole-input processing.
* Added lessons on measuring end-to-end performance, reporting row-count
statistics, delaying fallback planning until execution, and supporting
shared batch dimensions.
* Clarified that success benchmarks compare speed while keeping memory
use within the host’s plan.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <[email protected]>

🪭 Claude (einfold)

Part of #81: planning the GPT of xarray-sql's ML demos (xarray-sql's
`benchmarks/ml/02_nanogpt.py`, a GPT written and trained in SQL) took
longer with einfold on.

**Why.** Some fold nodes keep the host's own plan, to run where the
dense algorithm declines: masked folds, folds joined back to their
domain, fold pipelines, and every fold under
`DetectFolds::with_host_fallback`. `EinFoldPlanner` planned that
fallback along with the operator. A fallback is the fold's whole
subtree, so this planned the operator's inputs a second time, and within
them every inner fold's fallback again. In a deep plan that grows with
the square of the depth. The GPT's loss has 80 such folds, and with #86
none of them falls back, yet planning their fallbacks took 0.5 s of the
loss's 1.45 s of physical planning.

**What changed.** `EinFoldExec` keeps its fallback as a logical plan,
and plans it only when it runs. It plans it for the task that runs it:
with that task's session configuration and runtime, and DataFusion's
default physical optimizer rules. The rules are the one difference from
before: a session's own physical optimizer rules don't reach a fallback
now. einfold's Python extension already plans with the default rules,
since the session's rules see only proxies through the FFI.

`EinFoldExec::with_fallback` now takes a `LogicalPlan`, and `fallback()`
returns one. A fallback planned when it runs starts with fresh state, so
the operator no longer resets it, and its `reset_state` is DataFusion's
default again.

**Measured** on the GPT's loss at the demo's size, through
datafusion-python, on an Intel i7-8700. Each wheel ran twice,
alternating, while another job compiled, at a load average of 3–5:

| | #86 | this pull request |
|---|---:|---:|
| einfold's physical planning, inside the extension | 1.45 s | 0.9 s |
| the loss's planning, seen from Python (einfold off: 2.5 s) | 3.4–3.5 s
| 2.9 s |
| the loss's execution, after the first run (einfold off: 0.35–0.40 s) |
0.47–0.53 s | 0.57–0.58 s |

The executions are within the noise of the machine. Most of the planning
gap that's left is DataFusion's FFI: the session sends the logical plan,
about 10,000 lines of it, to the extension as protobuf (#58).

**Tests.** The existing tests of fallbacks pass unchanged:
- recursive queries whose fallback runs on every iteration;
- plans that are reset and run again;
- unique-domain checks that decline;
- the property test of detection against the host, with fallbacks on.

The two tests that checked a fallback holds no `EinFoldExec` now check
that its logical plan holds no fold node. The Python extension's tests
pass too.

## After CodeRabbit's review

- **A deferred fallback is planned with the planning session's
`ExecutionProps`,** so variables (`@name`) resolve there as they did in
the query. `fallbacks_plan_with_the_sessions_variables` failed before.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01M96xSez6PZwTmfaGh2eJgS

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>