Repository navigation
Conversation
experimental/mlx/ holds datafusion-accelerated-mlx, the contributor's MLX path, until the milestones replace it (issue #41). It's a project of its own, outside the core and its workspace, so that deleting the directory retires it. Its README says what each part is and the milestone that retires it, and credits Ray Cromwell. CI gains a job that runs its tests on x86-64 and ARM Linux, which need no Apple hardware, and the REUSE job's checkout is now pinned by SHA. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Retain coordinate axes, row-presence and NULL masks across supported numerical SQL stages. Expose an explicit host-invoked prototype with bounded intermediates and conservative refusal, and compare randomized joins and masked reductions with DataFusion. Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@452fda8 (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Intern ordered tensor expressions and contractions within one invocation, preserve typed scalar identities and integer-check provenance, and expose optional exact count proofs. Document matched training measurements and cold-start tradeoffs. Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@23fc63d (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Normalize typed Boolean expressions and attach an optional positive-mask fact to eligible exact counts across axis renames and gathers. Keep dynamic reduced-axis masks and overflow cases unchanged. Select Boolean normalization by default and retain presence proofs as an opt-in after measuring their allocation cost. Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@ed0a982 (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Expose private backend-neutral value, mask and coordinate metadata with an optional reduction lowering hook. Preserve the original aggregate closures and SQL row/NULL outputs when a host declines. Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@2249503 (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Reuse validated integer domains for dependent GPU grouping and prove row/NULL coverage from complete independent UNION branches. Retain numerical accumulation and reference-mode ablations. Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@805f50c (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Recognize complete static positional maps and keep the existing gather and scatter paths for other maps. Measure the opt-in layout proof independently of GPU gradient accumulation. Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@b10c8ff (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Describe the pre-scatter value and key as structural dependencies so a host can coalesce duplicate keys within a UNION branch while retaining the relational masks. Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@66b8f45 (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@7691744 (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@bdde066 (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@303d65c (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Keep row and NULL existence predicates symbolic when masks depend only on group axes, using bounded constant counts for full domains. The opt-in policy reduces measured full-model training from 602.30 to 483.11 ms with bitwise-identical final state. Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@3d85cf7 (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Retain numerical aggregates and count arithmetic while proving existence on each separable mask component. Differential SQL tests cover joined nullable and empty domains; the initial full-model comparison retains bitwise-identical state. Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@f3ed33a (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
Canonicalize pure pointwise operations, reductions, extrema, proved indexed sums and ordered union values while retaining each consumer’s SQL masks and coordinates. Indexed reduction reuse lowers measured full training from about 418 to 233–238 ms with bitwise-identical state. Co-Authored-By: GPT-6 <[email protected]> Replayed from cromwellian/einfold@3108f7a (issue #41), with its paths and its imports of einfold.dense_program rewritten for this package. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
The replayed files carried "SPDX-FileCopyrightText: 2026 OpenAI". They now carry the repository's copyright line, as issue #41 asks, with Ray credited in the package's README and as the author of each replayed commit. The package exports DenseProgram, its only public name. Co-Authored-By: Ray Cromwell <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01L8gkxyeFDs3SbL6S7w7UgX
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⚡ Claude (datafusion-accelerated)
This PR lands the contributor's MLX path here as an experimental package,
datafusion-accelerated-mlx, underexperimental/mlx/. The plan is #41, and the layout is the sketch on #41, which Alex agreed to. Like M0's #40, it's one PR for the whole plan: a sketch before each step, commits added as they land, and a merge commit at the end.Background. Ray Cromwell (@cromwellian), an XQL author, trained a 123.6M-parameter GPT-2 reproduction from SQL on an Apple M5 Ultra, at 238.95 ms per warm step against 185.40 ms for PyTorch (supplement S1.1). His code is spread across xarray-sql PRs, einfold PRs and a fork of einfold. This package collects it so that it runs from released code, with no forks, until the milestones replace it part by part.
Steps
cac7cab)DenseProgramand the fork's proofs, with their tests: Ray's 13 commits replayed, then the headers (dfc1c45to33b7daa)a1d237a(a sketch comes first, listing the hooks xarray-sql needs)Step 2
DensePrograminterprets datafusion-python's logical operators on compact arrays, which are NumPy arrays indexed by the SQL's coordinate columns. It imports only NumPy, PyArrow and the standard library, so it needs no einfold here.452fda8) and his 12 commits on top of it, to3108f7a, are replayed with his authorship, his messages and their dates. Each commit's message names the fork commit it came from. Only two kinds of change were made in replay: the files' paths, and the tests'from einfold.dense_programimports.dense_program.pyis byte-for-byte the fork's at3108f7a, apart from the header that33b7daachanges. The tests differ only in their import lines. Every replayed commit passes its own tests, from 9 tests at the first commit to 303 at the last.SPDX-FileCopyrightText: 2026 OpenAI.33b7daagives them the repository's line, as Land the contributor's MLX path as an experimental package #41 asks. It credits Ray with aCo-Authored-Bytrailer, and the README credits him too.DenseProgramfrom the fork's file and from the package both match DataFusion. Their medians over 100 alternating runs were 18.92 ms and 18.95 ms, on an Intel i7-8700 with Linux, CPython 3.12.14, datafusion 55.0.0 and NumPy 2.5.4. This is posted on xarray-sql#285.🤖 Generated with Claude Code