Skip to content

Commit b245813

Browse files
andygroveimtherealnaska
authored andcommitted
docs: add an agent skill for auditing datafusion-spark expressions (apache#23893)
## Which issue does this PR close? - Closes #. No issue tracks this. It is tooling, proposed for discussion. Related: https://github.com/apache/datafusion/issues/15914 (the `datafusion-spark` epic) and https://github.com/apache/datafusion/issues/23887 (version-specific SLT expectations, which this skill's cross-version findings link to). ## Rationale for this change The `datafusion-spark` crate reimplements Spark's built-in expressions. Correctness against Spark is the entire point of the crate, but there is no routine process for checking an implementation against the Spark source, and no record of which functions have been checked. The testing situation makes this harder than it looks. Comet, the sibling project, tests by differential execution: it runs each query in both engines and compares, so expected values never need to be written down. DataFusion's `.slt` files hard-code every expected result, with no Spark in the loop. Two consequences follow: 1. Someone has to determine what Spark actually returns before writing the expectation. Reading `nullSafeEval` and predicting the output is how wrong golden values get written, and once written they look like verified behavior. 2. A hard-coded expectation encodes exactly one Spark version, so every file has to be written with one version in mind. Many files under `test_files/spark/` were machine-generated from Sail's gold data and their values were never checked against a running Spark. An existing passing expectation is a claim, not evidence. ## What changes are included in this PR? One file: `.ai/skills/audit-datafusion-spark-expression/SKILL.md`, following the `.ai/skills/` convention already described in the repository `CLAUDE.md`. No production code changes. The skill audits one function per invocation, in eight steps: read the Spark source across 3.5.8, 4.0.4, 4.1.3 and 4.2.0 to detect cross-version divergence; harvest Spark's own unit tests and `sql-tests/results/*.sql.out` golden files; review the Rust implementation's signature and coercion, return nullability, null propagation, overflow, type dispatch and ANSI handling; cross-check Comet and Sail; review existing SLT coverage; build a gap matrix; establish ground truth; then apply findings. Three decisions are worth calling out for review: **Spark 4.2.0 is the declared baseline.** Older versions are read only to detect that behavior changed, never to set an expectation. **Expected values come from a real Spark.** The skill sets up a `pyspark==4.2.0` venv and runs every candidate query through it under both ANSI modes, including expectations that already exist in the file. If PySpark cannot be installed the audit continues from source-derived values, but every such expectation is marked unverified rather than presented as observed. **`--complete` is forbidden for files under `test_files/spark/`.** That mode fills expectations with whatever DataFusion currently returns, which on an audit would silently cement the bug being audited as the golden answer. The skill also establishes a convention for divergences it cannot fix: a commented-out SLT query carrying the correct Spark result and a link to the tracking issue, where uncommenting it is the contract for verifying a fix. That convention is new to this tree and is the piece most worth arguing about. ## Are these changes tested? The skill was run end to end against `next_day`, which is how it was debugged. That run produced apache#23892 and issues apache#23889, #23890 and apache#23891. Running it surfaced eleven defects in the skill itself, which are fixed here. Two are worth citing as evidence that the exercise was not circular: - The instruction to `unset SPARK_HOME` sat in a different code block from the command that needed it, so it never took effect and the run reproduced the exact `TypeError: 'JavaPackage' object is not callable` failure the text was written to prevent. - The ground-truth harness reported a Python `ValueError` as though it were a Spark error, which would have written a fabricated divergence into the `.slt` as observed fact. Review of the resulting `next_day` work then found a genuine SQL-reachable panic in the audited function that the audit had missed, and the gap that hid it, which is now a rule in the skill: cross the argument shapes with the NULL cases and with both ANSI modes rather than checking each axis alone. The skill carries no automated test. Its verification is that following it produces a correct audit, which is what apache#23892 demonstrates. It has since been run a second time, against `pmod`, which found three real bugs now fixed in apache#23898 and four more divergences filed as apache#23894, apache#23895, #23896 and #23897. That run surfaced four further skill gaps, folded in here. The most valuable is a section on Arrow kernel semantics not matching Java's: total-ordered float comparison hiding `-0.0`, checked rather than wrapping arithmetic, and Java's `byte`/`short` promotion to `int` each caused one of the `pmod` bugs, and the skill had said nothing about any of them. The `datafusion-spark` README already points contributors to Comet and Sail when implementing a function. This is the same idea applied to checking one.
1 parent 35c9214 commit b245813

1 file changed

Lines changed: 996 additions & 0 deletions

File tree

  • .ai/skills/audit-datafusion-spark-expression

0 commit comments

Comments
 (0)