Repository navigation
docs: fix SQL examples that the default dialect cannot parse - #26373
Merged
Merged
Conversation
The COMPRESSION TYPE clause now raises a parser error, so the compressed-CSV example uses 'format.compression' in OPTIONS. The struct coercion page wrote struct types as STRUCT(a INT), which only the DuckDB dialect accepts; use STRUCT<a INT>.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #26373 +/- ##
==========================================
- Coverage 82.85% 82.85% -0.01%
==========================================
Files 1151 1151
Lines 455996 455996
Branches 455996 455996
==========================================
- Hits 377805 377797 -8
- Misses 55239 55244 +5
- Partials 22952 22955 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 22, 2026
neilconway
enabled auto-merge
October 23, 2026 01:27
zhuqi-lucas
pushed a commit
to zhuqi-lucas/arrow-datafusion
that referenced
this pull request
Oct 28, 2026
## Which issue does this PR close? Related to apache#25148. Inspired by apache/airflow#75619. ## Rationale for this change Merge queue runs continue using runners after their group is removed or replaced. The Rust workflow gives each merge-group SHA a separate concurrency group, so a replacement does not cancel the old run. Jobs still waiting for a runner can start long after the queue has stopped using their results. A read-only analysis of runs created on October 19–25, 2026 (UTC) found: | Measurement | Result | | --- | ---: | | Merge-group workflow runs inspected | 1,170 | | Distinct merge-group commits / PRs | 195 / 154 | | Removed or superseded groups with subsequent runner use | 36 | | Workflows / jobs with subsequent runner use | 91 / 473 | | Runner time after removal or replacement | 20.3 hours | | Jobs that started after removal or replacement | 312, using 12.7 runner-hours | For the Rust and extended-test workflows, this was 19.0 of 369.4 runner-hours, or 5.2% of their sampled runner time. Cancellation taking effect one minute after obsolescence would have avoided 17.9 runner-hours; at five minutes, 11.9 hours; at a fixed fifteen-minute delay, 9.4 hours. A separate model using the next exact quarter-hour boundary estimates 11.2 hours. Examples: - [apache#26562](apache#26562): after manual queue removal, 34 jobs used another 121.6 runner-minutes in parallel. They finished within about 14 minutes. [Rust run](https://github.com/apache/datafusion/actions/runs/40069843292). - [apache#26373](apache#26373): the group was removed for failed checks at 21:21 UTC on October 22, but its MSRV job did not start until 23:20 UTC. [Rust run](https://github.com/apache/datafusion/actions/runs/39855621556). The cutoff was an exact-SHA queue-removal event or the creation of a replacement group for the same PR. For each job, the analysis summed time after the later of its start and the cutoff plus the assumed cancellation delay. It excluded skipped jobs, deduplicated successful jobs copied into rerun records, and excluded seven rerun attempts started after obsolescence that could have been deliberate debugging. All Rust and extended-test jobs were fetched for the denominator; other workflows were inspected when their run metadata showed activity after the cutoff. Merged groups and all rerun attempts contributed zero to the reported avoidable time, so preserving both leaves these estimates unchanged. These are historical runner times and estimated avoidable portions, not deployed savings or billing estimates. Of the 312 late-starting jobs, 42 used Runs-On (2.4 runner-hours); 270 used GitHub Linux runners. We have not measured whether cancelling queued jobs immediately terminates instances already starting. The models also exclude ref-deletion latency, cancellation cleanup, scheduler delays, and cleanup-workflow overhead. Recent runner changes make this sample unsuitable as an annual forecast. ## What changes are included in this PR? - Scan when a new merge group is created and every 15 minutes. Ref deletion can lag the event, so prompt cancellation is not guaranteed. - Ship in observation mode. Cancellation requires the repository Actions variable `MERGE_QUEUE_CLEANUP_ENABLED=true`. Document at least one week of observation before enabling it; there is no automatic activation. - Log candidate run IDs, SHAs, attempts, reasons, and UTC observation times for comparison with eventual outcomes. Each merge-group scan has its own concurrency group, keyed by the group SHA, so a waiting scan on a group commit is never replaced and shown as cancelled. Scheduled and manual scans keep one group per mode, so a manual dry run cannot replace a waiting cancellation scan. - Target only first-attempt, unfinished `merge_group` runs in `apache/datafusion` whose exact queue ref is missing or points elsewhere. Recheck the run and ref, then compare with its target branch to preserve commits already merged. Preserve reruns, other event types, and cleanup itself. - Keep exact-ref detection while the GraphQL queue-entry-to-group-SHA mapping remains unvalidated. A changed ref target is a safeguard for a reused queue branch; the suffix is not treated as the run SHA. - If an Actions run listing reaches the 1,000-result cap, stop the scan without cancelling any runs. Active merge-group runs stay far below the cap. - Share the ref lookup and the target-branch comparison among the workflows of one group. - API errors preserve affected runs and allow other candidates to be checked. Merge-group scans warn; scheduled/manual scans fail visibly. Recheck HTTP 409 responses and retry unresolved cancellations on later scans. - Check out the default branch's script. The workflow definition itself still comes from the triggering revision. - Add behavioral tests to Dev CI and the local lint suite; remove the test that repeated YAML literals. ## What is the testing strategy for this PR? - 35 cleanup tests cover active/merged groups, missing/replaced refs, target branches containing slashes, reruns, races, dry runs, API errors, error reporting by trigger, pagination, the 1,000-result cap, and shared comparisons. Together with existing fork-CI tests, 59 Node tests pass. - Mutation checks confirm the merged-commit, rerun, listing-cap and shared-comparison regressions are detected. - `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and the complete `./dev/rust_lint.sh` suite pass, including generated docs, links, dependency checks, Rust docs, and the HTML documentation build. - actionlint 1.7.12 passes for both changed workflows. - A live read-only probe completed five listing calls with no eligible candidates. No workflow was cancelled during validation. The observation period, production cancellation latency, and instance shutdown behavior remain to be validated after deployment. ## Are there any user-facing changes? Initially the workflow reports candidates only. Once enabled by a maintainer, it cancels obsolete first-attempt merge-queue runs while preserving merged commits and deliberate reruns. Contributors can rerun obsolete jobs for debugging. No query behavior or public API changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
None; documentation fix.
Closes #.
Rationale for this change
Two SQL examples in the user guide fail to parse with the default dialect:
ddl.md, "Example: Compression":COMPRESSION TYPE GZIPnow raisesParserError("COMPRESSION TYPE clause is no longer in use. Please use the OPTIONS clause with 'format.compression' set appropriately ...").struct_coercion.mdwrites struct types asSTRUCT(a INT, b INT), in bothCREATE TABLEcolumn definitions andCAST. Only the DuckDB dialect parses that form; the default dialect givesParserError("Expected: ',' or ')' after column definition, found: ("). Nine of the page's SQL blocks fail this way.What changes are included in this PR?
'format.compression' 'gzip'inOPTIONSinstead.STRUCT<a INT, b INT>, which the default dialect accepts. Thestruct()value constructor is unchanged.What is the testing strategy for this PR?
Ran every SQL block on both pages with
datafusion-clibuilt frommain, default dialect. Before, the compression example and nine struct blocks failed to parse; after, the remaining errors are fragments that reference tables or columns from other blocks. The compression example reads a gzipped CSV back correctly../ci/scripts/doc_prettier_check.shpasses. Docs only, no test.Are there any user-facing changes?
Documentation only.
Generated with Claude Code (claude-opus-5-5). It came from parsing the SQL examples in the docs against
main, not from an issue.