Skip to content

Commit cadc9fc

Browse files
committed
Merge remote-tracking branch 'upstream/main' into pr/delta-A1-spi
2 parents c6acc4f + 1142c3a commit cadc9fc

368 files changed

Lines changed: 22772 additions & 7447 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/skills/bug-triage/SKILL.md‎

Lines changed: 64 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: bug-triage
3-
description: Triage open Comet issues marked `requires-triage` per the project bug triage guide. Applies the recommended priority and area labels, removes `requires-triage`, and files a dated summary issue listing what was done. A human reviews the summary issue and closes it when satisfied.
3+
description: Triage open Comet issues marked `requires-triage` per the project bug triage guide. Classifies each issue as a bug or an enhancement, applies the recommended type (`bug`/`enhancement`), priority, and area labels, removes `requires-triage`, and files a dated summary issue listing what was done. A human reviews the summary issue and closes it when satisfied.
44
---
55

66
<!--
@@ -29,10 +29,16 @@ Run a bug triage pass for the `apache/datafusion-comet` repository.
2929
This skill triages every open issue carrying the `requires-triage` label. For
3030
each one it:
3131

32-
1. Decides a priority and area labels using the project's triage guide.
33-
2. Applies those labels via `gh`.
34-
3. Removes the `requires-triage` label.
35-
4. Records the decision (with rationale) in a single dated summary issue.
32+
1. Decides whether the issue is a bug or an enhancement (feature request).
33+
2. Decides a priority (bugs only) and area labels using the project's triage
34+
guide.
35+
3. Applies those labels via `gh` (`bug` or `enhancement`, plus priority/area),
36+
ensuring `bug` and `enhancement` are never both present.
37+
4. Removes the `requires-triage` label.
38+
5. Records the decision (with rationale) in a single dated summary issue.
39+
40+
`requires-triage` is auto-applied to **every** new issue, not just bug reports,
41+
so the first decision for each issue is always bug vs. enhancement.
3642

3743
A human reviewer reads the summary issue, sanity-checks the calls, and closes
3844
it when satisfied. Any label correction is done by the reviewer directly on the
@@ -74,26 +80,42 @@ file an empty summary issue and do not modify any labels.
7480

7581
For each issue, review the title and body and determine:
7682

77-
1. **Priority label** (exactly one): apply the decision tree from the guide.
83+
1. **Type label** (exactly one of `bug` / `enhancement`):
84+
- `bug`: something is broken. Wrong results, crashes, panics, regressions,
85+
test/CI failures, or any behavior that differs from what Comet should do.
86+
- `enhancement`: a request for new functionality or improvement that is not
87+
yet expected to work. New expression/operator support, a new config, a
88+
performance optimization request, a refactor, or a docs addition.
89+
- Do not rely on the title wording alone. An issue titled "bug: ..." may
90+
actually be a feature request, and an issue with no "bug" in the title may
91+
be a genuine defect. Classify from the actual content.
92+
- A bug and an enhancement are mutually exclusive: an issue must never carry
93+
both `bug` and `enhancement`. If the issue already has the wrong type label,
94+
remove it (see Step 5).
95+
2. **Priority label** (exactly one, **bugs only**): apply the decision tree from
96+
the guide.
7897
- `priority:critical` for correctness issues (silent wrong results, data
7998
corruption) and security vulnerabilities
8099
- `priority:high` for crashes, panics, segfaults, NPEs on supported paths
81100
- `priority:medium` for functional bugs / performance regressions with
82101
workarounds
83102
- `priority:low` for test-only, CI flakes, tooling, cosmetic
84-
2. **Area labels** (zero or more): from the area table in the guide
103+
3. **Area labels** (zero or more): from the area table in the guide
85104
(`area:writer`, `area:shuffle`, `area:aggregation`, `area:scan`,
86105
`area:expressions`, `area:ffi`, `area:ci`) plus the pre-existing area
87-
indicators (`spark 4`, `spark sql tests`).
88-
3. **Escalation note**: if the issue matches an escalation trigger from the
106+
indicators (`spark 4`, `spark sql tests`). Area labels apply to both bugs
107+
and enhancements.
108+
4. **Escalation note**: if the issue matches an escalation trigger from the
89109
guide (e.g., a `priority:high` crash that may also produce wrong results),
90110
note it in the summary.
91111

92112
## Step 4: Skip Issues You Cannot Confidently Classify
93113

94-
If an issue lacks reproduction steps or is otherwise too ambiguous to classify
95-
with confidence:
114+
If an issue is too ambiguous to classify with confidence (you cannot tell
115+
whether it is a bug or an enhancement, or a bug lacks reproduction steps and the
116+
priority is unclear):
96117

118+
- **Do not** apply a type label (`bug`/`enhancement`).
97119
- **Do not** apply a priority label.
98120
- **Do not** remove `requires-triage`.
99121
- **Do not** comment on the issue or ask the reporter for more info from this
@@ -106,17 +128,35 @@ Guessing is worse than skipping.
106128
## Step 5: Apply Labels
107129

108130
For each issue you classified in Step 3, apply the labels and remove
109-
`requires-triage` in a single `gh` call:
131+
`requires-triage` in a single `gh` call.
132+
133+
For a bug, add the `bug` type label and a priority label:
134+
135+
```bash
136+
gh issue edit <NUMBER> \
137+
--repo apache/datafusion-comet \
138+
--add-label "bug,priority:high,area:expressions" \
139+
--remove-label "requires-triage,enhancement"
140+
```
141+
142+
For an enhancement, add the `enhancement` type label and no priority label:
110143

111144
```bash
112145
gh issue edit <NUMBER> \
113146
--repo apache/datafusion-comet \
114-
--add-label "priority:high,area:expressions" \
115-
--remove-label "requires-triage"
147+
--add-label "enhancement,area:expressions" \
148+
--remove-label "requires-triage,bug"
116149
```
117150

118151
Notes:
119152

153+
- Always set exactly one type label. Add `bug` for bugs, `enhancement` for
154+
enhancements.
155+
- Always remove the opposite type label so an issue never carries both `bug`
156+
and `enhancement`. `--remove-label` is a no-op if the label is not present,
157+
so it is safe to remove the opposite type unconditionally.
158+
- Apply a priority label only to bugs. Do not add a priority label to
159+
enhancements.
120160
- Pass the labels as a single comma-separated string (no spaces around commas).
121161
- Quote labels that contain spaces (e.g., `"spark 4"`).
122162
- Only add labels that already exist in the repo. If a label from the guide is
@@ -141,11 +181,12 @@ Title: `Bug triage results: ${TRIAGE_DATE}`
141181
Body: a markdown report with these sections, in this order:
142182

143183
1. **Header**
144-
- Date, total issues processed, and counts per priority
184+
- Date, total issues processed, count of bugs vs. enhancements, and counts
185+
per priority
145186
- Link to `docs/source/contributor-guide/bug_triage.md`
146187
- Note that labels have already been applied; the reviewer should spot-check
147188
and close this issue when satisfied
148-
2. **Triaged** — one subsection per priority, ordered highest priority first
189+
2. **Bugs** — one subsection per priority, ordered highest priority first
149190
(`priority:critical`, then `priority:high`, then `priority:medium`, then
150191
`priority:low`). Omit any subsection whose count is zero. Do **not** use a
151192
markdown table anywhere in this section; use nested bullet lists only.
@@ -163,13 +204,17 @@ Body: a markdown report with these sections, in this order:
163204
The issue number (not the title) is the link target. The title is plain
164205
text. If there are no area labels, write `Area labels: none`.
165206

166-
3. **Escalations to consider** (omit section if empty) — bullet per issue with
207+
3. **Enhancements** (omit section if empty) — one top-level bullet per issue in
208+
the same `<title> ([#N](url))` form, with an `Area labels:` sub-bullet and a
209+
one-sentence rationale for classifying it as an enhancement. Enhancements
210+
have no priority subsections.
211+
4. **Escalations to consider** (omit section if empty) — bullet per issue with
167212
the same `<title> ([#N](url))` form, plus a sub-bullet explaining the
168213
trigger from the guide.
169-
4. **Skipped — needs more info** (omit if empty) — bullet per issue with the
214+
5. **Skipped — needs more info** (omit if empty) — bullet per issue with the
170215
same `<title> ([#N](url))` form, plus a sub-bullet explaining what is
171216
missing.
172-
5. **Failed to label** (omit if empty) — bullet per issue with the same
217+
6. **Failed to label** (omit if empty) — bullet per issue with the same
173218
`<title> ([#N](url))` form, plus a sub-bullet quoting the `gh` error.
174219

175220
File the issue with `gh`. Use a temp file for the body to keep quoting sane:

‎.github/workflows/ci.yml‎

Lines changed: 35 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,9 @@ jobs:
5656
github.event_name != 'pull_request' ||
5757
github.event.action != 'labeled' ||
5858
github.event.label.name == 'run-spark-3.4-tests' ||
59-
6059
github.event.label.name == 'run-spark-4.0-tests' ||
61-
github.event.label.name == 'run-spark-4.1-tests'
60+
github.event.label.name == 'run-spark-4.1-tests' ||
61+
github.event.label.name == 'run-iceberg-tests'
6262
steps:
6363
- uses: actions/checkout@v7
6464

@@ -117,6 +117,7 @@ jobs:
117117
iceberg_1_8: ${{ steps.compute.outputs.iceberg_1_8 }}
118118
iceberg_1_9: ${{ steps.compute.outputs.iceberg_1_9 }}
119119
iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }}
120+
iceberg_1_11: ${{ steps.compute.outputs.iceberg_1_11 }}
120121
steps:
121122
- uses: actions/checkout@v7
122123
with:
@@ -136,7 +137,7 @@ jobs:
136137
run: |
137138
set -euo pipefail
138139
if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
139-
for key in build_linux build_macos benchmark docs spark_3_4 spark_3_5 spark_4_0 spark_4_1 iceberg_1_8 iceberg_1_9 iceberg_1_10; do
140+
for key in build_linux build_macos benchmark docs spark_3_4 spark_3_5 spark_4_0 spark_4_1 iceberg_1_8 iceberg_1_9 iceberg_1_10 iceberg_1_11; do
140141
echo "${key}=true" >> "$GITHUB_OUTPUT"
141142
done
142143
exit 0
@@ -265,10 +266,13 @@ jobs:
265266
iceberg_1_8:
266267
name: Iceberg Spark SQL Tests (Iceberg 1.8)
267268
needs: changes
268-
# Main-only; never runs on PR events.
269+
# Main-only by default; PRs need the `run-iceberg-tests` label.
269270
if: |
270271
needs.changes.outputs.iceberg_1_8 == 'true' &&
271-
(github.event_name == 'push' || github.event_name == 'workflow_dispatch')
272+
(github.event_name == 'push' ||
273+
github.event_name == 'workflow_dispatch' ||
274+
(github.event_name == 'pull_request' &&
275+
contains(github.event.pull_request.labels.*.name, 'run-iceberg-tests')))
272276
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
273277
with:
274278
iceberg-short: '1.8'
@@ -280,10 +284,13 @@ jobs:
280284
iceberg_1_9:
281285
name: Iceberg Spark SQL Tests (Iceberg 1.9)
282286
needs: changes
283-
# Main-only; never runs on PR events.
287+
# Main-only by default; PRs need the `run-iceberg-tests` label.
284288
if: |
285289
needs.changes.outputs.iceberg_1_9 == 'true' &&
286-
(github.event_name == 'push' || github.event_name == 'workflow_dispatch')
290+
(github.event_name == 'push' ||
291+
github.event_name == 'workflow_dispatch' ||
292+
(github.event_name == 'pull_request' &&
293+
contains(github.event.pull_request.labels.*.name, 'run-iceberg-tests')))
287294
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
288295
with:
289296
iceberg-short: '1.9'
@@ -295,15 +302,35 @@ jobs:
295302
iceberg_1_10:
296303
name: Iceberg Spark SQL Tests (Iceberg 1.10)
297304
needs: changes
305+
# Main-only by default; PRs need the `run-iceberg-tests` label. Iceberg 1.11
306+
# (Spark 4.1) is the PR-gated Iceberg job; 1.10 covers the Spark 3.5 path.
298307
if: |
299308
needs.changes.outputs.iceberg_1_10 == 'true' &&
300309
(github.event_name == 'push' ||
301310
github.event_name == 'workflow_dispatch' ||
302-
github.event_name == 'pull_request')
311+
(github.event_name == 'pull_request' &&
312+
contains(github.event.pull_request.labels.*.name, 'run-iceberg-tests')))
303313
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
304314
with:
305315
iceberg-short: '1.10'
306316
iceberg-full: '1.10.0'
307317
spark-short: '3.5'
308318
spark-full: '3.5.8'
309319
java: 17
320+
321+
iceberg_1_11:
322+
name: Iceberg Spark SQL Tests (Iceberg 1.11)
323+
needs: changes
324+
# Runs on every PR: Iceberg 1.11 is our only Spark 4.1 Iceberg coverage.
325+
if: |
326+
needs.changes.outputs.iceberg_1_11 == 'true' &&
327+
(github.event_name == 'push' ||
328+
github.event_name == 'workflow_dispatch' ||
329+
github.event_name == 'pull_request')
330+
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
331+
with:
332+
iceberg-short: '1.11'
333+
iceberg-full: '1.11.0'
334+
spark-short: '4.1'
335+
spark-full: '4.1.2'
336+
java: 17

‎.github/workflows/iceberg_spark_test_reusable.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ jobs:
7575
jdk-version: 17
7676

7777
- name: Restore Cargo cache
78-
uses: actions/cache/restore@v5
78+
uses: actions/cache/restore@v6
7979
with:
8080
path: |
8181
~/.cargo/registry
@@ -93,7 +93,7 @@ jobs:
9393
RUSTFLAGS: "-Ctarget-cpu=x86-64-v3 -Clink-arg=-fuse-ld=bfd"
9494

9595
- name: Save Cargo cache
96-
uses: actions/cache/save@v5
96+
uses: actions/cache/save@v6
9797
if: github.ref == 'refs/heads/main'
9898
with:
9999
path: |

‎.github/workflows/pr_benchmark_check.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ jobs:
6060
cargo check --benches
6161
6262
- name: Cache Maven dependencies
63-
uses: actions/cache@v5
63+
uses: actions/cache@v6
6464
with:
6565
path: |
6666
~/.m2/repository

0 commit comments

Comments
 (0)