Skip to content

Commit 50c7b53

Browse files
committed
# Conflicts: # .github/workflows/ci.yml
2 parents 5db3fc7 + 424c31a commit 50c7b53

26 files changed

Lines changed: 1982 additions & 319 deletions

‎.github/labeler.yml‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# Path rules for the pull request labeler (.github/workflows/label_prs.yml).
19+
# A label is added when a changed file matches one of its globs; labels are never removed,
20+
# so hand-applied labels stay. area:ci and documentation apply only when every changed file
21+
# matches, since most PRs touch a workflow or a doc in passing.
22+
23+
area:shuffle:
24+
- changed-files:
25+
- any-glob-to-any-file:
26+
- 'native/shuffle/**'
27+
- 'spark/src/main/**/shuffle/**'
28+
- 'spark/src/main/scala/org/apache/comet/shuffle/**'
29+
- 'common/src/main/**/*Shuffle*'
30+
31+
area:scan:
32+
- changed-files:
33+
- any-glob-to-any-file:
34+
- 'native/core/src/parquet/**'
35+
- 'native/core/src/cloud/**'
36+
- 'native/core/src/execution/operators/scan.rs'
37+
- 'native/core/src/execution/operators/csv_scan.rs'
38+
- 'native/core/src/execution/operators/iceberg_scan.rs'
39+
- 'native/core/src/execution/operators/iceberg_common.rs'
40+
- 'spark/src/main/**/parquet/**'
41+
- 'common/src/main/**/parquet/**'
42+
- 'spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala'
43+
- 'spark/src/main/scala/org/apache/comet/rules/CometScanContrib.scala'
44+
- 'spark/src/main/scala/org/apache/spark/sql/comet/*Scan*.scala'
45+
- 'spark/src/main/scala/org/apache/comet/objectstore/**'
46+
- 'spark/src/main/scala/org/apache/comet/serde/operator/*Scan*.scala'
47+
- 'contrib/**'
48+
49+
area:writer:
50+
- changed-files:
51+
- any-glob-to-any-file:
52+
- 'native/core/src/execution/operators/parquet_writer.rs'
53+
- 'native/core/src/execution/operators/iceberg_write.rs'
54+
- 'spark/src/main/scala/org/apache/spark/sql/comet/*Write*.scala'
55+
- 'spark/src/main/scala/org/apache/spark/sql/comet/*Commit*.scala'
56+
- 'spark/src/main/scala/org/apache/comet/serde/operator/*Writ*.scala'
57+
- 'spark/src/main/scala/org/apache/comet/serde/operator/CometSink.scala'
58+
- 'spark/src/main/scala/org/apache/comet/serde/operator/CometDataWritingCommand.scala'
59+
- 'spark/src/main/**/*WriteFiles*'
60+
61+
area:aggregation:
62+
- changed-files:
63+
- any-glob-to-any-file:
64+
- 'native/spark-expr/src/agg_funcs/**'
65+
- 'spark/src/main/scala/org/apache/comet/serde/aggregates.scala'
66+
- 'spark/src/main/scala/org/apache/comet/serde/CometAggregateExpressionSerde.scala'
67+
- 'native/core/src/execution/merge_as_partial.rs'
68+
69+
area:expressions:
70+
- changed-files:
71+
- all-globs-to-any-file:
72+
- 'native/spark-expr/src/**'
73+
- '!native/spark-expr/src/agg_funcs/**'
74+
- '!native/spark-expr/src/jvm_udf/**'
75+
- changed-files:
76+
- all-globs-to-any-file:
77+
- 'spark/src/main/scala/org/apache/comet/serde/*.scala'
78+
- '!spark/src/main/scala/org/apache/comet/serde/aggregates.scala'
79+
- '!spark/src/main/scala/org/apache/comet/serde/CometAggregateExpressionSerde.scala'
80+
- changed-files:
81+
- any-glob-to-any-file:
82+
- 'native/core/src/execution/expressions/**'
83+
- 'spark/src/main/scala/org/apache/comet/expressions/**'
84+
- 'spark/src/main/scala/org/apache/comet/codegen/**'
85+
86+
area:joins:
87+
- changed-files:
88+
- any-glob-to-any-file:
89+
- 'native/core/src/execution/operators/dynamic_filter.rs'
90+
- 'native/core/src/execution/operators/dynamic_filter/**'
91+
- 'spark/src/main/scala/org/apache/comet/rules/RewriteJoin.scala'
92+
- 'spark/src/main/scala/org/apache/comet/rules/CometPlanAdaptiveDynamicPruningFilters.scala'
93+
- 'spark/src/main/**/*Join*'
94+
95+
area:ffi:
96+
- changed-files:
97+
- any-glob-to-any-file:
98+
- 'native/jni-bridge/**'
99+
- 'native/core/src/execution/columnar_to_row.rs'
100+
- 'spark/src/main/**/vector/**'
101+
- 'common/src/main/**/vector/**'
102+
- 'spark/src/main/scala/org/apache/comet/NativeColumnarToRowConverter.scala'
103+
- 'spark/src/main/scala/org/apache/spark/sql/comet/CometColumnarToRowExec.scala'
104+
- 'spark/src/main/scala/org/apache/spark/sql/comet/CometNativeColumnarToRowExec.scala'
105+
- 'spark/src/main/scala/org/apache/spark/sql/comet/CometSparkToColumnarExec.scala'
106+
107+
area:memory:
108+
- changed-files:
109+
- any-glob-to-any-file:
110+
- 'native/core/src/execution/memory_pools/**'
111+
- 'spark/src/main/**/*MemoryManager*'
112+
- 'spark/src/main/**/*MemoryAllocator*'
113+
- 'common/src/main/**/*MemoryAllocator*'
114+
- 'common/src/main/**/*OutOfMemory*'
115+
116+
area:udf:
117+
- changed-files:
118+
- any-glob-to-any-file:
119+
- 'native/spark-expr/src/jvm_udf/**'
120+
- 'spark/src/main/scala/org/apache/comet/udf/**'
121+
- 'spark/src/main/**/python/**'
122+
- 'spark/src/main/**/*MapInBatch*'
123+
- '.github/workflows/pyarrow_udf_test.yml'
124+
125+
area:Iceberg:
126+
- changed-files:
127+
- any-glob-to-any-file:
128+
- '**/*[Ii]ceberg*'
129+
- '**/*[Ii]ceberg*/**'
130+
131+
area:ci:
132+
- changed-files:
133+
- any-glob-to-all-files: '{.github/**,dev/ci/**,dev/release/**,dev/*.sh,dev/*.py,dev/*.xml,dev/*.toml}'
134+
135+
documentation:
136+
- changed-files:
137+
- any-glob-to-all-files: '{docs/**,**/*.md}'

‎.github/workflows/README.md‎

Lines changed: 69 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -30,16 +30,17 @@ is a `workflow_call` reusable invoked from the umbrella.
3030
| heavy job) |
3131
+-----------+-----------+
3232
|
33-
+-----------+-----------+-----------+-----------+-----------+-----------+
34-
| | | | | | |
35-
v v v v v v v
36-
pr_build_ pr_build_ pr_benchmark_ docs spark_3_5 spark_4_0 iceberg_1_10
37-
linux macos check (push) (PR+push) (PR+push) (PR+push)
38-
(PR+push) (PR+push) (PR+push)
39-
| | |
40-
v v v
41-
spark_3_4 / spark_4_1 iceberg_1_8 / 1_9
42-
(push or PR + label) (push only)
33+
+-----------------------------------+-----------------------------------+
34+
| | |
35+
v v v
36+
every PR + push push to main only PR with label, or push
37+
--------------- ----------------- ----------------------
38+
pr_build_linux docs spark_3_4 run-spark-3.4-tests
39+
pr_build_macos spark_4_0 run-spark-4.0-tests
40+
pr_benchmark_check iceberg_1_8 run-iceberg-tests
41+
spark_3_5 iceberg_1_9 run-iceberg-tests
42+
spark_4_1 iceberg_1_10 run-iceberg-tests
43+
iceberg_1_11
4344
4445
reusable workflows invoked via `uses:`:
4546
pr_build_linux.yml spark_sql_test_reusable.yml
@@ -50,25 +51,50 @@ is a `workflow_call` reusable invoked from the umbrella.
5051

5152
## What runs when
5253

53-
| Job in `ci.yml` | Triggered by | Path filter source |
54-
| -------------------- | ------------------------------------------------ | ----------------------------------- |
55-
| `preflight` | every PR / push to main / dispatch | none (always runs) |
56-
| `changes` | every PR / push to main / dispatch | runs `dev/ci/compute-changes.py` |
57-
| `pr_build_linux` | PR or push, paths matched | `dev/ci/compute-changes.py` |
58-
| `pr_build_macos` | PR or push, paths matched | `dev/ci/compute-changes.py` |
59-
| `pr_benchmark_check` | PR or push, paths matched | benchmark sources only |
60-
| `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` |
61-
| `spark_3_5` | PR or push, paths matched | Spark 3.5 sources |
62-
| `spark_4_0` | PR or push, paths matched | Spark 4.0 sources |
63-
| `spark_3_4` | push, **or** PR with `run-spark-3.4-tests` label | Spark 3.4 sources |
64-
| `spark_4_1` | push, **or** PR with `run-spark-4.1-tests` label | Spark 4.1 sources |
65-
| `iceberg_1_10` | PR or push, paths matched | Iceberg sources |
66-
| `iceberg_1_8` | push only | Iceberg sources |
67-
| `iceberg_1_9` | push only | Iceberg sources |
54+
| Job in `ci.yml` | Triggered by | Path filter source |
55+
| -------------------- | --------------------------------------------------- | ----------------------------------- |
56+
| `preflight` | every PR / push to main / dispatch / PR label added | none (always runs) |
57+
| `changes` | every PR / push to main / dispatch / PR label added | runs `dev/ci/compute-changes.py` |
58+
| `pr_build_linux` | PR or push, paths matched | `dev/ci/compute-changes.py` |
59+
| `pr_build_macos` | PR or push, paths matched | `dev/ci/compute-changes.py` |
60+
| `pr_benchmark_check` | PR or push, paths matched | benchmark sources only |
61+
| `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` |
62+
| `spark_3_5` | PR or push, paths matched | Spark 3.5 sources |
63+
| `spark_4_1` | PR or push, paths matched | Spark 4.1 sources |
64+
| `spark_3_4` | push, **or** PR with `run-spark-3.4-tests` label | Spark 3.4 sources |
65+
| `spark_4_0` | push, **or** PR with `run-spark-4.0-tests` label | Spark 4.0 sources |
66+
| `iceberg_1_11` | PR or push, paths matched | Iceberg sources |
67+
| `iceberg_1_8` | push, **or** PR with `run-iceberg-tests` label | Iceberg sources |
68+
| `iceberg_1_9` | push, **or** PR with `run-iceberg-tests` label | Iceberg sources |
69+
| `iceberg_1_10` | push, **or** PR with `run-iceberg-tests` label | Iceberg sources |
6870

6971
A heavy job appears in the PR's checks list as a `skipped` entry whenever
7072
its path filter or event criteria don't match. Skipped checks count as
71-
passing for branch protection.
73+
passing for branch protection, so a name that can report `skipped` is not
74+
safe to make a required check.
75+
76+
### Label events
77+
78+
`ci.yml` also fires on `pull_request.types: [labeled]`, so applying
79+
`run-spark-3.4-tests`, `run-spark-4.0-tests` or `run-iceberg-tests` starts the
80+
job that label gates without needing a new push. GitHub cannot filter a
81+
`pull_request` trigger by label name, so **every** label added to a PR starts a
82+
run, including labels that gate nothing.
83+
84+
Two rules keep those runs from corrupting the PR's status:
85+
86+
- `preflight` and `changes` carry no event guard and run every time. A job held
87+
back by `if:` still publishes a check run under its own name with conclusion
88+
`skipped`, and the newest check run for a name is what the merge box,
89+
`gh pr checks` and required-status-check evaluation read. Guarding
90+
`preflight` on the label name used to let any unrelated label overwrite the
91+
commit run's real `Preflight` verdict with `skipped`, see
92+
[#5007](https://github.com/apache/datafusion-comet/issues/5007).
93+
- Every heavy job excludes `labeled` events unless the label just added is the
94+
one that gates it. Without that, applying a single label re-ran the entire
95+
heavy pipeline at a commit that had already been tested.
96+
97+
`run-spark-4.1-tests` gates nothing: `spark_4_1` already runs on every PR.
7298

7399
## Standalone workflows (not under the umbrella)
74100

@@ -83,17 +109,18 @@ umbrella doesn't watch, or operate independently of the rest of CI:
83109
| `stale.yml` | Daily stale-PR closer. |
84110
| `take.yml` | Issue-comment trigger for `take` / `untake`. |
85111
| `label_new_issues.yml` | Issue trigger to apply `requires-triage`. |
112+
| `label_prs.yml` | Runs on `pull_request_target` so it can label pull requests opened from forks. |
86113

87114
## Reusable workflows (called by `ci.yml`)
88115

89-
| File | Called from `ci.yml` job(s) |
90-
| --------------------------------- | -------------------------------------------------- |
91-
| `pr_build_linux.yml` | `pr_build_linux` |
92-
| `pr_build_macos.yml` | `pr_build_macos` |
93-
| `pr_benchmark_check.yml` | `pr_benchmark_check` |
94-
| `docs.yaml` | `docs` |
95-
| `spark_sql_test_reusable.yml` | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1` |
96-
| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10` |
116+
| File | Called from `ci.yml` job(s) |
117+
| --------------------------------- | ------------------------------------------------------------ |
118+
| `pr_build_linux.yml` | `pr_build_linux` |
119+
| `pr_build_macos.yml` | `pr_build_macos` |
120+
| `pr_benchmark_check.yml` | `pr_benchmark_check` |
121+
| `docs.yaml` | `docs` |
122+
| `spark_sql_test_reusable.yml` | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1` |
123+
| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10`, `iceberg_1_11` |
97124

98125
## Modifying path filters
99126

@@ -171,3 +198,11 @@ umbrella exposes per-job names like `CI / pr_build_linux / Lint`,
171198
protection rules to point at the new names; the old standalone workflow
172199
names (`Spark SQL Tests (Spark 3.5)`, `PR Build (Linux)`, ...) no longer
173200
exist as top-level workflows.
201+
202+
`.asf.yaml` currently declares no `required_status_checks` for `main`, only
203+
`required_approving_review_count: 1`. Anything added there must be a name that
204+
never legitimately reports `skipped`, because GitHub counts a skipped check as
205+
passing. The bare caller-job names (`PR Build (Linux)`, `Spark SQL Tests
206+
(Spark 3.5)`, ...) are not such names: a reusable workflow that actually runs
207+
publishes only its child jobs, so the bare name shows up solely when the caller
208+
was skipped.

0 commit comments

Comments
 (0)