Skip to content

Commit 2d1ee49

Browse files
authored
ci: run every tier on release-branch pull requests, and the full suite before an RC (apache#6218)
* docs: document CI coverage on release branches and run the full suite before an RC Release branches get only the PR tier: GitHub fires schedules only on the default branch, the merge queue covers only main, and ci.yml runs on push only for main. Document that, add a step that dispatches ci.yml and miri.yml on the release branch before tagging each release candidate, and correct the stale claim that the release branch runs the same CI as main. Also keep the docs deploy to main. A dispatch runs every job, so dispatching ci.yml on a release branch would have published that branch's docs over the website. * ci: run every tier on release-branch pull requests A release branch has no merge queue and no nightly: the queue ruleset targets the default branch, ci.yml runs on push only for main, and GitHub fires schedule only on the default branch. So a tier that holds a suite back for the queue or the nightly drops it there instead. A pull request against branch-N.M now runs every pr, queue and nightly job, still routed by the path filters. docs stays push-only and Spark 3.4 stays label-only, so a labeled run there adds Spark 3.4 or nothing. The base branch reaches compute-changes.py as PR_BASE_REF. check-ci-config.py now pins every variable event_from_env reads to the github context expression ci.yml sets it from, because a dropped or mistyped one reads as an empty string and changes the routing without failing anything. * ci: pin the docs deploy guard, and check it before dispatching a release branch A dispatch runs the release branch's own ci.yml, docs job included, so the promise that it leaves the website alone holds only on a branch that carries the refs/heads/main guard. branch-1.1 was cut without it. Both dispatch instructions now start by printing the branch's docs job, and check-ci-config.py fails if the guard is dropped from ci.yml. The dispatch result check now queries each workflow on its own. With --limit 2 across both, a second ci.yml dispatch could push the Miri run out of the list. Also correct two statements: the documentation and change log pull requests change only Markdown, so they run no heavy suite, and a CI change on main reaches a release branch only if it is backported.
1 parent 605051a commit 2d1ee49

6 files changed

Lines changed: 319 additions & 9 deletions

File tree

‎.github/workflows/README.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,7 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
152152
| `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only |
153153
| `delta_build_gate` | merge group, **or** PR with `run-delta-build-gate` | main sources, poms, `contrib/delta` |
154154
| `pyarrow_udf_test` | merge group, **or** PR with `run-pyarrow-udf-tests` | map-in-batch and Python runner code |
155-
| `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` |
155+
| `docs` | push to main, paths matched, **or** dispatch on `main` | `.asf.yaml`, `docs/**`, `docs.yaml` |
156156
| `spark_3_5` | nightly, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources |
157157
| `spark_4_1` | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests` | Spark 4.1 sources |
158158
| `spark_3_4` | PR with `run-spark-3.4-tests`, or dispatch | Spark 3.4 sources |
@@ -169,6 +169,15 @@ its path filter or event criteria don't match. Skipped checks count as
169169
passing for branch protection, so a name that can report `skipped` is not
170170
safe to make a required check.
171171

172+
A pull request whose base is a release branch (`branch-N.M`) is the one
173+
exception to the table: it runs every job the table puts in the PR, merge
174+
group or nightly tier, because a release branch has no merge queue and no
175+
nightly to run the last two later. `docs` stays push-only and `spark_3_4`
176+
still needs its label, so a `labeled` run there adds `spark_3_4` or nothing.
177+
`changes` hands the base branch to `compute-changes.py` as `PR_BASE_REF`.
178+
`check-ci-config.py` pins that wiring, because a dropped variable reads as an
179+
empty string and would route the pull request as if it targeted `main`.
180+
172181
### Label events
173182

174183
`ci_label.yml` fires on `pull_request.types: [labeled]` and calls `ci.yml`

‎.github/workflows/ci.yml‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,11 @@
4040
# Spark 3.4 is deprecated and sits outside all three tiers: it runs only when
4141
# a pull request carries `run-spark-3.4-tests`, or from a manual dispatch.
4242
#
43+
# Release branches (`branch-N.M`) get neither the queue nor the nightly: the
44+
# queue ruleset covers only the default branch, `push` below is main-only, and
45+
# GitHub fires `schedule` only on the default branch. So a pull request that
46+
# targets one runs all three tiers at once, still routed by the path filters.
47+
#
4348
# Which tier a job sits in is POLICY in dev/ci/compute-changes.py, not an
4449
# expression here. Heavy jobs deliberately have no `push` tier: the queue
4550
# already tested the exact commit that lands, so re-running the pipeline on
@@ -265,6 +270,7 @@ jobs:
265270
EVENT_ACTION: ${{ github.event.action }}
266271
LABEL_NAME: ${{ github.event.label.name }}
267272
PR_LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }}
273+
PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
268274
PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
269275
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
270276
MQ_BASE_SHA: ${{ github.event.merge_group.base_sha }}
@@ -345,6 +351,7 @@ jobs:
345351
#
346352
# pull request linux, full -> profiles: pr
347353
# ... with the label linux, full, all -> profiles: all
354+
# ... to branch-N.M linux, full, all -> profiles: all
348355
# `labeled` run all -> profiles: nightly
349356
# merge queue linux, full -> profiles: pr
350357
# nightly all -> profiles: nightly
@@ -400,8 +407,11 @@ jobs:
400407
docs:
401408
name: Deploy Comet site
402409
needs: changes
403-
# docs deploys to asf-site, so only run on push-to-main (or a manual dispatch).
404-
if: needs.changes.outputs.docs == 'true'
410+
# docs deploys to asf-site, so only run on push-to-main (or a manual
411+
# dispatch of main). The ref check is for dispatch: it runs every job, and
412+
# a dispatch on a release branch would otherwise publish that branch's docs
413+
# over the site.
414+
if: needs.changes.outputs.docs == 'true' && github.ref == 'refs/heads/main'
405415
uses: ./.github/workflows/docs.yaml
406416

407417
spark_3_4:

‎dev/ci/check-ci-config.py‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,13 @@
2727
# could not be tested; POLICY_CASES below is the test it never had. The
2828
# expected sets are transcribed from the `if:` expressions ci.yml carried
2929
# before the policy moved, so a regression here is a behaviour change.
30+
# The event reaches the script as environment variables set by the
31+
# `Compute outputs` step of ci.yml's `Detect changes` job, and a variable
32+
# dropped there reads as an empty string, which is a valid value (no
33+
# label, no base branch), so that wiring is pinned as well. So is the
34+
# `refs/heads/main` guard on the `docs` job: a dispatch routes every job,
35+
# docs included, and that guard is all that keeps a dispatch on a release
36+
# branch from publishing that branch's docs over the site.
3037
#
3138
# 3. Required-check coverage. `Required Checks` in ci.yml is the job that
3239
# `.asf.yaml` can name in `required_status_checks` for main. A heavy job
@@ -390,6 +397,62 @@
390397
},
391398
set(),
392399
),
400+
# A pull request against a release branch runs the queue and nightly tiers
401+
# as well, because a release branch has no queue and no nightly to run them
402+
# later. The site deploy still never runs from a pull request, and the
403+
# deprecated Spark 3.4 suite still waits for its label.
404+
(
405+
{"name": "pull_request", "action": "opened", "labels": [], "base": "branch-1.1"},
406+
QUEUE_TIER | NIGHTLY_TIER,
407+
),
408+
(
409+
{"name": "pull_request", "action": "synchronize", "labels": [], "base": "branch-2.0"},
410+
QUEUE_TIER | NIGHTLY_TIER,
411+
),
412+
(
413+
{
414+
"name": "pull_request",
415+
"action": "synchronize",
416+
"labels": ["run-spark-3.4-tests"],
417+
"base": "branch-1.1",
418+
},
419+
QUEUE_TIER | NIGHTLY_TIER | SPARK_DEPRECATED,
420+
),
421+
# There the commit run already covered everything a label gates except
422+
# Spark 3.4, so a `labeled` run adds Spark 3.4 or nothing.
423+
(
424+
{
425+
"name": "pull_request",
426+
"action": "labeled",
427+
"label": "run-spark-3.4-tests",
428+
"labels": ["run-spark-3.4-tests"],
429+
"base": "branch-1.1",
430+
},
431+
SPARK_DEPRECATED,
432+
),
433+
(
434+
{
435+
"name": "pull_request",
436+
"action": "labeled",
437+
"label": "run-iceberg-tests",
438+
"labels": ["run-iceberg-tests"],
439+
"base": "branch-1.1",
440+
},
441+
set(),
442+
),
443+
# Only `branch-N.M` is a release branch. A pull request against main, or
444+
# one stacked on another branch in the repository, keeps the PR tier.
445+
({"name": "pull_request", "action": "synchronize", "labels": [], "base": "main"}, PR_TIER),
446+
({"name": "pull_request", "action": "synchronize", "labels": [], "base": "pr-5654"}, PR_TIER),
447+
(
448+
{
449+
"name": "pull_request",
450+
"action": "synchronize",
451+
"labels": [],
452+
"base": "branch-1.1-backports",
453+
},
454+
PR_TIER,
455+
),
393456
]
394457

395458

@@ -654,6 +717,8 @@ def check_event_policy():
654717
label += f"/{event['action']}"
655718
if event.get("label"):
656719
label += f" +{event['label']}"
720+
if event.get("base"):
721+
label += f" base={event['base']}"
657722
failures.append(
658723
f"{label} labels={event.get('labels', [])}: "
659724
f"unexpectedly allowed {sorted(actual - expected) or 'nothing'}, "
@@ -688,6 +753,98 @@ def check_event_policy():
688753
return not failures
689754

690755

756+
# The environment variables `event_from_env` in compute-changes.py reads, and
757+
# the `github` context expression ci.yml's `Compute outputs` step must set each
758+
# one from. A typo in an expression is as silent as a missing variable: both
759+
# arrive as an empty string.
760+
EVENT_ENV_SOURCES = {
761+
"EVENT_NAME": "github.event_name",
762+
"EVENT_ACTION": "github.event.action",
763+
"LABEL_NAME": "github.event.label.name",
764+
"PR_LABELS": "toJSON(github.event.pull_request.labels.*.name)",
765+
"PR_BASE_REF": "github.event.pull_request.base.ref",
766+
}
767+
ENV_READ = re.compile(r'os\.environ\.get\("([A-Z_]+)"')
768+
ENV_SET = re.compile(r"^\s+([A-Z_]+):\s+\$\{\{\s*(.+?)\s*\}\}\s*$")
769+
770+
771+
def compute_step_env():
772+
"""{variable: expression} from the `env:` of ci.yml's `Compute outputs` step."""
773+
env, in_step, in_env = {}, False, False
774+
for line in CI_WORKFLOW.read_text(encoding="utf-8").splitlines():
775+
if re.match(r"^\s+- name: Compute outputs\s*$", line):
776+
in_step = True
777+
elif in_step and not in_env:
778+
if re.match(r"^\s+- ", line):
779+
break # the next step; this one has no `env:`
780+
in_env = bool(re.match(r"^\s+env:\s*$", line))
781+
elif in_env:
782+
match = ENV_SET.match(line)
783+
if not match:
784+
break
785+
env[match.group(1)] = match.group(2)
786+
return env
787+
788+
789+
def check_event_env():
790+
"""Every variable POLICY reads is set by `Detect changes`, from the right context.
791+
792+
Without PR_BASE_REF, for instance, every pull request against a release
793+
branch would quietly fall back to the PR tier and every check would pass.
794+
"""
795+
failures = []
796+
source = Path("dev/ci/compute-changes.py").read_text(encoding="utf-8")
797+
body = source.split("\ndef event_from_env", 1)[1].split("\ndef ", 1)[0]
798+
read = set(ENV_READ.findall(body))
799+
env = compute_step_env()
800+
for name in sorted(read - set(EVENT_ENV_SOURCES)):
801+
failures.append(
802+
f"event_from_env reads {name}, which EVENT_ENV_SOURCES does not "
803+
f"list; add the `github` context expression it comes from"
804+
)
805+
for name in sorted(set(EVENT_ENV_SOURCES) - read):
806+
failures.append(f"EVENT_ENV_SOURCES lists {name}, which event_from_env no longer reads")
807+
for name in sorted(read & set(EVENT_ENV_SOURCES)):
808+
expected = EVENT_ENV_SOURCES[name]
809+
if name not in env:
810+
failures.append(
811+
f"{CI_WORKFLOW}: the `Compute outputs` step does not set {name}, "
812+
f"so compute-changes.py reads it as empty"
813+
)
814+
elif env[name] != expected:
815+
failures.append(
816+
f"{CI_WORKFLOW}: the `Compute outputs` step sets {name} from "
817+
f"`{env[name]}`, not `{expected}`"
818+
)
819+
for failure in failures:
820+
print(f"event env: {failure}")
821+
return not failures
822+
823+
824+
# The site deploy's job-level `if:` in ci.yml. It has to be on the job rather
825+
# than in POLICY, because a dispatch routes every job (see POLICY_CASES).
826+
DOCS_JOB = "docs"
827+
DOCS_MAIN_GUARD = re.compile(r"^ if:.*github\.ref\s*==\s*'refs/heads/main'")
828+
829+
830+
def check_docs_deploy_guard():
831+
"""The site deploy runs from main only, whatever the event.
832+
833+
docs.yaml rsyncs the built site over asf-site with --delete and falls back
834+
to `git push --force`, and the release process dispatches ci.yml on the
835+
release branch before every release candidate.
836+
"""
837+
_, guarded = guarded_jobs(CI_WORKFLOW, DOCS_MAIN_GUARD)
838+
if DOCS_JOB in guarded:
839+
return True
840+
print(
841+
f"docs deploy: the `{DOCS_JOB}` job in {CI_WORKFLOW} must require "
842+
f"github.ref == 'refs/heads/main' in its `if:`, or a dispatch on a "
843+
f"release branch publishes that branch's docs over the site"
844+
)
845+
return False
846+
847+
691848
def artifact_names(path):
692849
"""Return ([upload names], [download names]) for one workflow file."""
693850
uploads, downloads = [], []
@@ -1277,6 +1434,8 @@ def check_cache_save_scope():
12771434
if __name__ == "__main__":
12781435
ok = check_change_filters()
12791436
ok = check_event_policy() and ok
1437+
ok = check_event_env() and ok
1438+
ok = check_docs_deploy_guard() and ok
12801439
ok = check_spark_sql_modules() and ok
12811440
ok = check_linux_test_profiles() and ok
12821441
ok = check_artifact_names() and ok

‎dev/ci/compute-changes.py‎

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -435,6 +435,17 @@
435435
# Adding "push" back to a test job would make every merge run it twice, once
436436
# in the queue and once after, which is the thing the queue was adopted to
437437
# avoid.
438+
#
439+
# A pull request that targets a release branch (`branch-N.M`) is the one
440+
# exception: it runs every "pr", "queue" and "nightly" job. On main the tiers
441+
# spread the suites over three events, and every change still meets all of
442+
# them. A release branch has only the pull request: the merge queue covers the
443+
# default branch alone, ci.yml runs on push only for main, and GitHub fires
444+
# `schedule` only on the default branch. A tier there would not defer a suite,
445+
# it would drop it. And the tiers exist because of main's volume, which a
446+
# release branch does not have: branch-1.0 took 28 pull requests in its first
447+
# two months. "push" jobs (the site deploy) and label-only jobs (Spark 3.4)
448+
# keep their rules on a release branch.
438449
POLICY = {
439450
# The one test job that also runs on push to main, and only because of
440451
# actions/cache scoping: a pull request can restore caches saved on its
@@ -518,16 +529,23 @@
518529
}
519530

520531

532+
# Release branches are named `branch-<major>.<minor>`, as in branch-0.17 and
533+
# branch-1.0. A pull request against one runs these tiers; see the end of the
534+
# comment above POLICY.
535+
RELEASE_BRANCH = re.compile(r"branch-\d+\.\d+")
536+
RELEASE_BRANCH_TIERS = ("pr", "queue", "nightly")
537+
538+
521539
def gating_labels(job):
522540
return [t[len("label:"):] for t in POLICY[job] if t.startswith("label:")]
523541

524542

525543
def event_allows(job, event):
526544
"""Does `event` permit `job` to run, ignoring which files changed?
527545
528-
`event` is {"name", "action", "label", "labels"}: the workflow event name,
529-
the pull_request action, the label just added on a `labeled` event, and the
530-
labels currently on the pull request.
546+
`event` is {"name", "action", "label", "labels", "base"}: the workflow event
547+
name, the pull_request action, the label just added on a `labeled` event,
548+
the labels currently on the pull request, and the branch it targets.
531549
"""
532550
tiers = POLICY[job]
533551
name = event.get("name")
@@ -542,6 +560,8 @@ def event_allows(job, event):
542560
return "nightly" in tiers
543561
if name != "pull_request":
544562
return False
563+
if RELEASE_BRANCH.fullmatch(event.get("base", "")):
564+
return release_branch_allows(job, event)
545565

546566
gates = gating_labels(job)
547567
if gates:
@@ -560,6 +580,19 @@ def event_allows(job, event):
560580
return True
561581

562582

583+
def release_branch_allows(job, event):
584+
"""event_allows for a pull request that targets a release branch."""
585+
tiers = POLICY[job]
586+
gates = gating_labels(job)
587+
automatic = any(tier in tiers for tier in RELEASE_BRANCH_TIERS)
588+
# The opened/synchronize run already ran every automatic job at this
589+
# commit, so a `labeled` run adds only a job that nothing else runs there,
590+
# which leaves the label-only Spark 3.4 suite.
591+
if event.get("action") == "labeled":
592+
return not automatic and event.get("label") in gates
593+
return automatic or any(label in event.get("labels", []) for label in gates)
594+
595+
563596
def compute(files, event):
564597
"""Return {job: bool}, folding the path filter and the event policy."""
565598
return {
@@ -575,6 +608,7 @@ def event_from_env():
575608
"action": os.environ.get("EVENT_ACTION", ""),
576609
"label": os.environ.get("LABEL_NAME", ""),
577610
"labels": json.loads(labels) if labels.strip() else [],
611+
"base": os.environ.get("PR_BASE_REF", ""),
578612
}
579613

580614

‎docs/source/contributor-guide/ci.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,9 @@ Which tier a job belongs to is the `POLICY` table in `dev/ci/compute-changes.py`
105105
filters are the `FILTERS` table in the same file, and `dev/ci/check-ci-config.py` holds the test
106106
cases that pin both down.
107107

108+
A pull request against a release branch runs all three tiers at once; see
109+
[Release branches](#release-branches).
110+
108111
## Opting a pull request into a suite the PR tier skips
109112

110113
Each suite outside the PR tier has a label that runs it on a pull request:
@@ -341,6 +344,41 @@ and Iceberg 1.11. If everything is skipped, open the run's `Detect changes` job:
341344
`Nightly base:` commit it diffed against and the list of changed files, which is enough to tell a
342345
genuinely quiet day from a base that has drifted.
343346

347+
## Release branches
348+
349+
Release branches (`branch-N.M`) start with the workflow files `main` had when they were cut, but have
350+
no merge queue and no nightly run. The merge queue covers only `main`, `ci.yml` runs on push only for `main`, and GitHub
351+
fires scheduled workflows only on the default branch, so a release branch gets no scheduled `ci.yml`,
352+
Miri or CodeQL run, and nothing runs after a pull request merges.
353+
354+
On `main`, the tiers hold the expensive suites back for the queue and the nightly run, which still
355+
test every change. A release branch has neither, so holding a suite back there would mean never
356+
running it. A pull request that targets a release branch, such as a backport, therefore runs the PR,
357+
queue and nightly tiers together. Release branches get few pull requests, so this costs little. The
358+
path filters still apply, so a documentation-only change runs none of the heavy suites. The Spark
359+
SQL suite for Spark 3.4 still needs its `run-spark-3.4-tests` label, and the other `run-*` labels
360+
have nothing to add there.
361+
362+
A release branch runs the workflow files committed on it, so a change to these rules on `main`
363+
reaches a release branch only if it is backported. `branch-1.0` predates the tiers and follows its
364+
own, older rules.
365+
366+
Each pull request is tested against the release branch as it was when its run started, and with no
367+
merge queue, nothing tests the result of merging it. Two backports that pass on their own can still
368+
break the branch once both have landed. To run every suite against a release branch as it stands,
369+
dispatch `ci.yml` on it. A dispatch runs every job in that branch's own `ci.yml`, including `docs`,
370+
which publishes the website. So first check that the `if:` of the branch's `docs` job requires
371+
`github.ref == 'refs/heads/main'`. A branch without that guard publishes its own docs over the site.
372+
373+
```sh
374+
git show apache/branch-N.M:.github/workflows/ci.yml | sed -n '/^ docs:/,/uses:/p'
375+
gh workflow run ci.yml --repo apache/datafusion-comet --ref branch-N.M
376+
```
377+
378+
The release process does this before tagging each release candidate; see
379+
[Run the Full CI Suite](release_process.md#run-the-full-ci-suite). A failed dispatched run opens no
380+
`ci-nightly-failure` issue.
381+
344382
## Reproducing a suite failure locally
345383

346384
`dev/local-ci.sh` builds the same sandbox a runner builds and runs the Spark SQL or Iceberg

0 commit comments

Comments
 (0)