Skip to content

Commit 29ce01d

Browse files
committed
Merge remote-tracking branch 'apache/main' into release-plan-stop-tokio-task
# Conflicts: # spark/src/test/scala/org/apache/spark/CometExecIteratorLifecycleSuite.scala
2 parents 5d24232 + e897f8a commit 29ce01d

101 files changed

Lines changed: 7912 additions & 366 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.asf.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,12 @@ github:
110110
branch-0.17:
111111
required_pull_request_reviews:
112112
required_approving_review_count: 1
113+
branch-1.0:
114+
required_pull_request_reviews:
115+
required_approving_review_count: 1
116+
branch-1.1:
117+
required_pull_request_reviews:
118+
required_approving_review_count: 1
113119
# GitHub's merge queue. Classic `protected_branches` above has no way to
114120
# express this, but .asf.yaml rulesets accept a raw Rulesets API payload, so
115121
# no INFRA ticket is needed. Same approach as apache/iceberg-python and

‎.github/workflows/README.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,7 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
152152
| `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only |
153153
| `delta_build_gate` | merge group, **or** PR with `run-delta-build-gate` | main sources, poms, `contrib/delta` |
154154
| `pyarrow_udf_test` | merge group, **or** PR with `run-pyarrow-udf-tests` | map-in-batch and Python runner code |
155-
| `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` |
155+
| `docs` | push to main, paths matched, **or** dispatch on `main` | `.asf.yaml`, `docs/**`, `docs.yaml` |
156156
| `spark_3_5` | nightly, **or** PR with `run-spark-3.5-tests` | Spark 3.5 sources |
157157
| `spark_4_1` | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests` | Spark 4.1 sources |
158158
| `spark_3_4` | PR with `run-spark-3.4-tests`, or dispatch | Spark 3.4 sources |
@@ -169,6 +169,15 @@ its path filter or event criteria don't match. Skipped checks count as
169169
passing for branch protection, so a name that can report `skipped` is not
170170
safe to make a required check.
171171

172+
A pull request whose base is a release branch (`branch-N.M`) is the one
173+
exception to the table: it runs every job the table puts in the PR, merge
174+
group or nightly tier, because a release branch has no merge queue and no
175+
nightly to run the last two later. `docs` stays push-only and `spark_3_4`
176+
still needs its label, so a `labeled` run there adds `spark_3_4` or nothing.
177+
`changes` hands the base branch to `compute-changes.py` as `PR_BASE_REF`.
178+
`check-ci-config.py` pins that wiring, because a dropped variable reads as an
179+
empty string and would route the pull request as if it targeted `main`.
180+
172181
### Label events
173182

174183
`ci_label.yml` fires on `pull_request.types: [labeled]` and calls `ci.yml`

‎.github/workflows/ci.yml‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,11 @@
4040
# Spark 3.4 is deprecated and sits outside all three tiers: it runs only when
4141
# a pull request carries `run-spark-3.4-tests`, or from a manual dispatch.
4242
#
43+
# Release branches (`branch-N.M`) get neither the queue nor the nightly: the
44+
# queue ruleset covers only the default branch, `push` below is main-only, and
45+
# GitHub fires `schedule` only on the default branch. So a pull request that
46+
# targets one runs all three tiers at once, still routed by the path filters.
47+
#
4348
# Which tier a job sits in is POLICY in dev/ci/compute-changes.py, not an
4449
# expression here. Heavy jobs deliberately have no `push` tier: the queue
4550
# already tested the exact commit that lands, so re-running the pipeline on
@@ -265,6 +270,7 @@ jobs:
265270
EVENT_ACTION: ${{ github.event.action }}
266271
LABEL_NAME: ${{ github.event.label.name }}
267272
PR_LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }}
273+
PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
268274
PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
269275
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
270276
MQ_BASE_SHA: ${{ github.event.merge_group.base_sha }}
@@ -345,6 +351,7 @@ jobs:
345351
#
346352
# pull request linux, full -> profiles: pr
347353
# ... with the label linux, full, all -> profiles: all
354+
# ... to branch-N.M linux, full, all -> profiles: all
348355
# `labeled` run all -> profiles: nightly
349356
# merge queue linux, full -> profiles: pr
350357
# nightly all -> profiles: nightly
@@ -400,8 +407,11 @@ jobs:
400407
docs:
401408
name: Deploy Comet site
402409
needs: changes
403-
# docs deploys to asf-site, so only run on push-to-main (or a manual dispatch).
404-
if: needs.changes.outputs.docs == 'true'
410+
# docs deploys to asf-site, so only run on push-to-main (or a manual
411+
# dispatch of main). The ref check is for dispatch: it runs every job, and
412+
# a dispatch on a release branch would otherwise publish that branch's docs
413+
# over the site.
414+
if: needs.changes.outputs.docs == 'true' && github.ref == 'refs/heads/main'
405415
uses: ./.github/workflows/docs.yaml
406416

407417
spark_3_4:

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -525,6 +525,7 @@ jobs:
525525
org.apache.comet.exec.CometAggregateSuite
526526
org.apache.comet.exec.CometExec3_4PlusSuite
527527
org.apache.comet.exec.CometExecSuite
528+
org.apache.comet.exec.CometTopKSuite
528529
org.apache.comet.exec.CometEmptyRelationExecSuite
529530
org.apache.comet.exec.CometInMemoryCacheSuite
530531
org.apache.comet.exec.CometInMemoryCacheKryoSuite

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,7 @@ jobs:
173173
org.apache.comet.exec.CometAggregateSuite
174174
org.apache.comet.exec.CometExec3_4PlusSuite
175175
org.apache.comet.exec.CometExecSuite
176+
org.apache.comet.exec.CometTopKSuite
176177
org.apache.comet.exec.CometEmptyRelationExecSuite
177178
org.apache.comet.exec.CometInMemoryCacheSuite
178179
org.apache.comet.exec.CometInMemoryCacheKryoSuite

‎dev/ci/check-ci-config.py‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,13 @@
2727
# could not be tested; POLICY_CASES below is the test it never had. The
2828
# expected sets are transcribed from the `if:` expressions ci.yml carried
2929
# before the policy moved, so a regression here is a behaviour change.
30+
# The event reaches the script as environment variables set by the
31+
# `Compute outputs` step of ci.yml's `Detect changes` job, and a variable
32+
# dropped there reads as an empty string, which is a valid value (no
33+
# label, no base branch), so that wiring is pinned as well. So is the
34+
# `refs/heads/main` guard on the `docs` job: a dispatch routes every job,
35+
# docs included, and that guard is all that keeps a dispatch on a release
36+
# branch from publishing that branch's docs over the site.
3037
#
3138
# 3. Required-check coverage. `Required Checks` in ci.yml is the job that
3239
# `.asf.yaml` can name in `required_status_checks` for main. A heavy job
@@ -390,6 +397,62 @@
390397
},
391398
set(),
392399
),
400+
# A pull request against a release branch runs the queue and nightly tiers
401+
# as well, because a release branch has no queue and no nightly to run them
402+
# later. The site deploy still never runs from a pull request, and the
403+
# deprecated Spark 3.4 suite still waits for its label.
404+
(
405+
{"name": "pull_request", "action": "opened", "labels": [], "base": "branch-1.1"},
406+
QUEUE_TIER | NIGHTLY_TIER,
407+
),
408+
(
409+
{"name": "pull_request", "action": "synchronize", "labels": [], "base": "branch-2.0"},
410+
QUEUE_TIER | NIGHTLY_TIER,
411+
),
412+
(
413+
{
414+
"name": "pull_request",
415+
"action": "synchronize",
416+
"labels": ["run-spark-3.4-tests"],
417+
"base": "branch-1.1",
418+
},
419+
QUEUE_TIER | NIGHTLY_TIER | SPARK_DEPRECATED,
420+
),
421+
# There the commit run already covered everything a label gates except
422+
# Spark 3.4, so a `labeled` run adds Spark 3.4 or nothing.
423+
(
424+
{
425+
"name": "pull_request",
426+
"action": "labeled",
427+
"label": "run-spark-3.4-tests",
428+
"labels": ["run-spark-3.4-tests"],
429+
"base": "branch-1.1",
430+
},
431+
SPARK_DEPRECATED,
432+
),
433+
(
434+
{
435+
"name": "pull_request",
436+
"action": "labeled",
437+
"label": "run-iceberg-tests",
438+
"labels": ["run-iceberg-tests"],
439+
"base": "branch-1.1",
440+
},
441+
set(),
442+
),
443+
# Only `branch-N.M` is a release branch. A pull request against main, or
444+
# one stacked on another branch in the repository, keeps the PR tier.
445+
({"name": "pull_request", "action": "synchronize", "labels": [], "base": "main"}, PR_TIER),
446+
({"name": "pull_request", "action": "synchronize", "labels": [], "base": "pr-5654"}, PR_TIER),
447+
(
448+
{
449+
"name": "pull_request",
450+
"action": "synchronize",
451+
"labels": [],
452+
"base": "branch-1.1-backports",
453+
},
454+
PR_TIER,
455+
),
393456
]
394457

395458

@@ -654,6 +717,8 @@ def check_event_policy():
654717
label += f"/{event['action']}"
655718
if event.get("label"):
656719
label += f" +{event['label']}"
720+
if event.get("base"):
721+
label += f" base={event['base']}"
657722
failures.append(
658723
f"{label} labels={event.get('labels', [])}: "
659724
f"unexpectedly allowed {sorted(actual - expected) or 'nothing'}, "
@@ -688,6 +753,98 @@ def check_event_policy():
688753
return not failures
689754

690755

756+
# The environment variables `event_from_env` in compute-changes.py reads, and
757+
# the `github` context expression ci.yml's `Compute outputs` step must set each
758+
# one from. A typo in an expression is as silent as a missing variable: both
759+
# arrive as an empty string.
760+
EVENT_ENV_SOURCES = {
761+
"EVENT_NAME": "github.event_name",
762+
"EVENT_ACTION": "github.event.action",
763+
"LABEL_NAME": "github.event.label.name",
764+
"PR_LABELS": "toJSON(github.event.pull_request.labels.*.name)",
765+
"PR_BASE_REF": "github.event.pull_request.base.ref",
766+
}
767+
ENV_READ = re.compile(r'os\.environ\.get\("([A-Z_]+)"')
768+
ENV_SET = re.compile(r"^\s+([A-Z_]+):\s+\$\{\{\s*(.+?)\s*\}\}\s*$")
769+
770+
771+
def compute_step_env():
772+
"""{variable: expression} from the `env:` of ci.yml's `Compute outputs` step."""
773+
env, in_step, in_env = {}, False, False
774+
for line in CI_WORKFLOW.read_text(encoding="utf-8").splitlines():
775+
if re.match(r"^\s+- name: Compute outputs\s*$", line):
776+
in_step = True
777+
elif in_step and not in_env:
778+
if re.match(r"^\s+- ", line):
779+
break # the next step; this one has no `env:`
780+
in_env = bool(re.match(r"^\s+env:\s*$", line))
781+
elif in_env:
782+
match = ENV_SET.match(line)
783+
if not match:
784+
break
785+
env[match.group(1)] = match.group(2)
786+
return env
787+
788+
789+
def check_event_env():
790+
"""Every variable POLICY reads is set by `Detect changes`, from the right context.
791+
792+
Without PR_BASE_REF, for instance, every pull request against a release
793+
branch would quietly fall back to the PR tier and every check would pass.
794+
"""
795+
failures = []
796+
source = Path("dev/ci/compute-changes.py").read_text(encoding="utf-8")
797+
body = source.split("\ndef event_from_env", 1)[1].split("\ndef ", 1)[0]
798+
read = set(ENV_READ.findall(body))
799+
env = compute_step_env()
800+
for name in sorted(read - set(EVENT_ENV_SOURCES)):
801+
failures.append(
802+
f"event_from_env reads {name}, which EVENT_ENV_SOURCES does not "
803+
f"list; add the `github` context expression it comes from"
804+
)
805+
for name in sorted(set(EVENT_ENV_SOURCES) - read):
806+
failures.append(f"EVENT_ENV_SOURCES lists {name}, which event_from_env no longer reads")
807+
for name in sorted(read & set(EVENT_ENV_SOURCES)):
808+
expected = EVENT_ENV_SOURCES[name]
809+
if name not in env:
810+
failures.append(
811+
f"{CI_WORKFLOW}: the `Compute outputs` step does not set {name}, "
812+
f"so compute-changes.py reads it as empty"
813+
)
814+
elif env[name] != expected:
815+
failures.append(
816+
f"{CI_WORKFLOW}: the `Compute outputs` step sets {name} from "
817+
f"`{env[name]}`, not `{expected}`"
818+
)
819+
for failure in failures:
820+
print(f"event env: {failure}")
821+
return not failures
822+
823+
824+
# The site deploy's job-level `if:` in ci.yml. It has to be on the job rather
825+
# than in POLICY, because a dispatch routes every job (see POLICY_CASES).
826+
DOCS_JOB = "docs"
827+
DOCS_MAIN_GUARD = re.compile(r"^ if:.*github\.ref\s*==\s*'refs/heads/main'")
828+
829+
830+
def check_docs_deploy_guard():
831+
"""The site deploy runs from main only, whatever the event.
832+
833+
docs.yaml rsyncs the built site over asf-site with --delete and falls back
834+
to `git push --force`, and the release process dispatches ci.yml on the
835+
release branch before every release candidate.
836+
"""
837+
_, guarded = guarded_jobs(CI_WORKFLOW, DOCS_MAIN_GUARD)
838+
if DOCS_JOB in guarded:
839+
return True
840+
print(
841+
f"docs deploy: the `{DOCS_JOB}` job in {CI_WORKFLOW} must require "
842+
f"github.ref == 'refs/heads/main' in its `if:`, or a dispatch on a "
843+
f"release branch publishes that branch's docs over the site"
844+
)
845+
return False
846+
847+
691848
def artifact_names(path):
692849
"""Return ([upload names], [download names]) for one workflow file."""
693850
uploads, downloads = [], []
@@ -1277,6 +1434,8 @@ def check_cache_save_scope():
12771434
if __name__ == "__main__":
12781435
ok = check_change_filters()
12791436
ok = check_event_policy() and ok
1437+
ok = check_event_env() and ok
1438+
ok = check_docs_deploy_guard() and ok
12801439
ok = check_spark_sql_modules() and ok
12811440
ok = check_linux_test_profiles() and ok
12821441
ok = check_artifact_names() and ok

‎dev/ci/check-suites.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ def file_to_class_name(path: Path) -> str | None:
3737
"org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite", # manual test suite (loads libhdfs, see #5023)
3838
"org.apache.comet.IcebergReadFromS3Suite", # manual test suite
3939
"org.apache.comet.cloud.s3.CometS3CredentialBridgeSuite", # manual test suite
40+
"org.apache.comet.cloud.s3.HadoopS3ACredentialProviderAdapterBridgeSuite", # manual test suite
4041
"org.apache.comet.shuffle.CelebornReflectionCompatibilitySuite", # dedicated version matrix
4142
"org.apache.spark.sql.comet.CometPlanStabilitySuite", # abstract
4243
"org.apache.spark.sql.comet.ParquetDatetimeRebaseSuite", # abstract

‎dev/ci/compute-changes.py‎

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -435,6 +435,17 @@
435435
# Adding "push" back to a test job would make every merge run it twice, once
436436
# in the queue and once after, which is the thing the queue was adopted to
437437
# avoid.
438+
#
439+
# A pull request that targets a release branch (`branch-N.M`) is the one
440+
# exception: it runs every "pr", "queue" and "nightly" job. On main the tiers
441+
# spread the suites over three events, and every change still meets all of
442+
# them. A release branch has only the pull request: the merge queue covers the
443+
# default branch alone, ci.yml runs on push only for main, and GitHub fires
444+
# `schedule` only on the default branch. A tier there would not defer a suite,
445+
# it would drop it. And the tiers exist because of main's volume, which a
446+
# release branch does not have: branch-1.0 took 28 pull requests in its first
447+
# two months. "push" jobs (the site deploy) and label-only jobs (Spark 3.4)
448+
# keep their rules on a release branch.
438449
POLICY = {
439450
# The one test job that also runs on push to main, and only because of
440451
# actions/cache scoping: a pull request can restore caches saved on its
@@ -518,16 +529,23 @@
518529
}
519530

520531

532+
# Release branches are named `branch-<major>.<minor>`, as in branch-0.17 and
533+
# branch-1.0. A pull request against one runs these tiers; see the end of the
534+
# comment above POLICY.
535+
RELEASE_BRANCH = re.compile(r"branch-\d+\.\d+")
536+
RELEASE_BRANCH_TIERS = ("pr", "queue", "nightly")
537+
538+
521539
def gating_labels(job):
522540
return [t[len("label:"):] for t in POLICY[job] if t.startswith("label:")]
523541

524542

525543
def event_allows(job, event):
526544
"""Does `event` permit `job` to run, ignoring which files changed?
527545
528-
`event` is {"name", "action", "label", "labels"}: the workflow event name,
529-
the pull_request action, the label just added on a `labeled` event, and the
530-
labels currently on the pull request.
546+
`event` is {"name", "action", "label", "labels", "base"}: the workflow event
547+
name, the pull_request action, the label just added on a `labeled` event,
548+
the labels currently on the pull request, and the branch it targets.
531549
"""
532550
tiers = POLICY[job]
533551
name = event.get("name")
@@ -542,6 +560,8 @@ def event_allows(job, event):
542560
return "nightly" in tiers
543561
if name != "pull_request":
544562
return False
563+
if RELEASE_BRANCH.fullmatch(event.get("base", "")):
564+
return release_branch_allows(job, event)
545565

546566
gates = gating_labels(job)
547567
if gates:
@@ -560,6 +580,19 @@ def event_allows(job, event):
560580
return True
561581

562582

583+
def release_branch_allows(job, event):
584+
"""event_allows for a pull request that targets a release branch."""
585+
tiers = POLICY[job]
586+
gates = gating_labels(job)
587+
automatic = any(tier in tiers for tier in RELEASE_BRANCH_TIERS)
588+
# The opened/synchronize run already ran every automatic job at this
589+
# commit, so a `labeled` run adds only a job that nothing else runs there,
590+
# which leaves the label-only Spark 3.4 suite.
591+
if event.get("action") == "labeled":
592+
return not automatic and event.get("label") in gates
593+
return automatic or any(label in event.get("labels", []) for label in gates)
594+
595+
563596
def compute(files, event):
564597
"""Return {job: bool}, folding the path filter and the event policy."""
565598
return {
@@ -575,6 +608,7 @@ def event_from_env():
575608
"action": os.environ.get("EVENT_ACTION", ""),
576609
"label": os.environ.get("LABEL_NAME", ""),
577610
"labels": json.loads(labels) if labels.strip() else [],
611+
"base": os.environ.get("PR_BASE_REF", ""),
578612
}
579613

580614

0 commit comments

Comments
 (0)