Skip to content

CI: wire .pfnci test stage to GHA-built wheels - #10058

Merged
kmaehashi merged 16 commits into
cupy:mainfrom
leofang:ci/wire-pfnci-wheels
Aug 17, 2026
Merged

kmaehashi merged 16 commits into
cupy:mainfrom
leofang:ci/wire-pfnci-wheels

Conversation

@leofang

@leofang leofang commented Jun 30, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #10049 and the second half of #9794's build/test split: the PFN CI (FlexCI) test stage now pip-installs the wheel artifact produced by the GHA build stage instead of building CuPy from source — for the CUDA targets that have a matching wheel.

Which targets fetch vs. build from source

  • .pfnci/linux/tests/actions/build.sh is unchanged — the original source build. It stays the path for ROCm (no CUDA wheel exists) and for the CUDA targets a released wheel can't represent: cuda-build (its job is to test the source build + import-without-CUDA), cuda12x-cuda-python and cuda13x-cuda-python (CUPY_USE_CUDA_PYTHON is a compile-time flag not baked into the wheels), and cuda-head (builds against pre-release deps).
  • .pfnci/linux/tests/actions/fetch-wheel.sh (new) installs the GHA-built wheel and is used by the standard CUDA 12/13 targets. generate.py routes each target to one or the other via a wheel: matrix flag (defaulted on for CUDA).

Fetching the wheel

fetch-wheel.sh (Linux) and windows/test.ps1 resolve the commit under test from the checkout (HEAD, falling back to HEAD^2 for merge checkouts — matching the artifact_sha ci.yml's guard uses on the GHA side), rebuild the artifact name cupy-cuda{12,13}x-py<ver>-<plat>-pr<N>-<sha> (<N> from FLEXCI_BRANCH; bare <sha> post-merge), then:

  1. Look the artifact up by that unique name via the repo Actions-artifacts API — one call that returns the artifact id and its producing run id.
  2. Producer-pin that run with a single GET /actions/runs/<id>: it must be a successful ci.yml run for the expected event at the exact SHA. The name is a free string any run could choose, so the run metadata (not the name) is what proves provenance — a same-named artifact from a sibling workflow or a failed run is rejected.
  3. Download by the immutable artifact id and pip install.

The pinned gh CLI (v2.95.0) is installed by the Linux test Dockerfiles (via generate.py) and by windows/test.ps1.

Keeping the token out of the logs

  • On Linux the token1 is handed to fetch-wheel.sh via a mounted file (never an env var), so it never lands in the container's env dump; the script reads it, deletes the file, and unsets GH_TOKEN before the (PR-controlled) test scripts run. On Windows it is removed from the environment before pip/pytest.
  • The host-side env dumps (main-flexci.sh, -prep, -self-hosted) redact the token, and main-flexci-stable.sh drops it before delegating to the stable branch.

How to test / merge

  1. CI: build cupy-cuda{12,13}x wheels on GH-hosted runners, gated behind /test #10049 is merged, so ci.yml on main already handles /test.
  2. /test mini (etc.) on this PR exercises the full chain pre-merge in one shot: guard → wheel build (keyed on the PR head SHA) → FlexCI dispatch → artifact fetch → GPU tests.
  3. Merge normally; the post-merge run builds wheels for the merge commit and FlexCI fetches those.

Note for reviewers

  • Aside from a one-line doc-comment fix to ci-trigger.yml (correcting the GH App's required permission), no workflow logic changes here — the /test orchestration, artifact naming, and build→dispatch serialization all live in CI: build cupy-cuda{12,13}x wheels on GH-hosted runners, gated behind /test #10049.
  • FlexCI targets whose (CUDA, Python) tuple is outside the wheel-build matrix fail at the fetch step by design, until the matrix expands.

Footnotes

  1. CUPY_CI_GITHUB_TOKEN is provisioned on the FlexCI side as a long-lived fine-grained PAT (Actions:Read on cupy/cupy). FlexCI only replays the GitHub webhook to the job, surfacing the ref (→ PR number) and the checked-out commit (→ SHA) — enough to rebuild the artifact name and look it up; the GHA run/artifact id isn't handed over directly. ↩

@leofang leofang added cat:test Test code / CI to-be-backported Pull-requests to be backported to stable branch st:blocked-by-another-pr Blocked by another pull-request prio:high labels Jun 30, 2026
@leofang
leofang force-pushed the ci/wire-pfnci-wheels branch 3 times, most recently from 8931387 to ec7bc41 Compare June 30, 2026 04:10
leofang added a commit to leofang/cupy that referenced this pull request Jul 9, 2026
Adds a pull_request trigger + a matching guard case so pushes to a PR
against main/v* auto-fire the GHA wheel build. Fork PRs have no access
to secrets on pull_request events, so dispatch-flexci is gated off and
treated as expected-skipped by the checks job.

Purpose: let us push commits to leofang:ci/wire-pfnci-wheels to verify
wheel build succeeds, then /test mini on the PR uses main's flexci.yml
to dispatch FlexCI with secrets and fetch the wheels built by this
trigger.

REVERT this commit before merging PR cupy#10058 into main.

Refs: cupy#9794, cupy#9974
leofang added a commit to leofang/cupy that referenced this pull request Jul 18, 2026
…y#9794, cupy#9974)

Folds the FlexCI dispatch (formerly flexci.yml) into ci.yml so the whole
build+dispatch pipeline sits behind a single guard:

- Triggers reduced to issue_comment + pull_request_target(closed).
  Auto-build on pull_request open/push is dropped: except for pre-flight
  checks (pretest.yml etc.), PR updates do not start wheel builds.
- guard job: on issue_comment, requires a PR comment starting with /test
  from an OWNER/MEMBER and resolves the PR merge ref; on
  pull_request_target, requires merged && no skip-ci label and resolves
  the merge_commit_sha.
- guard also resolves artifact_sha -- the SHA keying the wheel artifact
  names the test stage looks up: the PR head SHA for /test runs
  (recoverable on the FlexCI side as HEAD, or HEAD^2 of a merge
  checkout), or the merge commit SHA for post-merge runs. build-wheel.yml
  gains an artifact-sha input and names artifacts with it instead of
  github.sha, which is the default-branch tip for issue_comment events
  and the transient merge SHA for pull_request events -- neither
  reconstructible from the FlexCI side.
- dispatch-flexci (lifted from flexci.yml: same dual trusted/untrusted
  checkout, same dispatcher invocation) now needs the build jobs, so
  FlexCI only starts once the artifacts it will fetch exist.

The .pfnci side that consumes these artifacts lands in cupy#10058.
leofang added a commit to leofang/cupy that referenced this pull request Jul 18, 2026
Adds a pull_request trigger + matching guard case so pushes to PRs
against main/v* auto-fire the wheel-build side, letting this PR (and
cupy#10058 stacked on it) validate the build pipeline before ci.yml is on
the default branch. Fork PRs get no secrets on pull_request events, so
dispatch-flexci is gated off and the checks job treats it as
expected-skipped.

REVERT this commit before merging this PR.
@leofang
leofang force-pushed the ci/wire-pfnci-wheels branch 2 times, most recently from 03800ba to 55ec468 Compare July 18, 2026 21:31
@leofang leofang changed the title CI: fold FlexCI into ci.yml + wire .pfnci to GHA-built wheels CI: wire .pfnci test stage to GHA-built wheels Jul 18, 2026
leofang added a commit to leofang/cupy that referenced this pull request Jul 19, 2026
…on push

Two defenses against pushes racing the /test flow (third review round):

- flexci.yml refuses to dispatch when the PR no longer points at the
  revision the completed build tested: the dispatcher resolves the
  PR's current head at dispatch time, so a push landing during the
  build window could otherwise ride the earlier /test vouch into
  FlexCI (and, as of cupy#10058, pair that head's FlexCI run with the
  older head's wheels). The untrusted config.tags.json checkout is
  likewise pinned to the tested revision instead of the live merge
  ref. Residual race: the seconds between this check and the
  dispatcher's own lookup -- the same window the pre-split flexci.yml
  always had.

- A push (synchronize) now cancels any in-flight labeled build and
  clears a stale ci:triggered label, so the label's presence keeps
  meaning "the current head was vouched". The cancellation is free:
  synchronize runs enter the same CI-<pr>-pull_request concurrency
  group as the labeled build, and cancel-in-progress evicts it; the
  guard makes those runs no-ops otherwise. The label removal lives on
  the pull_request_target synchronize leg, because fork-PR
  pull_request runs hold a read-only GITHUB_TOKEN; that leg executes
  base-branch workflow code with a writable token and checks out no
  PR code. Label removal with GITHUB_TOKEN fires no workflow, so
  there is no recursion.
leofang added a commit to leofang/cupy that referenced this pull request Jul 20, 2026
…kflows

Replaces the previous single-workflow orchestration (reviewed over five
design rounds; see PR discussion) with three workflows, each with a
one-sentence story:

  ci-trigger.yml  Turns /test comments from an OWNER/MEMBER into
                  ci:triggered label events, and keeps the label honest
                  on pushes/reopens.
  ci.yml          Builds wheels when a PR is vouched (labeled) or when a
                  merge lands (push); the push leg also dispatches
                  FlexCI in-run.
  flexci.yml      After a successful /test build run, dispatches FlexCI
                  with the recorded request.

Design bar: as safe as the old flexci.yml with the same UX -- not
safer -- except that the new wheel build stage's shared compile cache
must not be poisonable by vouched PR code, which is the single
requirement forcing /test builds into PR-scoped cache context.

/test flow: the trigger leg validates the comment (PR open, OWNER or
MEMBER, explicit target), records the raw event payload as a
dispatch-request-<pr>-<head_sha> artifact, and bounces the ci:triggered
label with a GitHub App token (GITHUB_TOKEN label events do not trigger
workflows). The labeled build runs in the PR's cache scope (writes
isolated per PR, restores fall back to base/default scope -- since
GitHub's 2026-06-26 change, untrusted default-scope contexts are cache
read-only anyway) from the event-pinned merge commit, tags its wheel
artifacts with the PR number (same-head PRs against different bases
never share wheels), and uploads a build sentinel only when every build
succeeded. flexci.yml reacts to the run's completion: sentinel =
liveness (no-op label runs can never dispatch); PR identity comes only
from trusted API metadata (pulls?head=<head repo>:<branch>); per
candidate PR it requires a request recorded by a default-branch run of
this repository (fork runs cannot forge that provenance), and verifies
the PR still points at the tested head and base before feeding the
recorded payload to flexci_dispatcher.py unchanged.

Post-merge: push events (main, v*, hotfix-*) are trusted, carry
secrets, may write this branch's cache scope, and are the dispatcher's
native input -- so build and dispatch live in one run with plain needs:
ordering, no artifacts, no relay. The guard maps the pushed commit to
its merged PR to honor skip-ci, and skips branch deletions. Push runs
have no shared concurrency group: a burst of merges must not cancel
intermediate runs, each of which carries its own dispatch.

A push to a PR (synchronize) cancels any in-flight labeled build via
the shared ci-build-<PR> concurrency group and clears the stale label;
adding an existing label fires no event, so the bounce removes it
first. All artifact uploads set overwrite so "re-run all jobs" stays
viable across immutable v4+ artifact attempts.

TEMPORARY: flexci.yml carries a legacy pull_request_target(closed)
dispatch job for bases not yet backported the new push workflow
(allowlist, currently v14); each allowlist entry is removed by the same
PR that backports the workflows to that base.

The reusable build workflow no longer receives secrets: it needs none,
and labeled runs of same-repo PRs would otherwise expose repository
secrets to PR code.

Requires (already provisioned or requested): the ci:triggered label;
GitHub App cupy-ci-trigger (Issues:R/W, webhook off) with repository
variable CI_TRIGGER_APP_ID and secret CI_TRIGGER_APP_PRIVATE_KEY;
secrets.FLEXCI_WEBHOOK_SECRET; CUPY_CI_GITHUB_TOKEN on the FlexCI side.
The .pfnci wheel fetch (PR cupy#10058) must pin artifact producers and use
the PR-tagged names; the v14 backport of these workflows lands in
lockstep with this PR's merge.
leofang added a commit to leofang/cupy that referenced this pull request Jul 20, 2026
…kflows

Replaces the previous single-workflow orchestration (reviewed over five
design rounds; see PR discussion) with three workflows, each with a
one-sentence story:

  ci-trigger.yml  Turns /test comments from an OWNER/MEMBER into
                  ci:triggered label events, and keeps the label honest
                  on pushes/reopens.
  ci.yml          Builds wheels when a PR is vouched (labeled) or when a
                  merge lands (push); the push leg also dispatches
                  FlexCI in-run.
  flexci.yml      After a successful /test build run, dispatches FlexCI
                  with the recorded request.

Design bar: as safe as the old flexci.yml with the same UX -- not
safer -- except that the new wheel build stage's shared compile cache
must not be poisonable by vouched PR code, which is the single
requirement forcing /test builds into PR-scoped cache context.

/test flow: the trigger leg validates the comment (PR open, OWNER or
MEMBER, explicit target), records the raw event payload as a
dispatch-request-<pr>-<head_sha> artifact, and bounces the ci:triggered
label with a GitHub App token (GITHUB_TOKEN label events do not trigger
workflows). The labeled build runs in the PR's cache scope (writes
isolated per PR, restores fall back to base/default scope -- since
GitHub's 2026-06-26 change, untrusted default-scope contexts are cache
read-only anyway) from the event-pinned merge commit, tags its wheel
artifacts with the PR number (same-head PRs against different bases
never share wheels), and uploads a build sentinel only when every build
succeeded. flexci.yml reacts to the run's completion: sentinel =
liveness (no-op label runs can never dispatch); PR identity comes only
from trusted API metadata (pulls?head=<head repo>:<branch>); per
candidate PR it requires a request recorded by a default-branch run of
this repository (fork runs cannot forge that provenance), and verifies
the PR still points at the tested head and base before feeding the
recorded payload to flexci_dispatcher.py unchanged.

Post-merge: push events (main, v*, hotfix-*) are trusted, carry
secrets, may write this branch's cache scope, and are the dispatcher's
native input -- so build and dispatch live in one run with plain needs:
ordering, no artifacts, no relay. The guard maps the pushed commit to
its merged PR to honor skip-ci, and skips branch deletions. Push runs
have no shared concurrency group: a burst of merges must not cancel
intermediate runs, each of which carries its own dispatch.

A push to a PR (synchronize) cancels any in-flight labeled build via
the shared ci-build-<PR> concurrency group and clears the stale label;
adding an existing label fires no event, so the bounce removes it
first. All artifact uploads set overwrite so "re-run all jobs" stays
viable across immutable v4+ artifact attempts.

TEMPORARY: flexci.yml carries a legacy pull_request_target(closed)
dispatch job for bases not yet backported the new push workflow
(allowlist, currently v14); each allowlist entry is removed by the same
PR that backports the workflows to that base.

The reusable build workflow no longer receives secrets: it needs none,
and labeled runs of same-repo PRs would otherwise expose repository
secrets to PR code.

Requires (already provisioned or requested): the ci:triggered label;
GitHub App cupy-ci-trigger (Issues:R/W, webhook off) with repository
variable CI_TRIGGER_APP_ID and secret CI_TRIGGER_APP_PRIVATE_KEY;
secrets.FLEXCI_WEBHOOK_SECRET; CUPY_CI_GITHUB_TOKEN on the FlexCI side.
The .pfnci wheel fetch (PR cupy#10058) must pin artifact producers and use
the PR-tagged names; the v14 backport of these workflows lands in
lockstep with this PR's merge.
@leofang
leofang force-pushed the ci/wire-pfnci-wheels branch 3 times, most recently from 0d66468 to 6b96f0c Compare July 23, 2026 18:31
@leofang

leofang commented Jul 23, 2026

Copy link
Copy Markdown
Member Author

FYI, only the last commit is new; the rest comes from #10049 and it should be merged/backported first.

@mergify

mergify Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

This pull request is now in conflicts. Could you fix it @leofang? 🙏

The FlexCI test pipelines (.pfnci/{linux,windows}) now pip-install the
wheel artifact produced by the GHA build stage (added in cupy#10049) instead
of source-building CuPy in every test job.

Wheel fetch (linux/tests/actions/build.sh + windows/test.ps1):

- Derives the artifact suffix ci-guard.sh emits: `pr<N>-<head sha>` for
  /test-driven pull_request builds, `<sha>` post-merge. On Linux,
  PULL_REQUEST is forwarded by run.sh from FlexCI; on Windows,
  IsPullRequestTest + GetPullRequestNumber parse FLEXCI_BRANCH
  (refs/pull/<N>/head).

- Producer pin: the wheel must come from a successful ci.yml run
  (workflow file by path) at the tested SHA under the expected event
  (pull_request for /test, push post-merge), via GET
  /actions/workflows/ci.yml/runs. `path` and `event` are API metadata
  the artifact producer can't forge -- a sibling workflow that shares
  the tested head SHA cannot satisfy this filter.

- Download by immutable artifact ID via GET /actions/artifacts/<id>/zip,
  not by name, so re-run-attempt artifacts can't shadow the intended
  one. Windows routes the binary download through cmd.exe /c to bypass
  PowerShell's UTF-16 stdout redirect (which would corrupt the zip).

- FlexCI's checkout may be the tested commit directly, or a merge of
  that commit into its base -- both are tried (HEAD, then HEAD^2).

Token + tools plumbing:

- linux/run.sh forwards CUPY_CI_GITHUB_TOKEN into the test container.
  Name-only --env + ${VAR+x} set-check keeps the token value out of
  the `set -x` trace.

- generate.py + Linux test Dockerfiles install a pinned `gh` CLI
  (v2.95.0).

CUPY_CI_GITHUB_TOKEN is provisioned FlexCI-side as a fine-grained PAT
(Actions:Read on cupy/cupy). FlexCI targets whose (CUDA, Python) tuple
is outside the wheel-build matrix fail at the fetch step by design,
until the matrix expands.
@leofang
leofang force-pushed the ci/wire-pfnci-wheels branch from 6b96f0c to 404083b Compare August 10, 2026 08:28
Merging main into this branch introduced the new cuda13x-cuda-python target
(CUDA 13.2, CUPY_USE_CUDA_PYTHON=1). It carried no `wheel` key, so
generate.py's `fetch-wheel.sh if (cuda and wheel) else build.sh` routing
would have made it fetch a GHA wheel -- but the CUPY_USE_CUDA_PYTHON
compile-time flag is not baked into released wheels, so it must build from
source, exactly like cuda12x-cuda-python. Set `wheel: false` on it and
regenerate.

Regeneration also restores the gh-CLI install line on
cuda13x-cuda-python.Dockerfile (main's copy of that new target predated the
wheel-fetch machinery). `./generate.py --dry-run` is clean.
@leofang leofang removed the st:blocked-by-another-pr Blocked by another pull-request label Aug 10, 2026
@leofang leofang added this to the v14.2.0 milestone Aug 10, 2026
@leofang

leofang commented Aug 10, 2026

Copy link
Copy Markdown
Member Author

/test mini

@cupy-ci-trigger cupy-ci-trigger Bot added the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Aug 12, 2026
@leofang
leofang requested a review from kmaehashi August 12, 2026 02:50
@leofang

leofang commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

@kmaehashi this is ready for review!

@leofang

leofang commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

/test mini,windows

@cupy-ci-trigger cupy-ci-trigger Bot added ci:triggered Triggers the CI build pipeline (bounced automatically by /test) and removed ci:triggered Triggers the CI build pipeline (bounced automatically by /test) labels Aug 12, 2026
@leofang

leofang commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

/test force-skip

@cupy-ci-trigger cupy-ci-trigger Bot added ci:triggered Triggers the CI build pipeline (bounced automatically by /test) and removed ci:triggered Triggers the CI build pipeline (bounced automatically by /test) labels Aug 12, 2026
@github-actions

Copy link
Copy Markdown

The following tests were force-skipped:

@leofang leofang added the skip-ci Skip extensive CI after merging label Aug 17, 2026
@leofang

leofang commented Aug 17, 2026

Copy link
Copy Markdown
Member Author

Let me add skip-ci to save resources when merging to main. We'll get this tested during backport.

@kmaehashi
kmaehashi merged commit 409f2c7 into cupy:main Aug 17, 2026
164 of 165 checks passed
leofang pushed a commit that referenced this pull request Aug 17, 2026
CI: wire .pfnci test stage to GHA-built wheels
@leofang
leofang deleted the ci/wire-pfnci-wheels branch August 17, 2026 19:41
leofang added a commit that referenced this pull request Aug 18, 2026
leofang added a commit that referenced this pull request Aug 18, 2026
leofang added a commit that referenced this pull request Aug 18, 2026
…wups

[backport] CI: PFN/FlexCI follow-ups on top of #10058
@kmaehashi kmaehashi modified the milestones: v14.2.0, v15.0.0a1 Aug 18, 2026
saladpalad pushed a commit to saladpalad/cupy that referenced this pull request Aug 24, 2026
CI: wire .pfnci test stage to GHA-built wheels
saladpalad pushed a commit to saladpalad/cupy that referenced this pull request Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cat:test Test code / CI ci:triggered Triggers the CI build pipeline (bounced automatically by /test) prio:high skip-ci Skip extensive CI after merging to-be-backported Pull-requests to be backported to stable branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants