Repository navigation
CI: wire .pfnci test stage to GHA-built wheels - #10058
Merged
Merged
Conversation
leofang
force-pushed
the
ci/wire-pfnci-wheels
branch
3 times, most recently
from
June 30, 2026 04:10
8931387 to
ec7bc41
Compare
leofang
added a commit
to leofang/cupy
that referenced
this pull request
Jul 9, 2026
Adds a pull_request trigger + a matching guard case so pushes to a PR against main/v* auto-fire the GHA wheel build. Fork PRs have no access to secrets on pull_request events, so dispatch-flexci is gated off and treated as expected-skipped by the checks job. Purpose: let us push commits to leofang:ci/wire-pfnci-wheels to verify wheel build succeeds, then /test mini on the PR uses main's flexci.yml to dispatch FlexCI with secrets and fetch the wheels built by this trigger. REVERT this commit before merging PR cupy#10058 into main. Refs: cupy#9794, cupy#9974
leofang
added a commit
to leofang/cupy
that referenced
this pull request
Jul 18, 2026
…y#9794, cupy#9974) Folds the FlexCI dispatch (formerly flexci.yml) into ci.yml so the whole build+dispatch pipeline sits behind a single guard: - Triggers reduced to issue_comment + pull_request_target(closed). Auto-build on pull_request open/push is dropped: except for pre-flight checks (pretest.yml etc.), PR updates do not start wheel builds. - guard job: on issue_comment, requires a PR comment starting with /test from an OWNER/MEMBER and resolves the PR merge ref; on pull_request_target, requires merged && no skip-ci label and resolves the merge_commit_sha. - guard also resolves artifact_sha -- the SHA keying the wheel artifact names the test stage looks up: the PR head SHA for /test runs (recoverable on the FlexCI side as HEAD, or HEAD^2 of a merge checkout), or the merge commit SHA for post-merge runs. build-wheel.yml gains an artifact-sha input and names artifacts with it instead of github.sha, which is the default-branch tip for issue_comment events and the transient merge SHA for pull_request events -- neither reconstructible from the FlexCI side. - dispatch-flexci (lifted from flexci.yml: same dual trusted/untrusted checkout, same dispatcher invocation) now needs the build jobs, so FlexCI only starts once the artifacts it will fetch exist. The .pfnci side that consumes these artifacts lands in cupy#10058.
leofang
added a commit
to leofang/cupy
that referenced
this pull request
Jul 18, 2026
Adds a pull_request trigger + matching guard case so pushes to PRs against main/v* auto-fire the wheel-build side, letting this PR (and cupy#10058 stacked on it) validate the build pipeline before ci.yml is on the default branch. Fork PRs get no secrets on pull_request events, so dispatch-flexci is gated off and the checks job treats it as expected-skipped. REVERT this commit before merging this PR.
leofang
force-pushed
the
ci/wire-pfnci-wheels
branch
2 times, most recently
from
July 18, 2026 21:31
03800ba to
55ec468
Compare
leofang
added a commit
to leofang/cupy
that referenced
this pull request
Jul 19, 2026
…on push Two defenses against pushes racing the /test flow (third review round): - flexci.yml refuses to dispatch when the PR no longer points at the revision the completed build tested: the dispatcher resolves the PR's current head at dispatch time, so a push landing during the build window could otherwise ride the earlier /test vouch into FlexCI (and, as of cupy#10058, pair that head's FlexCI run with the older head's wheels). The untrusted config.tags.json checkout is likewise pinned to the tested revision instead of the live merge ref. Residual race: the seconds between this check and the dispatcher's own lookup -- the same window the pre-split flexci.yml always had. - A push (synchronize) now cancels any in-flight labeled build and clears a stale ci:triggered label, so the label's presence keeps meaning "the current head was vouched". The cancellation is free: synchronize runs enter the same CI-<pr>-pull_request concurrency group as the labeled build, and cancel-in-progress evicts it; the guard makes those runs no-ops otherwise. The label removal lives on the pull_request_target synchronize leg, because fork-PR pull_request runs hold a read-only GITHUB_TOKEN; that leg executes base-branch workflow code with a writable token and checks out no PR code. Label removal with GITHUB_TOKEN fires no workflow, so there is no recursion.
leofang
added a commit
to leofang/cupy
that referenced
this pull request
Jul 20, 2026
…kflows
Replaces the previous single-workflow orchestration (reviewed over five
design rounds; see PR discussion) with three workflows, each with a
one-sentence story:
ci-trigger.yml Turns /test comments from an OWNER/MEMBER into
ci:triggered label events, and keeps the label honest
on pushes/reopens.
ci.yml Builds wheels when a PR is vouched (labeled) or when a
merge lands (push); the push leg also dispatches
FlexCI in-run.
flexci.yml After a successful /test build run, dispatches FlexCI
with the recorded request.
Design bar: as safe as the old flexci.yml with the same UX -- not
safer -- except that the new wheel build stage's shared compile cache
must not be poisonable by vouched PR code, which is the single
requirement forcing /test builds into PR-scoped cache context.
/test flow: the trigger leg validates the comment (PR open, OWNER or
MEMBER, explicit target), records the raw event payload as a
dispatch-request-<pr>-<head_sha> artifact, and bounces the ci:triggered
label with a GitHub App token (GITHUB_TOKEN label events do not trigger
workflows). The labeled build runs in the PR's cache scope (writes
isolated per PR, restores fall back to base/default scope -- since
GitHub's 2026-06-26 change, untrusted default-scope contexts are cache
read-only anyway) from the event-pinned merge commit, tags its wheel
artifacts with the PR number (same-head PRs against different bases
never share wheels), and uploads a build sentinel only when every build
succeeded. flexci.yml reacts to the run's completion: sentinel =
liveness (no-op label runs can never dispatch); PR identity comes only
from trusted API metadata (pulls?head=<head repo>:<branch>); per
candidate PR it requires a request recorded by a default-branch run of
this repository (fork runs cannot forge that provenance), and verifies
the PR still points at the tested head and base before feeding the
recorded payload to flexci_dispatcher.py unchanged.
Post-merge: push events (main, v*, hotfix-*) are trusted, carry
secrets, may write this branch's cache scope, and are the dispatcher's
native input -- so build and dispatch live in one run with plain needs:
ordering, no artifacts, no relay. The guard maps the pushed commit to
its merged PR to honor skip-ci, and skips branch deletions. Push runs
have no shared concurrency group: a burst of merges must not cancel
intermediate runs, each of which carries its own dispatch.
A push to a PR (synchronize) cancels any in-flight labeled build via
the shared ci-build-<PR> concurrency group and clears the stale label;
adding an existing label fires no event, so the bounce removes it
first. All artifact uploads set overwrite so "re-run all jobs" stays
viable across immutable v4+ artifact attempts.
TEMPORARY: flexci.yml carries a legacy pull_request_target(closed)
dispatch job for bases not yet backported the new push workflow
(allowlist, currently v14); each allowlist entry is removed by the same
PR that backports the workflows to that base.
The reusable build workflow no longer receives secrets: it needs none,
and labeled runs of same-repo PRs would otherwise expose repository
secrets to PR code.
Requires (already provisioned or requested): the ci:triggered label;
GitHub App cupy-ci-trigger (Issues:R/W, webhook off) with repository
variable CI_TRIGGER_APP_ID and secret CI_TRIGGER_APP_PRIVATE_KEY;
secrets.FLEXCI_WEBHOOK_SECRET; CUPY_CI_GITHUB_TOKEN on the FlexCI side.
The .pfnci wheel fetch (PR cupy#10058) must pin artifact producers and use
the PR-tagged names; the v14 backport of these workflows lands in
lockstep with this PR's merge.
leofang
added a commit
to leofang/cupy
that referenced
this pull request
Jul 20, 2026
…kflows
Replaces the previous single-workflow orchestration (reviewed over five
design rounds; see PR discussion) with three workflows, each with a
one-sentence story:
ci-trigger.yml Turns /test comments from an OWNER/MEMBER into
ci:triggered label events, and keeps the label honest
on pushes/reopens.
ci.yml Builds wheels when a PR is vouched (labeled) or when a
merge lands (push); the push leg also dispatches
FlexCI in-run.
flexci.yml After a successful /test build run, dispatches FlexCI
with the recorded request.
Design bar: as safe as the old flexci.yml with the same UX -- not
safer -- except that the new wheel build stage's shared compile cache
must not be poisonable by vouched PR code, which is the single
requirement forcing /test builds into PR-scoped cache context.
/test flow: the trigger leg validates the comment (PR open, OWNER or
MEMBER, explicit target), records the raw event payload as a
dispatch-request-<pr>-<head_sha> artifact, and bounces the ci:triggered
label with a GitHub App token (GITHUB_TOKEN label events do not trigger
workflows). The labeled build runs in the PR's cache scope (writes
isolated per PR, restores fall back to base/default scope -- since
GitHub's 2026-06-26 change, untrusted default-scope contexts are cache
read-only anyway) from the event-pinned merge commit, tags its wheel
artifacts with the PR number (same-head PRs against different bases
never share wheels), and uploads a build sentinel only when every build
succeeded. flexci.yml reacts to the run's completion: sentinel =
liveness (no-op label runs can never dispatch); PR identity comes only
from trusted API metadata (pulls?head=<head repo>:<branch>); per
candidate PR it requires a request recorded by a default-branch run of
this repository (fork runs cannot forge that provenance), and verifies
the PR still points at the tested head and base before feeding the
recorded payload to flexci_dispatcher.py unchanged.
Post-merge: push events (main, v*, hotfix-*) are trusted, carry
secrets, may write this branch's cache scope, and are the dispatcher's
native input -- so build and dispatch live in one run with plain needs:
ordering, no artifacts, no relay. The guard maps the pushed commit to
its merged PR to honor skip-ci, and skips branch deletions. Push runs
have no shared concurrency group: a burst of merges must not cancel
intermediate runs, each of which carries its own dispatch.
A push to a PR (synchronize) cancels any in-flight labeled build via
the shared ci-build-<PR> concurrency group and clears the stale label;
adding an existing label fires no event, so the bounce removes it
first. All artifact uploads set overwrite so "re-run all jobs" stays
viable across immutable v4+ artifact attempts.
TEMPORARY: flexci.yml carries a legacy pull_request_target(closed)
dispatch job for bases not yet backported the new push workflow
(allowlist, currently v14); each allowlist entry is removed by the same
PR that backports the workflows to that base.
The reusable build workflow no longer receives secrets: it needs none,
and labeled runs of same-repo PRs would otherwise expose repository
secrets to PR code.
Requires (already provisioned or requested): the ci:triggered label;
GitHub App cupy-ci-trigger (Issues:R/W, webhook off) with repository
variable CI_TRIGGER_APP_ID and secret CI_TRIGGER_APP_PRIVATE_KEY;
secrets.FLEXCI_WEBHOOK_SECRET; CUPY_CI_GITHUB_TOKEN on the FlexCI side.
The .pfnci wheel fetch (PR cupy#10058) must pin artifact producers and use
the PR-tagged names; the v14 backport of these workflows lands in
lockstep with this PR's merge.
leofang
force-pushed
the
ci/wire-pfnci-wheels
branch
3 times, most recently
from
July 23, 2026 18:31
0d66468 to
6b96f0c
Compare
Member
Author
|
FYI, only the last commit is new; the rest comes from #10049 and it should be merged/backported first. |
Contributor
|
This pull request is now in conflicts. Could you fix it @leofang? 🙏 |
The FlexCI test pipelines (.pfnci/{linux,windows}) now pip-install the
wheel artifact produced by the GHA build stage (added in cupy#10049) instead
of source-building CuPy in every test job.
Wheel fetch (linux/tests/actions/build.sh + windows/test.ps1):
- Derives the artifact suffix ci-guard.sh emits: `pr<N>-<head sha>` for
/test-driven pull_request builds, `<sha>` post-merge. On Linux,
PULL_REQUEST is forwarded by run.sh from FlexCI; on Windows,
IsPullRequestTest + GetPullRequestNumber parse FLEXCI_BRANCH
(refs/pull/<N>/head).
- Producer pin: the wheel must come from a successful ci.yml run
(workflow file by path) at the tested SHA under the expected event
(pull_request for /test, push post-merge), via GET
/actions/workflows/ci.yml/runs. `path` and `event` are API metadata
the artifact producer can't forge -- a sibling workflow that shares
the tested head SHA cannot satisfy this filter.
- Download by immutable artifact ID via GET /actions/artifacts/<id>/zip,
not by name, so re-run-attempt artifacts can't shadow the intended
one. Windows routes the binary download through cmd.exe /c to bypass
PowerShell's UTF-16 stdout redirect (which would corrupt the zip).
- FlexCI's checkout may be the tested commit directly, or a merge of
that commit into its base -- both are tried (HEAD, then HEAD^2).
Token + tools plumbing:
- linux/run.sh forwards CUPY_CI_GITHUB_TOKEN into the test container.
Name-only --env + ${VAR+x} set-check keeps the token value out of
the `set -x` trace.
- generate.py + Linux test Dockerfiles install a pinned `gh` CLI
(v2.95.0).
CUPY_CI_GITHUB_TOKEN is provisioned FlexCI-side as a fine-grained PAT
(Actions:Read on cupy/cupy). FlexCI targets whose (CUDA, Python) tuple
is outside the wheel-build matrix fail at the fetch step by design,
until the matrix expands.
leofang
force-pushed
the
ci/wire-pfnci-wheels
branch
from
August 10, 2026 08:28
6b96f0c to
404083b
Compare
Merging main into this branch introduced the new cuda13x-cuda-python target (CUDA 13.2, CUPY_USE_CUDA_PYTHON=1). It carried no `wheel` key, so generate.py's `fetch-wheel.sh if (cuda and wheel) else build.sh` routing would have made it fetch a GHA wheel -- but the CUPY_USE_CUDA_PYTHON compile-time flag is not baked into released wheels, so it must build from source, exactly like cuda12x-cuda-python. Set `wheel: false` on it and regenerate. Regeneration also restores the gh-CLI install line on cuda13x-cuda-python.Dockerfile (main's copy of that new target predated the wheel-fetch machinery). `./generate.py --dry-run` is clean.
Member
Author
|
/test mini |
Member
Author
|
@kmaehashi this is ready for review! |
Member
Author
|
/test mini,windows |
Member
Author
|
/test force-skip |
|
The following tests were force-skipped:
|
This was referenced Aug 12, 2026
Member
Author
|
Let me add |
kmaehashi
approved these changes
Aug 17, 2026
leofang
pushed a commit
that referenced
this pull request
Aug 17, 2026
CI: wire .pfnci test stage to GHA-built wheels
leofang
added a commit
that referenced
this pull request
Aug 18, 2026
CI: PFN/FlexCI follow-ups on top of #10058
leofang
added a commit
that referenced
this pull request
Aug 18, 2026
CI: PFN/FlexCI follow-ups on top of #10058
leofang
added a commit
that referenced
this pull request
Aug 18, 2026
…wups [backport] CI: PFN/FlexCI follow-ups on top of #10058
saladpalad
pushed a commit
to saladpalad/cupy
that referenced
this pull request
Aug 24, 2026
CI: wire .pfnci test stage to GHA-built wheels
saladpalad
pushed a commit
to saladpalad/cupy
that referenced
this pull request
Aug 24, 2026
CI: PFN/FlexCI follow-ups on top of cupy#10058
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #10049 and the second half of #9794's build/test split: the PFN CI (FlexCI) test stage now pip-installs the wheel artifact produced by the GHA build stage instead of building CuPy from source — for the CUDA targets that have a matching wheel.
Which targets fetch vs. build from source
.pfnci/linux/tests/actions/build.shis unchanged — the original source build. It stays the path for ROCm (no CUDA wheel exists) and for the CUDA targets a released wheel can't represent:cuda-build(its job is to test the source build + import-without-CUDA),cuda12x-cuda-pythonandcuda13x-cuda-python(CUPY_USE_CUDA_PYTHONis a compile-time flag not baked into the wheels), andcuda-head(builds against pre-release deps)..pfnci/linux/tests/actions/fetch-wheel.sh(new) installs the GHA-built wheel and is used by the standard CUDA 12/13 targets.generate.pyroutes each target to one or the other via awheel:matrix flag (defaulted on for CUDA).Fetching the wheel
fetch-wheel.sh(Linux) andwindows/test.ps1resolve the commit under test from the checkout (HEAD, falling back toHEAD^2for merge checkouts — matching theartifact_shaci.yml's guard uses on the GHA side), rebuild the artifact namecupy-cuda{12,13}x-py<ver>-<plat>-pr<N>-<sha>(<N>fromFLEXCI_BRANCH; bare<sha>post-merge), then:GET /actions/runs/<id>: it must be a successfulci.ymlrun for the expected event at the exact SHA. The name is a free string any run could choose, so the run metadata (not the name) is what proves provenance — a same-named artifact from a sibling workflow or a failed run is rejected.pip install.The pinned
ghCLI (v2.95.0) is installed by the Linux test Dockerfiles (viagenerate.py) and bywindows/test.ps1.Keeping the token out of the logs
fetch-wheel.shvia a mounted file (never an env var), so it never lands in the container'senvdump; the script reads it, deletes the file, and unsetsGH_TOKENbefore the (PR-controlled) test scripts run. On Windows it is removed from the environment beforepip/pytest.envdumps (main-flexci.sh,-prep,-self-hosted) redact the token, andmain-flexci-stable.shdrops it before delegating to the stable branch.How to test / merge
ci.ymlon main already handles/test./test mini(etc.) on this PR exercises the full chain pre-merge in one shot: guard → wheel build (keyed on the PR head SHA) → FlexCI dispatch → artifact fetch → GPU tests.Note for reviewers
ci-trigger.yml(correcting the GH App's required permission), no workflow logic changes here — the/testorchestration, artifact naming, and build→dispatch serialization all live in CI: build cupy-cuda{12,13}x wheels on GH-hosted runners, gated behind /test #10049.Footnotes
CUPY_CI_GITHUB_TOKENis provisioned on the FlexCI side as a long-lived fine-grained PAT (Actions:Read on cupy/cupy). FlexCI only replays the GitHub webhook to the job, surfacing the ref (→ PR number) and the checked-out commit (→ SHA) — enough to rebuild the artifact name and look it up; the GHA run/artifact id isn't handed over directly. ↩