Skip to content

Commit 286dc85

Browse files
jpascucci-nvjpascucci-nv
committed
ci: temporarily pause Pixi lockfile maintenance
1 parent 0ba120d commit 286dc85

10 files changed

Lines changed: 145 additions & 1071 deletions

‎.github/workflows/ci-nightly.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ jobs:
4444

4545
- name: Run CI tools unit tests
4646
run: |
47-
python -m pip install pytest
47+
python -m pip install pytest pyyaml==6.0.3
4848
# Standalone CI tool tests; skip repo-root conftest.py (imports cuda.pathfinder).
4949
python -m pytest -v --noconftest ci/tools/tests
5050

‎.github/workflows/ci-pixi-lockfile-freshness-check.yml‎

Lines changed: 10 additions & 330 deletions
Large diffs are not rendered by default.

‎.github/workflows/ci-pixi-lockfile-refresh.yml‎

Lines changed: 10 additions & 211 deletions
Original file line numberDiff line numberDiff line change
@@ -2,18 +2,11 @@
22
#
33
# SPDX-License-Identifier: Apache-2.0
44

5-
# Scheduled (and on-demand) pixi lockfile refresh. All workspaces are updated
6-
# in one job and, when anything changes, one PR. See #2298 and #2849.
7-
5+
# Scheduled refreshes are paused. Manual dispatch reports the suspension only.
6+
# Re-enable only after the released Pixi validation described in ci/README.md.
87
name: "CI: pixi lockfile refresh"
98

109
on:
11-
schedule:
12-
# Monday 03:17 Eastern. GitHub handles the DST offset for the IANA zone,
13-
# and a non-zero minute avoids the start-of-hour window where GitHub
14-
# documents scheduled runs being delayed or dropped.
15-
- cron: "17 3 * * 1"
16-
timezone: "America/New_York"
1710
workflow_dispatch: {}
1811

1912
defaults:
@@ -25,212 +18,18 @@ permissions: {}
2518
jobs:
2619
refresh:
2720
name: pixi update (all workspaces)
28-
if: ${{ github.repository_owner == 'nvidia' }}
2921
runs-on: ubuntu-latest
30-
timeout-minutes: 195
31-
permissions:
32-
contents: write
33-
pull-requests: write
34-
concurrency:
35-
group: pixi-lock-refresh
36-
cancel-in-progress: false
22+
timeout-minutes: 5
3723
steps:
38-
- name: Checkout ${{ github.event.repository.name }}
39-
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
40-
with:
41-
# Full history and tags for package metadata, without historical
42-
# file contents. See ci/README.md for the checkout policy.
43-
fetch-depth: 0
44-
filter: blob:none
45-
ref: ${{ github.event.repository.default_branch }}
46-
47-
- name: Setup pixi
48-
id: setup-pixi
49-
uses: ./.github/actions/setup-pixi
50-
51-
# Pixi 0.73.0's update and lock-read paths can serialize the same solve
52-
# differently (#2804). Canonicalize with the same operation used by the
53-
# freshness workflow before opening the PR. If cuda_core hits the exact
54-
# samples-source pruning defect guarded there, retain its committed lock
55-
# and continue refreshing the other workspaces.
56-
- name: Update and canonicalize all lockfiles
57-
id: update
58-
env:
59-
PIXI_VERSION: ${{ steps.setup-pixi.outputs.pixi-version }}
60-
WORKSPACE_TIMEOUT: 30m
24+
- name: Report temporary suspension
6125
run: |
62-
workspaces="$(python3 ci/tools/list_pixi_workspaces.py)"
63-
workspace_rows="$(
64-
jq -ce '
65-
if type != "array" or length == 0 then
66-
error("workspace inventory must be a non-empty array")
67-
elif any(.[];
68-
type != "object"
69-
or (.manifest | type != "string")
70-
or (.lockfile | type != "string")
71-
or .manifest == ""
72-
or .lockfile == ""
73-
) then
74-
error("each workspace must have non-empty string manifest and lockfile fields")
75-
else
76-
.[]
77-
end
78-
' <<<"${workspaces}"
79-
)"
80-
81-
while IFS= read -r workspace; do
82-
manifest="$(jq -r '.manifest' <<<"${workspace}")"
83-
lockfile="$(jq -r '.lockfile' <<<"${workspace}")"
84-
echo "::group::pixi update (${manifest})"
85-
86-
workspace_status=0
87-
# The single-quoted script expands its variables in the child shell.
88-
# shellcheck disable=SC2016
89-
timeout --kill-after=1m "${WORKSPACE_TIMEOUT}" \
90-
bash --noprofile --norc -xeuo pipefail -c '
91-
manifest="$1"
92-
lockfile="$2"
93-
pixi_version="$3"
94-
95-
committed_blob="$(git hash-object "${lockfile}")"
96-
pixi update --no-install --manifest-path "${manifest}"
97-
# The detector reads this uncommitted post-update state through
98-
# git cat-file, so explicitly persist the blob in the runner-local
99-
# temporary object database.
100-
updated_blob="$(git hash-object -w "${lockfile}")"
101-
102-
check_status=0
103-
pixi lock --check --manifest-path "${manifest}" \
104-
|| check_status=$?
105-
if ((check_status != 0)); then
106-
defect_status=1
107-
if [[ "${manifest}" == "cuda_core" ]]; then
108-
defect_status=0
109-
python3 ci/tools/check_pixi_samples_source_pruning.py \
110-
--original-blob "${updated_blob}" \
111-
--lockfile "${lockfile}" \
112-
--manifest-path "${manifest}/pixi.toml" \
113-
--pixi-version "${pixi_version}" \
114-
--check-status "${check_status}" \
115-
|| defect_status=$?
116-
fi
117-
118-
if ((defect_status == 0)); then
119-
echo "::warning title=Known Pixi samples lockfile bug::Pixi ${pixi_version} removed only the nine samples local-source references after updating ${lockfile}. Keeping the committed cuda_core lockfile and continuing with the other workspaces; all other refresh failures remain strict."
120-
git cat-file blob "${committed_blob}" > "${lockfile}"
121-
echo "known-defect-skipped=true" >> "${GITHUB_OUTPUT}"
122-
exit 0
123-
elif [[ "${manifest}" == "cuda_core" ]] && ((defect_status != 1)); then
124-
echo "::error::Could not determine whether ${lockfile} matches the known Pixi samples-source pruning defect. Resolve the detector error above and rerun."
125-
exit "${defect_status}"
126-
fi
127-
128-
exit "${check_status}"
129-
fi
130-
131-
canonical_blob="$(git hash-object "${lockfile}")"
132-
verify_status=0
133-
pixi lock --check --manifest-path "${manifest}" \
134-
|| verify_status=$?
135-
verified_blob="$(git hash-object "${lockfile}")"
136-
if ((verify_status != 0)) || [[ "${canonical_blob}" != "${verified_blob}" ]]; then
137-
echo "::error::pixi ${pixi_version} did not reach a successful byte-stable check for ${lockfile} (verification exit ${verify_status}, canonical blob ${canonical_blob}, verified blob ${verified_blob}); refusing to open an unstable refresh PR."
138-
exit 1
139-
fi
140-
' bash "${manifest}" "${lockfile}" "${PIXI_VERSION}" \
141-
|| workspace_status=$?
142-
143-
if ((workspace_status == 124 || workspace_status == 137)); then
144-
echo "::error::Timed out after ${WORKSPACE_TIMEOUT} while refreshing '${manifest}'."
145-
elif ((workspace_status != 0)); then
146-
echo "::error::Failed to refresh '${manifest}' (exit ${workspace_status})."
147-
fi
148-
echo "::endgroup::"
149-
if ((workspace_status != 0)); then
150-
exit "${workspace_status}"
151-
fi
152-
done <<<"${workspace_rows}"
153-
26+
echo "::warning title=Pixi lockfile maintenance suspended::Pixi lockfile freshness enforcement and automated refreshes are temporarily suspended pending a verified released Pixi fix. Freshness was not checked."
15427
{
155-
echo "lockfiles<<EOF"
156-
jq -r '.[].lockfile' <<<"${workspaces}"
157-
echo "EOF"
158-
} >> "${GITHUB_OUTPUT}"
159-
160-
- name: Compose PR body
161-
id: pr-body
162-
env:
163-
KNOWN_DEFECT_SKIPPED: ${{ steps.update.outputs.known-defect-skipped }}
164-
PIXI_VERSION: ${{ steps.setup-pixi.outputs.pixi-version }}
165-
run: |
166-
body_path="${RUNNER_TEMP}/pixi-lock-refresh-body.md"
167-
{
168-
echo "Automated lockfile refresh using pixi" \
169-
"\`${PIXI_VERSION}\` (\`pixi update --no-install\`, followed by" \
170-
"byte-stable \`pixi lock --check\` canonicalization for each" \
171-
"refreshed workspace)."
172-
if [[ "${KNOWN_DEFECT_SKIPPED}" == "true" ]]; then
173-
echo
174-
echo "> [!NOTE]"
175-
echo "> \`cuda_core/pixi.lock\` was left unchanged because Pixi" \
176-
"\`${PIXI_VERSION}\` reproduced the exact known samples-source" \
177-
"pruning defect. Other workspaces were refreshed normally."
178-
fi
179-
echo
180-
echo "Lockfiles covered by the refresh:"
181-
python3 ci/tools/list_pixi_workspaces.py \
182-
| jq -r '.[] | "- `\(.lockfile)`"'
28+
echo "### Pixi lockfile maintenance temporarily suspended"
18329
echo
184-
echo "See #2298 and #2849."
30+
echo "Freshness enforcement and automated refreshes are temporarily suspended. This successful job reports the suspension; it does not verify manifest/lock consistency."
18531
echo
186-
echo "> [!IMPORTANT]"
187-
echo "> This PR was opened with \`GITHUB_TOKEN\`. A maintainer with write access must:"
188-
echo ">"
189-
echo "> 1. Select **Approve workflows to run** in the merge box."
190-
echo "> 2. Assign themselves to this PR. The human-generated \`assigned\` event"
191-
echo "> creates the required **PR has assignee, labels, and milestone** check"
192-
echo "> and gives the PR a clear owner."
32+
echo "Source builds and tests continue using the committed lockfiles with \`PIXI_FROZEN=true\`. Dependency changes require manual review and validation."
19333
echo
194-
echo "If the diff looks wrong, close this PR and leave the branch deleted; the"
195-
echo "next scheduled run opens a fresh one if an update is still needed."
196-
} > "${body_path}"
197-
echo "path=${body_path}" >> "${GITHUB_OUTPUT}"
198-
199-
- name: Open or update a lockfile-only PR
200-
id: cpr
201-
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
202-
with:
203-
token: ${{ github.token }}
204-
add-paths: ${{ steps.update.outputs.lockfiles }}
205-
commit-message: "chore: refresh pixi lockfiles"
206-
signoff: true
207-
sign-commits: true
208-
branch: ci/pixi-lock-refresh/all
209-
delete-branch: true
210-
title: "chore: refresh pixi lockfiles"
211-
labels: |
212-
CI/CD
213-
dependencies
214-
body-path: ${{ steps.pr-body.outputs.path }}
215-
author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
216-
committer: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
217-
218-
- name: Warn about required maintainer action
219-
if: ${{ steps.cpr.outputs.pull-request-operation == 'created' || steps.cpr.outputs.pull-request-operation == 'updated' }}
220-
run: echo "::warning::The refresh PR requires a maintainer to approve its workflow runs and assign themselves to create the required metadata check."
221-
222-
- name: Summarize
223-
run: |
224-
{
225-
echo "### pixi lock refresh (all workspaces)"
226-
echo ""
227-
echo "- Operation: \`${{ steps.cpr.outputs.pull-request-operation }}\`"
228-
if [ -n "${{ steps.cpr.outputs.pull-request-url }}" ]; then
229-
echo "- PR: ${{ steps.cpr.outputs.pull-request-url }}"
230-
else
231-
echo "- No lockfile diff; PR not opened."
232-
fi
233-
if [[ "${{ steps.update.outputs.known-defect-skipped }}" == "true" ]]; then
234-
echo "- \`cuda_core/pixi.lock\` retained its committed state because the exact known Pixi samples-source pruning defect was detected."
235-
fi
236-
} >> "$GITHUB_STEP_SUMMARY"
34+
echo "See [CI documentation](https://github.com/NVIDIA/cuda-python/blob/main/ci/README.md#temporary-pixi-lockfile-maintenance-pause) for the tradeoff and re-enable criteria."
35+
} >> "${GITHUB_STEP_SUMMARY}"

‎.github/workflows/ci-pixi-source-test.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,9 +65,9 @@ env:
6565
# Temporarily bypass Pixi's workspace-wide freshness check: the unrelated
6666
# samples environment oscillates because excluded PyPI dependencies are
6767
# still checked. Frozen mode keeps the committed solution and source builds,
68-
# including nested `pixi run` calls. The separate freshness workflow rejects
69-
# every change except the exact known samples-source pruning defect.
70-
# Restore PIXI_LOCKED after the upstream fix is released and verified.
68+
# including nested `pixi run` calls. Freshness enforcement and automated
69+
# refreshes are temporarily suspended; dependency changes need manual review.
70+
# Restore PIXI_LOCKED after the released-fix validation in ci/README.md.
7171
PIXI_FROZEN: "true"
7272

7373
jobs:

‎ci/README.md‎

Lines changed: 30 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,7 @@ For those jobs, use `fetch-depth: 0` with `filter: blob:none`. The checkout
1010
action fetches complete ancestry for all branches and tags, while the blob
1111
filter skips historical file contents. This avoids downloading the large
1212
historical `gh-pages` contents discussed in issue #2197. Git retrieves file
13-
contents as needed when checking out the requested commit or creating the
14-
freshness check's base-commit worktree.
13+
contents as needed when checking out the requested commit.
1514

1615
CI change detection uses this full-history checkout for PR mirrors, so the
1716
actual PR base branch is available when computing its merge base. Non-PR runs
@@ -25,44 +24,35 @@ The pinned checkout action's
2524
[`README`](https://github.com/actions/checkout/blob/3d3c42e5aac5ba805825da76410c181273ba90b1/README.md#fetch-all-history-for-all-tags-and-branches)
2625
documents `fetch-depth: 0` for fetching all history, branches, and tags.
2726

28-
## Pixi Lockfile Freshness
29-
30-
The freshness check verifies every committed Pixi workspace against the pinned
31-
Pixi version. For PRs, a stale lockfile is attributed to the PR when its own
32-
`pixi.toml` changed but the lockfile was left unchanged, even if the base needs
33-
the same repair. The failure asks the author to regenerate and commit the
34-
lockfile in the PR. Changes to another workspace's manifest do not affect this
35-
attribution. A manifest edit that leaves its lockfile fresh still passes.
36-
37-
## Temporary Pixi Samples Lockfile Workaround
38-
39-
Pixi 0.73.0 checks PyPI dependencies that the `samples` environment explicitly
40-
excludes, then drops its local Conda source references during a PyPI-only
41-
refresh. The next refresh restores them, so repeated lock commands do not
42-
converge. A general maintenance refresh cannot repair this defect.
43-
44-
Source-build CI temporarily uses `PIXI_FROZEN=true`: it builds the checkout's
45-
source packages using the committed dependency solution without triggering
46-
Pixi's workspace-wide freshness check.
47-
48-
The separate freshness workflow remains strict except for one exact pattern:
49-
Pixi 0.73.0 exits 1 and removes precisely the nine `samples` source references,
50-
with every other lockfile byte unchanged. The detector also verifies the
51-
expected manifest overrides, local paths, platforms, and source-record
52-
definitions. It rejects the reverse cycle, which would add missing references.
53-
For PRs, the base must produce identical original and repaired blobs, and no
54-
`pixi.toml`, `pyproject.toml`, or Pixi-version input may differ from the base.
55-
Recognized cases emit a warning and restore the committed references.
56-
57-
The refresh workflow uses the same exact detector after updating `cuda_core`.
58-
On a match, it restores the pre-update committed lockfile and continues
59-
refreshing the other workspaces. This postpones `cuda_core` dependency updates
60-
without allowing an unstable lockfile into the generated refresh PR; every
61-
unrecognized refresh failure remains fatal.
62-
63-
Remove the detector and exception, and restore `PIXI_LOCKED=true`, once a
64-
released Pixi passes repeated lock and check commands without changing either
65-
the reduced upstream reproducer or our full workspaces.
27+
## Temporary Pixi Lockfile Maintenance Pause
28+
29+
Pixi lockfile freshness enforcement and automated refreshes are temporarily
30+
suspended because of [Pixi #7215](https://github.com/prefix-dev/pixi/issues/7215)
31+
and [#7000](https://github.com/prefix-dev/pixi/issues/7000). Repeated lockfile
32+
operations can change the committed solution without converging, so a general
33+
refresh is not a reliable repair.
34+
35+
The `CI: pixi lockfile freshness check` workflow still runs on every PR with its
36+
existing `pixi lock --check (all workspaces)` check name. It succeeds with a
37+
warning annotation and job summary explicitly reporting the suspension;
38+
success does not mean freshness passed. The refresh workflow has no schedule.
39+
Manual dispatch emits the same notice without solving dependencies, rewriting
40+
lockfiles, pushing a branch, or creating a PR.
41+
42+
Manifest/lock consistency is not automatically enforced during this pause.
43+
Necessary dependency changes require manual review and validation of the
44+
affected manifests and lockfiles. Real source builds and tests continue using
45+
`PIXI_FROZEN=true`, including nested `pixi run` calls, to build the checkout's
46+
source packages against the committed dependency solution. The pinned Pixi
47+
version and committed lockfiles are unchanged.
48+
49+
Re-enable maintenance only after a released Pixi passes both upstream issue
50+
reproducers and repeated byte-stable update/lock/check cycles across all six
51+
workspaces: the repository root, `cuda_pathfinder`, `cuda_bindings`, `cuda_core`,
52+
`benchmarks/cuda_bindings`, and `benchmarks/cuda_core`. Verify both `pixi update
53+
--no-install` and `pixi lock`, followed by repeated `pixi lock --check` commands,
54+
converge to unchanged lockfile bytes. Then restore strict freshness checks,
55+
scheduled refreshes, and `PIXI_LOCKED=true` in source-build CI.
6656

6757
## Repository Customizations
6858

0 commit comments

Comments
 (0)