Skip to content

Commit fb8806e

Browse files
authored
CI: harden build-wheel.yml for the eMudhra/emSign CA incident (13.4.x backport) (#3305)
CI: harden build-wheel.yml for the eMudhra/emSign CA incident (13.4.x backport) Backport CVE-2026-111943 / CVE-2026-115557 hardening from main to 13.4.x: install-sccache with pinned sha256, upgrade-ca-certificates, setup-hardened-python (replaces setup-python + harden-pip; falls back to pinned pip for 3.10), cibuildwheel switched to build[uv], and fetch_ctk verifies the redistrib JSON and each archive against pinned sha256s. CTK cache keys prefixed 'verified' to force a miss on pre-verification entries. Ubuntu test container gets ca-certificates upgraded before its first HTTPS call. Driver sha256 required and checked. Refs #3195
1 parent dcc427f commit fb8806e

27 files changed

Lines changed: 1422 additions & 226 deletions

‎.github/actionlint.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,24 @@ paths:
2020
".github/workflows/release-cuda-pathfinder.yml":
2121
ignore:
2222
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
23+
".github/workflows/release.yml":
24+
ignore:
25+
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
2326
".github/workflows/test-wheel-linux.yml":
2427
ignore:
2528
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
2629
".github/workflows/test-wheel-windows.yml":
2730
ignore:
2831
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
32+
".github/workflows/build-wheel.yml":
33+
ignore:
34+
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
35+
".github/workflows/ci.yml":
36+
ignore:
37+
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
38+
".github/workflows/test-sdist-windows.yml":
39+
ignore:
40+
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
41+
".github/workflows/coverage.yml":
42+
ignore:
43+
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'

‎.github/actions/fetch_ctk/action.yml‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,11 @@ runs:
3838
--metadata-url "$CTK_JSON_URL")"
3939
4040
HASH=$(echo -n "${CTK_CACHE_COMPONENTS}" | sha256sum | awk '{print $1}')
41-
echo "CTK_CACHE_KEY=mini-ctk-${{ inputs.cuda-version }}-${{ inputs.host-platform }}-$HASH" >> $GITHUB_ENV
42-
echo "CTK_CACHE_FILENAME=mini-ctk-${{ inputs.cuda-version }}-${{ inputs.host-platform }}-$HASH.tar.gz" >> $GITHUB_ENV
41+
# "verified" prefix ensures cached entries from before hash-verification
42+
# was introduced never match; bump this prefix when the verification
43+
# logic changes in a way that old caches should not satisfy.
44+
echo "CTK_CACHE_KEY=mini-ctk-verified-${{ inputs.cuda-version }}-${{ inputs.host-platform }}-$HASH" >> $GITHUB_ENV
45+
echo "CTK_CACHE_FILENAME=mini-ctk-verified-${{ inputs.cuda-version }}-${{ inputs.host-platform }}-$HASH.tar.gz" >> $GITHUB_ENV
4346
echo "CTK_CACHE_COMPONENTS=${CTK_CACHE_COMPONENTS}" >> $GITHUB_ENV
4447
4548
- name: Install dependencies
@@ -75,6 +78,11 @@ runs:
7578
CTK_JSON_URL="$CTK_BASE_URL/redistrib_${{ inputs.cuda-version }}.json"
7679
CTK_JSON_FILE="$CACHE_TMP_DIR/redistrib.json"
7780
curl -LSs "$CTK_JSON_URL" -o "$CTK_JSON_FILE"
81+
# Pin the metadata JSON: the per-archive sha256 checks below are only
82+
# trustworthy if the JSON that carries those hashes is itself trusted.
83+
python "$CTK_REDIST_TOOL" verify-metadata \
84+
--cuda-version "${{ inputs.cuda-version }}" \
85+
--metadata-path "$CTK_JSON_FILE"
7886
if [[ "${{ inputs.host-platform }}" == linux* ]]; then
7987
function extract() {
8088
tar -xvf $1 -C $CACHE_TMP_DIR --strip-components=1
@@ -102,6 +110,13 @@ runs:
102110
CTK_COMPONENT_URL="${CTK_BASE_URL}/${CTK_COMPONENT_REL_PATH}"
103111
CTK_COMPONENT_COMPONENT_FILENAME="$(basename $CTK_COMPONENT_REL_PATH)"
104112
download $CTK_COMPONENT_URL $CTK_COMPONENT_COMPONENT_FILENAME
113+
# Verify the downloaded archive against the sha256 in the (pinned)
114+
# metadata before extracting; fail closed on any mismatch.
115+
CTK_COMPONENT_SHA256="$(python "$CTK_REDIST_TOOL" component-sha256 \
116+
--host-platform "${{ inputs.host-platform }}" \
117+
--component "$CTK_COMPONENT" \
118+
--metadata-path "$CTK_JSON_FILE")"
119+
echo "${CTK_COMPONENT_SHA256} ${CTK_COMPONENT_COMPONENT_FILENAME}" | sha256sum --check --strict -
105120
extract $CTK_COMPONENT_COMPONENT_FILENAME
106121
rm $CTK_COMPONENT_COMPONENT_FILENAME
107122
}

‎.github/actions/harden-pip/action.yml‎

Lines changed: 0 additions & 28 deletions
This file was deleted.
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
name: Install pinned rapidsai/sccache
6+
description: Download, verify, and install a pinned rapidsai/sccache binary on Linux
7+
8+
# Regenerate hashes when bumping SCCACHE_VERSION:
9+
# for arch in x86_64 aarch64; do
10+
# curl -fsSL "https://github.com/rapidsai/sccache/releases/download/<tag>/sccache-${arch}-unknown-linux-musl.tar.gz" | sha256sum
11+
# done
12+
# Cross-check against the `.sha256` files published on the GitHub release.
13+
#
14+
# Security note (eMudhra/emSign CA incident):
15+
# - Every version bump MUST repeat the embedded-roots check on the extracted
16+
# `sccache` binary:
17+
# grep -ac 'ISRG Root X1' sccache # >0 means a root list is embedded
18+
# grep -ac 'emSign ECC Root CA - G3' sccache # must be 0
19+
# Pass criteria: either no embedded root list (both return 0; binary defers
20+
# to the OS trust store via `rustls-platform-verifier`, as verified for
21+
# `rapidsai/sccache` v0.18.0-rapids.7), or an embedded `webpki-roots` list
22+
# at 1.0.10+ where `ISRG Root X1` matches and `emSign ECC Root CA - G3`
23+
# returns 0. Match the full root name: 1.0.10 removes only that emSign root
24+
# and still ships other emSign roots.
25+
# - Do NOT switch to or bump `mozilla/sccache` to v0.19.0 (or later without
26+
# repeating the roots check): `mozilla/sccache` v0.19.0 re-embeds
27+
# `webpki-roots` 1.0.5, which still trusts `emSign ECC Root CA - G3`.
28+
29+
runs:
30+
using: composite
31+
steps:
32+
- name: Install pinned rapidsai/sccache
33+
shell: bash --noprofile --norc -xeuo pipefail {0}
34+
env:
35+
SCCACHE_VERSION: v0.18.0-rapids.7
36+
run: |
37+
if [[ "$(uname -s)" != "Linux" ]]; then
38+
echo "::error::install-sccache only supports Linux (got $(uname -s))" >&2
39+
exit 1
40+
fi
41+
42+
arch="$(uname -m)"
43+
case "${arch}" in
44+
x86_64)
45+
expected_sha256="44106a2e01aeff99767e66b1a5ae6be074306c518458b367f75287de4532974c"
46+
;;
47+
aarch64)
48+
expected_sha256="1bad10208481786f2c2e943d7cca8006eaa7e620d122859ef2d61861a733d3d1"
49+
;;
50+
*)
51+
echo "::error::Unsupported architecture for install-sccache: ${arch}" >&2
52+
exit 1
53+
;;
54+
esac
55+
56+
tarball="sccache-${arch}-unknown-linux-musl.tar.gz"
57+
url="https://github.com/rapidsai/sccache/releases/download/${SCCACHE_VERSION}/${tarball}"
58+
59+
tmp_dir="$(mktemp -d)"
60+
trap 'rm -rf "${tmp_dir}"' EXIT
61+
62+
curl -fsSL -o "${tmp_dir}/${tarball}" "${url}"
63+
(
64+
cd "${tmp_dir}"
65+
echo "${expected_sha256} ${tarball}" | sha256sum --check --strict -
66+
)
67+
68+
if [[ "$(id -u)" -eq 0 ]]; then
69+
tar -C /usr/local/bin -xvzf "${tmp_dir}/${tarball}" --wildcards --strip-components=1 -x '*/sccache'
70+
else
71+
sudo tar -C /usr/local/bin -xvzf "${tmp_dir}/${tarball}" --wildcards --strip-components=1 -x '*/sccache'
72+
fi
73+
74+
echo "SCCACHE_PATH=/usr/local/bin/sccache" >> "${GITHUB_ENV}"
75+
/usr/local/bin/sccache --version
Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
name: Set up hardened Python
6+
description: >-
7+
Install CPython with a pinned uv, create a venv from it, and put the venv
8+
first on PATH. setup-uv verifies the uv binary and uv verifies the
9+
python-build-standalone archive against embedded checksums, so the
10+
interpreter's integrity does not depend on TLS. The job then fails unless
11+
the interpreter's ssl module rejects the compromised eMudhra/emSign CA keys
12+
(CVE-2026-115557; first fixed releases: 3.11.18, 3.12.16, 3.13.18, 3.14.10,
13+
3.15.2), and pip is updated through that ssl module. Unlike setup-python,
14+
this provides patched Windows builds of maintenance series, for which
15+
python.org publishes no installers. Series that have no fixed release
16+
(3.10 and older, which this branch still builds and tests) are installed
17+
with a warning instead, and their pip is replaced by a sha256-pinned
18+
wheel, so that the pip update does not depend on TLS. See
19+
https://github.com/NVIDIA/cuda-python/issues/3195 and
20+
https://discuss.python.org/t/109678.
21+
22+
inputs:
23+
python-version:
24+
description: >-
25+
uv Python request, e.g. "3.12" or "3.14t" (free-threaded). The newest
26+
patch known to the pinned uv is installed. Series with a fixed release
27+
listed in the interpreter check (3.11-3.15) must carry it. Series
28+
without a fixed release (3.10 and older) are accepted with a warning,
29+
and pip is installed from a sha256-pinned wheel.
30+
required: true
31+
venv:
32+
description: Directory for the venv. Defaults to a new directory under RUNNER_TEMP.
33+
default: ""
34+
stack-reserve:
35+
description: >-
36+
Windows only. Raise the base interpreter's main-thread stack reserve to at
37+
least this many bytes. Empty (default) leaves the interpreter unchanged.
38+
default: ""
39+
40+
outputs:
41+
python-path:
42+
description: Absolute path of the venv's interpreter.
43+
value: ${{ steps.venv.outputs.python-path }}
44+
python-version:
45+
description: Full version of the interpreter, e.g. "3.14.10".
46+
value: ${{ steps.venv.outputs.python-version }}
47+
48+
runs:
49+
using: composite
50+
steps:
51+
- name: Install uv
52+
uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
53+
with:
54+
# The newest uv whose checksum ships inside the pinned setup-uv, so
55+
# the binary is verified against the SHA-pinned action itself. The uv
56+
# release determines the newest patch of each CPython series, so
57+
# bumping the setup-uv SHA also moves the CPython patch versions.
58+
version: "latest-known"
59+
enable-cache: false
60+
# Some callers (e.g. release verification) have no checkout.
61+
ignore-empty-workdir: true
62+
63+
- name: Install Python and create venv
64+
id: venv
65+
shell: bash --noprofile --norc -euo pipefail {0}
66+
env:
67+
PY_REQUEST: ${{ inputs.python-version }}
68+
VENV_DIR: ${{ inputs.venv }}
69+
# A per-job install directory ensures the newest patch known to the
70+
# pinned uv is used, never an older interpreter left on a persistent
71+
# self-hosted runner.
72+
UV_PYTHON_INSTALL_DIR: ${{ runner.temp }}/setup-hardened-python/interpreters
73+
UV_NO_CONFIG: "1"
74+
run: |
75+
uv python install --no-bin --no-registry "${PY_REQUEST}"
76+
base="$(uv python find --managed-python --no-project "${PY_REQUEST}")"
77+
venv="${VENV_DIR}"
78+
if [[ -z "${venv}" ]]; then
79+
# A fresh directory per call, so a second call in the same job
80+
# (e.g. build-wheel.yml) leaves the first venv intact.
81+
venv="$("${base}" -c 'import sys, tempfile; print(tempfile.mkdtemp(prefix="venv-", dir=sys.argv[1]))' "${RUNNER_TEMP}")"
82+
fi
83+
# The stdlib venv seeds pip from the interpreter's bundled wheel, so
84+
# nothing is downloaded before the ssl check below.
85+
"${base}" -m venv --clear "${venv}"
86+
if [[ "${RUNNER_OS}" == Windows ]]; then
87+
py="${venv}/Scripts/python.exe"
88+
else
89+
py="${venv}/bin/python"
90+
fi
91+
# Let the interpreter report native absolute paths (Windows included).
92+
python_path="$("${py}" -c 'import sys; print(sys.executable)')"
93+
"${py}" -c 'import os, sys; print(os.path.dirname(sys.executable))' >> "${GITHUB_PATH}"
94+
{
95+
echo "python-path=${python_path}"
96+
echo "python-version=$("${py}" -c 'import platform; print(platform.python_version())')"
97+
} >> "${GITHUB_OUTPUT}"
98+
99+
- name: Check Python interpreter is patched (CVE-2026-115557)
100+
id: check
101+
# Runs the venv's python through the PATH entry added above.
102+
shell: python {0}
103+
env:
104+
PYTHON: ${{ steps.venv.outputs.python-path }}
105+
run: |
106+
import os
107+
import platform
108+
import sys
109+
110+
if os.path.normcase(sys.executable) != os.path.normcase(os.environ["PYTHON"]):
111+
print(f"::error::PATH resolves python to {sys.executable}, not the venv interpreter {os.environ['PYTHON']}.")
112+
sys.exit(1)
113+
114+
FIXED = {(3, 11): 18, (3, 12): 16, (3, 13): 18, (3, 14): 10, (3, 15): 2}
115+
# Series that will never get a fixed release (end of life before the
116+
# fix). Their pip is installed from a pinned wheel instead.
117+
UNPATCHABLE_BELOW = (3, 11)
118+
ssl_patched = "true"
119+
v = sys.version_info
120+
series = (v.major, v.minor)
121+
name = f"{v.major}.{v.minor}"
122+
full = platform.python_version()
123+
print("sys.version:", sys.version)
124+
if series in FIXED:
125+
floor = f"{name}.{FIXED[series]}"
126+
if v.micro < FIXED[series]:
127+
print(f"::error::Python {full} is older than {floor}, the first {name} release whose ssl module rejects the compromised eMudhra CA keys (CVE-2026-115557).")
128+
sys.exit(1)
129+
print(f"Python {full}: ssl module carries the fix (first fixed: {floor}).")
130+
elif series < UNPATCHABLE_BELOW:
131+
ssl_patched = "false"
132+
print(f"::warning::Python {full}: no {name} release rejects the compromised eMudhra CA keys (CVE-2026-115557); pip will be installed from a sha256-pinned wheel instead.")
133+
else:
134+
# Fail closed: a newer series is only accepted once its fixed release is listed.
135+
print(f"::error::Python {full}: no {name} release is known to reject the compromised eMudhra CA keys (CVE-2026-115557).")
136+
sys.exit(1)
137+
138+
with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as f:
139+
f.write(f"ssl-patched={ssl_patched}\n")
140+
141+
- name: Update pip
142+
shell: bash --noprofile --norc -xeuo pipefail {0}
143+
env:
144+
PYTHON: ${{ steps.venv.outputs.python-path }}
145+
SSL_PATCHED: ${{ steps.check.outputs.ssl-patched }}
146+
run: |
147+
if [[ "${SSL_PATCHED}" == "true" ]]; then
148+
"${PYTHON}" -m pip install -U pip
149+
else
150+
# The interpreter's ssl module still trusts the compromised CA keys,
151+
# so do not fetch pip through it. The sha256 fragment guarantees the
152+
# wheel's integrity regardless of the transport.
153+
"${PYTHON}" -m pip install \
154+
"pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"
155+
fi
156+
157+
- name: Raise the interpreter's stack reserve
158+
if: runner.os == 'Windows' && inputs.stack-reserve != ''
159+
shell: bash --noprofile --norc -xeuo pipefail {0}
160+
env:
161+
PYTHON: ${{ steps.venv.outputs.python-path }}
162+
STACK_RESERVE: ${{ inputs.stack-reserve }}
163+
run: |
164+
# sys.stdout.write avoids a trailing CR from Windows Python's CRLF.
165+
base="$("${PYTHON}" -c 'import sys; sys.stdout.write(sys._base_executable)')"
166+
# Windows cannot overwrite a running executable. Prepare a copy and
167+
# replace the original after the patching interpreter exits.
168+
uv run --no-project --isolated --with pefile==2024.8.26 --python "${base}" python - "${base}" "${STACK_RESERVE}" <<'EOF'
169+
import sys
170+
171+
import pefile
172+
173+
path, reserve = sys.argv[1], int(sys.argv[2])
174+
if reserve <= 0:
175+
raise ValueError("stack-reserve must be a positive number of bytes")
176+
with pefile.PE(path) as pe:
177+
pe.OPTIONAL_HEADER.SizeOfStackReserve = max(pe.OPTIONAL_HEADER.SizeOfStackReserve, reserve)
178+
pe.write(path + ".stack-reserve")
179+
EOF
180+
mv -f -- "${base}.stack-reserve" "${base}"
181+
# Verify the running interpreter's stack, rather than just its header.
182+
"${base}" - "${STACK_RESERVE}" <<'EOF'
183+
import ctypes
184+
import sys
185+
186+
lo, hi = ctypes.c_size_t(), ctypes.c_size_t()
187+
get_limits = ctypes.windll.kernel32.GetCurrentThreadStackLimits
188+
get_limits.argtypes = [ctypes.POINTER(ctypes.c_size_t), ctypes.POINTER(ctypes.c_size_t)]
189+
get_limits.restype = None
190+
get_limits(ctypes.byref(lo), ctypes.byref(hi))
191+
print("main-thread stack reserve:", hi.value - lo.value)
192+
sys.exit(hi.value - lo.value < int(sys.argv[1]))
193+
EOF

0 commit comments

Comments
 (0)