Skip to content

Commit f447fc3

Browse files
committed
CI: switch to update-pip and distrust compromised eMudhra CAs
Follow-up on review of NVIDIA#413 (and matching cupy/cupy#10438 + NVIDIA/cuda-python#3270 + #3275). The hash-pinned pip bootstrap moves one layer up: setup-python resolves a CPython whose ssl module blocks the compromised eMudhra/emSign CAs (CVE-2026-115557), and the composite only has to upgrade pip from there. - Rename .github/actions/harden-pip -> update-pip. New body: shell:python guard asserts sys.version_info is at or above the per-series patched floor (3.11.18, 3.12.16, 3.13.18, 3.14.10, 3.15.2), then pip install -U pip. Copied from cupy/cupy@main. - Add .github/actions/upgrade-ca-certificates composite (plus emudhra_intermediates.ps1 and impacted-intermediates.json, verbatim SHA-verified from NVIDIA/cuda-python@main). Upgrades ca-certificates on GHA-hosted Linux; distrusts 16 compromised intermediates on GHA-hosted Windows. No-op on self-hosted. - install_unix_deps: pair the apt ca-certificates install with an explicit distrust via /etc/ca-certificates.conf + update-ca-certificates. The apt install stays so a future Ubuntu 22.04 security update is picked up automatically, but today it is a no-op there and the roots remain trusted without the explicit distrust (Juengling, review of NVIDIA#413). - Every setup-python call: tilde-range floor on single-version sites (build-wheel.yml host -> ~3.14.10, build-docs.yml -> ~3.12.16), check-latest: true on matrix sites so the newest patch resolves. - build-llvm.yml: prepend the manylinux Python's bin dir to PATH so the update-pip action's shell: python picks it up. - build-wheel.yml: drop CIBW_MANYLINUX_X86_64_IMAGE and _AARCH64_IMAGE overrides so cibuildwheel 4.3.4 uses its own bundled SHA-pinned manylinux images instead of the floating quay.io tags (Juengling, review of NVIDIA#413). - publish.yaml: run upgrade-ca-certificates in each ubuntu-latest job after actions/checkout, before any PyPI or gh release upload. Windows 3.11/3.12/3.13 wheel-test lanes will fail update-pip's guard because actions/python-versions ships no eMudhra-fixed Windows build for those three series (3.11.9, 3.12.10, 3.13.17 are the ceilings); tracked upstream at actions/setup-python#1383.
1 parent 71d70eb commit f447fc3

17 files changed

Lines changed: 807 additions & 67 deletions

‎.github/actions/harden-pip/action.yml‎

Lines changed: 0 additions & 32 deletions
This file was deleted.

‎.github/actions/install_unix_deps/action.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,28 @@ runs:
4343
SUDO apt update
4444
SUDO apt install -y ca-certificates
4545
46+
# Explicitly distrust the compromised eMudhra/emSign root CAs. Keeping
47+
# the apt install above picks up a future Ubuntu security update that
48+
# removes them from the shipped bundle, but today Ubuntu 22.04 has not
49+
# published a fixed ca-certificates build, so the apt install is a
50+
# no-op there and the roots remain trusted. Prefixing the entry in
51+
# /etc/ca-certificates.conf with '!' deactivates the cert; the
52+
# grep guard makes this a no-op when the entry is already absent or
53+
# already deactivated.
54+
_ca_changed=0
55+
for cert in \
56+
"mozilla/emSign_ECC_Root_CA_-_G3.crt" \
57+
"mozilla/emSign_Root_CA_-_G1.crt"; do
58+
if grep -qxF "${cert}" /etc/ca-certificates.conf 2>/dev/null; then
59+
SUDO sed -i "s|^${cert}$|!${cert}|" /etc/ca-certificates.conf
60+
echo "Distrusted: ${cert}"
61+
_ca_changed=1
62+
fi
63+
done
64+
if [[ ${_ca_changed} -eq 1 ]]; then
65+
SUDO update-ca-certificates
66+
fi
67+
4668
not_found=0
4769
for dep in ${dependent_exes[@]}; do
4870
if ! (command -v $dep 2>&1 >/dev/null); then
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
name: Update pip
5+
description: >-
6+
Assert that the interpreter's ssl module blocks the compromised
7+
eMudhra/emSign CAs, then upgrade pip to the latest release.
8+
The assertion catches regressions if the python-version spec in the calling
9+
workflow ever drifts below the required patch. Use a tilde-range spec
10+
(e.g. python-version: '~3.13.18') so setup-python enforces the floor while
11+
automatically picking up newer security patches. Operates on the `python`
12+
found on PATH, i.e. the one installed by setup-python.
13+
See https://discuss.python.org/t/109576
14+
15+
runs:
16+
using: composite
17+
steps:
18+
- name: Assert Python blocks compromised CAs
19+
shell: python
20+
run: |
21+
import sys
22+
23+
# Fail if the interpreter predates the CPython security releases whose
24+
# ssl module rejects the compromised eMudhra/emSign keys. The same
25+
# patch floors appear in ci/versions.yml and the python-version specs
26+
# of the calling workflows; update them all together.
27+
minima = {11: 18, 12: 16, 13: 18, 14: 10, 15: 2}
28+
v = sys.version_info
29+
if v.major != 3 or v.minor not in minima or v.micro < minima[v.minor]:
30+
sys.exit(
31+
f"Python {sys.version} is below the minimum patch release "
32+
"whose ssl module blocks the compromised eMudhra/emSign CAs. "
33+
"Update the python-version spec in the calling workflow."
34+
)
35+
- name: Upgrade pip
36+
shell: bash
37+
run: |
38+
python -m pip install --disable-pip-version-check --upgrade pip
39+
python -m pip --version
Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# Temporary stopgap for the eMudhra/emSign CA key compromise (CVE-2026-111943)
6+
# on GitHub-hosted runners until GitHub ships fixed images.
7+
#
8+
# On Linux: upgrades ca-certificates via apt so the compromised emSign ECC Root
9+
# CA - G3 anchor is removed from /etc/ssl/certs/ca-certificates.crt. Ubuntu's
10+
# bundle does not include AAA Certificate Services, so the Sectigo cross-signs
11+
# do not chain to a trusted root there.
12+
# On Windows: adds the eight compromised intermediates and their eight Sectigo
13+
# cross-signs (impacted-intermediates.json) to Cert:\LocalMachine\Disallowed so
14+
# schannel-based clients (git, gh, PowerShell/.NET) reject chains through them,
15+
# including chains to AAA Certificate Services, which Windows still trusts.
16+
#
17+
# This action is a no-op on self-hosted runners (runner.environment != 'github-hosted')
18+
# because self-hosted runner operators own and update their own host images.
19+
#
20+
# Adapted from NVIDIA/cuda-python#3270.
21+
22+
name: Harden OS certificate trust (eMudhra/emSign CA stopgap)
23+
description: >
24+
Temporary per-job stopgap for CVE-2026-111943. Upgrades ca-certificates on
25+
GitHub-hosted Linux runners and adds the compromised eMudhra intermediates
26+
and cross-signs to the Disallowed store on GitHub-hosted Windows runners.
27+
No-op on self-hosted.
28+
29+
runs:
30+
using: composite
31+
steps:
32+
33+
# -------------------------------------------------------------------------
34+
# Windows path: add the compromised intermediates to LocalMachine\Disallowed.
35+
# Takes effect immediately for all schannel-based clients in subsequent steps
36+
# (git, gh, PowerShell/.NET, WinHTTP). Must precede actions/checkout.
37+
# -------------------------------------------------------------------------
38+
- name: Distrust compromised eMudhra intermediates (Windows, GitHub-hosted)
39+
if: ${{ runner.os == 'Windows' && runner.environment == 'github-hosted' }}
40+
shell: pwsh
41+
env:
42+
ACTION_PATH: ${{ github.action_path }}
43+
run: |
44+
$ErrorActionPreference = 'Stop'
45+
. (Join-Path $env:ACTION_PATH 'emudhra_intermediates.ps1')
46+
47+
$certs = @(Get-EmudhraIntermediates)
48+
try {
49+
# Adding is idempotent, so no early exit when the image is already fixed.
50+
$store = [System.Security.Cryptography.X509Certificates.X509Store]::new('Disallowed', 'LocalMachine')
51+
$store.Open('ReadWrite')
52+
try {
53+
foreach ($cert in $certs) { $store.Add($cert) }
54+
} finally {
55+
$store.Close()
56+
}
57+
58+
$notDistrusted = @($certs | Where-Object { -not (Test-ExplicitlyDistrusted $_) } | ForEach-Object { $_.EmudhraLabel })
59+
if ($notDistrusted.Count -gt 0) {
60+
Write-Error "Not explicitly distrusted after adding to LocalMachine\Disallowed: $($notDistrusted -join '; ')"
61+
exit 1
62+
}
63+
Write-Host "Distrusted $($certs.Count) compromised eMudhra intermediates and cross-signs."
64+
} finally {
65+
$certs | ForEach-Object { $_.Dispose() }
66+
}
67+
68+
# -------------------------------------------------------------------------
69+
# Linux path: upgrade ca-certificates via apt.
70+
# The Ubuntu archive GPG check does not depend on the trust store being
71+
# fixed, so this bootstrap is safe. Retries guard against transient mirror
72+
# failures. Must precede actions/checkout.
73+
# -------------------------------------------------------------------------
74+
- name: Upgrade ca-certificates (Linux, GitHub-hosted)
75+
if: ${{ runner.os == 'Linux' && runner.environment == 'github-hosted' }}
76+
shell: bash
77+
run: |
78+
set -euo pipefail
79+
EMSIGN_G3_FP="86:A1:EC:BA:08:9C:4A:8D:3B:BE:27:34:C6:12:BA:34:1D:81:3E:04:3C:F9:E8:A8:62:CD:5C:57:A3:6B:BE:6B"
80+
CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt"
81+
VERSION_ID="$(. /etc/os-release && echo "${VERSION_ID}")"
82+
83+
# Fail closed if openssl is missing or the bundle cannot be parsed.
84+
if ! command -v openssl >/dev/null 2>&1; then
85+
echo "::error::openssl not found; cannot verify CA bundle"
86+
exit 1
87+
fi
88+
if [[ ! -f "${CA_BUNDLE}" ]]; then
89+
echo "::error::CA bundle not found: ${CA_BUNDLE}"
90+
exit 1
91+
fi
92+
93+
fingerprints_in_bundle() {
94+
awk '/-----BEGIN CERTIFICATE-----/{cert=""} {cert=cert $0 "\n"}
95+
/-----END CERTIFICATE-----/{print cert | "openssl x509 -noout -fingerprint -sha256 2>/dev/null"; close("openssl x509 -noout -fingerprint -sha256 2>/dev/null")}' \
96+
"${CA_BUNDLE}"
97+
}
98+
99+
fps="$(fingerprints_in_bundle)"
100+
cert_count="$(grep -c . <<< "${fps}" || true)"
101+
if (( cert_count == 0 )); then
102+
echo "::error::No certificates parsed from ${CA_BUNDLE}; openssl may have failed"
103+
exit 1
104+
fi
105+
106+
if grep -qiF "${EMSIGN_G3_FP}" <<< "${fps}"; then
107+
before_state="not fixed"
108+
else
109+
before_state="already fixed"
110+
fi
111+
before_ver="$(dpkg-query -W -f='${Version}' ca-certificates 2>/dev/null || echo unknown)"
112+
echo "::notice::emSign ECC Root CA - G3 on ${VERSION_ID}: ${before_state} (ca-certificates ${before_ver})"
113+
114+
if [ "${before_state}" = "already fixed" ]; then
115+
exit 0
116+
fi
117+
118+
for attempt in 1 2 3; do
119+
if sudo apt-get update -q && sudo apt-get install --only-upgrade -y ca-certificates; then
120+
break
121+
fi
122+
if [ "${attempt}" -lt 3 ]; then
123+
echo "apt-get attempt ${attempt} failed; retrying in 10s..."
124+
sleep 10
125+
else
126+
echo "::error::apt-get upgrade failed after ${attempt} attempts"
127+
exit 1
128+
fi
129+
done
130+
131+
after_ver="$(dpkg-query -W -f='${Version}' ca-certificates 2>/dev/null || echo unknown)"
132+
after_fps="$(fingerprints_in_bundle)"
133+
after_count="$(grep -c . <<< "${after_fps}" || true)"
134+
if (( after_count == 0 )); then
135+
echo "::error::No certificates parsed after upgrade; cannot verify fix"
136+
exit 1
137+
fi
138+
if grep -qiF "${EMSIGN_G3_FP}" <<< "${after_fps}"; then
139+
echo "::error::emSign ECC Root CA - G3 still present after upgrade (ca-certificates ${after_ver} on ${VERSION_ID})"
140+
exit 1
141+
fi
142+
echo "ca-certificates upgraded to ${after_ver}; emSign ECC Root CA - G3 absent."
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# Helpers for the compromised eMudhra intermediates and their Sectigo
6+
# cross-signs listed in impacted-intermediates.json. Dot-sourced by this
7+
# action's Windows step.
8+
#
9+
# impacted-intermediates.json is the disclosure's file, unmodified (SHA-256
10+
# f67b3abe9b4879f4e91874ed041beccd71257f1db41c588c724ccf12aa90c59e). The set
11+
# is closed: new certificates for these keys would have to be issued by
12+
# another CA, and all existing ones are revoked.
13+
14+
$EmudhraIntermediatesPath = Join-Path $PSScriptRoot 'impacted-intermediates.json'
15+
$EmudhraIntermediatesCount = 16
16+
17+
# Returns the certificates as X509Certificate2 objects; callers dispose them.
18+
function Get-EmudhraIntermediates {
19+
$entries = @(Get-Content -LiteralPath $EmudhraIntermediatesPath -Raw | ConvertFrom-Json)
20+
if ($entries.Count -ne $EmudhraIntermediatesCount) {
21+
throw "Expected $EmudhraIntermediatesCount certificates in $EmudhraIntermediatesPath, found $($entries.Count)."
22+
}
23+
foreach ($entry in $entries) {
24+
$cert = [System.Security.Cryptography.X509Certificates.X509Certificate2]::CreateFromPem($entry.pem)
25+
if ($cert.GetCertHashString('SHA256') -ne $entry.sha256.ToUpperInvariant()) {
26+
$cert.Dispose()
27+
throw "SHA-256 mismatch for $($entry.name) (issued by $($entry.issued_by))."
28+
}
29+
Add-Member -InputObject $cert -NotePropertyName EmudhraLabel -NotePropertyValue "$($entry.name) (issued by $($entry.issued_by))"
30+
$cert
31+
}
32+
}
33+
34+
# True if Windows chain building reports the certificate as explicitly
35+
# distrusted. This covers both a local LocalMachine\Disallowed entry and
36+
# Microsoft's automatically updated disallowed list. Revocation checking and
37+
# AIA downloads are off so the result reflects the local trust state only.
38+
function Test-ExplicitlyDistrusted([System.Security.Cryptography.X509Certificates.X509Certificate2]$Cert) {
39+
$explicitDistrust = [System.Security.Cryptography.X509Certificates.X509ChainStatusFlags]::ExplicitDistrust
40+
$chain = [System.Security.Cryptography.X509Certificates.X509Chain]::new()
41+
try {
42+
$chain.ChainPolicy.RevocationMode = [System.Security.Cryptography.X509Certificates.X509RevocationMode]::NoCheck
43+
$chain.ChainPolicy.DisableCertificateDownloads = $true
44+
[void]$chain.Build($Cert)
45+
$statuses = @($chain.ChainStatus) + @($chain.ChainElements | ForEach-Object { $_.ChainElementStatus })
46+
return [bool]($statuses | Where-Object { $_.Status.HasFlag($explicitDistrust) })
47+
} finally {
48+
$chain.Dispose()
49+
}
50+
}

0 commit comments

Comments
 (0)