Skip to content

Commit 4fb742c

Browse files
authored
[no-ci] Fix Windows coverage test stack overflows (NVIDIA#3342)
Closes NVIDIA#3341. Windows coverage tests overflow the default stack in uv-managed Python. The existing `run_pytest_with_stack.py` enlarges only the pytest worker thread's stack; samples, subprocess tests, and multiprocessing workers start fresh interpreters with the executable's original stack reserve. Running pytest on a worker thread can also delay deferred cleanup scheduled on Python's main thread. Add an opt-in `stack-reserve` input to `setup-hardened-python`. Both Windows coverage jobs request at least 8 MiB. The action uses pinned `pefile` to prepare a patched executable, replaces the original after Python exits, and verifies the running interpreter's stack with `GetCurrentThreadStackLimits`. This consolidates stack preparation without requiring MSVC on GPU runners. Remove the custom PE parser and its tests, replace the build job's `editbin` step, and run pytest directly on the main thread. Keep `continue-on-error: true`. Validation: - Verified patching of real PE32 and PE32+ executables, including preservation of larger existing reserves. - Exercised runtime-check success and failure paths with a simulated Windows API. - YAML, Bash, embedded Python syntax, and Ruff checks passed. - Native Windows coverage CI confirmation remains pending.
1 parent 4862cc1 commit 4fb742c

3 files changed

Lines changed: 55 additions & 72 deletions

File tree

‎.github/actions/setup-hardened-python/action.yml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,11 @@ inputs:
2626
venv:
2727
description: Directory for the venv. Defaults to a new directory under RUNNER_TEMP.
2828
default: ""
29+
stack-reserve:
30+
description: >-
31+
Windows only. Raise the base interpreter's main-thread stack reserve to at
32+
least this many bytes. Empty (default) leaves the interpreter unchanged.
33+
default: ""
2934

3035
outputs:
3136
python-path:
@@ -123,3 +128,41 @@ runs:
123128
PYTHON: ${{ steps.venv.outputs.python-path }}
124129
run: |
125130
"${PYTHON}" -m pip install -U pip
131+
132+
- name: Raise the interpreter's stack reserve
133+
if: runner.os == 'Windows' && inputs.stack-reserve != ''
134+
shell: bash --noprofile --norc -xeuo pipefail {0}
135+
env:
136+
PYTHON: ${{ steps.venv.outputs.python-path }}
137+
STACK_RESERVE: ${{ inputs.stack-reserve }}
138+
run: |
139+
# sys.stdout.write avoids a trailing CR from Windows Python's CRLF.
140+
base="$("${PYTHON}" -c 'import sys; sys.stdout.write(sys._base_executable)')"
141+
# Windows cannot overwrite a running executable. Prepare a copy and
142+
# replace the original after the patching interpreter exits.
143+
uv run --no-project --isolated --with pefile==2024.8.26 --python "${base}" python - "${base}" "${STACK_RESERVE}" <<'EOF'
144+
import sys
145+
146+
import pefile
147+
148+
path, reserve = sys.argv[1], int(sys.argv[2])
149+
if reserve <= 0:
150+
raise ValueError("stack-reserve must be a positive number of bytes")
151+
with pefile.PE(path) as pe:
152+
pe.OPTIONAL_HEADER.SizeOfStackReserve = max(pe.OPTIONAL_HEADER.SizeOfStackReserve, reserve)
153+
pe.write(path + ".stack-reserve")
154+
EOF
155+
mv -f -- "${base}.stack-reserve" "${base}"
156+
# Verify the running interpreter's stack, rather than just its header.
157+
"${base}" - "${STACK_RESERVE}" <<'EOF'
158+
import ctypes
159+
import sys
160+
161+
lo, hi = ctypes.c_size_t(), ctypes.c_size_t()
162+
get_limits = ctypes.windll.kernel32.GetCurrentThreadStackLimits
163+
get_limits.argtypes = [ctypes.POINTER(ctypes.c_size_t), ctypes.POINTER(ctypes.c_size_t)]
164+
get_limits.restype = None
165+
get_limits(ctypes.byref(lo), ctypes.byref(hi))
166+
print("main-thread stack reserve:", hi.value - lo.value)
167+
sys.exit(hi.value - lo.value < int(sys.argv[1]))
168+
EOF

‎.github/workflows/coverage.yml‎

Lines changed: 12 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -242,22 +242,15 @@ jobs:
242242
submodules: true
243243

244244
- name: Set up Python
245-
uses: ./.github/actions/setup-hardened-python
245+
uses: $/.github/actions/setup-hardened-python
246246
with:
247247
python-version: ${{ env.PY_VER }}
248248
venv: .venv
249+
stack-reserve: 8388608
249250

250251
- name: Set up MSVC
251252
uses: step-security/msvc-dev-cmd@22c98154b708dbd743e6f27a933cf6ceba3305c4 # v1.13.1
252253

253-
- name: Raise the interpreter's stack reserve to 8 MiB
254-
# Importing the coverage-built cuda.bindings.driver overflows the
255-
# 2,000,000-byte main-thread stack of python-build-standalone (#3330).
256-
run: |
257-
base_py="$(.venv/Scripts/python -c 'import sys; print(sys._base_executable)')"
258-
editbin -nologo -stack:8388608 "$base_py"
259-
dumpbin -nologo -headers "$base_py" | grep -i "size of stack reserve"
260-
261254
- name: Set up mini CTK
262255
uses: ./.github/actions/fetch_ctk
263256
continue-on-error: false
@@ -381,10 +374,11 @@ jobs:
381374
nvidia-smi
382375
383376
- name: Set up Python
384-
uses: ./.github/actions/setup-hardened-python
377+
uses: $/.github/actions/setup-hardened-python
385378
with:
386379
python-version: ${{ env.PY_VER }}
387380
venv: .venv
381+
stack-reserve: 8388608
388382

389383
- name: Set up mini CTK
390384
uses: ./.github/actions/fetch_ctk
@@ -435,22 +429,22 @@ jobs:
435429
cd "${{ steps.install-root.outputs.INSTALL_ROOT }}"
436430
"$GITHUB_WORKSPACE/.venv/Scripts/pytest" -v --cov=./cuda --cov-append --cov-context=test --cov-config="$GITHUB_WORKSPACE/.coveragerc" "$GITHUB_WORKSPACE/cuda_pathfinder/tests"
437431
438-
# Cython linetrace under coverage on Windows needs more stack than the
439-
# default 1 MB thread size. The helper runs pytest on an 8 MB thread.
440-
- name: Run cuda.bindings tests (with 8MB stack)
432+
# The patched interpreter provides the larger stack on the main thread
433+
# and in child interpreters. Deferred cleanup needs pytest on the main thread.
434+
- name: Run cuda.bindings tests
441435
continue-on-error: true
442436
run: |
443-
"$GITHUB_WORKSPACE/.venv/Scripts/python" "$GITHUB_WORKSPACE/ci/tools/run_pytest_with_stack.py" \
444-
--cwd "${{ steps.install-root.outputs.INSTALL_ROOT }}" \
437+
cd "${{ steps.install-root.outputs.INSTALL_ROOT }}"
438+
"$GITHUB_WORKSPACE/.venv/Scripts/python" -m pytest \
445439
-v --cov=./cuda --cov-append --cov-context=test \
446440
--cov-config="$GITHUB_WORKSPACE/.coveragerc" \
447441
"$GITHUB_WORKSPACE/cuda_bindings_13/tests"
448442
449-
- name: Run cuda.core tests (with 8MB stack)
443+
- name: Run cuda.core tests
450444
continue-on-error: true
451445
run: |
452-
"$GITHUB_WORKSPACE/.venv/Scripts/python" "$GITHUB_WORKSPACE/ci/tools/run_pytest_with_stack.py" \
453-
--cwd "${{ steps.install-root.outputs.INSTALL_ROOT }}" \
446+
cd "${{ steps.install-root.outputs.INSTALL_ROOT }}"
447+
"$GITHUB_WORKSPACE/.venv/Scripts/python" -m pytest \
454448
-v --cov=./cuda --cov-append --cov-context=test \
455449
--cov-config="$GITHUB_WORKSPACE/.coveragerc" \
456450
"$GITHUB_WORKSPACE/cuda_core/tests"

‎ci/tools/run_pytest_with_stack.py‎

Lines changed: 0 additions & 54 deletions
This file was deleted.

0 commit comments

Comments
 (0)