[gn] Use `python3` as `script_executable` in `engine/src/.gn` by mdebbar · Pull Request #193954 · flutter/flutter

mdebbar · 2026-10-07T16:35:15Z

Linux local_engine_builds fails on ~13.5% of postsubmit runs when a Windows sub-build (display builds|windows/*) aborts at ACTION //flutter/third_party/angle:angle_commit_id (FAILED: gen/angle/angle_commit.h). engine/src/.gn sets script_executable = "vpython3", spawning >100 concurrent vpython3.exe processes during the initial Ninja COPY/ACTION burst that collide in vpython-root.0\store while angle_commit_id walks up to flutter/third_party/angle/vpython.toml. This PR sets script_executable = "python3" in engine/src/.gn so GN rules run CIPD python3.exe directly.

The flake

Where	Failure rate	Window
Postsubmit, last 200 builds	13.5% (27/200)	Oct 1–Oct 6 on `Linux local_engine_builds`
Try, without the fix (logging-only build)	14.3% (1/7)	one wave on `65b69d1e0db`
Try, with the fix	0/64	see Proof of fix

Example postsubmit failures (FAILURE in display builds|windows/* with FAILED: gen/angle/angle_commit.h): 4146, 4125, 4113.

Root cause

engine/src/.gn sets script_executable = "vpython3" (set in buildroot PR 590 / issue 108122 so local builds resolve depot_tools Python on PATH), which GN binds to $python_path for Windows tool("copy") (tool_wrapper.py recursive-mirror) and ACTION rules.
vpython3 locates its spec by walking upward from the target script's directory. engine/src has no root .vpython3 (root_vpython3=False), and commit_id.py imports only Python 3 standard-library modules (pathlib, subprocess, sys, os), but flutter/third_party/angle/ has vpython.toml + vpython.toml.uv.lock (rolled in PR 193133, replacing ANGLE's .vpython3) declaring 80+ standalone test/infra wheels.
Under RBE (ninja -j 1000), the first ~340 Ninja steps are almost all COPY and ACTION rules, spawning >100 concurrent vpython3.exe processes (peak_vpython_procs=101) against vpython-root.0\store.
When ACTION //flutter/third_party/angle:angle_commit_id (../../flutter/third_party/angle/src/commit_id.py) runs during that >100-process burst, vpython3.exe walks up to flutter/third_party/angle/vpython.toml and intermittently fails due to internal cipkg package-reference/lock contention (inferred):

FAILED: gen/angle/angle_commit.h
C:/b/s/w/ir/cipd_bin_packages/vpython3.exe ../../flutter/third_party/angle/src/commit_id.py gen gen/angle/angle_commit.h gen/angle/angle_commit.h.d
failed to generate venv derivation: failed to reference the package: vpython_requirements+6a6r820h3nqlaqp6gjqd7r2ecc: package does not exist

Why it passes ~86.5% of the time: pass vs. fail depends on vpython3.exe concurrency and angle_commit_id scheduling relative to the initial ~335-step COPY burst. In the failing sub-build of 10993, angle_commit_id runs at step [260/10139] while peak_vpython_procs=101 (vp_samples=[101, 2, 2, 2, 2, 2, 2]). In passing sub-builds of 10993, angle_commit_id either runs at [1302/10053] after the burst drains to 4 processes (peak_vpython_procs=87, vp_samples=[87, 4, 4, 4, 4, 4]) or finishes at [335/9855] (angle_commit_h_ms=2619, peak_vpython_procs=116) without colliding.

Why it started now: unknown. The flake spans the full 200-build baseline window (27/200 = 13.5%); PR 193133 rolled ANGLE from .vpython3 to vpython.toml on Oct 2, changing the failing cipkg package name to vpython_requirements+6a6r820h3nqlaqp6gjqd7r2ecc.

Proof of root cause (before the fix)

Build	Result	What to look for
10993 (log)	failed	`[DEFLAKE_PROBE]` logs `root_vpython3=False angle_vpython3=False` and `peak_vpython_procs=101`; step `[260/10139]` fails with `vpython_requirements+6a6r820h3nqlaqp6gjqd7r2ecc: package does not exist`
10993 (log)	passed (control)	`[DEFLAKE_PROBE]` logs `peak_vpython_procs=87 vp_samples=[87, 4, 4, 4, 4, 4]`; `angle_commit_id` runs at `[1302/10053]` after the `COPY` burst drains
10993 (log)	passed (control)	`[DEFLAKE_PROBE]` logs `peak_vpython_procs=116 vp_samples=[116, 3, 2] angle_commit_h_ms=2619`; `angle_commit_id` completes at `[335/9855]`

The fix

Setting script_executable = "python3" in engine/src/.gn makes GN exec_script(), ACTION, and COPY rules invoke CIPD python3.exe directly (matching Dart's .gn and all 14 hooks in DEPS), removing vpython3.exe virtualenv derivation from Ninja build steps (vpython_procs=1, only the outer recipe runner). Trade-off: local developer builds rely on depot_tools (or a valid python3) preceding any broken system python3 stub on PATH (issue 108122); see Suggested Alternative (cpython3 in DEPS) below to preserve PATH-independent hermeticity.

Rejected alternatives

Alternative	Why not (TL;DR)	Evidence
Provision hermetic `cpython3` in `DEPS` (Suggested Alternative)	Preserves `PATH`-independent local hermeticity without `vpython3`, but requires new cross-platform `DEPS` CIPD entries; deferred to keep this fix minimal.	`precedent`: `ce8851b40f3`, `45c55079f22`, issue 108122
Add a root `engine/src/.vpython3` or `vpython.toml` spec	Does not intercept `angle_commit_id`: `vpython3` walks upward from `angle/src/commit_id.py` and hits `flutter/third_party/angle/vpython.toml` before `engine/src/`.	`constraint`: `angle/BUILD.gn`, `angle/vpython.toml`
Keep `vpython3` in `.gn` and pre-warm `angle/vpython.toml`	Forces every engine build to download/extract 80+ unused ANGLE test wheels (`opencv`, `pandas`, `grpcio`) and spawns >100 `vpython3.exe` wrappers on Windows `COPY` steps.	`history`: issue 88719, buildroot 507, buildroot 590, 10993

Details and evidence

Suggested Alternative — Provision hermetic cpython3 in DEPS (script_executable = "//third_party/cpython3/host/bin/python3"). In Aug 2021, buildroot PR 501 switched .gn to vpython3 so Fuchsia's gen_dart_package_config.py could import yaml. That caused Windows vpython cache flakes (issue 88719), so buildroot PR 507 reverted .gn to python3 and engine PR 28413 vendored third_party/pyyaml into DEPS. In July 2022, buildroot PR 590 / issue 108474 re-landed script_executable = "vpython3" in .gn not for wheel management, but so local workstations would resolve depot_tools's bundled Python even if a broken system python3 (or Windows Store stub) preceded depot_tools on PATH (issue 108122 comment). If we want to preserve that PATH-independent hermeticity without invoking vpython3 during GN/Ninja builds, we can follow Chromium (ce8851b40f3, Aug 2026) and ANGLE (45c55079f22, Sept 2026): fetch infra/3pp/tools/cpython3/${{platform}} via DEPS (as noted in issue 108122) and point script_executable at the hermetic cpython3 binary.

Add a root engine/src/.vpython3 or vpython.toml spec. vpython3 resolves specs by walking upward from the target script's directory. Because angle_commit_id runs ../../flutter/third_party/angle/src/commit_id.py, vpython3 walks up to flutter/third_party/angle/vpython.toml before ever reaching engine/src/.

Keep vpython3 in .gn and pre-warm angle/vpython.toml. Neither Chromium nor ANGLE runs GN action() scripts under vpython3 (angle/.gn uses cpython3). Keeping script_executable = "vpython3" in engine/src/.gn forces every engine build that compiles ANGLE to materialize 80+ standalone ANGLE test wheels (opencv-python, pandas, numpy, grpcio, selenium, cryptography, pywin32) from angle/vpython.toml just to run 40-line stdlib commit_id.py, while still spawning >100 concurrent vpython3.exe wrapper processes across Windows COPY steps (10993).

Proof of fix (after the fix)

Commit	Builds	Result	What to look for
`1fe39ff406e` (fix + telemetry)	11003–11037	32/32 green	`[DEFLAKE_PROBE]` in 11036 (log) logs `exe=C:\b\s\w\ir\cipd_bin_packages\cpython3\bin\python3.exe` and `vpython_procs=1`
`991dfddcce4` (this PR's diff)	11038–11069	32/32 green	no telemetry; 32/32 green within normal ~26–44m build duration

If the fix did nothing, 64/64 green would have probability (1 − 0.135)^64 ≈ 0.0093% (9.3 × 10^-5).

All 64 verification builds

11003, 11004, 11005, 11006, 11007, 11008, 11009, 11010, 11011, 11012, 11014, 11015, 11016, 11017, 11018, 11019, 11020, 11021, 11022, 11023, 11025, 11026, 11027, 11028, 11029, 11030, 11031, 11032, 11033, 11034, 11036, 11037, 11038, 11039, 11040, 11041, 11042, 11043, 11044, 11045, 11046, 11047, 11048, 11049, 11050, 11051, 11052, 11053, 11054, 11055, 11056, 11057, 11058, 11059, 11060, 11061, 11062, 11063, 11064, 11065, 11066, 11067, 11068, 11069

Investigation branch (instrumented history): https://github.com/mdebbar/flutter/tree/vivid_halo_climbs_18h12
Follow-up: Upstream LUCI vpython3.exe (cipkg) race on Windows when >100 processes concurrently access vpython-root.0\store — not filed

Fixes #193175

mdebbar · 2026-10-15T18:11:53Z

Superseded by #194454

[gn] Use python3 as script_executable in engine/src/.gn

3efc985

github-actions Bot added the engine flutter/engine related. See also e: labels. label Oct 7, 2026

mdebbar mentioned this pull request Oct 15, 2026

Use hermetic cpython3 for GN actions and roll GN to f9eca75ae554 #194454

Merged

mdebbar closed this Oct 15, 2026

Navigation Menu

Repository navigation

[gn] Use `python3` as `script_executable` in `engine/src/.gn` - #193954

[gn] Use `python3` as `script_executable` in `engine/src/.gn`#193954
mdebbar wants to merge 1 commit into
flutter:masterfrom
mdebbar:gn-script-executable-python3

mdebbar commented Oct 7, 2026 •

edited

Loading

Uh oh!

mdebbar commented Oct 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mdebbar commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The flake

Root cause

Proof of root cause (before the fix)

The fix

Rejected alternatives

Proof of fix (after the fix)

Uh oh!

mdebbar commented Oct 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mdebbar commented Oct 7, 2026 •

edited

Loading