Use hermetic `cpython3` for GN actions and roll GN to f9eca75ae554 by mdebbar · Pull Request #194454 · flutter/flutter

mdebbar · 2026-10-15T16:12:12Z

Depends on #194471

Linux local_engine_builds fails on 16% of postsubmit runs when a Windows sub-build dies at ACTION //flutter/third_party/angle:angle_commit_id with FAILED: gen/angle/angle_commit.h. engine/src/.gn runs every GN action through vpython3, and on Windows more than 100 concurrent vpython3.exe processes contend on the shared virtualenv store. This PR fetches a hermetic cpython3 through DEPS and points script_executable at it, the same way Chromium and ANGLE do. This replaces #193954, which used python3 from PATH.

The flake

Where	Failure rate	Window
Postsubmit, last 200 builds	16.0% (32/200)	`display builds\|windows/*` failing on `angle_commit.h` or `venv derivation`, Oct 3 to Oct 15
Try, without the fix (logging-only build)	1/7	one wave on `65b69d1e0db`
Try, with the fix	0/52	see Proof of fix

Example postsubmit failures (FAILURE in display builds|windows/*): 4322, 4314, 4310.

Root cause

engine/src/.gn sets script_executable = "vpython3", so every GN action() and the Windows tool("copy") run through vpython3.exe.
vpython3 finds its spec by walking up from the script. For angle/src/commit_id.py (stdlib only) it finds angle/vpython.toml, which declares 80+ test wheels.
Under RBE (ninja -j 1000), the first ~340 steps are mostly COPY/ACTION, so more than 100 vpython3.exe processes run at once against vpython-root.0\store.
When angle_commit_id runs inside that burst, cipkg fails to reference the venv package (inferred: lock/reference contention):

failed to generate venv derivation: failed to reference the package: vpython_requirements+6a6r820h3nqlaqp6gjqd7r2ecc: package does not exist

Why it passes ~84% of the time: it depends on whether angle_commit_id gets scheduled during the burst. In the failing sub-build it ran at step [260/10139] with peak_vpython_procs=101. Passing sub-builds ran it after the burst drained ([1302/10053], 4 processes) or finished it early ([335/9855]).

Why it started now: unknown. It spans the whole baseline window.

Proof of root cause (before the fix)

Build	Result	What to look for
10993 (log)	failed	`[DEFLAKE_PROBE]` logs `peak_vpython_procs=101`; step `[260/10139]` fails with `vpython_requirements+6a6r820h3nqlaqp6gjqd7r2ecc: package does not exist`
10993 (log)	passed (control)	`[DEFLAKE_PROBE]` logs `vp_samples=[87, 4, 4, 4, 4, 4]`; `angle_commit_id` runs at `[1302/10053]` after the burst drains
10993 (log)	passed (control)	`[DEFLAKE_PROBE]` logs `peak_vpython_procs=116 angle_commit_h_ms=2619`; `angle_commit_id` finishes at `[335/9855]`

The fix

This removes vpython3 from the build graph entirely. Every GN action now runs ../../flutter/third_party/cpython3/bin/python3, a plain CPython 3.11.9 from CIPD (infra/3pp/tools/cpython3, the same version Chromium pins). There's no virtualenv, no spec lookup, and no shared store. It also stays independent of whatever python3 is on a developer's PATH, which was the reason for switching to vpython3 in flutter-team-archive/buildroot#590.

Pieces that come with it:

GN roll 81b24e01531e → f9eca75ae554. The old GN treats script_executable as a literal path. Support for // paths landed in gn@4a060c55dcc5.
gn format: the new GN rewraps comments and adds blank lines in 49 .gn files under engine/src/flutter. No semantic changes.
License checker: licenses_cpp crawls every non-git DEPS directory, so I excluded flutter/third_party/cpython3.

After pulling this, you need to run gclient sync.

Rejected alternatives

Alternative	Why not (TL;DR)	Evidence
`script_executable = "python3"` (PR 193954)	Fixes the flake (64/64 green) but local builds then depend on `PATH` order again	`history`: buildroot 590, issue 108122
Roll GN to tip (Chromium's `0817afa57742`)	Turns on `check_includes_strict` by default; `gn gen --check` fails on every engine config	`constraint`: gn@526295467bd1
Add a root `engine/src/.vpython3` spec	`vpython3` hits `angle/vpython.toml` first when walking up from `commit_id.py`	`constraint`: `angle/BUILD.gn`
Keep `vpython3` and pre-warm `angle/vpython.toml`	Still spawns 100+ wrappers per build and installs 80+ unused wheels	`history`: issue 88719, buildroot 507

Details and evidence

python3 from PATH. #193954 verified 64/64 green with script_executable = "python3". But .gn moved from python3 to vpython3 in buildroot 590 so workstations wouldn't pick up a broken system python3 or the Windows Store stub ahead of depot_tools (issue 108122). A pinned cpython3 keeps that guarantee without vpython3.

GN tip. Chromium's pin includes gn@526295467bd1 "Enable check_includes_strict by default". Running flutter/tools/gn with it fails gen --check on host, Android, web, and Fuchsia configs with Public headers cannot include private dependencies and Can't include this header from here. f9eca75ae554 is its parent. Whoever rolls GN next will have to either set check_includes_strict = false or fix those includes.

Root .vpython3 / pre-warming. vpython3 resolves the nearest spec walking up from the script, and angle_commit_id lives under flutter/third_party/angle/. Pre-warming still runs every Windows COPY through vpython3.exe, and vpython cache flakes on Windows are why buildroot 507 reverted the first move to vpython3 (issue 88719).

Proof of fix (after the fix)

Commit	Builds	Result	What to look for
`d44d28f3df7` (`cpython3` + GN roll)	11360–11369	10/10 green	all `SUCCESS`
`f6c011ba36e` (this PR's diff)	11371–11424	42/42 Windows green	39 `SUCCESS`; 11376, 11378, 11389 are `wasm_*` 60m task timeouts after slow `Mount caches` (same as prod 4148)

If the fix did nothing, 52/52 green Windows sub-builds would have probability (1 − 0.16)^52 ≈ 1.2 × 10^-4.

Since script_executable only points at //flutter/third_party/cpython3/bin/python3, every green Windows sub-build ran its actions on cpython3. One build each of windows_unopt, windows_host_engine, windows_arm_host_engine, windows_host_engine_test, mac_unopt, mac_host_engine, mac_ios_engine, linux_unopt, linux_host_engine, linux_arm_host_engine, linux_web_engine, linux_fuchsia, linux_android_aot_engine, linux_clang_tidy and linux_license (among others) also passed on these commits.

All 52 verification builds

11360, 11361, 11362, 11363, 11364, 11365, 11366, 11367, 11368, 11369, 11371, 11372, 11373, 11374, 11375, 11376, 11377, 11378, 11379, 11380, 11386, 11387, 11388, 11389, 11390, 11391, 11392, 11393, 11394, 11395, 11396, 11399, 11400, 11401, 11402, 11403, 11404, 11405, 11406, 11407, 11408, 11409, 11415, 11416, 11417, 11418, 11419, 11420, 11421, 11422, 11423, 11424

Investigation branch: https://github.com/mdebbar/flutter/tree/cpython3-flutter-flake-fix-2

Fixes #193175

flutter-dashboard · 2026-10-15T16:12:24Z

It looks like this pull request may not have tests. Please make sure to add tests or get an explicit test exemption before merging.

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. If both you and the PR author believe this PR qualifies for a test exemption, escalate to the tech lead for the changed area to review and apply the test-exempt label. All reviewers should feel empowered to ask for tests before escalating.

gemini-code-assist

Code Review

This pull request updates the GN configuration to use the hermetic Python 3 binary from DEPS instead of vpython3 as the script executable, aiming to reduce virtualenv store contention under high Ninja parallelism. It also adds the hermetic Python path to the license check exclusion list. The remaining changes consist of minor formatting updates, such as comment re-wrapping and empty line insertions, across numerous GN and GNI files. There are no review comments, so no additional feedback is provided.

gaaclarke

can you pull out the formatting changes into a different PR please?

gaaclarke

lgtm!

Rolls GN from `81b24e01531e` to `f9eca75ae554` (last rolled in flutter#176119). flutter#194454 needs this to point `script_executable` at a `//` path, which landed in https://gn.googlesource.com/gn/+/4a060c55dcc5380837a8684949b74400e929b85e. The new `gn format` rewraps some comments and adds blank lines, so this PR also reformats 49 `.gn`/`.gni` files under `engine/src/flutter`. No semantic changes. I stopped at `f9eca75ae554` instead of GN tip because its child https://gn.googlesource.com/gn/+/526295467bd18f5f8d9cb11f03a1c56cb40b7b28 enables `check_includes_strict` by default, which breaks `gn gen --check` on most engine configs.

auto-submit · 2026-10-16T04:10:18Z

autosubmit label was removed for flutter/flutter/194454, because - This pull request has the missing-tests label and does not have the test-exempt label. Please add tests or obtain a test exemption from a tech lead before re-applying the autosubmit label.

eyebrowsoffire · 2026-10-16T16:38:30Z

Test exempt: only changes build files.

Rolls GN from `81b24e01531e` to `f9eca75ae554` (last rolled in flutter#176119). flutter#194454 needs this to point `script_executable` at a `//` path, which landed in https://gn.googlesource.com/gn/+/4a060c55dcc5380837a8684949b74400e929b85e. The new `gn format` rewraps some comments and adds blank lines, so this PR also reformats 49 `.gn`/`.gni` files under `engine/src/flutter`. No semantic changes. I stopped at `f9eca75ae554` instead of GN tip because its child https://gn.googlesource.com/gn/+/526295467bd18f5f8d9cb11f03a1c56cb40b7b28 enables `check_includes_strict` by default, which breaks `gn gen --check` on most engine configs.

Use hermetic cpython3 for GN actions and roll GN to f9eca75ae554

4c93706

mdebbar requested review from a team as code owners October 15, 2026 16:12

mdebbar requested review from camsim99 and removed request for a team October 15, 2026 16:12

flutter-dashboard Bot added CICD Run CI/CD missing-tests labels Oct 15, 2026

gemini-code-assist Bot reviewed Oct 15, 2026

View reviewed changes

Merge branch 'master' into gn-cpython3-script-executable

75d2231

mdebbar mentioned this pull request Oct 15, 2026

[gn] Use python3 as script_executable in engine/src/.gn #193954

Closed

mdebbar requested a review from gaaclarke October 15, 2026 18:13

gaaclarke reviewed Oct 15, 2026

View reviewed changes

Move GN roll and formatting to a separate PR

4a6cfff

mdebbar mentioned this pull request Oct 15, 2026

Roll GN to f9eca75ae554 #194471

Merged

github-actions Bot removed platform-android Android applications specifically platform-fuchsia Fuchsia code specifically labels Oct 15, 2026

gaaclarke approved these changes Oct 15, 2026

View reviewed changes

Merge branch 'master' into gn-cpython3-script-executable

eba292e

mdebbar added the autosubmit Merge PR when tree becomes green via auto submit App label Oct 16, 2026

auto-submit Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Oct 16, 2026

eyebrowsoffire added the test-exempt Exempt from tests. Overrides `missing-tests` label label Oct 16, 2026

mdebbar added the autosubmit Merge PR when tree becomes green via auto submit App label Oct 16, 2026

auto-submit Bot added this pull request to the merge queue Oct 16, 2026

Merged via the queue into flutter:master with commit 6dc3c6b Oct 16, 2026
34 checks passed

flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Oct 16, 2026

Navigation Menu

Repository navigation

Use hermetic `cpython3` for GN actions and roll GN to f9eca75ae554 - #194454

Use hermetic `cpython3` for GN actions and roll GN to f9eca75ae554#194454
auto-submit[bot] merged 4 commits into
flutter:masterfrom
mdebbar:gn-cpython3-script-executable

mdebbar commented Oct 15, 2026 •

edited

Loading

Uh oh!

flutter-dashboard Bot commented Oct 15, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gaaclarke left a comment

Uh oh!

gaaclarke left a comment

Uh oh!

auto-submit Bot commented Oct 16, 2026

Uh oh!

eyebrowsoffire commented Oct 16, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mdebbar commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The flake

Root cause

Proof of root cause (before the fix)

The fix

Rejected alternatives

Proof of fix (after the fix)

Uh oh!

flutter-dashboard Bot commented Oct 15, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gaaclarke left a comment

Choose a reason for hiding this comment

Uh oh!

gaaclarke left a comment

Choose a reason for hiding this comment

Uh oh!

auto-submit Bot commented Oct 16, 2026

Uh oh!

eyebrowsoffire commented Oct 16, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mdebbar commented Oct 15, 2026 •

edited

Loading