Repository navigation
Failed releases don't block flutter/packages repo #194870
Description
It currently looks like the flutter/packages doesn't block when the release action fails, regardless if the commit even needed to publish to pub.dev or not.
This has been a reoccurring issue for awhile, but was rare enough that we haven't investigated or made changes to CI to block the tree because of it. Now it seems to occur at least once a day causing gaps in version releases.
Below is the error from the release action when flutter/packages#13341 failed to publish in post-submit. (I reran the job and then it published). Note that rerunning the action seems to works everytime, it just happens to fail the first time and then go unnoticed.
Output:
Matching checks have not been found yet, will check again in 30 seconds. (Limit: 60s)
Matching checks have not been found yet, will check again in 30 seconds. (Limit: 60s)
The requested check was never run against this ref, exiting...
Activity
bparrishMines commentedon Oct 20, 2026
cc @tarrinneal @Piinks
Piinks commentedon Oct 20, 2026
We should definitely not fail silently
Piinks commentedon Oct 21, 2026
I went down the rabbit hole on this to see if I could find out why this was flaking to begin with.
I found things...
TL;DR
release.ymlis failing almost daily because it gives LUCI checks ~2m45s to show up on a merged commit, while Cocoon routinely takes ~3–4 minutes to mirror the commit to Git-on-Borg and register those checks. Whenrelease.ymlfails—either from that race or from a post-submit LUCI flake—Cocoon'stree-statusstays green because it only tracks.ci.yamlLUCI tasks, not GitHub Actions. The next commit only checksHEAD~..HEAD, so the missed version is silently skipped onpub.dev.Why
release.ymlIs Failing So OftenLooking at the last 10 failed
release.ymlruns onmain, there are two failure modes:Failure Mode A: A ~3-minute race with Git-on-Borg mirroring (9 of 10 failures)
In
release.yml(L34–L52), the workflow runs ~40s of setup, sleeps60s, and then useslewagon/wait-on-check-actionto wait forLinux ci_yaml packages rollerto appear. Since#11388bumpedwait-on-check-actionto v1.6.0+ (upstreame183d72),checks-discovery-timeoutdefaults to60s—giving checks ~2m45s total to register before the step exits with(Limit: 60s).On the Cocoon side (
webhook_subscription.dartL37, L144–L152), post-submit LUCI checks can't be scheduled until the commit mirrors from GitHub to Git-on-Borg (GoB). The initial merge webhook arrives before GoB has synced, so Cocoon intentionally fails it and relies on Cloud Pub/Sub exponential backoff (up to 15 minutes, with a 6-hourvacuum-github-commitscron as fallback).In practice, Cocoon often registers all 93 LUCI checks just seconds after
release.ymlgives up:#13301(dbb784d6): Run39325214097attempt 1 failed at21:11:49Z; LUCI checks registered 3 seconds later at21:11:52Z.4a08a545: Run39667937325failed at20:12:50Z; LUCI checks registered 6 seconds later at20:12:56Z.#13347(250879b6): Run39455827439failed at ~2m47s; LUCI checks registered at +3m38s.13834275: Run39526055358missed webhook retries completely;ci_yamldidn't register until +1h43m via cron (so a longer timeout alone won't catch every case).Failure Mode B: Post-submit LUCI flakes (1 of 10 failures)
release.ymlwaits 30–85 minutes for all LUCI shards to pass (Wait on all tests). On#13243(0563606e), run38976753824failed after 1h35m becauseLinux_web web_dart_unit_test_shard_1 masterflaked two minutes after a newer commit (e01c3432) had already landed. Cocoon skips auto-retries for non-tip-of-tree commits (postsubmit_luci_subscription.dartL200–L211), and oncee01c3432passedLinux_web,tree-statuswent right back to green while0563606e's release stayed dead.Other recent failed / manually re-run
release.ymlrunsmain:39272706028,38984889236,38976353620,38943989232,38933648080,38843825442(all ~2m40s–2m51s discovery timeouts).39325214097,39463075497,39452044073,38983385077,38982309308.Why Failures Go Unnoticed & Create Version Gaps
PushBuildStatusToGithubandBuildStatusService.calculateCumulativeStatusonly check.ci.yamlLUCI tasks in Firestore. A redrelease.ymlnever turnstree-statusred.HEAD~..HEAD:release.yml(L71) runsPublishCommandwith--all-changed --base-sha=HEAD~(and tagsHEAD), so the next commit ignores any missed release before it.pub.dev:VersionAndChangelogValidatorchecks the previous version onmainin git. That's how#13341(run39464016871) was able to bump and publishin_app_purchase_storekitright over#13301's unpublished version, creating a version gap until#13301's release job was manually re-run.What About Batch-Released Packages?
cupertino_ui,go_router, andmaterial_ui(batch: true) publish weekly fromrelease-<pkg>-<version>branches viarelease_from_branches.yml→reusable_release.yml.Wait for test check-inandWait on all testsonly run wheninputs.branch-name == 'main'(reusable_release.ymlL52–L76), thoughsleep 60sat L43–L45 still runs unnecessarily.publishfails on arelease-*branch:sync_to_mainwon't run (release_from_branches.ymlL24, added in#12581), andInFlightReleaseCheckCommandwill block the next Monday's cut (like run38223606104), but nothing alerts anyone in the meantime.unblock_synccan still cause a version gap: Added in#12868,unblock_sync(batch_release_pr.ymlL94–L111) assumes any leftoverrelease-<pkg>-<version>branch already published and callsreusable_sync_to_main.ymlto openSync release-<pkg>-<version> to main—even ifpublishfailed. Merging that sync PR deletespending_changelogs/onmainand permanently skips<version>.How We Should Fix It
Step 1: Stop the daily timeout & deduplicate
release.yml(flutter/packages)checks-discovery-timeout: 900(15m, matching Cocoon's GoB sync window) inreusable_release.yml, and guardsleep 60swithif: inputs.branch-name == 'main'.release.ymlcallreusable_release.ymlinstead of maintaining two copies.Step 2: Block merges when a release fails
Unlike normal LUCI test columns,
C2's release passing doesn't fixC1's failed release—so any recent commit whose latestreleaseattempt failed needs to block merges until re-run, except commits that were reverted onmain(whose broken LUCI checks can't pass and shouldn't be published).flutter/packagesnow (no Cocoon changes needed):auto-submitalready treats the'submit-queue'commit status onflutter/packagesPRs just like'tree-status'(ci_successful.dartL17–L20, L176–L204): whenfailure, it holdsautosubmitPRs without stripping the label and respectsemergency.workflow_run(release,Batch Release) +pull_request_targetthat checks the last ~20maincommits (ignoring reverted commits) and activerelease-*branches.submit-queue: failure(linking straight to the failed run) on open PRs, and flips them back tosuccessonce re-run.flutter/cocoon:Teach
BuildStatusService.calculateCumulativeStatusto check per-commitreleasecheck-runs forflutter/packages(and addpackagesSlugtoreposWithTreeStatus). This turnstree-statusitself red and posts to#tree-statuson Discord, thoughflutter-dashboardwill also need a way to surface the GitHub Action link since gardeners can't re-run Actions from the LUCI grid.Step 3: Close the version-gap holes in repo tooling (
flutter/packages)reusable_sync_to_main.yml, check that tag<pkg>-v<version>exists onoriginbefore creating aSync release-* to mainPR, and fail loudly if it doesn't.publish-checkguard: InPublishCheckCommand(which already queriespub.devinLinux repo_checks), verify that a package's previous version (base-sha) is already onpub.devbefore allowing a new version bump—and for batch sync-back PRs (override: batch-<pkg>intomain), verify the synced version itself is onpub.dev.[ci] Increase check-discovery timeout in release workflows (#13392)
[-]Failed releases doesn't block flutter/packages repo[/-][+]Failed releases don't block flutter/packages repo[/+]