[CI] - Automate Go toolchain updates #3013

Activity

1. Grant McCloskey (MushuEE)

   changed the title [-]Automate Go toolchain updates[/-] [+][CI] - Automate Go toolchain updates[/+] on Oct 20, 2026
2. MushuEE commented on Oct 20, 2026

   

   Grant McCloskey (MushuEE)

   on Oct 20, 2026

   CollaboratorAuthor

   More actions

   .go-version only moves the version into one file. It doesn't fix the real problem (nothing bumps Go), and it costs:

   1. The go command ignores it. Local builds, go run, and IDEs lose the automatic "at least 1.27.3" that the toolchain lines give for free.
   2. Renovate needs a custom rule for it. If that rule ever stops matching, bumps silently stop.
   3. It doesn't make release builds exact. An exact pin comes from setting GOTOOLCHAIN in CI and publish.sh, and that works the same with the toolchain lines.

   The 15 toolchain lines aren't kept in sync today. Probably want a script as presubmit check here.
3. MushuEE commented on Oct 20, 2026

   

   Grant McCloskey (MushuEE)

   on Oct 20, 2026

   CollaboratorAuthor

   More actions

   Proposal

   Decision: keep the toolchain directive that #2984 added, and don't switch to a .go-version file.

   • The go command reads toolchain, so local builds, go run, and IDEs get at least the pinned version with no setup. Nothing reads .go-version except CI tooling.
   • Renovate supports toolchain natively, whereas .go-version would need a custom regex rule that fails silently if it stops matching.
   • Neither option makes builds exact by itself. An exact pin comes from setting GOTOOLCHAIN in the build, and that works the same with either.

   Correction to the description: there are 15 modules, not 16. The 16th go.mod is a third-party copy under _LICENSES/. Today nothing keeps their toolchain lines in sync; they match because #2984 edited them all by hand.

   Phase 1: a verify check that the pins agree (no approvals needed)

   Add a script under hack/verify/, which hack/verify-all.sh and CI run automatically. It fails when:

   • any module's toolchain line differs from the root go.mod, or a module has none (this catches new modules)
   • the Go version in benchmarking/automation/Dockerfile (ARG GO_VERSION) or benchmarking/locust/Dockerfile (golang: image) differs from the root's

   A short allowlist covers deliberate exceptions. Each entry names the module, the reason, and a removal date.

   Phase 2: Renovate, Go toolchain only

   The config can land before the app is installed. Renovate ignores it until then. On install, it sees an existing config and skips the onboarding PR.

   Setup:

   • An agent-substrate org admin installs the Mend Renovate app, after any approval needed for third-party apps.
   • Someone owns reviewing and merging its PRs.
   • For the first runs, set "dependencyDashboardApproval": true at the top level. Renovate then lists what it would do in a dashboard issue and opens nothing until someone approves it there. Remove it once the output looks right.

   Expected volume: about one PR per Go patch release, roughly monthly, per branch. Each PR covers every pin. Minor upgrades (1.28) wait for approval in the dashboard. Ordinary dependency updates, Actions pins, and image digest refreshes stay off.

   Draft .github/renovate.json5. The option names are checked against Renovate's current schema, but run renovate-config-validator and do a dashboard-only first run before relying on it:

   {
     $schema: "https://docs.renovatebot.com/renovate-schema.json",
     extends: ["config:recommended"],
     // Only go.mod, Dockerfiles, and the regex rule below. No Actions, npm, etc.
     enabledManagers: ["gomod", "dockerfile", "custom.regex"],
     baseBranchPatterns: ["main", "release-0.5"],
     customManagers: [
       {
         // ARG GO_VERSION=1.27.3 in benchmarking/automation/Dockerfile
         customType: "regex",
         managerFilePatterns: ["/(^|/)Dockerfile$/"],
         matchStrings: ["ARG GO_VERSION=(?<currentValue>\\S+)"],
         depNameTemplate: "go",
         datasourceTemplate: "golang-version",
       },
     ],
     packageRules: [
       // Off by default: no ordinary dependency updates.
       { matchPackageNames: ["*"], enabled: false },
       // Back on: the go.mod toolchain line, the golang image, and the ARG.
       { matchDepTypes: ["toolchain"], matchDepNames: ["go"], enabled: true },
       {
         matchDatasources: ["docker", "golang-version"],
         matchPackageNames: ["golang", "go"],
         enabled: true,
       },
       // One PR for every Go pin, opened as soon as Go releases.
       {
         matchDatasources: ["docker", "golang-version"],
         matchPackageNames: ["golang", "go"],
         groupName: "go toolchain",
         minimumReleaseAge: "0 days",
       },
       // Patches open PRs; minor bumps (1.28) wait for a tick in the dashboard.
       {
         matchPackageNames: ["golang", "go"],
         matchUpdateTypes: ["minor", "major"],
         dependencyDashboardApproval: true,
       },
       // No PRs just because the golang image was rebuilt on a new base.
       { matchPackageNames: ["golang"], matchUpdateTypes: ["digest"], enabled: false },
     ],
   }

   Later: exact Go for release images

   Once release images come from a single build script, it should set GOTOOLCHAIN to the root go.mod's toolchain value. It should also fail if the Go it actually used (go env GOVERSION) differs. Builds from a laptop and from CI then match exactly.

   govulncheck stays mandatory as the backstop. It scans the standard library at the pinned version, so a pin left behind a known vulnerability turns CI red even if Renovate stops working.
4. MushuEE commented on Oct 20, 2026

   

   Grant McCloskey (MushuEE)

   on Oct 20, 2026

   CollaboratorAuthor

   More actions

   On CNCF and Renovate: other CNCF projects already use it. OpenFeature has run the free Mend app for years, and Cilium's config is a good reference. Mend also has an optional Community (OSS) plan that projects can request (OpenFeature's request). It adds more concurrency and memory, plus Merge Confidence. A toolchain-only config doesn't need any of that, so the free app works for us.
5. BenTheElder commented on Oct 20, 2026

   

   Benjamin Elder (BenTheElder)

   on Oct 20, 2026

   Collaborator

   More actions

   FWIW, in most cases we don't care that all of hack/tools are on the same go version as us, and it might even break some of the tools. They're only used at dev-time.

   For the root module we really care that it stays very patched, along with any plugin modules at least until they move out to their own repos.

   Just a quick note, I didn't absorb all the other details yet.

   👍React with 👍1Reacted by Grant McCloskey
6. MushuEE commented on Oct 20, 2026

   

   Grant McCloskey (MushuEE)

   on Oct 20, 2026

   CollaboratorAuthor

   More actions

   Ack, makes sense we can let things drift for hack and tools. Mostly trying to prevent any chance that GOTOOLCHAIN=local ever fails to build because the code is on a new go version etc which .go-version might encourage mismatch. Plan is to see if we can onboard to Renovate and let it dynamically handle the patches.
7. Aditya Shantanu (aditya-shantanu)

   added

   kind/featureAn enhancement / feature request or implementationAn enhancement / feature request or implementation

   area/dev-infra

   on Oct 21, 2026
8. Quentin Bisson (QuentinBisson)

   added a commit that references this issue on Oct 23, 2026

   docs(fork): record that upstream pins the Go toolchain at 1.27.3 (#539)

   ...

   13c8208