Summary

Shards the Tier 3 KWOK recipe × deployer matrix into batches fanned out to a reusable workflow, so the job stays under GitHub Actions' hard cap of 256 matrix configurations per job.

Motivation / Context

The test-tier3 job crossed every testable recipe with all four deployers (72 × 4 = 288), exceeding GitHub's 256-config limit. GitHub rejected the matrix before any leg ran, so the post-merge/nightly full-matrix backstop silently stopped executing.

Fixes: #1171
Related: #1172 (follow-up: deployer-list / step-block de-duplication across tiers)

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• Docs/examples (docs/, examples/)
• Other: CI workflows (.github/workflows/kwok-recipes.yaml, .github/workflows/kwok-tier3-shard.yaml)

Implementation Notes

• discover now builds the {recipe, deployer} cross-product and chunks it into batches of TIER3_BATCH_SIZE (200, with headroom under 256), emitting a new tier3_batches output of {id, pairs} objects. A fail-closed guard errors loudly if the batch size is ever raised past 256, rather than resurfacing GitHub's opaque rejection mid-fan-out.
• test-tier3 is now a thin matrix over those batches that fans each one out to the new kwok-tier3-shard.yaml reusable workflow (matching the repo's existing workflow_call pattern), which expands its batch as its own ≤ 256 matrix. Batch count grows automatically as overlays are added — no manual job duplication.
• Concurrency: the group gains a -${{ matrix.batch.id }} suffix so every shard of a single run lands in its own group and runs in parallel; the SHA key still keeps successive merges independent (ADR-003's no-cancel-on-successive-merges guarantee preserved). The inner pair-matrix in the shard workflow has no concurrency block, so it is plainly parallel.
• Edge cases: workflow_dispatch emits tier3_batches=[] so the output contract is total; test-tier3's if: guard is keyed on tier3_batches (the output it actually consumes).
• ADR-003 amended: updated the concurrency snippet and added a "Tier 3 matrix sharding (GitHub 256-config cap)" subsection and refreshed structure diagram.

Testing

This is a CI-workflow + docs change (no Go); the merge gate for workflows is actionlint + yamllint, run directly:

# actionlint pinned to the CI version (v1.7.11), exact CI flags — clean repo-wide
actionlint -color -shellcheck=

# yamllint with repo config — clean
yamllint -c .yamllint.yaml .github/workflows/kwok-recipes.yaml .github/workflows/kwok-tier3-shard.yaml

Batching logic validated locally with synthetic inputs (0, 1, 50, 72, 200, 201, 400 recipes × 4 deployers): no dropped or duplicated pairs, contiguous integer batch ids, every batch ≤ 200. Reviewed via a three-persona review (CI/security, distributed-systems correctness, maintainability) — no blockers; all feedback applied.

Note: Tier 3 only runs on push-to-main and the nightly schedule (never on PRs, by design), so the sharded matrix is first exercised end-to-end on the post-merge run.

Risk Assessment

• Low — Isolated CI/docs change, no product code; easy to revert (single commit).

Rollout notes: No migration. KWOK Test Summary remains the required check (per ADR-003), so the renamed/nested Tier 3 matrix job names do not affect branch protection.

Checklist

• Tests pass locally (make test with -race) — N/A (no Go changes); ran actionlint + yamllint instead
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality — N/A (CI workflow change)
• I updated docs if user-facing behavior changed (ADR-003)
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)