Test running the CI on RunsOn runners by shahar1 · Pull Request #74714 · apache/airflow

shahar1 · 2026-10-14T09:42:12Z

Experiment to compare per-run cost and duration of running the Tests (AMD) / Tests (ARM) jobs on RunsOn (runs-on.com) EC2 runners instead of GitHub-hosted ones. Opened as a draft; it is not intended to be merged as-is.

The branch routes the CI jobs to RunsOn size classes, adds S3-backed stash actions, and includes the supporting changes (breeze --github-repository for k8s commands, hermetic AWS no-credentials tests, capped SFTP concurrency test workers).

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Sonnet 5.5)

Generated-by: Claude Code (Sonnet 5.5) following the guidelines

🤖 Generated with Claude Code

Fork-only experiment: route every Tests (AMD) / Tests (ARM) job to RunsOn (runs-on.com) EC2 runners instead of GitHub-hosted ones, so the per-run cost and duration can be compared with the EKS runner experiment. Each wrapper's `runner-type` output carries the architecture-specific RunsOn label template (families, image, volume); jobs fill in the size class they need with `format(<template>, cpu, ram)`. The classes are the EC2 equivalents of the ones measured for EKS: small 4 vCPU / 16 GB, medium 8 / 32, xl 32 / 128. xl gets 128 GB rather than 64 because xdist parallelism follows the host's 32 vCPUs (EKS capped it at 16), and the non-DB providers suite already peaked at 35 GB with 16 workers. `build-info`, `print-platform`, `notify-slack` and the Windows smoke test stay on GitHub-hosted runners as upstream. The stash actions are S3-backed drop-ins for the ASF stash action that write to the RunsOn stack's cache bucket (the only S3 prefix the instance role can reach), so image tars and caches never leave the region. The RunsOn action is added to the breeze composite action to put the EC2 cost of every job next to the GitHub-hosted price in the job summary. Fork prerequisites carried over from the EKS branch: breeze k8s commands honour `--github-repository` so the K8s tests find the fork's prod image, the apache remote always points at apache/airflow, helm-chart tags are fetched from apache/airflow, e2e tests skip pulling fork ghcr images, and the SFTP concurrency test caps its workers instead of using the host CPU count. Static checks pin PYTHONWARNINGS because prek runs 32 concurrent breeze-backed hooks on the xl class and they clobber the shared docker env file otherwise.

The job still consumed the raw `runner-type` output, which on this branch is the RunsOn label template, so it asked EC2 for `cpu={0}/ram={1}` and RunsOn fell back to a runner that fails the job by design.

The TypeScript SDK tests job and the agent framework tests landed on main after this branch was cut, so they still passed the raw `runner-type` output through. On this branch that output is the RunsOn label template, which RunsOn rejects without a cpu/ram fill-in.

The stash actions defaulted to the fork stack's bucket, so on any other RunsOn stack the instance role was denied PutObject and the CI image build failed. RunsOn exports its stack's cache bucket and region into every job, so read them from there instead of hardcoding either.

vitest starts one worker per CPU minus one, and every worker funnels its module transforms through the single vite server thread. On the 32-CPU class the 31 workers starve the full `monaco-editor/editor` import that the `beforeAll` hook of `pythonFStrings.test.ts` performs: the test file took 24s on the one run that passed and hit the hook's 30s timeout on the two that failed, while it takes 3.6s on the 4-CPU GitHub-hosted runner. Measured locally on a 16-core host (partial suite): 3 workers 4.5s, 7 workers 8.0s, 15 workers timed out. Scaling the two CI points linearly gives 7-8s at 7 workers, so the 8-CPU class keeps the hook far below the timeout.

shahar1 · 2026-10-16T18:22:58Z

Great success, real thing on its way :)

`store_directory_concurrently` and `retrieve_directory_concurrently` default to `os.cpu_count()` workers, and each worker opens its own SSH connection, one after another, before any transfer starts. The test's runtime therefore grows with the host's CPU count. On the self-hosted EKS runners of an earlier CI experiment `os.cpu_count()` reported 32-48 (it ignores cgroup CPU limits) and the test hit the 60s per-test timeout in 14 jobs, including on a dedicated node. Locally, 64 workers take 24s for their 128 sequential handshakes, so slower or throttled hosts cross the limit. The 32-vCPU class of the RunsOn runners (apache#74714) has the same exposure. sshd's `MaxStartups` is not involved: the connections authenticate one at a time, and 32 and 64 workers both pass locally against its default. Four workers is what the test uses on today's 4-vCPU GitHub-hosted runners, so it keeps exercising the same path whatever the host.

On EC2 types with local NVMe disks (the family prefixes also match the `d` variants, e.g. `m6id` or `m7gd`), RunsOn assembles them into a RAID0, mounts it at /mnt/ephemeral and /var/lib/docker, and keeps /home/runner and /tmp on it. `rm -rf /mnt/*` expanded to that mount point, emptied it and only then failed with "Device or resource busy", taking the runner's own files with it: the CI image build on apache#74889 lost its runner right after this step with "Could not find file '/home/runner/_diag/pages/...'". Whether it hit depended on which spot type the job got, which is why the apache#74714 runs passed. On self-hosted runners make_mnt_writeable.sh now keeps mount points under /mnt, keeps rm out of other filesystems mounted further down, and chowns only /mnt itself, since a recursive chown would re-own docker's store on that disk. move_docker_to_mnt.sh leaves /var/lib/docker alone when it is already a mount there: emptying it could hit the same disk, and moving docker to the EBS root volume would only make it slower. GitHub-hosted runners keep the previous commands unchanged.

`store_directory_concurrently` and `retrieve_directory_concurrently` default to `os.cpu_count()` workers, and each worker opens its own SSH connection, one after another, before any transfer starts. The test's runtime therefore grows with the host's CPU count. On the self-hosted EKS runners of an earlier CI experiment `os.cpu_count()` reported 32-48 (it ignores cgroup CPU limits) and the test hit the 60s per-test timeout in 14 jobs, including on a dedicated node. Locally, 64 workers take 24s for their 128 sequential handshakes, so slower or throttled hosts cross the limit. The 32-vCPU class of the RunsOn runners (#74714) has the same exposure. sshd's `MaxStartups` is not involved: the connections authenticate one at a time, and 32 and 64 workers both pass locally against its default. Four workers is what the test uses on today's 4-vCPU GitHub-hosted runners, so it keeps exercising the same path whatever the host.

GitHub-hosted runners cap every job at 4 vCPU / 16 GB, and the full provider and core suites spend most of their time waiting on that. The RunsOn experiment in apache#74714 ran the whole CI on EC2 runners in the project's AWS account and came back green, so this moves the trusted part of CI there: pushes, schedules and dispatches in apache/airflow, and PRs from apache/airflow branches. Code running on those runners can reach the RunsOn stack's AWS resources, so PRs from forks, PRs opened by bots and PRs from Dependabot branches (code nobody has reviewed yet) stay on GitHub-hosted runners. A fork PR moves only when a committer adds the `use self-hosted runners` label; triage collaborators can label PRs too, so the label counts only if the PR's events show a committer added it and nobody removed it since. `use public runners` sends any PR back to GitHub-hosted runners, e.g. when RunsOn is unavailable. The routing lives in selective checks, which already know the event, repository, labels and platform, so the decision is unit-tested instead of spread over workflow expressions. Its `runner-type` output is either the public label or a RunsOn label template, and each job picks its size class with `format(<label>, cpu, ram)`, which leaves a public label untouched. The classes are the EC2 equivalents of those measured in the EKS runner experiment: small 4 vCPU / 16 GB, medium 8 / 32, xl 32 / 128. xl gets 128 GB rather than 64 because xdist runs one worker per vCPU and the non-DB providers suite already peaked at 35 GB with 16 workers. React UI tests get the 8-CPU class, not xl: vitest starts one worker per CPU minus one, and on 32 CPUs the workers starved the `monaco-editor` import in a `beforeAll` hook past its 30s timeout (24s on the one passing run, 3.6s on the 4-CPU GitHub-hosted runner; locally 3 workers 4.5s, 7 workers 8.0s, 15 workers timed out). On RunsOn the multi-GB image tarballs go to the stack's in-region S3 cache bucket instead of leaving AWS as GitHub artifacts. Every RunsOn job can write to that bucket, so it is not trusted: a tarball is stored under its sha256 and only the digest is stashed as a GitHub artifact, which GitHub scopes to the run and branch. Without that, a PR's job could replace an image a canary run restores in the jobs that publish docs and push constraints. The small caches (mount cache, prek, docs inventory) keep using the ASF stash action on both kinds of runners, so PRs on GitHub-hosted runners still get the ones canary runs save. The image stash actions are loaded from the workflow's own checkout because ci-image-build also builds older release refs that predate them. The EC2 cost of each job is added to its summary, on RunsOn only. `amd-runners` / `arm-runners` stay on public runners: the Docker Hub release workflow consumes them and runs with release secrets.