Skip to content

Run shared distribution tests in a single CI job - #74566

Merged
potiuk merged 3 commits into
apache:mainfrom
austenstone:collapse-shared-distributions-matrix
Oct 18, 2026
Merged

potiuk merged 3 commits into
apache:mainfrom
austenstone:collapse-shared-distributions-matrix

Conversation

@austenstone

Copy link
Copy Markdown
Contributor

Every PR starts 14 separate ~20s "Shared tests" jobs, using 14 of the 20 concurrent jobs ASF allows per workflow. Last week that was about 1 in 5 jobs in Tests (AMD), mostly spent on checkout and uv setup rather than tests.

This runs the shared distributions sequentially in one job, each in its own log group, and fails with a list of every distribution that failed. The 14 per-distribution checks become one "Shared distributions tests" check; none were required.

Verified with actionlint, shellcheck and zizmor; locally the loop passes all 14 distributions in ~66s and reports failures by name.


Was generative AI tooling used to co-author this PR?
  • Yes (GitHub Copilot, Claude Opus 5.5)

Generated-by: GitHub Copilot (Claude Opus 5.5) following the guidelines

@boring-cyborg

boring-cyborg Bot commented Oct 12, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://s.apache.org/airflow-slack

Comment thread .github/workflows/basic-tests.yml Resolved

@Andrushika Andrushika left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update!

austenstone and others added 3 commits October 18, 2026 19:00
The shared distributions matrix starts one runner per directory in
shared/ (14 today) on every PR. Each leg spends ~15s on checkout and uv
setup to run ~2-4s of tests, and the fan-out takes 14 of the 20
concurrent jobs ASF allows per workflow.

Running the distributions sequentially in one job keeps the same
coverage, reports every failing distribution, and frees those slots.
@potiuk
potiuk force-pushed the collapse-shared-distributions-matrix branch from 7860260 to c118542 Compare October 18, 2026 17:00
@potiuk
potiuk merged commit 6b42e65 into apache:main Oct 18, 2026
284 of 285 checks passed
@boring-cyborg

boring-cyborg Bot commented Oct 18, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

potiuk added a commit that referenced this pull request Oct 25, 2026
…, #75615) (#75617)

* Keep a Breeze test's --dry-run from leaking into later tests (#75589)

The dry-run, verbose and forced-answer options are process-wide, so a
test invoking a command with --dry-run made run_command return empty
output for every later test on the same xdist worker. Which tests failed
depended on how xdist distributed them, so the Breeze unit tests failed
intermittently, as in the scheduled ARM run.

Generated-by: Claude Opus 5

(cherry picked from commit 0c1fa8e)

* Run shared distribution tests in a single CI job (#74566)

* Run shared distribution tests in a single CI job

The shared distributions matrix starts one runner per directory in
shared/ (14 today) on every PR. Each leg spends ~15s on checkout and uv
setup to run ~2-4s of tests, and the fan-out takes 14 of the 20
concurrent jobs ASF allows per workflow.

Running the distributions sequentially in one job keeps the same
coverage, reports every failing distribution, and frees those slots.

* Run each shared distribution with uv --exact for isolation

* Wrap the shared distributions test loop line to satisfy yamllint

Generated-by: Claude Opus 5

---------

Co-authored-by: Jarek Potiuk <[email protected]>

(cherry picked from commit 6b42e65)

* Run maintainer and canary CI on self-hosted RunsOn runners (#74889)

This PR implements AIP-118 for integrating self-hosted runners in Apache Airflow's CI, utilizing the project's AWS instance and runs-on.com's runners.

(cherry picked from commit b5bc253)
Generated-by: Claude Opus 5

* Check newsfragment PR numbers in static checks instead of a separate workflow (#75612)

* Check newsfragment PR numbers in static checks instead of a separate workflow

The separate workflow ran outside the fork CI gate (AIP-120), needed its own
required check, and needed its own approval for first-time contributors.
As part of the existing newsfragment hook it runs wherever static checks
run, in project CI and in a contributor's fork, and fails CI passed like any
other static check.

A newsfragment is usually written before its pull request exists, so a
local run cannot know the right number and accepts any.

Generated-by: Claude Opus 5

* Address review of the newsfragment PR number check

A pull request inside a fork carries the fork's number, so only apache/airflow pull requests are checked. A GitHub API error is reported without a traceback, and the instructions say who re-runs the checks when CI runs in the contributor's fork.

Generated-by: Claude Opus 5

(cherry picked from commit 757898c)

The documentation conflicts were resolved by keeping the release branch's text and
updating only its sentences about the newsfragment check.

* Report EC2 Spot interruptions on pull requests and keep them out of Slack (#75615)

Jobs on the self-hosted RunsOn runners fail with exit code 137 when AWS
reclaims their Spot instance. Nothing in the job's log says why, so the
failure looks like a broken test or an out-of-memory kill, although RunsOn
re-runs the job by itself once the run finishes.

A pull request should say so as soon as it happens, and the canary should
not page anyone on Slack until RunsOn has given up retrying. The Slack
notification also never reported how RunsOn's re-run ended, because it only
ran for the first attempt.

Generated-by: Claude Opus 5

(cherry picked from commit 4386cb1)
@potiuk potiuk added this to the Airflow 3.4.0 milestone Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants