Reduce Github Action Usage

We received a note on the mailing list today:

Mailing list item: https://lists.apache.org/thread/07rr90dpwdc0mqk2mxd4drkco50d334x

n 2024, the ASF introduced the policy for GitHub Actions usage
across the foundation1. The ASF Github shared pool of
Github-hosted runners has been at, or very close to the limit of
900 jobs most of the time in the past few weeks and this is the
case again today.

Your project has been identified as being among the top 5 consumers of
build time over the past 7 days and we request that you bring your
usage down by stream-lining long-running builds. Contact Infra for
a consultation if you are unable to streamline your builds further.

You can use the infra reporting tool2 to monitor your GHA usage as you
work on stream-lining, as well as locate any bottlenecks in the workflows.

Infra will allow you two weeks time (till the 8th of June, 2026) to
progress this, but should you still be above the limits by then,
without a viable path forward, we will be limiting your GHA usage.

Kind regards,
Bob Thomson, on behalf of ASF Infrastructure.

Here is a snapshot of the report from this morning (need a login to see)

Yikes, thanks for opening up the issue here. 😬

@andygrove

It looks like 78% or more of this usage is from Comet;

Any ideas how to reduce this usage @andygrove or @mbutrovich ?

The biggest DataFusion core contributor looks like

12%: Rust: https://github.com/apache/datafusion/blob/main/.github/workflows/rust.yml
3%: extended tests

Any ideas how to reduce this usage @andygrove or @mbutrovich ?

We've been discussing in apache/datafusion-comet#4389 and apache/datafusion-comet#4289 but this will definitely light a fire under the Comet project to get it sorted out.

In the short term, we can consider reducing the matrix that runs on PRs and run the full matrix on main - not ideal, since regressions will slip through and create extra work for us - but will buy us time until we can implement apache/datafusion-comet#4289, which seems like the best long term solution to me.

In the short term, we can consider reducing the matrix that runs on PRs and run the full matrix on main - not ideal, since regressions will slip through and create extra work for us - but will buy us time until we can implement apache/datafusion-comet#4289, which seems like the best long term solution to me.

One way to mitigate the extra work would be to have an automatic action that makes a revert PR of you detect a failure 🤔

@blaginin

Also @blaginin has the (very sweet) merge queue setup in the main datafusion repo

I wonder if we could move some of the heavier tests from running on PR push / creation and instead only run when they are put into the merge queue (right before merge to main)

This should reduce the number of heavy test runs (basically 2x per commit to main, rather than per push to each PR)

🤔 I wonder if we could do this for extended tests on datafusion main too 🤔

One way to mitigate the extra work would be to have an automatic action that makes a revert PR of you detect a failure 🤔

Spark SQL tests like to fail spuriously, so this might not be something we want to automate yet. Maybe the action can at least file an issue to bring attention to the failure.

I created apache/datafusion-comet#4406 to track the Comet-specific discussions

I wonder if we could move some of the heavier tests from running on PR push / creation and instead only run when they are put into the merge queue (right before merge to main)

🤔 I wonder if we could do this for extended tests on datafusion main too 🤔

Will do!

Some of our ci jobs are failing regularly:

I asked codex about it:

  the job  failed in actions/checkout, before Setup
  Rust toolchain or any Rust tests started. The errors are submodule fetch
  failures for parquet-testing and apache/arrow-testing: git exit 128, “did not
  contain <commit>”, “the remote end hung up unexpectedly”, and “could not read
  Username for https://github.com: terminal prompts disabled”. I sampled several
  other failed Rust workflow runs from May 23, 2026 across unrelated PRs and saw
  the same pattern in different jobs, including Linux and macOS. I also checked
  one rerun-success case where attempt 1 failed in checkout and attempt 2 passed
  with no code change.

  Why this keeps happening: the Rust workflow uses actions/checkout@v6 with
  submodule checkout enabled in many jobs, and GitHub’s checkout docs say
  fetch-depth defaults to 1, submodules are fetched when that input is enabled,
  and the auth token is persisted in local git config by default. That means
  these jobs are doing shallow submodule fetches during checkout, which is
  exactly the phase that is flaking here. GitHub Status also shows recent GitHub
  Actions degradation incidents on May 20, 2026 and May 15, 2026, so
  platform-side instability is a plausible contributor. (github.com
  (https://github.com/actions/checkout/blob/main/README.md))

  My conclusion: the real issue is flaky submodule checkout against GitHub-
  hosted repos, not failing DataFusion tests and not a bad PR. The “could not
  read Username” message looks like a fallback symptom after the fetch breaks,
  not the main cause. The shared risk factors are actions/checkout, submodules:
  true, shallow fetch, and GitHub-side transport instability.

New snapshot - not directly comparable to previous one since we're into the weekend, but we've made good progress in Comet to reduce Spark SQL runs.

The ceiling per https://infra.apache.org/github-actions-policy.html is

250,000 minutes over 7 days (calendar week)
216,000 minutes over 5 days

The average number of minutes a project uses per calendar week MUST NOT exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours).

The average number of minutes a project uses in any consecutive five-day period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes, or 3,600 hours).

Datafusion seems to be sitting right below that 😄

Past 7 day usage is 216,110 minutes
Past 5 day usage is 120,929 minutes

In Comet we are working on making some CI jobs run in contributors accounts (same approach as Spark takes). We have done about as much as we can for what runs on each PR.

@comphead

Thanks @comphead -- shall we close this issue now?

Thanks @comphead -- shall we close this issue now?

I dont think we can close until we get good feedback from Bob, current update is we are on right track, but we still some things to do as part of this issue, namely:

Merge chore: run CI on contributor forks datafusion-comet#4570 when its ready

@alamb

I think this issue can be closed for now, @alamb I'm leaving the final decision to you

Last response from Bob, DF is not a top concern but still concern. We need to take CI seriously

On 2026/06/11 08:48:46 Bob Thomson wrote:
> Hi,
> 
> Reviewing the picture today, the GitHub hosted runners utilisation overall is still maxxing out daily. WIth respect to datafusion we can see over the last 7 days that the project has dropped out of the top 5 to 7 which is good progress. Your continued work and vigilence on this area helps all projects and is appreciated.
> 
> Please note that there is now a page for sharing tips and best practice for optimising workflows with respect to utilisation:
> 
> https://cwiki.apache.org/confluence/display/INFRA/GitHub+Actions+Recommended+Practices
> 
> There is also a Slack channel for community discussion and sharing ideas: project-workflow-optimisations
> 
> Thanks.
> 
> Kind regards,
> -Bob Thomson,
> ASF Infrastructure

I'll try to spend some time on a local one-click script that mirrors CI as closely as possible. The goal is to make it easier to validate fixes locally before pushing, so we can reduce CI re-runs due to test failures.

Add Local Scripts to Reproduce Full CI #21048

I'll try to spend some time on a local one-click script that mirrors CI as closely as possible. The goal is to make it easier to validate fixes locally before pushing, so we can reduce CI re-runs due to test failures.

Add Local Scripts to Reproduce Full CI #21048

You may want to explore our experiments with contributors forks we investigated in Comet, as Comet runs much more heavy weight pipelines. apache/datafusion-comet#4570

@2010YOUY01

@2010YOUY01 @comphead earlier I had raised the proposal in the discord for updating the PR template to have a checklist with instructions on running tests, and linters locally. There was some consensus the only concern raised was if this will create friction for new contributors. Looking at the current situation and since @2010YOUY01 is working on the script for local test runners should we think about the PR template approach. Let me know what you all think.

@2010YOUY01 @comphead earlier I had raised the proposal in the discord for updating the PR template to have a checklist with instructions on running tests, and linters locally. There was some consensus the only concern raised was if this will create friction for new contributors. Looking at the current situation and since @2010YOUY01 is working on the script for local test runners should we think about the PR template approach. Let me know what you all think.

I think many contributors these days are using coding agents, so perhaps we can provide clear instructions for how to do verify/fix / automate the process and I bet it won't add too much over head

There was some consensus the only concern raised was if this will create friction for new contributors.

Currently it seems we have to read the CI .yml scripts to fully reproduce it locally, this creates friction for experienced contributors or even AI agents (waste tokens and execution time)

But even without a better script for it, we can update the agents instruction to below for now

1. run quick tests to ensure things mostly working after commit
2. Before pushing, try to fully reproduce CI from `.github/workflows`

But even without a better script for it, we can update the agents instruction to below for now

I feel we already discussed something like local resources involvement to catch low hanging issues, before AI we explored git pre hooks https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks to validate initial commit

WDYT folks if we get back to this idea in combination with AI, so AI can help a person to make a commit happen by running/fixing tests, which would reduce chances of running CI multiple times on server side?

Note: it can be issues on running pre hooks for gh

But even without a better script for it, we can update the agents instruction to below for now

I feel we already discussed something like local resources involvement to catch low hanging issues, before AI we explored git pre hooks https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks to validate initial commit

WDYT folks if we get back to this idea in combination with AI, so AI can help a person to make a commit happen by running/fixing tests, which would reduce chances of running CI multiple times on server side?

Note: it can be issues on running pre hooks for gh

I do like git pre hooks, I use in all my workflows though I am not sure there will be consensus on this as people have very strong opinions on it.

I think some kind of AI instruction would be less intrusive.

I think some kind of AI instruction would be less intrusive.

Agree, it won't hurt. let me create the PR, and also I'll ping ASF infra to see if we can access historical ASF resource usage to identify a trend. this would help to understand if we getting any actual benefits

@2010YOUY01

@2010YOUY01 Please have a look on #23313

chore: extend pre commit instructions for AI agents (apache#23313)

It happened again -- see

Reduce GitHub Actions usage (DataFusion is currently the top ASF consumer of GHA minutes) #25148

Navigation Menu

Repository navigation

Reduce Github Action Usage #22455

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Reduce Github Action Usage #22455

Description

Activity

alamb commented on May 22, 2026

mbutrovich commented on May 22, 2026

alamb commented on May 22, 2026

mbutrovich commented on May 22, 2026

andygrove commented on May 22, 2026

alamb commented on May 22, 2026

alamb commented on May 22, 2026

mbutrovich commented on May 22, 2026

andygrove commented on May 22, 2026

blaginin commented on May 22, 2026

kumarUjjawal commented on May 23, 2026

andygrove commented on May 23, 2026

kevinjqliu commented on May 26, 2026

andygrove commented on May 28, 2026

2 remaining items

alamb commented on Jun 4, 2026

comphead commented on Jun 4, 2026

comphead commented on Jun 11, 2026

2010YOUY01 commented on Jun 14, 2026

comphead commented on Jun 15, 2026

kumarUjjawal commented on Jun 15, 2026

alamb commented on Jun 15, 2026

2010YOUY01 commented on Jun 16, 2026

comphead commented on Jun 18, 2026

kumarUjjawal commented on Jun 18, 2026

comphead commented on Jun 18, 2026

comphead commented on Jul 3, 2026

alamb commented on Sep 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions