feat: implement "concurrency" block in Forgejo Actions at the workflow level #9434

Merged
earl-warren merged 11 commits from mfenniak/forgejo:workflow-level-concurrency into forgejo 2025-10-03 18:43:13 +02:00
Member

Currently references a pre-release version of code.forgejo.org/forgejo/runner/v11, pending release of https://code.forgejo.org/forgejo/runner/pulls/1026.

Fixes #5914.

This PR is quite large, but it can be reviewed commit-by-commit in relatively small, logical chunks.

Adds support for workflows with a concurrency block, and submembers group and cancel-in-progress. For example:

on:
  workflow_dispatch:
jobs:
  rust-checks:
    runs-on: debian-latest
    steps:
      - run: sleep 300
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: false

The concurrency block effectively ends up with four supported behaviors that users will want to choose from:

  • Backwards compatibility / default -- if omitted completely, the existing Forgejo behavior will be implemented. That behavior is that push and pull request synchronize events will cancel all previous runs on the same repository, branch, and workflow.
  • Unlimited concurrency -- if the cancel-in-progress value is set to false and no group is provided, then the previously described Forgejo behavior will be disabled and an unlimited number of workflows can be executed simultaneously (to the maximum supported by the Forgejo Runner capacity).
  • Queue-behind -- if a group is provided and cancel-in-progress: false is set, then every new action run with in the same repository with the same group value will be queued behind previous workflow runs, allowing only one workflow to execute at a time in the group, but allowing all workflows to finish naturally.
  • Cancel-in-progress -- if a group is provided and cancel-in-progress: true is set, then every new action run with in the same repository with the same group value will cause previously queued or running runs to be cancelled, allowing only one workflow to execute at a time in the group, but preferring execution of the most recent workflow.

Both the group and cancel-in-progress values can access values from the github, inputs and vars context for dynamic behavior.

Checklist

The contributor guide contains information that will be helpful to first time contributors. There also are a few conditions for merging Pull Requests in Forgejo repositories. You are also welcome to join the Forgejo development chatroom.

Tests

  • I added test coverage for Go changes...
    • in their respective *_test.go for unit tests.
    • in the tests/integration directory if it involves interactions with a live Forgejo server.
  • I added test coverage for JavaScript changes...

Documentation

  • I created a pull request to the documentation to explain to Forgejo users how to use this change.
  • I did not document these changes and I do not expect someone else to do it.

Release notes

  • I do not want this change to show in the release notes.
  • I want the title to show in the release notes with a link to this pull request.
  • I want the content of the release-notes/<pull request number>.md to be be used for the release notes instead of the title.

Release notes

  • Features
    • PR: implement "concurrency" block in Forgejo Actions at the workflow level
Currently references a pre-release version of `code.forgejo.org/forgejo/runner/v11`, pending release of https://code.forgejo.org/forgejo/runner/pulls/1026. Fixes #5914. This PR is quite large, but it can be reviewed commit-by-commit in relatively small, logical chunks. Adds support for workflows with a `concurrency` block, and submembers `group` and `cancel-in-progress`. For example: ``` on: workflow_dispatch: jobs: rust-checks: runs-on: debian-latest steps: - run: sleep 300 concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: false ``` The concurrency block effectively ends up with four supported behaviors that users will want to choose from: - Backwards compatibility / default -- if omitted completely, the existing Forgejo behavior will be implemented. That behavior is that push and pull request synchronize events will cancel all previous runs on the same repository, branch, and workflow. - Unlimited concurrency -- if the `cancel-in-progress` value is set to `false` and no `group` is provided, then the previously described Forgejo behavior will be disabled and an unlimited number of workflows can be executed simultaneously (to the maximum supported by the Forgejo Runner capacity). - Queue-behind -- if a `group` is provided and `cancel-in-progress: false` is set, then every new action run with in the same repository with the same group value will be queued behind previous workflow runs, allowing only one workflow to execute at a time in the group, but allowing all workflows to finish naturally. - Cancel-in-progress -- if a `group` is provided and `cancel-in-progress: true` is set, then every new action run with in the same repository with the same group value will cause previously queued or running runs to be cancelled, allowing only one workflow to execute at a time in the group, but preferring execution of the most recent workflow. Both the `group` and `cancel-in-progress` values can access values from the `github`, `inputs` and `vars` context for dynamic behavior. ## Checklist The [contributor guide](https://forgejo.org/docs/next/contributor/) contains information that will be helpful to first time contributors. There also are a few [conditions for merging Pull Requests in Forgejo repositories](https://codeberg.org/forgejo/governance/src/branch/main/PullRequestsAgreement.md). You are also welcome to join the [Forgejo development chatroom](https://matrix.to/#/#forgejo-development:matrix.org). ### Tests - I added test coverage for Go changes... - [x] in their respective `*_test.go` for unit tests. - [x] in the `tests/integration` directory if it involves interactions with a live Forgejo server. - I added test coverage for JavaScript changes... - [ ] in `web_src/js/*.test.js` if it can be unit tested. - [ ] in `tests/e2e/*.test.e2e.js` if it requires interactions with a live Forgejo server (see also the [developer guide for JavaScript testing](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/tests/e2e/README.md#end-to-end-tests)). ### Documentation - [x] I created a pull request [to the documentation](https://codeberg.org/forgejo/docs) to explain to Forgejo users how to use this change. - https://codeberg.org/forgejo/docs/pulls/1513 - [ ] I did not document these changes and I do not expect someone else to do it. ### Release notes - [ ] I do not want this change to show in the release notes. - [x] I want the title to show in the release notes with a link to this pull request. - [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title. <!--start release-notes-assistant--> ## Release notes <!--URL:https://codeberg.org/forgejo/forgejo--> - Features - [PR](https://codeberg.org/forgejo/forgejo/pulls/9434): <!--number 9434 --><!--line 0 --><!--description aW1wbGVtZW50ICJjb25jdXJyZW5jeSIgYmxvY2sgaW4gRm9yZ2VqbyBBY3Rpb25zIGF0IHRoZSB3b3JrZmxvdyBsZXZlbA==-->implement "concurrency" block in Forgejo Actions at the workflow level<!--description--> <!--end release-notes-assistant-->
mfenniak force-pushed workflow-level-concurrency from 50a4c2e428
Some checks failed
issue-labels / cascade (pull_request_target) Has been skipped
issue-labels / backporting (pull_request_target) Has been skipped
testing / frontend-checks (pull_request) Successful in 57s
testing / backend-checks (pull_request) Failing after 1m16s
testing / test-unit (pull_request) Has been skipped
testing / test-e2e (pull_request) Has been skipped
testing / test-mysql (pull_request) Has been skipped
testing / test-pgsql (pull_request) Has been skipped
testing / test-sqlite (pull_request) Has been skipped
testing / test-remote-cacher (redis) (pull_request) Has been skipped
testing / test-remote-cacher (valkey) (pull_request) Has been skipped
testing / test-remote-cacher (garnet) (pull_request) Has been skipped
testing / test-remote-cacher (redict) (pull_request) Has been skipped
testing / security-check (pull_request) Has been skipped
requirements / merge-conditions (pull_request) Successful in 2s
issue-labels / release-notes (pull_request_target) Has been skipped
to 211abc077e
Some checks failed
requirements / merge-conditions (pull_request) Successful in 2s
issue-labels / release-notes (pull_request_target) Has been skipped
testing / frontend-checks (pull_request) Successful in 56s
testing / backend-checks (pull_request) Failing after 1m18s
testing / test-unit (pull_request) Has been skipped
testing / test-e2e (pull_request) Has been skipped
testing / test-mysql (pull_request) Has been skipped
testing / test-pgsql (pull_request) Has been skipped
testing / test-sqlite (pull_request) Has been skipped
testing / test-remote-cacher (redis) (pull_request) Has been skipped
testing / test-remote-cacher (valkey) (pull_request) Has been skipped
testing / test-remote-cacher (garnet) (pull_request) Has been skipped
testing / test-remote-cacher (redict) (pull_request) Has been skipped
testing / security-check (pull_request) Has been skipped
2025-09-27 03:45:31 +02:00
Compare
mfenniak force-pushed workflow-level-concurrency from 211abc077e
Some checks failed
requirements / merge-conditions (pull_request) Successful in 2s
issue-labels / release-notes (pull_request_target) Has been skipped
testing / frontend-checks (pull_request) Successful in 56s
testing / backend-checks (pull_request) Failing after 1m18s
testing / test-unit (pull_request) Has been skipped
testing / test-e2e (pull_request) Has been skipped
testing / test-mysql (pull_request) Has been skipped
testing / test-pgsql (pull_request) Has been skipped
testing / test-sqlite (pull_request) Has been skipped
testing / test-remote-cacher (redis) (pull_request) Has been skipped
testing / test-remote-cacher (valkey) (pull_request) Has been skipped
testing / test-remote-cacher (garnet) (pull_request) Has been skipped
testing / test-remote-cacher (redict) (pull_request) Has been skipped
testing / security-check (pull_request) Has been skipped
to 8e2fb30e2b
All checks were successful
requirements / merge-conditions (pull_request) Successful in 2s
issue-labels / release-notes (pull_request_target) Has been skipped
2025-09-27 04:01:04 +02:00
Compare
Contributor

@mfenniak wrote in #9434 (comment):

  • That behavior is that push and pull request synchronize events will cancel all previous runs on the same repository, branch, and workflow.

I have not observed that it actually works on pull requests. Do you have a reproducer where a push to a pull request cancels the previous run?

https://code.forgejo.org/forgejo/end-to-end/pulls/135/files is a tentative reproducer from some time ago but it would need to be updated.

@mfenniak wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issue-2436940: > * That behavior is that push and pull request synchronize events will cancel all previous runs on the same repository, branch, and workflow. I have not observed that it actually works on pull requests. Do you have a reproducer where a push to a pull request cancels the previous run? https://code.forgejo.org/forgejo/end-to-end/pulls/135/files is a tentative reproducer from some time ago but it would need to be updated.
Author
Member

@earl-warren wrote in #9434 (comment):

I have not observed that it actually works on pull requests. Do you have a reproducer where a push to a pull request cancels the previous run?

https://code.forgejo.org/forgejo/end-to-end/pulls/135/files is a tentative reproducer from some time ago but it would need to be updated.

This statement reflects the existing code as written, but no, I haven't observed it. I suspect that if you've observed the absence of it, it's likely because of the cancel check is specific to the synchronized "subevent" (for lack of a better term). That may not be the event type when on: pull_request is used in the workflow...

I can take a closer look tomorrow. Since this behavior was just copied from the existing implementation, I didn't dig into it in detail, I was just concerned with not changing anything that existed in that path. 🤔

@earl-warren wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issuecomment-7399585: > > I have not observed that it actually works on pull requests. Do you have a reproducer where a push to a pull request cancels the previous run? > > https://code.forgejo.org/forgejo/end-to-end/pulls/135/files is a tentative reproducer from some time ago but it would need to be updated. This statement reflects the existing code as written, but no, I haven't observed it. I suspect that if you've observed the absence of it, it's likely because of the cancel check is specific to the synchronized "subevent" (for lack of a better term). That may not be the event type when `on: pull_request` is used in the workflow... I can take a closer look tomorrow. Since this behavior was just copied from the existing implementation, I didn't dig into it in detail, I was just concerned with not changing anything that existed in that path. 🤔
Contributor

Very nicely done and an easy to read series of commits. I also took a look at your todo list mfenniak/forgejo#1 out of curiosity but this PR stands on its own. I'll review again once the matching runner PR is merged.

Very nicely done and an easy to read series of commits. I also took a look at your todo list https://codeberg.org/mfenniak/forgejo/pulls/1 out of curiosity but this PR stands on its own. I'll review again once the matching runner PR is merged.
mfenniak force-pushed workflow-level-concurrency from 8e2fb30e2b
All checks were successful
requirements / merge-conditions (pull_request) Successful in 2s
issue-labels / release-notes (pull_request_target) Has been skipped
to f94c345349
Some checks failed
requirements / merge-conditions (pull_request) Successful in 1s
testing / backend-checks (pull_request) Failing after 3m8s
testing / test-remote-cacher (valkey) (pull_request) Has been skipped
testing / test-remote-cacher (garnet) (pull_request) Has been skipped
testing / test-remote-cacher (redict) (pull_request) Has been skipped
testing / frontend-checks (pull_request) Successful in 51s
testing / test-unit (pull_request) Has been skipped
testing / test-e2e (pull_request) Has been skipped
testing / test-mysql (pull_request) Has been skipped
testing / test-pgsql (pull_request) Has been skipped
testing / test-sqlite (pull_request) Has been skipped
testing / test-remote-cacher (redis) (pull_request) Has been skipped
testing / security-check (pull_request) Has been skipped
2025-09-27 18:13:18 +02:00
Compare
Author
Member

@earl-warren wrote in #9434 (comment):

I also took a look at your todo list mfenniak/forgejo#1 out of curiosity but this PR stands on its own.

Regarding the TODO list, the only item unsatisfied is that I'd love to performance test the concurrency group task picking database queries. (Query in question: mfenniak/forgejo#1 (comment))

I'm not very worried about the changes because:

  • There's a shut-off ini setting, in the worst-case scenario
  • FetchTask API calls are shortcut if the tasks version hasn't been incremented in the DB, so these queries aren't continuous when no server-side operations occur
  • The main search in the query is for action_run_job records with task_id=0 and status=StatusWaiting, and the additional complicated subqueries on concurrency groups will only apply to that small set of waiting tasks.

Despite that I had thought it might be good to test the performance with a large-scale action_run and action_run_job table like Codeberg's... but then there's a complication that it requires the new concurrency_group and concurrency_type fields... and I'd have to synthetically populate those in order to create a test dataset. Once you get to doing performance testing with synthetically generated data, the value of that testing varies wildly depending on how lucky you are in guessing the data that might generate problems.

I'm reaching two possible conclusions:

  • It wouldn't be as simple as asking someone to test a query, so I'd need to borrow a copy of action_run and action_run_job (which could have event_payload and workflow_payload redacted for its sensitive data), load it into a test DB, perform schema changes and synthetic data generation, and hope to generate a reasonable test case.
  • It's not worth it.

and I'm leaning towards it's not worth it.

@earl-warren wrote in #9434 (comment):

I'll review again once the matching runner PR is merged.

I've bumped the forgejo->runner reference up to the latest main commit in this branch now, and there will be a few CI tweaks to resolve. A follow-up review should hold until I get a nice green build. 🙂

@earl-warren wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issuecomment-7399786: > I also took a look at your todo list mfenniak/forgejo#1 out of curiosity but this PR stands on its own. Regarding the TODO list, the only item unsatisfied is that I'd love to performance test the concurrency group task picking database queries. (Query in question: https://codeberg.org/mfenniak/forgejo/pulls/1#issuecomment-7381471) I'm not very worried about the changes because: - There's a shut-off ini setting, in the worst-case scenario - FetchTask API calls are shortcut if the tasks version hasn't been incremented in the DB, so these queries aren't continuous when no server-side operations occur - The main search in the query is for `action_run_job` records with `task_id=0` and `status=StatusWaiting`, and the additional complicated subqueries on concurrency groups will only apply to that small set of waiting tasks. Despite that I had thought it might be good to test the performance with a large-scale `action_run` and `action_run_job` table like Codeberg's... but then there's a complication that it requires the new `concurrency_group` and `concurrency_type` fields... and I'd have to synthetically populate those in order to create a test dataset. Once you get to doing performance testing with synthetically generated data, the value of that testing varies wildly depending on how lucky you are in guessing the data that might generate problems. I'm reaching two possible conclusions: - It wouldn't be as simple as asking someone to test a query, so I'd need to borrow a copy of `action_run` and `action_run_job` (which could have `event_payload` and `workflow_payload` redacted for its sensitive data), load it into a test DB, perform schema changes and synthetic data generation, and hope to generate a reasonable test case. - It's not worth it. and I'm leaning towards it's not worth it. @earl-warren wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issuecomment-7399786: > I'll review again once the matching runner PR is merged. I've bumped the forgejo->runner reference up to the latest `main` commit in this branch now, and there will be a few CI tweaks to resolve. A follow-up review should hold until I get a nice green build. 🙂
mfenniak force-pushed workflow-level-concurrency from f94c345349
Some checks failed
requirements / merge-conditions (pull_request) Successful in 1s
testing / backend-checks (pull_request) Failing after 3m8s
testing / test-remote-cacher (valkey) (pull_request) Has been skipped
testing / test-remote-cacher (garnet) (pull_request) Has been skipped
testing / test-remote-cacher (redict) (pull_request) Has been skipped
testing / frontend-checks (pull_request) Successful in 51s
testing / test-unit (pull_request) Has been skipped
testing / test-e2e (pull_request) Has been skipped
testing / test-mysql (pull_request) Has been skipped
testing / test-pgsql (pull_request) Has been skipped
testing / test-sqlite (pull_request) Has been skipped
testing / test-remote-cacher (redis) (pull_request) Has been skipped
testing / security-check (pull_request) Has been skipped
to f2cadf5409
Some checks failed
requirements / merge-conditions (pull_request) Successful in 1s
issue-labels / release-notes (pull_request_target) Has been skipped
testing / frontend-checks (pull_request) Successful in 1m9s
testing / backend-checks (pull_request) Successful in 3m43s
testing / test-sqlite (pull_request) Failing after 4m2s
testing / test-unit (pull_request) Successful in 5m43s
testing / test-remote-cacher (valkey) (pull_request) Successful in 1m52s
testing / test-remote-cacher (redis) (pull_request) Successful in 1m54s
testing / test-remote-cacher (garnet) (pull_request) Successful in 1m52s
testing / test-remote-cacher (redict) (pull_request) Successful in 1m54s
testing / test-mysql (pull_request) Failing after 17m16s
testing / test-e2e (pull_request) Successful in 18m38s
testing / test-pgsql (pull_request) Failing after 26m38s
testing / security-check (pull_request) Has been skipped
2025-09-27 18:33:21 +02:00
Compare
mfenniak force-pushed workflow-level-concurrency from f2cadf5409
Some checks failed
requirements / merge-conditions (pull_request) Successful in 1s
issue-labels / release-notes (pull_request_target) Has been skipped
testing / frontend-checks (pull_request) Successful in 1m9s
testing / backend-checks (pull_request) Successful in 3m43s
testing / test-sqlite (pull_request) Failing after 4m2s
testing / test-unit (pull_request) Successful in 5m43s
testing / test-remote-cacher (valkey) (pull_request) Successful in 1m52s
testing / test-remote-cacher (redis) (pull_request) Successful in 1m54s
testing / test-remote-cacher (garnet) (pull_request) Successful in 1m52s
testing / test-remote-cacher (redict) (pull_request) Successful in 1m54s
testing / test-mysql (pull_request) Failing after 17m16s
testing / test-e2e (pull_request) Successful in 18m38s
testing / test-pgsql (pull_request) Failing after 26m38s
testing / security-check (pull_request) Has been skipped
to 4cc122047a
Some checks failed
requirements / merge-conditions (pull_request) Successful in 1s
testing / frontend-checks (pull_request) Successful in 1m20s
testing / backend-checks (pull_request) Failing after 1m43s
testing / test-unit (pull_request) Has been skipped
testing / test-e2e (pull_request) Has been skipped
testing / test-mysql (pull_request) Has been skipped
testing / test-pgsql (pull_request) Has been skipped
testing / test-sqlite (pull_request) Has been skipped
testing / test-remote-cacher (redis) (pull_request) Has been skipped
testing / test-remote-cacher (valkey) (pull_request) Has been skipped
testing / test-remote-cacher (garnet) (pull_request) Has been skipped
testing / test-remote-cacher (redict) (pull_request) Has been skipped
testing / security-check (pull_request) Has been skipped
2025-09-27 19:27:52 +02:00
Compare
mfenniak force-pushed workflow-level-concurrency from 4cc122047a
Some checks failed
requirements / merge-conditions (pull_request) Successful in 1s
testing / frontend-checks (pull_request) Successful in 1m20s
testing / backend-checks (pull_request) Failing after 1m43s
testing / test-unit (pull_request) Has been skipped
testing / test-e2e (pull_request) Has been skipped
testing / test-mysql (pull_request) Has been skipped
testing / test-pgsql (pull_request) Has been skipped
testing / test-sqlite (pull_request) Has been skipped
testing / test-remote-cacher (redis) (pull_request) Has been skipped
testing / test-remote-cacher (valkey) (pull_request) Has been skipped
testing / test-remote-cacher (garnet) (pull_request) Has been skipped
testing / test-remote-cacher (redict) (pull_request) Has been skipped
testing / security-check (pull_request) Has been skipped
to 1d2ef03cf8
Some checks failed
testing / frontend-checks (pull_request) Successful in 1m17s
testing / backend-checks (pull_request) Successful in 3m18s
testing / test-unit (pull_request) Successful in 8m59s
testing / test-remote-cacher (redis) (pull_request) Successful in 1m50s
testing / test-remote-cacher (valkey) (pull_request) Successful in 1m53s
testing / test-remote-cacher (garnet) (pull_request) Successful in 1m50s
testing / test-remote-cacher (redict) (pull_request) Successful in 1m47s
testing / test-mysql (pull_request) Successful in 24m20s
testing / test-e2e (pull_request) Successful in 25m15s
testing / test-sqlite (pull_request) Successful in 28m15s
testing / test-pgsql (pull_request) Successful in 31m50s
testing / security-check (pull_request) Successful in 56s
requirements / merge-conditions (pull_request) Successful in 1s
issue-labels / backporting (pull_request_target) Has been skipped
issue-labels / release-notes (pull_request_target) Successful in 38s
issue-labels / cascade (pull_request_target) Failing after 4m54s
2025-09-27 19:54:25 +02:00
Compare
mfenniak changed title from WIP: feat: implement "concurrency" block in Forgejo Actions at the workflow level to feat: implement "concurrency" block in Forgejo Actions at the workflow level 2025-09-27 21:32:29 +02:00
Where does that come from? The following is a preview of the release notes for this pull request, as they will appear in the upcoming release. They are derived from the content of the `release-notes/9434.md` file, if it exists, or the title of the pull request. They were also added at the bottom of the description of this pull request for easier reference.

This message and the release notes originate from a call to the release-notes-assistant.

@@ -51,2 +51,10 @@
 - [x] I want the title to show in the release notes with a link to this pull request.
 - [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title.
+
+<!--start release-notes-assistant-->
+
+## Release notes
+<!--URL:https://codeberg.org/forgejo/forgejo-->
+- Features
+  - [PR](https://codeberg.org/forgejo/forgejo/pulls/9434): <!--number 9434 --><!--line 0 --><!--description aW1wbGVtZW50ICJjb25jdXJyZW5jeSIgYmxvY2sgaW4gRm9yZ2VqbyBBY3Rpb25zIGF0IHRoZSB3b3JrZmxvdyBsZXZlbA==-->implement "concurrency" block in Forgejo Actions at the workflow level<!--description-->
+<!--end release-notes-assistant-->

Release notes

  • Features
    • PR: implement "concurrency" block in Forgejo Actions at the workflow level
<details> <summary>Where does that come from?</summary> The following is a preview of the release notes for this pull request, as they will appear in the upcoming release. They are derived from the content of the `release-notes/9434.md` file, if it exists, or the title of the pull request. They were also added at the bottom of the description of this pull request for easier reference. This message and the release notes originate from a call to the [release-notes-assistant](https://code.forgejo.org/forgejo/release-notes-assistant). ```diff @@ -51,2 +51,10 @@ - [x] I want the title to show in the release notes with a link to this pull request. - [ ] I want the content of the `release-notes/<pull request number>.md` to be be used for the release notes instead of the title. + +<!--start release-notes-assistant--> + +## Release notes +<!--URL:https://codeberg.org/forgejo/forgejo--> +- Features + - [PR](https://codeberg.org/forgejo/forgejo/pulls/9434): <!--number 9434 --><!--line 0 --><!--description aW1wbGVtZW50ICJjb25jdXJyZW5jeSIgYmxvY2sgaW4gRm9yZ2VqbyBBY3Rpb25zIGF0IHRoZSB3b3JrZmxvdyBsZXZlbA==-->implement "concurrency" block in Forgejo Actions at the workflow level<!--description--> +<!--end release-notes-assistant--> ``` </details> <!--start release-notes-assistant--> ## Release notes <!--URL:https://codeberg.org/forgejo/forgejo--> - Features - [PR](https://codeberg.org/forgejo/forgejo/pulls/9434): <!--number 9434 --><!--line 0 --><!--description aW1wbGVtZW50ICJjb25jdXJyZW5jeSIgYmxvY2sgaW4gRm9yZ2VqbyBBY3Rpb25zIGF0IHRoZSB3b3JrZmxvdyBsZXZlbA==-->implement "concurrency" block in Forgejo Actions at the workflow level<!--description--> <!--end release-notes-assistant-->
Author
Member

@earl-warren wrote in #9434 (comment):

I have not observed that it actually works on pull requests. Do you have a reproducer where a push to a pull request cancels the previous run?

You are right to raise a concern here. The existing behavior...

First attempts to CancelPreviousJobs if the run's Event is HookEventPullRequestSync:

// cancel running jobs if the event is push or pull_request_sync
if run.Event == webhook_module.HookEventPush ||
run.Event == webhook_module.HookEventPullRequestSync {
if err := CancelPreviousJobs(
ctx,
run.RepoID,
run.Ref,
run.WorkflowID,
run.Event,
); err != nil {
log.Error("CancelPreviousJobs: %v", err)
}
}

And then attempts to find the runs to cancel based upon querying TriggerEvent, not Event.

// CancelPreviousJobs cancels all previous jobs of the same repository, reference, workflow, and event.
// It's useful when a new run is triggered, and all previous runs needn't be continued anymore.
func CancelPreviousJobs(ctx context.Context, repoID int64, ref, workflowID string, event webhook_module.HookEventType) error {
// Find all runs in the specified repository, reference, and workflow with non-final status
runs, total, err := db.FindAndCount[actions_model.ActionRun](ctx, actions_model.FindRunOptions{
RepoID: repoID,
Ref: ref,
WorkflowID: workflowID,
TriggerEvent: event,
Status: []actions_model.Status{actions_model.StatusRunning, actions_model.StatusWaiting, actions_model.StatusBlocked},
})

When a push occurs to a branch with an open PR, the Event is HookEventPullRequestSync, causing CancelPreviousJobs to be invoked. But the TriggerEvent in the database for previous runs is actually HookEventPullRequest:

=> select id, event, trigger_event from action_run where ref = 'refs/pull/21/head' order by id;
  id  |       event       | trigger_event
------+-------------------+---------------
 1249 | pull_request      | pull_request
 1250 | pull_request_sync | pull_request
 1251 | pull_request_sync | pull_request
 1252 | pull_request_sync | pull_request
 1253 | pull_request_sync | pull_request
 1254 | pull_request_sync | pull_request
 1255 | pull_request_sync | pull_request
 1256 | pull_request_sync | pull_request
(8 rows)

And so the existing behavior is that the run generated by opening the pull request is the only one that will ever be cancelled by future pushes. Once that run is complete, subsequent pushes to the branch will not effect each other.

This seems like it is clearly a bug...

The "default" behavior in this PR works as I would expect, and as it seems was intended in the original implementation: push and pull request synchronize events cancel all previous runs on the same repository, branch, event, and workflow.

If we're OK with the change: Is it possible that we could document this as a "breaking bug fix" in the release notes on this PR? I'm not sure if that's possible with the release notes assistant to have two entries, one for a feature, and one for a breaking bug fix?

Otherwise: I could restore the broken behavior... or remove any reference to HookEventPullRequestSync and make it the cancel only apply to Push.

@earl-warren wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issuecomment-7399585: > I have not observed that it actually works on pull requests. Do you have a reproducer where a push to a pull request cancels the previous run? You are right to raise a concern here. The existing behavior... First attempts to `CancelPreviousJobs` if the run's `Event` is `HookEventPullRequestSync`: https://codeberg.org/forgejo/forgejo/src/commit/5da9c6ce64a6720702f3f3f23aac4b859d54fd52/services/actions/notifier_helper.go#L390-L402 And then attempts to find the runs to cancel based upon querying `TriggerEvent`, not `Event`. https://codeberg.org/forgejo/forgejo/src/commit/5da9c6ce64a6720702f3f3f23aac4b859d54fd52/services/actions/schedule_tasks.go#L184-L194 When a push occurs to a branch with an open PR, the `Event` is `HookEventPullRequestSync`, causing `CancelPreviousJobs` to be invoked. But the `TriggerEvent` in the database for previous runs is actually `HookEventPullRequest`: ```sql => select id, event, trigger_event from action_run where ref = 'refs/pull/21/head' order by id; id | event | trigger_event ------+-------------------+--------------- 1249 | pull_request | pull_request 1250 | pull_request_sync | pull_request 1251 | pull_request_sync | pull_request 1252 | pull_request_sync | pull_request 1253 | pull_request_sync | pull_request 1254 | pull_request_sync | pull_request 1255 | pull_request_sync | pull_request 1256 | pull_request_sync | pull_request (8 rows) ``` And so the existing behavior is that the run generated by opening the pull request is the only one that will ever be cancelled by future pushes. Once that run is complete, subsequent pushes to the branch will not effect each other. This seems like it is clearly a bug... The "default" behavior in this PR works as I would expect, and as it seems was intended in the original implementation: push and pull request synchronize events cancel all previous runs on the same repository, branch, event, and workflow. If we're OK with the change: Is it possible that we could document this as a "breaking bug fix" in the release notes on this PR? I'm not sure if that's possible with the release notes assistant to have two entries, one for a feature, and one for a breaking bug fix? Otherwise: I could restore the broken behavior... or remove any reference to `HookEventPullRequestSync` and make it the cancel only apply to `Push`.
Contributor

@mfenniak wrote in #9434 (comment):

and I'm leaning towards it's not worth it.

The absence of performance tests make me nervous on occasions. But given your analysis of what actually happens, I think it will be fine and performance testing would be overkill. That said, it would be fine by me to trust you with the requested data from code.forgejo.org if you are motivated to run tests.

@mfenniak wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issuecomment-7407070: > and I'm leaning towards it's not worth it. The absence of performance tests make me nervous on occasions. But given your analysis of what actually happens, I think it will be fine and performance testing would be overkill. That said, it would be fine by me to trust you with the requested data from code.forgejo.org if you are motivated to run tests.
Contributor

@mfenniak wrote in #9434 (comment):

If we're OK with the change: Is it possible that we could document this as a "breaking bug fix" in the release notes on this PR? I'm not sure if that's possible with the release notes assistant to have two entries, one for a feature, and one for a breaking bug fix?

It is possible to have multiple lines when using a release-notes/*.md file, which can be used for instance when upgrading a dependency that has multiple user facing changes worth a release note.

feat: mermaid: [Add support for iconify icons](https://github.com/mermaid-js/mermaid/pull/5793).
feat: mermaid: [Allow multi-line relationship labels](https://github.com/mermaid-js/mermaid/pull/5711).
feat: mermaid: [Adds architecture diagrams which allows users to show relations between services](https://github.com/mermaid-js/mermaid/pull/5452).

@mfenniak wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issuecomment-7410280: > If we're OK with the change: Is it possible that we could document this as a "breaking bug fix" in the release notes on this PR? I'm not sure if that's possible with the release notes assistant to have two entries, one for a feature, and one for a breaking bug fix? It is possible to have multiple lines when using a `release-notes/*.md` file, which can be used for instance when upgrading a dependency that has multiple user facing changes worth a release note. https://codeberg.org/forgejo/forgejo/src/commit/0a18c04a4b789d6a50267343f43c2ab5c4d0120c/release-notes/5205.md?display=source#L1-L3
Contributor

Otherwise: I could restore the broken behavior... or remove any reference to HookEventPullRequestSync and make it the cancel only apply to Push.

IMHO it is not worth restoring the broken behavior. Nice sleuth work.

> Otherwise: I could restore the broken behavior... or remove any reference to HookEventPullRequestSync and make it the cancel only apply to Push. IMHO it is not worth restoring the broken behavior. Nice sleuth work.
earl-warren requested changes 2025-09-28 12:31:11 +02:00
Dismissed
earl-warren left a comment
Contributor

A minor change request after a first review pass. I'll need to spend time on a thorough review now that I get the overall logic.

A minor change request after a first review pass. I'll need to spend time on a thorough review now that I get the overall logic.
go.mod Outdated
@ -11,3 +11,3 @@
code.forgejo.org/forgejo/levelqueue v1.0.0
code.forgejo.org/forgejo/reply v1.0.2
code.forgejo.org/forgejo/runner/v11 v11.1.1
code.forgejo.org/forgejo/runner/v11 v11.1.2-0.20250927055528-46a955d1ffec
Contributor

Warning

do not merge as is, wait for v11.1.2

> **Warning** do not merge as is, wait for v11.1.2
Author
Member

Now updated to v11.1.2. aeb0de1269..5669e49025

Now updated to v11.1.2. https://codeberg.org/forgejo/forgejo/compare/aeb0de1269efa03973257d1c95df87b7ad69f5dd..5669e4902525cb41e3e4c2943a4ea4cdb7aec836
@ -242,0 +295,4 @@
// computationally expensive on the database. To manage the risk that this might have on large-scale deployments
// When this feature is initially released, it can be disabled in the ini file by setting
// `CONCURRENCY_GROUP_QUEUE_ENABLED = false` in the `[actions]` section. If disabled, then actions with a
// concurrency group and `cancel-in-progress: false` will run simultaneously rather than being queued.
Contributor

If I understand correctly, the idea behind CONCURRENCY_GROUP_QUEUE_ENABLED is

  • provide a way for instance to switch the feature off in case they experience unforeseen performance issues
  • to make it so existing workflows do not suddenly change their behavior because this feature was previously a noop and is now doing something that may have undesirable side effects.

If I interpreted that correctly, it should be documented in custom/conf/app.example.ini

If I understand correctly, the idea behind `CONCURRENCY_GROUP_QUEUE_ENABLED` is - provide a way for instance to switch the feature off in case they experience unforeseen performance issues - to make it so existing workflows do not suddenly change their behavior because this feature was previously a noop and is now doing something that may have undesirable side effects. If I interpreted that correctly, it should be documented in `custom/conf/app.example.ini`
Author
Member

CONCURRENCY_GROUP_QUEUE_ENABLED was only intended to manage the performance risks from the database query change. It doesn't disable the entire capability of concurrency -- just "group queue" functionality. The cancellation capabilities still work.

Documentation added covering the existing implementation: !9434 (commit c52860cc3d)

"... to make it so existing workflows do not suddenly change their behavior" is not the intended use of the setting, however, it's an interesting thought. That need wouldn't be satisfied by the current setting because of the limited scope -- for example, an existing workflow with cancel-in-progress: true would be valid and run, and have its behavior change with this feature regardless of the setting.

I'm thinking about it... and I don't think it's necessary to create a setting that disables the entire feature. If someone has an existing concurrency block that suddenly becomes functional and they don't want it, they can just remove it to get the old behavior. It isn't breaking without any solution available.

`CONCURRENCY_GROUP_QUEUE_ENABLED` was only intended to manage the performance risks from the database query change. It doesn't disable the entire capability of `concurrency` -- just "group queue" functionality. The cancellation capabilities still work. Documentation added covering the existing implementation: https://codeberg.org/forgejo/forgejo/pulls/9434/commits/c52860cc3d5945beb3ad8010d7f8bd7093e68758 "... to make it so existing workflows do not suddenly change their behavior" is not the intended use of the setting, however, it's an interesting thought. That need wouldn't be satisfied by the current setting because of the limited scope -- for example, an existing workflow with `cancel-in-progress: true` would be valid and run, and have its behavior change with this feature regardless of the setting. I'm thinking about it... and I don't think it's necessary to create a setting that disables the entire feature. If someone has an existing `concurrency` block that suddenly becomes functional and they don't want it, they can just remove it to get the old behavior. It isn't breaking without any solution available.
@ -227,0 +228,4 @@
// Reaching a finalized result for a task can cause other tasks in the same concurrency group to become
// unblocked. Increasing task version here allows all applicable runners to requery to the DB for that state.
// Because it is only useful for that condition, and it has system performance risks, only enable it when
// concurrency group queuing is enabled.
Contributor

Note to self: here is a comment explaining the reason for having ConcurrencyGroupQueueEnabled

Note to self: here is a comment explaining the reason for having `ConcurrencyGroupQueueEnabled`
@ -120,2 +120,4 @@
// v39 -> v40
NewMigration("Add index for release sha1", AddIndexForReleaseSha1),
// v40 -> v41
NewMigration("Add action_run concurrency fields", AddActionRunConcurrency),
Contributor

I don't get why the upgrade test fail. 🤔

tmp/forgejo-end-to-end/forgejo-work-path-out.log:Your Forgejo database (migration version: 41) is for a newer version of Forgejo, you cannot use the newer database for this old Forgejo release (40).
I don't get why [the upgrade test fail](https://code.forgejo.org/forgejo/end-to-end/actions/runs/4158/jobs/4/attempt/2#jobstep-3-1046). 🤔 ``` tmp/forgejo-end-to-end/forgejo-work-path-out.log:Your Forgejo database (migration version: 41) is for a newer version of Forgejo, you cannot use the newer database for this old Forgejo release (40). ```
Contributor

I think it has nothing to do with this PR.

I think it has nothing to do with this PR.
earl-warren marked this conversation as resolved
Collaborator
cascading-pr updated at https://code.forgejo.org/forgejo/end-to-end/pulls/1072
Contributor

The upgrade failure of the cascade is because the tag for v14 dev is not in this branch, it can be ignored. What matters really is that the actions tests run fine and show it does not break anything.

The upgrade test will run fine once this PR is rebased.

The upgrade failure of the cascade is because the tag for v14 dev is not in this branch, it can be ignored. What matters really is that the actions tests run fine and show it does not break anything. The upgrade test will run fine once this PR is rebased.
Contributor

Unfinished thoughts.

Tthe query getConcurrencyCondition is both complex and potentially expensive. It feels like something is missing that would make it more efficient and simpler.

In the current state of the codebase CreateTaskForRunner draws the first available row from the action_run_job table and it sits in the action_task table until it is picked by a runner with the appropriate label.

With this PR the getConcurrencyCondition on the action_run row will decide which action_run_job is eligible based on the concurrency logic.

Since the concurrency rules are at the workflow level, could they be implemented so that no action_run_job is created rather than at a lower level where it regulates how an action_task is created from an action_run_job?

models/actions/run.go
193:func InsertRun(ctx context.Context, run *ActionRun, jobs []*jobparser.SingleWorkflow) error {
services/actions/notifier_helper.go
404:		if err := actions_model.InsertRun(ctx, run, jobs); err != nil {
services/actions/schedule_tasks.go
176:	if err := actions_model.InsertRun(ctx, run, workflows); err != nil {
services/actions/workflows.go
146:	return run, jobNames, actions_model.InsertRun(ctx, run, jobs)
Unfinished thoughts. Tthe query `getConcurrencyCondition` is both complex and potentially expensive. It feels like something is missing that would make it more efficient and simpler. In the current state of the codebase `CreateTaskForRunner` draws the first available row from the `action_run_job` table and it sits in the `action_task` table until it is picked by a runner with the appropriate label. With this PR the `getConcurrencyCondition` on the `action_run` row will decide which `action_run_job` is eligible based on the concurrency logic. Since the concurrency rules are at the workflow level, could they be implemented so that no `action_run_job` is created rather than at a lower level where it regulates how an `action_task` is created from an `action_run_job`? ``` models/actions/run.go 193:func InsertRun(ctx context.Context, run *ActionRun, jobs []*jobparser.SingleWorkflow) error { services/actions/notifier_helper.go 404: if err := actions_model.InsertRun(ctx, run, jobs); err != nil { services/actions/schedule_tasks.go 176: if err := actions_model.InsertRun(ctx, run, workflows); err != nil { services/actions/workflows.go 146: return run, jobNames, actions_model.InsertRun(ctx, run, jobs) ```
mfenniak force-pushed workflow-level-concurrency from 1d2ef03cf8
Some checks failed
testing / frontend-checks (pull_request) Successful in 1m17s
testing / backend-checks (pull_request) Successful in 3m18s
testing / test-unit (pull_request) Successful in 8m59s
testing / test-remote-cacher (redis) (pull_request) Successful in 1m50s
testing / test-remote-cacher (valkey) (pull_request) Successful in 1m53s
testing / test-remote-cacher (garnet) (pull_request) Successful in 1m50s
testing / test-remote-cacher (redict) (pull_request) Successful in 1m47s
testing / test-mysql (pull_request) Successful in 24m20s
testing / test-e2e (pull_request) Successful in 25m15s
testing / test-sqlite (pull_request) Successful in 28m15s
testing / test-pgsql (pull_request) Successful in 31m50s
testing / security-check (pull_request) Successful in 56s
requirements / merge-conditions (pull_request) Successful in 1s
issue-labels / backporting (pull_request_target) Has been skipped
issue-labels / release-notes (pull_request_target) Successful in 38s
issue-labels / cascade (pull_request_target) Failing after 4m54s
to 28ae1c339f
All checks were successful
issue-labels / release-notes (pull_request_target) Successful in 43s
2025-09-29 03:07:14 +02:00
Compare
Author
Member

@earl-warren wrote in #9434 (comment):

Unfinished thoughts.

The query getConcurrencyCondition is both complex and potentially expensive. It feels like something is missing that would make it more efficient and simpler.

In the current state of the codebase CreateTaskForRunner draws the first available row from the action_run_job table and it sits in the action_task table until it is picked by a runner with the appropriate label.

Just a minor correction: action_task is created after an action_run_job is picked for a runner. Before jobs are picked, there are just action_run (a workflow) and action_run_job (a job).

With this PR the getConcurrencyCondition on the action_run row will decide which action_run_job is eligible based on the concurrency logic.

Since the concurrency rules are at the workflow level, could they be implemented so that no action_run_job is created rather than at a lower level where it regulates how an action_task is created from an action_run_job?

It's an interesting thought, but I think we'd run into two problems:

  • Later on when a task is complete, you would need to query the database in virtually same way in order to find out which jobs are now available for execution.
  • Assuming we kept the existing tables action_run and action_run_job, then at this later time you would need to parse the workflow and generate the N jobs from the run into the action_run_job to make them runnable, and you'd end up with some code duplication at this time.
    • This could probably be avoided by continuing to put the jobs in the table but having a flag on them that makes them disappear from eligibility for picking.

If CreateTaskForRunner was invoked for every fetch, I think the best argument for what you're suggesting is that fetches happen more frequently than tasks finishing, and so the expensive query would happen less frequently. But with the task version layer in the API that prevents CreateTaskForRunner from being invoked for most fetches, it isn't invoked too often.

I think it's a fair risk to be concerned about the complexity and performance of the queuing database query -- that is why I created the targeted enable/disable flag. But... I think it might be a premature to be re-architecting based upon just the worry.

As Forgejo is deployed on larger systems and runs into scalability problems, I'm confident that the current implementation with CreateTaskForRunner pulling all the pending tasks and check them against the runner's labels in the application server will be a performance problem before the concurrency querying is. 😬

@earl-warren wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issuecomment-7415206: > Unfinished thoughts. > > The query `getConcurrencyCondition` is both complex and potentially expensive. It feels like something is missing that would make it more efficient and simpler. > > In the current state of the codebase `CreateTaskForRunner` draws the first available row from the `action_run_job` table and it sits in the `action_task` table until it is picked by a runner with the appropriate label. Just a minor correction: `action_task` is created after an `action_run_job` is picked for a runner. Before jobs are picked, there are just `action_run` (a workflow) and `action_run_job` (a job). > With this PR the `getConcurrencyCondition` on the `action_run` row will decide which `action_run_job` is eligible based on the concurrency logic. > > Since the concurrency rules are at the workflow level, could they be implemented so that no `action_run_job` is created rather than at a lower level where it regulates how an `action_task` is created from an `action_run_job`? It's an interesting thought, but I think we'd run into two problems: - Later on when a task is complete, you would need to query the database in virtually same way in order to find out which jobs are now available for execution. - Assuming we kept the existing tables `action_run` and `action_run_job`, then at this later time you would need to parse the workflow and generate the N jobs from the run into the `action_run_job` to make them runnable, and you'd end up with some code duplication at this time. - This could probably be avoided by continuing to put the jobs in the table but having a flag on them that makes them disappear from eligibility for picking. *If* `CreateTaskForRunner` was invoked for every fetch, I think the best argument for what you're suggesting is that fetches happen more frequently than tasks finishing, and so the expensive query would happen less frequently. But with the task version layer in the API that prevents `CreateTaskForRunner` from being invoked for most fetches, it isn't invoked too often. I think it's a fair risk to be concerned about the complexity and performance of the queuing database query -- that is why I created the targeted enable/disable flag. But... I think it might be a premature to be re-architecting based upon just the worry. As Forgejo is deployed on larger systems and runs into scalability problems, I'm confident that the current implementation with CreateTaskForRunner pulling *all* the pending tasks and check them against the runner's labels in the application server will be a performance problem before the concurrency querying is. 😬
mfenniak force-pushed workflow-level-concurrency from c52860cc3d
Some checks failed
requirements / merge-conditions (pull_request) Successful in 1s
issue-labels / release-notes (pull_request_target) Successful in 40s
testing / frontend-checks (pull_request) Successful in 1m7s
testing / backend-checks (pull_request) Failing after 1m17s
testing / test-unit (pull_request) Has been skipped
testing / test-e2e (pull_request) Has been skipped
testing / test-mysql (pull_request) Has been skipped
testing / test-pgsql (pull_request) Has been skipped
testing / test-sqlite (pull_request) Has been skipped
testing / test-remote-cacher (redis) (pull_request) Has been skipped
testing / test-remote-cacher (valkey) (pull_request) Has been skipped
testing / test-remote-cacher (garnet) (pull_request) Has been skipped
testing / test-remote-cacher (redict) (pull_request) Has been skipped
testing / security-check (pull_request) Has been skipped
to 722cb9a9ed
All checks were successful
requirements / merge-conditions (pull_request) Successful in 2s
issue-labels / release-notes (pull_request_target) Successful in 40s
testing / frontend-checks (pull_request) Successful in 1m4s
testing / backend-checks (pull_request) Successful in 3m13s
testing / test-unit (pull_request) Successful in 5m37s
testing / test-remote-cacher (redis) (pull_request) Successful in 1m56s
testing / test-remote-cacher (valkey) (pull_request) Successful in 1m55s
testing / test-remote-cacher (garnet) (pull_request) Successful in 1m56s
testing / test-remote-cacher (redict) (pull_request) Successful in 1m57s
testing / test-mysql (pull_request) Successful in 17m24s
testing / test-sqlite (pull_request) Successful in 22m33s
testing / test-pgsql (pull_request) Successful in 25m38s
testing / security-check (pull_request) Successful in 53s
testing / test-e2e (pull_request) Successful in 17m36s
2025-09-29 03:37:25 +02:00
Compare
Contributor

@mfenniak wrote in #9434 (comment):

In the current state of the codebase CreateTaskForRunner draws the first available row from the action_run_job table and it sits in the action_task table until it is picked by a runner with the appropriate label.

Just a minor correction: action_task is created after an action_run_job is picked for a runner. Before jobs are picked, there are just action_run (a workflow) and action_run_job (a job).

I stand corrected 👍

@mfenniak wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issuecomment-7423186: > > In the current state of the codebase `CreateTaskForRunner` draws the first available row from the `action_run_job` table and it sits in the `action_task` table until it is picked by a runner with the appropriate label. > > Just a minor correction: `action_task` is created after an `action_run_job` is picked for a runner. Before jobs are picked, there are just `action_run` (a workflow) and `action_run_job` (a job). I stand corrected 👍
Contributor

@mfenniak wrote in #9434 (comment):

  • Later on when a task is complete, you would need to query the database in virtually same way in order to find out which jobs are now available for execution.

But you could also do nothing different than now when a task completes. You would only do something different when a run completes. Only then would you look for other runs eligible according to the concurrency rules. And once you find one, you call notify() on it and it spawns action_run_job via actions_model.InsertRun

The existing logic of CreateTaskForRunner would not need to be changed, it would continue to pick jobs the way it currently does. Only it would not see any action_run_job for runs that are queued because of concurrency rules.

Maybe we can chat about that later, the code logic is not as clear in my head as it is in yours and I may be dreaming too much about this "let's do this at a higher level" idea. 😊

@mfenniak wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issuecomment-7423186: > * Later on when a task is complete, you would need to query the database in virtually same way in order to find out which jobs are now available for execution. But you could also do nothing different than now when a task completes. You would only do something different when a run completes. Only then would you look for other runs eligible according to the concurrency rules. And once you find one, you call notify() on it and it spawns `action_run_job` via `actions_model.InsertRun` The existing logic of `CreateTaskForRunner` would not need to be changed, it would continue to pick jobs the way it currently does. Only it would not see any `action_run_job` for runs that are queued because of concurrency rules. Maybe we can chat about that later, the code logic is not as clear in my head as it is in yours and I may be dreaming too much about this "let's do this at a higher level" idea. 😊
Contributor

@mfenniak wrote in #9434 (comment):

But... I think it might be a premature to be re-architecting based upon just the worry.

I agree. My motivation here is to simplify rather than refactor. If a refactor is needed, that would defeat the idea I'm proposing.

@mfenniak wrote in https://codeberg.org/forgejo/forgejo/pulls/9434#issuecomment-7423186: > But... I think it might be a premature to be re-architecting based upon just the worry. I agree. My motivation here is to simplify rather than refactor. If a refactor is needed, that would defeat the idea I'm proposing.
mfenniak force-pushed workflow-level-concurrency from 722cb9a9ed
All checks were successful
requirements / merge-conditions (pull_request) Successful in 2s
issue-labels / release-notes (pull_request_target) Successful in 40s
testing / frontend-checks (pull_request) Successful in 1m4s
testing / backend-checks (pull_request) Successful in 3m13s
testing / test-unit (pull_request) Successful in 5m37s
testing / test-remote-cacher (redis) (pull_request) Successful in 1m56s
testing / test-remote-cacher (valkey) (pull_request) Successful in 1m55s
testing / test-remote-cacher (garnet) (pull_request) Successful in 1m56s
testing / test-remote-cacher (redict) (pull_request) Successful in 1m57s
testing / test-mysql (pull_request) Successful in 17m24s
testing / test-sqlite (pull_request) Successful in 22m33s
testing / test-pgsql (pull_request) Successful in 25m38s
testing / security-check (pull_request) Successful in 53s
testing / test-e2e (pull_request) Successful in 17m36s
to aeb0de1269
All checks were successful
requirements / merge-conditions (pull_request) Successful in 4s
testing / frontend-checks (pull_request) Successful in 1m51s
issue-labels / release-notes (pull_request_target) Successful in 1m11s
testing / backend-checks (pull_request) Successful in 4m35s
testing / test-unit (pull_request) Successful in 7m19s
testing / test-remote-cacher (redis) (pull_request) Successful in 2m37s
testing / test-remote-cacher (valkey) (pull_request) Successful in 2m36s
testing / test-remote-cacher (garnet) (pull_request) Successful in 2m33s
testing / test-remote-cacher (redict) (pull_request) Successful in 2m1s
testing / test-mysql (pull_request) Successful in 21m7s
testing / test-e2e (pull_request) Successful in 23m23s
testing / test-sqlite (pull_request) Successful in 25m51s
testing / test-pgsql (pull_request) Successful in 29m54s
testing / security-check (pull_request) Successful in 55s
2025-10-01 02:20:33 +02:00
Compare
mfenniak force-pushed workflow-level-concurrency from aeb0de1269
All checks were successful
requirements / merge-conditions (pull_request) Successful in 4s
testing / frontend-checks (pull_request) Successful in 1m51s
issue-labels / release-notes (pull_request_target) Successful in 1m11s
testing / backend-checks (pull_request) Successful in 4m35s
testing / test-unit (pull_request) Successful in 7m19s
testing / test-remote-cacher (redis) (pull_request) Successful in 2m37s
testing / test-remote-cacher (valkey) (pull_request) Successful in 2m36s
testing / test-remote-cacher (garnet) (pull_request) Successful in 2m33s
testing / test-remote-cacher (redict) (pull_request) Successful in 2m1s
testing / test-mysql (pull_request) Successful in 21m7s
testing / test-e2e (pull_request) Successful in 23m23s
testing / test-sqlite (pull_request) Successful in 25m51s
testing / test-pgsql (pull_request) Successful in 29m54s
testing / security-check (pull_request) Successful in 55s
to 5669e49025
All checks were successful
requirements / merge-conditions (pull_request) Successful in 3s
testing / frontend-checks (pull_request) Successful in 1m1s
testing / backend-checks (pull_request) Successful in 3m18s
testing / test-unit (pull_request) Successful in 5m30s
testing / test-remote-cacher (redis) (pull_request) Successful in 1m55s
testing / test-remote-cacher (valkey) (pull_request) Successful in 1m56s
testing / test-remote-cacher (garnet) (pull_request) Successful in 1m58s
testing / test-remote-cacher (redict) (pull_request) Successful in 1m59s
testing / test-mysql (pull_request) Successful in 17m26s
testing / test-e2e (pull_request) Successful in 20m0s
testing / test-sqlite (pull_request) Successful in 22m36s
testing / test-pgsql (pull_request) Successful in 26m51s
testing / security-check (pull_request) Successful in 56s
issue-labels / backporting (pull_request_target) Has been skipped
milestone / set (pull_request_target) Successful in 5s
2025-10-03 16:14:55 +02:00
Compare
Owner

This is one of my blocker to use the runner for more jobs on out own instance 😏

This is one of my blocker to use the runner for more jobs on out own instance 😏
Sign in to join this conversation.
No reviewers
No labels
arch
riscv64
backport/v1.19
backport/v1.20
backport/v1.21/forgejo
backport/v10.0/forgejo
backport/v11.0/forgejo
backport/v12.0/forgejo
backport/v13.0/forgejo
backport/v14.0/forgejo
backport/v7.0/forgejo
backport/v8.0/forgejo
backport/v9.0/forgejo
breaking
bug
bug
confirmed
bug
duplicate
bug
needs-more-info
bug
new-report
bug
reported-upstream
code/actions
code/api
code/auth
code/auth/faidp
code/auth/farp
code/email
code/federation
code/git
code/migrations
code/packages
code/wiki
database
MySQL
database
PostgreSQL
database
SQLite
dependency-upgrade
dependency
certmagic
dependency
chart.js
dependency
Chi
dependency
Chroma
dependency
citation.js
dependency
codespell
dependency
css-loader
dependency
devcontainers
dependency
dropzone
dependency
editorconfig-checker
dependency
elasticsearch
dependency
enmime
dependency
F3
dependency
ForgeFed
dependency
garage
dependency
Git
dependency
git-backporting
dependency
Gitea
dependency
gitignore
dependency
go-ap
dependency
go-enry
dependency
go-gitlab
dependency
Go-org
dependency
go-rpmutils
dependency
go-sql-driver mysql
dependency
go-swagger
dependency
go-version
dependency
go-webauthn
dependency
gocron
dependency
Golang
dependency
goldmark
dependency
goquery
dependency
Goth
dependency
grpc-go
dependency
happy-dom
dependency
Helm
dependency
image-spec
dependency
jsonschema
dependency
KaTeX
dependency
lint
dependency
MariaDB
dependency
Mermaid
dependency
minio-go
dependency
misspell
dependency
Monaco
dependency
PDFobject
dependency
playwright
dependency
postcss
dependency
postcss-plugins
dependency
pprof
dependency
prometheus client_golang
dependency
protobuf
dependency
relative-time-element
dependency
renovate
dependency
reply
dependency
ssh
dependency
swagger-ui
dependency
tailwind
dependency
temporal-polyfill
dependency
terminal-to-html
dependency
tests-only
dependency
text-expander-element
dependency
urfave
dependency
vfsgen
dependency
vite
dependency
Woodpecker CI
dependency
x tools
dependency
XORM
Discussion
duplicate
enhancement/feature
forgejo/accessibility
forgejo/branding
forgejo/ci
forgejo/commit-graph
forgejo/documentation
forgejo/furnace cleanup
forgejo/i18n
forgejo/interop
forgejo/moderation
forgejo/privacy
forgejo/release
forgejo/scaling
forgejo/security
forgejo/ui
Gain
High
Gain
Nice to have
Gain
Undefined
Gain
Very High
good first issue
i18n/backport-stable
impact
large
impact
medium
impact
small
impact
unknown
Incompatible license
issue
closed
issue
do-not-exist-yet
issue
open
manual test
Manually tested during feature freeze
OS
FreeBSD
OS
Linux
OS
macOS
OS
Windows
problem
QA
regression
release blocker
Release Cycle
Feature Freeze
release-blocker
v7.0
release-blocker
v7.0.1
release-blocker
v7.0.2
release-blocker
v7.0.3
release-blocker
v7.0.4
release-blocker
v8.0.0
release-blocker/v9.0.0
run-all-playwright-tests
run-end-to-end-tests
test
manual
test
needed
test
needs-help
test
not-needed
test
present
untested
User research - time-tracker
valuable code
worth a release-note
User research - Accessibility
User research - Blocked
User research - Community
User research - Config (instance)
User research - Errors
User research - Filters
User research - Future backlog
User research - Git workflow
User research - Labels
User research - Moderation
User research - Needs input
User research - Notifications/Dashboard
User research - Rendering
User research - Repo creation
User research - Repo units
User research - Security
User research - Settings (in-app)
No milestone
No project
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
forgejo/forgejo!9434
No description provided.