feat!: validate workflows with a schema #170

Merged
earl-warren merged 12 commits from earl-warren/act:wip-interpolation-error into main 2025-07-12 17:03:55 +00:00
Contributor
  • add a schema and validate workflows before acting on them
  • add enable-email-notifications to the schema
  • Forgejo has support for support services..cmd
  • the forge context is equivalent to the github context
  • secrets can be used in jobs.*.if expressions
  • expressions in env can use the env context, that works

It is a breaking change because it will fail existing workflows that contain errors that were previously undetected.
The associated release note include a strategy and example to verify workflows from the command line.

That verify was applied most repositories in the following organizations and only found a few mistakes that were fixed (e.g. actions/setup-forgejo@ef6e5bef04).

There is proof that the end-to-end run is using this pull request because it failed on both enable-email-notifications and cmd in services which do not exist in GitHub Actions and were not in the schema cherry-picked from Nektos ACT.

Note that the error returned is displayed in context with the companion pull request of the runner that can be merged afterwards. Without it the error message are displayed in the pre-step output of an invalid workflow but tey are a oneliner that is very difficult to put in context unless you know exactly what it is about.

Closes forgejo/act#169
Closes forgejo/runner#317

- add a schema and validate workflows before acting on them - add enable-email-notifications to the schema - Forgejo has support for support services.<id>.cmd - the forge context is equivalent to the github context - secrets can be used in jobs.*.if expressions - expressions in env can use the env context, that works It is a breaking change because it will fail existing workflows that contain errors that were previously undetected. The associated [release note](https://code.forgejo.org/forgejo/runner/src/branch/main/RELEASE-NOTES.md#8-0-0-not-published-yet) include a strategy and example to verify workflows from the command line. That verify was applied most repositories in the following organizations and only found a few mistakes that were fixed (e.g. https://code.forgejo.org/actions/setup-forgejo/commit/ef6e5bef04fe9ed9d79702d1ccef4222ea95579d). - https://codeberg.org/forgejo - https://code.forgejo.org/forgejo - https://code.forgejo.org/forgejo-helm/ - https://code.forgejo.org/renovate/ There is proof that the end-to-end run is using this pull request because it failed on both `enable-email-notifications` and `cmd` in services which do not exist in GitHub Actions and were not in the schema cherry-picked from Nektos ACT. Note that the error returned is displayed in context with the [companion pull request of the runner that can be merged afterwards](https://code.forgejo.org/forgejo/runner/pulls/666/files). Without it the error message are displayed in the pre-step output of an invalid workflow but tey are a oneliner that is very difficult to put in context unless you know exactly what it is about. Closes forgejo/act#169 Closes forgejo/runner#317
Contributor

cascading-pr updated at forgejo/runner#663

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/663
earl-warren changed title from feat!: validate workflows with a schema to feat!: validate workflows with a schema [skip cascade] 2025-07-05 20:36:39 +00:00
earl-warren changed title from feat!: validate workflows with a schema [skip cascade] to WIP: feat!: validate workflows with a schema [skip cascade] 2025-07-05 20:41:21 +00:00
earl-warren changed title from WIP: feat!: validate workflows with a schema [skip cascade] to WIP: feat!: validate workflows with a schema 2025-07-05 20:47:34 +00:00
Contributor

cascading-pr updated at forgejo/runner#663

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/663
earl-warren force-pushed wip-interpolation-error from 0c15daba39
Some checks failed
checks / unit (pull_request) Successful in 1m13s
/ cascade (pull_request_target) Failing after 1m58s
checks / integration (pull_request) Successful in 1m3s
to 1ff7540828
Some checks failed
checks / unit (pull_request) Successful in 1m13s
/ cascade (pull_request_target) Failing after 1m28s
checks / integration (pull_request) Successful in 1m7s
2025-07-05 21:26:15 +00:00
Compare
Contributor

cascading-pr updated at forgejo/runner#663

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/663
Contributor

cascading-pr updated at forgejo/runner#663

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/663
earl-warren force-pushed wip-interpolation-error from d66c8a80ec
Some checks failed
checks / unit (pull_request) Successful in 1m19s
/ cascade (pull_request_target) Failing after 2m2s
checks / integration (pull_request) Successful in 1m7s
to 0dc6f254ac
Some checks failed
checks / unit (pull_request) Successful in 1m48s
/ cascade (pull_request_target) Failing after 2m20s
checks / integration (pull_request) Successful in 1m14s
2025-07-05 22:07:13 +00:00
Compare
Contributor

cascading-pr updated at forgejo/runner#663

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/663
earl-warren force-pushed wip-interpolation-error from 0dc6f254ac
Some checks failed
checks / unit (pull_request) Successful in 1m48s
/ cascade (pull_request_target) Failing after 2m20s
checks / integration (pull_request) Successful in 1m14s
to a97ed12456
Some checks failed
checks / unit (pull_request) Successful in 1m30s
/ cascade (pull_request_target) Failing after 2m5s
checks / integration (pull_request) Successful in 1m10s
2025-07-05 22:16:07 +00:00
Compare
Contributor

cascading-pr updated at forgejo/runner#663

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/663
earl-warren force-pushed wip-interpolation-error from a97ed12456
Some checks failed
checks / unit (pull_request) Successful in 1m30s
/ cascade (pull_request_target) Failing after 2m5s
checks / integration (pull_request) Successful in 1m10s
to 21c5eccc5e
Some checks failed
checks / unit (pull_request) Successful in 1m34s
checks / integration (pull_request) Successful in 1m37s
/ cascade (pull_request_target) Failing after 7m37s
2025-07-05 22:21:29 +00:00
Compare
Contributor

cascading-pr updated at forgejo/runner#665

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/665
earl-warren force-pushed wip-interpolation-error from 21c5eccc5e
Some checks failed
checks / unit (pull_request) Successful in 1m34s
checks / integration (pull_request) Successful in 1m37s
/ cascade (pull_request_target) Failing after 7m37s
to a8efb77605
All checks were successful
checks / unit (pull_request) Successful in 1m25s
checks / integration (pull_request) Successful in 1m8s
/ cascade (pull_request_target) Successful in 37m13s
2025-07-05 23:13:07 +00:00
Compare
Contributor

cascading-pr updated at forgejo/runner#665

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/665
earl-warren changed title from WIP: feat!: validate workflows with a schema to feat!: validate workflows with a schema 2025-07-06 09:02:00 +00:00
earl-warren force-pushed wip-interpolation-error from a8efb77605
All checks were successful
checks / unit (pull_request) Successful in 1m25s
checks / integration (pull_request) Successful in 1m8s
/ cascade (pull_request_target) Successful in 37m13s
to d196d59e62
Some checks failed
checks / unit (pull_request) Successful in 1m21s
checks / integration (pull_request) Successful in 1m12s
/ cascade (pull_request_target) Failing after 13m33s
2025-07-06 12:16:28 +00:00
Compare
Contributor

cascading-pr updated at forgejo/runner#665

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/665
earl-warren force-pushed wip-interpolation-error from d196d59e62
Some checks failed
checks / unit (pull_request) Successful in 1m21s
checks / integration (pull_request) Successful in 1m12s
/ cascade (pull_request_target) Failing after 13m33s
to a128183f74
Some checks failed
checks / unit (pull_request) Successful in 2m3s
checks / integration (pull_request) Successful in 1m31s
/ cascade (pull_request_target) Failing after 15m5s
2025-07-06 12:31:28 +00:00
Compare
Contributor

cascading-pr updated at forgejo/runner#665

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/665
earl-warren force-pushed wip-interpolation-error from a128183f74
Some checks failed
checks / unit (pull_request) Successful in 2m3s
checks / integration (pull_request) Successful in 1m31s
/ cascade (pull_request_target) Failing after 15m5s
to 34a06b01e1
All checks were successful
checks / unit (pull_request) Successful in 1m14s
checks / integration (pull_request) Successful in 1m8s
/ cascade (pull_request_target) Successful in 34m10s
2025-07-06 13:10:44 +00:00
Compare
Contributor

cascading-pr updated at forgejo/runner#665

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/665
earl-warren force-pushed wip-interpolation-error from 34a06b01e1
All checks were successful
checks / unit (pull_request) Successful in 1m14s
checks / integration (pull_request) Successful in 1m8s
/ cascade (pull_request_target) Successful in 34m10s
to a134916d8e
All checks were successful
checks / unit (pull_request) Successful in 1m45s
checks / integration (pull_request) Successful in 1m47s
/ cascade (pull_request_target) Successful in 41m3s
2025-07-07 20:32:17 +00:00
Compare
Contributor

cascading-pr updated at forgejo/runner#665

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/665
Author
Contributor

@wetneb asking for your review because you contributed a change to improve error reporting on workflows in Forgejo. This is an entirely different codebase though, feel free to decline.

@wetneb asking for your review because you contributed a change to improve error reporting on workflows in Forgejo. This is an entirely different codebase though, feel free to decline.
earl-warren force-pushed wip-interpolation-error from a134916d8e
All checks were successful
checks / unit (pull_request) Successful in 1m45s
checks / integration (pull_request) Successful in 1m47s
/ cascade (pull_request_target) Successful in 41m3s
to 65406c85e3
All checks were successful
checks / unit (pull_request) Successful in 1m15s
checks / integration (pull_request) Successful in 1m7s
/ cascade (pull_request_target) Successful in 34m51s
2025-07-11 12:12:34 +00:00
Compare
Contributor

cascading-pr updated at forgejo/runner#689

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/689
fix: secrets can be used in jobs.*.if expressions
All checks were successful
checks / unit (pull_request) Successful in 1m16s
checks / integration (pull_request) Successful in 1m7s
/ cascade (pull_request_target) Successful in 36m3s
4f8ec351da
Contributor

cascading-pr updated at forgejo/runner#689

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/689
viceice left a comment
Owner

i havn't checked the full schme yet. it's very big. we should somehow validate all our existing workflows against this

i havn't checked the full schme yet. it's very big. we should somehow validate all our existing workflows against this
@ -85,0 +91,4 @@
}).UnmarshalYAML(node); err != nil {
return err
}
type ActionDefault Action

why this type def? 🤔

why this type def? 🤔
earl-warren marked this conversation as resolved
@ -95,0 +102,4 @@
}).UnmarshalYAML(node); err != nil {
return errors.Join(err, fmt.Errorf("Forgejo Actions YAML Schema validation error"))
}
type WorkflowDefault Workflow

same here

same here
Author
Contributor

I do not know the rationale for this. It reads like a noop to me but it does not hurt. And maybe I'm missing something subtle.

🤷‍♂️

I do not know the rationale for this. It reads like a noop to me but it does not hurt. And maybe I'm missing something subtle. 🤷‍♂️

oh, I see it comes from upstream. I don't know go good enough 🫣 so probably leave it as is for compabillity

oh, I see it comes from upstream. I don't know go good enough 🫣 so probably leave it as is for compabillity
viceice marked this conversation as resolved
earl-warren changed title from feat!: validate workflows with a schema to WIP: feat!: validate workflows with a schema 2025-07-12 15:09:34 +00:00
Author
Contributor

@viceice wrote in #170 (comment):

i havn't checked the full schme yet. it's very big. we should somehow validate all our existing workflows against this

I will do that using forgejo-runner exec --event unknown --workflows on all the repos in Forgejo space. Switching to WIP in the meantime. I may take a minute.

@viceice wrote in https://code.forgejo.org/forgejo/act/pulls/170#issuecomment-46881: > i havn't checked the full schme yet. it's very big. we should somehow validate all our existing workflows against this I will do that using `forgejo-runner exec --event unknown --workflows ` on all the repos in Forgejo space. Switching to WIP in the meantime. I may take a minute.
Owner

@earl-warren wrote in #170 (comment):

@viceice wrote in #170 (comment):

i havn't checked the full schme yet. it's very big. we should somehow validate all our existing workflows against this

I will do that using forgejo-runner exec --event unknown --workflows on all the repos in Forgejo space. Switching to WIP in the meantime. I may take a minute.

sounds like a good idea.

@earl-warren wrote in https://code.forgejo.org/forgejo/act/pulls/170#issuecomment-46911: > @viceice wrote in #170 (comment): > > > i havn't checked the full schme yet. it's very big. we should somehow validate all our existing workflows against this > > I will do that using `forgejo-runner exec --event unknown --workflows ` on all the repos in Forgejo space. Switching to WIP in the meantime. I may take a minute. sounds like a good idea.
Author
Contributor

Went through all repositories in:

Found and fixed the following:

I also did not browse the entirety of the schema. I adapted it a few times (see the individual commits) and it was rather straightforward. I think it will be a major improvement for people who are faced with silent false positive when they have a typo in the name of a context.

Went through all repositories in: - https://codeberg.org/forgejo - https://code.forgejo.org/forgejo - https://code.forgejo.org/forgejo-helm/ - https://code.forgejo.org/renovate/ Found and fixed the following: - https://code.forgejo.org/forgejo/release-notes-assistant/commit/c848a4db31387c1f378c6e5fc2bb699ae29b96be I also did not browse the entirety of the schema. I adapted it a few times (see the individual commits) and it was rather straightforward. I think it will be a major improvement for people who are faced with silent false positive when they have a typo in the name of a context.
feat: expressions in env can use the env context, that works
Some checks failed
checks / unit (pull_request) Successful in 1m17s
checks / integration (pull_request) Successful in 1m10s
/ cascade (pull_request_target) Failing after 56s
2dd9f34c9e
Contributor

cascading-pr updated at forgejo/runner#689

cascading-pr updated at https://code.forgejo.org/forgejo/runner/pulls/689
earl-warren changed title from WIP: feat!: validate workflows with a schema to feat!: validate workflows with a schema 2025-07-12 15:53:07 +00:00
Author
Contributor

The description was updated to reflect the latest developments.

The description was updated to reflect the latest developments.
viceice approved these changes 2025-07-12 16:07:02 +00:00
Author
Contributor

That's a big breaking change, there will be consequences. But I'm cautiously optimistic. Thanks for taking the time to review, it was not trivial.

That's a big breaking change, there will be consequences. But I'm cautiously optimistic. Thanks for taking the time to review, it was not trivial.
earl-warren deleted branch wip-interpolation-error 2025-07-12 17:03:55 +00:00
Commenting is not possible because the repository is archived.
No reviewers
No milestone
No project
No assignees
3 participants
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
forgejo/act!170
No description provided.