Skip to content

fix(ci): upload main firmware to S3 again when no seeder runs - #29271

Open
fjvbn2003 wants to merge 1 commit into
PX4:mainfrom
fjvbn2003:fix-ci-s3-upload-no-seeder
Open

fjvbn2003 wants to merge 1 commit into
PX4:mainfrom
fjvbn2003:fix-ci-s3-upload-no-seeder

Conversation

@fjvbn2003

@fjvbn2003 fjvbn2003 commented Oct 16, 2026 •

Copy link
Copy Markdown

Solved Problem

Since #28278, the Upload Artifacts job of Build all targets is skipped on every push where no seeder runs. On main that is almost every push, so Firmware/master/ on S3 (the developer firmware QGC offers) went without an update from 2026-08-18 to 2026-11-06, and has not been updated by any later push.

#28278 made the seed stage conditional on has_cold. On a normal main push every family cache is warm, so all seeders are skipped. setup copes with that through !failure() && !cancelled(), but artifacts has no status function in its if:, so GitHub adds an implicit success(), and that is false when any upstream job was skipped (actions/runner#491, actions/runner#2205).

The run history shows when it started:

https://px4-travis.s3.amazonaws.com/Firmware/master/px4_fmu-v6x_default.px4 reported Last-Modified: Tue, 18 Aug 2026 18:02:20 GMT (the end of run 32166883191) until that run, and now reports Fri, 06 Nov 2026 16:14:59 GMT.

Version tags cut from main are affected as well. filter_cold_seeders.py probes the refs/heads/main cache scope for every ref (this workflow does not set SEEDER_PROBE_SCOPE), so a tag made after #28278 finds warm caches and skips the seeders. It then loses both the S3 upload and the draft GitHub Release, because Upload Artifacts to GitHub Release is a step of the same artifacts job. v1.18.0-rc1 (34518662756) still seeded and uploaded only because release/1.18 branched before #28278, and its seed job has no condition.

Solution

Give artifacts an explicit condition, !cancelled() && needs.setup.result == 'success' && (<existing branch/tag check>). This is the same condition the deploy job in dev_container.yml uses. The job still only runs when every build leg succeeded. A failed seeder still stops it, because setup is then skipped. PR builds are unchanged because their branch name is pr-<N>.

Changelog Entry

For release notes:

Bugfix: CI uploads main branch firmware (Firmware/master/ on S3) again

Alternatives

Adding always() alone would also let the job start, but it would upload even when a build leg failed. Checking needs.setup.result keeps the old "upload only when every build succeeded" rule.

Test coverage

I could not run the changed workflow myself, because the upload job only runs on pushes to this repository. The first main push after merging should show Upload Artifacts running and a new Last-Modified on the S3 objects, and the next version tag should create its draft GitHub Release again.

Context

actions/runner#491, actions/runner#2205 (implicit success() with skipped needs).

Assisted-by: Claude:claude-opus-5-5

@github-actions github-actions Bot added kind:bug Something is broken or behaving incorrectly. scope:infrastructure CI workflows, checks, automation, or build infrastructure. labels Oct 16, 2026
@farhangnaderi
farhangnaderi requested a review from mrpollo October 16, 2026 19:17

@Saibernard Saibernard left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked this against the run history and the bucket. The condition is right!

The upload did run once since August. On 6 November a cold imx9 cache made the seed job run (run 42244062992) and it uploaded, so the master build on S3 is from that day now.

Tags cut from main are affected as well. filter_cold_seeders.py probes the main cache scope for every ref, so a tag made after #28278 finds a warm cache, skips the seed and loses the S3 upload and the GitHub release upload with it, that step is in the same job. v1.18.0-rc1 only seeded because release/1.18 branched before #28278 and its seed job has no condition. Could you update the description for these two?

And deploy-aws isn't quite the same condition, it uses always(). The deploy job in dev_container.yml is the exact match if you want to point at one.

@fjvbn2003

Copy link
Copy Markdown
Author

Thanks for checking this against the run history and the bucket. All three points are right, and I confirmed each one:

  • Run 42244062992 (243f20a, 2026-11-06) ran Seed [imx9] and Upload Artifacts, and Firmware/master/px4_fmu-v6x_default.px4 now reports Last-Modified: Fri, 06 Nov 2026 16:14:59 GMT.
  • filter_cold_seeders.py defaults to the refs/heads/main scope and build_all_targets.yml does not set SEEDER_PROBE_SCOPE, so a tag from main skips the seeders, and Upload Artifacts to GitHub Release is in the same artifacts job. The seed job on release/1.18 has no if:.
  • deploy-aws starts with always(). deploy in dev_container.yml has exactly !cancelled() && needs.setup.result == 'success' && (...).

I updated the description for all three. The commit message repeats two of the same statements (the deploy-aws comparison and "tag pushes were unaffected"). Since review has started I did not rewrite the commit, but I can push a reworded one if you prefer.

@mrpollo

mrpollo commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

I'll take a look.

@Saibernard

Copy link
Copy Markdown
Contributor

Thanks for updating it! Yes, I'd reword the commit as well so it matches the description.

Since the seed stage became conditional (PX4#28278), it is skipped on every
push where all chip family caches are already warm, which is almost
every push to main. The artifacts job had no status function in its
condition, so GitHub applied an implicit success(), which is false when
any upstream job was skipped. Since then Upload Artifacts has only run
on main when a seeder happened to run (once, on 2026-11-06, when the
imx9 cache was cold).

Version tags cut from main are affected the same way.
filter_cold_seeders.py probes the main cache scope for every ref, so
such a tag skips the seeders and loses both the S3 upload and the draft
GitHub Release, which is a step of the same job.

Check the build result explicitly, as the deploy job in
dev_container.yml already does.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: YoungjuKim <[email protected]>
@fjvbn2003
fjvbn2003 force-pushed the fix-ci-s3-upload-no-seeder branch from f096e04 to 52ed7a6 Compare November 11, 2026 23:33
@fjvbn2003

Copy link
Copy Markdown
Author

Done. I reworded the commit (now 52ed7a6) so it matches the description: the one upload on 2026-11-06, tags cut from main losing the S3 upload and the draft release, and deploy in dev_container.yml as the matching condition. The diff is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:bug Something is broken or behaving incorrectly. scope:infrastructure CI workflows, checks, automation, or build infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants