CRON job to trigger daily build of ros2 images by mikaelarguedas · Pull Request #348 · osrf/docker_images

mikaelarguedas · 2019-12-22T18:55:51Z

ros2 images have not been rebuilt since Dec 4th (20 days ago).
It was suggested in the past to have a cron job to trigger these builds daily to not rely on the currently flaky setup. The PR for it has been blocked for a while #312

This PR implements a cron job using Github Actions thus allowing us to use it without extra permissions given to a third party app.

How it compares to #312:

the secrets have to be defined in this repo. They are encrypted by github and filtered out of the logs
- cons:
  - any matching string in the logs will be redacted, so it could actually leak password if the same string is printed to the logs from code (though unlikely). I believe the logs are not public, so leaking is prevented
  - Logs are not public?
the images are built in the actions and not on dockerhub:
- pro:
  - we get notified on build failure
  - the job starts instantly
- cons:
  - slower (maybe?) as pulling, building and pushing will require some bandwidth and latency is likely lower on dockerhub
  - I'm discovering github actions and hopefully things can be optimized and improved later on

Leaving this in draft for a couple days the time it proves to run reliably at https://github.com/mikaelarguedas/docker_images/actions

ruffsl · 2019-12-23T19:22:43Z

I see you've opted to have the docker engine in the CI build in image, instead of that of the automated registry. One reason in #312 that I opted to use the registry API to trigger the build was that it would subsequent trigger any automated repo build pointed at osrf/ros2 repo upon success. Could you verify that pushing remotely to the registry instead still triggers downstream docker hub repos to rebuild?

mikaelarguedas · 2019-12-23T20:33:29Z

was that it would subsequent trigger any automated repo build pointed at osrf/ros2 repo upon success

Oh right, it's definitely a potential downside, I havent tried it but I would expect dockerhub to trigger downstream repo only from successful automated builds (and not from image pushes).
I'll give it a shot soon and report here.

If we end up having to back to triggering them instead of pushing them, just saw that triggering nightly yeasterday did not trigger the downstream nightly-rmw* images .. So we'll need to trigger all images from the cron job.

ruffsl · 2019-12-23T23:10:49Z

just saw that triggering nightly yeasterday did not trigger the downstream nightly-rmw* images

Hmm, that was not what I've seen in the past, as this post_push hook was working when I setup the the docker hub repo settings. You triggered the nightly tag, yet on build/push completion by docker hub, the webhook did not trigger/cascade to build the next tag?

docker_images/ros2/nightly/nightly/hooks/post_push

Lines 1 to 4 in c857f6c

    
           #!/bin/bash 
        
           curl -H "Content-Type: application/json" \ 
        
               --data '{"docker_tag": "nightly-rmw"}' \ 
        
               -X POST ${POST_PUSH_TRIGGER}

ruffsl · 2019-12-23T23:16:50Z

Looks like the trigger api URL has migrated from https://cloud.docker.com/api/build/v1/source/.../call to https://hub.docker.com/api/build/v1/source/.../call.

I've updated the URL in the POST_PUSH_TRIGGER for the BUILD ENVIRONMENT VARIABLES.

ruffsl · 2019-12-23T23:24:24Z

Also, looks like the github link to the osrf/docker_images repo has been unset. It doesn't seem like I have the github permissions to re-link it: ping @tfoote or @nuclearsandwich

This perhaps explains why my fork with the CircleCI cron job (using #312) triggering was ineffective while still correctly calling the API daily.

https://circleci.com/gh/ruffsl/docker_images/tree/master

mikaelarguedas · 2019-12-24T08:16:14Z

Also, looks like the github link to the osrf/docker_images repo has been unset.

This seem to be fixed

mikaelarguedas · 2019-12-24T11:50:40Z

was that it would subsequent trigger any automated repo build pointed at osrf/ros2 repo upon success

It actually works, downstream repos get triggered when new mages are pushed 🎉. tested on this repo

(unrelated to the push/build on dockerhub)Note that it works only with repos where the base image is explicitly set: FROM osrf/ros2:devel. Doesn't work for files that have a configurable base image

ARG FROM_IMAGE=osrf/ros2:devel
FROM $FROM_IMAGE

One possible downside of this approach if that the build logs will not be on docker hub anymore. Not a big bummer though

I also thought a bit about the triggering approach and a few questions come up compared to this github actions approach:

How would we confirm that the build trigger worked? (to avoid surprises like CRON job to trigger daily build of ros2 images #348 (comment))
How would we get notified if the automated build fails? (there's no webhook for build failure on DockerHub).
- Potential solution: We can enable email notifications on build failure. These are per organization and not per repo, so we'll receive email for any failed builds on the osrf org (currently the only one failing is sros). It may be worth enabling regardless of the approach we take.

mikaelarguedas · 2019-12-26T15:26:30Z

Potential solution: We can enable email notifications on build failure. These are per organization and not per repo, so we'll receive email for any failed builds on the osrf org (currently the only one failing is sros). It may be worth enabling regardless of the approach we take.

#350 fixes the build of the sros image

Signed-off-by: Mikael Arguedas <[email protected]>

mikaelarguedas · 2020-01-17T18:46:10Z

Follow-up of offline discussion:

Email notifications on dockerhub have been enabled after thumbsup from Tully
Made a change to go back to triggering builds on dockerhub insteal of building in github actions mikaelarguedas@88e3569

I'll let it run for a day or 2 before forcepushin the change to this branch

ruffsl · 2020-01-17T22:34:33Z

I'll let it run for a day or 2 before forcepushin the change to this branch

Ok, I've just now disabled the CircleCI on my fork to check yours is working fine.

mikaelarguedas · 2020-01-19T02:11:46Z

Ok, I've just now disabled the CircleCI on my fork to check yours is working fine.

👍 Seems to have ran successfully today.

Regarding the implementation. I couldn't get curl to expand $dockerTag if it was provided as an env var in the build matrix. So I had to duplicate the jobs... Now this workflow has 2 jobs one to trigger the devel image and one for the nightly one

Signed-off-by: Mikael Arguedas <[email protected]>

mikaelarguedas · 2020-01-22T11:05:51Z

Relevant secret added to this repo.

This is now ready for review

ruffsl

LGTM, just a nitpick that might help release the nightly images sooner together. If it gets complex for actions to have different cron job times for the two, then nevermind.

Signed-off-by: Mikael Arguedas <[email protected]>

This was referenced Dec 22, 2019

Adding a source tag to ROS2 automated repo #294

Closed

Use cron jobs from Circleci to trigger Docker Hub builds #312

Closed

mikaelarguedas force-pushed the docker_cron_job_to_pr branch from 17ec903 to 5e7437b Compare December 29, 2019 09:46

mikaelarguedas marked this pull request as ready for review December 29, 2019 09:46

triggering builds on dockerhub

88e3569

Signed-off-by: Mikael Arguedas <[email protected]>

mikaelarguedas force-pushed the docker_cron_job_to_pr branch from 5e7437b to 88e3569 Compare January 19, 2020 02:09

update secret name

b45cf69

Signed-off-by: Mikael Arguedas <[email protected]>

mikaelarguedas requested a review from ruffsl January 22, 2020 11:06

mikaelarguedas changed the title ~~CRON job to build and publish ros2 images~~ CRON job to trigger daily build of ros2 images Jan 22, 2020

ruffsl reviewed Jan 22, 2020

View reviewed changes

Comment thread .github/workflows/cron.yaml Outdated

ruffsl approved these changes Jan 22, 2020

View reviewed changes

mikaelarguedas added 2 commits January 22, 2020 21:34

trigger devel at different time than nightly

c788ed3

Signed-off-by: Mikael Arguedas <[email protected]>

add trigger for push on specific subpaths

f0a676e

Signed-off-by: Mikael Arguedas <[email protected]>

mikaelarguedas merged commit b5f4849 into osrf:master Jan 23, 2020

mikaelarguedas deleted the docker_cron_job_to_pr branch January 23, 2020 10:38

Conversation

mikaelarguedas commented Dec 22, 2019

Uh oh!

ruffsl commented Dec 23, 2019

Uh oh!

mikaelarguedas commented Dec 23, 2019

Uh oh!

ruffsl commented Dec 23, 2019

Uh oh!

ruffsl commented Dec 23, 2019

Uh oh!

ruffsl commented Dec 23, 2019

Uh oh!

mikaelarguedas commented Dec 24, 2019

Uh oh!

mikaelarguedas commented Dec 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mikaelarguedas commented Dec 26, 2019

Uh oh!

mikaelarguedas commented Jan 17, 2020

Uh oh!

ruffsl commented Jan 17, 2020

Uh oh!

mikaelarguedas commented Jan 19, 2020

Uh oh!

mikaelarguedas commented Jan 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ruffsl left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mikaelarguedas commented Dec 24, 2019 •

edited

Loading

mikaelarguedas commented Jan 22, 2020 •

edited

Loading