Skip to content

Conversation

@mschoettle
Copy link
Contributor

@mschoettle mschoettle commented Dec 2, 2024

Description

Build and push the devel tag for new commits on master. Further work is completed at #512.

Related Issue

As discussed in #504 (comment)

Motivation and Context

I wanted to run the latest version locally and struggled to get it done. Since we discussed doing it in CI I went ahead and added another workflow based on a copy of release.docker.yml.

How Has This Been Tested?

Not yet since it only runs on master. We could temporarily add pull_request to on to see if it runs successfully.

Screenshots (if appropriate):

@semanticdiff-com
Copy link

Review changes with  SemanticDiff

@mschoettle mschoettle marked this pull request as draft December 2, 2024 21:35
@mschoettle
Copy link
Contributor Author

I tried it out locally by passing --build-arg SERVER_VERSION=devel which does not work because it then resolves into vdevel and there's is also no release. I think the best would be to add a build step to the devel dockerfile.

@joseluisq
Copy link
Collaborator

We could temporarily add pull_request to on to see if it runs successfully.

I'm fine with it because we need to test this anyway.

I tried it out locally by passing --build-arg SERVER_VERSION=devel which does not work because it then resolves into vdevel and there's is also no release. I think the best would be to add a build step to the devel dockerfile.

First, you are not compiling the project in the workflow so you have to add that step before start building the Docker image.
The reason why release.docker.yml works is because release.yml does it first, so release.docker.yml just needs to download the binaries per target.
We will need to replicate something like that for release.docker.devel.yml as well.

Aside, I thought you wanted to use the devel Dockerfiles? Do you think we need to build all targets on every merge or can the amd64 be enough?
The devel Dockefiles are amd64 already so we can just compile once, copy the binary (it does it) and push it. Otherwise, we have to go a bit more complicated and perform cross-compilation and replicate several steps of release.yaml workflow for all targets and modify the Dockerfile to use accordingly.

In my opinion, just building for amd64 should be fine per merge for now. Maybe we can try that out first and enhance it in the future with more targets if needed.

--

Additionally, we don't always want to trigger a devel build (e.g. if we changed a test, doc or another file). So let's restrict this workflow to source code only.

Note that for now, we can comment the paths section to be able to test it on pull_request as discussed. Before the merge, we can enable it again and remove the pull_request action.

name: release-docker
on:
  push:
    branches:
      - master
    paths:
      - release.docker.devel.yml
      - .cargo/config.toml
      - Cargo.lock
      - Cargo.toml
      - src/**

@joseluisq joseluisq added v2 v2 release enhancement New feature or request ci Related to CI/CD labels Dec 3, 2024
@mschoettle
Copy link
Contributor Author

Thanks for the feedback, @joseluisq.

First, you are not compiling the project in the workflow so you have to add that step before start building the Docker image.
The reason why release.docker.yml works is because release.yml does it first, so release.docker.yml just needs to download the binaries per target.
We will need to replicate something like that for release.docker.devel.yml as well.

Aside, I thought you wanted to use the devel Dockerfiles?

The reason why I tried to use the other Dockerfiles is that I saw it contains a build stage. I overlooked that it doesn't actually build but downloads the release.

So I will switch to the devel Dockerfiles. In theory it should be possible to have a multi-step Dockerfile that has a build stage where the project is compiled.

Do you think we need to build all targets on every merge or can the amd64 be enough?

amd64 should be sufficient, at least for now. My machine is arm64 but to run devel I can explicitly request amd64 and run that instead. It's probably not a common case to run the devel tag anyway. I was only planning to run it to test the redirects fix on my website.

@mschoettle
Copy link
Contributor Author

@joseluisq

I am at the point where I can't continue anymore. The login to Docker hub fails (commented out right now) and the push of the image fails with the token having insufficient scopes (although I am currently investigating the tags since it seems to be incorrect).

I added building of the binary which is passed via cache to the other two jobs (building debian and scratch). I left the alpine image out because it requires a musl binary. Looking at the Makefile it seems that the musl binary is used for all three images?

I also updated the versions of the actions to their latest major version.

@mschoettle
Copy link
Contributor Author

I am also testing building in the devel Dockerfile but there seem to be some system packages missing. Do you know which packages are required?

FROM rust:1.83.0-alpine AS build

ENV RUST_BACKTRACE=1

RUN apk add --no-cache clang lld musl-dev

WORKDIR /app

COPY ./src ./src
COPY Cargo.toml .
COPY Cargo.lock .

RUN cargo build --locked --bin static-web-server -vv --release --features=all

@joseluisq
Copy link
Collaborator

I am at the point where I can't continue anymore. The login to Docker hub fails (commented out right now) and the push of the image fails with the token having insufficient scopes (although I am currently investigating the tags since it seems to be incorrect).

Your username simply does not have permission to push packages to ghcr.io/static-web-server. What I saw is that you can specify those permissions in the Job section. See https://docs.github.com/en/actions/use-cases-and-examples/publishing-packages/publishing-docker-images#publishing-images-to-github-packages

permissions:
      contents: read
      packages: write

I added building of the binary which is passed via cache to the other two jobs (building debian and scratch). I left the alpine image out because it requires a musl binary. Looking at the Makefile it seems that the musl binary is used for all three images?

Yes, musl-based binaries are used on Alpine Docker images.
Please do not guide you from Makefile tasks (I mean you can but this is not used on the build system (I'm only the user and some tasks like building are a bit messy). Instead, take release.yaml and the Debian/Scratch Dockerfiles available as a reference.

About Alpine, let's forget about it for now and concentrate on debian/scratch only.

I am also testing building in the devel Dockerfile but there seem to be some system packages missing. Do you know which packages are required?

Yes, you will need some setup and dependencies. But I wouldn't try that. We have GitHub Actions with Ubuntu so let's keep using that. It is just fine to use and should compile without any issues for Linux amd64.
Basically, the approach of copying the binary and building the Docker image is fine to me.

--

I quickly looked at the PR release.docker.devel.yml and I saw extra tags that wanted to be pushed. Let's just focus on pushing a simple/static devel tag.

I'm also fine with pushing to ghcr.io only for now. We don't need to push to Docker Hub, maybe in the future if we need it.

@mschoettle
Copy link
Contributor Author

Your username simply does not have permission to push packages to ghcr.io/static-web-server. What I saw is that you can specify those permissions in the Job section. See https://docs.github.com/en/actions/use-cases-and-examples/publishing-packages/publishing-docker-images#publishing-images-to-github-packages

Thanks. That is true. It's not being applied though. Maybe because it is coming from a fork? See: https://github.com/static-web-server/static-web-server/actions/runs/12248375532/job/34167957103?pr=508#step:1:19

I found some posts that mention changing the setting in the repo to "read and write" by default. Although that changes it for everything I believe.

About Alpine, let's forget about it for now and concentrate on debian/scratch only.

Sounds good.

Basically, the approach of copying the binary and building the Docker image is fine to me.

Sounds good.

I quickly looked at the PR release.docker.devel.yml and I saw extra tags that wanted to be pushed. Let's just focus on pushing a simple/static devel tag.

Yes, it's because I reused the metadata action which didn't work with master because this is a PR. So for testing purposes I used that. I changed it so that the tag is hardcoded now.

I'm also fine with pushing to ghcr.io only for now. We don't need to push to Docker Hub, maybe in the future if we need it.

That's fine. I removed the login to dockerhub. You might want to consider removing the old devel tags on Docker Hub though (last updated 4 months ago): https://hub.docker.com/r/joseluisq/static-web-server/tags?name=devel

@mschoettle mschoettle marked this pull request as ready for review December 10, 2024 02:51
Copy link
Collaborator

@joseluisq joseluisq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. That is true. It's not being applied though. Maybe because it is coming from a fork? See: https://github.com/static-web-server/static-web-server/actions/runs/12248375532/job/34167957103?pr=508#step:1:19

Yes, what I'm going to do is branch out from master and merge this PR onto that new branch (master-ci) just for verification purposes and conduct several tests there. I will manage this.

Just address the PR comments and switch to master-ci as the merging target. The rest of the stuff looks fine to me.
When done, then I will merge it and start testing the changes in a new PR.

@joseluisq joseluisq changed the base branch from master to master-ci December 10, 2024 06:27
@mschoettle mschoettle requested a review from joseluisq December 10, 2024 10:45
Copy link
Collaborator

@joseluisq joseluisq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@joseluisq joseluisq merged commit f181f08 into static-web-server:master-ci Dec 10, 2024
@mschoettle mschoettle deleted the build-devel-image branch December 10, 2024 14:23
joseluisq added a commit that referenced this pull request Dec 10, 2024
)

* chore: build and push devel image on master changes via ci workflow (#508) by @mschoettle

* ci: build and push devel image on master

* Temporarily enable devel docker release on pull request

* add build release

* ci: build only amd64 and use devel dockerfiles

* temporarily disable login to docker hub

* fix tag

* add cache for binary

* test release binary

* move binary to the right place

* update action versions

* disable docker hub image

* test tags

* test tags

* test tags

* add permissions to scratch job

* apply feedback

* remove test step

* Address comments

* refactor: prefer job matrix strategy and x86_64-unknown-linux-musl

---------

Co-authored-by: Matthias Schoettle <[email protected]>
@joseluisq
Copy link
Collaborator

@mschoettle FYI I finished the new workflow and tested it successfully on both branches. See #512 description and comments for more details.

@mschoettle
Copy link
Contributor Author

@joseluisq Thanks a lot for finishing it! And thanks for adding the co-authored-by :)

@joseluisq joseluisq modified the milestones: v2.34.1, v2.35.0 Jan 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Related to CI/CD enhancement New feature or request v2 v2 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants