allow exporting to github cache backend by tonistiigi · Pull Request #535 · docker/buildx

tonistiigi · 2021-02-06T08:21:43Z

Client-side support for moby/buildkit#1974

--cache-to type=gha[,scope=foo]
--cache-from type=gha[,scope=foo]

If called outside github environment and required env are not present then the configuration will be skipped. Token/URL can also be passed directly in csv but I would expect only a program wrapping buildx to use that method.

Signed-off-by: Tonis Tiigi [email protected]

tonistiigi · 2021-03-23T00:27:35Z

Added caps detection so that type=gha can be used even if current buildkit does not expose that backend without getting a fatal error. We should be able to merge this PR now and make it easier to test buildkit PR.

depends on #565

Signed-off-by: Tonis Tiigi <[email protected]>

tonistiigi · 2021-03-23T16:00:38Z

I've taken this out of draft. If the caps detection works and we are ok with the ux we can merge this and continue testing/updating the Buildkit PR.

crazy-max

Ux is fine to me to start with. Thanks

JanJakes · 2021-04-29T09:51:02Z

Is there any way to use this with https://github.com/docker/setup-buildx-action yet? Or do we need to wait when Buildx is tagged?

douglasduteil · 2021-04-30T09:29:29Z

Are we waiting for 0.6.0 version @crazy-max ?
Or...

Is it already working somehow

https://github.com/crazy-max/test-docker-action/blob/46a3fac1d1df70169dc7f72ef93b500e762e8a03/.github/workflows/gha-cache.yml#L44-L53

        name: Build
        uses: docker/build-push-action@master
        with:
          context: .
          file: ./Dockerfile
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.docker_meta.outputs.tags }}
          labels: ${{ steps.docker_meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha

crazy-max · 2021-04-30T09:38:58Z

@JanJakes @douglasduteil We need the server-side first moby/buildkit/pull/1974

Is it already working somehow

I was using a staging build to test the feature.

cdeneen · 2021-06-25T17:22:14Z

Does anyone have a complete example of using this, are you still required to setup the cache in an action?

adrienjoly · 2021-06-26T15:59:08Z

Here's how I ended up caching of our "admin-api" Docker layers, on GitHub Actions:

  e2e:
    name: End-to-end tests against Docker containers
    runs-on: ubuntu-latest
    env:
      DOCKER_BUILDKIT: '1' # to enable persistent docker cache
      COMPOSE_DOCKER_CLI_BUILD: '1' # so docker-compose commands benefits from buildkit/buildx's docker cache
    steps:
      - uses: actions/checkout@v2

      # Use GitHub's Docker registry to cache layers and reduce image build time from a run to the other
      - run: echo ${{ secrets.GITHUB_TOKEN }} | docker login docker.pkg.github.com -u $GITHUB_ACTOR --password-stdin
      - run: docker pull docker.pkg.github.com/$GITHUB_REPOSITORY/admin-api-cached-build || true
      - run: docker build . --file Dockerfile-local --tag admin-api --cache-from=docker.pkg.github.com/$GITHUB_REPOSITORY/admin-api-cached-build --build-arg BUILDKIT_INLINE_CACHE=1
      - run: docker tag admin-api docker.pkg.github.com/$GITHUB_REPOSITORY/admin-api-cached-build && docker push docker.pkg.github.com/$GITHUB_REPOSITORY/admin-api-cached-build || true
      # source: https://dev.to/dtinth/caching-docker-builds-in-github-actions-which-approach-is-the-fastest-a-research-18ei

      - name: 'Build admin-api with docker-compose' # so the test suite does not have to
        run: docker-compose -f docker-compose-local.yml build admin-api
      - name: Run end-to-end tests
        run: yarn test:e2e:docker # starts the APIs and services with docker-compose before running tests

It adds and updates the resulting Docker image to the "Packages" section of our GitHub repository.

Hope this helps.

valentijnscholten · 2021-06-26T17:25:47Z

@adrienjoly this PR is about a new "GitHub native"cache backend being implemented that should be faster / better with regards to expiration etc. Caching to github package registry is kinda old school and slow in my experience.

adrienjoly · 2021-06-27T08:36:57Z

My bad, thanks for the clarification.

cdeneen · 2021-06-28T15:23:17Z

Yeah, I was just looking to see the entire cache system in use in an example. @douglasduteil posted a piece of code from @crazy-max but the repo is no longer exists or is public.

name: Build
        uses: docker/build-push-action@master
        with:
          context: .
          file: ./Dockerfile
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.docker_meta.outputs.tags }}
          labels: ${{ steps.docker_meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha

crazy-max · 2021-06-28T15:33:26Z

Will be available when moby/buildkit#1974 is merged.

benesch · 2021-07-13T14:56:26Z

If anyone else is looking to give this a shot without having to build docker buildx from source, I put together https://github.com/benesch/setup-docker-caching. It can set up buildx/Buildkit with the new GHA backend in <10s. It also exports the ACTIONS_ env vars so that you can use docker build directly instead of going through docker/build-push-action.

patroza · 2021-07-13T18:53:15Z

If anyone else is looking to give this a shot without having to build docker buildx from source, I put together https://github.com/benesch/setup-docker-caching. It can set up buildx/Buildkit with the new GHA backend in <10s. It also exports the ACTIONS_ env vars so that you can use docker build directly instead of going through docker/build-push-action.

Thanks.

Im getting:

time="2021-07-13T18:39:55Z" level=warning msg="No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load"
15
error: failed to solve: rpc error: code = Unknown desc = url not set for github actions cache

crazy-max · 2021-07-13T18:55:31Z

@patroza You can test the gha cache exporter using this workflow from the official Build Push Action while waiting for buildx 0.6 and BuildKit 0.9 to be GA.

See docker/build-push-action#406, docker/buildx#535.

Changes: * https://github.com/docker/buildx/releases/tag/v0.6.0 * https://github.com/docker/buildx/releases/tag/v0.6.1 * https://github.com/docker/buildx/releases/tag/v0.6.2 * https://github.com/docker/buildx/releases/tag/v0.6.3 * https://github.com/docker/buildx/releases/tag/v0.7.0 Significantly, this introduces support for GHA caches, see docker/buildx#535 (this will require separate changes). Signed-off-by: Stephen Kitt <[email protected]>

This was referenced Mar 22, 2021

caps: add contains method moby/buildkit#2031

Merged

vendor: update buildkit to 8effd45b #565

Merged

tonistiigi force-pushed the github-actions-cache branch from 74f47e8 to 6de12ae Compare March 23, 2021 00:24

malobre mentioned this pull request Mar 23, 2021

Pretty big cache docker/build-push-action#252

Open

tonistiigi added 2 commits March 23, 2021 08:58

allow exporting to github cache backend

3fef64f

Signed-off-by: Tonis Tiigi <[email protected]>

gha cache caps detection

d7964be

Signed-off-by: Tonis Tiigi <[email protected]>

tonistiigi force-pushed the github-actions-cache branch from 6de12ae to d7964be Compare March 23, 2021 15:59

tonistiigi marked this pull request as ready for review March 23, 2021 15:59

crazy-max approved these changes Mar 23, 2021

View reviewed changes

AkihiroSuda approved these changes Apr 8, 2021

View reviewed changes

tonistiigi merged commit 5ca0cbf into docker:master Apr 9, 2021

Niek mentioned this pull request Apr 12, 2021

Add cache cleanup option to local cache exporter moby/buildkit#1896

Open

crazy-max mentioned this pull request Jul 12, 2021

Example with gha cache #670

Merged

crazy-max mentioned this pull request Jul 13, 2021

Cache backend API example docker/build-push-action#406

Merged

crazy-max mentioned this pull request Jul 23, 2021

fix: should ignore nil client #686

Merged

jrheard mentioned this pull request Aug 1, 2021

tweak github actions setup so that docker layers are cached better jrheard/rask#8

Open

lopopolo added a commit to artichoke/docker-artichoke-nightly that referenced this pull request Aug 7, 2021

Add GitHub Actions cache mode to docker/build-push-action

87f6b22

See docker/build-push-action#406, docker/buildx#535.

lopopolo mentioned this pull request Aug 7, 2021

Add GitHub Actions cache mode to docker/build-push-action artichoke/docker-artichoke-nightly#47

Merged

skitt mentioned this pull request Nov 16, 2021

Bump Docker buildx to 0.7.0 submariner-io/shipyard#679

Merged

crazy-max added the area/github-actions label Sep 10, 2022

Conversation

tonistiigi commented Feb 6, 2021

Uh oh!

tonistiigi commented Mar 23, 2021

Uh oh!

tonistiigi commented Mar 23, 2021

Uh oh!

crazy-max left a comment

Choose a reason for hiding this comment

Uh oh!

JanJakes commented Apr 29, 2021

Uh oh!

douglasduteil commented Apr 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crazy-max commented Apr 30, 2021

Uh oh!

cdeneen commented Jun 25, 2021

Uh oh!

adrienjoly commented Jun 26, 2021

Uh oh!

valentijnscholten commented Jun 26, 2021

Uh oh!

adrienjoly commented Jun 27, 2021

Uh oh!

cdeneen commented Jun 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crazy-max commented Jun 28, 2021

Uh oh!

benesch commented Jul 13, 2021

Uh oh!

patroza commented Jul 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crazy-max commented Jul 13, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

douglasduteil commented Apr 30, 2021 •

edited

Loading

cdeneen commented Jun 28, 2021 •

edited

Loading

patroza commented Jul 13, 2021 •

edited

Loading