Skip to content

Enhanced Dockerfile for cross compilation #43529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
crazy-max wants to merge 24 commits into from
Closed

Enhanced Dockerfile for cross compilation #43529

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6b60df6
Dockerfile: use TARGETVARIANT for frozen images script
crazy-max Sep 2, 2022
cc49291
Dockerfile: handle multi base image for riscv64 and armel support
crazy-max Aug 19, 2022
7fad041
Dockerfile: remove hardcoded platform for vpnkit stage
crazy-max Aug 20, 2022
5877916
Dockerfile: add cross compilation helper
crazy-max Aug 19, 2022
439e7e7
Dockerfile: pin criu version and build from source
crazy-max Aug 19, 2022
2c3d94e
Dockerfile: add dummy stage
crazy-max Aug 19, 2022
10f4f72
Dockerfile: containerd cross
crazy-max Sep 4, 2022
af7ab21
Dockerfile: runc cross
crazy-max Aug 19, 2022
65cc91d
Dockerfile: tini cross
crazy-max Aug 19, 2022
16f6dca
Dockerfile: rootlesskit cross
crazy-max Aug 19, 2022
0effae3
Dockerfile: containerutility cross
crazy-max Aug 19, 2022
8b56a2e
Dockerfile: verify and better cache for go-swagger stage
crazy-max Aug 19, 2022
4f5c955
Dockerfile: align deps format and output
crazy-max Aug 19, 2022
22c3b89
Dockerfile: use TARGETPLATFORM for delve stage and verify
crazy-max Aug 19, 2022
2a7f462
Dockerfile: verify and better cache for dockercli stage
crazy-max Aug 19, 2022
79a20f2
Dockerfile: use global scope platform args for registry stage and verify
crazy-max Aug 19, 2022
7fe6900
Dockerfile: better cache for crun stage
crazy-max Aug 19, 2022
750330e
Dockerfile: GO111MODULE=on by default
crazy-max Aug 19, 2022
86f9e1e
Dockerfile: enhanced for cross compilation
crazy-max Aug 20, 2022
679bb5f
Dockerfile: add "all" stage to build binaries and extra tools
crazy-max Sep 8, 2022
5100cd9
Dockerfile: smoke test stage and gha workflow
crazy-max Sep 22, 2022
b97bb43
Dockerfile: refactor Dockerfile.simple and create gha workflow
crazy-max Aug 20, 2022
d511c1e
riscv64 cross comp support
crazy-max Aug 20, 2022
9bfbfda
docs: contributing docs update for cross comp
crazy-max Aug 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
.git
Copy link
Copy Markdown
Member

@thaJeztah thaJeztah Jul 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This concerns me a bit; this can be quite large, and contain a lot of files;

du -sh ./.git
221M	./.git

find ./.git -type f | wc -l
    6062
  • This would also mean we'd invalidate the cache on any change in the .git directory (which for me happens a lot, e.g. add / remove branches)
  • Besides potential metadata (which commit did we build from for inclusion in docker version), are there other reasons we would need this?
  • ^^ as, in general, the build should not depend on git (i.e., it should be possible to build from a source tarball, which could be downloaded from the releases page)

Copy link
Copy Markdown
Member Author

@crazy-max crazy-max Jul 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would also mean we'd invalidate the cache on any change in the .git directory (which for me happens a lot, e.g. add / remove branches)

Yes indeed cache would be invalidated quite often to load the context here: https://github.com/moby/moby/pull/43529/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R656. I have checked on ci (fetch depth 0) and loading the context with .git + building dockerd and docker-proxy takes ~30sec: https://github.com/moby/moby/runs/7425238835?check_suite_focus=true#step:6:2230. Locally it's even faster for me.

Besides potential metadata (which commit did we build from for inclusion in docker version), are there other reasons we would need this?

Yes this is just to invoke git in https://github.com/moby/moby/pull/43529/files#diff-9eace97aaee3a26c826f37605437498f930565a99a4f072cc476bfeac6199e73R28-R36 if VERSION, GITCOMMIT or BUILDTIME is empty.

^^ as, in general, the build should not depend on git (i.e., it should be possible to build from a source tarball, which could be downloaded from the releases page)

Yes we could only rely on ARGs to set VERSION, GITCOMMIT and BUILDTIME for sandboxed builds.

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah Jul 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is just to invoke git in https://github.com/moby/moby/pull/43529/files#diff-9eace97aaee3a26c826f37605437498f930565a99a4f072cc476bfeac6199e73R28-R36 if VERSION, GITCOMMIT or BUILDTIME is empty.

(let me copy that code, as the link will break on rebase)

if [ -z "$VERSION" ]; then
 	VERSION=$(git describe --match 'v[0-9]*' --dirty='.m' --always --tags | sed 's/^v//' 2> /dev/null || echo "unknown-version")
 fi
 if [ -z "$GITCOMMIT" ]; then
 	GITCOMMIT=$(git rev-parse --short HEAD 2> /dev/null || true)
 fi
 if [ -z "$BUILDTIME" ]; then
 	BUILDTIME=$(date -u --date="@${SOURCE_DATE_EPOCH:-$(date +%s)}" +"%Y-%m-%dT%H:%M:%SZ")
 fi

Yes we could only rely on ARGs to set VERSION, GITCOMMIT and BUILDTIME for sandboxed builds.

Yeah, leaning toward that, as;

  • we could already (and already do?) pass that information as argument
  • 30 seconds is still 30 seconds 😅
  • for local builds, I don't think the information is really relevant (dev as "version" is fine, and I know I just built a binary, so I don't really care about build-time)

Of course happy to hear other opinions

Copy link
Copy Markdown
Member

@cpuguy83 cpuguy83 Oct 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COPY --ignore would be real nice :)

I think for local dev we'd typically have the source mounted in anyway, in which case we skip the COPY step anyway.

.go-pkg-cache
.gopath
bundles
Expand Down
11 changes: 6 additions & 5 deletions .github/workflows/buildkit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ on:
- '[0-9]+.[0-9]{2}'
pull_request:

env:
BUNDLES_OUTPUT: ./bundles

jobs:
validate-dco:
uses: ./.github/workflows/.dco.yml
Expand All @@ -23,6 +20,8 @@ jobs:
runs-on: ubuntu-20.04
needs:
- validate-dco
env:
DESTDIR: ./build
steps:
-
name: Checkout
Expand All @@ -35,12 +34,14 @@ jobs:
uses: docker/bake-action@v2
with:
targets: binary
env:
DOCKER_LINKMODE: static
-
name: Upload artifacts
uses: actions/upload-artifact@v3
with:
name: binary
path: ${{ env.BUNDLES_OUTPUT }}
path: ${{ env.DESTDIR }}
if-no-files-found: error
retention-days: 1

Expand Down Expand Up @@ -109,7 +110,7 @@ jobs:
env:
CONTEXT: "."
TEST_DOCKERD: "1"
TEST_DOCKERD_BINARY: "./build/moby/binary-daemon/dockerd"
TEST_DOCKERD_BINARY: "./build/moby/dockerd"
TESTPKGS: "${{ matrix.pkg }}"
TESTFLAGS: "-v --parallel=1 --timeout=30m --run=//worker=dockerd$"
SKIP_INTEGRATION_TESTS: "${{ matrix.skip-integration-tests }}"
Expand Down
81 changes: 60 additions & 21 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ on:
pull_request:

env:
BUNDLES_OUTPUT: ./bundles
PLATFORM: "Moby Engine"
PRODUCT: "Moby"
DEFAULT_PRODUCT_LICENSE: "Moby"
PACKAGER_NAME: "Moby"

jobs:
validate-dco:
Expand All @@ -25,12 +28,14 @@ jobs:
runs-on: ubuntu-20.04
needs:
- validate-dco
env:
DESTDIR: ./build
strategy:
fail-fast: false
matrix:
target:
- binary
- dynbinary
linkmode:
- static
- dynamic
steps:
-
name: Checkout
Expand All @@ -45,32 +50,55 @@ jobs:
uses: docker/bake-action@v1
with:
targets: ${{ matrix.target }}
env:
DOCKER_LINKMODE: ${{ matrix.linkmode }}
-
name: List artifacts
run: |
tree -nh ${{ env.DESTDIR }}
-
name: Check artifacts
run: |
find ${{ env.DESTDIR }} -type f -exec file -e ascii -- {} +
-
name: Upload artifacts
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.target }}
path: ${{ env.BUNDLES_OUTPUT }}
name: binary-${{ matrix.linkmode }}
path: ${{ env.DESTDIR }}
if-no-files-found: error
retention-days: 7

prepare:
runs-on: ubuntu-latest
needs:
- validate-dco
outputs:
matrix: ${{ steps.platforms.outputs.matrix }}
steps:
-
name: Checkout
uses: actions/checkout@v3
-
name: Create matrix
id: platforms
run: |
echo ::set-output name=matrix::$(docker buildx bake binary-cross --print | jq -cr '.target."binary-cross".platforms')
-
name: Show matrix
run: |
echo ${{ steps.platforms.outputs.matrix }}

cross:
runs-on: ubuntu-20.04
needs:
- validate-dco
- prepare
env:
DESTDIR: ./build
strategy:
fail-fast: false
matrix:
platform:
- linux/amd64
- linux/arm/v5
- linux/arm/v6
- linux/arm/v7
- linux/arm64
- linux/ppc64le
- linux/s390x
- windows/amd64
- windows/arm64
platform: ${{ fromJson(needs.prepare.outputs.matrix) }}
steps:
-
name: Checkout
Expand All @@ -82,21 +110,32 @@ jobs:
run: |
platform=${{ matrix.platform }}
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
-
name: Set up QEMU
uses: docker/setup-qemu-action@v1
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
-
name: Build
uses: docker/bake-action@v1
with:
targets: cross
env:
DOCKER_CROSSPLATFORMS: ${{ matrix.platform }}
targets: all
set: |
*.platform=${{ matrix.platform }}
-
name: List artifacts
run: |
tree -nh ${{ env.DESTDIR }}
-
name: Check artifacts
run: |
find ${{ env.DESTDIR }} -type f -exec file -e ascii -- {} +
-
name: Upload artifacts
uses: actions/upload-artifact@v2
with:
name: cross-${{ env.PLATFORM_PAIR }}
path: ${{ env.BUNDLES_OUTPUT }}
path: ${{ env.DESTDIR }}
if-no-files-found: error
retention-days: 7
38 changes: 38 additions & 0 deletions .github/workflows/simple.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: simple

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

on:
workflow_dispatch:
push:
branches:
- 'master'
- '[0-9]+.[0-9]{2}'
pull_request:

jobs:
validate-dco:
uses: ./.github/workflows/.dco.yml

build:
runs-on: ubuntu-20.04
needs:
- validate-dco
steps:
-
name: Checkout
uses: actions/checkout@v3
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
-
name: Build simple image
uses: docker/bake-action@v2
with:
targets: simple
-
name: Build dynbinary
run: |
docker run --rm docker:simple hack/make.sh dynbinary
53 changes: 53 additions & 0 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -499,3 +499,56 @@ jobs:
name: Create summary
run: |
teststat -markdown $(find /tmp/reports -type f -name '*.json' -print0 | xargs -0) >> $GITHUB_STEP_SUMMARY

prepare-smoke:
runs-on: ubuntu-20.04
needs:
- validate-dco
outputs:
matrix: ${{ steps.platforms.outputs.matrix }}
steps:
-
name: Checkout
uses: actions/checkout@v3
-
name: Create matrix
id: platforms
run: |
echo ::set-output name=matrix::$(docker buildx bake binary-smoketest --print | jq -cr '.target."binary-smoketest".platforms')
-
name: Show matrix
run: |
echo ${{ steps.platforms.outputs.matrix }}

smoke:
runs-on: ubuntu-20.04
needs:
- prepare-smoke
strategy:
fail-fast: false
matrix:
platform: ${{ fromJson(needs.prepare-smoke.outputs.matrix) }}
steps:
-
name: Checkout
uses: actions/checkout@v3
-
name: Prepare
run: |
platform=${{ matrix.platform }}
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
-
name: Set up QEMU
uses: docker/setup-qemu-action@v2
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
-
name: Test
uses: docker/bake-action@v2
with:
targets: binary-smoketest
set: |
*.platform=${{ matrix.platform }}
env:
DOCKER_LINKMODE: static
Loading