Intro

With #1087, it's now possible to config one architecture when you run a pipeline.
The aim is to improve that to :

• Add another layer of configuration that targets one pipeline and not the whole plan.
• Specify architecture directly in cue
• Manage multiple architectures in a single plan.

This way, we will be able to :

• Build & push multi-architecture images and then resolve Move to dagger when multi-arch images are supported dubo-dubon-duponey/docker-images#11
• Create multi-arch CI/CD for multi-platform application

There are already two issues (#106 and #437) related to that topic but I prefer writing that one as a proposal.

Proposal

As discussed in #1071, we will need to improve our init operation to support a flag arch.

💡 I consider that an init operation is an op.#Op used at the beginning of a pipeline, for instance: #FetchContainer, #DockerBuild.

It has no sense to give an architecture constraint to #FetchHTTP or FetchGit so they are omitted.

DockerBuild

op.#DockerBuild already supports a platforms flag.

#DockerBuild: {  
 do: "docker-build"  
 // We accept either a context, a Dockerfile or both together  
 context?:        _  
 dockerfilePath?: string // path to the Dockerfile (defaults to "Dockerfile")  
 dockerfile?:     string  
  
 platforms?: [...string]  
 buildArg?: {  
 // FIXME: should be `[string]: string | #Secret` (circular import)  
 [string]: string | _ @dagger(secret)  
 } label?: [string]: string  
 target?: string  
 hosts?: [string]: string  
}

First, we will need to fix #437, here platforms is an array of platforms. It's really cool because with that flag you can build an image in multiple architectures.
But, it doesn't work on dagger because #up is single architecture.

As a fix, we should rewrite platform as platform: string and allow only 1 architecture constraint.

💡 It's not relevant to create a type #Arch that lists each architecture available because we certainly going to lose performance.
The error will be caught in dagger, not in cue (for the moment)

Then if you want to build a multi-arch image, you can simply do

images: [arch=string]: dagger.#Artifact

images: amd64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/amd64"
    }
]

images: arm64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/arm64"
    }
]

// ... Do other stuff

FetchContainer

Thanks DockerBuild we can build an image with a specific architecture, then it's should also be possible to fetch an image with a specific architecture

Below the current definition of FetchContainer

#FetchContainer: {  
 do:  "fetch-container"  
 ref: string  
}

We will be able to config the architecture by adding an optional field arch to the definition.

E.g

#FetchContainer: {  
 do:  "fetch-container"  
 ref: string
 arch?: string
}

💡 arch must be optional since by default the architecture is the linux/amd64 OR the one configured in values.yaml.

PushContainer

As well, @dubo-dubon-duponey needs to push a multi-architecture image.
To do so, it's required to improve PushContainer to handle it.

Here's below his current definition :

#PushContainer: {  
 do:  "push-container"  
 ref: string  
}

I made a simple proof of concept to push a multi-architecture image. The only required informations are the state and the image configuration.
Those pieces of information can be extracted from a dagger.#Artifact.

So, to support a multi-architecture image, we can just add an optional field that could take as parameters artifact and then we push it instead of pushing the current pipeline state.

#PushContainer: {  
 do:  "push-container"  
 ref: string
 images?: [arch=string]: dagger.#Artifact
}

💡 It's necessary to use a map instead of an array to keep dependency working well. Either, images will always be an empty array.

I don't know if it's a cue issue but it's required to make it works.

Details

The main challenge to correctly push multi-architecture images is to push the manifest list.

This way we can get something like this

docker buildx imagetools inspect docker.io/alpine:latest                                                                                  
Name:      docker.io/library/alpine:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:e1c082e3d3c45cccac829840a25941e679c25d438cc8412c2fa221cf1a824e6a
           
Manifests: 
  Name:      docker.io/library/alpine:latest@sha256:69704ef328d05a9f806b6b8502915e6a0a4faa4d72018dc42343f511490daf8a
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64
             
  Name:      docker.io/library/alpine:latest@sha256:18c29393a090ba5cde8a5f00926e9e419f47cfcfd206cc3f7f590e91b19adfe9
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v6
             
  Name:      docker.io/library/alpine:latest@sha256:e12ff876f0075740ed3d7bdf788107ae84c1b3dd6dc98b3baea41088aba5236f
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7
             
  Name:      docker.io/library/alpine:latest@sha256:b06a5cf61b2956088722c4f1b9a6f71dfe95f0b1fe285d44195452b8a1627de7
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64/v8

To do so, we must :

• Retrieve all states & images to push
• Push each image one by one
• Push a manifest list containing metadata about images.

Our current Export function can only push a single image.

We will need to create a new function named Exports to push multiple images.

Example

Then let me show you an example of a plan that pushes a multi-architecture image

images: [arch=string]: dagger.#Artifact

// NOTE: It can be done through a for loop
// 
images: amd64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/amd64"
    }
]

images: arm64: #up: [
    op.#DockerBuild & { 
         ...
        platform: "linux/arm64"
    }
]

remoteImages: #up: [
	op.#PushContainer & {
		ref: "docker.io/example/example:multi-arch"
		"images": images
	}
]

Side effects

• We will improve the docker package to expose those features to the user.

Don't hesitate to ping me if you see more side effects.
I'll keep that issue updated.

Thank you, don't hesitate to share your opinion, especially about the cue integration.

/cc @aluzzardi @talentedmrjones @samalba @shykes @dubo-dubon-duponey