Configure architecture of a running pipeline #1071
Description
Issue
To support the multi-architecture with Dagger and solve #106, it's necessary to improve the engine to handle other architecture than the remote one.
Here's a repro case to understand the problem
My computer supports amd64 so we got the following output :
package main
import (
"alpha.dagger.io/dagger/op"
)
// Alpîne amd64
amd64: #up: [
op.#FetchContainer & {
ref: "docker.io/library/alpine:latest@sha256:69704ef328d05a9f806b6b8502915e6a0a4faa4d72018dc42343f511490daf8a"
},
op.#Exec & {
args: ["/bin/sh", "-c", "uname -a"]
always: true
}
]dagger up --log-format plain
4:15PM INF amd64 | computing
4:15PM INF amd64 | #3 0.098 Linux buildkitsandbox 5.11.0-37-generic #41~20.04.2-Ubuntu SMP Fri Sep 24 09:06:38 UTC 2021 x86_64 Linux
4:15PM INF amd64 | completed duration=100ms
Output Value DescriptionBut if I use an arm64 image, it fails
package main
import (
"alpha.dagger.io/dagger/op"
)
// Alpine ARM64
arm64: #up: [
op.#FetchContainer & {
ref: "docker.io/library/alpine:latest@sha256:b06a5cf61b2956088722c4f1b9a6f71dfe95f0b1fe285d44195452b8a1627de7"
},
op.#Exec & {
args: ["/bin/sh", "-c", "uname -a"]
always: true
}
]
dagger up --log-format plain
4:16PM INF arm64 | computing
4:16PM INF amd64 | computing
4:16PM INF amd64 | #6 0.102 Linux buildkitsandbox 5.11.0-37-generic #41~20.04.2-Ubuntu SMP Fri Sep 24 09:06:38 UTC 2021 x86_64 Linux
4:16PM INF amd64 | completed duration=200ms
4:16PM ERR arm64 | failed: process "/bin/sh -c uname -a" did not complete successfully: exit code: 1 duration=200ms
4:16PM INF arm64 | #5 0.102 standard_init_linux.go:228: exec user process caused: exec format error
4:16PM FTL system | failed to up environment: task failed: process "/bin/sh -c uname -a" did not complete successfully: exit code: 1Support
When Buildkit Marshal LLB into a definition, it's possible to apply an architecture constraint, this way, it's possible to execute ARM program in AMD processor and vice-versa.
It should work for every architecture supported by
buildkitd
Below, our Marshal function that must be update.
func (s Solver) Marshal(ctx context.Context, st llb.State) (*bkpb.Definition, error) {
def, err := st.Marshal(ctx)Here's a couple of example about how we can force an architecture
For ARM64
func (s Solver) Marshal(ctx context.Context, st llb.State) (*bkpb.Definition, error) {
def, err := st.Marshal(ctx, llb.LinuxArm64)For AMD64
func (s Solver) Marshal(ctx context.Context, st llb.State) (*bkpb.Definition, error) {
def, err := st.Marshal(ctx, llb.LinuxAmd64)For x390x
func (s Solver) Marshal(ctx context.Context, st llb.State) (*bkpb.Definition, error) {
def, err := st.Marshal(ctx, llb.LinuxS390x)Below you can see an exec on an ARM64 image tested with my computer and a constraint llb.LinuxArm64
dagger up --log-format plain
4:17PM INF arm64 | computing
4:17PM INF arm64 | completed duration=200ms
Output Value Description
4:17PM INF arm64 | #3 0.146 Linux buildkitsandbox 5.11.0-37-generic #41~20.04.2-Ubuntu SMP Fri Sep 24 09:06:38 UTC 2021 aarch64 LinuxProposal
Basically, we must allow the user to configure in which architecture he wants to run his pipeline.
To do so, we can add a new field in our values.yaml named architecture
plan:
package: ./alpine/tests
name: alpine
architecture: linux/amd64 # Our new fieldThen it would be possible to configure our architecture with a simple dagger edit.
We could also add a new flag to dagger new to specify an architecture when creating the package.
dagger new -p test -a linux/arm64 .By default we will use the host architecture (no change from now)
Details
Architectures supported by buildkitd
[
"linux/amd64",
"linux/arm64",
"linux/riscv64",
"linux/ppc64le",
"linux/s390x",
"linux/386",
"linux/arm/v7",
"linux/arm/v6"
]Engine
We will need to update the Pipeline structure to store the architecture
type Pipeline struct {
code *compiler.Value
name string
s solver.Solver // We want to keep one solver
state llb.State
arch llb.ConstraintsOpt // Architecture constraint
result bkgw.Reference
image dockerfile2llb.Image
computed *compiler.Value
}This way, we will just forward it to the Solver and we will support multi-architecture.
Don't hesitate to share your opinion 🚀