What is the problem you're trying to solve
With multi-stage builds, it's useful to be able to e.g. access an SSH server only from within the build image. Docker now supports that with BuildKit. It adds new command line flags and Dockerfile syntax. Compose should allow defining build secrets, just like it supports runtime secrets.
Building using BuildKit is enabled through an environment variable (DOCKER_BUILDKIT=1) or a Docker configuration file setting.
Compose already supports runtime secrets, which are files "mounted" inside a container. BuildKit seems very similar, but with some differences:
--ssh, which simply mounts a Unix socket and sets the SSH_AUTH_SOCK environment variable for RUN commands that request it.- The
dst flag, which specifies an alternative directory (default is still /run/secrets/).
- Though it's undocumented, it seems setting owner and mode is supported.
Describe the solution you'd like
This is a high-level discussion around the requirements of a solution. A more concrete suggestion would probably be more suited for a PR.
Tie in to the existing secrets. A file secret is defined by
- Type, but only
file is currently supported
- ID (used to reference the secret in a
RUN
- Source (the file path)
An SSH secret is defined by
Both of them are referenced/dispatched by specifying
- Source (only for file secrets; implied for SSH)
- Target
- Required (boolean)
- UID
- GID
- Mode
This seems similar enough that it should be possible to extend the secrets section, and allow referencing it in an image's build section. The SSH type is specific to BuildKit, so parsing code would have to know which type is allowed in build/runtime. Using a separate section for build secrets is a possibility.
Lastly, adding a "builder" selector that allows choosing "buildkit" as the builder could set the right environment variable to allow parsing these flags. (The Dockerfile still needs to have the syntax line to support the RUN --mount variant.)
Additional context
A spec PR was suggested as the first step, in docker/compose#7296 (comment).
The --secret command line option is parsed into a github.com/moby/buildkit/session/secrets/secretsprovider.FileSource. The --ssh option is parsed into a github.com/moby/buildkit/session/sshforward/sshprovider.AgentConfig.
The secret mount is dispatched at https://github.com/moby/buildkit/blob/195f9e598cc7ba82aabefe60d564df7329cb38dd/frontend/dockerfile/dockerfile2llb/convert_secrets.go, and the ssh mount is dispatched at https://github.com/moby/buildkit/blob/195f9e598cc7ba82aabefe60d564df7329cb38dd/frontend/dockerfile/dockerfile2llb/convert_ssh.go.
What is the problem you're trying to solve
With multi-stage builds, it's useful to be able to e.g. access an SSH server only from within the build image. Docker now supports that with BuildKit. It adds new command line flags and
Dockerfilesyntax. Compose should allow defining build secrets, just like it supports runtime secrets.Building using BuildKit is enabled through an environment variable (
DOCKER_BUILDKIT=1) or a Docker configuration file setting.Compose already supports runtime secrets, which are files "mounted" inside a container. BuildKit seems very similar, but with some differences:
--ssh, which simply mounts a Unix socket and sets theSSH_AUTH_SOCKenvironment variable forRUNcommands that request it.dstflag, which specifies an alternative directory (default is still/run/secrets/).Describe the solution you'd like
This is a high-level discussion around the requirements of a solution. A more concrete suggestion would probably be more suited for a PR.
Tie in to the existing
secrets. A file secret is defined byfileis currently supportedRUNAn SSH secret is defined by
Both of them are referenced/dispatched by specifying
This seems similar enough that it should be possible to extend the
secretssection, and allow referencing it in an image'sbuildsection. The SSH type is specific to BuildKit, so parsing code would have to know which type is allowed in build/runtime. Using a separate section for build secrets is a possibility.Lastly, adding a "builder" selector that allows choosing "buildkit" as the builder could set the right environment variable to allow parsing these flags. (The Dockerfile still needs to have the syntax line to support the
RUN --mountvariant.)Additional context
A spec PR was suggested as the first step, in docker/compose#7296 (comment).
The
--secretcommand line option is parsed into agithub.com/moby/buildkit/session/secrets/secretsprovider.FileSource. The--sshoption is parsed into agithub.com/moby/buildkit/session/sshforward/sshprovider.AgentConfig.The secret mount is dispatched at https://github.com/moby/buildkit/blob/195f9e598cc7ba82aabefe60d564df7329cb38dd/frontend/dockerfile/dockerfile2llb/convert_secrets.go, and the ssh mount is dispatched at https://github.com/moby/buildkit/blob/195f9e598cc7ba82aabefe60d564df7329cb38dd/frontend/dockerfile/dockerfile2llb/convert_ssh.go.