Could we reuse some of this code in k8s driver? cc @morlay

copyBuildKitConfToContainer could.

In k8s, we have to read file data and store it in ConfigMap or Secrets, and mount it into running containter.

I prefix add an interface for Driver to add files, like

inteface Driver {
   AddFile(mountPath string, data []byte, secure bool)
}

Then diffrent drivers do each mounting.

And copyBuildKitConfToContainer(...) could be copyBuildKitConfForDriver(d Dirver, ....) to reuse

Actually, this part might be fine to be driver-specific but the part that modifies the toml file and copies certs under .docker should not be specific to the driver but shared.

So let driver have ability to add file will be useful.

if the driver not need those files, it could skip AddFile

But driver like docker-container and kubernetes need, it could be easy to AddFile with different ways.

At least, copyBuildKitConfToContainer is needed to kubernetes driver too.

But consider to read file data to memory and write it the temp path in copyToContainer (may rename as AddFile).

cc @crazy-max

@tonistiigi

I don't understand why you have two volumes in your example when they both modify /etc/buildkit

it is not modify /etc/buildkit, it is modify files and sub dirs of /etc/buildkit.

if only mount single /etc/buildkit, we have to store toml and files in ConfigMap like

data:
    buildkit.toml:  xxx
    key.pem: xxx  
    certs.pem: xxx

The mounted /etc/buildkit will be

/etc/buildkit/
  buildkit.toml
  key.pem
  certs.pem

With my example, it could mount any file in any location.

even put all files under /etc/buildkit, it could be same as container-driver did

/etc/buildkit/
  certs/myregistry.io/
     key.pem
     certs.pem
  buildkit.toml

we can't copy a file in a folder that does not exist in the container with the CopyToContainer Docker API.

Yes, but we could write to temp files before CopyToContainer in AddFile

If worry about copy cost. we could design the API like https://docs.min.io/docs/golang-client-api-reference.html#PutObject

type PutFileOptions struct {
   Secure bool
}

type Driver interface {
   PutFile(ctx context.Context, containerAbsPath string, r io.Reader, putOpts PutFileOptions) error
}

So configmap can not contain subdirectories?

@tonistiigi Yes. / is not allowed in ConfigMap data key, it could be a flatten KV.

For subdirectories, we have to rename the absolute path, and resolve it back with volumes and volumeMounts like my example did.

This why I prefer have a PutFile() interface to make driver could do mounting files,
Then we could just convert and mount files for diffrent drivers in PutFile().

Yes we could use an interface but like @tonistiigi said we can't use the same paths defined on the host for ca and user tls registries certs inside the container, that's why the buildkitd toml needs to be modified anyway.

Maybe we could impl a buildkitd.override.toml on BuildKit.

why temp directory? just save it under .docker/buildx or do you want to do this separately?

yes will do that in a follow-up to keep the current state

I don't think we need any of this between 218-278. Just calling archive.Tar to get ReadCloser should be enough for this case.

^

Done

I think you can remove more. As I said, I think only a single archive.Tar call is needed here.

I don't think this is needed either. Nor the /. prefix.

nit: dockerarchive.Tar simpler for this case

dockerarchive.Tar requires a Compression arg, that's why I use dockerarchive.TarWithOptions instead. That API feels odd and should be changed I think.

dockerarchive.Uncompressed

If it is default anyway then don't set it.