@tonistiigi I prefer use single one ConfigMap

if we could multi ConfigMap.
When next deployment with less configurations will not delete some ConfigMap
https://github.com/docker/buildx/blob/master/driver/kubernetes/driver.go#L168-L182

Example

# first release
/etc/buildkit
/etc/buildkit/certs   

# second release
# the dir list is from new config content, once certs not includes, /etc/buildkit/certs will not includes.
/etc/buildkit
# ConfigMap for /etc/buildkit/certs will be remain in k8s

If still keep multiple ConfigMaps, need to add label in ConfigMaps, and List by label before delete

For single one ConfigMap. could be:

configFiles := []struct {
	name    string
	absPath string
}{}

d.Spec.Template.Spec.Containers[0].VolumeMounts = make([]corev1.VolumeMount, len(configFiles))
d.Spec.Template.Spec.Volumes = make([]corev1.Volume, len(configFiles))

for i, cf := range configFiles {
	d.Spec.Template.Spec.Containers[0].VolumeMounts[i] = corev1.VolumeMount{
		Name:      "config-" + cf.name,
		MountPath: cf.name,
		SubPath:   filepath.Base(cf.absPath),
	}

	d.Spec.Template.Spec.Volumes[i] = corev1.Volume{
		Name: "config-" + cf.name,
		VolumeSource: corev1.VolumeSource{
			ConfigMap: &corev1.ConfigMapVolumeSource{
				LocalObjectReference: corev1.LocalObjectReference{
					Name: "config", // from same ConfigMap
				},
				Items: []corev1.KeyToPath{
					{
						Key: cf.name,
						Path: filepath.Base(cf.absPath),
					},
				},
			},
		}}
}

Which one do you prefer ?

I'd like to keep the file structure the same as container driver(+rest in future) and reuse the same code.

I didn't quite get in what case the second release with config and no certs can happen. Also not sure how they remain mounted if they are removed from container spec. On deletion cleanup by id should still work. I added incrementing numbers to names.

If you think deletion by a label is safer feel free to add commit for that.

@tonistiigi
Sure. This PR could be ok to merge.
I may add the deletion by a label later today if I have time.

@tonistiigi Could the spec be a map where we also define the base dir. So instead of:

{
  "Nodes": [
    {
      "Files": {
        "buildkitd.toml": "...",
        "certs/docker.io/cert.pem": "...",
        "certs/docker.io/key.pem": "...",
        "certs/docker.io/myca.pem": "..."
      }
    }
  ],
}

we have:

{
  "Nodes": [
    {
      "Files": [
        {
          "/etc/buildkit": {
            "buildkitd.toml": "...",
            "certs/docker.io/cert.pem": "...",
            "certs/docker.io/key.pem": "...",
            "certs/docker.io/myca.pem": "..."
          }
        }
      ]
    }
  ]
}

to be able to handle other use cases in the future outside buildkitd config?

@crazy-max It could be though that we want to change the etc dir in the future. Eg. Rootless could be one case