For now, I'll focus on layer caches and add a summary of my tests to let everyone follow the progress.

First, I'll try to understand how cache work with a simple example

// tests/plan/cache/test-cache.cue
package test

import (
	"dagger.io/dagger/engine"
)

engine.#Plan & {
	actions: {
		image: engine.#Pull & {
			source: "alpine"
		}

		test: engine.#Exec & {
			input: image.output
			args: ["/bin/sh", "-c", """
					sleep 15 && echo test
				"""]
		}
	}
}

We use --log-format plain flag to see CACHED keyword

Native buildkit cache

First run with no cache : > 15 seconds

dagger --europa up test-cache.cue -l debug --log-format plain
9:35PM DBG system | detected buildkit config    haveHostNetwork=true isActive=true version=v0.9.3
9:35PM DBG system | loading plan    args=[
    "test-cache.cue"
]
9:35PM DBG system | vendoring packages    mod=/Users/tomchauveau/Documents/DAGGER/dagger/tests
9:35PM DBG system | spawning buildkit job    attrs=null localdirs={}
9:35PM INF actions.image | computing
9:35PM DBG actions.image | #1 load metadata for docker.io/library/alpine:latest
9:35PM DBG actions.image | #1 sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6
9:35PM DBG actions.image | #1 DONE 1.5s
9:35PM DBG actions.image | 
9:35PM DBG actions.image | #2 Pull alpine
9:35PM DBG actions.image | #2 sha256:665ba8b2cdc0cb0200e2a42a6b3c0f8f684089f4cd1b81494fbb9805879120f7
9:35PM DBG actions.image | #2 resolve docker.io/library/alpine:latest
9:35PM INF actions.image | completed    duration=1.8s
9:35PM DBG actions.image | merging task result    result={
    output: {
        $dagger: {
            fs: {}
        }
    }
    digest: "sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300"
    config: {
        env: {
            PATH: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        }
        cmd: ["/bin/sh"]
    }
}
9:35PM INF actions.test | computing
9:35PM DBG actions.test | dependency detected    dependency=actions.image
9:35PM DBG actions.image | #2 resolve docker.io/library/alpine:latest 0.2s done
9:35PM DBG actions.image | #2 sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 0B / 2.82MB 0.2s
9:35PM DBG actions.image | #2 sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 1.05MB / 2.82MB 1.7s
9:35PM DBG actions.image | #2 sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 2.10MB / 2.82MB 2.9s
9:35PM DBG actions.image | #2 sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 2.82MB / 2.82MB 3.6s done
9:35PM DBG actions.image | #2 extracting sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 0.1s done
9:35PM DBG actions.image | #2 DONE 3.7s
9:35PM DBG actions.test | 
9:35PM DBG actions.test | #3 Exec ["/bin/sh", "-c", "\tsleep 15 && echo test"]
9:35PM DBG actions.test | #3 sha256:887f42c9fac1ade4cfb2f15a0514e5a104b58c25c5bc76174d6a6ecf89795255
9:35PM INF actions.test | #3 15.06 test
9:35PM DBG actions.test | #3 DONE 15.1s
9:35PM INF actions.test | completed    duration=18.8s
9:35PM DBG actions.test | merging task result    result={
    exit: 0
    output: {
        $dagger: {
            fs: {}
        }
    }
}
9:35PM DBG system | computed values    computed={
    actions: {
        image: {
            output: {
                $dagger: {
                    fs: {}
                }
            }
            digest: "sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300"
            config: {
                env: {
                    PATH: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
                }
                cmd: ["/bin/sh"]
            }
        }
        test: {
            exit: 0
            output: {
                $dagger: {
                    fs: {}
                }
            }
        }
    }
}

Second run with buildkit cache : < 3 seconds

dagger --europa up test-cache.cue -l debug --log-format plain
9:32PM DBG system | detected buildkit config    haveHostNetwork=true isActive=true version=v0.9.3
9:32PM DBG system | loading plan    args=[
    "test-cache.cue"
]
9:32PM DBG system | vendoring packages    mod=/Users/tomchauveau/Documents/DAGGER/dagger/tests
9:32PM DBG system | spawning buildkit job    attrs=null localdirs={}
9:32PM INF actions.image | computing
9:32PM DBG actions.image | #1 load metadata for docker.io/library/alpine:latest
9:32PM DBG actions.image | #1 sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6
9:32PM DBG actions.image | #1 DONE 0.2s
9:32PM DBG actions.image | 
9:32PM DBG actions.image | #2 Pull alpine
9:32PM DBG actions.image | #2 sha256:665ba8b2cdc0cb0200e2a42a6b3c0f8f684089f4cd1b81494fbb9805879120f7
9:32PM DBG actions.image | #2 resolve docker.io/library/alpine:latest
9:32PM INF actions.image | completed    duration=400ms
9:32PM DBG actions.image | merging task result    result={
    output: {
        $dagger: {
            fs: {}
        }
    }
    digest: "sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300"
    config: {
        env: {
            PATH: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        }
        cmd: ["/bin/sh"]
    }
}
9:32PM INF actions.test | computing
9:32PM DBG actions.test | dependency detected    dependency=actions.image
9:32PM INF actions.test | completed    duration=0s
9:32PM DBG actions.test | merging task result    result={
    output: {
        $dagger: {
            fs: {}
        }
    }
    exit: 0
}
9:32PM DBG system | computed values    computed={
    actions: {
        image: {
            output: {
                $dagger: {
                    fs: {}
                }
            }
            digest: "sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300"
            config: {
                env: {
                    PATH: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
                }
                cmd: ["/bin/sh"]
            }
        }
        test: {
            output: {
                $dagger: {
                    fs: {}
                }
            }
            exit: 0
        }
    }
}
9:32PM DBG actions.image | #2 resolve docker.io/library/alpine:latest 0.2s done
9:32PM DBG actions.image | #2 CACHED
9:32PM DBG actions.test | 
9:32PM DBG actions.test | #3 Exec ["/bin/sh", "-c", "\tsleep 15 && echo test"]
9:32PM DBG actions.test | #3 sha256:887f42c9fac1ade4cfb2f15a0514e5a104b58c25c5bc76174d6a6ecf89795255
9:32PM DBG actions.test | #3 CACHED

If we stop and remove the container with the dagger-buildkitd volume, the cache is not save.
Something possible with buildkit is to save that in multiple position :

Local
Registry
GitHub

Before starting debugging GitHub cache, it's a good idea to start with a local cache to see how it behave and how it's configured.

Native buildkit cache

Local cache

Support cache export to github action #1020

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions