Europa: Example configuration for changelog.com CI#1227
Europa: Example configuration for changelog.com CI#1227shykes merged 2 commits intodagger:mainfrom shykes:europa-example-thechangelog
Conversation
|
✔️ Deploy Preview for devel-docs-dagger-io ready! 🔨 Explore the source changes: 615a9a663276127bd588adf3ebab9cdf2b5eeabb 🔍 Inspect the deploy log: https://app.netlify.com/sites/devel-docs-dagger-io/deploys/61b9b925e4f9610007eed0c8 😎 Browse the preview: https://deploy-preview-1227--devel-docs-dagger-io.netlify.app |
|
✔️ Deploy Preview for devel-docs-dagger-io ready! 🔨 Explore the source changes: 5dbfa7beaaa9bfc09e9bbd13f47e8dccfef5289a 🔍 Inspect the deploy log: https://app.netlify.com/sites/devel-docs-dagger-io/deploys/61d4f6834818ce0007949f47 😎 Browse the preview: https://deploy-preview-1227--devel-docs-dagger-io.netlify.app |
There was a problem hiding this comment.
@shykes Can you make it work with something like this?
#Set: { config: #ImageConfig input: #Image output: { rootfs: input.rootfs "config": { // loop fields from input.config that are *not present* in config // loop fields from config } } }
There was a problem hiding this comment.
Yes that is one approach - wrap the mutation below in an engine operation perhaps.
The other approach would be to avoid mutation - by controlling what fields get filled into the base imageconfig, ensuring that the fields we want to set are not concrete.
There was a problem hiding this comment.
Here's a POC: https://cuelang.org/play/?id=QGI_MUToOqa#cue@export@cue
It starts to get a bit hairy when you need to drill down on struct values and lists, if you want to do it all in CUE.
This is not a mutation though. You're not modifying, you're creating a new instance.
I like the idea of having an engine op that merges (overwriting values) two structs and spits out a new one. If they're the same definition, the result will also be valid for that definition. Downside is maybe cue eval?
There was a problem hiding this comment.
Yes you’re right, it’s not a mutation. What I meant is that it gives the developer the familiar abstraction of mutation (“set this value”) but actually implements it as a DAG node (immutable inputs and outputs).
I like the idea of having an engine op that merges (overwriting values) two structs and spits out a new one. If they're the same definition, the result will also be valid for that definition.
It’s definitely a fun idea :) cc @aluzzardi
Downside is maybe cue eval?
Could you elaborate?
There was a problem hiding this comment.
Well, if you were to check if output.config has the intended keys resulting from the merge, that won't be available in pure cue.
Although, looking at the spec (are we supporting dockerfile as well?), I don't think you want to merge all maps/lists, just a few. You want to combine the Env lists and Labels maps, but not a Cmd list, for example. Anything other then those specific ones that need merging, would just be replaced.
I think this is doable in pure cue. I've implemented the map merging, but not lists. For Env you can make a temporary struct out of it, merge it, and spit out a list in the end. As for OnBuild, you'd just append, even if it causes duplicates.
Besides Env, Labels and OnBuild, seems to me everything else should be replaced (easy). Only Env requires a bit more cue code to do the merging.
There was a problem hiding this comment.
I'll send a playground link when I can for you to see this in action.
There was a problem hiding this comment.
I've not handled OnBuild because their strings are Dockerfile instructions, that's another discussion.
There was a problem hiding this comment.
There was a problem hiding this comment.
@helderco I would like to take your proof-of-concept, and make it the official solution. Instead of a generic operation to merge any 2 CUE values, I think we should start with a domain-specific implementation in the docker package. That allows us to fix the immediate problem with the #ImageConfig schema, and postpone the harder work to make a perfectly general implementation. I will open an issue.
There was a problem hiding this comment.
@shykes Yes, and I don't think you will be able to have a fully generic solution in this case. For one, Env is special and you'd have to convert to a struct first (it's a list of strings in the key=value format). Also, there are some maps/lists that don't make sense to be merged, and should be replaced. You'd have to input exactly which keys you want merged in the generic solution.
|
Will look into this next week 🙌🏻 |
|
@shykes @gerhard What I built is quite a bit different than this config, and only uses I think it's a good place to start as working config. And I think we should build up the package main
import (
// "alpha.dagger.io/europa/dagger"
"alpha.dagger.io/europa/dagger/engine"
)
runtime_image_ref: string | *"thechangelog/runtime:2021-05-29T10.17.12Z"
engine.#Plan & {
inputs: directories: app: {
path: "."
exclude: [
".circleci",
".dagger",
".git",
".github",
"2021",
"2022",
"_build/dev",
"_build/test",
"assets/node_modules",
"cue.mod",
"dev_docker",
"docker",
"import",
"nginx",
"priv/db",
"priv/uploads",
"script",
"tmp",
".all-contributorsrc",
".autocomplete",
".credo.exs",
".dockerignore",
".formatter.exs",
".envrc",
".env",
".gitattributes",
".gitignore",
"README.md",
"coveralls.json",
"start_dev_stack.sh",
".kube",
"erl_crash.dump",
"deps",
"_build",
"dagger",
"main.cue",
]
}
inputs: directories: docker: {
path: "."
include: [
"docker/Dockerfile.production",
".dockerignore",
]
}
actions: {
runtimeImage: engine.#Pull & {
source: runtime_image_ref
}
depsCache: engine.#CacheDir & {
id: "depsCache"
}
depsCacheMount: "depsCache": {
dest: *"/app/deps/" | string
contents: depsCache
}
buildCacheTest: engine.#CacheDir & {
id: "buildCacheTest"
}
buildCacheTestMount: "buildCacheTest": {
dest: *"/app/_build/test" | string
contents: buildCacheTest
}
buildCacheProd: engine.#CacheDir & {
id: "buildCacheProd"
}
buildCacheProdMount: "buildCacheProd": {
dest: *"/app/_build/prod" | string
contents: buildCacheProd
}
nodeModulesCache: engine.#CacheDir & {
id: "nodeModulesCache"
}
nodeModulesCacheMount: "nodeModulesCache": {
dest: *"/app/assets/node_modules" | string
contents: nodeModulesCache
}
appImage: engine.#Copy & {
input: runtimeImage.output
source: {
root: inputs.directories.app.contents
}
dest: "/app"
}
deps: engine.#Exec & {
input: appImage.output
mounts: depsCacheMount
workdir: "/app"
args: ["bash", "-c", " mix deps.get"]
}
// testDeps: engine.#Exec & {
// input: deps.output
// mounts: depsCacheMount
// args: ["bash", "-c", "ls -al /app /app/deps"]
// }
assetsCompile: engine.#Exec & {
input: depsCompileProd.output
mounts: depsCacheMount & nodeModulesCacheMount
workdir: "/app/assets"
env: PATH: "/usr/local/lib/nodejs/node-v14.17.0-linux-x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
args: ["bash", "-c", "yarn install --frozen-lockfile && yarn run compile"]
}
#depsCompile: engine.#Exec & {
input: deps.output
mounts: depsCacheMount
workdir: "/app"
args: ["bash", "-c", "mix do deps.compile, compile"]
}
depsCompileTest: #depsCompile & {
env: MIX_ENV: "test"
mounts: buildCacheTestMount
}
depsCompileProd: #depsCompile & {
env: MIX_ENV: "prod"
mounts: buildCacheProdMount
}
assetsDigest: engine.#Exec & {
input: assetsCompile.output
mounts: depsCacheMount & buildCacheProdMount & nodeModulesCacheMount
env: MIX_ENV: "prod"
workdir: "/app"
args: ["bash", "-c", "mix phx.digest"]
}
imageProdCacheCopy: engine.#Exec & {
input: assetsDigest.output
mounts: (depsCacheMount & {depsCache: dest: "/mnt/app/deps/"} )
mounts: (buildCacheProdMount & {buildCacheProd: dest: "/mnt/app/_build/prod"} )
args: ["bash", "-c", "cp -Rp /mnt/app/deps/* /app/deps/ && cp -Rp /mnt/app/_build/prod/* /app/_build/prod/"]
}
imageProdDockerCopy: engine.#Copy & {
input: imageProdCacheCopy.output
source: {
root: inputs.directories.docker.contents
}
dest: "/"
}
imageProd: engine.#Build & {
source: imageProdDockerCopy.output
dockerfile: path: "/docker/Dockerfile.production"
buildArg: {
APP_FROM_PATH: "/app"
GIT_AUTHOR: "joel"
GIT_SHA: "abcdef"
APP_VERSION: "main"
BUILD_URL: "longtine.io/build"
}
}
testImageProdCacheCopy: engine.#Exec & {
input: imageProdDockerCopy.output
always: true
args: ["bash", "-c", "ls -al / /docker /app/deps/ /app/_build/prod/ /app/priv/static"]
}
}
} |
Signed-off-by: Solomon Hykes <[email protected]>
Signed-off-by: Solomon Hykes <[email protected]>
|
As discussed with @jlongtine and @gerhard:
Thanks all! |
Continuing the experiment started in https://github.com/thechangelog/changelog.com/pull/401… cc @gerhard :)