Skip to content

Europa: Example configuration for changelog.com CI#1227

Merged
shykes merged 2 commits intodagger:mainfrom
shykes:europa-example-thechangelog
Jan 6, 2022
Merged

Europa: Example configuration for changelog.com CI#1227
shykes merged 2 commits intodagger:mainfrom
shykes:europa-example-thechangelog

Conversation

@shykes
Copy link
Copy Markdown
Contributor

@shykes shykes commented Dec 15, 2021

@netlify
Copy link
Copy Markdown

netlify bot commented Dec 15, 2021

✔️ Deploy Preview for devel-docs-dagger-io ready!

🔨 Explore the source changes: 615a9a663276127bd588adf3ebab9cdf2b5eeabb

🔍 Inspect the deploy log: https://app.netlify.com/sites/devel-docs-dagger-io/deploys/61b9b925e4f9610007eed0c8

😎 Browse the preview: https://deploy-preview-1227--devel-docs-dagger-io.netlify.app

@netlify
Copy link
Copy Markdown

netlify bot commented Dec 15, 2021

✔️ Deploy Preview for devel-docs-dagger-io ready!

🔨 Explore the source changes: 5dbfa7beaaa9bfc09e9bbd13f47e8dccfef5289a

🔍 Inspect the deploy log: https://app.netlify.com/sites/devel-docs-dagger-io/deploys/61d4f6834818ce0007949f47

😎 Browse the preview: https://deploy-preview-1227--devel-docs-dagger-io.netlify.app

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From https://github.com/thechangelog/changelog.com/pull/401/files#r767925747

@shykes Can you make it work with something like this?

#Set: {
    config: #ImageConfig
    input: #Image
    output: {
        rootfs: input.rootfs
        "config": {
            // loop fields from input.config that are *not present* in config
            // loop fields from config
        }
    }
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that is one approach - wrap the mutation below in an engine operation perhaps.

The other approach would be to avoid mutation - by controlling what fields get filled into the base imageconfig, ensuring that the fields we want to set are not concrete.

Copy link
Copy Markdown
Contributor

@helderco helderco Dec 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's a POC: https://cuelang.org/play/?id=QGI_MUToOqa#cue@export@cue

It starts to get a bit hairy when you need to drill down on struct values and lists, if you want to do it all in CUE.

This is not a mutation though. You're not modifying, you're creating a new instance.

I like the idea of having an engine op that merges (overwriting values) two structs and spits out a new one. If they're the same definition, the result will also be valid for that definition. Downside is maybe cue eval?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you’re right, it’s not a mutation. What I meant is that it gives the developer the familiar abstraction of mutation (“set this value”) but actually implements it as a DAG node (immutable inputs and outputs).

I like the idea of having an engine op that merges (overwriting values) two structs and spits out a new one. If they're the same definition, the result will also be valid for that definition.

It’s definitely a fun idea :) cc @aluzzardi

Downside is maybe cue eval?

Could you elaborate?

Copy link
Copy Markdown
Contributor

@helderco helderco Dec 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, if you were to check if output.config has the intended keys resulting from the merge, that won't be available in pure cue.

Although, looking at the spec (are we supporting dockerfile as well?), I don't think you want to merge all maps/lists, just a few. You want to combine the Env lists and Labels maps, but not a Cmd list, for example. Anything other then those specific ones that need merging, would just be replaced.

I think this is doable in pure cue. I've implemented the map merging, but not lists. For Env you can make a temporary struct out of it, merge it, and spit out a list in the end. As for OnBuild, you'd just append, even if it causes duplicates.

Besides Env, Labels and OnBuild, seems to me everything else should be replaced (easy). Only Env requires a bit more cue code to do the merging.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll send a playground link when I can for you to see this in action.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://cuelang.org/play/?id=Wn_IAXkW_ZP#cue@export@cue

I've not handled OnBuild because their strings are Dockerfile instructions, that's another discussion.

Copy link
Copy Markdown
Contributor

@helderco helderco Dec 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@helderco I would like to take your proof-of-concept, and make it the official solution. Instead of a generic operation to merge any 2 CUE values, I think we should start with a domain-specific implementation in the docker package. That allows us to fix the immediate problem with the #ImageConfig schema, and postpone the harder work to make a perfectly general implementation. I will open an issue.

Copy link
Copy Markdown
Contributor

@helderco helderco Jan 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shykes Yes, and I don't think you will be able to have a fully generic solution in this case. For one, Env is special and you'd have to convert to a struct first (it's a list of strings in the key=value format). Also, there are some maps/lists that don't make sense to be merged, and should be replaced. You'd have to input exactly which keys you want merged in the generic solution.

@gerhard
Copy link
Copy Markdown
Contributor

gerhard commented Dec 15, 2021

Will look into this next week 🙌🏻

@jlongtine
Copy link
Copy Markdown
Contributor

@shykes @gerhard What I built is quite a bit different than this config, and only uses engine.* tasks. I've also verified it runs with latest dagger. I think it's building the prod container correctly.

I think it's a good place to start as working config. And I think we should build up the docker + mix packages as we get them working.

package main

import (
	// "alpha.dagger.io/europa/dagger"
	"alpha.dagger.io/europa/dagger/engine"
)

runtime_image_ref: string | *"thechangelog/runtime:2021-05-29T10.17.12Z"

engine.#Plan & {
	inputs: directories: app: {
		path: "."
		exclude: [
			".circleci",
			".dagger",
			".git",
			".github",
			"2021",
			"2022",
			"_build/dev",
			"_build/test",
			"assets/node_modules",
			"cue.mod",
			"dev_docker",
			"docker",
			"import",
			"nginx",
			"priv/db",
			"priv/uploads",
			"script",
			"tmp",
			".all-contributorsrc",
			".autocomplete",
			".credo.exs",
			".dockerignore",
			".formatter.exs",
			".envrc",
			".env",
			".gitattributes",
			".gitignore",
			"README.md",
			"coveralls.json",
			"start_dev_stack.sh",
			".kube",
			"erl_crash.dump",
			"deps",
			"_build",
			"dagger",
			"main.cue",
		]
	}
	inputs: directories: docker: {
		path: "."
		include: [
			"docker/Dockerfile.production",
			".dockerignore",
		]
	}

	actions: {
		runtimeImage: engine.#Pull & {
			source: runtime_image_ref
		}

		depsCache: engine.#CacheDir & {
			id: "depsCache"
		}

		depsCacheMount: "depsCache": {
			dest:     *"/app/deps/" | string
			contents: depsCache
		}

		buildCacheTest: engine.#CacheDir & {
			id: "buildCacheTest"
		}

		buildCacheTestMount: "buildCacheTest": {
			dest:     *"/app/_build/test" | string
			contents: buildCacheTest
		}

		buildCacheProd: engine.#CacheDir & {
			id: "buildCacheProd"
		}

		buildCacheProdMount: "buildCacheProd": {
			dest:     *"/app/_build/prod" | string
			contents: buildCacheProd
		}

		nodeModulesCache: engine.#CacheDir & {
			id: "nodeModulesCache"
		}

		nodeModulesCacheMount: "nodeModulesCache": {
			dest:     *"/app/assets/node_modules" | string
			contents: nodeModulesCache
		}

		appImage: engine.#Copy & {
			input: runtimeImage.output
			source: {
				root: inputs.directories.app.contents
			}
			dest: "/app"
		}

		deps: engine.#Exec & {
			input:   appImage.output
			mounts:  depsCacheMount
			workdir: "/app"
			args: ["bash", "-c", " mix deps.get"]
		}

		// testDeps: engine.#Exec & {
		// 	input:  deps.output
		// 	mounts: depsCacheMount
		// 	args: ["bash", "-c", "ls -al /app /app/deps"]
		// }

		assetsCompile: engine.#Exec & {
			input:   depsCompileProd.output
			mounts:  depsCacheMount & nodeModulesCacheMount
			workdir: "/app/assets"
			env: PATH: "/usr/local/lib/nodejs/node-v14.17.0-linux-x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
			args: ["bash", "-c", "yarn install --frozen-lockfile && yarn run compile"]
		}

		#depsCompile: engine.#Exec & {
			input:   deps.output
			mounts:  depsCacheMount
			workdir: "/app"
			args: ["bash", "-c", "mix do deps.compile, compile"]
		}

		depsCompileTest: #depsCompile & {
			env: MIX_ENV: "test"
			mounts: buildCacheTestMount
		}

		depsCompileProd: #depsCompile & {
			env: MIX_ENV: "prod"
			mounts: buildCacheProdMount
		}

		assetsDigest: engine.#Exec & {
			input:  assetsCompile.output
			mounts: depsCacheMount & buildCacheProdMount & nodeModulesCacheMount
			env: MIX_ENV: "prod"
			workdir: "/app"
			args: ["bash", "-c", "mix phx.digest"]
		}

		imageProdCacheCopy: engine.#Exec & {
			input:  assetsDigest.output
			mounts: (depsCacheMount & {depsCache: dest:           "/mnt/app/deps/"} )
			mounts: (buildCacheProdMount & {buildCacheProd: dest: "/mnt/app/_build/prod"} )
			args: ["bash", "-c", "cp -Rp /mnt/app/deps/* /app/deps/ && cp -Rp /mnt/app/_build/prod/* /app/_build/prod/"]
		}

		imageProdDockerCopy: engine.#Copy & {
			input: imageProdCacheCopy.output
			source: {
				root: inputs.directories.docker.contents
			}
			dest: "/"
		}

		imageProd: engine.#Build & {
			source: imageProdDockerCopy.output
			dockerfile: path: "/docker/Dockerfile.production"
			buildArg: {
				APP_FROM_PATH: "/app"
				GIT_AUTHOR:    "joel"
				GIT_SHA:       "abcdef"
				APP_VERSION:   "main"
				BUILD_URL:     "longtine.io/build"
			}
		}

		testImageProdCacheCopy: engine.#Exec & {
			input:  imageProdDockerCopy.output
			always: true

			args: ["bash", "-c", "ls -al / /docker /app/deps/ /app/_build/prod/ /app/priv/static"]
		}
	}
}

@shykes
Copy link
Copy Markdown
Contributor Author

shykes commented Jan 6, 2022

As discussed with @jlongtine and @gerhard:

  • I’m merging this now and continue developing iteratively. This is to facilitate collaborative development without slowing us down.
  • I moved my implementation in europa-universe/examples/changelog.com/highlevel and will continue to develop there
  • I created europa-universe/examples/changelog.com/lowlevel for @jlongtine to add his implementation, and continue developing there.

Thanks all!

@shykes shykes merged commit 8b5a801 into dagger:main Jan 6, 2022
@shykes shykes deleted the europa-example-thechangelog branch January 6, 2022 00:58
@shykes shykes changed the title [draft] Europa: Example configuration for changelog.com CI Europa: Example configuration for changelog.com CI Jan 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants