Canary in a coal mine

This project provides a tool for exercising participating projects with a pinned bleeding-edge ClojureScript compiler version. The goal is to have some smoke tests which allow us to detect issues early before releasing a new ClojureScript version.

How to Participate

If you are using Travis for CI, participating in Canary builds involves a few simple steps. (Before doing this, ask someone in the cljs-oss organization for commit rights.)

1. Revise your .travis.yml to fetch the Canary-built ClojureScript JAR.

   before_install:
     - curl -sSL https://raw.githubusercontent.com/cljs-oss/canary/master/scripts/install-canary.sh | bash
   

2. Configure your project's build so that it honors the CANARY_CLOJURESCRIPT_VERSION environment variable. If using Leiningen, this can be accomplished by revising your ClojureScript dependency in project.clj to look like:

   [org.clojure/clojurescript ~(or (System/getenv "CANARY_CLOJURESCRIPT_VERSION") "1.9.946")]
   

3. Obtain Travis Access token:

   Probably the easiest way is to copy it from your Travis UI web interface (look under Settings/Settings/API authentication).

   Note that an alternative way to obtain a Travis Access token is by doing gem install travis && travis login && travis token. See their api reference for updated info on how to retrieve access tokens.

4. Set up Canary with a Travis access token. Replace YOUR_PROJECT with the name of your project and deadbeef below with your Travis Token obtained above.

   git clone --branch jobs [email protected]:cljs-oss/canary.git
   cd canary
   travis encrypt CANARY_YOUR_PROJECT_TRAVIS_TOKEN=deadbeef --add env.global
   git add .travis.yml
   git commit -m "add CANARY_YOUR_PROJECT_TRAVIS_TOKEN [ci skip]"
   git push origin jobs
   

   Warning: Please make sure that you run travis encrypt inside your canary checkout of the jobs branch as shown above. Running it outside seems to produce some encrypted value, but the result is corrupted when actually used by travis.

5. Add a Canary task to trigger CI builds of your project. This is done by adding a ^:task to a namespace in the master branch under runner/src/canary/projects, like this (replacing github-yourname, your-project and YOUR_PROJECT):

   (ns canary.projects.github-yourname
     (:require [canary.runner.travis :as travis]))
   
   (defn ^:task your-project [options]
     (travis/request-build! "github-yourname/your-project" "CANARY_YOUR_PROJECT_TRAVIS_TOKEN" options))
   

After setting this up, you can force an out-of-cycle test of your job going into the jobs branch and making a commit like this (replacing your-project) and pushing it:

git commit --allow-empty -m "job --only your-project"

More Details

Canary provides a script which can be used to run a job. Each job is assigned a ClojureScript compiler version. It builds compiler jar, uploads it and then runs exercises of individual projects in parallel (we call them tasks). Finally it waits for task results and generates a report for archiving.

Typically you don't run canary script on your local machine but rather invoke it on some machine in the cloud. For your convenience a new job can be triggered by committing an empty commit into the jobs branch. Travis machine will then execute canary script possibly triggering more child Travis builds for individual project tasks. At the end, results are generated and archived in the results branch. Also look into GitHub releases of this repo where individual built compiler versions get published.

Quick test

via docker

./scripts/docker-build.sh
./scripts/docker-run.sh help
./scripts/docker-run.sh list
./scripts/docker-run.sh job
./scripts/docker-clean.sh # to start from scratch

directly

./runner/run.sh job -v -r master
./scripts/blow-local-caches.sh # to start from scratch

Vocabulary

• the script is called runner
• ClojureScript compiler is usually simply referred to as the compiler
• a list of relevant projects/libraries is called projects
• a single request for a complete round of tests is called a job
• a single project test is called a task

Main ideas

A single repo

Instead of maintaining multiple git repos, let's have just a single mono-repo canary with following branches:

1. master where the source code for the runner lives
2. jobs where anyone with commit rights can trigger a new job (as a new commit)
3. results where the runner will produce one commit per job

All participating project authors will get commit access to this repo and can collaborate on the master branch.

The job workflow

When a job is triggered. The runner goes through following steps:

1. prepares compiler's jar according to requested parameters (repo/version)
2. determines which tasks should be part of the job based on request parameters
3. spawns all tasks (in parallel) instructing them to use the compiler
4. waits for tasks results and collects them
5. generates a report and commits it in the results branch

Implemented in Clojure

The runner is implemented in Clojure for convenience. Each project author gets own place (function/script) for implementing functionality specific for their project. Authors are expected to trigger test builds on their own repos (e.g. via Travis) and simply collect results back. But anyone can use an escape hatch to invoke a shell script or do whatever they need in Clojure. For example cloning their repo and running tests directly in the context of the runner.

Docker

We wrap runner in a Docker container to provide well-controlled environment for tasks (possibly shell scripts).

No need for fancy publishing

We are developers and we have git and GitHub which is a great publishing platform on its own. Travis alone will provide some trace of a job (depending on what runner script and individual tasks output to stdout). But ultimately a commit into results branch presents all interesting results also for archivation purposes. Then anyone can use their git-fu to follow those or process them further.

FAQ

How to run a job?

Please read the readme in the jobs branch with details on that.

How to run a job with specific ClojureScript fork?

No problem. You can point Canary to your own fork of ClojureScript by specifying --compiler-repo and --compiler-rev parameters.

Can you give me more details on how can I can participate with my project?

You will need to write a new task for your project. First look how existing projects are implemented. Ask for commit access. You can write your task as a Clojure function or as a shell script.

In case of a Clojure function you have to annotate it with ^:task metadata so that runner recognizes it. In general you can do whatever you need to do in your task (you can block, it is running on a separate thread) - this will depend on your project setup. To ease some common scenarios we provide some helper functions. For example canary.runner.travis namespace might be very useful for triggering a Travis build for your own project.

In case of a shell script. You simply create your_name.sh in the projects directory. Script has to return with zero exit code to be considered as passing. Standard outputs will be embedded into the final report, so don't be too verbose there.

You can test your task locally. Running jobs without --production flag should do no harm. When using travis/request-build! it will mock it by default. You will have to add --production as the last step for fine-tuning final version of the code.

How can I trigger Travis build of my project?

For inspiration here is the task for cljs-devtools, and here is the adaptation needed in the project itself. Please note that child Travis build triggered by travis/request-build! is configured with bunch of extra env variables prefixed with CANARY_ - those have to be taken into account by the participating project (an example here). Also for us to be able to trigger builds of your Travis projects you will also have to provide a Travis API token for triggering the builds on your behalf.

You should encrypt your token and commit it into .travis.yml in jobs branch (replace YOUR_PROJECT and deadbeef):

git clone --branch jobs [email protected]:cljs-oss/canary.git
cd canary
travis encrypt CANARY_YOUR_PROJECT_TRAVIS_TOKEN=deadbeef --add env.global
git add .travis.yml
git commit -m "add CANARY_YOUR_PROJECT_TRAVIS_TOKEN [ci skip]"
git push origin jobs

How can I trigger Circle CI build of my project?

You are on your own. We haven't implemented similar helpers as for Travis yet. Want to contribute them? That would be great.

---

Credits

Inspired by ideas of @deraen and @mfikes.

You might discuss this in #cljs-dev channel on clojurians.net Slack.