This article explains how to implement the pipeline described in this article (and summed up in the diagram below) for Docker images with Jenkins (I used version 2.145 for my tests).

We assume Jenkins runs on nodes. The pipeline can be adapted to run in containers. However, running tests on Docker images might be harder. Solutions like Docker in Docker, and most of all, Docker outside of Docker (share the host’s Docker daemon inside a container to create siblings) may work. However, running tests of Docker containers is easier when the Jenkins agent runs directly on a node. And Docker outside of Docker is not considered as a safe practice.

Let’s see what we need for Jenkins:

• A Jenkins master, with the Pipeline plug-in.
• Jenkins agents that run on (virtual) machines.
• At least one agent node with a Docker daemon installed. It will be used to build images and run tests.
• The same agent node should have access to a Clair instance (through Clair-Scanner) or to a Dagda install.

Here is the Jenkins topology with Clair.
In this diagram, the Clair server would be managed by the security team/role. Not every project should have its own instance of Clair, that would make no sense. It is something that should be common.

The topology would be almost the same with Dadga.
To integrate Dadga in a CI process, you have two options. Dagda being a Python script, either you create a web service to invoke during the build process (that will execute the Python script), or you deploy its database on a separate VM, and you execute Dadga as a container on the Jenkins slave. This container should connect to the remote database. That would be the best option to have quick scans. For the notice, Dadga uses MongoDB instead of PostgreSQL.

It is also important to secure the Docker daemon on the node.
As I do not want to spend too much time on it, here are some links:

• https://docs.docker.com/engine/security/
• https://docs.docker.com/engine/security/security/#other-kernel-security-features
• https://thenewstack.io/5-docker-security-best-practices/
• https://github.com/docker/docker-bench-security

Pipeline

With Jenkins, the convention is to define pipelines in a file named Jenkinsfile.
To make ours simple, we provide an all-in-one package. Projects only have to provide parameters and their own tests for their Docker image. The following sample gives an idea about what such a project repository would look like…

image/
- Dockerfile
- ...
tests/
- execute-tests.sh
- ...
Jenkinsfile
readme.md

The project repository contains resources to build the image, resources to test it, and a Jenkinsfile, that describes the build process for Jenkins. Here is the content of this file.

allInOne(
    imageName: 'my-image-name',
    imageVersion: '1.0.0'
)

allInOne is not something Jenkins recognizes by default.
In fact, it refers to a Jenkins shared library we need to define. There will be 3 of them: one for the composite pipeline (« allInOne »), one for the security team, and one for the governance team. Each shared library is defined in its own Git repository, each one having its own permissions. This way, the security team can be sure only it can access/commit about security stuff. Same thing for the Software governance team. And they can all have read access to the shared library that agregates the various stages.

Shared library for the composite pipeline

This is the shared library to use for the « allInOne » tag.
Jenkins pipeline allows to define three things in a shared library: helpers, a pipeline or a step. What would have been better would have been defining stages. But this is not possible. Our allInOne library will thus declare a pipeline with parameters. Here is the main code:

// In vars/allInOne.groovy (shared library that defines the generic pipeline)
def call(Map config) { 
 
	node { 
		def timeStamp = Calendar.getInstance().getTime().format('YYYYMMdd-hhmmss', TimeZone.getTimeZone('Europe/Paris'))
		def buildId = "${config.imageVersion}-${timeStamp}"

		stage('Checkout') { 
			echo "Checking out the sources..." 
			checkout scm 
		}

		stage('Build Image') {
			// Enforce the shape of the repository and assume the Dockerfile is always under image/
			sh 'docker build -t "${config.imageName}:${buildId}" image/'
		}

		stage('Project tests') {
			def scriptFileContent = libraryResource( 'com/linagora/execute-project-tests.sh' )
			sh scriptFileContent
		}

		stage('Security checks') {
			echo "Checking security..."
			securityInspection( "${config.imageName}", "${buildId}" )
		}

		stage('Software Governance') {
			echo "Handling Software checks..."
			softwareCheck( "${config.imageName}", "${buildId}" )
		}

		stage('Promotion') {
			echo "Promoting the local image to a trusted repository..."
			def scriptFileContent = libraryResource( 'com/linagora/promote-image.sh' )
			sh scriptFileContent
		}
	}
}

All the stages are sequential. If one fails, everything stops. The securityInspection and softwareCheck steps are defined farther as global shared libraries. The promote-image.sh and execute-project-tests.sh scripts are provided as resources of the shared library. You can find the whole project online, on Github.

Notice the build stage is very basic.
In this example, we do not handle build parameters. This part can easily be adapted.

Project tests

How can a project specify tests for its Docker image?
Well, this is not complicated. I already faced the issue with a previous project. The idea is the following:

1. Have a script that launches tests.
2. This script instantiates the image in detached mode, with a shared volume.
3. We then execute another script inside the container, by using docker exec.
4. This second script is in charge of verifying assertions inside the container. Example: verify some process is running, verify a given file was created, etc. When an assertion fails, a message is written inside a file, located in the shared volume. When the container terminates, this file will remain available on the host system.
5. Once the script has completed, the first script can verify the content of the error file. If there are errors, it can just fail the build.
6. It is also possible to verify some assertions about the container from the outside (e.g. try to ping or reach a given port).

Here is another repository that illustrates this strategy.
It was used to test RPM and Debian packages in Docker containers. The script that launches the tests and verifies the result is here. Scripts that verify assertions in the containers are located in this directory.

This kind of tests works for functional tests, but it can also be applied for security checks (e.g. permissions on files) or governance ones. The difference here is that these tests lie in the same Git repository than the Dockerfile.

To run them, the allInOne pipeline must verify some script test exists and start it. A naming convention is enough.

Security checks

Security checks are maintained by the security team/role as a shared library, hosted in its own Git repository. Security checks can be verified with a script (e.g. permissions, etc). But for Docker images, there also exist solutions that inspect images and detect leaks.

Clair, from CoreOS, is one of them.
This project uses a database of known vulnerabilities, and scans images. It then provides a dashboard, indicating which CVE were found and for which images.

Dadga is another solution.
Although it is more recent, it provides additional features than Clair. It works the same way, but also uses ClamAV to search for virus, trojans and so on. What takes time for both Dadga and Clair is to update the database of known vulnerabilities. This is why the database cannot be deployed on the fly during the build process. It must preexist to pipeline executions.

Both solutions can also be used to regularly control Docker images. But since the focus of these articles is about delivery pipelines, you will have to dig this part yourself. What is interesting is to integrate these tools in the delivery pipeline, as a validation step. To keep the example simple, I only include Clair. Clair is great, but using it as a command-line is not easy. The best option is to use Clair Scanner as a complement.

And here is our shared library…

// In vars/securityInspection.groovy (shared library for the security role)
def call(String imageName, String buildId) { 

	// We assume clair-scanner is available in the path
	def host = sh(returnStdout: true, script: 'hostname -i').trim()
	clair-scanner -c <CLAIR_SERVER_URL> --ip ${host} --t High ${imageName}:${buildId}
}

Here, Clair will scan the given image. If vulnerabilities are found with a severity higher or equal to high, then clair-scanner will return a non-zero exit code and thus fail the build.

If you use Dadgda instead of Clair, you simply run a Python script. The installation is a little bit different, but the pipeline step would remain simple. You can also add custom scripts to perform additional verifications (just add new steps in the pipeline).

In addition, one could also use Anchore, an open source solution to perform static analysis and check custom policies against Docker images. I have found it after writing this article, so I just put a mention here.

Software Governance

Software governance can be managed in the same way that previous stages.
Since it depends on the organization itself, I have no generic tool to suggest. Instead, I assume there is some REST end-point somewhere that can be contacted during the build. The goal is to extract information and send them to a remote web service that will store them, and optionally trigger an alert or a build failure in case of exotic findings.

So, here is an example of associated shared library…

// In vars/softwareCheck.groovy (shared library for the Software Governance role)
def call(String imageName, String buildId) { 

	def scriptFileContent = libraryResource( 'com/linagora/analyze-dockerfile.sh' )
	sh scriptFileContent
	sh 'echo "imageName: ${imageName}" >> /tmp/gov.results.txt'
	sh 'echo "imageVersion: ${buildId}" >> /tmp/gov.results.txt'
	sh 'curl --data-binary "@/tmp/gov.results.txt" -X POST...'
	sh 'rm -rf /tmp/gov.results.txt'
}

Pipeline for existing Docker images

The initial pipeline includes a stage to build an image.
However, a project team may want to use an existing image. Community images benefit from various feedbacks and contributors. How to guarantee a project can safely use it inside an organization with its own guidelines?

Well, our generic pipeline, with project tests, security checks and governance, perfectly fits such a use case. The guidelines should be enforced in the automated pipeline. The only difference is that we do not build an image, we use an existing one from the outside. So, let’s adapt our allInOne shared library to cover such a scenario.

// In vars/allInOne.groovy (shared library that defines the generic pipeline, upgraded to support existing images)
def call(Map config) { 

	node { 
		def timeStamp = Calendar.getInstance().getTime().format('YYYYMMdd-hhmmss', TimeZone.getTimeZone('Europe/Paris'))
		def buildId = "${config.imageVersion}-${timeStamp}"

		// Alway checkout the sources, as they may include tests
		stage('Checkout') { 
			echo "Checking out the sources..." 
			checkout scm 
		}

		if (config.existing == true) {
			stage('Docker pull') {
				def buildId = "${config.imageVersion}"
				sh 'docker pull "${config.imageName}:${buildId}"'
			}
		}

		if (config.existing != true) {
			stage('Build Image') {
				// Enforce the shape of the repository and assume it is always under image/
				sh 'docker build -t "${config.imageName}:${buildId}" image/'
			}
		}

		stage('Project tests') {
			def scriptFileContent = libraryResource( 'com/linagora/execute-project-tests.sh' )
			sh scriptFileContent
		}

		stage('Security checks') {
			echo "Checking security..."
			securityInspection( "${config.imageName}", "${buildId}" )
		}

		stage('Software Governance') {
			echo "Handling Software checks..."
			softwareCheck( "${config.imageName}", "${buildId}" )
		}

		stage('Promotion') {
			echo "Promoting the local image to a trusted repository..."
			def scriptFileContent = libraryResource( 'com/linagora/promote-image.sh' )
			sh scriptFileContent
		}
	} 
}

If we use an existing image we simply pull it. Otherwise, we build it. The other parts of the pipeline are the same.
A project using an existing image would then declare its Jenkinsfile as…

allInOne(
    imageName: 'my-image-name',
    imageVersion: '1.0.0',
    existing: true
)

Integration in Jenkins

The most simple solution is to define our 3 shared libraries as global shared libraries. Besides, the shared libraries above all expose global variables, which avoids using import declarations in our Jenkinsfiles. To do so, go into Jenkins administration, system configuration and find the pipeline section. The shared library will be loaded on the fly from a git repository, in every job. It is never cached. For security reasons, the (Jenkins) user that pulls the repository should only have read access.

Here is a screenshot that shows how a shared library is defined in Jenkins.

You can add as many ones as necessary. This article has defined 3 shared libraries, so you would add 3 into Jenkins. It is possible to set versions to shared libraries, but I think it is not necessary for global shared libraries. If stages had to differ between projects, you would define different composite pipelines. And the stages are managed on the fly, in the same fashion than 12 factors.

Perimeter

Two questions can arise once we are here.

• Where does this pipeline lead?
• How to prevent a project from by-passing these checks?

The answer to the first question influences the second one.
In my opinion, this pipeline promotes the built image into a trusted Docker registry, that can then be used in Kubernetes environments. You cannot test this image in a K8s cluster before (or if you can, then it must be a sandbox cluster). Once this is clear, the second answer becomes obvious. A project cannot by-pass this pipeline because otherwise, it cannot import its Docker images in the trusted registry. The allInOne shared-library is the only part that can access the trusted registry. It cannot be done from anywhere else: the credentials are kept secret and a single user (Jenkins) should have the write permissions to push an image in the trusted registry. All the other users have read-only access.

Summary

This article has shown how to use Jenkins shared libraries to build a validation process for Docker images, process that is under different responsibilities. I skipped some details to not lost readers, but I believe the main aspects are all explained here.

The next article is about Helm packages: how to verify quality criteria (e.g. linting), test them and so on. All of this with a Jenkins pipeline and the same distribution of responsibilities.