Navigating all the content around Jenkins can look like an impossible challenge, especially for newbies. In this post we'll explore one of its most successful and useful plugins - Jenkins Config as Code.
If you are using Jenkins as your CI server you've probably heard about the Jenkins config as code plugin. It enables automation of the tool setup whilst providing better control and visibility of your configuration. The plugin (https://plugins.jenkins.io/configuration-as-code/) makes it possible to configure Jenkins instances using YAML files for the configuration. I would highly recommend that anyone administrating Jenkins master(s) should check it out, and this blog should help you to get up and running.
We've observed that for newcomers to Jenkins, or even experienced admins, adopting config as code can be a big challenge because most online examples are quite complicated and use orchestrators like Kubernetes, adding an extra layer of difficulty to the setup. So, we thought we'd strip back all the layers of difficulty that are not associated with Jenkins, and provide some basic examples using only Docker. After all, we still need some kind of environment to run Jenkins in :)
Our tutorial should give you a very basic initial setup for you to build on. We will take you through the following steps:
-
Build a Docker image based on the community Jenkins image with a minimal config-as-code setup
-
Extend the config-as-code setup
-
Talk about next steps and tips
Prerequisite#
The only prerequisite for this tutorial is that you have Docker installed and are able to run the community Jenkins image.
1docker run --rm --name jenkins jenkins/jenkins:lts-jdk11
1. Minimal Config as Code#
If you follow the documentation on the config-as-code plugin page you'll find concise information on getting started. To summarize, there are 2 things that we need to do:
-
Install the config-as-code plugin to Jenkins
-
Setup the environment variable
CASC_JENKINS_CONFIGto tell Jenkins where to find the YAML configuration
How this looks in a Dockerfile is as follows, with some useful comments:
1# Dockerfile
2
3FROM jenkins/jenkins:lts-jdk11
4
5# copy the list of plugins we want to install
6COPY plugins.txt /usr/share/jenkins/plugins.txt
7# run the install-plugins script to install the plugins
8RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt
9
10# disable the setup wizard as we will set up jenkins as code :)
11ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
12
13# copy the config-as-code yaml file into the image
14COPY jenkins-casc.yaml /usr/local/jenkins-casc.yaml
15# tell the jenkins config-as-code plugin where to find the yaml file
16ENV CASC_JENKINS_CONFIG /usr/local/jenkins-casc.yaml
And the list of plugins can be listed in a plugins.txt file which looks as follows:
1# plugins.txt
2
3configuration-as-code
And a basic jenkins-casc.yaml file which can contain something basic like:
1# jenkins-casc.yaml
2
3jenkins:
4 securityRealm:
5 local:
6 allowsSignup: false
7 users:
8 # create a user called admin
9 - id: "admin"
10 password: "admin"
11 authorizationStrategy: loggedInUsersCanDoAnything
This creates a base Jenkins master with a username called admin who can login and do anything because of the authorizationStrategy: loggedInUsersCanDoAnything.
Let's test that this basic setup works.
1# build the docker image
2docker build -t jenkins-casc:0.1 .
3
4# run the docker image and map ports 8080
5docker run --rm --name jenkins-casc -p 8080:8080 jenkins-casc:0.1
6# ...
7# ...
8# INFO hudson.WebAppMain$3#run: Jenkins is fully up and running
Once you see the message Jenkins is fully up and running you can go to your browser <localhost:8080> and you should see your Jenkins instance running. Success!
Now let's continue to develop the Jenkins instance further.
2. Extending config-as-code setup#
The above minimal example should have given you a basic setup to develop further.
Now we are going to add two basic things to this setup:
-
Add another user
devthat is not an administrator and setup permissions using the matrix-auth plugin -
Create some initial Jenkins pipelines
2.1 Add a dev user and permissions#
First we need to install the matrix-auth plugin.
To add plugins simply add them to the plugins.txt
1# plugins.txt
2
3configuration-as-code
4
5matrix-auth
Now that the plugin is available we can add the configurations to the config-as-code YAML file.
1# jenkins-casc.yaml
2
3jenkins:
4 securityRealm:
5 local:
6 allowsSignup: false
7 users:
8 - id: "admin"
9 password: "admin"
10 - id: "dev"
11 password: "dev"
12 authorizationStrategy:
13 globalMatrix:
14 permissions:
15 - "Job/Build:dev"
16 - "Job/Cancel:dev"
17 - "Job/Read:dev"
18 - "Job/Workspace:dev"
19 - "Overall/Administer:admin"
20 - "Overall/Read:authenticated"
21 - "Run/Replay:dev"
22 - "Run/Update:dev"
2.2 Create some initial pipeline jobs#
The Jenkins config-as-code plugin uses the Jenkins job-dsl plugin plugin for handling jobs in Jenkins.
There is a nice interactive API for this plugin so you can check what's available: https://jenkinsci.github.io/job-dsl-plugin/
This is obviously specific to your Jenkins instance and the plugins that you have available, and the API viewer is available for your instance once you have installed the plugin at <localhost:8080/plugin/job-dsl/api-viewer/index.html>
Let's add this plugin to plugins.txt as well as the pipeline plugins that we need to create some pipeline jobs in Jenkins
1# plugins.txt
2
3configuration-as-code
4job-dsl
5
6blueocean
7workflow-job
8workflow-cps
9
10matrix-auth
Now that we have the necessary plugins installed, let's add the configuration to the config-as-code YAML file.
I always like to split the job-dsl groovy scripts to create jobs away from the YAML, and this can be done by specifying a path to a file that will exist inside our Jenkins Docker image.
1# jenkins-casc.yaml
2---
3# specify the seedjob groovy script that will create the pipelines for us
4jobs:
5 - file: /usr/local/seedjob.groovy
See the full config-as-code YAML file here:
1# jenkins-casc.yaml
2
3jenkins:
4 securityRealm:
5 local:
6 allowsSignup: false
7 users:
8 - id: "admin"
9 password: "admin"
10 - id: "dev"
11 password: "dev"
12 # authorizationStrategy: loggedInUsersCanDoAnything
13 authorizationStrategy:
14 globalMatrix:
15 permissions:
16 - "Job/Build:dev"
17 - "Job/Cancel:dev"
18 - "Job/Read:dev"
19 - "Job/Workspace:dev"
20 - "Overall/Administer:admin"
21 - "Overall/Read:authenticated"
22 - "Run/Replay:dev"
23 - "Run/Update:dev"
24
25 # make sure our jenkins master has 1 executor so that we can run our pipelines
26 numExecutors: 1
27
28# specify the seedjob groovy script that will create the pipelines for us
29jobs:
30 - file: /usr/local/seedjob.groovy
Here is an example seedjob.groovy script to show you some of the things you can do:
1// seedjob.groovy
2
3// create an array with our two pipelines
4pipelines = ["first-pipeline", "another-pipeline"]
5
6// iterate through the array and call the create_pipeline method
7pipelines.each { pipeline ->
8 println "Creating pipeline ${pipeline}"
9 create_pipeline(pipeline)
10}
11
12// a method that creates a basic pipeline with the given parameter name
13def create_pipeline(String name) {
14 pipelineJob(name) {
15 definition {
16 cps {
17 sandbox(true)
18 script("""
19// this is an example declarative pipeline that says hello and goodbye
20pipeline {
21 agent any
22 stages {
23 stage("Hello") {
24 steps {
25 echo "Hello from pipeline ${name}"
26 }
27 }
28 stage("Goodbye") {
29 steps {
30 echo "Goodbye from pipeline ${name}"
31 }
32 }
33 }
34}
35
36""")
37 }
38 }
39 }
40}
Finally we need to update our Dockerfile to copy the seedjob.groovy file into it:
1# Dockerfile
2
3FROM jenkins/jenkins:lts-jdk11
4
5# copy the list of plugins we want to install
6COPY plugins.txt /usr/share/jenkins/plugins.txt
7# run the install-plugins script to install the plugins
8RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt
9
10# disable the setup wizard as we will set up jenkins as code :)
11ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
12
13# copy the seedjob file into the image
14COPY seedjob.groovy /usr/local/seedjob.groovy
15# copy the config-as-code yaml file into the image
16COPY jenkins-casc.yaml /usr/local/jenkins-casc.yaml
17# tell the jenkins config-as-code plugin where to find the yaml file
18ENV CASC_JENKINS_CONFIG /usr/local/jenkins-casc.yaml
Let's test this extra config as before.
1# build the docker image, we can give it a new version
2docker build -t jenkins-casc:0.2 .
3
4# run the docker image and map ports 8080
5docker run --rm --name jenkins-casc -p 8080:8080 jenkins-casc:0.2
6# ...
7# ...
8# INFO hudson.WebAppMain$3#run: Jenkins is fully up and running
Once you see the message Jenkins is fully up and running you can go to your browser <localhost:8080> and you should see your Jenkins instance running.
Login with user admin and password admin and you should see your two pipelines.
You can also login as user dev with password dev if you want the experience of a developer.
3. Next steps & tips#
Hopefully, this tutorial has given you a very basic introduction to how config-as-code works and will help you configure a Jenkins instance automatically.
Given that all the configurations are version controlled, and there's a process behind handling updates and deployments of Jenkins masters, config-as-code should be the way people now maintain their Jenkins instances. It's the easiest way to avoid typical problems like Jenkinstein and having to manage updates in a maintenance window and hope that nothing goes wrong :)
So, next steps.
3.1 Adding more configuration to your config-as-code YAML#
My approach to configuring Jenkins and plugins/features that I have not used before is to always do things manually first.
Read the docs, spin up a test instance, play around with the UI and try to get things working. Once I have a working recipe, check the config-as-code configuration for my Jenkins instance at <localhost:8080/configuration-as-code/viewExport>
There are nice examples in the configuration-as-code github repository demo folder.
Also, if you search the repository for any snippets you are looking for there are likely some tests for that feature that you could copy bits of working YAML from.
But, don't expect this to work first time - you'll probably need to be patient and experiment a little.
Finally, you can check the "documentation" for your Jenkins instance. I've still not fully figured out how the syntax should look even after 2+ years working with the plugin.
The link is available on your running instance of Jenkins at <localhost:8080/configuration-as-code/reference>
3.2 Test, staging, production, etc environments#
The types of environments you want to define are up to you, but you should have at least one other environment separate from production.
Test things before you deploy to production, and have a test environment for you to play around in.
Try not to experiment with new features or possible breaking changes in
the production instance.
However mild the changes are, things always go wrong :)
3.3 Build into image or make a configuration#
A common question that comes up in our projects is whether the Jenkins
config-as-code YAML files and groovy scripts should be built into the
Docker image (as we did in our example), or if they should be mounted to
the container at runtime, using something like Kubernetes ConfigMaps, Nomad templates, or Docker configs.
My general rule-of-thumb is that maintaining Docker images is an extra overhead and most of these configs are runtime configs.
So, I would remove them from the Docker build stage and make them some kind of mountable config.
The exception to this rule is the plugins.txt.
Plugins can get messy and I have ended up in many debates about strictly versioning the plugins you install vs getting the latest. Let's take an example:
1# plugins.txt
2
3configuration-as-code
4
5# vs
6
7configuration-as-code:1.43
In the first case Jenkins will get the latest version of the plugin. In the second case it specifically installs version 1.43.
The main reason for not getting the latest is to make things deterministic.
However, if you build the plugins.txt and the plugin installation process into the Docker image, then this process is deterministic because the plugins are installed into the image and are not installed at runtime.
So, no matter how or where you run the Docker image, you will always have the same plugin versions.
On the flip side, managing specific versions of plugins means you have to continuously check the versions and update them.
Personally, I go with the former method where we use the latest versions
of plugins and build them into the Docker image, mainly because it
feels like less maintenance whilst remaining deterministic.
3.4 Orchestrators#
In a project with proper infrastructure you are likely using some kind of orchestrator like Kubernetes, Nomad, or Docker Swarm.
The purpose of this tutorial was to strip away the added complexity that comes with these tools.
My hope would be that you can now use some of the orchestrators' capabilities and community support to improve your deployment and configuration of Jenkins.
For example, if you are using Kubernetes, then there is a very powerful Helm chart available for Jenkins.
We have been using Flux to deploy Jenkins using HelmReleases.
For newcomers to this environment, there are a lot of extra layers between the deployment and Jenkins config-as-code, so it is not exactly a gentle introduction. It does provide a much better experience from the operations side, though.
Consider this when you move your Jenkins config-as-code to production.
3.5 Plugins#
One of the things that made Jenkins so popular and easy to use is its plugin
ecosystem. Conversely, the plugin ecosystem can become a bit of a nightmare
when ensuring cross-plugin compatibility, maintenance, upgrades, and so on.
So, the question arises: should you use plugins? Using Jenkins without
them would make it pretty useless because everything is implemented as a
plugin, including pipelines, config-as-code, job-dsl, docker, kubernetes, etc.
The most important thing is to ensure you have a minimal set of plugins to help you achieve your goals. Also, and this goes for any OSS tool, framework, library,
package, etc. - check the stability of the project and the community around it. Is
it being maintained? Are issues resolved? Are there any any other factors that can help
you reach a good judgement on whether to use a plugin or not.
Fortunately, if you have made it this far in the tutorial you will be jubilant
to hear that managing Jenkins using Config as Code is a VERY GOOD approach
to avoiding Jenkins plugin bloat and ending up with a Jenkinstein managing
your CI process.
Conclusion#
Hopefully this tutorial has given you a basic introduction to Jenkins config-as-code and given you a working setup to develop. We will be publishing a tutorial on Jenkins pipelines as a natural follow-up to this.
