Jenkins is a well-proven CI tool and a key link in the software delivery chain of many organisations, but managing one or more Jenkins instances adds an overhead to your teams. Defining your Jenkins instances as code and automating their scaling is a great way to reduce this management overhead, while improving the maintainability, stability and disaster recovery of your deployments.
In this blog we use Terraform to create a Kubernetes Cluster in Azure Cloud onto which we install our Jenkins Controller using Helm. Both the Jenkins Controller and the Agents are defined as code; our Jenkins Controller using JCasC (Jenkins Configuration as Code), and our Jenkins Agents as part of our Declarative Pipelines.
Introduction#
Jenkins Configuration-as-Code (JCasC) has been covered in great detail already; this blog will focus on the running of both Jenkins Controllers (previously Masters) and Jenkins Agents in Kubernetes. The really exciting thing about this approach is that Jenkins Agents can be defined per-job and spun up when needed! Together with an auto-scaling Kubernetes cluster, this means we have an auto-scaling Jenkins deployment which will run at minimal cost while idling and offer performance when necessary.
Terraforming our Kubernetes Cluster#
In this blog, we will use Terraform to create our Kubernetes Cluster in Azure Cloud, but the approach will be the same for whichever IaC tool and Cloud Platform you choose.
We will skip over provider configuration, remote state etc. in this blog to keep things focused on the problem at hand. You can look at the complete example in the GitHub repository if you are interested in seeing the full configuration:
Github - Jenkins in Kubernetes example
1resource "azurerm_kubernetes_cluster" "jenkinsk8sexample" {
2 name = "aks-jenkinsk8sexample"
3 location = azurerm_resource_group.jenkinsk8sexample.location
4 resource_group_name = azurerm_resource_group.jenkinsk8sexample.name
5 dns_prefix = "aks-jenkinsk8sexample"
6
7 default_node_pool {
8 name = "default"
9 node_count = 1
10 vm_size = "Standard_B2s"
11 enable_auto_scaling = true
12 min_count = 1
13 max_count = 4
14 }
15
16 service_principal {
17 client_id = var.client_id
18 client_secret = var.client_secret
19 }
20}
As seen above, we’re creating a Kubernetes Cluster with a single auto-scaling nodepool.
Once we’ve run
1terraform apply
and the cluster is created, we can create the Jenkins namespace:
1kubectl create namespace jenkins
Our cluster is now ready to have Jenkins installed using Helm.
Installing our Jenkins Controller#
We will use Helm to install our Jenkins Controller in the Kubernetes Cluster. Helm is best thought of as a package manager for Kubernetes. First we need to add the Jenkins repo to our local Helm configuration:
1helm repo add jenkins https://charts.jenkins.io
2helm repo update
We can then run the install (upgrade) command:
1helm upgrade --install jenkins jenkins/jenkins \
2 --namespace jenkins \
3 --version 4.1.8 \
4 --set controller.adminUser="admin" \
5 --set controller.adminPassword=$JENKINS_ADMIN_PASSWORD \
6 -f custom_values.yaml
Note that our admin password is pre-configured in environment variable JENKINS_ADMIN_PASSWORD. Our custom_values.yaml file contains the JCasC values:
1controller:
2 installPlugins:
3 - kubernetes:3600.v144b_cd192ca_a_
4 - workflow-aggregator:581.v0c46fa_697ffd
5 - git:4.11.3
6 - configuration-as-code:1429.v09b_044a_c93de
7 - job-dsl:1.79
8 JCasC:
9 defaultConfig: true
10 configScripts:
11 welcome-message: |
12 jenkins:
13 systemMessage: Welcome to Jenkins, you are. Managed as code, this instance is.
14 example-job: |
15 jobs:
16 - script: >
17 multibranchPipelineJob('jenkins-in-kubernetes-example-pipeline') {
18 branchSources {
19 git {
20 id('jenkins-in-kubernetes-example-pipeline')
21 remote('https://github.com/verifa/jenkins-in-kubernetes-example-pipeline.git')
22 }
23 }
24 }
25 securityRealm: |-
26 local:
27 allowsSignup: false
28 enableCaptcha: false
29 users:
30 - id: "admin"
31 name: "Jenkins Admin"
32 password: "${chart-admin-password}"
33 authorizationStrategy: |-
34 loggedInUsersCanDoAnything:
35 allowAnonymousRead: false
Each Helm chart comes with a default set of values (values.yaml) which is also the set of supported values which can be overriden. This is what our custom_values.yaml file does, so for example we are overriding the list of plugins to be installed, adding the job-dsl plugin so that we can declare our pipeline as code.
Of most interest in the custom_values.yaml file is probably the example-job value, which lists the jobs to be created upon Jenkins Controller instantiation. As you can see, we are creating a Multi-branch Pipeline Job with Git sources in another repository.
Once you have executed the helm upgrade command, the Jenkins Controller Stateful Set should be created in your cluster, which will trigger the creation of the Jenkins Controller Pod, hosting your Jenkins Controller instance. Once Jenkins finishes starting up, it will scan the Git repo defined in the Multi-branch Pipeline Job and run a build on each discovered branch.
The Jenkins deployment above will be deployed with ClusterIP services only, meaning it is not accessible from outside the cluster. This is fine for testing purposes, and the simplest way to access Jenkins locally is to run a Kubernetes port-forward:
1kubectl port-forward svc/jenkins 8080:8080
This will forward traffic from 127.0.0.1:8080 (localhost) to svc/jenkins:8080 (Jenkins Controller ClusterIP), so while the kubectl port-forward command is running, you can navigate to http://localhost:8080 to access your Jenkins instance.
Configuring our Jenkins Agent#
Let’s inspect the master branch of the Git repo configured as the source of our Multi-branch Pipeline Job:
1pipeline {
2 agent {
3 kubernetes {
4 yaml '''
5 apiVersion: v1
6 kind: Pod
7 spec:
8 containers:
9 - name: maven
10 image: maven:alpine
11 command:
12 - cat
13 tty: true
14 - name: busybox
15 image: busybox
16 command:
17 - cat
18 tty: true
19 '''
20 }
21 }
22 stages {
23 stage('Run maven') {
24 steps {
25 container('maven') {
26 sh 'mvn -version'
27 }
28 container('busybox') {
29 sh '/bin/busybox'
30 }
31 }
32 }
33 }
34}
What you see in the pipeline block above are two blocks:
- agent
- stages
The agent block defines the agent which will execute the steps in the stages block. As you can see in the agent block, we are able to use YAML to declare the Kubernetes Pod which will execute the job. Note that we declare two separate containers, one running the maven:alpine image and one running the busybox image.
In the stages block you can see that we run one command in each container. This is an example of how you can break up your pipelines into small tasks which can run in specialized containers, instead of building bloated container images containing all the tools you need.
Auto-scaling#
Let’s take a look at Jenkinsfile in the large-pod branch of the pipeline Git repo:
1pipeline {
2 agent {
3 kubernetes {
4 yaml '''
5 apiVersion: v1
6 kind: Pod
7 spec:
8 containers:
9 - name: busybox
10 image: busybox
11 command:
12 - cat
13 tty: true
14 resources:
15 requests:
16 memory: "2Gi"
17 cpu: "1000m"
18 '''
19 }
20 }
21 stages {
22 stage('Run') {
23 steps {
24 container('busybox') {
25 sh '/bin/busybox'
26 }
27 }
28 }
29 }
30}
In the example above, a single busybox container is declared with large resource requests (2 GB memory and 1 CPU core). This exceeds the resources available on the single Node in the Kubernetes cluster initially, so we can see in the log that a second Node is created on which the Pod is then created and the Job scheduled:
1Started by user Jenkins Admin
2[Pipeline] Start of Pipeline
3[Pipeline] podTemplate
4[Pipeline] {
5[Pipeline] node
6Created Pod: kubernetes jenkins/test-2-v4rqd-2w5b2-nvfs2
7[Warning][jenkins/test-2-v4rqd-2w5b2-nvfs2][FailedScheduling] 0/1 nodes are available: 1 Insufficient memory.
8Still waiting to schedule task
9'test-2-v4rqd-2w5b2-nvfs2' is offline
10[Normal][jenkins/test-2-v4rqd-2w5b2-nvfs2][TriggeredScaleUp] pod triggered scale-up: [{aks-default-19934684-vmss 1->2 (max: 4)}]
11[Warning][jenkins/test-2-v4rqd-2w5b2-nvfs2][FailedScheduling] 0/1 nodes are available: 1 Insufficient memory.
12[Warning][jenkins/test-2-v4rqd-2w5b2-nvfs2][FailedScheduling] 0/1 nodes are available: 1 Insufficient memory.
13[Normal][jenkins/test-2-v4rqd-2w5b2-nvfs2][Scheduled] Successfully assigned jenkins/test-2-v4rqd-2w5b2-nvfs2 to aks-default-19934684-vmss000001
14[Normal][jenkins/test-2-v4rqd-2w5b2-nvfs2][Pulling] Pulling image "busybox"
15[Normal][jenkins/test-2-v4rqd-2w5b2-nvfs2][Pulled] Successfully pulled image "busybox" in 2.945562307s
16...
Once the job is complete, the Pod is destroyed. The AKS cluster auto-scaler will then, after a period of low pod activity, scale down the node pool back to the single node hosting the Jenkins Controller.
Going Further#
Security#
The example shown here is still far from production-ready, and there are many considerations to make, such as
- Persistent storage - what size and performance is required, and how should they be backed up?
- Networking security - what kind of Ingress/Egress rules should we take? What other restrictions should be put in place?
- Container security - we should (at minimum) prevent any container from running as root. How do we avoid potentially harmful images from being run?
Many of these considerations can be implemented using policies, perhaps your team or organization has a set of default cluster policies to use; if not, there’s no time like the present to create them.
Automation#
Although we have automated many steps in a typical Jenkins deployment, we are still triggering the deploy itself manually. The deployment commands themselves can be automated - such as in Continuous Delivery.
One commonly-used setup is that each commit to the main branch in your Git repository triggers a deployment. This is a great way to
- enforce good SCM practices (developers never commit directly to
main) - ensure “what you see in
mainis what is deployed in production”
Summary#
In this minimalist example, we’ve looked at how you can define both your infrastructure and Jenkins deployment as code, how to auto-scale both infrastructure and Jenkins agents and how Jenkins agents can be defined as part of your build pipelines to give complete freedom to developers on what kind of Jenkins agent their build pipeline needs. Hopefully this has given you the information you need to develop a solution like this on your own.
