In a previous post we showed how to build and store Docker images in GitLab. Building on that, we will demonstrate a workflow to Dockerise a custom tool which first needs to be compiled while minimising time to deployment by storing artifacts in the Package Registry.
The repo for this post is here, if you wish to jump straight to the final results.
Introduction#
We often need custom tools in our pipelines. Whether they are modified versions of larger open-source projects or tools developed entirely in-house, simple packagers or complex compilers, these custom tools need to be built and made available to our CI pipelines. Deploying these tools as images leverages the ease of deployment and reliability of Docker.
For this demonstration, we will build and deploy a compilation-like tool, the Godot engine. The Godot engine is a game development editor and compiler tool that is growing in popularity. One of the key reasons for this is that it is free and open-source. This means that, while we could use it as provided, we are also able to create customised variants of it in order to better suit our projects.
We will compile and deploy the Godot engine in a way which allows us to automatically run tests on a project created in Godot. We will also minimise the time it takes to deploy the image, which is valuable when improving or fixing the tool.
Compiling the Tool#
In order to compile the Godot engine we’ll need a slightly more elaborate than usual compilation environment. Therefore, first we will create a Docker image. The headless server artifact is compiled on a Linux environment, which is convenient for our purposes.
The compilation environment image itself is fairly straightforward; we install scons, which is needed to compile Godot, and the libraries needed to compile for the target platform.
1#Linux
2FROM ubuntu:20.04
3LABEL Author="Zach Laster <zlaster@verifa.io"
4
5RUN apt-get update && apt-get install -y --no-install-recommends \
6 ca-certificates \
7 build-essential \
8 scons \
9 pkg-config \
10 libx11-dev \
11 libxcursor-dev \
12 libxinerama-dev \
13 libgl1-mesa-dev \
14 libglu-dev \
15 libasound2-dev \
16 libpulse-dev \
17 libfreetype6-dev \
18 libudev-dev \
19 libxi-dev \
20 libxrandr-dev \
21 git \
22 && rm -rf /var/lib/apt/lists/*
23
24# Specify the python version in the scons script file
25RUN sed -i "/\#\! \/usr\/bin\/python/c\\#\! \/usr\/bin\/python3" $(which scons)
We’ve already gone over how to build this Docker image and store it in a GitLab registry, so we’ll skip over that part here.
Now that we have our environment image, we can compile Godot! For our CI file we’ll utilise a few techniques which will make the CI jobs easier to work with.
First, we’ll specify a workflow block, which allows us to specify high level rules for when any jobs are run.
Second, we’ll move most of the logic for the artifact job into a hidden job. Hidden jobs are denoted by a prefix of ., to let GitLab know that this isn’t a job to execute. Instead, they are very useful for definitions blocks. We can use GitLab’s extends keyword to base our actual job on hidden jobs, thereby reusing definitions and separating concerns.
The hidden job specifies our stage, where the artifacts will be found and how long to keep them, and two extra script steps. This separation already helps to make responsibilities more clear, as we can see at a glance what is common, boilerplate logic and what is the specific instruction and configuration for building. This will be a significant benefit when we have more artifacts.
As to the actual compilation, our instruction is a single line which we run on the image we built previously. We’ll use before_script to clone the official Godot repository from GitHub, and after_script to move the artifacts to a bin folder in the root of the workspace.
1workflow:
2 rules:
3 - if: $CI_MERGE_REQUEST_ID
4 when: never
5 - when: always
6
7stages:
8 - artifacts
9
10variables:
11 GODOT_REPO_URL: https://github.com/godotengine/godot.git
12 GODOT_REPO_BRANCH: 3.x
13
14.artifact:
15 stage: artifacts
16 artifacts:
17 expire_in: 1 day
18 paths:
19 - bin/
20 before_script:
21 - git clone --depth 1 --branch "$GODOT_REPO_BRANCH" "$GODOT_REPO_URL"
22 after_script:
23 - if [ -d godot/bin/ ]; then mkdir bin && mv godot/bin/* bin/; fi #Relocate artifacts folder to root
24
25headless:
26 extends: .artifact
27 image: $CI_REGISTRY_IMAGE/environments/linux:latest
28 script:
29 - scons --directory=godot profile=../editor.py platform=server tools=yes target=release_debug
Pushing this file triggers the pipeline which will build our headless artifact.
Deploying the Image#
Now that our artifact can be built, it’s time to put it into a Docker image for external use.
The following GitLab CI configuration adds a headless-image job that depends on the headless build job and artifact. Again, we use hidden jobs to simplify the structure and move the boilerplate elsewhere. This can be very useful in extending the pipeline.
1.dind:
2 image: docker:23.0.1
3 services:
4 - docker:23.0.1-dind
5
6.image:
7 extends: .dind
8 stage: images
9 variables:
10 DOCKER_IMAGE_BUILD_TAG: $CI_REGISTRY_IMAGE/$CI_JOB_NAME_SLUG:$CI_COMMIT_REF_SLUG
11 DOCKER_FILE: Dockerfile
12 script:
13 - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
14 - docker build --pull -t $DOCKER_IMAGE_BUILD_TAG -f $DOCKER_FILE .
15 - docker push $DOCKER_IMAGE_BUILD_TAG
16
17headless-image:
18 extends: .image
19 variables:
20 DOCKER_IMAGE_BUILD_TAG: $CI_REGISTRY_IMAGE/headless:$CI_COMMIT_REF_SLUG
21 DOCKER_FILE: headless.dockerfile
22 needs:
23 - job: headless
24 artifacts: true
The image is created using the specification in headless.dockerfile, which is very simple: We use the same version of Ubuntu as before, ensure we have python and git, and copy in our artifact to the user binary folder.
1FROM ubuntu:20.04
2LABEL Author="Zach Laster <zlaster@verifa.io>"
3
4RUN apt-get update && apt-get install -y --no-install-recommends \
5 ca-certificates \
6 git \
7 python \
8 python-openssl \
9 && rm -rf /var/lib/apt/lists/*
10
11COPY bin/godot_server.x11.opt.tools.64 /usr/local/bin/godot
12
13RUN mkdir ~/.cache && mkdir -p ~/.config/godot
This should work, but the compilation of the artifact takes much too long to be practical here. We need to improve on this.
Improving Godot Compilation Time#
While this is a complex and advanced subject, and how to improve the compilation time of an artifact tends to depend a lot on the artifact, the usual answers are to either reduce what we are building or to cache something.
Godot provides numerous flags for controlling the build output and process. These will affect both the artifact size and build time. There is a file in the repo which specifies what we want to build.
More impactfully, we can make use of GitLab caching to significantly improve build times after the first time. Since scons can cache its work to speed up compilation on subsequent attempts, we can inform it to do that and have GitLab track that cache.
1.scons_cache:
2 variables: &scons_cache_variables
3 SCONS_CACHE: ../.scons-cache/ #Up a directory since we are working in ./godot/
4 # Limit to 7 GiB to avoid having the extracted cache fill the disk.
5 SCONS_CACHE_LIMIT: 7168
6 cache:
7 - key: "$CI_COMMIT_REF_SLUG-scons-$CI_JOB_NAME_SLUG"
8 paths:
9 - .scons-cache/
10 policy: pull-push
We’ll use a mixture of YAML anchors and !reference for the scons_cache hidden block. Using !reference gives us more control and makes it easier to merge with other logic, but it doesn’t work well with variables blocks if we have more than one imported set of variables.
Adding these lines to .artifact adds the cache.
1variables:
2 <<: *scons_cache_variables
3 cache:
4 - !reference [.scons_cache, cache]
scons caching drastically reduces our compilation times, even when some of the code changes. However, the linking portion can still take ten minutes or more. A long time to wait if we don’t actually need to do anything.
Reusing Artifacts#
Rather than compile Godot every time we run the job, we could first check if we have already compiled the current version. The easiest check for this is if we already have a compiled artifact of the current git commit SHA which we are about to clone.
To accomplish this, we need to store our compiled artifacts in a way than enables us to retrieve them. We could utilise GitLab job artifacts, but there’s some limitations with retrieving specific versions and the API is not streamlined for this use case.
Instead, we can utilise another GitLab feature, the Package Registry.
To use the GitLab package registry, the only tool we’ll need is wget. Pushing to and retrieving from the registry is handled via a REST API.
wget --header="JOB-TOKEN:$CI_JOB_TOKEN" -O ${ARTIFACT_PATH}${ARTIFACT_FILE} $ARTIFACT_URL
The above line will use our current job token to download an artifact file from a given URL.
ARTIFACT_URL=${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/$GENERIC_PACKAGE_NAME/$SRC_REMOTE_SHA/$ARTIFACT_FILE
The artifact URL is fairly predefined; we only decide the last three values. The structure of <package name>/<version>/<filename> is fixed, but we can use almost anything for them. For our case, we’ll use the job name slug as the package name, the Godot repo SHA we are compiling as the version, and the name of the artifact as the actual file name.
Uploading works the same way as retrieval: wget --header="JOB-TOKEN:$CI_JOB_TOKEN" --method=PUT --body-file="${ARTIFACT_PATH}$ARTIFACT_FILE" "$ARTIFACT_URL". With very little around these we can push and pull on the registry.
1.artifact_package:
2 variables: &artifact_package_variables
3 ARTIFACT_FILE: example_file
4 ARTIFACT_PATH: bin/
5 GENERIC_PACKAGE_NAME: $CI_JOB_NAME_SLUG
6 download: #Bash commands we will execute to download the artifact based on variables
7 - if [ -z $SRC_REMOTE_SHA ]; then SRC_REMOTE_SHA=`git ls-remote $GODOT_REPO_URL $GODOT_REPO_BRANCH | head -1 | sed "s/\t.*//" | tee godot.sha`; fi; echo "Compiling Godot repo:"" $SRC_REMOTE_SHA"
8 - ARTIFACT_URL=${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/$GENERIC_PACKAGE_NAME/$SRC_REMOTE_SHA/$ARTIFACT_FILE; echo $ARTIFACT_URL
9 - (wget --header="JOB-TOKEN:$CI_JOB_TOKEN" -q --spider $ARTIFACT_URL && mkdir -p ${ARTIFACT_PATH} && wget --header="JOB-TOKEN:$CI_JOB_TOKEN" -O ${ARTIFACT_PATH}${ARTIFACT_FILE} $ARTIFACT_URL) || true
10 check: #Bash command to check if the artifact exists and exit if so
11 - if [ -f ${ARTIFACT_PATH}$ARTIFACT_FILE ]; then echo Retrieved":" $ARTIFACT_FILE from package registry; exit 0; fi
12 upload: #Upload the artifact
13 - ls "${ARTIFACT_PATH}$ARTIFACT_FILE"
14 - SRC_REMOTE_SHA=`cat godot.sha`
15 - ARTIFACT_URL=${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/$GENERIC_PACKAGE_NAME/$SRC_REMOTE_SHA/$ARTIFACT_FILE; echo $ARTIFACT_URL
16 - wget --header="JOB-TOKEN:$CI_JOB_TOKEN" --method=PUT --body-file="${ARTIFACT_PATH}$ARTIFACT_FILE" "$ARTIFACT_URL"
With this last piece, our .artifact block looks like this
1.artifact:
2 stage: artifact
3 tags:
4 - offload
5 artifacts:
6 expire_in: 1 day
7 paths:
8 - bin/
9 variables:
10 <<: *scons_cache_variables
11 <<: *artifact_package_variables
12 cache:
13 - !reference [.scons_cache, cache]
14 before_script:
15 - !reference [.artifact_package, download] # We import the bash commands
16 - !reference [.artifact_package, check]
17 - git clone --depth 1 --branch "$GODOT_REPO_BRANCH" "$GODOT_REPO_URL"
18 after_script:
19 - if [ ! -d godot/bin/ ]; then exit 0; fi
20 - mkdir -p bin && mv godot/bin/* bin/ #Relocate artifacts folder to root
21 - !reference [.artifact_package, upload]
Now when we next run the compilation job, it will first check if we’ve already pushed a compiled artifact to the registry and, if so, use that. If there isn’t a package for the current SHA, then the job runs as before and then pushes the resulting artifact to the registry. This cuts our compilation job time down from 15 minutes to 50 seconds 🎉 in the case of having previously compiled the artifact for this version of Godot.
Summary#
We now have a Godot image which we can use to run tests on our Godot projects. We’ve verified that the image works and the contained Godot executable runs and knows its name and version.
This process and workflow is applicable to any number of tools we might wish to compile and deploy using docker images. The ability to run containers containing custom-built tooling can have a massive impact on our CI jobs.
In future blog posts, we will explore how to handle compiling for multiple architectures, as well as handling images which contain multiple artifacts.
