Building multi-architecture container images on GitLab CI

Turns out that macOS on Apple Silicon is pretty popular right now. Especially among software engineers. From a ‘works on my machine’ perspective - one operating system among many developers - it greatly simplifies support. However Apple Silicon is not x86-64, but ARM64. Fortunately, many of the widely used images, especially published in public registries like Docker Hub, provide variants for different architectures.

Unfortunately, for internal images, we only had the amd64 version because those were built on the amd64 GitLab runners. As a result, performance for macOS users was… bad due to emulation.

While the easiest solution would be to add a new runner with an arm64 processor and build images on it, I wanted to avoid that. I did not want to spin up a new runner that would be used very rarely (a couple of times a year). I did not want to build them locally and push them to our registry either - that would be boring and ugly, and require somebody with a dedicated device to build them. And allowing that person to push images directly to the registry is a no-go from a security perspective as this would bypass the proper code review and CI process.

So I searched for a solution using emulation.

And I found one! Using the cross-platform emulator collection distributed using Docker images available at tonistiigi/binfmt. It registers QEMU emulators with the host kernel through binfmt_misc.

So, below are the steps needed to run building multi-platform container images in GitLab CI (GitLab Runner with the Docker executor) with Buildah¹.

1. Install the required emulation support

To install emulation that will be also available to containers, I needed to run (to install only the emulation support for ARM64²):

docker run --privileged --rm tonistiigi/binfmt --install arm64

After that, I verified if everything is working using the following command:

docker run --rm --platform linux/arm64 alpine uname -m

It should print the following:

aarch64

Note that registration is not persisted between reboots. The Docker image does not install any host service to reapply registration on boot.³

2. Update GitLab Runner configuration

The next step is to update the configuration of the GitLab Runner using the Docker executor and relax some of its security restrictions.

Since this weakens container isolation, do it at your own risk. At minimum, make sure this is a controlled runner that executes only trusted jobs.

[[runners]]
# ...other configuration ommited...
  [runners.docker]
    # To allow running buildah in the container
    security_opt = ["seccomp:unconfined", "apparmor:unconfined"]

Is it 100% secure? Nope.

However, for our self-hosted runner it was an acceptable trade-off compared to building images manually on a developer’s machine.

Spinning up a dedicated ARM64 runner would provide native builds (which would be faster). Yet considering that it would be used at most a couple of times a year, it would also add operational overhead and increase infrastructure costs.

3. Configure the image step using Buildah

Last but not least, the a GitLab CI job. As mentioned at the beginning I used Buildah and its support for building images for different CPU architectures⁴.

I store the images in GitLab Registry, so the first step (in before_script) is to log in to the registry. Then the job builds the image for specified CPU platforms and pushes manifest with platform-specific images to the registry.

images:multiplatform:
  stage: images
  image: quay.io/buildah/stable
  variables:
    IMAGE_DIRECTORY: "myimage"
    IMAGE_NAME: "corp/myimage"
    IMAGE_VERSION: "1.2.3"
    STORAGE_DRIVER: vfs
    BUILDAH_FORMAT: docker
    BUILDAH_ISOLATION: chroot
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
  script:
    - buildah build
      --platform=linux/amd64,linux/arm64
      --manifest "$CI_REGISTRY_IMAGE/${IMAGE_NAME}:${IMAGE_VERSION}" 
      "${IMAGE_DIRECTORY}"
    - buildah manifest push --all "$CI_REGISTRY_IMAGE/${IMAGE_NAME}:${IMAGE_VERSION}" "docker://$CI_REGISTRY_IMAGE/${IMAGE_NAME}:${IMAGE_VERSION}"

You may ask: why so many variables? To make the job easy to reuse. All I need to do to add a new image is to create a new directory with Dockerfile or Containerfile and create a new entry in .gitlab-ci.yml.

That is all - now all developers can benefit from prebuilt images that have native performance on Apple Silicon. :-)