How we streamlined CI pipelines with Gitlab components

At Bonito, we initially had each project’s gitlab-ci.yml files copy each other for deployment configurations as it was one of the fastest and easiest solutions to implement. We had no problem with this setup at first — that is, until we needed to make major changes to deployments or builds, like when we moved platforms or needed to update authentication changes across the board. This, in addition to the fact that this implementation required someone to spend some time writing (or copy and pasting) all the files needed, started to flag to us the need to explore other solutions.

During a recent review of our CI pipelines, we found that GitLab introduced CI/CD components, which looked like a good solution to templating a large part of those deployment files. These components are like modular units or parts of configuration that we can import and reuse for projects in their gitlab-ci.yml files, which can then be given new parameters as inputs to replace default variables or behavior.

One problem we had with not having a project as a source of truth for our deployment files was that we couldn’t be sure we’re always adding the new lines verbatim. This sometimes meant that if you check our gitlab-ci.yml files, you’ll see similar (but not exact copy) lines.

Spot the difference:

script:
  - docker build . 
    --tag asia-docker.pkg.dev/data:$CI_COMMIT_SHA
  - docker build . 
    --tag asia-docker.pkg.dev/data:$CI_COMMIT_REF_NAME
  - docker push asia-docker.pkg.dev/data:$CI_COMMIT_SHA
  - docker push asia-docker.pkg.dev/data:$CI_COMMIT_REF_NAME

^{project0/.gitlab-ci.yml}

script:
  - docker build . -t asia-docker.pkg.dev/app:$CI_COMMIT_SHA
  - docker build . -t asia-docker.pkg.dev/app:$CI_COMMIT_REF_NAME
  - docker push asia-docker.pkg.dev/app:$CI_COMMIT_SHA
  - docker push asia-docker.pkg.dev/app:$CI_COMMIT_REF_NAME

_{project1/.gitlab-ci.yml}

That one is actually a real example of when Google deprecated their Container Registry and we had to transition from it to their new Artifact Registry. We hadn’t set up any templates using components yet at the time so we had to change all of our CI files manually, looking for instances docker build -t and docker push in every project and then updating each domain name. The little difference here of setting the tag for Docker was just a minor issue and was easy to do, but it took a longer time than it should have because it was a manual endeavour. There have also been times where pipelines failed because we forgot to change a CI variable in the manifest, or because we missed a command for git-crypt in the script section when a secret was added in for the first time in a project.

Now, here’s an example that fixes those problems. This is a component named build.yml because it’s for our build stages, but you could also just name it anything. The template here is for getting Docker to build and push a project’s images to Artifact Registry:

spec:
  inputs:
    as:
      default: build
    stage:
      default: build

    image_registry:
      default: asia-docker.pkg.dev
    stage_image:
      default: docker:latest
    image_name:
      default: $CI_PROJECT_NAME
    image_tag:
      default: $CI_COMMIT_REF_NAME

    changes:
	    type: array
      description: 'An array of files that when modified should trigger a build'

---

'$[[ inputs.as ]]':
  stage: $[[ inputs.stage ]]
  image: $[[ inputs.image_registry ]]/$[[ inputs.stage_image ]]
  variables:
	  GIT_STRATEGY: clone
  script:
	  - git crypt unlock
    - docker build
      -t $[[ inputs.image_registry ]]/$[[ inputs.image_name ]]:$CI_COMMIT_SHA
      -t $[[ inputs.image_registry ]]/$[[ inputs.image_name ]]:$[[ inputs.image_tag ]]
    - docker push
      -t $[[ inputs.image_registry ]]/$[[ inputs.image_name ]]:$CI_COMMIT_SHA
      -t $[[ inputs.image_registry ]]/$[[ inputs.image_name ]]:$[[ inputs.image_tag ]]
  rules:
	  changes: $[[ inputs.changes ]]

^{componentsproject/templates/build.yml}

The spec first half of build.yml where we wrote inputs get referenced later on in the next part through $[[ inputs.<keyword> ]]. To use the component, I created a dedicated component project, and put build.yml in a directory named templates because GitLab requires you to have the project’s file structure like this:

componentsproject/
	README.md
	templates/
		build.yml
		deploy.yml

You’ll notice that all templates must be in the templates directory. They are imported into other projects without the directory name in those project’s gitlab-ci.yml like this:

stages:
  - build

include:
  - component: gitlab.com/bonitotech/componentsproject/build@~latest
    inputs:
	    changes:
		    - Dockerfile
		    - main.go

You’ll notice that all templates must be in the templates directory. They are imported into other projects without the directory name in those project’s gitlab-ci.yml like this:

stages:
  - build

include:
  - component: gitlab.com/bonitotech/componentsproject/build@~latest
    inputs:
	    changes:
		    - Dockerfile
		    - main.go

^{project2/.gitlab-ci.yml}

You’ll see that we didn’t write in anything else for the new project’s gitlab-ci.yml except for changes and stages. Any input where we set a default value won’t need interaction when the component is imported using include, if you’re okay with the default. If you look at the example component, you’ll also notice that we omitted a default value for changes because we want this to be provided explicitly when writing the pipeline manifest. We also set the type explicitly for changes because normally any input without its type set is treated automatically as a string. Additionally we also included a description, which just helps everyone else reading the file know what the input is.

It’s also recommended we have a table that describes each input in the component README so that it’s easier for someone else to use:

By utilizing GitLab CI/CD components, we at Bonito Tech were able to achieve significant improvements in development workflow efficiency and consistency. This approach not only streamlined manual efforts and reduced the risk of errors, but it also ensured our CI pipelines remained up-to-date and adaptable to future changes.

Need expert digital solutions? Bonito Tech can help! We offer tailored services to help you succeed in the digital landscape, creating thoughtful solutions done with integrity. Contact us today to learn more!