GitOps Pipeline

Planning

From the very start we wanted to build an industry standard pipeline capable of automating all the necessary tasks related to CI/CD.

Following good GitOps practices, our vision was that repositories should have at least three branches; production, staging and features. Production and staging branches are protected which means that the only way to commit to a branch is via merge request. Developers create a new branch for every feature which they can use to develop locally. When they are finished with the feature, they will create a merge request which the lead developer will then accept into the staging branch. After the staging branch has all the needed features, the lead developer will merge it into the production branch.

At every merge, the pipeline will run automated tests. Docker is used for simple unit tests in feature branches before merge and when merged, Testkube running inside Kubernetes will run integration and site acceptance tests. Depending on the criticality of the tests, a failed test can block the merge. Staging and production branches will have their own environments and will be deployed on separate MK8s and K8s clusters.

We opted to use a multi-repository approach for the application management. GitLab should have at least two repositories; one for applications themselves and another one for the configurations. This is to ensure correct deployment into the clusters. Developers have access only to the development repository and the pipeline automatically adjusts the manifests inside the configuration repository as needed. The Mysticons’ infrastructure team has access to all repositories.

Using commit hashes as container tags we can easily implement a rollback system with Argo CD. If the newest container doesn’t start properly, Argo CD can then automatically pull the previous container from the GitLab.

Starting the work

First we tried to use the GitLab pipeline with Helm to deploy applications to Kubernetes. Helm manifests were stored in the GitLab repository and runners would push the configuration defined by the manifests to the cluster. GitLab has easy-to-install agents to help integrate Kubernetes into Gitlab so this solution was nice to get familiar with deployment methods.

Getting the hang of Helm manifests took some time. After that we managed to create a CI/CD pipeline that started every time a change was committed to the repository. Using the K8s GitLab agent installed inside the cluster, the pipeline was then able to deploy necessary configurations to the cluster. Then the agent would pull the image from the GitLab registry and deploy it. Soon we discovered that this solution was not viable for actual production use; it was hard to automate with multiple teams in regards to security and isolation. In addition, we didn't want to use a push-based deployment method for the configuration.

To streamline our work, we started researching different CI/CD-tools:

• Flux CD
• Argo CD
• Jenkins

Flux CD seemed like a good choice since GitLab itself promotes it. Before we managed to implement it in any way, we had a meeting with a DevOps architect from NorthCode. He told us they were using Argo CD in their workflows and due to this we decided to primarily focus on Argo CD.

Deploying Argo CD into the cluster was very straightforward in MicroK8s as it was available as a community addon. Argo CD provided a very good user interface which we could use to deploy containers from GitLab repositories. We set up a repository which contained configuration manifests for applications that the developers wanted to deploy. Argo CD then could pull the docker containers from the respective project repository registry.

We had issues with multiple projects using the same ip address. Most applications didn’t work correctly if they were not situated in the root url. To solve this we requested a wildcard domain name for our project. We managed to deploy frontend-applications from two teams to our cluster, but connecting them to the backend-services did not work yet.

Customizing automation

At this point Argo CD still required manual action for the pull action to actually happen. The reason for this was that all the containers used the same latest tag and thus Argo CD thought that nothing changed even if the container was different. To solve this, we needed to configure the pipeline in such a way that it would generate unique ids for the containers and then update the files in the configuration repository automatically.

We modified the pipeline to use the GitLab repository files API. This allowed us to connect the pipeline in the application repository to the configuration repository. Figuring out the syntax for the Sed stream editor took us a great deal of time, but now we finally had a pipeline from start to finish.

To further control applications from different teams in our cluster we needed more manifests. We created new configuration files which included namespaces and quota limits for the teams. Deploying the configuration happened automatically via Argo CD thanks to the pipeline we had built.

Pipeline for infrastructure

We designed our cluster to consist of approximately 40 nodes in multiple high-availability clusters. Trying to provision and configure them manually was out of the question from the very beginning. Following the examples set by the previous Mysticons team we started to steer towards Terraform and Ansible.

We spent a lot of time trying to figure out how to use Terraform and Ansible the right way. Both seemed to be able to do almost everything the other one was doing. After some research we made initial plans to provision virtual machines with Terraform and then configure them with Ansible.

It took some time to get used to Terraform's syntax, particularly when attempting more complex tasks like executing commands on both master and worker nodes simultaneously. Ansible appeared better suited for these types of tasks, but we didn’t manage to integrate it with the dynamic inventory created by Terraform. No further work was done on this part due to time constraints.