Software testing

Right from the very start of WIMMA Lab there were people interested in testing. So to encourage teamwork and communication between teams, the tester tribe was formed. The purpose of the tribe was to allow those interested in testing to come together to learn and test.

At the beginning, the tools chosen for testing were Robot Framework and SquashTM. Most of our time was spent on learning the chosen tools and the basics of testing. Learning process included trying out various libraries inside Robot Framework and watching recordings of previous guests.

After the open doors event it was decided that the SquashTM would be abandoned. We found out it was a hassle to do anything with it. We also got a hint from the company NorthCode that Testkube could be something we could take a look at. Thus it was decided that Testkube would be the new testing platform.

Installing Testkube

The tester tribe got to work setting up Testkube. We found official instructions about the installation and followed those. In said instructions, the Kubernetes they were using was Minikube, so we chose the same.

The first problem we encountered was with Minikube drivers. We didn’t know this at the time, but Minikube can be deployed as a VM, a container, or on host, depending on the chosen driver. We tried using the virtualbox drivers, but we didn’t get it to work as there were problems with virtualization. The error suggested that we should use Docker if we were running Minikube on a VM. We then installed Docker and changed driver settings accordingly, and we got Minikube working.

Next step was to install and initialize Testkube. The installation succeeded without problems, but the same couldn’t be said about the initialization. Whenever we tried, it would give us an error about missing api-server or endpoint, context is not set or missing files. There were times when we thought that “This is it”, but that just ended up in a new error.

Eventually we made the decision to change to Microk8s, because we couldn’t get Testkube to work on Minikube. This resulted in the same errors as with Minikube, but after some time we managed to get both the dashboard and the API server to work. We did this by opening the required ports from the CSC, and port forwarding both the dashboard and the API server. We also inserted the correct API endpoint to the dashboard. The same solution would likely work on Minikube as well, but we were too excited to go back.

Now we had a working Testkube ready for testing. Timing of this was unfortunate, however, as the set up took too much time after the change from SquashTM. Other teams were using their own testing tools, for example Robot Framework, and making manual reports by using a template. If we had had more time, we would have provided Testkube as a primary tool for testing and we could have configured it to its full potential.

Testing

Now that the testing environment was working, it was time to do some testing. When first opening Testkube, it proved to be surprisingly intuitive. This was especially nice as we had little if any experience with testing.

Creation of a new test was quite simple: test name, what type of test is it and where the test comes from. Depending on the test source there could be more options, for example if it’s from string, you just type the test. But if it’s from Git, you need a Git token, username etc. You can also add labels to your tests to easily differentiate them.

Now we knew how to make and run tests, but what if we wanted to run multiple tests. This is where test suites come in. Test suites are basically a collection of tests. You make a test suite and choose what pre-existing tests you would like it to run.

Now that we could make and run tests manually, it was time for automation. In Testkube, test automation would be handled by scheduling and triggers. Scheduling means that a test or test suite can be automated to run on certain intervals, like once a month or every ten minutes. It is quite handy for tests that need to be repeated on certain intervals. Triggers differ from scheduling by the fact that they don’t have a set time, but are run when a “trigger” is activated.

In the picture above there is a working trigger. The ingress refers to the ingress.yaml file in Kubernetes that the Tetris game uses. When a modification is made to that file, trigger activates and runs a test called tetris, which is a test to curl the URL the game uses. Also, the labels mentioned before come in handy here, since they can be used to identify cluster resources. The example ingress trigger is Name to Name, meaning namespace/resource-name.

Integration with Argo CD

Now we were at a stage where it was time to integrate Testkube with GitLab and Argo CD. GitLab would host the tests for Testkube to use when something activates an automated trigger in Kubernetes. Argo CD would monitor and keep Kubernetes resources up to date according to GitLab configurations. Since our team was already somewhat familiar with the Argo CD, this was straightforward to implement. We did manage to get the GitLab, Testkube and Argo CD integrations to work, and we had an integration triangle.

What was missing was feedback from Testkube to Argo CD and GitLab. Test results didn’t affect deployment at all no matter how critical the errors were. We also couldn’t get the reports to show on GitLab. It should be also considered whether Argo CD sync hooks should be used instead of Testkube triggers.. However, this integration part happened when WiMMA Lab was coming to an end, so there wasn’t enough time to do all the things we would have liked.