Autoscaling your GitLab runners on Fuga has many advantages. For example, you can only have runners active during office hours or just when you need them. The unused runners will be removed automatically. Since Fuga Cloud is pay for what you use, the total costs for your GitLab CI/CD will go down. In this tutorial, I will explain how to add autoscaling GitLab runners to your GitLab installation using GitLab CI and Docker Machine.
When a commit is made, a CI/CD pipeline will be started. GitLab will notify the runner-manager that a new job is available. The runner-manager will pass down this job to an available runner, if there are none available, the runner-manager will create a new instance with a runner. This pipeline can be customized to your liking, for example how many idle runners are available or the number of jobs each runner can handle concurrently.
The flow looks like this:
- GitLab installed on Fuga. See our other GitLab tutorial
- SSH keys added to your GitLab admin account
Setting up the security groups
In order to autoscale your runners, Docker Machine requires port 2376 to be opened for every new runner that gets automatically deployed. In this section is explained how you can open this port with a security group.
To open this port, go to Networking and click on the Security Groups tab.
Click on + Create Security Group to create a new security group.
Enter a fitting name for this Security Group, for example, gitlabrunner. You can enter a description for clarification if you’d like to.
Click on Create Security Group to create the new security group.
Once the new security group has been created, click on the Manage Rules button that belongs to the new security group you just made.
In the top right corner, click on + Add Rule to add a new security rule.
In the Port field enter 2376 and keep the rest of the form at the default values. It should look like this:
Click on Add to add the security rule.
Creating new SSH key pair for GitLab
Between the runner-manager and the runner instances, an SSH connection is used by Docker-Machine. Instead of using your personal private SSH keys, a new key is generated and used specifically for the runners.
On your Fuga dashboard go to Compute and click on the Key Pairs tab.
Click on + Create Key Pair.
Choose a fitting name for the Key Pair, this tutorial uses name gitlab.
Save this file to your machine, it will be needed later for the SSH connections between the runners.
Installing and registering the GitLab runner
In this section the installation and the registration of the GitLab runner will be done. This runner, called the runner-manager, will be used to connect to your GitLab and manages the runners with Docker Machine. These runners will receive the jobs from the runner-manager which received the jobs from GitLab.
Launch a new instance on Fuga, in this tutorial the instance is called runner-manager. The flavor c2.small is chosen with the image Ubuntu 18.04 LTS.
Log in to your runner-manager instance using the following example command in your terminal:
For other examples of SSH follow the Getting Started guide.
Start by updating your repositories by using the command:
$ sudo apt update
Install Docker with the following command:
$ sudo apt-get install \
Add Docker’s official GPG key:
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
Verify that you now have the key with the fingerprint 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88, by searching for the last 8 characters of the fingerprint.
$ sudo apt-key fingerprint 0EBFCD88
Set up the stable repository:
$ sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
Update your repositories:
$ sudo apt-get update
Install the latest version of Docker CE and containerd
$ sudo apt-get install docker-ce docker-ce-cli containerd.io
Add the GitLab Runner repository:
$ curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
Now install the GitLab runner package:
$ sudo apt install gitlab-runner
Install Docker-Machine with the following command:
curl -L $base/docker-machine-$(uname -s)-$(uname -m) >/tmp/docker-machine &&
sudo install /tmp/docker-machine /usr/local/bin/docker-machine
Log in to your GitLab with an admin account (the default admin account is root).
Click on the wrench icon Admin Area located in the top bar of GitLab.
Go to Overview -> Runners in the left menu bar. Keep this page open because the URL and registration token is needed in the following steps.
Enter the following command in your GitLab runner instance to start the registration:
sudo gitlab-runner register
When asked for your GitLab URL, enter the URL of your GitLab. For example http://gitlab.example.comor http://185.xx.xx.xx.
When asked for a GitLab-ci token, enter the token on the page of step 9.
When asked for a description of your GitLab runner, enter a fitting description.
When asked for tags for your GitLab runner, enter tags that you deem fitting.
When asked to lock your GitLab runner to the project, enter false.
When asked for an Executor for your GitLab runner enter docker+machine.
When asked for a default Docker image, enter nginx. You can choose anything you like but for this tutorial the nginx image is used.
Setting up SSH between the runner-manager and the runners
Once the GitLab runner is installed, some additional configuration is necessary. By default our configuration will look for a gitlab.pem file in the home/ubuntu/.ssh folder. You need to transfer your new .pem file to the runner-manager instance, so it can be used for the SSH connections between the runners.
Show the contents from the pem file you downloaded from the section Creating new SSH key pair for GitLab with the following command:
Copy the entire contents to your clipboard.
In your .ssh folder on your runner-manager instance, create a new gitlab.pem file:
Open the newly created gitlab.pem file with your favorite text editor and paste in the contents from your clipboard.
Save the file.
Configuring the runner-manager
Once everything is installed, the last changes to the runner-manager instance have to be fulfilled.
Open the configuration file /etc/gitlab-runner/config.toml in your runner-manager instance with your favorite text editor, for example:
sudo nano /etc/gitlab-runner/config.toml
Copy the name, url and token under the [[runners]] and save them temporarily on your machine.
Copy and paste the following code:
concurrent = 8
check_interval = 0
name = "NAME_OF_YOUR_MANAGER_INSTANCE"
limit = 4
url = "YOUR_GITLAB_URL_OR_IP_WITH_HTTP(S)"
token = "YOUR_GITLAB_RUNNER_MANAGER_TOKEN"
executor = "docker+machine"
tls_verify = false
image = "nginx"
privileged = true
disable_cache = true
volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
shm_size = 0
IdleCount = 1
IdleTime = 1800
MaxBuilds = 100
MachineDriver = "openstack"
MachineName = "gitlab-ci-as-%s"
MachineOptions = [
"openstack-image-name=Ubuntu 18.04 LTS",
OffPeakPeriods = ["* * 0-7,19-23 * * mon-fri *", "* * * * * sat,sun *"]
OffPeakTimezone = "Europe/Amsterdam"
OffPeakIdleCount = 0
OffPeakIdleTime = 1200
Change the NAME_OF_YOUR_MANAGER_INSTANCE to the name you saved in step 2.
Change the YOUR_GITLAB_URL_OR_IP_WITH_HTTP(S) to the url you saved in step 2.
Change the YOUR_GITLAB_RUNNER_MANAGER_TOKENto the token you saved in step 2.
Under MachineOptions, change openstack-username to your Fuga email address.
Under MachineOptions, change openstack-password to your Fuga password.
Under MachineOptions, change openstack-tenant-name to your Fuga username.
Under MachineOptions, change openstack-net-name to your Fuga Network name.
Save the file.
This config will by default always have a runner active during office hours. You can change the settings to anything you want under the OffPeakPeriods. For more information about the settings in this file, see the official GitLab documentation under Advanced configuration.
Testing the configuration
Your GitLab configuration should be ready to be used. To test all the changes you can create a test project and see if the jobs will successfully run.
To create a new project on your GitLab, click the plus icon located on the top of the screen next to the search bar and click New project.
Give it a name and click on Create project.
Open the new project and go to Settings -> CI/CD, which is located in the left menu bar.
Expand the Secret variables menu and enter your access key as ACCESS_TOKEN and your secret key as SECRET_TOKEN. Also enter your project ID as PROJECT_ID. You can find your project ID in Account -> Details. There you’ll find a list with all your projects. The values for these keys should be the tokens from the Fuga Object Store tokens section. It should look like the picture below:
Click on Save variables.
Clone your new project to your system using the git link located on the project page:
git clone firstname.lastname@example.org:root/your-project.git
Open your new git folder inside your terminal:
Create a new file that will contain our CI/CD settings:
Open the file in your favorite text editor and copy paste the following code inside:
- echo "This is the test stage."
- apt update
- apt install -y s3cmd
- s3cmd --quiet --no-check-certificate --access_key "$ACCESS_TOKEN" --secret_key "$SECRET_TOKEN" --host object.api.ams.fuga.cloud --host-bucket object.api.ams.fuga.cloud/$PROJECT_ID/%\(bucket\) --exclude ".git/*" put -r ./ s3://YOUR_CONTAINER
Make sure to replace YOUR_CONTAINER with the name of your container.
Save the file and we’re now ready to test our CI/CD pipeline.
Enter the following command to create a test file:
echo Testfile >> testfile
Execute the following command to add all the files to your git staged changes:
git add .
Execute the following command to create a new commit:
git commit -m "Initial commit"
Push all the changes to your GitLab:
Go to your GitLab project and in the left menu bar click on CI/CD.
There should be two green check marks indicating both of your CI/CD stages as successful.