Configuring GitLab CI on AWS EC2 Using Docker

GitLab CI can be very cryptic to learn, even if the maintainers say that it’s easy. And the reason is that you need to go through hoops such as configuring a “Runner”, and then create a file which you barely know what to put inside.

In this post, I continue from [Tutorial — Guide] Installing GitLab, GitLab CI on AWS EC2 from Zero, and show you how to use a Docker runner with GitLab.

Posts:

1. [Tutorial — Guide] Installing GitLab, GitLab CI on AWS EC2 from Zero.
2. Configure GitLab CI on AWS EC2 Using Docker (This Post)
3. Configuring .gitlab-ci.yml
4. Troubleshooting GitLab and GitLab CI

I will go through each one of those steps. Feedback is appreciated, in case something needs clarification. If you’re here for the .gitlab-ci.yml file and how to use it, go here.

From the first part: you learned how to set up GitLab on AWS EC2 instance… But what about GitLab CI? Well, it’s a long process as well. But if you’ve made it this far, pat yourself in the back and let’s continue!

I’m going to assume that you:

• Already have setup GitLab.
• You went ahead, and created or imported a project (Please, do this after you assign an Elastic IP to the GitLab instance).

#1- Introduction

Things have changed ever since GitLab 10.3. GitLab introduced a Kubernetes Cluster that allows you to run CI in there. Kubernetes is an open-source container management platform (You can deploy Docker images inside it). This tutorial will not cover this. This will cover running a separate Docker instance that will run the CI. I will make a future tutorial on how to set it up with Google Cloud Platform.

For GitLab to use the CI, it needs help from what it calls a GitLab runner. It’s basically a scipt that is executed to run the CI jobs. Yes, they usually reside in a different server than your GitLab installation (Yup, another additional server to spin). They can also be installed in your current machine and do the process there. Haven’t looked into that, yet. I wouldn’t recommend it anyways, since GitLab is resource-intensive by its own.

The good news is that they’re cheap to run. You can run them in micro, nano, and even spot instances! If you’re not familiar with the latter, Amazon has what they call Spot Instances. Basically, since the runner will not always run (It only runs when someone with the CI config file pushes to the server), you don’t need to have a server that is 100% active. You can just rent it per limited time frames (Spot Instances), and you’ll be good to go.

#2- Creating the instance.

You’re more than welcome to create a dedicated instance, and shut it down when it doesn’t need to. You do not need to assign an Elastic IP to the spot instance or the dedicated one that the GitLab runner is using. It’s the runner that communicates to GitLab, not the other way around.

Let’s go to the EC2 Dashboard

You can arrive here, by clicking “Services” on the top and then clicking “EC2”

Click where it says “Launch Instance”

On the next screen select the Ubuntu Server 64-bit, which has a free tier (If you’re applicable). The version of Linux you choose doesn’t matter. Ubuntu is just more convenient to use.

You want to select the t2.micro since it could potentially give you a free tier. You don’t need a lot of power (This is just at first. If you need more power and speed, go ahead and enable the better instances. For occasional commits (1–2 per day) you won’t die)

Then, select “Next: Configure Instance Details”

In here, you have the option to “Request Spot Instances”. I recommend you to do it, since it can save you some money. You can read more about EC2 Spot Instances, here.

Do not underestimate the next step. Add as much SSD as the free tier gives you. GitLab CI can use Docker (And it’s going to be the method we’re going to use. You don’t have to learn Docker in order to use it), and it’s going to fill up the space really fast (Which you’ll have to give maintenance over time, and delete the images, more on that later).

After that, click on “Next: Add Tags”

Tags aren’t mandatory, and you can safely skip this step.

I always recommend the following step because it secures it, and adds a layer of security to prevent unwanted access. Give access only to your IP. Understand, that you’re going to have to do this process each time your IP changes and you want to connect to the machine (If the IP remains the same, then you don’t need to do anything).

Check that everything is OK, and Launch it!

The following screen will ask you whether you want to use an existing or a new one. If you want to have the maximum security use a new key specifically for the GitLab Runner.

#3 Installing the GitLab Runner

Once you have your instance up and running, connect to it via PuTTY. It’s the same procedure as in the previous article. Put the IPv4 Address into PuTTY

Put it into PuTTY (And remember about the .ppk file as well!)

Click on Yes.

Once you’re logged in:

Run in the command:

sudo apt-get update

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash

sudo apt-get install gitlab-runner

If you ever want to update the runner, you just do:

sudo apt-get updatesudo apt-get install gitlab-runner

Now you must register the runner:

sudo gitlab-runner register

You’ll be prompted with the following

For the URL, type the IPv4 address with the trailing slash (Slash at the end). For example:

http://ec2–28-251–133-100.us-east-2.compute.amazonaws.com/

Then, we need to head to GitLab. Go to the project that you want to setup CI, and head to the main repo page. Then, in the left sidebar, there’s going to be a “Settings” link. Click there. It’ll show you CI/CD on the bottom. Once you’re there, look for “Runners settings”, and click “Collapse”.

Then it’s going to ask you for a description, you can just type “Docker Runner”

#3.1 A quick note about tags.

Tags are used in the GitLab CI (Continuous Integration) environment to classify, and control access (From the projects in GitLab) to the runners. They are configured in both: .gitlab-ci.yml (More on that later), and the runner itself. We’re going to leave this empty for simplicity. You’re more than welcome to add tags to the runner.

When it asks you to whether lock the Runner to current project [true/false], I’ll leave that one up to you. If you want to use that same instance to process many projects:

Choose false…

(false) It’ll save you money. This means that many projects (a.k.a repositories) will use the same runner to process the Integration pipeline. It’s perfect if you have a small team, and you’re committing at different schedules and few times a day. true)

Choose true…

This means that you’ll have to register a new runner for each project. Yes, you can register multiple runners in a same instance, although you have to remember that each runner will eat the CPU when running (no pun intended). Therefore, I recommend, whenever you have medium-big team (or you’re integrating multiple projects multiple times per day), have separate runners. For you to add separate runners, is the same procedure, than what we’re doing right now.

For simplicity, we’re going to choose false, and we’ll prevent the Runner from being locked to the current project.

Once, you’ve done that, it’s going to ask you:

You’re going to type shell. Don’t type Docker. The reason is that if we specify Docker, we’ll be specifying what we call a Docker Image from the Docker hub, and we will specify ours in the config file that we’ll create later.

There we go:

#3.2- Removing unused runners

Running multiple times gitlab-runner register will create multiple runners and it won’t delete them for you. If you want to delete unused runners, then you can run:

sudo gitlab-runner verify — delete

To verify that the runner is working:

sudo gitlab-runner start

Navigate to the page you were before (Project Settings => CI/CD), and verify that the runner is registered and running.

#3.3 Installing Docker on the GitLab runner

This follows the normal installation of Docker in an Ubuntu server. Check this link for the official method of installation. I’ll go through the steps here:

1. Update the apt package index:

sudo apt-get update

2. Install packages to allow apt to use a repository over HTTPS:

sudo apt-get install \    apt-transport-https \    ca-certificates \    curl \    software-properties-common

3. Add Docker’s official GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

4. Setup the repository:

sudo add-apt-repository \   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \   $(lsb_release -cs) \   stable"

5. Update the package

sudo apt-get update

6. Install the latest version of Docker cE:

sudo apt-get install docker-ce

This will bring a yes/no question: Select yes to install.

7. Verify that Docker got installed correctly by running:

sudo docker run hello-world

This is it! Finally, the runner has been configured 🎉🎉

With that in place, the runner should've been successfully configured.

#4 Configuring the CI pipeline in GitLab

This is the moment, everyone! The moment to finally configure the CI pipeline!

😱: “Hold on Joe! You’re telling me that we haven’t done *** yet?”

🔥:“That’s right! We’ve been merely getting the ingredients for the dish. Now we’re preparing it.”

Everything we’ve done so far was to setup the environment for the CI to work. All the Continuous Integration (CI) will work through a .gitlab-ci.yml file that you need to create in the project.

The beauty of this file is that the CI pipeline is only coupled (dependent) on that file, and you are open to do anything you want.

The whole specification of the file can be found here:

Configuration of your jobs with .gitlab-ci.yml | GitLab_Documentation for GitLab Community Edition, GitLab Enterprise Edition, Omnibus GitLab, and GitLab Runner._docs.gitlab.com

Unfortunately, the post has become so long that I had to split it, into a secondary one.

Let’s continue in the following one:

Configuring .gitlab-ci.yml with AWS EC2 for Continuous Integration (CI) or Continuous Deplyment (CD)