

IT Enthusiast
This lab is a step-by-step guide, so you should be able to follow along and copy paste the snippets and commands without any problem. However, having some knowledge of any of the following technologies will help you understand what is happening behind the scenes.
Linux CLI / SSH
Vagrant
Ansible
Docker / Docker Compose
Jenkins
Before starting you have to first set up the following:
These tools are available for all platforms. For Windows users, just follow the links and download the exe file. Same for Mac OS users, follow the links and download the dmg file, although it is recommended to use Homebrew. Finally, for Linux users, just use your default package manager.
Alright! Now that your setup is ready, let's start by giving you some insights on what we are building.
This is the architecture of the project. We are going to use Vagrant to create these two Virtual Machines. The Manager Node will contain Ansible and will configure the 2 Virtual Machines using a Playbook. It will also contain Jenkins to setup the pipeline and Docker to perform the necessary builds and tests. The Dev Node will contain Docker to run the app using port 80.
You can find the demo application we are trying to deploy in this repository. As illustrated via the following architecture, itβs a web application with a Flask backend using a MariaDB database and communicating with a React frontend through a REST API.
Enough "blah blah", show me some code!
That covers the basic theory behind the project. We can now start setting up the Infrastructure of the project. I will be using Linux commands so for Windows users please use "Git Bash" to follow along, it's installed with Git.
First, we need to clone the Infrastructure repository.
git clone https://github.com/RedaLegzali/MovieAppInfra.git
Now that the project is cloned you are gonna need to make some small changes. You should probably open this folder using your favorite text editor like Atom or Visual Studio Code.
Let's start by investing the Vagrantfile. You will find it here :
MovieAppInfra/Vagrant/Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
image = "bento/ubuntu-20.04"
config.vm.define "manager" do |manager|
manager.vm.box = image
manager.vm.hostname = "manager"
manager.vm.network "public_network"
manager.vm.network "private_network", ip: "192.168.33.10"
manager.vm.synced_folder "../Ansible/", "/home/vagrant/Ansible/",
create: true, owner: "vagrant", group: "vagrant", mount_options: ["dmode=755","fmode=755"]
manager.vm.synced_folder "../Jenkins/", "/home/vagrant/Jenkins/",
create: true, owner: "vagrant", group: "vagrant", mount_options: ["dmode=755","fmode=755"]
manager.vm.provision "shell" do |s|
s.path = "provisions/manager.sh"
end
manager.vm.provider "virtualbox" do |vb|
vb.gui = false
vb.memory = 2048
vb.cpus = 2
end
end
config.vm.define "dev" do |dev|
dev.vm.box = image
dev.vm.hostname = "dev"
dev.vm.network "private_network", ip: "192.168.33.11"
dev.vm.provision "shell" do |s|
s.path = "provisions/dev.sh"
end
dev.vm.provider "virtualbox" do |vb|
vb.gui = false
vb.memory = 1024
vb.cpus = 1
end
end
end
The file is written in a programming language called Ruby. Basically, it will create two Virtual Machines running Ubuntu 20.04 inside VirtualBox. It will also automatically install Ansible inside the Manager server and mount the Jenkins and Ansible directories so that you can find them inside the server without doing anything. The only thing you can change in this file is :
vb.memory
and vb.cpus
depending on your system resources. Don't change anything else as you might break things :)Next, let's check the Jenkins configuration file :
MovieAppInfra/Jenkins/Jenkins.yaml
---
jenkins:
systemMessage: Jenkins as Code
unclassified:
shell:
shell: /bin/bash
credentials:
system:
domainCredentials:
- credentials:
- usernamePassword:
id: DockerCreds
username: dockerhub_username
password: dockerhub_password
description: Credentials for Docker Hub
scope: GLOBAL
- usernamePassword:
id: GitCreds
username: github_username
password: github_password
description: Credentials for Git Hub
scope: GLOBAL
- basicSSHUserPrivateKey:
scope: GLOBAL
id: DevServer
username: vagrant
description: Private Key for SSH Agent
privateKeySource:
directEntry:
privateKey: "readFile:${/var/lib/jenkins/.ssh/private_key}"
tool:
git:
installations:
- name: Default
home: /usr/bin/git
dockerTool:
installations:
- name: Default
home: /usr/bin/
This YAML file describes the required configuration for Jenkins. You must put your GitHub and DockerHub credentials in here. Make sure your credentials are correct and save the file.
And finally, there are a number of files inside the
MovieAppInfra/Ansible
directory. We will only have a look inside the playbook.yaml
to have an idea of what's happening---
- name: Setup System
hosts: all
become: yes
tasks:
- include: tasks/system.yaml
- name: Setup Docker
hosts: all
become: yes
tasks:
- include: tasks/docker.yaml
handlers:
- include: handlers/docker.yaml
- name: Setup Docker-Compose
hosts: all
become: yes
tasks:
- include: tasks/dockercompose.yaml
- name: Register Dev Server Private Key
hosts: dev
tasks:
- name: Share Public Key
shell: cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
- name: Register Private Key
shell: cat ~/.ssh/id_rsa
register: private_key
- name: Setup Jenkins
hosts: manager
become: yes
vars:
private_key: "{{ hostvars[dev].private_key.stdout }}"
tasks:
- include: tasks/jenkins.yaml
handlers:
- include: handlers/jenkins.yaml
The file is pretty self-explanatory. In short, it does the following:
Okay now that we have an overview of what this repository contains, we can resume our work. Get back to your terminal and navigate to the
Vagrant
directory.cd MovieAppInfra/Vagrant
Now that you are on the right directory you can run the
Vagrantfile
. This will create our two Virtual Machines. It's an operation that will take a few moments the first time you run it, since it has to download the Ubuntu image.vagrant up
If you have multiple network interfaces in your system you will receive this message. You have to choose your wifi or ethernet interface depending on what you use to connect to Internet. It's usually the first one. Press Enter after you have entered the number of your interface and let Vagrant do its magic.
If you open VirtualBox you should see your two Virtual Machines. You don't have to use VirtualBox at all. So you can close it. Your entire work will be done through Vagrant and SSH
You can always run one of these two commands if you wish to remove or shutdown the Virtual Machines. Just make sure you are in the right directory.
vagrant destroy // Shutdown and Erase the VMs
vagrant halt // Shutdown the VMs
Now that our servers are up and running, let's connect to the Manager server. Give it a second and you should be connected to the server. Your prompt will become :
vagrant@manager
Now let's navigate to the
Ansible
directory. Remember that Vagrant mounted them automatically.cd Ansible/
Now we need to share the Manager's public key to be able to run the Ansible Playbook. We will do so by running an Ansible Ad-Hoc command. This is the only time we are going to use a password. The password used in our demo is :
vagrant
ansible -k -m authorized_key -a "user='vagrant' key='{{ lookup('file','~/.ssh/id_rsa.pub') }}'" all
To test that Ansible is working correctly, let's Ping Pong !
ansible -m ping all
We're all set.
Let's run the Playbook. Give it some time ( just like Vagrant ), it will take some time in the first time.
ansible-playbook playbook.yaml
First, let's retrieve the Jenkins initial password. To do so, execute this command.
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Store the key that this command returns somewhere or simply copy it as we will need it later in order to setup Jenkins.
Open your web browser and visit this repository, make sure you are connected to GitHub, then fork this project. If you are unfamiliar with the term forking in GitHub, it simply means that you will take someone's project and make it your own. Gotta love Open Source :)
This will redirect you to your repository. Now you should be able to modify the project.
.env
fileredalegzali
with your DockerHub username.Now open a new tab in your browser and visit the following URL : http://192.168.33.10:8080
Paste the Jenkins password we copied earlier.
Then click on Install suggested plugins and give it some time. Don't worry if some plugins don't get installed, just continue we will go over the important plugins again later. For the next steps just enter your information and leave the default Jenkins URL. Jenkins will restart and ask you to login. Once you login, you will be redirected to the Jenkins dashboard.
Go to Manage Jenkins
Then to Manage Plugins
Make sure you are in the Available tab and install this list of plugins. This time if it fails you must try again.
When all the installations are done, you will be redirected to the Dashboard. Click on Manage Jenkins again.
This time you should be able to see the Configuration as Code plugin. Instead of doing our entire Jenkins configuration manually, we will simply give this plugin our Jenkins.yaml file and let it handle the configuration for us.
Once you click on it, you should see this page. Just paste the path to the configuration file :
/home/vagrant/Jenkins/Jenkins.yaml
And then click on : Apply new configuration
You should get this message as a confirmation that the configuration has been set up successfully. You can go back to the Dashboard
We are now ready to create our pipeline. Go back to the Dashboard and Click on New Item.
Then select Pipeline and give your Job a name then press OK.
Start by checking the GitHub Project and enter your repository URL.
Next, enter the following parameters : registry = Your Docker Hub username remoteUser = vagrant remoteHost = 192.168.33.11
Then check this option so that the build can be triggered via GitHub Webhook.
And finally, tell the Pipeline to get the Jenkinsfile from the repository. Again, enter your Git repository URL. Select your Git Credentials which have been created with the configuration as code plugin. And finally change the branch to main. Now you can press Save and start the build.
Click on Build with Parameters and leave the default values. The build will take some time the first time you run it.
While it is running, let's talk a bit about what this pipeline does. First, here are the different stages
Creates a Python virtual environment and installs the dependencies.
Creates a temporary MariaDB container and run some Unit Tests in Python.
Builds the Docker images and pushes them in your Docker registry. If you go check your Docker registry after the Pipeline is completed, you will find these images.
Executes the Docker Compose in the Dev server which pulls the Docker images from the registry and starts the containers.
Now that the Pipeline is finished, give it a few moments the time for Docker Compose to settle and visit this URL : http://192.168.33.11
And there you go! Your web application is deployed successfully.
Alright, we have come a long way. The last thing we are going to do is to create a GitHub Webhook. This will basically trigger our Pipeline every time we make a Push. Go to your repository in GitHub -> Settings -> Webhooks -> Add Webhook
Now go back to your Manager server. If you closed your SSH session or closed your terminal, just follow the previous steps to connect to the Manager server again. Once you are connected, execute the following command.
ngrok http 8080
You should get an output that looks like this. Copy the HTTPS link in my case it's :
https://d096ac97f292.ngrok.io
This is a temporary link, the purpose of this is to simulate a public ip address. You wouldn't do this in a production environment.Now go back to GitHub. Paste your NGROK URL followed by
/github-webhook/
For my case https://d096ac97f292.ngrok.io/github-webhook/
Choose application/json and leave the rest as default and Click on Add Webhook.You will be redirected. Refresh the page and if you get this green checkbox you are good to go.
Now let's make a Push to our repository. Just modify the README file inside GitHub and Commit changes.
Boom ! Our pipeline is triggered automatically.
This wraps up our lab. From here you can extend this pipeline and add one or two more Dev servers and use some container orchestrator like Kubernetes. You can also monitor your infrastructure and application with some tools like Prometheus or Elastic Search. And don't forget about security, for the sake of simplicity we used plain text passwords in this lab but you should probably use some secrets management tool like git-secret, git-crypt or Vault.
Contributors
Samy Simon
Allaki Driss
Create your free account to unlock your custom reading experience.