Introduction to DevOps Automation Tools [ Hands-on Tutorial ]

March 23rd 2021 326 reads

@redalegzaliReda Legzali

IT Enthusiast

In this lab we are going to automate the process of creating the Infrastructure. We will also create a CI/CD Pipeline which will deploy a web application in a container-based environment. We're going to accomplish this using some interesting automation tools. Let's get started !

Pre-requisites

This lab is a step-by-step guide, so you should be able to follow along and copy paste the snippets and commands without any problem. However, having some knowledge of any of the following technologies will help you understand what is happening behind the scenes.

Linux CLI / SSH

Vagrant

Ansible

Docker / Docker Compose

Jenkins

Requirements

Before starting you have to first set up the following:

• Create a GitHub Account
• Create a DockerHub Account
• Install VirtualBox
• Install Vagrant
• Install Git

These tools are available for all platforms. For Windows users, just follow the links and download the exe file. Same for Mac OS users, follow the links and download the dmg file, although it is recommended to use Homebrew. Finally, for Linux users, just use your default package manager.

Introduction

Alright! Now that your setup is ready, let's start by giving you some insights on what we are building.

This is the architecture of the project. We are going to use Vagrant to create these two Virtual Machines. The Manager Node will contain Ansible and will configure the 2 Virtual Machines using a Playbook. It will also contain Jenkins to setup the pipeline and Docker to perform the necessary builds and tests. The Dev Node will contain Docker to run the app using port 80.

You can find the demo application we are trying to deploy in this repository. As illustrated via the following architecture, it’s a web application with a Flask backend using a MariaDB database and communicating with a React frontend through a REST API.

Infra as Code

Enough "blah blah", show me some code!

That covers the basic theory behind the project. We can now start setting up the Infrastructure of the project. I will be using Linux commands so for Windows users please use "Git Bash" to follow along, it's installed with Git.

First, we need to clone the Infrastructure repository.

git clone https://github.com/RedaLegzali/MovieAppInfra.git

Now that the project is cloned you are gonna need to make some small changes. You should probably open this folder using your favorite text editor like Atom or Visual Studio Code.

Let's start by investing the Vagrantfile. You will find it here :

MovieAppInfra/Vagrant/Vagrantfile  

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  image  = "bento/ubuntu-20.04"
  
  config.vm.define "manager" do |manager|
    manager.vm.box = image
    manager.vm.hostname = "manager"
    manager.vm.network "public_network"
    manager.vm.network "private_network", ip: "192.168.33.10"
    manager.vm.synced_folder "../Ansible/", "/home/vagrant/Ansible/",
    create: true, owner: "vagrant", group: "vagrant", mount_options: ["dmode=755","fmode=755"]
    manager.vm.synced_folder "../Jenkins/", "/home/vagrant/Jenkins/", 
    create: true, owner: "vagrant", group: "vagrant", mount_options: ["dmode=755","fmode=755"]
    manager.vm.provision "shell" do |s|
      s.path = "provisions/manager.sh"
    end
    manager.vm.provider "virtualbox" do |vb|
      vb.gui = false
      vb.memory = 2048
      vb.cpus = 2
    end
  end

  config.vm.define "dev" do |dev|
    dev.vm.box = image
    dev.vm.hostname = "dev"
    dev.vm.network "private_network", ip: "192.168.33.11"
    dev.vm.provision "shell" do |s|
      s.path = "provisions/dev.sh"
    end
    dev.vm.provider "virtualbox" do |vb|
      vb.gui = false
      vb.memory = 1024
      vb.cpus = 1
    end
  end

end

The file is written in a programming language called Ruby. Basically, it will create two Virtual Machines running Ubuntu 20.04 inside VirtualBox. It will also automatically install Ansible inside the Manager server and mount the Jenkins and Ansible directories so that you can find them inside the server without doing anything. The only thing you can change in this file is :

vb.memory

vb.cpus

Next, let's check the Jenkins configuration file :

MovieAppInfra/Jenkins/Jenkins.yaml

---
jenkins:
  systemMessage: Jenkins as Code 

unclassified:
  shell:
    shell: /bin/bash

credentials:
  system:
    domainCredentials:
    - credentials:
      - usernamePassword:
          id: DockerCreds
          username: dockerhub_username
          password: dockerhub_password
          description: Credentials for Docker Hub
          scope: GLOBAL
      - usernamePassword:
          id: GitCreds
          username: github_username
          password: github_password
          description: Credentials for Git Hub
          scope: GLOBAL
      - basicSSHUserPrivateKey:
          scope: GLOBAL
          id: DevServer
          username: vagrant
          description: Private Key for SSH Agent
          privateKeySource:
            directEntry:
              privateKey: "readFile:${/var/lib/jenkins/.ssh/private_key}"

tool:
  git:
    installations:
      - name: Default
        home: /usr/bin/git
  dockerTool:
    installations:
      - name: Default
        home: /usr/bin/

This YAML file describes the required configuration for Jenkins. You must put your GitHub and DockerHub credentials in here. Make sure your credentials are correct and save the file.

And finally, there are a number of files inside the

MovieAppInfra/Ansible

playbook.yaml

---
- name: Setup System
  hosts: all
  become: yes
  tasks:
    - include: tasks/system.yaml

- name: Setup Docker
  hosts: all
  become: yes
  tasks:
    - include: tasks/docker.yaml
  handlers:
    - include: handlers/docker.yaml

- name: Setup Docker-Compose
  hosts: all
  become: yes
  tasks:
    - include: tasks/dockercompose.yaml

- name: Register Dev Server Private Key
  hosts: dev
  tasks:
    - name: Share Public Key
      shell: cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    - name: Register Private Key
      shell: cat ~/.ssh/id_rsa
      register: private_key

- name: Setup Jenkins
  hosts: manager
  become: yes
  vars:
    private_key: "{{ hostvars[dev].private_key.stdout }}"
  tasks:
    - include: tasks/jenkins.yaml
  handlers:
    - include: handlers/jenkins.yaml

The file is pretty self-explanatory. In short, it does the following:

1. Upgrade the system and install some useful packages like NGROK which we will discuss later.
2. Install Docker and Docker Compose
3. Fetch the SSH private key from the Dev server.
4. Install Jenkins and place the Dev server's private key in the Jenkins home directory so that we can use it later.

Vagrant

Okay now that we have an overview of what this repository contains, we can resume our work. Get back to your terminal and navigate to the

Vagrant

cd MovieAppInfra/Vagrant

Now that you are on the right directory you can run the

Vagrantfile

vagrant up

If you have multiple network interfaces in your system you will receive this message. You have to choose your wifi or ethernet interface depending on what you use to connect to Internet. It's usually the first one. Press Enter after you have entered the number of your interface and let Vagrant do its magic.

If you open VirtualBox you should see your two Virtual Machines. You don't have to use VirtualBox at all. So you can close it. Your entire work will be done through Vagrant and SSH

You can always run one of these two commands if you wish to remove or shutdown the Virtual Machines. Just make sure you are in the right directory.

vagrant destroy // Shutdown and Erase the VMs
vagrant halt // Shutdown the VMs

Ansible

Now that our servers are up and running, let's connect to the Manager server. Give it a second and you should be connected to the server. Your prompt will become :

vagrant@manager

Now let's navigate to the

Ansible

cd Ansible/

Now we need to share the Manager's public key to be able to run the Ansible Playbook. We will do so by running an Ansible Ad-Hoc command. This is the only time we are going to use a password. The password used in our demo is :

vagrant

ansible -k -m authorized_key -a "user='vagrant' key='{{ lookup('file','~/.ssh/id_rsa.pub') }}'" all

To test that Ansible is working correctly, let's Ping Pong !

ansible -m ping all

We're all set.

Let's run the Playbook. Give it some time ( just like Vagrant ), it will take some time in the first time.

ansible-playbook playbook.yaml

Jenkins

First, let's retrieve the Jenkins initial password. To do so, execute this command.

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Store the key that this command returns somewhere or simply copy it as we will need it later in order to setup Jenkins.

Open your web browser and visit this repository, make sure you are connected to GitHub, then fork this project. If you are unfamiliar with the term forking in GitHub, it simply means that you will take someone's project and make it your own. Gotta love Open Source :)

This will redirect you to your repository. Now you should be able to modify the project.

1. Click on the

   .env

   file
2. Click on the small pencil on the right to edit the file.
3. Replace

   redalegzali

   with your DockerHub username.
4. Scroll Down and press Commit changes.

Now open a new tab in your browser and visit the following URL : http://192.168.33.10:8080

Paste the Jenkins password we copied earlier.

Then click on Install suggested plugins and give it some time. Don't worry if some plugins don't get installed, just continue we will go over the important plugins again later. For the next steps just enter your information and leave the default Jenkins URL. Jenkins will restart and ask you to login. Once you login, you will be redirected to the Jenkins dashboard.

Go to Manage Jenkins

Then to Manage Plugins

Make sure you are in the Available tab and install this list of plugins. This time if it fails you must try again.

• Credentials Plugin
• Git Plugin
• GitHub Plugin
• Pipeline
• Docker
• Docker Pipeline
• Configuration as Code
• SSH Agent

When all the installations are done, you will be redirected to the Dashboard. Click on Manage Jenkins again.

This time you should be able to see the Configuration as Code plugin. Instead of doing our entire Jenkins configuration manually, we will simply give this plugin our Jenkins.yaml file and let it handle the configuration for us.

Once you click on it, you should see this page. Just paste the path to the configuration file :

/home/vagrant/Jenkins/Jenkins.yaml

And then click on : Apply new configuration

You should get this message as a confirmation that the configuration has been set up successfully. You can go back to the Dashboard

We are now ready to create our pipeline. Go back to the Dashboard and Click on New Item.

Then select Pipeline and give your Job a name then press OK.

Start by checking the GitHub Project and enter your repository URL.

Next, enter the following parameters : registry = Your Docker Hub username remoteUser = vagrant remoteHost = 192.168.33.11

Then check this option so that the build can be triggered via GitHub Webhook.

And finally, tell the Pipeline to get the Jenkinsfile from the repository. Again, enter your Git repository URL. Select your Git Credentials which have been created with the configuration as code plugin. And finally change the branch to main. Now you can press Save and start the build.

Click on Build with Parameters and leave the default values. The build will take some time the first time you run it.

While it is running, let's talk a bit about what this pipeline does. First, here are the different stages

Build

Creates a Python virtual environment and installs the dependencies.

Test

Creates a temporary MariaDB container and run some Unit Tests in Python.

Deploy.

Builds the Docker images and pushes them in your Docker registry. If you go check your Docker registry after the Pipeline is completed, you will find these images.

Run

Executes the Docker Compose in the Dev server which pulls the Docker images from the registry and starts the containers.

Now that the Pipeline is finished, give it a few moments the time for Docker Compose to settle and visit this URL : http://192.168.33.11

And there you go! Your web application is deployed successfully.

WebHook

Alright, we have come a long way. The last thing we are going to do is to create a GitHub Webhook. This will basically trigger our Pipeline every time we make a Push. Go to your repository in GitHub -> Settings -> Webhooks -> Add Webhook

Now go back to your Manager server. If you closed your SSH session or closed your terminal, just follow the previous steps to connect to the Manager server again. Once you are connected, execute the following command.

ngrok http 8080

You should get an output that looks like this. Copy the HTTPS link in my case it's :

https://d096ac97f292.ngrok.io 

Now go back to GitHub. Paste your NGROK URL followed by

/github-webhook/

https://d096ac97f292.ngrok.io/github-webhook/

You will be redirected. Refresh the page and if you get this green checkbox you are good to go.

Now let's make a Push to our repository. Just modify the README file inside GitHub and Commit changes.

Boom ! Our pipeline is triggered automatically.

Conclusion

This wraps up our lab. From here you can extend this pipeline and add one or two more Dev servers and use some container orchestrator like Kubernetes. You can also monitor your infrastructure and application with some tools like Prometheus or Elastic Search. And don't forget about security, for the sake of simplicity we used plain text passwords in this lab but you should probably use some secrets management tool like git-secret, git-crypt or Vault.

Contributors

Samy Simon

Allaki Driss