Site Color

Text Color

Ad Color

Text Color

Evergreen

Duotone

Mysterious

Classic

Sign Up to Save Your Colors

or

Introduction to DevOps Automation Tools [ Hands-on Tutorial ] by@redalegzali

Introduction to DevOps Automation Tools [ Hands-on Tutorial ]

image
Reda Legzali Hacker Noon profile picture

Reda Legzali

IT Enthusiast

In this lab we are going to automate the process of creating the Infrastructure. We will also create a CI/CD Pipeline which will deploy a web application in a container-based environment. We're going to accomplish this using some interesting automation tools. Let's get started !

Pre-requisites

This lab is a step-by-step guide, so you should be able to follow along and copy paste the snippets and commands without any problem. However, having some knowledge of any of the following technologies will help you understand what is happening behind the scenes.

Linux CLI / SSH
Vagrant
Ansible
Docker / Docker Compose
Jenkins

Requirements

Before starting you have to first set up the following:

These tools are available for all platforms. For Windows users, just follow the links and download the exe file. Same for Mac OS users, follow the links and download the dmg file, although it is recommended to use Homebrew. Finally, for Linux users, just use your default package manager.

Introduction

Alright! Now that your setup is ready, let's start by giving you some insights on what we are building.

image

This is the architecture of the project. We are going to use Vagrant to create these two Virtual Machines. The Manager Node will contain Ansible and will configure the 2 Virtual Machines using a Playbook. It will also contain Jenkins to setup the pipeline and Docker to perform the necessary builds and tests. The Dev Node will contain Docker to run the app using port 80.

You can find the demo application we are trying to deploy in this repository. As illustrated via the following architecture, it’s a web application with a Flask backend using a MariaDB database and communicating with a React frontend through a REST API.

image

Infra as Code

Enough "blah blah", show me some code!

That covers the basic theory behind the project. We can now start setting up the Infrastructure of the project. I will be using Linux commands so for Windows users please use "Git Bash" to follow along, it's installed with Git.

First, we need to clone the Infrastructure repository.

git clone https://github.com/RedaLegzali/MovieAppInfra.git

Now that the project is cloned you are gonna need to make some small changes. You should probably open this folder using your favorite text editor like Atom or Visual Studio Code.

Let's start by investing the Vagrantfile. You will find it here :

MovieAppInfra/Vagrant/Vagrantfile  

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  image  = "bento/ubuntu-20.04"
  
  config.vm.define "manager" do |manager|
    manager.vm.box = image
    manager.vm.hostname = "manager"
    manager.vm.network "public_network"
    manager.vm.network "private_network", ip: "192.168.33.10"
    manager.vm.synced_folder "../Ansible/", "/home/vagrant/Ansible/",
    create: true, owner: "vagrant", group: "vagrant", mount_options: ["dmode=755","fmode=755"]
    manager.vm.synced_folder "../Jenkins/", "/home/vagrant/Jenkins/", 
    create: true, owner: "vagrant", group: "vagrant", mount_options: ["dmode=755","fmode=755"]
    manager.vm.provision "shell" do |s|
      s.path = "provisions/manager.sh"
    end
    manager.vm.provider "virtualbox" do |vb|
      vb.gui = false
      vb.memory = 2048
      vb.cpus = 2
    end
  end

  config.vm.define "dev" do |dev|
    dev.vm.box = image
    dev.vm.hostname = "dev"
    dev.vm.network "private_network", ip: "192.168.33.11"
    dev.vm.provision "shell" do |s|
      s.path = "provisions/dev.sh"
    end
    dev.vm.provider "virtualbox" do |vb|
      vb.gui = false
      vb.memory = 1024
      vb.cpus = 1
    end
  end

end

The file is written in a programming language called Ruby. Basically, it will create two Virtual Machines running Ubuntu 20.04 inside VirtualBox. It will also automatically install Ansible inside the Manager server and mount the Jenkins and Ansible directories so that you can find them inside the server without doing anything. The only thing you can change in this file is :

vb.memory
and
vb.cpus
depending on your system resources. Don't change anything else as you might break things :)

Next, let's check the Jenkins configuration file :

MovieAppInfra/Jenkins/Jenkins.yaml

---
jenkins:
  systemMessage: Jenkins as Code 

unclassified:
  shell:
    shell: /bin/bash

credentials:
  system:
    domainCredentials:
    - credentials:
      - usernamePassword:
          id: DockerCreds
          username: dockerhub_username
          password: dockerhub_password
          description: Credentials for Docker Hub
          scope: GLOBAL
      - usernamePassword:
          id: GitCreds
          username: github_username
          password: github_password
          description: Credentials for Git Hub
          scope: GLOBAL
      - basicSSHUserPrivateKey:
          scope: GLOBAL
          id: DevServer
          username: vagrant
          description: Private Key for SSH Agent
          privateKeySource:
            directEntry:
              privateKey: "readFile:${/var/lib/jenkins/.ssh/private_key}"

tool:
  git:
    installations:
      - name: Default
        home: /usr/bin/git
  dockerTool:
    installations:
      - name: Default
        home: /usr/bin/

This YAML file describes the required configuration for Jenkins. You must put your GitHub and DockerHub credentials in here. Make sure your credentials are correct and save the file.

And finally, there are a number of files inside the

MovieAppInfra/Ansible
directory. We will only have a look inside the
playbook.yaml
to have an idea of what's happening

---
- name: Setup System
  hosts: all
  become: yes
  tasks:
    - include: tasks/system.yaml

- name: Setup Docker
  hosts: all
  become: yes
  tasks:
    - include: tasks/docker.yaml
  handlers:
    - include: handlers/docker.yaml

- name: Setup Docker-Compose
  hosts: all
  become: yes
  tasks:
    - include: tasks/dockercompose.yaml

- name: Register Dev Server Private Key
  hosts: dev
  tasks:
    - name: Share Public Key
      shell: cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    - name: Register Private Key
      shell: cat ~/.ssh/id_rsa
      register: private_key

- name: Setup Jenkins
  hosts: manager
  become: yes
  vars:
    private_key: "{{ hostvars[dev].private_key.stdout }}"
  tasks:
    - include: tasks/jenkins.yaml
  handlers:
    - include: handlers/jenkins.yaml

The file is pretty self-explanatory. In short, it does the following:

  1. Upgrade the system and install some useful packages like NGROK which we will discuss later.
  2. Install Docker and Docker Compose
  3. Fetch the SSH private key from the Dev server.
  4. Install Jenkins and place the Dev server's private key in the Jenkins home directory so that we can use it later.

Vagrant

Okay now that we have an overview of what this repository contains, we can resume our work. Get back to your terminal and navigate to the

Vagrant
directory.

cd MovieAppInfra/Vagrant

Now that you are on the right directory you can run the

Vagrantfile
. This will create our two Virtual Machines. It's an operation that will take a few moments the first time you run it, since it has to download the Ubuntu image.

vagrant up

If you have multiple network interfaces in your system you will receive this message. You have to choose your wifi or ethernet interface depending on what you use to connect to Internet. It's usually the first one. Press Enter after you have entered the number of your interface and let Vagrant do its magic.

image

If you open VirtualBox you should see your two Virtual Machines. You don't have to use VirtualBox at all. So you can close it. Your entire work will be done through Vagrant and SSH

image

You can always run one of these two commands if you wish to remove or shutdown the Virtual Machines. Just make sure you are in the right directory.

vagrant destroy // Shutdown and Erase the VMs
vagrant halt // Shutdown the VMs

Ansible

Now that our servers are up and running, let's connect to the Manager server. Give it a second and you should be connected to the server. Your prompt will become :

[email protected]

Now let's navigate to the

Ansible
directory. Remember that Vagrant mounted them automatically.

cd Ansible/

Now we need to share the Manager's public key to be able to run the Ansible Playbook. We will do so by running an Ansible Ad-Hoc command. This is the only time we are going to use a password. The password used in our demo is :

vagrant

ansible -k -m authorized_key -a "user='vagrant' key='{{ lookup('file','~/.ssh/id_rsa.pub') }}'" all
image

To test that Ansible is working correctly, let's Ping Pong !

ansible -m ping all
image

We're all set.

Let's run the Playbook. Give it some time ( just like Vagrant ), it will take some time in the first time.

ansible-playbook playbook.yaml

Jenkins

First, let's retrieve the Jenkins initial password. To do so, execute this command.

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Store the key that this command returns somewhere or simply copy it as we will need it later in order to setup Jenkins.

Open your web browser and visit this repository, make sure you are connected to GitHub, then fork this project. If you are unfamiliar with the term forking in GitHub, it simply means that you will take someone's project and make it your own. Gotta love Open Source :)

image

This will redirect you to your repository. Now you should be able to modify the project.

  1. Click on the
    .env
    file
  2. Click on the small pencil on the right to edit the file.
  3. Replace
    redalegzali
    with your DockerHub username.
  4. Scroll Down and press Commit changes.

Now open a new tab in your browser and visit the following URL : http://192.168.33.10:8080

Paste the Jenkins password we copied earlier.

image
image

Then click on Install suggested plugins and give it some time. Don't worry if some plugins don't get installed, just continue we will go over the important plugins again later. For the next steps just enter your information and leave the default Jenkins URL. Jenkins will restart and ask you to login. Once you login, you will be redirected to the Jenkins dashboard.

Go to Manage Jenkins

image

Then to Manage Plugins

image

Make sure you are in the Available tab and install this list of plugins. This time if it fails you must try again.

  • Credentials Plugin
  • Git Plugin
  • GitHub Plugin
  • Pipeline
  • Docker
  • Docker Pipeline
  • Configuration as Code
  • SSH Agent
    image

When all the installations are done, you will be redirected to the Dashboard. Click on Manage Jenkins again.

This time you should be able to see the Configuration as Code plugin. Instead of doing our entire Jenkins configuration manually, we will simply give this plugin our Jenkins.yaml file and let it handle the configuration for us.

image

Once you click on it, you should see this page. Just paste the path to the configuration file :

/home/vagrant/Jenkins/Jenkins.yaml

And then click on : Apply new configuration

image

You should get this message as a confirmation that the configuration has been set up successfully. You can go back to the Dashboard

image

We are now ready to create our pipeline. Go back to the Dashboard and Click on New Item.

image

Then select Pipeline and give your Job a name then press OK.

image

Start by checking the GitHub Project and enter your repository URL.

image

Next, enter the following parameters : registry = Your Docker Hub username remoteUser = vagrant remoteHost = 192.168.33.11

image
image

Then check this option so that the build can be triggered via GitHub Webhook.

image

And finally, tell the Pipeline to get the Jenkinsfile from the repository. Again, enter your Git repository URL. Select your Git Credentials which have been created with the configuration as code plugin. And finally change the branch to main. Now you can press Save and start the build.

image
image

Click on Build with Parameters and leave the default values. The build will take some time the first time you run it.

image

While it is running, let's talk a bit about what this pipeline does. First, here are the different stages

image

Build

Creates a Python virtual environment and installs the dependencies.

Test

Creates a temporary MariaDB container and run some Unit Tests in Python.

Deploy.

Builds the Docker images and pushes them in your Docker registry. If you go check your Docker registry after the Pipeline is completed, you will find these images.

Run

Executes the Docker Compose in the Dev server which pulls the Docker images from the registry and starts the containers.

image

Now that the Pipeline is finished, give it a few moments the time for Docker Compose to settle and visit this URL : http://192.168.33.11

image

And there you go! Your web application is deployed successfully.

WebHook

Alright, we have come a long way. The last thing we are going to do is to create a GitHub Webhook. This will basically trigger our Pipeline every time we make a Push. Go to your repository in GitHub -> Settings -> Webhooks -> Add Webhook

image
image
image

Now go back to your Manager server. If you closed your SSH session or closed your terminal, just follow the previous steps to connect to the Manager server again. Once you are connected, execute the following command.

ngrok http 8080
image

You should get an output that looks like this. Copy the HTTPS link in my case it's :

https://d096ac97f292.ngrok.io 
This is a temporary link, the purpose of this is to simulate a public ip address. You wouldn't do this in a production environment.

Now go back to GitHub. Paste your NGROK URL followed by

/github-webhook/
For my case
https://d096ac97f292.ngrok.io/github-webhook/
Choose application/json and leave the rest as default and Click on Add Webhook.

image

You will be redirected. Refresh the page and if you get this green checkbox you are good to go.

image

Now let's make a Push to our repository. Just modify the README file inside GitHub and Commit changes.

image
image

Boom ! Our pipeline is triggered automatically.

Conclusion

This wraps up our lab. From here you can extend this pipeline and add one or two more Dev servers and use some container orchestrator like Kubernetes. You can also monitor your infrastructure and application with some tools like Prometheus or Elastic Search. And don't forget about security, for the sake of simplicity we used plain text passwords in this lab but you should probably use some secrets management tool like git-secret, git-crypt or Vault.

Contributors

Samy Simon

Allaki Driss

Tags