with Ansible, Vagrant and Continuous Integration
Problem
Developers need a specific set of software tools to get the job done.
You might have new starters in your team OR you need to fundamentally change the software environment that your current team uses in some way — perhaps migrating to a different OS?
The list of software tools we all need can be quite extensive. While some of these software tools you can obtain from package managers (e.g. Aptitude, Snap, Homebrew, etc…), they can still have quite complex installation steps (which could randomly change and break!).
These steps might be documented, they might not. There’s also the problem that having a developer maintain their own base software environment can mean huge inconsistencies between different dev machines that get worse over time.
Addressing the problem can create a huge hit to productivity. For on-boarding new starters it can mean a whole lot of effort lining up knowledgeable people in your team supporting your new starter for the next X days, or a long and drawn out migration plan if your problem is needing to change a dev team’s existing software environment.
A Solution!
- Leverage the power of Ansible and script up your development environment (with all of its tools and dependencies) in a playbook.
- Use Vagrant to spin up a VirtualBox of your base machine image (of the OS / Distro you’re using).
- Run Ansible as a provisioner in the build of the virtual machines and use the playbook you’ve wrote.
- Output success or failure of provisioning steps and run in a Continuous Integration tool.
- Communicate updates and releases of this playbook to your team and ask them to pull and run the updates.
What do you get? A base development environment that will be consistent across your team, that you know will have everything a developer needs to get started and you can rely on every time it’s installed (providing it’s actively maintained).
What the heck is Ansible?
Ansible is an IT automation tool which can be used for a number of different tasks and IT needs. It uses a declarative DSL written in YAML and can be run locally, within the provisioning of a virtual machine, cloud environment or via ssh against a list of known hosts.
Writing a Playbook
For the sake of brevity for this article, I’m going to assume some Ansible knowledge — there’s a fantastic zero-to-hero tutorial below which will take you through the Ansible syntax and some of the built-in plugins it uses:
An Ansible2 Tutorial_Updated for Ansible2! Ansible is one of the simplest server provisioning and configuration management tools. This is a…_serversforhackers.com
We’re going to start with a basic playbook file which will reference the roles and tasks we need to create a development environment:
We’ll then build out the roles / tasks and vars we need. I’ll only really describe the first of these — you can checkout the playbook if you would like to see how the rest of the steps are implemented and the project should be structured.
Packages Task
Vars file for Packages Task
Running the Playbook
ansible-playbook -i "localhost," -c local configure.yml
[00:00:30][00:00:30] PLAY [all] *********************************************************************[00:00:30][00:00:30] TASK [Gathering Facts] *********************************************************[00:00:31] ok: [localhost][00:00:31][00:00:54] TASK [packages : Install required apt packages] ********************************[00:02:59] changed: [localhost][00:02:59][00:02:59] TASK [packages : Remove unrequired apt packages] *******************************[00:02:59] ok: [localhost]
If any of the steps fail, the playbook fails and returns an exit code of 0 (failure).
Stitching it Together (and Testing)
Great! We now have a playbook we can use to install and remove various aptitude packages — but as we extend it, how should we run it and test it to make sure it doesn’t brick our machine?
We don’t really want to risk breaking our current environment if we can help it. This is where Vagrant comes in handy.
What the heck is Vagrant?
Vagrant is an open source tool for initialising and configuring virtual environments. Vagrant provides a command-line interface for managing virtual environments, and uses a file called a Vagrantfile to provide the definition of the environment that needs to be built.
Here’s a very simple Vagrantfile which creates a VM containing an Ubuntu OS:
We can configure a Vagrantfile to run our Ansible playbook once a virtual machine of our choosing has booted up:
Bringing our Development Machine Up
Hashicorp take great pride in how simple it is to bring up a Vagrant machine — simply: vagrant up
Bringing machine 'dev' up with 'virtualbox' provider...==> dev: Checking if box 'bento/ubuntu-18.04' is up to date...==> dev: Running provisioner: ansible...
dev: Running ansible-playbook...PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false ANSIBLE_SSH_ARGS='-o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --connection=ssh --timeout=30 --limit="dev" --inventory-file=/home/andy/Projects/development-environment-playbook/.vagrant/provisioners/ansible/inventory -v configure.ymlUsing /home/andy/Projects/development-environment-playbook/ansible.cfg as config file
PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************ok: [dev]
(If it’s the first time you’ve done this, you’ll see a lot of output from the downloading of the appropriate VM image if it’s not available on your machine already).
Integrating Playbook Build into a CI Pipeline
You can probably guess what would come here if you’re familiar with the concept of Continuous Integration and the tools available. There will be many options and differences here depending on what CI solution you use. In this example I’ll briefly cover over how this could work in one of the more popular solutions: Jenkins — which I’ll also assume a working knowledge of — here’s a good knowledge-base around Jenkins pipe-lining.
Installing / Configuring Jenkins
I’ll assume a pre-existing installation of Jenkins — but if not, here’s the official docker image so you can try it out:
jenkinsci/docker_Docker official jenkins repo. Contribute to jenkinsci/docker development by creating an account on GitHub._github.com
The only special considerations here are to ensure that the Jenkins CI server, container or VM has all the dependencies it needs to run VMs via Vagrant. This will obviously differ dependent on the base OS and version. Here’s an example for installing Vagrant on Centos 7.
A Simple Pipeline
Assuming you actually want to know when your build fails, with the help of the Slack plugin for Jenkins — your Jenkinsfile could be as simple as:
pipeline {stages {stage ('Start') {steps {sh 'vagrant up'}}}post {success {slackSend (color: '#00FF00', message: "A new version of the development environment is available!")}
failure {slackSend (color: '#FF0000', message: "The latest build of the development environment FAILED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL})")}}}
And if all goes well? Voila!
Rolling it out
If you commit to this approach and embed this in your team — here’s what benefits you could see:
- You now have a comprehensive list of most of the software the team is using.
- You can on-board new starters much more quickly because all they need to do is run an automated playbook to have a working environment.
- When “developer accidents” happen and machines are irrevocably broken — a new and clean state isn’t far away and can be achieved in a short space of time.
- You can roll out updates of a standard development environment by creating a new release version of the playbook.
- If install steps change or become outdated, the build will fail and you will get early visibility that you need to update your steps before it becomes a problem to solve at an inconvenient time.
Other OS’s?
I’ve focused quite a lot on the specifics of implementing this strategy for a Linux based environment (specifically Ubuntu), but this strategy can really apply to any OS (in principle any that you could obtain a VM image for). Providing you have good scripting skills in your preferred environment — you could easily tailor this strategy to suit your needs.
Final Thoughts (and an example)
There are obviously many ways to achieve the same result as this, but hopefully this article will help those in need of a solution to this problem and who perhaps have constraints preventing them from adopting other approaches.
Here’s a full example I’ve whipped up for a development setup I’ve used in the past. It’s by no means perfect.
AndyMacDroo/ubuntu-development-environment-playbook_Ansible playbook for an ubuntu development environment. - AndyMacDroo/ubuntu-development-environment-playbook_github.com
Note: The cloud based CI integration I’ve used here to build the project, unfortunately doesn’t use VirtualBox / Vagrant due to some known limitations in this area (in a nutshell it amounts to issues running and deploying VMs inside the virtualised workers initialised in cloud CI pipelines)— the configuration I’ve applied here instead just runs the playbook against the Ubuntu Bionic VM provided by AppVeyor.
Ideas and Further Suggestions:
- Add replacements to
.bashrcto include any custom scripts or utilities that developers might need or would save them time into your playbook. - Add additional tests / health checks to confirm vital software and tools have been installed correctly and are functional to improve reliability of the CI build.
- Create CI build steps to run a build where each successive release of your new and shiny development environment playbook is applied cumulatively over your virtual machines in test (dev machines are rarely clean for very long — ensure your playbook can be used for ‘updates’ of an already existing environment as well as from clean builds).
- Split your playbook into a base playbook and a development environment specific one — if you’re tempted to use Ansible to provision your production environment, why not make sure that your team’s development machines also shares things in common with your production environment?
- Use a technology like Packer to create an image of the development environment and build a CI process around running the image rather than spawning virtual machines.
Thanks for reading! Hope you enjoyed it☺️