Building Elixir Applications as systemd Services

The story of deploying elixir applications in production is always a tough one, as BEAM applications love to live in VM or bare metal without most abstractions. If you don't want to use prepared services like gigalixer, deploying elixir applications to your own VMs is never trivial.

Beginning

As a developer and tinkerer I love being on the cutting edge of open source. Arch linux is an OS of choice, a tool of the trade. The projects that I work on always need to be built on Arch. However, more often than not, there will be problems with deploying a project built on a different linux distro or version. Libraries not matching (I am looking at you OpenSSL in Debian), tools not available, and so on.

One solution is to containerize everything into Docker or K8ns, deploy containers, and be done with it.

Since I am a Bad Dev, I ain't got time to learn new tech stack such as docker, and I like doing things the old way - deploying everything into VM directly.

So how do I deploy my projects into Debian or CentOS VM from an Arch machine? - By compiling them first inside a container with an OS identical to my production VM.

What are the options?

• Virtual Box - lol no
• Docker or K8s container - no, read above
• Vagrant - Too heavy. By default, it runs a virtual box.
• LXD/LXC containers - Maybe. A bunch of configs required though, and permissions set up
• Systemd-Nspawn - Simple, Light, perfect for Lazy Devs

Systemd

Nspawn is one of the components of systemd that allows you to run containers (even docker containers) in your system. It's like chroot on steroids.

Preparing Nspawn

Our production VMs run Debian Linux, so that's the container we will use. Nspawn comes with systemd, so no need to install anything.

cd /var/lib/machines
debootstrap --include=systemd-container --components=main,universe stable debian https://deb.debian.org/debian

Now let's ensure we have a password set in the container:

cd /var/lib/machines
systemd-nspawn -D ./debian
passwd
logout

Done. Now you have a debian container living in the /var/lib/machines directory you can start it manually with

systemd-nspawn

machinectl

To be able to build our application in the container using deploy tools like ansible, we need couple more things.

Ensure you won't require sudo to start the container. Let's add a line to sudoers:

user-name ALL=NOPASSWD: /bin/machinectl start debian

Where username is your user name and debian is the name of the nspawn container.

Let's create systemd machine file, which will allow us to start the container automatically on system boot (if required), but also will allow us to mount our project directory directly into the container

sudo vi /etc/systemd/nspawn/debian.nspawn

[Network]
VirtualEthernet=yes
Port=23
Private=true

[Files]
Bind=/home/user-name/projects/my-awesome-project:/home/user-name/my-awesome-project

[Exec]
PrivateUsers=false

When we do

machinectl start debian

Bringing this into CI

We've set up our container, now the only thing left is to trigger build inside of the container as part of the deploy process. Let's test if we can run

mix release

ssh user-name@debian -p 23 'bash -c "cd /home/user-name/projects/my-awesome-project && MIX_ENV=prod mix release --overwrite --force --path ../release"'

If successful you will find a new folder "release" with build artifacts, which you can deploy to your prod server as is.

Now part of ansible pipeline:

- name: Boot NSPAWN debian host
  shell: machinectl start debian

- name: Pause for 2 seconds waiting for machine to boot
  pause:
    seconds: 2

- name: Build App on NSPAWN host
  shell: ssh user-name@debian -p 23 'bash -c "cd /home/user-name/projects/my-awesome-project && MIX_ENV=prod mix release --overwrite --force --path ../release"'

- name: Poweroff NSPAWN debian host
  shell: machinectl poweroff debian

Pause is the only way I found to reliably wait for the container to boot, let me know if there is a better way.

That is how we deploy at pagerevew.io. You can adjust this for any type of CI you have, even simply use make file.