State of the art in deploying Elixir / Phoenix applications

This is opinionated guide based on running 3 to 4 Phoenix applications in production for over a year at Kontomatik. The largest of our apps is facing about 14K paying users daily.

I might be wrong and we might do things differently at Kontomatik in the future. However, to the best of my knowledge, this is the best way to deploy Phoenix Framework based applications as of March 2017.

Use asdf to manage Erlang and Elixir versions on your laptop

Assume working on multiple Elixir projects from the start. Even if you have only one production app right now, you will want to experiment with different versions of your app and with other open source projects. The bottom line is, you will inevitably need to manage Erlang and Elixir versions easily on your laptop.

Use asdf version manager and its .tool-versions file to pin specific Erlang and Elixir version for each of your projects.

Obviously, these must match the versions you are running in production.

The asdf version manager worked very well in my experience. The other tools (kiex etc) were lacking (for example by only supporting Elixir).

Use distillery to package releases

The Erlang/OTP release is a standard way of packaging projects to make them runnable on the Erlang VM.

Distillery is the best tool Elixir community has to offer for building releases.

It is important to understand Distillery does not touch servers or deployment. The end result is a local catalog (or gzipped file) containing the release.

Internally, release building is a complex beast. Most of the time, Distillery will completely abstract this complexity away. Once in a while, you will need to get your hands dirty and understand some aspects of it in more detail.

Elixir is built on top of the 31-years old Erlang/OTP ecosystem and deployment is one aspect where the legacy shit shines through.

Do not attempt to rsync Elixir source code “The Rails Way” to compile it on the server. It is a dead end and wrong approach in the Erlang ecosystem.

Do not install Erlang on the server — embed it into the release

Configure Distillery to embed Erlang in the release:

# rel/config.exsenvironment :prod doset include_erts: true

...

end

Do not install Erlang on the server.

Unfortunately, Erlang VM is not fully backwards compatible. Code compiled for old VM can crash on the newer VM. This is really annoying, and unheard of in the JVM and CLR world.

Practically, this means you cannot upgrade your server Erlang installation without significant downtime. The infrastructure work must be synchronized with deployment of the new app version (compiled for the new Erlang). This bi-directional deployment coupling feels like a stone age and is unacceptable in the modern technology environment.

Embedding Erlang into release solves this issue completely but introduces a new problem: different programmers have different, incompatible builds of Erlang (Ubuntu LTS vs Arch rolling release vs OS X etc).

One cannot simply take OS X Erlang binary (and dependencies) and deploy it to production Ubuntu LTS server.

We will tackle this in the next chapter.

Use docker to build locally for the production environment

I do not mean to build docker images. Docker is only necessary to have production-compatible building environment, regardless of your underlying development machine.

You want to build your Erlang/OTP release using Distillery inside a running docker container.

Keep your Dockerfile 100% close to what you use in production — the same Linux distribution version, the same Erlang and Elixir versions as pinned in your .tool-versions, etc. Be aware that build of the Erlang itself will have hardcoded specific versions of dependencies (like “libncursesw.so.6” on Arch and “libncursesw.so.5” on Ubuntu LTS) depending on the platform it was built. This is why you need docker even if everyone on your team uses Linux.

As a side note, Docker can also be used to quickly setup development environment. Although I am not a big fan of this. It has some uses but typically you will wan’t to have a direct control and simplicity coming from developing directly on your laptop w/o layers of complexity. Also, things like contextual test running in IntelliJ won’t work with containers. Prefer the local setup.

PS Why not use edeliver? The tool assumes a deployment server between programmer and production. There is nothing wrong with that except it’s an overkill for most projects. For small-to-midsize projects I tend to avoid that for speed and simplicity, to have less moving parts, and less servers to manage. That being said, true deployment pipeline is perfectly fine for larger teams and projects.

Do not use hot code upgrades

Erlang allows for updating the running app to achieve extreme uptimes.

The downside is that you need to migrate data structures in your application. Deployment is no longer a no-brainer (as it should be in the continuous deployment world). Suddenly, deployment must be carefully considered — or you risk facing hard to debug issues.

We don’t want that. It’s time consuming and risky.

Simply restart.

To achieve great uptime focus on quick restarts and — if necessary — try classic blue/green deployment or similar patterns.

Use ansible (or similar) to automate deployment

In the Ruby world Capistrano offers several proven flows for deployment. You will want to create something similar using your favorite provisioning tool. If you are clueless, pick ansible.

Some tips for designing your deployment flow:

• production servers should not have access to your source code repository
• as a first step, clone/pull the central repository to another catalog on your laptop (not your project root directory); the build will happen isolated in this catalog
• everything must be built and deployed from the central master (production) branch — not the local programmer’s branch); this is to guarantee that no private commit gets deployed to production
• do not gzip the releases; instead rsync the release catalog; this will allow rsync to only send the actual changes (much faster)
• use per-developer release cache on the server (much faster, avoids permission issues)
• make deployments close-to-atomic by preparing everything on the server and then only switching the symlink to the new release (and forcing a restart); this is know as “current” symlink in the Rails world
• to run database migrations, see: https://github.com/bitwalker/distillery/blob/master/docs/Running%20Migrations.md

Hey, this shit is complicated!

Reliable production deployment is never easy and Elixir adds some accidental complexity on top of it. This is probably the worst aspect of the otherwise wonderful and productive ecosystem we have been in love at Kontomatik for the last year.

On the bright side, Elixir / Phoenix apps:

• have a very low resource usage in terms of memory and CPU; this is exceptionally important when you host in the programmable cloud (things get expensive quickly over there)
• apps restart very quickly, minimizing downtime
• ecosystem is quickly maturing and things will partially streamline in the coming months and years

Follow the suggested best practices to minimize deployment pain and be quickly back to Elixir hacking fun! :)

Hacker Noon is how hackers start their afternoons. We’re a part of the @AMIfamily. We are now accepting submissions and happy to discuss advertising & sponsorship opportunities.

To learn more, read our about page, like/message us on Facebook, or simply, tweet/DM @HackerNoon.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!