1,933 reads

Building a Scalable URL Shortener Service using Swift with Vapor3

by Andrew ScottNovember 16th, 2018

Too Long; Didn't Read

After I finished a couple basic Vapor tutorials and built and wrote about <a href="https://hackernoon.com/building-an-excellent-mock-server-in-swift-with-vapor-7e25aaa985e6" target="_blank">creating a Mock server using Vapor</a> I struggled to think of what to do next. I ended up coming across this post on Medium which seemed like an excellent project to try to build using <a href="https://hackernoon.com/tagged/vapor" target="_blank">Vapor</a>.

Coin Mentioned

featured image - Building a Scalable URL Shortener Service using Swift with Vapor3

Moving beyond “Hello, world” using Swift with Vapor3

After I finished a couple basic Vapor tutorials and built and wrote about creating a Mock server using Vapor I struggled to think of what to do next. I ended up coming across this post on Medium which seemed like an excellent project to try to build using Vapor.

How to build a Tiny URL service that scales to billions?_Building a url shortner service which scales to billions is an software architecture challenge. This blog covers basic…_medium.com

In this post I’ll look at actually building a URL shortening service based on some of the designs in the post above by combining Vapor, Redis, Nginx, and Postgres and utilizing Docker to ease development and testing.

The Goal

The goal for this project is to build a scalable, quick URL shortener that will support both automatic and custom URL shortening capabilities using Vapor. The goal of creating enumeration-resistant keys will be outside of the scope for this project — the assumption being that the onus is on the link creator to ensure that a shared url is protected against unauthorized users.

The Design

I’ll break the design down by component and then look at how each piece fits into the logical flow of creating a new url and accessing an existing url.

Vapor Web Application

From routing logic to the shortening algorithm itself, the majority of the actual business logic will take place inside the Vapor app. I’ve tried to plan this in such a way that the number of instances of the Vapor app could be scaled up to meet demand if necessary.

Postgres Database

Postgres will serve as the persistent storage for the application. Postgres seemed like a good choice just due to its ease of use, integration with vapor using Fluent, and it being cloud agnostic.

Redis

Redis will be serving dual duties. First, it will act as an atomic counter in order to allow for multiple instances of the application to run in parallel. Second it will act as a fast cache for responding to frequent link requests.

Nginx

Nginx will act as a boundary reverse-proxy and round-robin requests to any number of Vapor instances.

Docker (docker-compose)

I’m using docker-compose in this example to bundle everything together and ease local development and testing.

Network diagram

The Code

Getting Started

This tutorial assumes you have docker-compose and Vapor installed already.

The first thing to do is create a new directory to store all of our project files in.

Then initialize a new Vapor project. This command should set you up with a new Vapor template that includes code for an example ToDo application.

Since I like to take advantage of XCode’s code completion I’ll also add an xcodeproj file and open our project in XCode.

Scaffolding

First we need to consider how docker is going to be used in this project. Since XCode really only cares about the package files, we’ll use the command line to create a Dockerfile and docker-compose.yml file. Since docker basics are a bit outside the scope of this tutorial I won’t go into too much detail, but everything we’ll use for development is pretty basic. I may look at configuring your application for a production deployment in a later post.

Create a new Dockerfile and docker-compose.yml file by running the following commands.

Paste the following into the Dockerfile. This code will build and launch our Vapor application within a docker container and expose the API on port 8080.

Next add the following to the docker-compose.yml file. There’s a bit more going on here, but at a high level it’s defining each of our images api for the Vapor App, db for the Postgres database, redis for redis, and nginx for nginx. It will also configure some basic authentication for Postgres and create separate networks for db operations and serving the API.

You’ll also need to define your configuration for nginx in order to get the forwarding to work as expected. My conf.d/api.conf file looks like this.

Vapor Web Application

The first bit of Vapor code to look at is our Package.swift file. As I mentioned earlier we’ll want to make sure our project has all the necessary dependencies needed to interact with Redis and Postgres. Replace the existing Package.swift file with the following.

Configuration

Next we can take a look at the configure.swift file to see how we’re using some of the environment variables described in the docker files within the Vapor app. This file is fairly straight-forward and the comments should clear up any ambiguity, so I won’t go into much detail here.

As for the rest of the boilerplate code, in your project there should be an app.swift and a boot.swift file, the defaults that are used here are fine for our purposes.

The Model

Next we will remove the Todo.swift file included in the template and create a new Link.swift file to describe our Link model. The Link model does not contain any unexpected properties. It has an id which is a special field in Fluent as well as a friendlyUrl, which represents a custom defined url, a shortUrl, which represents a shortened Url that will redirect to the originalUrl, a createdAt metadata attribute which vapor can populate automatically, and finally an expiresAt attribute that could be used to clean up old links.

I’ll go into how we plan to query using the short url and the id on this database in a second, but since we’ll also allow users to use a custom “friendly” url, we should add a unique index for that as well. In the App directory of the project create a new directory named Migrations. Within this directory create a new file named migrations.swift and add the following code. You can see above in configure.swift starting on line 56 the code that will actually run this migration.

Routing

Next we’ll look at routes.swift which is where Vapor expects your routes to be defined. Since this service really only has two functions; 1. Creating a redirect (link) and 2. Redirecting a request, our routes.swift file will be pretty straight-forward. Replace the autogenerated routes.swift file with the following.

The Controller

Finally we end up at the controller, the actual business logic that will serve our API. There are a few workflows here so I’ll walk through each one.

Saving a new Link

As you can see in routing.swift above, creating a new Link is accomplished by making a POST request to /links . The first thing it will do when a request is made is ask redis for a new id from its atomic counter. We will then use that numeric id and the rest of the payload supplied in the POST request to create a new instance of Link and save it to Postgres. Assuming the save is successful we will add this link to our redis cache and return the object.

2. Redirects

The second half of this service is redirecting all GET requests to the appropriate url, if one exists. I’ll go over the shortening algorithm this service utilizes in a moment, but generally the service will first check redis to see if it has the redirect cached. Assuming it does not find a match in redis it will pass the request off to lookup() which will attempt to look up the url based on our base62 encoded id, or else based on the user-defined friendly url. Regardless of the lookup method used, if a result is found we will construct a 307 redirect response which most browsers should follow. If a result is not found in redis or Postgres, a 404 error is returned.

Shortening the URLs

Since we’ll primarily be relying on the integer ids for our Link lookup, we need some way of shortening integers into a more compact format. Integers work fine until you get into the 100,000s or millions, and at that point you really may not be saving too much space. This is where Base62 encoding comes into play.

Base62, unsurprisingly, utilizes a 62 character mapping [a-zA-Z0–9]. What this means is that a long number such as 10,000,000 can be represented by a more compact “fxSK”.

I’ve also distinguished auto-generated vs. custom urls by prepending the auto-generated Base62 strings with a “.”, when parsing the provided path during redirect handling, if the path starts with a “.” we know to query on the id index, and if it does not we know to query using the friendlyUrl.

Running the Application

Running the application in development mode (more on production below) should be straight-forward assuming you already have docker-compose set up and have followed the steps above. You can find all my code here.

To start the application just run docker-compose up from your project’s root directory. It may take a few minutes to start up the first time because it needs to download all of Vapor’s dependencies and build the application.

To scale up the number of swift nodes you have running you simply need to add --scale api=Nwhere N is the number of nodes you desire. Docker should handling the routing between Nginx, Redis, and the Swift nodes.

What’s Next?

There are obviously a number of issues with this application that preclude it from being a production ready service. If I were to continue with this example there are a number of things I would add before deploying this application.

Add unit and component test coverage

This deserves a whole post of its own, since we would need to mock out our dependencies.

Production configuration changed

In our example all all of the postgres and redis configuration information is hardcoded into the application and the docker-compose file. For a production ready service we would want to harden this and likely not use some other orchestration for managing our data stores. But docker-compose is great for local development.

Improving resiliency

The app currently has OK error handling, but what do we do if our Postgres store become unreachable or our atomic counter is reset and conflicts with existing records — clearly there are improvements to be made.