Golang: Using systemd for Zero Downtime Restarts and Deploys

During restarts, your API may be unavailable for a short period of time to stop an old app instance and start a new one. You can fix that by introducing another app (proxy) that listens on behalf of the main app and somehow proxies the data to the main app.

A popular way to do that is to pass the listening socket as a file descriptor to a child process and then re-create the socket there using the descriptor. Systemd can do that for you via the feature called socket activation.

By configuring a systemd socket, you can tell systemd to listen on the configured ports and start your service with a copy of the listening sockets. During restarts, the data is not lost but stored in OS buffers.

Systemd Socket

You can configure a systemd socket by creating /lib/systemd/system/myapp.socket file, for example, the following socket listens on the port 80, but you can add as many ports as you need:

[Socket]
ListenStream = 80
#ListenStream = 443
BindIPv6Only = both
Service      = myapp.service

[Install]
WantedBy = sockets.target

You also need to configure an accompanying systemd service by creating myapp.service file:

[Unit]
Description = myapp
After       = network.target

[Service]
Type = simple

ExecStart = /bin/myapp
ExecStop  = /bin/kill $MAINPID
KillMode  = none

[Install]
WantedBy = multi-user.target

Using Systemd Sockets

The Go application should be modified to use the systemd socket instead of directly listening on the port. Let's install a Go module that will help us with that:

go get github.com/coreos/go-systemd/v22

And then use it like this:

import "github.com/coreos/go-systemd/v22/activation"

func main() {
	listeners, err := activation.Listeners()
	if err != nil {
		return err
	}

	httpLn := listeners[0]
	//httpsLn := listeners[1]

	httpServer := &http.Server{
		ReadHeaderTimeout: 5 * time.Second,
		ReadTimeout:	   5 * time.Second,
		WriteTimeout:	   5 * time.Second,
		IdleTimeout:	   60 * time.Second,
		Handler:		   router,
	}
	if err := httpServer.Serve(httpLn); err != nil {
		log.Printf("Serve failed: %s", err)
	}
}

As a Golang router, you can try bunrouter.

Restarting the App

First, you need to start the configured systemd socket:

sudo systemctl start myapp.socket

And then check that the socket was created successfully:

sudo systemctl status myapp.socket

Systemd will automatically start the service on the first request or if the service crashes. You can also manually restart the service during deploys:

sudo systemctl restart myapp.service

Uptrace

Uptrace is an open-source DataDog competitor with an intuitive query builder, rich dashboards, alerting rules, and integrations for most languages and frameworks. It can process billions of spans and metrics on a single server and allows you to monitor your applications at a 10x lower cost.

Uptrace uses the ClickHouse database to store traces, metrics, and logs. You can use it to monitor applications and set up automatic alerts to receive notifications via email, Slack, Telegram, and more.

You can get started with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.

Also published here