paint-brush
Golang: Using systemd for Zero Downtime Restarts and Deploys by@vmihailenco
4,010 reads
4,010 reads

Golang: Using systemd for Zero Downtime Restarts and Deploys

by Vladimir MihailencoDecember 21st, 2022
Read on Terminal Reader
Read this story w/o Javascript
tldt arrow

Too Long; Didn't Read

During restarts, your API may be unavailable for short period of time required to stop an old app instance and start a new one. You can fix that by introducing another app (proxy) that listens on behalf of the main app and somehow proxies the data. A popular way is to pass the listening socket as a file descriptor to a child process and then re-create the socket there.
featured image - Golang: Using systemd for Zero Downtime Restarts and Deploys
Vladimir Mihailenco HackerNoon profile picture

During restarts, your API may be unavailable for a short period of time to stop an old app instance and start a new one. You can fix that by introducing another app (proxy) that listens on behalf of the main app and somehow proxies the data to the main app.


A popular way to do that is to pass the listening socket as a file descriptor to a child process and then re-create the socket there using the descriptor. Systemd can do that for you via the feature called socket activation.


By configuring a systemd socket, you can tell systemd to listen on the configured ports and start your service with a copy of the listening sockets. During restarts, the data is not lost but stored in OS buffers.

Systemd Socket

You can configure a systemd socket by creating /lib/systemd/system/myapp.socket file, for example, the following socket listens on the port 80, but you can add as many ports as you need:


[Socket]
ListenStream = 80
#ListenStream = 443
BindIPv6Only = both
Service      = myapp.service

[Install]
WantedBy = sockets.target


You also need to configure an accompanying systemd service by creating myapp.service file:


[Unit]
Description = myapp
After       = network.target

[Service]
Type = simple

ExecStart = /bin/myapp
ExecStop  = /bin/kill $MAINPID
KillMode  = none

[Install]
WantedBy = multi-user.target

Using Systemd Sockets

The Go application should be modified to use the systemd socket instead of directly listening on the port. Let's install a Go module that will help us with that:


go get github.com/coreos/go-systemd/v22


And then use it like this:


import "github.com/coreos/go-systemd/v22/activation"

func main() {
	listeners, err := activation.Listeners()
	if err != nil {
		return err
	}

	httpLn := listeners[0]
	//httpsLn := listeners[1]

	httpServer := &http.Server{
		ReadHeaderTimeout: 5 * time.Second,
		ReadTimeout:	   5 * time.Second,
		WriteTimeout:	   5 * time.Second,
		IdleTimeout:	   60 * time.Second,
		Handler:		   router,
	}
	if err := httpServer.Serve(httpLn); err != nil {
		log.Printf("Serve failed: %s", err)
	}
}


As a Golang router, you can try bunrouter.

Restarting the App

First, you need to start the configured systemd socket:


sudo systemctl start myapp.socket


And then check that the socket was created successfully:


sudo systemctl status myapp.socket


Systemd will automatically start the service on the first request or if the service crashes. You can also manually restart the service during deploys:


sudo systemctl restart myapp.service

Uptrace

Uptrace is an open-source DataDog competitor with an intuitive query builder, rich dashboards, alerting rules, and integrations for most languages and frameworks. It can process billions of spans and metrics on a single server and allows you to monitor your applications at a 10x lower cost.


Uptrace uses the ClickHouse database to store traces, metrics, and logs. You can use it to monitor applications and set up automatic alerts to receive notifications via email, Slack, Telegram, and more.


You can get started with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.


Also published here