Graceful shutdown in Kubernetes

I’ve been doing a research on how to do graceful shutdown of HTTP services in Kubernetes. Surprisingly, I found contradictory opinions on the topic, so I decided to do my own testing. Read on if you are interested in the results.

Research

Before diving into different shutdown approaches, let’s see what the official Kubernetes documentation says about the topic[2]. Here is an excerpt from the doc (I’ve skipped the points that I consider irrelevant):

1. User sends command to delete Pod, with default grace period (30s)3. Pod shows up as “Terminating” when listed in client commands6. The processes in the Pod are sent the TERM signal.7. (simultaneous with 3), Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly can continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.

Reading this, it looks like there is possibility for new requests to come in after a SIGTERM is received (between 6 and 7). This means that if the process just listens for SIGTERM and exits upon reception, some requests will not be served.

That was not clear enough, so I started researching. The first post[1] I stumbled upon suggest the following flow:

1. Process receives SIGTERM.2. Process returns 5XX response for (failureThreshold * periodSeconds) to its readiness probe, so it will be unregistered from the list of valid endpoints.3. Process closes its listener, waits for all ongoing requests to finish and exists.

This approach suggests that the process should notify that it is exiting, by failing its readiness probe. It seems weird to expect that a process is aware of the underlying infrastructure it is running on. Even more, reading the comments adds more to the confusion. There is one comment that states it is not needed to notify that your process is exiting, but indeed there are some requests that will get routed to the container after SIGTERM being sent. Strange, indeed. Let’s keep digging.

I found a Stackoverflow question[3] regarding graceful shutdown, with the following answer.

I ran some experiments to find out exactly what happens.

The pod will briefly (<1s) continue receiving requests after shutdown is initiated, so you need to either catch SIGTERM or install a preStop hook so you can wait for them (and finish serving current requests).

However, once the shutdown has been initiated, the readiness probe no longer matters, you don't need to change its state to stop receiving requests. (But before that a failing readiness probe will lead to your pod receiving no more traffic.)

More research just yielded more results that suggest both approaches, making the things even more unclear. At this point I decided I’ll have to do some experiments myself.

Experiments

The first thing I wanted to check is whether requests are routed to the container after SIGTERM-ing it. I ended up spinning up a local cluster using minikube[6] and deploying test Go program (called kuber) that counts all received requests after getting SIGTERM. It was exposed via a service and an ingress.

I ran a test with ab (the Apache HTTP server benchmarking tool), and while the test was running I deleted the pod which was serving the requests.

ab -n 10000 -c 10 https://192.168.99.100/api/info & sleep 3 && kubectl delete pods/kuber-2370255394-v93lk

And this is the output of the test program:

$ kubectl logs -f kuber-2263169569-r527k2017/05/07 18:26:47 Serving...2017/05/07 18:27:24 Shutting down due to terminated2017/05/07 18:27:39 Requests served after shutdown signal: 22162017/05/07 18:27:39 Exiting

Turns out that it is indeed possible to get requests after receiving shutdown signal. For the 15 seconds wait period before stopping the HTTP server, more than 2000 requests were served.

However, I did not find any correlation between the duration that the program waited before actually exiting and the number of requests. Next in my tests was playing with the readiness probes. I enhanced the test program, so it starts returning 503 Service Unavailable once the termination signal is received. This did not change the behavior either.

Future work

The test above concerts the case where a single pod is deleted using kubectl on a single node Kubernetes setup. It is a good idea to test what would happen in a multi-node Kubernetes setup, when one of the nodes is terminated. This may yield different results regarding the time needed to stop routing requests to shutdown pods.

Takeaways

Here are some obvious and not so obvious takeaways from my experiments

• Make sure your process is actually receiving the termination signal. If you use a shell command to start it, e.g. /bin/sh -c myapp, make sure that the shell you use (sh in this example) is forwarding the TERM signal to your program. For example, on Alpine, /bin/sh does not forward signals.
• Add some grace period before closing the HTTP server to avoid failing requests.
• It did not make any difference if the app failed it’s readiness probe after SIGTERM, but having that mechanism in place wouldn’t hurt and may actually be the right thing to do. It may depend on the implementation of the load-balancing layer you’re using.
• If you’re using Go, you should subscribe for syscall.SIGTERM instead of os.Interrupt.

If you have any experience with the topic, please share it in the comments section below, so this confusion is driven away once and for all.

Sources

[1] https://blog.risingstack.com/graceful-shutdown-node-js-kubernetes/

[2] https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods

[3] http://stackoverflow.com/questions/40545581/do-kubernetes-pods-still-receive-requests-after-receiving-sigterm

[4] http://stackoverflow.com/questions/31574038/how-can-i-ensure-graceful-scaling-in-kubernetes

[5] http://www.bite-code.com/2015/07/27/implementing-graceful-shutdown-for-docker-containers-in-go-part-2/

[6] https://github.com/kubernetes/minikube

Hacker Noon is how hackers start their afternoons. We’re a part of the @AMIfamily. We are now accepting submissions and happy to discuss advertising & sponsorship opportunities.

To learn more, read our about page, like/message us on Facebook, or simply, tweet/DM @HackerNoon.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!