Graceful shutdown in Kubernetes

I’ve been doing a research on how to do graceful shutdown of HTTP services in . Surprisingly, I found contradictory opinions on the topic, so I decided to do my own testing. Read on if you are interested in the results. Kubernetes Research Before diving into different shutdown approaches, let’s see what the official Kubernetes documentation says about the topic[2]. Here is an excerpt from the doc (I’ve skipped the points that I consider irrelevant): 1. User sends command to delete Pod, with default grace period (30s)3. Pod shows up as “Terminating” when listed in client commands6. The processes in the Pod are sent the TERM signal.7. (simultaneous with 3), Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly can continue to serve traffic as load balancers (like the service proxy) remove them from their rotations. Reading this, it looks like there is possibility for new requests to come in after a is received (between 6 and 7). This means that if the process just listens for and exits upon reception, some requests will not be served. SIGTERM SIGTERM That was not clear enough, so I started researching. The first post[1] I stumbled upon suggest the following flow: 1. Process receives SIGTERM.2. Process returns 5XX response for (failureThreshold * periodSeconds) to its readiness probe, so it will be unregistered from the list of valid endpoints.3. Process closes its listener, waits for all ongoing requests to finish and exists. This approach suggests that the process should notify that it is exiting, by failing its readiness probe. It seems weird to expect that a process is aware of the underlying infrastructure it is running on. Even more, reading the comments adds more to the confusion. There is one comment that states it is needed to notify that your process is exiting, but indeed there are some requests that will get routed to the container after being sent. Strange, indeed. Let’s keep digging. not SIGTERM I found a Stackoverflow question[3] regarding graceful shutdown, with the following answer. I ran some experiments to find out exactly what happens. The pod will briefly (<1s) continue receiving requests after shutdown is initiated, so you need to either catch SIGTERM or install a preStop hook so you can wait for them (and finish serving current requests). However, once the shutdown has been initiated, the readiness probe no longer matters, you don't need to change its state to stop receiving requests. (But before that a failing readiness probe will lead to your pod receiving no more traffic.) More research just yielded more results that suggest both approaches, making the things even more unclear. At this point I decided I’ll have to do some experiments myself. Experiments The first thing I wanted to check is whether requests are routed to the container after -ing it. I ended up spinning up a local cluster using minikube[6] and deploying test Go program (called ) that counts all received requests after getting . It was exposed via a service and an ingress. SIGTERM kuber SIGTERM I ran a test with (the Apache HTTP server benchmarking ), and while the test was running I deleted the pod which was serving the requests. ab tool ab -n 10000 -c 10 & sleep 3 && kubectl delete pods/kuber-2370255394-v93lk https://192.168.99.100/api/info And this is the output of the test program: $ kubectl logs -f kuber-2263169569-r527k2017/05/07 18:26:47 Serving...2017/05/07 18:27:24 Shutting down due to terminated2017/05/07 18:27:39 Requests served after shutdown signal: 22162017/05/07 18:27:39 Exiting Turns out that it is indeed possible to get requests after receiving shutdown signal. For the 15 seconds wait period before stopping the HTTP server, more than 2000 requests were served. However, I did not find any correlation between the duration that the program waited before actually exiting and the number of requests. Next in my tests was playing with the readiness probes. I enhanced the test program, so it starts returning once the termination signal is received. This did not change the behavior either. 503 Service Unavailable Future work The test above concerts the case where a single pod is deleted using on a single node Kubernetes setup. It is a good idea to test what would happen in a multi-node Kubernetes setup, when one of the nodes is terminated. This may yield different results regarding the time needed to stop routing requests to shutdown pods. kubectl Takeaways Here are some obvious and not so obvious takeaways from my experiments Make sure your process is actually receiving the termination signal. If you use a shell command to start it, e.g. , make sure that the shell you use ( in this example) is forwarding the signal to your program. For example, on Alpine, does not forward signals. /bin/sh -c myapp sh TERM /bin/sh Add some grace period before closing the HTTP server to avoid failing requests. It did not make any difference if the app failed it’s readiness probe after , but having that mechanism in place wouldn’t hurt and may actually be the right thing to do. It may depend on the implementation of the load-balancing layer you’re using. SIGTERM If you’re using Go, you should subscribe for instead of . syscall.SIGTERM os.Interrupt If you have any experience with the topic, please share it in the comments section below, so this confusion is driven away once and for all. Sources [1] https://blog.risingstack.com/graceful-shutdown-node-js-kubernetes/ [2] https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods [3] http://stackoverflow.com/questions/40545581/do-kubernetes-pods-still-receive-requests-after-receiving-sigterm [4] http://stackoverflow.com/questions/31574038/how-can-i-ensure-graceful-scaling-in-kubernetes [5] http://www.bite-code.com/2015/07/27/implementing-graceful-shutdown-for-docker-containers-in-go-part-2/ [6] https://github.com/kubernetes/minikube is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising & sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories