Handling an Ungraceful Service in Kubernetes by Bash

by Kaveh Mousavi ZamaniSeptember 21st, 2019

Too Long; Didn't Read

Kubernetes has several mechanisms to check if your app is able to handle connections and in return if it wants to kill you app (scale down, or end of life for your preemptible/spot-instance node,… it will signal your app to wrap up and shutdown. If your server does not handle shutdowns gracefully or it is not adjustable in ways you need to make it work in k8s (shutdown delay, …), notice you can always wrap it with bash.

featured image - Handling an Ungraceful Service in Kubernetes by Bash

Kubernetes has several mechanisms to check if your app is able to handle connections and in return if Kubernetes wants to kill you app (scale down, or end of life for your preemptible/spot-instance node,…) it will signal your app to wrap up and shutdown.

If your server does not handle shutdowns gracefully or it is not adjustable in ways you need to make it work in k8s (shutdown delay, …) , notice you can always wrap it with bash.

Assume a deployment like

apiVersion: extensions/v1beta1
kind: Deployment
...
      containers:
      - args:
        - /bin/bash
        - -c
        - tensorflow_serving
        image: tensorflow/serving:1
        name: tfserving
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
...

You can use your bash-wrapper and run the server there. Your bash script can handle the termination signal and do what you want.

In this case you just need to instruct Kubernetes to use a file for readinessProbe or livenessProbe.

apiVersion: extensions/v1beta1
kind: Deployment
...
      containers:
      - args:
        - /bin/bash
        - -c
        - /init.sh tensorflow_serving
        image: tensorflow/serving:1
        name: tfserving
        readinessProbe:
          exec:
            command:
            - cat
            - /tmp/ready
...

And a sample of an init.sh can look like:

#!/bin/bash
set -e

COMMAND="$@"
INIT_NAME=${INIT_NAME:-myapp}
HEALTH_PORT=${HEALTH_PORT:-8080}
HEALTH_PATH=${HEALTH_PATH:-ping}
HEALTH_PERIOD_SECONDS=${HEALTH_PERIOD_SECONDS:-1}
HEALTHY_INDICATOR=${HEALTHY_INDICATOR:-/tmp/ready}
APP_STOP_DELAY=${APP_STOP_DELAY:-5}
SIGNAL_RECIEVED_INDICATOR=$(mktemp --dry-run /tmp/$INIT_NAME-teminating.XXXXXX)

_terminating() {
  touch ${SIGNAL_RECIEVED_INDICATOR}
  rm  -f ${HEALTHY_INDICATOR}
}

_term() {
  ls /tmp
  echo "Caught SIGTERM signal!"
  _terminating
  sleep ${APP_STOP_DELAY}
  echo "stopping the child ${child}"
  kill -9 ${child}
}

_health_check() {
  while [ ! -f ${SIGNAL_RECIEVED_INDICATOR} ]
  do
    # either you check the port (nc version) or you are checking the http call (curl version)
    # if ! nc -vz locahost ${HEALTH_PORT} >/dev/null 2>&1
    if ! curl --fail --silent http://localhost:${HEALTH_PORT}/${HEALTH_PATH} > /dev/null
    then
      rm  -f ${HEALTHY_INDICATOR}
    else
      touch rm  -f ${HEALTHY_INDICATOR}
    fi
    sleep ${HEALTH_PERIOD_SECONDS}
  done
}

trap _term SIGTERM
${COMMAND} &
child=$!

_health_check &
wait "${child}"
_terminating