At Debricked we have been using Symfony for our web backend for a while now. It has served us very well and when the Messenger component back in Symfony 4.1, we were eager to try it out. Since then we have used the component for asynchronous queuing emails. they announced Lately the need arose to separate our handling of GitHub events, which we receive from our , from the web backend to a separate microservice for performance reasons. We decided to utilise the which the Messenger component provides as it lets us asynchronously dispatch the various events to a queue and then immediately acknowledge the event to GitHub. GitHub integration producer/consumer pattern However, compared to sending emails, handling some GitHub events can be time consuming. We also have no control of when these events will happen, so the load is both unpredictable and irregular - we needed auto scaling of our consumers. Kubernetes Autoscaling to the rescue As we already had most of our infrastructure deployed to Kubernetes on Google Cloud, it made sense to try to utilise that for our consumers. Kubernetes offers something called , which lets you automatically scale your pods up and down depending on some metric. Horizontal Pod Autoscaler The autoscaler has one metric built in, the CPU metric. It lets us set a target CPU load across our pods and then Kubernetes will automatically adjust the amount of pods to match the target. We will use this metric to make sure we always have a reasonable amount of pods, running our consumers, available. Preparing Docker image running consumers Concluding that Kubernetes could help us out we now need a suitable Docker image for our pods to run. We base our consumer image on our base image, which is based on Debian and contains our backend logic, including the logic for the GitHub event consumer/message handler. Symfony a tool called “Supervisor” for controlling the execution of the consumer, hence we add this to our image and starts it in Docker’s CMD directive, see example code below: recommends FROM your-registry.com: /your_base_image:latest

USER root

WORKDIR /app

RUN apt update && apt install -y supervisor

# Cleanup
RUN rm -rf / /lib/apt/lists 5555 var /* && apt clean

COPY ./pre_stop.sh /pre_stop.sh
COPY ./supervisord_githubeventconsumer.conf /etc/supervisord.conf
COPY ./supervisord_githubeventconsumer.sh /supervisord_githubeventconsumer.sh

CMD supervisord -c /etc/supervisord.conf Config file If you look carefully we also add two files which are related to running Supervisor(d). The files looks like the following: [supervisord]
nodaemon= user=root

[program:consume-github-events]
command=bash /supervisord_githubeventconsumer.sh
directory= dev/stdout
stdout_logfile_maxbytes= stderr_logfile= stderr
stderr_logfile_maxbytes= true /app
autostart=true
# Restart on unexpected exit codes
autorestart=unexpected
# Expect 37 exit code, returned when stop file exists
exitcodes=37
startretries=99999
startsecs=0
# Your user
user=www-data
killasgroup=true
stopasgroup=true
# Number of consumers to run. We use a high number because we are bound by network IO
numprocs=70
process_name=%(program_name)s_%(process_num)02d
stdout_logfile=/ 0 /dev/ 0 Bash script [ -f ]; then
  rm -rf /tmp/debricked-stop-work.txt
  exit php bin/ messenger:consume -m --time-limit= --memory-limit= M githubevents --env=prod
fi if "/tmp/debricked-stop-work.txt" 37 else console 100 3600 150 It is a fairly standard Supervisor config with a few items worthy of note. We are executing a bash script, which in turn either exits with code 37, more on that in next section, or executes the Messenger component’s consume command using our GitHub events consumer. We also configure Supervisor to automatically restart on unexpected failures, which is any status code not being 37. In our case we will run a high number of consumers concurrently, 70, due to the load being very network IO bound. By running 70 consumers concurrently we are able to fully saturate our CPU. This is required for the Horizontal Pod Autoscaler’s CPU metric to work correctly, as the load would otherwise be too low - making the scaling being stuck at the minimum configured replicas, no matter how long the queue is. Gracefully scale down pods/consumers When the autoscaler decides the load is too high it starts new pods. Thanks to the asynchronous nature of the messenger component we do not need to worry about concurrency issues such as race conditions. It will just work out of the box with multiple consumers, hence the increased pods/consumers count won’t cause any issues, but what happens when the load is too low and the scaler decides to scale down the instance? By default the autoscaler will just abruptly terminate a running pod if it decides it is not needed anymore. This causes a problem with the consumer as it might be in the middle of handling a message. We need a way of gracefully shutting down the pod, consuming the message we are currently dealing with and then exit. In the previous section in the Dockerfile you might have noticed that we copied a file called ` ` to our image. That file looks as the following: pre_stop.sh # Script to execute when pod is about to be terminated

touch /tmp/debricked-stop-work.txt
chown www-data:www-data /tmp/debricked-stop-work.txt
# Tell workers to stop
php bin/ messenger:stop-workers --env=prod
# Wait until file has been deleted
until [ ! -f /tmp/debricked-stop-work.txt ] echo sleep done

echo console do "Stop file still exists" 5 "Stop file found, exiting" When executed, this bash script will create a ` ` file. Because the script also calls ` `, it will gracefully stop the current workers/consumers, causing the ` ` to be restarted by Supervisord. When the script it restarted it will now immediately exit with status code 37 because the ` ` exists. In turn causing Supervisor to exit, because 37 is our expected exit code. /tmp/debricked-stop-work.txt php /app/bin/console messenger:stop-workers supervisord_githubeventconsumer.sh /tmp/debricked-stop-work.txt As soon as Supervisor exits, so will our Docker image as Supervisor is our CMD and the ` ` will also terminate because ` ` deletes the ` ` before exiting with 37. We have achieved a graceful shutdown! pre_script.sh supervisord_githubeventconsumer.sh /tmp/debricked-stop-work.txt But when is ` ` executed? You might be asking yourself. We will execute it on Kubernetes container’s `PreStop` . pre_script.sh lifecycle event This event occurs whenever a container is about to be terminated, such as when being terminated by the auto scaler. It is a blocking event, meaning that the container will not be deleted before this script is complete, just what we want. To configure the lifecycle event we just need to a few lines of code to our deployment config, like in the example below: ---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
	name: gheventconsumer
	namespace: labels:
    	app: gheventconsumer
    	tier: backend
spec:
	replicas: selector:
    		matchLabels:
        			app: gheventconsumer
	template:
    	metadata:
        	labels:
            	app: gheventconsumer
    	spec:
        	terminationGracePeriodSeconds: # Consuming might be slow, allow minutes graceful shutdown
        	containers:
            	- name: gheventconsumer
              	  image: your-registry.com: /your_base_image:latest
              	  imagePullPolicy: Always
              	  Lifecycle: # ← ’s shut down gracefully
                  	  preStop:
                      		  exec:
                          		  command: [ , ]
              	  resources:
                  	  requests:
                      	  cpu: m
                      	  memory: Mi
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
	name: gheventconsumer-hpa
	namespace: labels:
    	app: gheventconsumer
    	tier: backend
spec:
	scaleTargetRef:
    	kind: Deployment
    	name: gheventconsumer
    	apiVersion: apps/v1
	minReplicas: maxReplicas: metrics:
    	- type: Resource
      	  resource:
          	  name: cpu
          	  targetAverageUtilization: default 1 240 for 4 5555 this let "bash" "/pre_stop.sh" 0.490 6500 default 1 5 60 Overwhelmed? Here is an image of the shutdown flow: Conclusion In this article we have learnt how to dynamically scale Symfony Messenger consumers depending on load, including gracefully shutting them down. Resulting in high throughput of messages in a cost effective manner.

Flow

Google

Target

Using Workload Identity to Handle Keys in Google Kubernetes Engine

https://debricked.com

Too Long; Didn't Read

Make resilience your competitive advantage

Scaling Symfony Consumers using Kubernetes [A How To Guide]

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

How and Why We Choose to Clone all Data on Github

20,000 Leagues Under Your Shell

47 Stories To Learn About Bash

A Basic Guide to MySQL Tests Automation

A Web Service Written in Pure Bash.

How and Why We Choose to Clone all Data on Github

20,000 Leagues Under Your Shell

47 Stories To Learn About Bash

A Basic Guide to MySQL Tests Automation

A Web Service Written in Pure Bash.

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps