Cheapskate's Journey to On-Demand Load Tests with Locust

Published with permission by Rahul Rai I want to stretch every dollar that I spend on the cloud. I run a handful of web applications on Heroku, and like everyone else, run a suite of smoke tests and load tests on every release increment in a non-production environment. Load tests are important: they help us not only to understand the limits of our systems but also bring up issues that arise due to concurrency, which often escape the realms of unit tests and integration tests. But since we run the tests often, we don’t want to pay a lot of money every time the tests run. In this article, I’ll show you how to set up cost-effective load tests. We’ll use Locust to make the testing robust, and to make running the tests easy and cost-effective. I’ll also show how you can use VS Code and Docker for development without installing dev dependencies on your system. Heroku What is Locust? is an open-source load testing tool written in Python. Locust tests can be distributed over multiple machines to simulate millions of users simultaneously, helping to determine just how many users your site or system can handle. Locust Locust was created to address issues that exist with two other leading solutions — and . Specifically, it was built to address the following limitations: JMeter Tsung Concurrency: JMeter is thread bound, creating a new thread for every user. This severely limits the number of users that can be simulated per machine. Locust, on the other hand, is event-based and can simulate thousands of users on one process. Ease of Coding: JMeter requires complicated callbacks. Tsung uses an XML-based DSL to define user behavior. Both are difficult to code. Locust scenarios, on the other hand, are written in plain Python and are easy to code. Terminology First, a little terminology. With Locust, you write user behavior tests in a set of locustfiles, and then execute the locustfiles concurrently on the target application. In terms of Locust, a collection of locust users (collectively called a Swarm, and individually called a Locust) will attack the target application and record the results. Each locust executes inside its sandboxed process called Greenlet. Considerations Before proceeding further, I recommend that you read the on load tests, which lists the restrictions that apply and the consequences. The guidance in this article is limited to executing low to medium level tests (less than 10,000 requests per second). For executing high-scale tests, you should either contact Heroku support first to ensure your systems are pre-warmed and will scale appropriately, or use Private Spaces to host your testbed (application under test and the test platform). For high-volume load tests, I recommend modeling your test setup on . For the latest pricing details and to estimate the cost of running your applications on Heroku, refer to the . guidance from Heroku this sample application repository Heroku website Prerequisites Here is the list of tools and cloud services that I used to build the sample application. My development machine runs Windows 10 Professional, however, the following tools are available on Mac as well. VS Code with extension Remote Development in which you can create apps on the standard tier A Heroku account A free Microsoft Azure subscription Docker Desktop for Windows (or Mac) Heroku CLI Azcopy The Applications The sample application that I have prepared for this demo, which we will refer to as , is a REST API written in Go. We also have a second application, which we will refer to as , that contains the load tests written in Python using . the Target API application Loadtest application Locust The is the REST API that we intend to test. Since the API is required to process HTTP requests, we host it on . Target API application web dynos The contains our . These are split into two categories based on the type of users supported by the . You can execute the two test suites in parallel or in sequence, thus varying the amount and nature of load that you apply on the . Since the dynos executing the tests are required only for the duration of test executions, we host them in Heroku’s . The one-off dynos are billed only for the time and resources that they consume, and an Administrator can spawn them using the tool. Loadtest application Locust tests Target API application Target API application one-off dynos Heroku CLI The following is the high-level design diagram of the applications and their components. High Level Design Diagram Heroku provides ephemeral storage to the application processes executing on the dyno, which may or may not exist. Also, because the storage is local to the process, we cannot access any files generated by the Heroku CLI since it creates another sandboxed process with its own storage on the dyno. Due to access restrictions, the process that generates the files will export them to a durable cloud storage service, or in the case of web dynos, make them available through an HTTP endpoint. By executing Locust with a flag (–csv), you can instruct locust to persist test results in CSV files locally. We use , which is a CLI tool used for copying binary data into and out of Azure storage to export the results generated by the Locust tests to an Azure blob storage. Azcopy Setting Up the Applications The source code of the applications is available in my . GitHub repository Target API Application Let’s first dissect the Target API application, which we want to test with our load test suite. Open the folder named in VS Code. In the file , I have defined three API endpoints: api main.go http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Println("Served home request") fmt.Fprintf(w, "Server OK") }) http.HandleFunc("/volatile", func(w http.ResponseWriter, r *http.Request) { // For every 10 requests, delay the response by 1 second, up to 5 seconds. currentCount := atomic.LoadInt32(&requestCount) atomic.AddInt32(&requestCount, 1) delay := currentCount / 10 if delay > 5 { atomic.StoreInt32(&requestCount, 0) delay = 5 } time.Sleep(time.Duration(delay) * time.Second) fmt.Fprintf(w, "Produced response after %d second/s", delay) fmt.Printf("Produced response after %d second/s \n", delay) }) http.HandleFunc("/buggy", func(w http.ResponseWriter, r *http.Request) { // After every 5 requests, throw error currentCount := atomic.LoadInt32(&requestCount) atomic.AddInt32(&requestCount, 1) if currentCount%5 == 0 { fmt.Printf("Returning error at %d request \n", currentCount) http.Error(w, http.StatusText(500), 500) return } fmt.Fprintf(w, "Ok for request %d", currentCount) }) The behavior of the three endpoints is as follows: “/“: Returns an HTTP 200 response with text OK. “/ “: Returns HTTP 200 response but successively delays the response by one second for every 10 requests. volatile “/ “: Returns an HTTP 500 fault message for every fifth request. buggy Remote Development Extension for Debugging You probably noticed that I did not mention installing Golang or Python as a prerequisite for this application. We will use the extension that you installed to VS Code to debug the Target API application. You can read about this extension in . However, in a nutshell, this extension allows you to use a container as your development environment. The extension searches for a folder named at the root and uses the (the container definition) and (for container settings) files to create a new container and mount the folder containing your code as a volume to the container. For debugging, the extension attaches the VS Code debugger to the process running in the container. I have already configured the container resources for you, so you just need to press the F1 key to bring up the command window and select the command: . Remote Development detail here .devcontainer Dockerfile devcontainer.json Remote-Containers: Open folder in container Open Folder In Container When asked which folder to open, select the ‘api’ folder and continue. Alternatively, you can spawn the command dialog by clicking on the green icon in the bottom left of the VS Code window. Once the container is ready, press F5 to start debugging the application. You will notice that the text in the bottom left corner of the VS Code window changes to : Go to denote that the application is currently executing in a remote container. You can now access the application endpoints from your browser by navigating to . Dev Container http://localhost:9000 Executing Application in A Remote Container Loadtest Application Now we are going to use VS Code to build the test suite inside a container and create a shell script that automates the process of setup and tear down of the test infrastructure. You can use this script to automate the spin up and tear down of the test grid and add it to your CI\CD pipeline. 1. Launch Loadtest Application Dev Container In another VS Code instance, open the folder loadtest and launch it in a dev container as well. In this application, you will notice that I created two sets of tests to model the behavior of two user types of the Target API application. Locustfiles for Test The user behavior of type is recorded in locustfile_scene_1.py. According to the test, a user of type APIUser accesses the default and the volatile endpoints of the Target API application after waiting for five to nine seconds between invocations. ApiUser The user behavior of type is recorded in locustfile_scene_2.py. This category of user accesses the default and the buggy endpoints of the Target API application after waiting for five to 15 seconds between invocations. AdminUser 2. Verify the Tests To verify the test scripts, execute the following command in the integrated terminal ( ). Ctrl + ~ $ locust -f locustfile_scene_1.py Navigate to to bring up the locust UI. In the form, enter the hostname and port of the Target API application along with the desired locust swarm configurations, and click the button to initiate the tests. http://localhost:8089 Start Swarming Locust UI 3. The Run Shell Script For executing the locust tests, we need to define a small workflow for each set of tests as follows. Execute the test without the web UI on a single worker node for a fixed duration and generate CSV reports of the test results. Use Azcopy to copy the test result files to Azure storage. (Of course, you can substitute this part for any cloud storage provider you may use. You would simply need to modify the following script to use a different utility instead of azcopy, and you would be copying to a different storage location.) The script in the load test project implements this workflow as follows: run.sh locust -f --headless -u 200 -r 10 --host= --csv= --run-time 1h -t 2s --stop-timeout 60 filename *.csv; [ -e ] || azcopy copy 0 #!/bin/bash $1 $TARGET_HOST " " $2_ $(date +%F_%T) for in do " " $filename continue " " $filename "https://locustloadtest.blob.core.windows.net/testresult/ \$SAS_TOKEN" $filename done exit In the previous code listing, after executing the command, which produces CSV results, we loop through the CSV files and use the Azcopy utility to upload each file to an Azure storage location — a container named in the account. You must change these values with the storage account that you created in your Azure subscription. You can see that this command relies on a Shared Access Secret (SAS) token for authentication, which we applied through an environment variable named SAS_TOKEN. We will add this environment variable to the application later. If you are not familiar with the Azcopy utility, please read more about using . locust testresult locustloadtest.blob.core.windows.net Azcopy with SAS tokens here Start the Target API Application and Create the Web Dyno Inside the root directory of each project, API and Loadtest, you will find a file named . Procfile In the API Procfile, the following command will instruct Heroku to create a web dyno and invoke the command to launch the application. locust-loadtest locust-loadtest web: In the Loadtest project, the Procfile for the Locust tests instructs Heroku to create two worker dynos and invoke Run.sh script with appropriate parameters as follows: worker_scene_1: bash ./run locustfile_scene_1 scene_1 worker_scene_2: bash ./run locustfile_scene_2 scene_2 .sh .py .sh .py Creating Applications in Heroku We will now create the two required applications in Heroku. There are two ways in which you can interact with Heroku: the user interface and the Heroku CLI. I will guide you through a mix of both approaches so that you get some experience with both. For creating the applications, we will use the Heroku user interface. We will create the Target API application first. Create Target API Application In your browser, navigate to and click on the button. https://dashboard.heroku.com/ New/Create new app Create a New Heroku App On the create app page, enter the name of the application (locust-heroku-target), choose the option, and the desired region. Note that the application name must be unique across all Heroku apps, and so this name be available. You can choose your own unique name for this application (and the test engine application lower down), making sure to reference these new names in all subsequent code and commands. If your customers are present in multiple geographies, you can create an additional test bed in a different location and test the performance of your application from that location as well. Click the button to create the application. Common Runtime may not Create app Create locust-heroku-target The next screen asks you to specify the deployment method. Since I am already using GitHub for source control, I can instruct Heroku to automatically deploy whenever I make changes to the branch. I recommend you don’t follow the same scheme for real-life applications. You should deploy to production from the branch and use another branch such as the release branch to deploy to test environments (Git flow) or from the master branch after approvals (GitHub flow). master master Link App to GitHub — locust-heroku-target Create Loadtest Application Now let’s set up the Loadtest application for our Locust tests. You can create another app (locust-heroku-testengine) for the test, like this: Create locust-heroku-testengine You may have noticed that I used the model to keep the Target API application and tests together in the same project. monorepo On the next screen, connect the deployment of the application you just created to the same repository. With this setup, whenever you make changes to either the Loadtest or the Target API application, both will be deployed to Heroku, which helps to avoid any conflicts between the versions of the Loadtest and the Target API application. Link App to GitHub — locust-heroku-testengine By default, the worker dynos of this application will use Standard-1x dynos, which are a great balance of cost and performance for our scenario. However, you can change the dyno type based on your requirements with the Heroku CLI or through the UI. Refer to the for the CLI command and types of dynos that you can use. Heroku documentation Adding Buildpacks via Heroku CLI Now let’s switch to the terminal and prepare the environment using the Heroku CLI. We’ll go through the buildpacks that our services need and add them one at a time. How the Buildpacks Work Heroku are responsible for transforming your code into a “slug.” In Heroku terms, a slug is a deployable copy of your application. Not every buildpack must generate binaries from your application code — buildpacks can be linked together such that each buildpack transforms the application code in some manner and feeds it to the next buildpack in the chain. However, after processing, the dyno manager must receive a slug as an output. buildpacks For example, since our source code is organized as a monorepo consisting of the Target API application and Loadtest application, the first buildpack in the buildpack chain, , extracts an application from the monorepo. The second buildpack in the chain builds the appropriate application. heroku-buildpack-monorepo Target API Buildpacks Let us consider the Target API application first. Use to extract the locust-heroku-target application from the monorepo. The next buildpack, , builds the Target API project. heroku-buildpack-monorepo heroku-buildpack-go Execute the following commands in the exact sequence to preserve their order of execution, and remember to change the name of the application in the command to what you specified in the Heroku User Interface earlier. $ heroku buildpack - locust-heroku-target http //github. /lstoll/heroku-buildpack-monorepo $ heroku buildpack - locust-heroku-target http //github. /heroku/heroku-buildpack- s:add a s: com s:add a s: com go Loadtest Buildpacks For the project, we need two buildpacks. The first buildpack is the one we used previously, . We will modify the parameter though, so it will extract the Locust test project ( ) from the monorepo. The second buildpack, , enables executing Python scripts on Heroku. locust-heroku-testengine heroku-buildpack-monorepo locust-heroku-testengine heroku-buildpack-python $ heroku buildpacks:add -a locust-heroku-testengine https://github.com/lstoll/heroku-buildpack-monorepo $ heroku buildpacks:add -a locust-heroku-testengine https://github.com/heroku/heroku-buildpack-python Configuring Environment Variables Via Heroku CLI Our applications require setting a few environment variables. Execute the following commands to add the environment variables to your applications. heroku set -a locust-heroku-target APP_BASE=api heroku set -a locust-heroku-target GOVERSION=go1. heroku set -a locust-heroku-testengine APP_BASE=loadtest heroku set -a locust-heroku-testengine PATH= heroku set -a locust-heroku-testengine SAS_TOKEN= heroku set -a locust-heroku-testengine TARGET_HOST= / $ config: $ config: 13 $ config: $ config: /usr/local /sbin:/usr /local/bin :/usr/sbin :/usr/bin :/sbin :/bin :/app/bin $ config: "?sv=2019-10-10&ss=bfqt&srt=sco&sp=rwdlacupx&se=2025-05-16T11:41:28Z&st=2020-05-15T03:41:28Z&spr=https&sig=\ " $ config: https: /locust-heroku-target.herokuapp.com/ Via Heroku User Interface As I mentioned above in this article, you can configure the applications through the user interface as well. You can find the settings that we applied under the tab as shown in the following screenshot of the section from the application. Settings locust-heroku-target Settings — locust-heroku-target Similarly, the following screenshot illustrates the settings that we applied to the locust-heroku-testengine application. Settings — locust-heroku-testengine Deploy the Applications Because of the existing GitHub integration, Heroku deploys our application whenever any changes are pushed to the branch. Push your application or changes to GitHub and wait for the build to complete. You can view the logs of the build under the tab of your application. master Activity Target API application After deployment, you can navigate to the tab and view the dyno hosting the application. You can scale out the dyno from this UI. Click on the button to launch the application. Resources Open app Open App — Locust Heroku Target Loadtest application If you navigate to the app, you will find that Heroku created two worker dynos by reading the instructions from the Loadtest project’s Procfile. locust-heroku-testengine Worker Dynos of locust-heroku-testengine Execute Tests To execute the tests hosted in the dynos, we kick them off using the Heroku CLI with the following commands. These start the one-off dynos, which then terminate right after they finish execution. $ heroku $ heroku run worker_scene_1 --app locust-heroku-testengine run worker_scene_2 --app locust-heroku-testengine After execution, the Azcopy utility copies the CSV files containing the test results to Azure storage, which you can extract using . The following image illustrates this process in action. Azure Storage Explorer Execute Load Tests You can use a custom visualizer or open the CSV files in Excel to read the test results. The following image presents part of a result that I received from the execution of worker_scene_2 dyno that executes the test present in the locustfile_scene_2.py file. Load Test Results The Results Let’s analyze the results to see how well our application is working. Every test run produces three files: The file lists the total number of failures encountered. In scenario 2 results, my run produced 28 errors from the endpoint, which were expected as this is how we programmed it. failures.csv GET /buggy The file lists the endpoints to which the tests send requests and the response time in milliseconds. My run for scenario 2 shows that the swarm sent 29 and 28 requests to the and endpoints respectively. On average, the locusts received a response from the two endpoints in 149 ms and 78 ms respectively. The percentile splits of average response time are the most valuable pieces of information generated by the load tests. From my test run I can see that 99% of the users of my API will receive a response from the and endpoints in 430 ms and 270 ms respectively. stats.csv GET / GET /buggy GET / GET /buggy The third file, , is similar to the file but gets a new row for every 10 seconds of the test run. By inspecting the results of this file, you can find whether your API response time is deteriorating as time passes. history.csv stats.csv Let’s also look at how much it costs to execute these tests. I hosted the tests on two Standard-1X dynos, which cost $25 each month. Therefore, if I were to let the tests execute continuously for a month, it would cost $50. Since my individual test runs lasted only two minutes, and Heroku charges for the processing time by the second, my incurred charges were so minuscule that they did not even show up on my dashboard. That’s great, but let’s approximate the charges that testing a real-life application might incur. Let’s say on average an API requires around 10 suites of tests, and hence 10 dynos. If these tests run every night and each run lasts for five minutes, each dyno will remain active for one dyno x 300 seconds x 30 days = 9,000 seconds; hence, each dyno will cost $0.086 each month. The total cost of running 10 load-test dynos (one-off dynos) for an entire month will be around $0.87. Conclusion You are now ready to execute load tests on Heroku using Locust. You’ll be able to test the stability and performance of every deployment. Since one-off dynos are charged only for the time and resources that they consume, you’ll get maximum value from every cent that you spend. Previously published at https://thecloudblog.net/post/cheapskates-journey-to-on-demand-load-tests-on-heroku-with-locust/