Learnings while benchmarking Java Template Engine with Spring Boot — 2018

Performance testing server side rendering using java template engines + Spring Boot and some learnings on the way

This article is about benchmarking server side rendering tools i.e the most commonly used java based templating engines i.e Velocity, Freemarker and HTTL (new to me) and lessons I learnt while doing so. I have already worked with Velocity and Freemarker as my server side render tools but after reading about HTTL, I wanted to get real insights of how actually these 3 will perform with a Spring Boot project.

Below is the system info where java server side render/template benchmarking will happen:

1. OS: MacOS Sierra
2. Processor: 1.6 GHz Intel Core i5
3. Java: Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
4. Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

For benchmarking we are using Apache bench. The code is a simple spring boot app exposing three endpoints and three spring based view resolvers like below snippet.

You can download the complete code from this github repo

@SpringBootApplication@Controllerpublic class TemplateBenchmark {

@Bean public HttlViewResolver httlViewResolver() { HttlViewResolver resolver = new HttlViewResolver(); … return resolver; } @Bean public ViewResolver vmViewResolver() { VelocityLayoutViewResolver bean = new VelocityLayoutViewResolver(); …. return bean; }  @RequestMapping(“/checkout-httl”) public String checkout(Model model) { model.addAttribute(“name”, “Apurav Chauhan”); return “checkout-httl”; }}

ROUND 1 — Simple key val + HTML render

HTTL Template engine for server side rendering

We warm up the server with 5000 requests in total with a concurrency of 5.

Finally, we set the bench to hit 25K requests with a concurrency of 25 using the following command

ab -n 25000 -c 25 -g httl.tsv -k http://localhost:9090/checkout-httl

-n flag for total requests | -c for concurrency | -g to save bench stats to a tab separated file | -k to enable keep alive connections

Learning 1

While doing benchmarking, if you see this error ”apr_socket_recv: Operation timed out (60)” its because load has consumed all the available 16,383 ports on MacOS.

As each port is used, it get’s put into a queue where it waits for the tcp “Maximum Segment Lifetime”, which is configured to be 15 seconds on osx. So if you use more than 16,383 ports in 15 seconds, you’re effectively going to get throttled by the OS on further connections. Depending on which process runs out of ports first, you will get connection errors from the server, or hangs from [_ab_](https://stackoverflow.com/a/30357879/949912).

To fix this, use the below command to check the default wait time in your macos terminal.

sysctl net.inet.tcp.msl

And update it to a lower value like 1000ms by using following command

sudo sysctl -w net.inet.tcp.msl=1000

And now here are the stats:

HTTL VisualVM Stats

A total of 25K requests were completed within ~3secs. CPU usage bumped up to 70%. Mean processing time for a single request was 0.134ms

Freemarker Template Engine for java server side render

The same step was repeated to fetch the below stats

FTL JVM Stats

A total of 25K requests were completed within ~12secs. CPU usage bumped up to 30%. Mean processing time for a single request was 0.476ms

Velocity Template Engine for java server side rendering

VM JVM Stats

A total of 25K requests were completed within ~6secs. CPU usage bumped up to 90%. Mean processing time for a single request was 0.253ms

GNUPLOT FOR BENCHMARK DATA

Using the tsv file that was generated via -g flag, we will now generate a graph to actually see how each of these engines performed in comparison to each other. You can check the template files in this and this github links that were used to create the below graphs. Run the below command :

gnuplot template.p

gnuplot template2.p

Benchmarks HTTL vs Freemarker vs Velocity

HTTL definitely shines, but Freemarker shows a huge downside. Lets dig deeper. While analyzing the ab results, we can see that freemarker didn’t use any keep alive requests causing the skewed behaviour by including the connection creation lag in the results. But why?

Learning 2

After some research, found that ab tool works on Http/1.0 protocol that defines the server to send a Content-Length header to respect the keep alive connection. Freemarker responds with a chunked encoded response thus NOT allowing the connection reuse using Connection: KeepAlive header.

Lets update the code to set content length header for our simple use case and see what we get:

Seems like Freemarker is back in the race! :)

Still not satisfied? Lets try to benchmark the three using jmeter now. I am using 25 users with a ramp-up period of 20 secs with a loop count of 25k and below are the results:

Jmeter stats for response time

Performance of HTTL and Freemarker seems to be comparable for a simple template render with a key value object. However, Velocity is completely out of the picture!

Here is the Jmeter test plan used to benchmark java server side templates discussed above.

Round 2 — Complex object + HTML rendering

Coming soon