At my job, I am evaluating some microservices platform options. A good way to test them is doing some benchmarks. Everybody likes benchmarks. So, why not? although I tried to make conditions fair and "scientific", it's an and I'm making this for fun. Please, don't take it too seriously. Disclaimer: amateur benchmark Why Quarkus? It's the "supersonic subatomic Java" framework, sponsored by Red Hat. My work place is a Red Hat shop and we have many subscriptions from them. So, Quarkus is not only a good fit, but we already use it on some projects. It promises a better startup time and memory footprint. Tested version: 1.6.1. Why Phoenix Well, personal preference. I like Elixir, it is a powerful and fun language. When you discover Erlang and what problems it was made to solve, you will find it amazing. WhatsApp, Discord and many others use it. The framework of choice for the Elixir language is Phoenix. Achieving is really impressive too. Tested version: 1.5.4. 2 million connections in a single machine The benchmark I couldn't find any real benchmarks on these technologies. There is , which are the technologies behind these frameworks, but using a "sleep" to simulate load is not ok. These frameworks handle databases and connection pools differently. Acquiring a database connection from a pool can timeout and throw an exception. A "sleep" command won't timeout. Serializing data to JSON also impose memory and CPU usage. this benchmark on Netty and Cowboy does something different, but it uses good hardware and low load. Its focus is on how many requests per second a framework can handle, and this is not ok too. So, speed isn't a concern at this point. Techempower I need to know what framework can handle more requests within constrained hardware, like a Heroku dyno. In this benchmark, I decided to use my work development database and data. It's an Oracle database and my test data is on a table with 5 columns and 6000 rows. Yes, it is a lot of JSON, so it can force some timeouts, CPU and memory usage. Although Oracle JDBC driver is well tested and supported, in Phoenix we must rely on package. It works well and it is maintained, but it's not "official". Oracle database driver is not a first class citizen on Quarkus either, and we can not compile it to a native image using GraalVM out of the box. jamdb_oracle We are going to use the tool. Quarkus will run on OpenJDK 11 by Red Hat and Phoenix on Erlang OTP 22, both on a Red Hat Linux 8 with 4 cores and 8 GB of RAM. But, we are pretending to be inside a Heroku dyno or docker container, so the memory limit will be 512 MB. siege On Heroku or docker, any memory allocation beyond this will use disk swap, reducing perfomance. Keep that in mind! Comparing Apples to Oranges Before we start, we have to remember this. We can not use these frameworks as they come out of the box. To compare apples to apples (and not to oranges) and make things fair, I will configure both frameworks with the same connection pool size and timeout. In Phoenix, the default connection timeout and pool size are very low, so it can fail fast and use less memory. In Quarkus, the connection timeout is longer and pool size bigger, so it can serve more requests at the cost of memory. Well, that is what I think, but I might be wrong. For the test, we had to impose some limits to the JVM, as suggested by Heroku: java -Xss512k -Xmx300m -jar <my-app>.jar We will use to hit the REST endpoint with 25, 50, 100 e 250 concurrent connections, 5 times each. siege siege -d1 -r5 -c250 http://<my-server> Here are the results: CPU Winner: Quarkus Both frameworks have a similar CPU usage profile and "warmup" peak. Quarkus and JVM peaked at 56% of CPU usage while Phoenix and Erlang VM peaked at 79% of CPU usage. OpenJDK uses less CPU over time and is the clear winner. Memory Winner: Phoenix (x2) Phoenix peaked at 364 MB and Quarkus peaked at 807 MB. Here we can see Quarkus and JVM problem on microservices. Java refuses to stay within the 512 MB limit. Actually, it uses more than twice the memory Phoenix used. As stated above, at Heroku or docker with a 512 MB of RAM limit, any memory beyond that will use disk swap, reducing performance. In this benchmark, I could simulate this, because I had memory available. not Why Phoenix won 2 times? I could not show it on the graph, but another thing to notice is that Erlang/Elixir gives back unused memory to the operating system. It peaked at 364 MB, but after the test, it was using only 152 MB. JVM, on the other hand, never return unused memory back to the operating system, it peaked at 807 MB and stayed there. So, Quarkus lost twice here. If you run your own cloud, like me, this is an important and desired behaviour. Speed Winner: Quarkus Quarkus took, at most, 34 secs to respond to all requests. Phoenix took 60 secs. Not much to argue here. Quarkus and JVM are faster. And in this benchmark, they are almost twice as fast than Phoenix. Quarkus is the clear winner. If your microservice have any speed requirement, Quarkus is a better fit than Phoenix. Timeouts Winner: Draw What? Why a draw? Well, handling 250 simultaneous connections, Quarkus timed out 110 times while Phoenix timed out 155 times. Quarkus should be the winner. The problem is the difference is not so big. As stated above, Quarkus used more memory than it was available. If it was at Heroku or docker, it would be using disk swap, having perfomance penalties and leading to more timeouts. But, I could not simulate that and Quarkus had this "advantage". So, I will consider it a draw. Conclusion As I said in the introduction, I was looking for a framework that could handle more simultaneous connections using less memory. If you are like me and need to deploy your microservices on premise hardware or on Heroku, memory usage comes first. And the ability to return unused memory back to the cluster is desirable. Requirements like speed and CPU usage comes after. Phoenix is the clear winner under this circuntances. Quarkus, on the other hand, shines where speed is a requirement and memory consumption is acceptable. The cool thing about a microservices archtecture is that you can use the best tool for the job. So, use both frameworks. They are new, comparable and capable. Also published at https://medium.com/@hlalvesbr/quarkus-vs-phoenix-62f6b1965037

Heroku

Oracle

Read My Stories

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Performance Battle: Quarkus vs Phoenix

Too Long; Didn't Read

Companies Mentioned

Hélio Bessoni Rodrigues

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Performance Battle: Quarkus vs Phoenix

Too Long; Didn't Read

Companies Mentioned

Hélio Bessoni Rodrigues

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES