Protobuf vs JSON in The Ruby World

On my current project, I work with protobuf not only for GRPC, but also as RabbitMQ message format. While the advantages of protobuf are not limited to its speed, I was wondering if it's really so fast compared to JSON libraries, especially in the ruby world. I decided to make some benchmarks to check it, but first I want to add a brief introduction to each format.

What is protobuf?

It's a fast, compact cross-platform message system, designed with forward-and backward compatibility in mind. It consists of a definition language and language-specific compilers.

It works perfectly for small object-like data, has great backward and forward compatibility, is fast(we are not sure yet), and is more compact than JSON, for example, but have some limitations like not supporting direct comparison(you need to deserialize objects to compare).

It's not compressed and some specific formats can work better for their data(for example JPEG). It's not self-describing.

See Official docs for more details.

What is JSON

JSON is an abbreviation for JavaScript object notation. A text-based data format, which was originally used in JavaScript, but later got widely spread as a communication format not only between JS apps and backend but even between microservices and has multiple other usages.

It uses strings as keys and has a string, number, boolean, object, array and nul as available types for value. The main advantage of it is that it is human-readable, and pretty easy to serialize and parse for programming language.

See the site for more details.

Benchmarks

I picked up three popular ruby JSON libraries. They are Oj, Yajl, and standard JSON library. For protobuf, I use standard google protoc with google ruby gem.

I will measure different specific types of payload to see which data type we will show the most difference, as long as complex payload with a mix of field types.

You can see all the code here https://github.com/alexstaro/proto-vs-json.

Benchmark setup

As a hardware I use laptop with AMD Ryzen 3 PRO 5450U and 16gb of ddr4 ram.

As an operating system, I use Ubuntu 22.10 kinetic.

Ruby version 3.2.1 was installed via asdf.

For benchmarking, I use benchmark/ips gem (https://github.com/evanphx/benchmark-ips)

The setup looks like this:

Benchmark.ips do |x|  
  x.config(time: 20, warmup: 5)  
  
  x.report('Yajl encoding') do  
    Yajl::Encoder.encode(data)  
  end  
  ...
  x.compare!  
end

Integers only

We will start with only integers. Numbers are pretty hard for JSON so we expect protobuf to be far away from other competitors.

The test data:

data = {  
  field1: 2312345434234,  
  field2: 31415926,  
  field3: 43161592,  
  field4: 23141596,  
  field5: 61415923,  
  field6: 323423434343443,  
  field7: 53141926,  
  field8: 13145926,  
  field9: 323423434343443,  
  field10: 43161592  
}

Benchmark results:

   protobuf encoding:  4146929.7 i/s
         Oj encoding:  1885092.0 i/s - 2.20x  slower
standard JSON encoding:   505697.5 i/s - 8.20x  slower
       Yajl encoding:   496121.7 i/s - 8.36x  slower

There is no doubt that protobuf is an absolute winner, but what if we make the test more closer to the real-world scenario - we almost always create proto messages only for serialization.

What would happen if we move model initialization under benchmark?

Here are the results:

   protobuf encoding:  4146929.7 i/s
         Oj encoding:  1885092.0 i/s - 2.20x  slower
standard JSON encoding:   505697.5 i/s - 8.20x  slower
       Yajl encoding:   496121.7 i/s - 8.36x  slower
protobuf with model init:   489658.0 i/s - 8.47x  slower

The result is not so obvious. I expected encoding with message initialization would be slower but not the slowest.

Let's check deserialization:

    protobuf parsing:   737979.5 i/s
          Oj parsing:   448833.9 i/s - 1.64x  slower
standard JSON parsing:   297127.2 i/s - 2.48x  slower
        Yajl parsing:   184361.1 i/s - 4.00x  slower

There are no surprises here.

In terms of payload size, protobuf is almost 4 times more compact compared to json:

JSON payload bytesize 201
Protobuf payload bytesize 58

Doubles only

Doubles are expected to be the hardest payloads for JSON, let's check this out.

Our payload:

data = {  
  field1: 2312.345434234,  
  field2: 31415.926,  
  field3: 4316.1592,  
  field4: 23141.596,  
  field5: 614159.23,  
  field6: 3234234.34343443,  
  field7: 53141.926,  
  field8: 13145.926,  
  field9: 323423.434343443,  
  field10: 43161.592  
}

Result:

protobuf encoding:  4814662.9 i/s
protobuf with model init:   444424.1 i/s - 10.83x  slower
         Oj encoding:   297152.0 i/s - 16.20x  slower
       Yajl encoding:   160251.9 i/s - 30.04x  slower
standard JSON encoding:   158724.3 i/s - 30.33x  slower

Protobuf is much faster even with model initialization. Let's check the deserialization:

Comparison:
    protobuf parsing:   822226.6 i/s
          Oj parsing:   395411.3 i/s - 2.08x  slower
standard JSON parsing:   241438.7 i/s - 3.41x  slower
        Yajl parsing:   157235.7 i/s - 5.23x  slower

Still no surprises here.

and the payload size:

JSON payload bytesize 211
Protobuf payload bytesize 90

Not four times, but still noticeable.

Strings only

Strings are expected to be easier for JSON, let's check this out.

payload:

data = {  
  field1: "2312.345434234",  
  field2: "31415.926",  
  field3: "4316.1592",  
  field4: "23141.596",  
  field5: "614159.23",  
  field6: "3234234.34343443",  
  field7: "53141.926",  
  field8: "13145.926",  
  field9: "323423.434343443",  
  field10: "43161.592"  
}

Bench results:

Comparison:
   protobuf encoding:  3990298.3 i/s
          oj encoder:  1848941.3 i/s - 2.16x  slower
        yajl encoder:   455222.0 i/s - 8.77x  slower
standard JSON encoding:   444245.6 i/s - 8.98x  slower
protobuf with model init:   368818.3 i/s - 10.82x  slower

Deserialization:

Comparison:
     protobuf parser:   631262.5 i/s
           oj parser:   378697.6 i/s - 1.67x  slower
standard JSON parser:   322923.5 i/s - 1.95x  slower
         yajl parser:   187593.4 i/s - 3.37x  slower

The payload size:

JSON payload bytesize 231
Protobuf payload bytesize 129

Integer array

Despite we have separated integers bench it's interesting how protobuf handles collections.

Here is the data:

data = {  
  field1: [  
    2312345434234, 31415926, 43161592, 23141596, 61415923, 323423434343443, 53141926, 13145926, 323423434343443, 43161592  
  ]  
}

Serialization bench:

Comparison:
   protobuf encoding:  4639726.6 i/s
          oj encoder:  2929662.1 i/s - 1.58x  slower
standard JSON encoding:   699299.2 i/s - 6.63x  slower
        yajl encoder:   610215.5 i/s - 7.60x  slower
protobuf with model init:   463057.9 i/s - 10.02x  slower

Deserialization bench:

Comparison:
           oj parser:  1190763.1 i/s
     protobuf parser:   760307.3 i/s - 1.57x  slower
standard JSON parser:   619360.4 i/s - 1.92x  slower
         yajl parser:   414352.4 i/s - 2.87x  slower

To be honest, the deserialization results are pretty unexpected here.

Let's check payload size:

JSON payload bytesize 121
Protobuf payload bytesize 50

Array of doubles

I decided to check if an array of doubles shares the same behavior.

data:

data = {  
  field1: [  
    2312.345434234, 31415.926, 4316.1592, 23141.596, 614159.23, 3234234.34343443,
    53141.926, 13145.926, 323423.434343443, 43161.592  
  ]  
}

Serialization:

Comparison:
   protobuf encoding:  7667558.9 i/s
protobuf with model init:   572563.4 i/s - 13.39x  slower
         Oj encoding:   323818.1 i/s - 23.68x  slower
       Yajl encoding:   183763.3 i/s - 41.73x  slower
standard JSON encoding:   182332.3 i/s - 42.05x  slower

Deserialization:

Comparison:
          Oj parsing:   953384.6 i/s
    protobuf parsing:   883899.0 i/s - 1.08x  slower
standard JSON parsing:   452799.0 i/s - 2.11x  slower
        Yajl parsing:   356091.2 i/s - 2.68x  slower

We got similar results here. It seems that protobuf has some issues with arrays.

Payload size:

JSON payload bytesize 131
Protobuf payload bytesize 82

Complex payload

As a "complex" payload I mocked some user data with posts and comments for those posts to make it more like real life application.

data = {  
  user_id: 12345,  
  username: 'johndoe',  
  email: '[email protected]',  
  date_joined: '2023-04-01T12:30:00Z',  
  is_active: true,  
  profile: {  
	full_name: 'John Doe',  
	age: 30,  
	address: '123 Main St, Anytown, USA',  
	phone_number: '+1-555-123-4567'  
  },  
  posts: [  
	{  
    	post_id: 1,  
    	title: 'My first blog post',  
    	content: 'This is the content of my first blog post.',  
    	date_created: '2023-04-01T14:00:00Z',  
    	likes: 10,  
    	tags: ['blog', 'first_post', 'welcome'],  
    	comments: [  
          {  
    	    comment_id: 101,  
    	    author: 'Jane',  
    		content: 'Great first post!',  
    		date_created: '2023-04-01T15:00:00Z',  
    		likes: 3  
    	  },  
    	  ... 
        ]  
    },  
    ...
  ] 
}

The results:

Comparison:
   protobuf encoding:  1038246.0 i/s
         Oj encoding:   296018.6 i/s - 3.51x  slower
       Yajl encoding:   125909.6 i/s - 8.25x  slower
protobuf with model init:   119673.2 i/s - 8.68x  slower
standard JSON encoding:   115773.4 i/s - 8.97x  slower


Comparison:
    protobuf parsing:   291605.9 i/s
          Oj parsing:    76994.7 i/s - 3.79x  slower
standard JSON parsing:    64823.6 i/s - 4.50x  slower
        Yajl parsing:    34936.4 i/s - 8.35x  slower

And payload size:

JSON payload bytesize 1700
Protobuf payload bytesize 876

We see here the expected behavior with pure protobuf encoding on the first place, however, if we look at our “real-world” example we see it is not faster that the standard JSON encoding.

Conclusion

If you are switching from JSON to Protobuf just for the speed, it's may not worth it.

The reason to use Protobuf should be the awesome cross-language schema definition for data exchange — not a performance boost.

The lead image for this article was generated by HackerNoon's AI Image Generator via the prompt "programming language".