What is the Lisk Codec and How Well Does it Perform?

[EDITOR NOTE] Schedule for Sun, Jun 13th

Serializing is the process of encoding information into bytes. Much fun! but oh so instrumental in blockchain. The topic of this blog post is a bit more technical than others, however, we will try to explain its main components in a clear and easy-to-understand way.

This blog post covers the reason why serialization is needed in the first place, and how it has been implemented up to now in the Lisk project. This is followed by an introduction to Lisk codec and the improvements it will bring to the SDK developer experience. We will conclude by showcasing some of the benefits, coupled with how it will enhance the Lisk network.

Glossary

• Message, object: A data structure, such as a JavaScript object. For example, in Lisk messages could represent transactions, blocks, or accounts.
• Binary message: A sequence of bytes. A serialization method outputs a binary message.
• Encoding, serializing: Converting a message into a binary message.
• Decoding, deserializing: Converting a binary message into a message.

In the diagram below, on the left-hand side, we show a message corresponding to a balance transfer transaction and on the right-hand side of the corresponding binary message. To improve readability, public keys and signature have been shortened and punctuation has been added to the binary message.

Serialization Goals and Uses

The Lisk blockchain uses several objects, such as transactions, blocks, and accounts. In most non-blockchain projects, the manner in which objects are converted to bytes by your device is not actually critical. The messages can be displayed, stored, copied, and modified without causing any unwanted issues.

However, in blockchain projects, it is essential that the same object is converted into the same byte sequence every time and by all users. Otherwise, properties such as signature or ID (obtained by hashing the binary message), would be invalid or change. If you sign a transaction on your device and this same transaction is then serialized differently by someone else, they would reject your signature and consider your transaction invalid. 

Besides this critical property, an efficient serialization method can benefit other parts of the ecosystem. One of the problems that blockchains are facing is the growing storage and network requirements. In this regard, transmitting, receiving, and storing fewer bytes makes the whole application lighter, and allows for faster synchronization. This in turn highlights the advantages of using a serialization method which generates small binaries.

Finally, to improve the decentralization of Lisk, it is important that the Lisk protocol can be implemented in various programming languages, and that different teams can work on creating tools for the ecosystem. Hence, the serialization method used in the Lisk protocol must be well defined, and implementation agnostic.

Shortcomings of the Current Solution

Up to version 4.0.0, the Lisk tool kit did not include a serialization method which satisfied all the above requirements.  Binary messages were only used for signing, and not for communication and storage, as such the serialization method was not optimized for speed and size.

Another pain-point in the current architecture is the serialization of custom transactions. To serialize mainnet transactions, Lisk implemented a getBytes function. For custom transactions, the application developers have to specify a custom transaction asset for which they have two serialization alternatives shown below: 

1. Serialize the asset as a JSON string.

assetToBytes(asset) = Buffer.from(JSON.stringify(asset), 'utf-8');

2. Write a custom assetToBytes function.

The first option is fast, however, it is not efficient in terms of size, and furthermore different node versions could have slightly different implementations. The second option is time-consuming and could lead to errors. In addition, there is no guarantee that the custom assetToBytes function is efficient in regard to its size and encoding speed.

In a unified ecosystem, different projects should use similar encodings for transmitting objects and for serializing them. If this burden is left on the shoulders of the developer, it is likely that the ecosystem will be filled with all kinds of unorthodox binaries. In the worst-case scenario, this could lead to unsafe behavior and maybe even loss of funds.

Lisk Codec

To address these problems, in version 5.0.0 of the SDK the Lisk team is introducing Lisk codec. Lisk codec is an efficient, light-weight module that allows users to encode and decode JavaScript objects. Using the codec, the application developer only needs to provide a JSON schema for each sidechain object. The schema is then used for validating, encoding, and decoding the object.

Another advantage of Lisk codec is its ability to support the Lisk transition to a key-value store. The binary sequences can directly be stored and only need to be decoded when used by the application. Storing and transmitting binary messages directly enables the network to run smoother and avoid delays due to unnecessary encoding and decoding. Finally, nodes can answer API requests by sending the byte value corresponding to the requested key directly.

Lisk codec is based on Protocol buffers (protobuf), and has been tailored to fit the needs of a blockchain project. Protobuf is a well-known serialization mechanism, meaning that the Lisk codec behavior for encoding and decoding can be reproduced by most protobuf implementations. However, the protobuf serialization is not always deterministic, and as such not directly suited for our use case.

The following example below illustrates serializing an empty block using Lisk codec:

blockData = {
  "header": {
    "version": 3,
    "timestamp": 180,
    "height": 16,
    "previousBlockID": "e194ce4e908c148ea4d11719cd40a016d07f393d31031ea150d7a8b7904a22d5",
    "transactionRoot": ,
    "generatorPublicKey": "ed3b9fd50b188d35f5d2ea3fef05cb894363931c5ba50a5967c224ae5b16b339",
    "reward": 100000000,
    "asset": {
      "maxHeightPreviouslyForged": 3,
      "maxHeightPrevoted": 10,
      "seedReveal": "8038ec83c421fa4844c5c65995cb2a66"
    },
    "signature": "bfc186b17132180057c8604640c276b85169fcaba72255bdc24f9220e620aa3e9731e6308a131f87097979e5696a7c38a25212bcb4779b099fab7df576b50207"
  },
  "payload": []
}

blockMsg = {
  0aac01: {
    08: 03,
    10: b401,
    18: 10,
    2220: e194ce4e908c148ea4d11719cd40a016d07f393d31031ea150d7a8b7904a22d5,
    2a00: ,
    3220: ed3b9fd50b188d35f5d2ea3fef05cb894363931c5ba50a5967c224ae5b16b339,
    38: 80c2d72f,
    4216: {
      08: 03,
      10: 0a,
      1a10: 8038ec83c421fa4844c5c65995cb2a66
    },
    4a40: bfc186b17132180057c8604640c276b85169fcaba72255bdc24f9220e620aa3e9731e6308a131f87097979e5696a7c38a25212bcb4779b099fab7df576b50207
  }
}

You can find more examples of transaction serialization in LIP 0028 and examples for blocks in LIP 0029.

Developer Experience

The use of Lisk codec will improve the experience of developing custom applications. 

All Lisk transactions now have a set of common baseTransaction properties (such as fee, nonce and sender public key), and a set of transaction-specific properties defined in the asset.

Custom transactions are similarly defined by a JSON schema specifying their asset. The serialization and validation with respect to this schema will happen in the background and does not require any additional developer input. This saves developing time and minimizes the risk of introducing errors. The SDK will also include tools to validate the syntax of your custom asset schemas.

The example below shows the asset schema of the balance transfer transaction:

balanceTransferAsset = {
  "type": "object",
  "properties": {
    "amount": {
      "dataType": "uint64",
      "fieldNumber": 1
    },
    "recipientAddress": {
      "dataType": "bytes",
      "fieldNumber": 2
    },
    "data": {
      "dataType": "string",
      "fieldNumber": 3
    }
  },
  "required": ["amount","recipientAddress","data"]
}

Alternatively, a custom transaction to announce the return of a bike and its new position is displayed below:

returnBikeAsset = {
  "type": "object",
  "properties": {
    "bikeID": {
      "dataType": "bytes",
      "fieldNumber": 1
    },
    "returnPosition": {
      "type": "object",
      "properties": {
         "longitude": {"dataType": "uint64","fieldNumber": 1},
         "latitude": {"dataType": "uint64","fieldNumber": 2}
      },
      "required": ["longitude","latitude"],
      "fieldNumber": 2
    },
    "bikeState": {
      "dataType": "string",
      "fieldNumber": 3
    }
  },
  "required": ["bikeID","returnPosition","bikeState"]
}

Performance of Lisk Codec

A benchmark of the performances of Lisk codec is available in the Lisk codec repository. Encoding and decoding of regular transactions can be done at a rate of 200,000 transactions per second. Full blocks can be encoded and decoded at a rate of 50,000 blocks per second. 

As mentioned above, using Lisk codec is also beneficial in terms of size for communicating between nodes and storing the blockchain. Currently, objects are saved and exchanged in JSON format. A regular balance transfer (no data field, no multisignature) is roughly 500 bytes. If encoded with Lisk codec the same object would be 150 bytes. This is a 70% reduction! Other objects would see a similar reduction in size, which is great news for the Lisk network and will help it scale in the future.

Conclusion

To summarize here, introducing Lisk codec provides the end-user with the ability to streamline the whole Lisk application, reduce network communications, and lower the global size of the blockchain. You can find a precise specification of Lisk codec in LIP 0027, while schemas used for transactions, blocks, and accounts can be found in LIP 0028, LIP 0029, and LIP 0030, respectively. 

For further questions and comments on the topic, we will host an AMA on Lisk.chat with Maxime Gagnebin (Research Scientist) this Friday, August the 21st at 4 pm CEST. We also invite all community members to go to the Lisk Research forum where we are always happy to hear your feedback and to participate in discussions on this and other topics.