The Results Are In - Public Testing of the Solutions for Privacy and Scalability in Ethereum

Blockchain (decentralized ledger) is an innovative technology that promises to improve many diverse areas of human life. It transfers real-life processes and products into the digital space, ensures the speed and reliability of financial operations and reduces their cost, and allows for the creation of advanced DAPP applications by using smart contracts in decentralized networks. 

With its many advantages and diverse applications, it may seem puzzling that this promising technology has not yet penetrated every industry. The problem is that modern decentralized blockchains lack scalability. Ethereum processes about 20 tx/s, which is not enough to satisfy the needs of today’s dynamic businesses. At the same time, companies leveraging blockchain technology hesitate to abandon Enthereum, due to its high degree of protection against hacking and network failures.

To ensure decentralization, security, and scalability in blockchain, thereby solving the Scalability Trilemma, the development team of Opporty has created Plasma Cash - a child chain consisting of a smart contract and a private Node.js-based network - periodically committing its state to the Ethereum root chain.

Key Processes in Plasma Cash

1. A user activates the smart contract's `deposit` function by replenishing it with the amount of ETH they wish to invest in the Plasma Cash token. The smart contract function creates a token and generates an event dedicated to it.

2. Plasma Cash nodes linked to the smart contract's events receive a notification about the deposit creation and add the transaction, signaling the token creation to the pool.

3. Periodically, special Plasma Cash nodes take all transactions (up to 1 million) from the pool, assemble them into a block, and calculate the Merkle Tree according to the hash. This block is sent to other nodes for verification. Nodes check whether the Merkle hash and transactions are valid (for example, whether the sender is the token owner). After verifying the block, the node activates the smart contract's `submitBlock` function, which then stores the block's number and Merkle hash in the root chain. A smart contract generates an event signaling the successful block addition. Transactions are deleted from the pool.

4. The nodes that have received the block submission notification begin to apply the transactions that were added to the block.

5. At some point, the token owner may decide to withdraw it from Plasma Cash. To do so, they must activate the `startExit` function by providing information about the last two token transactions, thus confirming that they are the token owner. Using a Merkle hash, the smart contract checks transactions in blocks and sends a token to the output, which takes two weeks.

6. If the token withdrawal operation was carried out with violations (for example, the token was spent after the start of the withdrawal procedure or the token belonged to someone else prior to withdrawal), the token owner has two weeks to refute the withdrawal.

Privacy Can Be Achieved in Two Ways:

1. The root chain knows nothing about transactions that are generated and transferred within the child chain. It is only publicly known who added or withdrew ETH to/from Plasma Cash.

2. A child chain allows organizing anonymous transactions using zk-SNARKs.

Technology Stack

The following technologies were used while developing Plasma Cash:

- NodeJS

- Redis

- Etherium

- Soild

Testing

We tested the speed of the system and got the following results:

- up to 35,000 transactions per second are being added to the pool;

- up to 1,000,000 transactions can be stored in a block.

Tests were performed on the following three servers:

1. Intel® Core™ i7-6700 Quad-Core Skylake incl. NVMe SSD - 512 GB, 64 GB DDR4 RAM

Three validator Plasma Cash nodes were raised.

2. AMD Ryzen 7 1700X Octa-Core "Summit Ridge" (Zen), SATA SSD - 500 GB, 64 GB DDR4 RAM

Ropsten Testnet ETH node and three validator Plasma Cash nodes were raised.

3. Intel® Core™ i9-9900K Octa-Core incl. NVMe SSD - 1 TB, 64 GB DDR4 RAM

One submit Plasma Cash node and three validator Plasma Cash nodes were raised.

A test for adding transactions to the Plasma Cash network was launched. 

In total, we obtained 10 Plasma Cash nodes in the network.

Test 1

We had a limit of 1 million transactions per block. These 1 million transactions were distributed between two blocks, since the system was able to submit a portion of the transactions in the transfer process.

Initial state: the last block # 7; 1 million transactions and tokens are stored in the database.

00:00 - Transaction generation script was launched.

01:37 - 1 million transactions were generated and transition to the node began.

01:46 - A submit node extracted 240 thousand transactions from the pool and formed block # 8. Also, 320 thousand transactions were added to the pool in 10 seconds.

01:58 - Block # 8 was signed and sent for validation.

02:03 - Block # 8 was validated and the `submitBlock` function of the smart contract with Merkle hash and block number was activated.

02:10 - Demo script that processed 1 million transactions in 32 seconds finished working.

02:33 - Nodes began to receive information that block # 8 had been added to the root chain. The processing of 240 thousand transactions started.

02:40 - 240 thousand transactions placed in block # 8 were deleted from the pool. 

02:56 - A submit node took the remaining 760 thousand transactions from the pool and began to calculate the Merkle hash and sign block # 9.

03:20 - All nodes contain 1 million, 240 thousand transactions and tokens.

03:35 - Block # 9 was signed and sent to other nodes for validation.

03:41 - A network error occurred.

04:40 - The validation of block # 9 stopped according to the timing.

04:54 - The submit node took the remaining 760 thousand transactions from the pool and began to calculate the Merkle hash and sign block # 9.

05:32 - Block # 9 was signed and sent for validation to other nodes.

05:53 - Block # 9 was validated and sent to the root chain.

06:17 - Nodes began to receive information that block # 9 had been added to the root chain. The processing of 760 thousand transactions started.

06:47 - Pool was cleared of transactions from block # 9.

09:06 - All nodes contain 2 million transactions and tokens.

Test 2

We had a limit of 350 thousand transactions per block. They were distributed among three blocks. 

Initial state: last block # 9; 2 million transactions and tokens are stored in the database.

00:00 - The transaction generation script was already running.

00:44 - 1 million transactions were generated and transfer to the node began. 

00:56 - A submit node took 320 thousand transactions from the pool and formed block # 10. Also, 320 thousand transactions were added to the pool in 10 seconds.

01:12 - Block # 10 was signed and sent to other nodes for validation.

01:18 - Demo script that transferred 1 million transactions in 34 seconds finished working.

01:20 - Block # 10 was validated and sent to the root chain.

01:51 - All nodes received information from the root chain that block # 10 had been added. They started to apply 320 thousand transactions.

02:01 - Pool was cleared of 320 thousand transactions that were added to block # 10.

02:15 - The submit node took 350 thousand transactions from the pool and formed block # 11.

02:34 - Block # 11 was signed and sent to other nodes for validation.

02:51 - Block # 11 was validated and sent to the root chain.

02:55 - The last node added and performed transactions from block # 10.

10:59 - The transaction with the submission of block # 9 continued running for a long while in the root chain. It finally finished. All the nodes received information about this and started to execute 350 thousand transactions.

11:05 - The pool cleared out 320 thousand transactions that were added to block # 11.

12:10 - All nodes contain 1 million, 670 thousand transactions and tokens.

12:17 - The submit node took 330 thousand transactions from the pool and formed block # 12.

12:32 - Block # 12 was signed and sent to other nodes for validation.

12:39 - Block # 12 was validated and sent to the root chain.

13:44 - All nodes received information from the root chain that block # 12 had been added. They began to apply 330 thousand transactions.

14:50 - All nodes contain 2 million transactions and tokens.

Test 3

One validator node was replaced by a submit node in the first and second servers.

Initial state: the last block # 84, 0 transactions and tokens are stored in the database.

00:00 - 3 scripts that generate and transfer 1 million transactions were launched.

01:38 - 1 million transactions were generated and transfer to submit node # 3 began. 

01:50 - Submit node # 3 received 330 thousand transactions from the pool and formed block # 85 (f21). Also, 350 thousand transactions were added to the pool in 10 seconds.

01:53 - 1 million transactions were generated and transfer to submit node # 1 began. 

02:01 - Submit node # 1 received 250 thousand transactions from the pool and formed block # 85 (65e).

02:06 - Block # 85 (f21) was signed and sent to other nodes for validation.

02:08 - Demo script of server # 3, which transferred 1 million transactions in 30 seconds, finished working.

02:14 - Block # 85 (f21) was validated and sent to the root chain.

02:19 - Block # 85 (65e) was signed and sent to other nodes for validation.

02:22 - 1 million transactions were generated and transfer to submit node # 2 began. 

02:27 - Block # 85 (65e) was validated and sent to the root chain.

02:29 - Submit node # 2 received 111855 transactions from the pool and formed block # 85 (256).

02:36 - Block # 85 (256) was signed and sent to other nodes for validation.

02:36 - The script of server # 1, which transferred 1 million transactions in 42.5 seconds, finished working.

02:38 - Block # 85 (256) was validated and sent to the root chain.

03:08 - The script of server # 2, which transferred 1 million transactions in 47 seconds, finished working.

03:38 - All the nodes received from the root chain information that blocks # 85 (f21), # 86 (65e), and # 87 (256) were added and started applying 330 thousand, 250 thousand, and 111855 transactions respectively.

03:49 - The pool was cleared of 330 thousand, 250 thousand, and 111855 transactions that were added to blocks # 85 (f21), # 86 (65e), and # 87 (256).

03:59 - Submit node # 1 received 888145 transactions from the pool and formed block # 88 (214); submit node # 2 received 750 thousand transactions from the pool and formed block # 88 (50a); submit node # 3 received 670 thousand transactions from the pool and formed block # 88 (d3b).

04:44 - Block # 88 (d3b) was signed and sent to other nodes for validation.

04:58 - Block # 88 (214) was signed and sent to other nodes for validation.

05:11 - Block # 88 (50a) was signed and sent to other nodes for validation.

05:11 - Block # 85 (d3b) was validated and sent to the root chain.

05:36 - Block # 85 (214) was validated and sent to the root chain.

05:43 - All the nodes received from the root chain information that blocks # 88 (d3b) and # 89 (214) were added and started applying 670 thousand and 750 thousand transactions.

06:50 - Due to a disconnection, block # 85 (50a) was not validated.

06:55 - Submit node # 2 received 888145 transactions from the pool and formed block # 90 (50a).

08:14 - Block # 90 (50a) was signed and sent to other nodes for validation.

09:04 - Block # 90 (50a) was validated and sent to the root chain.

11:23 - All the nodes received from the root chain information that block # 90 (50a) was added and started applying 888145 transactions; at the same time, server # 3 has long applied transactions from blocks # 88 (d3b) and # 89 (214).

12:11 - The pool is empty.

13:41 - All the nodes of server # 3 contain 3 million transactions and tokens.

14:35 - All the nodes of server # 1 contain 3 million transactions and tokens

19:24 - All the nodes of server # 2 contain 3 million transactions and tokens.

Challenges

In the process of developing Plasma Cash, we encountered some problems that were either solved, or are currently in the process of being solved:

1. A conflict of function interactions within the system. For example, the function for adding transactions to the pool blocked the submission and validation of blocks and vice versa. This led to a decrease in speed.

2. A large number of transactions had to be transferred to minimize data transfer costs.

3. Optimal data storage was needed to achieve high results.

4. It was necessary to organize a network between nodes, since the block with 1 million transactions took about 100 MB.

5. Work in the single-threaded mode was breaking the connection between nodes when lengthy calculations were carried out (for example, when building a Merkle tree and calculating its hash).

Solutions

The first version of the Plasma node was akin to a universal soldier, able to do everything at once: accept transactions, submit and validate blocks, and provide an API for accessing data. But since the original NodeJS is single-threaded, it failed to cope with such a large load. For example, the laborious function for calculating the Merkle tree blocked the function for adding a transaction. 

There were two options for solving this problem:

1. To run several NodeJS processes at once and assign different functions to them.

2. To use worker_threads and place part of the code in threads.

We took advantage of both options.

A node was logically divided into three parts that are able to work autonomously but are well-synced:

1. Submit node that accepts transactions in the pool and is engaged in block creation.

2. Validator node that verifies the validity of nodes.

3. API node that provides API for data access.

We can connect to each node via the Unix socket using CLI. 

Laborious operations, such as calculating the Merkle tree, are carried out in a separate thread.

In this way we have achieved the smooth and synchronous operation of all functions of Plasma Cash.

Speed

Once the system began to function in an orderly manner, we proceeded to test its speed. The results were unsatisfactory: 5,000 transactions per second and up to 50,000 transactions per block. We began searching for the underlying reasons.

To begin with, we tested the mechanism of communication with Plasma to determine the peak capacity of the system. Earlier, we wrote that a Plasma node provides a Unix socket interface. It was originally textual. JSON objects were transferred using `JSON.parse ()` and `JSON.stringify ()`.

```json
{
 "action": "sendTransaction",
  "payload":{
	"prevHash": "0x8a88cc4217745fd0b4eb161f6923235da10593be66b841d47da86b9cd95d93e0",
	"prevBlock": 41,
	"tokenId": "57570139642005649136210751546585740989890521125187435281313126554130572876445",
	"newOwner": "0x200eabe5b26e547446ae5821622892291632d4f4",
	"type": "pay",
	"data": "",
	"signature": "0xd1107d0c6df15e01e168e631a386363c72206cb75b233f8f3cf883134854967e1cd9b3306cc5c0ce58f0a7397ae9b2487501b56695fe3a3c90ec0f61c7ea4a721c"
  }
}
```

We measured the speed for exchanging such objects and got ~ 130 thousand transactions per second. We tried to replace the standard functions for working with JSON, but the performance did not improve. It is likely that the V8 engine is well optimized for such operations.

Work with transactions, tokens, and blocks was carried out through classes. While creating the classes, performance dropped twice. Therefore, OOP was proven unsuitable for our purposes. We had to rewrite all the code based on a purely functional approach.

Data Recording

Redis was initially selected for data storage, as the solution most closely met our requirements, making it possible to work with key-value storage, hash tables, and sets. 

We launched Redis-benchmark and got ~ 80 thousand operations per second in a single pipeline mode.

Also, we configured Redis to increase performance:

- Set a Unix socket connection.

- Disabled state saving to disk (for reliability, it is best to configure the replica and save a separate Redis to disk).

The pool is represented as a hash table in Redis, since all transactions need to be in one request and deleted one by one. We tried to use a regular list, but it was downloading too slowly. 

With the standard Node.js library Redis, we achieved a performance speed of 18 thousand transactions per second. The speed dropped by nine times. Since the benchmark had clearly shown us a five times higher potential, we began optimizing the system. 

The library was changed from Redis to ioredis, which ensured the performance speed of 25 thousand operations per second. 

We added transactions one by one using the `hset` command, thereby generating a large number of requests. Also, we merged transactions into batches and transferred them with a single `hmset` command, which resulted in a performance speed of 32 thousand operations per second. 

For several reasons to be described below, we used `Buffer` to process data. It turned out that translating data into text (`buffer.toString ('hex')`) before recording it improved performance. Thanks to this approach, the system speed was increased to 35 thousand operations per second. 

At this point, optimization was suspended.

We had to switch to a binary protocol, since:

1. The system often calculates hashes, signatures, etc. and needs data in `Buffer` for this.

2. Binary data weighs less than text when transferring between services. For example, when transferring a block with 1 million transactions, textual data can take more than 300 megabytes.

3. Continuous data conversion affects performance.

We used our own binary protocol as a basis for storing and transmitting data, in conjunction with the great `binary-data` library.

As a result, we obtained the following data structures:

Transaction

 ```json
  {
	prevHash: BD.types.buffer(20),
	prevBlock: BD.types.uint24le,
	tokenId: BD.types.string(null),
	type: BD.types.uint8,
	newOwner: BD.types.buffer(20),
	dataLength: BD.types.uint24le,
	data: BD.types.buffer(({current}) => current.dataLength),
	signature: BD.types.buffer(65),
	hash: BD.types.buffer(32),
    blockNumber: BD.types.uint24le,
	timestamp: BD.types.uint48le,
  }
  ```

Token

```json
  {
	id: BD.types.string(null),
	owner: BD.types.buffer(20),
	block: BD.types.uint24le,
	amount: BD.types.string(null),
  }
  ```

Block

  ```json
  {
	number: BD.types.uint24le,
	merkleRootHash: BD.types.buffer(32),
	signature: BD.types.buffer(65),
	countTx: BD.types.uint24le,
	transactions: BD.types.array(Transaction.Protocol, ({current}) => current.countTx),
	timestamp: BD.types.uint48le,
  }
  ```

With regular `BD.encode (block, Protocol) .slice ();` and `BD.decode (buffer, Protocol)` commands, we can convert data to `Buffer` to be saved in Redis or forwarded to another node and later retrieved.

We also have two binary protocols for transferring data between services.

- Protocol for interacting with Plasma Node via Unix socket

```json
  {
	type: BD.types.uint8,
	messageId: BD.types.uint24le,
	error: BD.types.uint8,
	length: BD.types.uint24le,
	payload: BD.types.buffer(({node}) => node.length)
  }
  ```

Clarifications:

• `type` - action to be performed, for example, `sendTransaction` or `getTransaction`;
•  `payload` - data that needs to be transferred to the corresponding function;
•  `messageId` - message ID for response identification.

- Protocol of interaction between nodes

  ```json
  {
	code: BD.types.uint8,
	versionProtocol: BD.types.uint24le,
	seq: BD.types.uint8,
	countChunk: BD.types.uint24le,
	chunkNumber: BD.types.uint24le,
	length: BD.types.uint24le,
	payload: BD.types.buffer(({node}) => node.length)
  }
  ```

Clarifications:

• `code` - message code, for example 6 - PREPARE_NEW_BLOCK, 7 - BLOCK_VALID, 8 - BLOCK_COMMIT;
• `versionProtocol` - protocol version is designated since nodes of different versions can be raised on the network and work in different ways;
• `seq` - message identifier;
• `countChunk` and `chunkNumber` are needed to split large messages;
•  `length` and `payload` designate the length of data and data itself.

Since we had typed the data in advance, our approach ensured much faster work than the `RLP` library by Ethereum. Unfortunately, we have not yet been able to abandon it, since we need it to finalize the smart contract. However, we plan to do so in the near future.

While we have achieved a speed of 35,000 transactions per second, we also need to promptly process them. Since an approximate block formation time is 30 seconds, we need to include 1,000,000 transactions in a block, which entails the transfer of more than 100 MB.

Initially, we used the `ethereumjs-devp2p` library to connect nodes, but it failed to cope with large data volumes. As an alternative, we used the `ws` library and set up binary data forwarding via WebSocket. Of course, we also faced challenges when sending large data batches. They were addressed by dividing batches into chunks.

Also, forming a Merkle tree and calculating a hash of 1,000,000 transactions took about 10 seconds of continuous calculation. The connection with all nodes was broken during this time, so we decided to run these operations in a separate thread.

In Conclusion

We acknowledge that our discoveries are by no means new,  but for some reason they were forgotten by many when developing blockchain products.

Some key thoughts:

- Using Functional Programming instead of Object-Oriented Programming increases productivity.

- A monolith is worse than a service architecture for creating a productive system on NodeJS.

- Using `worker_threads` for laborious computing improves the system’s responsiveness, especially when working with i/o operations.

- Unix socket is more stable and faster than HTTP requests.

- To quickly transfer large data volumes over the network, it is better to use WebSockets and binary data split into chunks. These data can be altered and merged into a single message if the goal has not been reached. 

Please check GitHub: https://github.com/opporty-com/Plasma-Cash/tree/new-version

The article was written in co-authorship with Oleksandr Nashyvan, a Senior Developer at Clever Solution Inc.