A Brief Overview of Blockchain Testing and Benchmarking Tools

Testing solutions for distributed systems today are usually built for a specific type of blockchain or its forks. Some of them are open source projects, others are SaaS, but the majority are in-house solutions. However, they all solve similar problems.

Blockchain has a lot in common with distributed databases, so you can use analogous testing tools and methods. To better understand how distributed databases are tested, take a look at a nice selection of articles from here. For example, latency is explained in detail here; to understand how bugs are identified in replication algorithms, I recommend reading this article.

I will review some popular solutions for blockchain testing and benchmarking. You are very welcome to mention other useful software products in the comments below.

A distributed system is considered reliable if it continues working in case of problems with servers and network issues. These can be network lags, storage overflow, unavailable external services (like DNS), hardware failure and hundreds of other reasons. To test if a system is stable, use Gremlin. It relies on a very efficient approach called Chaos Engineering.

Gremlin installs its network agent on the required number of machines and creates various issues: network lags, resource overload (CPU, disk, memory, network), disables different protocol parts, etc. So, developers and administrators can track what happens in case of a system crash in a controlled environment.

For blockchains, Gremlin can be used on testnet nodes, but the network must be configured and deployed in advance.

Gremlin is a convenient tool for architects, devops and security specialists and a universal solution for testing any launched distributed systems, including blockchains.

Hyperledger Caliper is a more specialized solution. At the moment, Caliper supports several blockchains at once - representatives of the Hyperledger family (Fabric, Sawtooth, Iroha, Burrow, Besu) as well as Ethereum and the FISCO BCOS network.

Caliper allows to specify blockchain topology and contracts for testing, as well as describe node configuration. Blockchain nodes are deployed in docker containers on one machine. Next, you can select the necessary test configurations and get a file with test results after launch. Metrics can also be viewed at Grafana. A complete list of Caliper metrics and benchmarking process can be found in very good article here , especially if you are interested in blockchain benchmarking .

Hyperledger Caliper is a tool designed for developers and system architects, as it provides high repetitiveness and automated testing and benchmarking. It is often used to develop a blockchain core, consensus algorithms, a virtual machine for processing smart contracts, peer-to-peer layers and other system mechanisms.

MixBytes Tank is a tool that appeared while developing consensus and finality algorithms for EOS-based networks and testing parachains based on Parity Substrate (Polkadot). It resembles Hyperledger Caliper in terms of functionality, since it collects many important metrics from the nodes of any distributed system and client machines that run benchmarking scripts.

MixBytes Tank supports several cloud services (Digital Ocean, Google Cloud Engine, etc.) for launching many nodes, performs preliminary configuration procedures, distributes nodes geographically, emulates packetloss problems, simultaneously runs several benchmarks on different machines, collects necessary metrics and automatically shuts down the whole test network.

MixBytes Tank allows to save money on cloud servers, automatically removing unnecessary resources after the test. Another distinguishing feature is the Molecule package use allowing a developer to test a complicated blockchain setup locally.

MixBytes Tank allows for early detection of bottlenecks and errors in algorithms that occur in real networks with a large number of geographically distributed servers and clients. Tank allows to track what is happening on blockchain nodes if clients are sending transactions at a given rate on repeat and integrate its results into CI/CD process.

Whiteblock Genesis was born as an Ethereum-based blockchain testing platform. This tool is of rich functionality: it allows to run a network, create a necessary number of accounts, launch the desired number of clients, configure the network topology, set bandwidth and packetloss parameters and run a test.

Whiteblock Genesis provides its own test facilities. It is enough for developers to specify test parameters, run them using a ready-made API and get results on the dashboard.

With Whiteblock Genesis you can configure a fairly detailed test, that will be automatically carried out for any significant code change. You will be able to find errors early and immediately assess the impact of changes on important network parameters, such as transaction speed and resources consumed by nodes.

Another interesting young product for testing distributed systems is Madt. It is written in Python and allows to create the network topology and the required number of servers and clients using a simple configuration script (example). After that, the service deploys the network in several docker containers and runs a web interface for tracking messages from servers and network clients.

Madt can be used for blockchain testing - there is a test for p2p networks based on the Kademlia protocol. The test is meant to check the status of the data in case of gradually increasing delays of data transfer to the nodes.

Though recently appeared, Madt may grow into a very effective product, given its very flexible architecture.

In-house solutions

What testing software do blockchain projects use the most? Their own solutions.

Almost any testing of the blockchain system part requires launch of preliminary scripts, preparing accounts and conditions for the test (testing consensus errors that may generate numerous chain forks; hard fork script scenarios, changing system parameters, etc.). These manipulations are performed differently in different blockchains, so it’s easier for teams to gradually “adapt” testing and benchmarking for their product, gradually increasing complexity and depth of testing procedures.

Conclusion

Summing up this overview, I will list the important characteristics of blockchain testing tools:

• Ability to automatically deploy a blockchain network in repeatable conditions. This factor is crucial for developing blockchain system parts: consensus algorithms, finality, system smart contracts.
• The price of system ownership, consumable resources and convenience for constant use. Obtain high-quality tests for reasonable money.
• Flexible and simple test configuration. This factor directly influences your ability to identify system problems - you are less likely to miss something important.
• Customization for specific blockchains. Developing a solution based on the existing one can greatly improve the quality and timeline.
• Convenience and availability of the results and their type (reports, metrics, graphs, logs, etc.). This is essential if you want to keep a history of product development, or need an in-depth analysis the blockchain network behaviour.

Good luck in testing and let your blockchains be fast and fault-tolerant!

(Disclaimer: The Author is associated with the MixBytes Team)