Experimenting with Chaos Engineering and Blockchain

In this article, we'll show how you can use the open-source Chaos Toolkit (CTK) to better understand Blockchain, stable-states, and what immutability really means. Yolanne Lee's full tutorial on CTK/Blockchain is here for reference for the experiment details.

By the end of this article, you'll understand two important things:

1. How to create and verify a blockchain transaction
2. How experiments in CTK can be incredibly powerful tools for both testing and automation

Typically in chaos engineering, your experiment automates the testing and reporting of your system's strengths and weaknesses. You can use this to build more reliable systems. Let's experiment with understanding the state of a blockchain transaction. For this tutorial you've got everything you need already configured, so let's jump in (with Python 3.6 or higher):

git clone https://github.com/yolannel/CTKBlockchain

pip install -U -r requirements.txt

Experiment 1: Understanding Rollbacks, Immutability and Stable States

If you are interested in Blockchain, you probably care a lot about building things that hold immutable ledgers. These ledgers create records that cannot be changed. Some important terms to understand are listed below.

• Node - user or computer within the blockchain architecture (each has an independent copy of the whole blockchain ledger)
• Transaction - the smallest building block of a blockchain system (records, information, etc.) that serves as the purpose of blockchain
• Block - a data structure used for keeping a set of transactions which is distributed to all nodes in the network
• Chain - a sequence of blocks in a specific order
• Miners - specific nodes which perform the block verification process before adding anything to the blockchain structure
• Consensus (consensus protocol) - a set of rules and arrangements to carry out blockchain operations

Below you can see an example of the linked lists that help to create this in Blockchain Architecture:

Often described as 'spreadsheets in the sky' - these present a really unique way to use Chaos Toolkit. The goal of this experiment is to check that a chain exists and can be called after a simple transaction. No rollbacks should be supported since a blockchain should be immutable. To test this here's our first experiment:

{
    "version": "1.0.0",
    "title": "Can we make a new transaction?",
    "description": "The system should respond to a transaction request.",
    "tags": ["tx"],
    ...

What we are trying to observe with our experiments is the Steady State Hypothesis, which is a fancy way of asking we will take a snapshot of what "normal" looks like for your service for or baseline, and then again following the experiment to observe any changes. This is the first phase of a full chaos engineering process.

We can check the steady state by introducing this probe to ensure that the chain exists after a blockchain transaction:

    "steady-state-hypothesis": {
        "title": "Chain exists",
        "probes": [
            {
                "type": "probe",
                "name": "chain-exists",
                "tolerance": 200,
                "provider": {
                    "type": "http",
                    "timeout": 5,
                    "url": "http://127.0.0.1:5000/chain"
                }
            }
        ]
    },

Actions vs Probes

The basic structure is simply a list of probes and actions.

Probes can help you to check the system state, but actions introduce new information or a change to the system being tested. For example, checking that the chain exists is a probe but creating a new transaction is an action.

The next step is to include the specific arguments called for by our blockchain.py file, defined below:

"method": [
        {
            "type": "action",
            "name": "make-new-transaction",
            "provider": {
                "type": "http",
                "timeout": 1,
                "url": "http://127.0.0.1:5000/transactions/new",
                "method": "POST",
                "headers": {
                    "Content-Type": "application/json"
                },
                "arguments": {
                    "sender": "me",
                    "recipient": "new-other-address",
                    "amount": 20
                }
            }
        },
        {
            "type": "probe",
            "name": "check-chain",
            "provider": {
                "type": "http",
                "url": "http://127.0.0.1:5000/mine"
            }
        },
        {
            "type": "action",
            "name": "mine-block",
            "provider": {
                "type": "http",
                "timeout": 3,
                "url": "http://127.0.0.1:5000/mine"
            }
        },
        {
            "type": "probe",
            "name": "check-chain",
            "provider": {
                "type": "http",
                "url": "http://127.0.0.1:5000/mine"
            }
        }
    ],

Finally, we reach the rollbacks! When designing an experiment, you should be aware of the capabilities of your system and also what it should be able to do.

For example, I could include code in my blockchain.py file that allows a user to delete a transaction which hasn’t been mined yet; however, this would violate the operation of a blockchain because blockchains derive trust from immutability.

A software production rollback is simply the returning of a codebase to an earlier, more stable version. If you are curious about the execution workflow that would typically result in a rollback, you can understand those conditions from the CTK reference guide. Then the snippet that usually handles rollbacks is simply:

"rollbacks": [
    ]}

Running the Transaction Experiment:

$ python blockchain.py

$ chaos run testTransaction.json

Understanding Decentralized and Distributed Consensus

The consensus mechanism is how you determine the global truth of a blockchain transaction. We will demo a simple Proof of Work, which essentially is an extremely difficult computation to solve a math puzzle. We will then start a second chain as an example of the branching that may occasionally occur; this blockchain resolves itself by taking the longest chain.

Centralized, decentralized and distributed network models by Paul Baran (1964), part of a RAND Institute study to create a robust and nonlinear military communication network . 

1. We should ensure that the chain exists on two separate nodes.
2. Simulate activity.
3. Check the chains exist still.
4. Resolve the chains to identify the global truth.

Steps 1 and 3 should be familiar to you and I invite you to try coding them yourself! Step 2 brings an opportunity to show another use case for CTK. So far we have used the http provider, but we may also use a python provider:

{
    "type": "action",
    "name": "simulate activity",
    "provider": {
        "type": "python",
        "module": "os",
            "func": "system",
            "arguments": {
                "command": "python -c \"import activity; activity.run(100)\""
            }
    }
}

If a function takes arguments, check the documentation so you know how the arguments are titled and you can list them in the standard JSON format. Here, the argument is a command and the input is '"python -c \"import activity; activity.run(100)\""

The activity.py file run by simulate activity randomly posts transactions from either of the two nodes and occasionally mines a block. For reference, the approximate probability of posting a transaction to any of the two chains is 75% and accordingly, the probability of mining from either of the two chains is 25%.

Finally, we want to resolve the chains. This is an http request which we’ve learned earlier, and I again invite you to try your hand at it!

Running the Consensus Experiment

In your command line, you should create the blockchain and start both nodes (here we use 127.0.0.1:5000 and 127.0.0.1:5001) before running the experiment:

$ python blockchain.py --port 5000

Then open a new terminal and run:

$ python blockchain.py --port 5001

Then, you can run the experiment by using the command:

$ chaos run testConsensus.json

Some Key Takeaways

1. The Chaos Toolkit tests what states are actually possible for your system, so that is context specific - walking through this tutorial, you have seen how the experiments should be tailored to how the system should work (in this case, no rollback options).

2. On an even more abstracted level, the CTK is an automation tool - note how the consensus test essentially automates a lot of usage and then checks. This actually makes it incredibly powerful even if you aren’t specifically running a chaos experiment because there is a set process that you create which is replicable. Think of experiments as blueprints for what you want to try!

Diving into blockchain with experimentation is a great way to flex your mind around immutability and stable states. If you have any questions or would like some insights into similar topics, find us at Reliably on Hackernoon.

Chaos Toolkit is an open-source project hosted on Github. If you have any issues then raise them on Github, and if you’d like to contribute, start here!

This Chaos Engineering with Blockchain tutorial was developed by Yolanne Lee. The blockchain tutorial linked in the beginning was developed by Daniel van Flymen.