Fuzzing Around: Smart Contract Testing Through Random Inputs

Testing smart contracts is really important. Why? Smart contracts are generally immutable in production, public, targets of hackers, and often involve significant financial implications. The short story? With smart contracts, you don’t want to screw up. You need to test a lot, test often, and test thoroughly. Smart contract testing involves all the usual suspects—unit testing, integration testing, automated testing— but also highly recommended is fuzzing—a way to bombard your code with a bunch of random inputs just to see what happens.

Let’s take a look at exactly what fuzzing is as a testing method and how you can easily include it in your software testing workflow using tools such as ConsenSys Diligence Fuzzing. We’ll walk through a tutorial using an example defi smart contact to see how it's done.

What is fuzzing testing and why would I use it?

Fuzzing testing (or fuzz testing) involves sending millions of invalid, unexpected, and (semi) random data against a smart contract in an effort to cause unexpected behavior and detect vulnerabilities. It’s a fast, efficient, brute-force way to test edge cases and scenarios you might not think of. It complements your other testing flows and your audits.

Fuzzing has been around for a while in traditional full-stack development, but a new class of tools is here that can apply fuzzing to smart contract testing in web3. Some of the fuzzing tools include the open source Echidna and MythX.

In this article, however, I will take a deep dive into Diligence Fuzzing, which offers fuzzing as a service—a pretty cool and easy way to fuzz.

To use Diligence Fuzzing, you annotate your smart contracts using Scribble. Basically, you use Scribble annotations in your code to tell the fuzzer what type of output to expect for that function.

It looks something like this:

/// #invariant {:msg "balances are in sync"} unchecked_sum(_balances) == _totalSupply;

You call the fuzzer from a web UI where you submit the smart contract code. You hit run, and approximately 10 minutes later, you get a report with the issues found, location of issues, coverage %, and more.

This is an easy and powerful way to check some edge cases.

Testing a DeFi Smart Contract

Let’s say we have a new DeFi protocol and corresponding token that we want to launch. DeFi is cutting-edge finance, fun to write, and it’s critical that we don’t have any bugs. DeFi smart contracts often involve millions (or more) in user funds. So we want to be sure our smart contracts are tested (and audited) as thoroughly as possible.

IMPORTANT NOTE: This code has vulnerabilities in it ON PURPOSE! It’s so we can show how fuzzing can catch these mistakes. Please don’t really use this code for anything.

Let’s get started.

An example using Diligence Fuzzing (FaaS)

Step 1: Set up a Diligence account.

First, we need our Diligence account. Sign up at Diligence. You get 10 hours of fuzzing for free to try things out.

Step 2: Create a new API key.

Click your account name at the top right, click “create new API key”, and give it a name. This API key allows us to connect to the FaaS.

Step 3: Set up your local environment.

Clone the following repository:

https://github.com/ConsenSys/scribble-exercise-1.git

From the command line, navigate to the repository folder, which will now be your project root folder. If necessary for your machine, activate a Python venv. Then, run the following commands to install your project dependencies:

$ npm i -g eth-scribble ganache truffle
$ pip3 install diligence-fuzzing

Step 4: Enter your API key.

Edit the .fuzz.yml file to use your Diligence account API key for the value of key.

Your resulting file should look like this:

.fuzz.yml

fuzz:
# Tell the CLI where to find the compiled contracts and compilation artifacts
build_directory: build/contracts

...
campaign_name_prefix: "ERC20 campaign"

# Point to your ganache node which holds the seed
rpc_url: "http://localhost:8545"

key: "DILIGENCE API KEY GOES HERE"

...

Step 5: Write and annotate your contract.

Now let’s check out our smart contract at contracts/vulnerableERC20.sol. Here we have a vulnerable token (as it’s named!) for our new DeFi protocol. We clearly need to test it as much as possible. So let’s add a check using Scribble that the fuzzer will hopefully catch.

Above the contract definition, add:

/// #invariant "balances are in sync" unchecked_sum(_balances) == _totalSupply;

And above the transfer function, add:

/// #if_succeeds msg.sender != _to ==> _balances[_to] == old(_balances[_to]) + _value;
/// #if_succeeds msg.sender != _to ==> _balances[msg.sender] == old(_balances[msg.sender]) - _value;
/// #if_succeeds msg.sender == _to ==> _balances[msg.sender] == old(_balances[_to]);
/// #if_succeeds old(_balances[msg.sender]) >= _value;

So now let’s see if the fuzzing tester will catch something. Notice the annotations we’ve added to the contract to help the tester understand what we expect to happen. And notice in the config the tester is set to run for 10 minutes max, so it might not catch everything.

Step 6: Fuzz around.

Now we’re ready to test! From the project root folder, run make fuzz to call the make file, which will compile, build, and send everything over to the FaaS.

Step 7: See the results.

Go back to your dashboard, and you should see something similar to the below. Wait several seconds for the campaign to start.

After it’s finished generating, you should see something similar to what is shown below:

We even see code coverage!

And after a few minutes, we see a failure!

Clicking through to properties, we see any violations. And yes we have a bug! Click on the line location to see the details of our potential code vulnerabilities.

We’ll stop there, but if you let the fuzzer keep running, it might find more!

Conclusion

Fuzzing can be a crucial step in your software development and testing process. It helps you identify potential vulnerabilities that otherwise may go unnoticed. By using it, you start to recognize and understand common pitfalls and challenges. Take advantage of tools such as Diligence Fuzzing, write better smart contracts, and become a better developer!