Access External Data in Solidity: A How-To Guide

Original Image by ipopba from Getty Images Pro

Accessing external data with solidity or smart contract can seem daunting when you first attempt this. How can you get non-determanistic data into a determanistic system? And more importantly, how can you do it in a decentralized, safe, secure way, so that your dApp doesn't depend on the trustworthiness of the data source?

These are some of the important questions that engineers ask themselves when building smart contracts. Let's just jump right into an example.

Accessing external food and drug data with solidity

For our example, let's get data on food enforcement (like recalls) in the United States. This might be used to power a dApp doing analytics on the health of the US, creating an immutable historical record, or maybe even making a bet. We can get all of this data from the FDA website using their API. APIs are ways that technologies communicate with each other, and is currently the standard for fetching and sending data on the web that you know and love today.

We can run a sample call of this data by hitting the following link. This will return the number of food recalls for the year 2015.

https://api.fda.gov/food/enforcement.json?search=report_date:[2015101+TO+20151231]&limit=1

Our sample return looks like this:

{
  "meta": {
    "disclaimer": "Do not rely on openFDA to make decisions regarding medical care. While we make every effort to ensure that data is accurate, you should assume all results are unvalidated. We may limit or otherwise restrict your access to the API in line with our Terms of Service.",
    "terms": "https://open.fda.gov/terms/",
    "license": "https://open.fda.gov/license/",
    "last_updated": "2020-11-01",
    "results": {
      "skip": 0,
      "limit": 1,
      "total": 9118
    }
  },
  "results": [
    {...
    }
  ]
}

We changed the object in the "results" section to just be `{...}` since we are only going to be looking at that `total` object.

We can see here, that there were 9,118 recalls for the year 2015, and we want to get this in our smart contract. If you want to get more customized data, you can learn how to change the request URL to get different data by checking the FDA documentation.

Centralized vs Decentralized

For this first example, we are going to use just one data source (the FDA API) and one blockchain oracle (we will explain what that is in a minute). For those of us who are familiar with smart contracts we know that we always want them decentralied, and we need to use decentralized oracles. For developing your smart contracts, starting centralized is fine, just remember to make it decentralized when you go to production! We will go over how to do that towards the end of this article as well, so don't worry.

Our Solidity Code

Awesome, let's look at what our simple solidity smart contract is going to look like.

pragma solidity ^0.6.0;

import "https://raw.githubusercontent.com/smartcontractkit/chainlink/develop/evm-contracts/src/v0.6/ChainlinkClient.sol";

contract FDAConsumer is ChainlinkClient {
  
    uint256 public enforcementActions;
    address private oracle;
    bytes32 private jobId;
    uint256 private fee;
    
    constructor() public {
        setPublicChainlinkToken();
        oracle = 0x2f90A6D021db21e1B2A077c5a37B3C7E75D15b7e;
        jobId = "29fa9aa13bf1468788b7cc4a500a45b8";
        fee = 0.1 * 10 ** 18; // 0.1 LINK
    }
    
    function requestEnforcementData() public returns (bytes32 requestId) 
    {
        Chainlink.Request memory request = buildChainlinkRequest(jobId, address(this), this.fulfill.selector);
    
        request.add("get", "https://api.fda.gov/food/enforcement.json?search=report_date:[2015101+TO+20151231]&limit=1");
        request.add("path", "meta.results.total");
        return sendChainlinkRequestTo(oracle, request, fee);
    }
    
    function fulfill(bytes32 _requestId, uint256 _enforcementActions) public recordChainlinkFulfillment(_requestId)
    {
        enforcementActions = _enforcementActions;
    }
}

This is actually the whole contract, and we will get the number of enforcement actions to be saved to the `enforcementActions` variable.

If you're familiar with remix and sending Chainlink requests, you can deploy this contract on Kovan right now by clicking this link! If you're unfamiliar, don't worry, we will step through it here.

The setup

The start of this smart contract is pretty basic, but we import a chainlink package and have our contract inherit it by saying that `FDAConsumer is ChainlinkClient`. This allows us to make our Chainlink API call. Now let's walk through the steps to make this call. If you'd like a beginner walkthrough you can check the Chainlink documentation. This will also go through the walkthrough as well, but if you get lost, feel free to reference the walkthrough.

1. Know your response type

We want to return a uint256, since we are looking for the number of enforcement actions taken in 2015, so this will be a number.

Now that we know our response type, we can choose our oracle.

2. Choose your oracle

We need to find an Chainlink oracle that can return a uint256 after making an httpget request. We can look at some node listing services like market.link to do this. We can then search for "get > uint256" to find an oracle that can make this for us.

Each oracle has a list of jobs that they can do, and we need to get:

1. The oracle address
2. The Job ID
3. The Chainlink node fee

In order to make a request through that oracle. This will be important later when you go to make this decentralized, since you'll want to pull data from many independent oracles. For now, we can use some of the testing Chainlink oracles on the Kovan network. We hard coded these two values in our example, they are as follows:

oracle = 0x2f90A6D021db21e1B2A077c5a37B3C7E75D15b7e;
jobId = "29fa9aa13bf1468788b7cc4a500a45b8";
fee = 0.1 * 10 ** 18; // 0.1 LINK

You'll notice the oracle variable is an address and the jobId is a bytes32. The fee is set per Chainlink node, and is the "oracle gas" of the transaction. It's similar to the transaction gas that you're used to, but for Oracles, and instead of paying ETH, you pay LINK. We will show you how to do that in a little. You'll also want to call the `setPublicChainlinkToken` method like we do in the full example.

3. Make the Request

All Chainlink API calls follow the Request and Recieve style. This means our smart contract will make a request in 1 transaction, and the chainlink node will then make a request back to our smart contract later with the data. This means we need to define:

1. A request function for us to call
2. A fulfill function for the chainlink oracle to callback

We build our main chainlink request like so:

    function requestEnforcementData() public returns (bytes32 requestId) 
    {
        Chainlink.Request memory request = buildChainlinkRequest(jobId, address(this), this.fulfill.selector);
        request.add("get", "https://api.fda.gov/food/enforcement.json?search=report_date:[2015101+TO+20151231]&limit=1");
        request.add("path", "meta.results.total");
        return sendChainlinkRequestTo(oracle, request, fee);
    }

You can see here, that we add the API URL that we got from the FDA website into the request.add parameter. We also only want the total number of enforcements so we can return a uint256, so we "walk the json path" of "meta -> results -> total" to get his number syntax. Remember that storing data in solidity is expensive, so we don't want to pull tons of data that we don't need!

In the

buildChainlinkRequest

this.fulfill.selector

fulfill

Then finally, at the bottom we send the request for the Chainlink node to fulfill. Great! Let's define the callback function now.

4. The Callback Function

Our callback function is really simple:

function fulfill(bytes32 _requestId, uint256 _enforcementActions) public recordChainlinkFulfillment(_requestId)
    {
        enforcementActions = _enforcementActions;
    }

All we do here is set the `enforcementActions` variable so that we can view it in our smart contract. Remember, the Chainlink node that we specificed is the one making this function call!

5. Deploy the smart contract

Now that we've set it all up, we can deploy the smart contract. If you want to just use the remix link here, you can, it has all the code we need and can be deployed from remix. But to get started, we need 3 things:

1. A metamask or eth wallet
2. Kovan ETH
3. Kovan LINK

You can get Kovan ETH using the ETH faucet and Kovan LINK using the LINK faucet. Then, make sure your web3 or metamask wallet is set to the kovan network.

Now, compile the contract, and then deploy to the kovan network. Make sure you use "Injected Web3"

Once it's deployed, you have to fund the contract with LINK. Copy the address with the little copy button, and then send it LINK from your web3 wallet. 2 Kovan LINK should be more than fine!

Once the transaction completes, you can run the

requestEnforcementData

You'll then have to wait a little after it confirms. The Chainlink node now has your request, and is waiting to send it back!

Once it does (maybe a minute or two), you can check the status by hitting the blue button to check on the result.

And fantastic! We've access the external FDA data in solidity!

Continuing on

This showed us how to access or fetch external data with solidity, but as we spoke, this is a centralized way. In order to make it decentralized we want to do two things:

1. Access external data through multiple Chainlink oracles
2. Access multiple external data sources

If you repeat this process with different Chainlink nodes and different APIs, the simplest way to make it decentralized. Or you can piggy back off someone else already setting this up! There are a number of data sources that already allow public access in a decentralized manner. Some of the most popular forms of data are cryptocurrency prices, and you can see a list of the on the Price Feeds page of Chainlink.

These are really easy to get into your code, since they are just a few functions. You can learn how to integrate the price feeds by checking the documentation.

And that's it! If you were successful, be sure to tag @patrickalphac on twitter with your successful deployment of this data! Join the Chainlink community and start building some amazing smart contracts!