Eternal Storage : Building Dapps For the Future

TLDR: Eternal Storage provides the stable and flexible base needed to build and maintain long-lived decentralized applications.

Once deployed to a network, smart contract logic cannot be altered. However, developers are not perfect. Bugs are bound to loom in our logic no matter how carefully we plan our logic, test it, and have it externally audited. Iteration is inevitable in any complicated application, decentralized or not. Smart contracts will need to be updated and redeployed during the lifetime of a decentralized app (dapp). In this post we will learn about the Eternal Storage pattern and why it's an elegant answer to our need for mutability in an immutable world.

This article makes the assumption that the reader has some basic understanding of writing smart contracts in Solidity and deploying and testing them locally (we like Truffle). Some patterns are based off examples from EIP#930, fravoll, and RocketPool.

To illustrate the typical evolution of a dapp, let's consider a potential implementation of Decentralized DNS. The most basic functionality required in a first iteration might include:

Claiming a domain

Only allowed if domain is not already claimed

Releasing a domain

Only allowed by the owner

Updating a domain's records

Type (A, CNAME, MX, etc)
Value
TTL

Listing all domains owned by a particular person

How might the logic for this feature set look like?

Initial Implementation

pragma solidity >=0.6.4 <0.7.0;
pragma experimental ABIEncoderV2;

contract DNS {
    struct DomainRecord {
        string _type; // "A", "CNAME", "MX", etc...
        string value;
        uint ttl;
    }

    struct Domain {
        address owner;
        DomainRecord[] records;
    }

    mapping(string => Domain) public domains;

    function claim(string memory name) public {
        address domainOwner = domains[name].owner;
        if (domainOwner == address(0)) {
            domains[name].owner = msg.sender;
            return;
        }
        require(domainOwner == msg.sender, "Domain has already been claimed");
    }

    function updateRecords(string memory name, DomainRecord[] memory records) public {
        claim(name);

        delete domains[name].records;
        uint size = records.length;
        for (uint i = 0; i < size; i++) {
          domains[name].records.push(records[i]);
        }
    }

    function release(string memory name) public {
        require(domains[name].owner == msg.sender, "Sender does not own this domain");
        domains[name].owner = address(0);
    }

    function recordsForName(string memory name) public view
            returns (DomainRecord[] memory recordsForName_)
    {
        recordsForName_ = domains[name].records;
    }
}

[View DNS_v1.sol]

This contract allows users to perform basic actions we associate with domain names: claim them, modify their records, and release them. The data associated with these actions are stored directly in the contract itself. Its deployed address is

0x00000001

. We register

test.com

with our wallet at address

0x99999999

and confirm we do own it:

truffle> let dns = await DNS.deployed()
undefined
truffle> dns.claim("test.com")
{...}
truffle> dns.domains("test.com")
'0x99999999'

Great start. Users start registering domain names in our dapp, but a few weeks later someone points out that

release

doesn’t clear the records on a domain before giving up ownership. We decide that's a bug, so we have to update the contract by updating the

release

method:

 function release(string memory name) public {
     require(domains[name].owner == msg.sender, "Sender does not own this domain");
+    delete domains[name].records; // now it clears records during release
     domains[name].owner = address(0);
 }

[View DNS_v2.sol]

We deploy this version of the contract with release clearing the records. Its deployed address is now

0x00000002

. The address has changed, that's bad. Let's test our new release logic against

test.com

anyway:

truffle> let dns = await DNS.deployed()
undefined
truffle> dns.release("test.com")
Uncaught:
Error: Returned error: VM Exception while processing transaction: revert Sender
does not own this domain -- Reason given: Sender does not own this domain.
truffle> dns.domains("test.com")
'0x00000000'

What happened? Didn't we claim

test.com

already? Unfortunately data stored in a contract doesn't automatically migrate over on subsequent deploys of new versions of the contract.

Two issues are now apparent:

Version 2 of the contract is missing all the data saved from user interaction with Version 1 of the contract.
Users need to know that Version 1 of the contract is now deprecated and that the newest version now lives at
```
0x00000002
```
.

Couldn’t we just migrate the data from Version 1 to Version 2 ourselves? Yes, but writes are expensive and paying the cost to copy data becomes prohibitively expensive as users create more data in our contracts. Not to mention we now must be careful not to introduce any errors as part of the data migration process.

What if we just let users know a new version has been deployed at

0x00000002

via our website or social media? It’s highly likely more than a few users will miss these announcements which will lead to very poor user experience and distrust in the dapp/developers.

Maybe we can get around this with some cleverness?

Proxy/Delegate

First, let's solve the problem where users of the old contract don't know about the newly deployed contract. We create a proxy contract, that points to another contract (our evolving dapp), which allows us to update the underlying contract at any point in the future.

pragma solidity >=0.5.14 <0.6.0;

contract Proxy {
    address private delegate;
    address private owner = msg.sender;

    // allows us to update where the proxy contract points to
    function upgradeDelegate(address newDelegateAddress) public {
        require(msg.sender == owner, "Sender is not owner");
        delegate = newDelegateAddress;
    }

    // fallback function
    function() external payable {
        assembly {
            let _target := sload(0)
            calldatacopy(0x0, 0x0, calldatasize)
            let result := delegatecall(gas(), _target, 0x0, calldatasize, 0x0, 0)
            returndatacopy(0x0, 0x0, returndatasize)
            switch result case 0 {revert(0, 0)} default {return (0, returndatasize)}
        }
    }
}

[View Proxy.sol]

The

Proxy

contract leverages Solidity's unnamed

function()

fallback to forward calls it receives to a contract at any delegate address set by us via

upgradeDelegate()

. It gets deployed and lives at

0x11111111

. We set the delegate address to

0x00000002

. We let users know they can use our dapp via

0x11111111

. By updating the

Proxy

instance to point to the newest version of the

DNS

contract every time we redeploy it, we now have a reliable static address that users can trust.

What about handling data across updates? One interesting property of

delegatecall

is that the

Proxy

contract becomes the context for storage and all read/writes. Storage operations that normally would happen on the

DNS

contract if we called it directly instead happen on the

Proxy

contract itself. This is a nice side effect because we no longer have to worry about migrating data each time we redeploy the

DNS

contract. However, there’s an important caveat to this. Data structure declarations cannot be removed, or even reordered, because existing stored data structure will not match and unintended behavior will likely occur (example exploit).

What if we tried completely separating our data storage and business logic?

Eternal Storage

Eternal Storage contracts consist of mappings from

bytes32

type to other usable base types along with their respective getters and setters. This forms a flat key-value store. Mappings give us full flexibility in how we label and organize our data.

pragma solidity >=0.5.14 <0.6.0;

contract EternalStorage {
    mapping(bytes32 => string)     private stringStorage;
    mapping(bytes32 => bool)       private boolStorage;
    ...

    // this will help us lock down access to only authorized addresses. ignore it for now
    modifier onlyNetworkContract() {
        if (boolStorage[keccak256(abi.encodePacked("contract.storage.initialized"))] == true) {
            require(boolStorage[keccak256(abi.encodePacked("access.role", "contract", msg.sender))] == true, "Sender is not a contract in the network");
        }
        _;
    }

    constructor() public {
        boolStorage[keccak256(abi.encodePacked("access.role", "owner", msg.sender))] = true;
    }

    // Get
    function getString(bytes32 _key) external view returns (string memory) {
        return stringStorage[_key];
    }
    function getBool(bytes32 _key) external view returns (bool) {
        return boolStorage[_key];
    }
    ...

    // Set
    function setString(bytes32 _key, string memory _value) public onlyNetworkContract {
        stringStorage[_key] = _value;
    }
    function setBool(bytes32 _key, bool _value) external onlyNetworkContract {
        boolStorage[_key] = _value;
    }
    ...

    // Delete
    function deleteString(bytes32 _key) external onlyNetworkContract {
        delete stringStorage[_key];
    }
    function deleteBool(bytes32 _key) external onlyNetworkContract {
        delete boolStorage[_key];
    }
    ...

    // And similar get/set/delete declarations and functions for the types:
    // "address", "bytes", "int", "bytes32"
}

[View EternalStorage.sol]

To keep things sane and organized, we will use dot notation to address our pseudo objects and their properties. Interacting with our Eternal Storage contract may look something like this:

eternalStorage.setAddress(keccak256(abi.encodePacked("foo.bar.owner"), msg.sender))
eternalStorage.setString(keccak256(abi.encodePacked("foo.bar.title"), “Hello World”))

Keccak is the algorithm that became SHA-3, but NIST changed an implementation detail in the standardization process. Chances of collision are extremely low, giving us 64^36 addressable keys for each mapping. The function

keccak256()

expects

bytes

array and returns

bytes32

. The function

abi.encodePacked()

expects any type of data and returns it encoded in the smallest byte representation possible via a

bytes

array. Hashing our keys like this allows us to stay at 32 bytes regardless of key length, helping save on gas costs.

Setters are locked down to allow access only from registered contracts in the network once admins mark the boolean map with key

“contract.storage.intialized”

to be true.

Unfortunately adding Eternal Storage increases the complexity in our contracts. See how our original

DNS

contract logic changes:

// Old
function updateRecords(string memory name, DomainRecord[] memory records) public {
    claim(name);

    delete domains[name].records;
    uint size = records.length;
    for (uint i = 0; i < size; i++) {
        domains[name].records.push(records[i]);
    }
}

// New
EternalStorage eternalStorage;
...
function updateRecords(string memory name, DomainRecord[] memory records) public {
    uint length = records.length;
    eternalStorage.setUint(keccak256(abi.encodePacked("domains", name, "records", "length")), length);

    for (uint i = 0; i < length; i++) {
        eternalStorage.setString(keccak256(abi.encodePacked("domains", name, "records", i, "type")), records[i].recordType);
        eternalStorage.setString(keccak256(abi.encodePacked("domains", name, "records", i, "value")), records[i].value);
        eternalStorage.setUint(keccak256(abi.encodePacked("domains", name, "records", i, "ttl")), records[i].ttl);
}

Luckily, we can utilize helper functions to keep our business logic free of the complexity introduced above by using Eternal Storage.

Storage Operations

One common data type we will have to reintroduce is lists/arrays.

StringListStorage

highlights the many steps involved in implementing a string array on top of Eternal Storage and why it is important to separate this into a helper.

pragma solidity >=0.5.14 <0.6.0;

contract StringListStorage {

    EternalStorage es;

    constructor(address _es) public {
        es = EternalStorage(_es);
    }


    // Get
    function getListLength(bytes32 _key) external view returns (uint) {
        return es.getUint(keccak256(abi.encodePacked(_key, "length")));
    }
    function getListItem(bytes32 _key, uint _index) external view returns (string memory) {
        return es.getString(keccak256(abi.encodePacked(_key, "item", _index)));
    }
    ...

    // Set
    function pushListItem(bytes32 _key, string memory _value) public {
        uint length = es.getUint(keccak256(abi.encodePacked(_key, "length")));
        es.setString(keccak256(abi.encodePacked(_key, "item", length)), _value);
        es.setUint(keccak256(abi.encodePacked(_key, "length")), length + 1);
    }
    function removeUnorderedListItem(bytes32 _key, uint _index) external {
        require(_index < es.getUint(keccak256(abi.encodePacked(_key, "length"))), "List index out of bounds");
        uint length = es.getUint(keccak256(abi.encodePacked(_key, "length")));
        if (_index < length - 1) {
            string memory lastItem = es.getString(keccak256(abi.encodePacked(_key, "item", length - 1)));
            es.setString(keccak256(abi.encodePacked(_key, "item", _index)), lastItem);
        }
        es.setUint(keccak256(abi.encodePacked(_key, "length")), length - 1);
    }
    ...

    // See complete contract linked below
}

[View StringListStorage_v1.sol]

Now that we have a nice wrapper for string arrays, we can create a helper around our previous idea of a

Domain

struct.

DomainModel

shows usage of the

StringListStorage

helper and how we are able to maintain friendly method interfaces through structs despite the underlying storage mechanisms having no notion of them.

pragma solidity >=0.5.14 <0.6.0;
pragma experimental ABIEncoderV2;

contract DomainModel {
    struct DomainRecord {
        string recordType; // "A", "CNAME", "MX", etc...
        string value;
        uint ttl;
    }

    EternalStorage es;
    StringListStorage sls;

    constructor(address _es) public {
        es = EternalStorage(_es);
        sls = new StringListStorage(_es);
    }


    function getOwner(string memory name) public view
        returns (address owner_)
    {
        address domainOwner = es.getAddress(keccak256(abi.encodePacked("domains", name, "owner")));
        owner_ = domainOwner;
    }

    function setOwner(string memory name, address owner) public {
        if (owner == address(0)) {
            return;
        }
        es.setAddress(keccak256(abi.encodePacked("domains", name, "owner")), owner);
        sls.pushListItem(keccak256(abi.encodePacked("owners", owner, "domains")), name);
    }

    ...

    // See complete contract linked below
}

[View DomainModel_v1.sol]

With

DomainModel

, we now have an idea of how we can organize larger data structures around Eternal Storage. Finally, let's use

DomainModel

inside our

DNS

contract.

pragma solidity >=0.5.14 <0.6.0;
pragma experimental ABIEncoderV2;

contract DNS {
    DomainsModel _domains;

    constructor(address _es) public {
        _domains = new DomainsModel(_es);
    }

    function claim(string memory name) public {
        address domainOwner = _domains.getOwner(name);
        if (domainOwner == address(0)) {
            _domains.setOwner(name, msg.sender);
            return;
        }
        require(domainOwner == msg.sender, "Domain has already been claimed");
    }

    function updateRecords(string memory name, DomainsModel.DomainRecord[] memory records) public {
        claim(name);
        _domains.setRecords(name, records);
    }

    function release(string memory name) public {
        address domainOwner = _domains.getOwner(name);
        require(domainOwner == msg.sender, "Sender does not own this domain");
        _domains.setRecords(name, []);
        _domains.deleteOwner(name);
    }

    function domains(string memory name) public view
        returns (address owner_)
    {
        address domainOwner = _domains.getOwner(name);
        owner_ = domainOwner;
    }


    function recordsForName(string memory name) public view
            returns (DomainsModel.DomainRecord[] memory recordsForName_)
    {
        DomainsModel.DomainRecord[] memory records = _domains.getRecords(name);
        recordsForName_ = records;
    }

    function domainNamesForOwner(address owner) public view
            returns (string[] memory domainNamesForOwner_)
    {
        string[] memory domainNames = _domains.getDomainNamesForOwner(owner);
        domainNamesForOwner_ = domainNames;
    }

    function domainNamesForSender() public view
            returns (string[] memory domainNamesForSender_)
    {
        string[] memory domainNames = _domains.getDomainNamesForOwner(msg.sender);
        domainNamesForSender_ = domainNames;
    }
}

[View DNS_v3.sol]

This isn't so bad. In fact it reads much more nicely than before. Now that we are happy with our storage, let's wrap it all up and secure our new network of contracts.

Authorization & Upgradeability

We cannot talk about authorization and upgradeability separately as each implementation depends on the other.

Authorization

Our contracts must be explicit about who can and cannot access its methods.
Eternal Storage must only allow storage access from certain contracts/senders for write operations.

Upgradeability

Without having unrestricted access to all data in the Eternal Storage contract, Admins must still retain the ability to add, upgrade, and authorize new contracts to the network.
Our contracts must be able to retrieve the most up-to-date address for any contract it wishes to make a call to.

Once we set

"contract.storage.intialized"

, only contracts registered at the boolean map with key

“access.role”, “contract”, CONTRACT_ADDRESS

will be able to modify storage. Unfortunately, admins will locked out and now unable to add/upgrade/authorize contracts. Additionally with our current logic, any sender can still bypass domain ownership checks in the

DNS

contract by calling the methods in

DomainModel

and

StringListStorage

directly.

Upgradeability, and any other actions admins may need to perform can be implemented as... contracts! Admins should never need fully unrestricted access to Eternal Storage. Limiting the power admins hold also reduces risk and improves overall trust in the system.

We need the following for authorization/upgradeability:

A way to lookup if a sender is a contract in the network or is an admin so we can limit access by role.

“access.role”, “contract”, _SenderAddress

(boolean)

“access.role”, “admin”, _SenderAddress

(boolean)

A way to lookup the address for the most recent version of any contract in the network so we can limit access to certain methods to a specific contract(s) and so we can make sure we are always calling the most up-to-date version of a contract.

“contract”, _contractName, “address”

(address)

Let’s jump ahead a little bit and add a

Base

contract that implements these checks.

pragma solidity >=0.5.14 <0.6.0;

contract Base {
    EternalStorage internal es;

    modifier onlyNetworkContracts() {
        require(
            hasRole("contract") == true,
            "Sender is not a contract in the network"
        );
        _;
    }

    modifier onlyNetworkContract(string memory _contractName) {
        address contractAddress = getContractAddress(_contractName);
        require(contractAddress == msg.sender, "Incorrect or outdated contract access used");
        _;
    }

    modifier onlyRole(string memory _role) {
        require(
            hasRole(_role) == true,
            "Sender doesn't have matching role"
        );
        _;
    }

    constructor(address _es) public {
        es = EternalStorage(_es);
    }


    function getContractAddress(string memory _contractName) public view returns(address) {
        address contractAddress = es.getAddress(
            abi.encodePacked(keccak256(abi.encodePacked("contracts", _contractName, "address")))
        );
        require(address(contractAddress) != address(0x0), "Contract not found");
        return contractAddress;
    }

    function hasRole(string memory _role) public view returns(bool) {
        return es.getBool(
            abi.encodePacked(keccak256(abi.encodePacked("access.role", _role, msg.sender)))
        ) ;
    }
}

[View Base.sol]

The

Base

contract gives us:

constructor that initializes a reference to Eternal Storage.
modifiers that allow us to guard methods to any or only specific contracts on the network or admins.
a method to get the address for the most recent version of any contract type deployed on the network.

pragma solidity >=0.5.14 <0.6.0;

contract ContractAdmin is Base {
    constructor(address _es) Base(_es) public {}

    function addContract(
        string memory _name,
        address _contractAddress,
        string memory _contractAbi
      ) public onlyRole("owner") {
        require(_contractAddress != address(0x0), "Invalid contract address");
        address existingContractName = es.getAddress(keccak256(abi.encodePacked("contracts", _name, "address")));
        require(existingContractName == address(0x0), "Contract name is already in use");
        bool existingContractAddress = es.getBool(keccak256(abi.encodePacked("access.role", "contract", _contractAddress)));
        require(!existingContractAddress, "Contract address is already in use");

        es.setAddress(keccak256(abi.encodePacked("contracts", _name, "address")), _contractAddress);
        es.setString(keccak256(abi.encodePacked("contracts", _name, "abi")), _contractAbi);
        es.setBool(keccak256(abi.encodePacked("access.role", "contract", _contractAddress)), true);
    }

    ...

    // See complete contract linked below
}

[View ContractAdmin.sol]

The

ContractAdmin

contract gives admins (via

onlyRole("owner")

) the ability to add and upgrade contracts with required authorization-related entries being updated automatically.

Now how will the rest of our contracts look after they inherit the

Base

contract and lock down access to their methods?

pragma solidity >=0.5.14 <0.6.0;

contract StringListStorage is Base {
    constructor(address _es) Base(_es) public {}

    // Getters remain the same
    function getListLength(bytes32 _key) external view returns (uint) {...}
    function getListItem(bytes32 _key, uint _index) external view returns (string memory) {...}

    // Setters have the onlyNetworkContracts modifier applied to them
    function pushListItem(bytes32 _key, string memory _value) public onlyNetworkContracts {...}
    function removeUnorderedListItem(bytes32 _key, uint _index) external onlyNetworkContracts {...}

    ...

    // See complete contract linked below
}

[View StringListStorage_v2.sol]

The

StringListStorage

contract locks down storage-modifying methods to only contracts in the network.

pragma solidity >=0.5.14 <0.6.0;
pragma experimental ABIEncoderV2;

contract DomainModel is Base {
    struct DomainRecord { ... }

    constructor(address _es) Base(_es) public {}

    ...

    // this is method and the other setters are now only accessible by the most recently deployed DNS contract
    function setOwner(string memory name, address owner) onlyNetworkContract("DNS") public {
        ...
        // notice how we are using getContractAddress to discover the address of the StringListStorage contract
        StringListStorage sls = StringListStorage(getContractAddress("StringListStorage"));
        ...
        sls.pushListItem(keccak256(abi.encodePacked("owners", owner, "domains")), name);
    }

    ...

    // See complete contract linked below
}

[View DomainModel_v2.sol]

The

DomainModel

contract locks down storage modifying methods to only the currently deployed DNS contract. Additionally, it now fetches the most current address of

StringListStorage

from Eternal Storage rather than having the address passed to its constructor.

pragma solidity >=0.5.14 <0.6.0;
pragma experimental ABIEncoderV2;

contract DNS is Base {
    constructor(address _es) public Base(_es) {}

    function claim(string memory name) public {
        // notice how we are using getContractAddress to discover the address of the DomainModel contract
        DomainModel dm = DomainModel(getContractAddress("DomainModel"));
        address domainOwner = dm.getOwner(name);
        if (domainOwner == address(0)) {
            dm.setOwner(name, msg.sender);
            return;
        }
        require(domainOwner == msg.sender, "Domain has already been claimed");
    }

    function updateRecords(string memory name, DomainModel.DomainRecord[] memory records) public {
        claim(name);
        // another example of using getContractAddress
        DomainModel dm = DomainModel(getContractAddress("DomainModel"));
        dm.setRecords(name, records);
    }

    function release(string memory name) public {
        DomainModel dm = DomainModel(getContractAddress("DomainModel"));
        address domainOwner = dm.getOwner(name);
        require(domainOwner == msg.sender, "Sender does not own this domain");
        dm.deleteOwner(name);
    }

    function domains(string memory name) public view
        returns (address owner_)
    {
        DomainModel dm = DomainModel(getContractAddress("DomainModel"));
        address domainOwner = dm.getOwner(name);
        owner_ = domainOwner;
    }


    function recordsForName(string memory name) public view
            returns (DomainModel.DomainRecord[] memory recordsForName_)
    {
        DomainModel dm = DomainModel(getContractAddress("DomainModel"));
        DomainModel.DomainRecord[] memory records = dm.getRecords(name);
        recordsForName_ = records;
    }

    function domainNamesForOwner(address owner) public view
            returns (string[] memory domainNamesForOwner_)
    {
        DomainModel dm = DomainModel(getContractAddress("DomainModel"));
        string[] memory domainNames = dm.getDomainNamesForOwner(owner);
        domainNamesForOwner_ = domainNames;
    }

    function domainNamesForSender() public view
            returns (string[] memory domainNamesForSender_)
    {
        DomainModel dm = DomainModel(getContractAddress("DomainModel"));
        string[] memory domainNames = dm.getDomainNamesForOwner(msg.sender);
        domainNamesForSender_ = domainNames;
    }
}

[View DNS_v4.sol]

The DNS contract does not lock down access to any of its methods since it’s directly user facing. It now fetches the most current address of the

DomainModel

contract from Eternal Storage rather than deploying an instance and storing that address in its constructor.

We now have a dapp with the following properties:

contracts can be upgraded over time.
existing contracts will be aware of newly deployed contracts.
data remains intact regardless of any contract upgrades.
old version of contracts are locked out of the network.
contract methods can be restricted by role/contract/address.

You can find all of the contracts and their revisions here.

Conclusion

Finally, we have a strong base for building and maintaining long-lived decentralized applications. This system isolates the trust to admins who can't directly access the eternal storage, and can only modify it by contract upgrades. Furthermore it allows for the implementation of clear and appropriate governance policies to further control the actions of the admins.

Hopefully this exercise has demonstrated why it's a good idea to use Eternal Storage in your future dapp.