TLDR: Eternal Storage provides the stable and flexible base needed to build and maintain long-lived decentralized applications.
Once deployed to a network, smart contract logic cannot be altered. However, developers are not perfect. Bugs are bound to loom in our logic no matter how carefully we plan our logic, test it, and have it externally audited. Iteration is inevitable in any complicated application, decentralized or not. Smart contracts will need to be updated and redeployed during the lifetime of a decentralized app (dapp). In this post we will learn about the Eternal Storage pattern and why it's an elegant answer to our need for mutability in an immutable world.
This article makes the assumption that the reader has some basic understanding of writing smart contracts in Solidity and deploying and testing them locally (we like Truffle). Some patterns are based off examples from EIP#930, fravoll, and RocketPool.
To illustrate the typical evolution of a dapp, let's consider a potential implementation of Decentralized DNS. The most basic functionality required in a first iteration might include:
How might the logic for this feature set look like?
pragma solidity >=0.6.4 <0.7.0;
pragma experimental ABIEncoderV2;
contract DNS {
struct DomainRecord {
string _type; // "A", "CNAME", "MX", etc...
string value;
uint ttl;
}
struct Domain {
address owner;
DomainRecord[] records;
}
mapping(string => Domain) public domains;
function claim(string memory name) public {
address domainOwner = domains[name].owner;
if (domainOwner == address(0)) {
domains[name].owner = msg.sender;
return;
}
require(domainOwner == msg.sender, "Domain has already been claimed");
}
function updateRecords(string memory name, DomainRecord[] memory records) public {
claim(name);
delete domains[name].records;
uint size = records.length;
for (uint i = 0; i < size; i++) {
domains[name].records.push(records[i]);
}
}
function release(string memory name) public {
require(domains[name].owner == msg.sender, "Sender does not own this domain");
domains[name].owner = address(0);
}
function recordsForName(string memory name) public view
returns (DomainRecord[] memory recordsForName_)
{
recordsForName_ = domains[name].records;
}
}
This contract allows users to perform basic actions we associate with domain names: claim them, modify their records, and release them. The data associated with these actions are stored directly in the contract itself. Its deployed address is
0x00000001
. We register test.com
with our wallet at address 0x99999999
and confirm we do own it:truffle> let dns = await DNS.deployed()
undefined
truffle> dns.claim("test.com")
{...}
truffle> dns.domains("test.com")
'0x99999999'
Great start. Users start registering domain names in our dapp, but a few weeks later someone points out that
release
doesn’t clear the records on a domain before giving up ownership. We decide that's a bug, so we have to update the contract by updating the release
method: function release(string memory name) public {
require(domains[name].owner == msg.sender, "Sender does not own this domain");
+ delete domains[name].records; // now it clears records during release
domains[name].owner = address(0);
}
We deploy this version of the contract with release clearing the records. Its deployed address is now
0x00000002
. The address has changed, that's bad. Let's test our new release logic against test.com
anyway:truffle> let dns = await DNS.deployed()
undefined
truffle> dns.release("test.com")
Uncaught:
Error: Returned error: VM Exception while processing transaction: revert Sender
does not own this domain -- Reason given: Sender does not own this domain.
truffle> dns.domains("test.com")
'0x00000000'
What happened? Didn't we claim
test.com
already? Unfortunately data stored in a contract doesn't automatically migrate over on subsequent deploys of new versions of the contract. Two issues are now apparent:
0x00000002
.Couldn’t we just migrate the data from Version 1 to Version 2 ourselves? Yes, but writes are expensive and paying the cost to copy data becomes prohibitively expensive as users create more data in our contracts. Not to mention we now must be careful not to introduce any errors as part of the data migration process.
What if we just let users know a new version has been deployed at
0x00000002
via our website or social media? It’s highly likely more than a few users will miss these announcements which will lead to very poor user experience and distrust in the dapp/developers.Maybe we can get around this with some cleverness?
First, let's solve the problem where users of the old contract don't know about the newly deployed contract. We create a proxy contract, that points to another contract (our evolving dapp), which allows us to update the underlying contract at any point in the future.
pragma solidity >=0.5.14 <0.6.0;
contract Proxy {
address private delegate;
address private owner = msg.sender;
// allows us to update where the proxy contract points to
function upgradeDelegate(address newDelegateAddress) public {
require(msg.sender == owner, "Sender is not owner");
delegate = newDelegateAddress;
}
// fallback function
function() external payable {
assembly {
let _target := sload(0)
calldatacopy(0x0, 0x0, calldatasize)
let result := delegatecall(gas(), _target, 0x0, calldatasize, 0x0, 0)
returndatacopy(0x0, 0x0, returndatasize)
switch result case 0 {revert(0, 0)} default {return (0, returndatasize)}
}
}
}
The
Proxy
contract leverages Solidity's unnamed function()
fallback to forward calls it receives to a contract at any delegate address set by us via upgradeDelegate()
. It gets deployed and lives at 0x11111111
. We set the delegate address to 0x00000002
. We let users know they can use our dapp via 0x11111111
. By updating the Proxy
instance to point to the newest version of the DNS
contract every time we redeploy it, we now have a reliable static address that users can trust.What about handling data across updates? One interesting property of
delegatecall
is that the Proxy
contract becomes the context for storage and all read/writes. Storage operations that normally would happen on the DNS
contract if we called it directly instead happen on the Proxy
contract itself. This is a nice side effect because we no longer have to worry about migrating data each time we redeploy the DNS
contract. However, there’s an important caveat to this. Data structure declarations cannot be removed, or even reordered, because existing stored data structure will not match and unintended behavior will likely occur (example exploit).What if we tried completely separating our data storage and business logic?
Eternal Storage contracts consist of mappings from
bytes32
type to other usable base types along with their respective getters and setters. This forms a flat key-value store. Mappings give us full flexibility in how we label and organize our data.pragma solidity >=0.5.14 <0.6.0;
contract EternalStorage {
mapping(bytes32 => string) private stringStorage;
mapping(bytes32 => bool) private boolStorage;
...
// this will help us lock down access to only authorized addresses. ignore it for now
modifier onlyNetworkContract() {
if (boolStorage[keccak256(abi.encodePacked("contract.storage.initialized"))] == true) {
require(boolStorage[keccak256(abi.encodePacked("access.role", "contract", msg.sender))] == true, "Sender is not a contract in the network");
}
_;
}
constructor() public {
boolStorage[keccak256(abi.encodePacked("access.role", "owner", msg.sender))] = true;
}
// Get
function getString(bytes32 _key) external view returns (string memory) {
return stringStorage[_key];
}
function getBool(bytes32 _key) external view returns (bool) {
return boolStorage[_key];
}
...
// Set
function setString(bytes32 _key, string memory _value) public onlyNetworkContract {
stringStorage[_key] = _value;
}
function setBool(bytes32 _key, bool _value) external onlyNetworkContract {
boolStorage[_key] = _value;
}
...
// Delete
function deleteString(bytes32 _key) external onlyNetworkContract {
delete stringStorage[_key];
}
function deleteBool(bytes32 _key) external onlyNetworkContract {
delete boolStorage[_key];
}
...
// And similar get/set/delete declarations and functions for the types:
// "address", "bytes", "int", "bytes32"
}
To keep things sane and organized, we will use dot notation to address our pseudo objects and their properties. Interacting with our Eternal Storage contract may look something like this:
eternalStorage.setAddress(keccak256(abi.encodePacked("foo.bar.owner"), msg.sender))
eternalStorage.setString(keccak256(abi.encodePacked("foo.bar.title"), “Hello World”))
Keccak is the algorithm that became SHA-3, but NIST changed an implementation detail in the standardization process. Chances of collision are extremely low, giving us 64^36 addressable keys for each mapping. The function
keccak256()
expects bytes
array and returns bytes32
. The function abi.encodePacked()
expects any type of data and returns it encoded in the smallest byte representation possible via a bytes
array. Hashing our keys like this allows us to stay at 32 bytes regardless of key length, helping save on gas costs.Setters are locked down to allow access only from registered contracts in the network once admins mark the boolean map with key
“contract.storage.intialized”
to be true.Unfortunately adding Eternal Storage increases the complexity in our contracts. See how our original
DNS
contract logic changes:// Old
function updateRecords(string memory name, DomainRecord[] memory records) public {
claim(name);
delete domains[name].records;
uint size = records.length;
for (uint i = 0; i < size; i++) {
domains[name].records.push(records[i]);
}
}
// New
EternalStorage eternalStorage;
...
function updateRecords(string memory name, DomainRecord[] memory records) public {
uint length = records.length;
eternalStorage.setUint(keccak256(abi.encodePacked("domains", name, "records", "length")), length);
for (uint i = 0; i < length; i++) {
eternalStorage.setString(keccak256(abi.encodePacked("domains", name, "records", i, "type")), records[i].recordType);
eternalStorage.setString(keccak256(abi.encodePacked("domains", name, "records", i, "value")), records[i].value);
eternalStorage.setUint(keccak256(abi.encodePacked("domains", name, "records", i, "ttl")), records[i].ttl);
}
Luckily, we can utilize helper functions to keep our business logic free of the complexity introduced above by using Eternal Storage.
One common data type we will have to reintroduce is lists/arrays.
StringListStorage
highlights the many steps involved in implementing a string array on top of Eternal Storage and why it is important to separate this into a helper.pragma solidity >=0.5.14 <0.6.0;
contract StringListStorage {
EternalStorage es;
constructor(address _es) public {
es = EternalStorage(_es);
}
// Get
function getListLength(bytes32 _key) external view returns (uint) {
return es.getUint(keccak256(abi.encodePacked(_key, "length")));
}
function getListItem(bytes32 _key, uint _index) external view returns (string memory) {
return es.getString(keccak256(abi.encodePacked(_key, "item", _index)));
}
...
// Set
function pushListItem(bytes32 _key, string memory _value) public {
uint length = es.getUint(keccak256(abi.encodePacked(_key, "length")));
es.setString(keccak256(abi.encodePacked(_key, "item", length)), _value);
es.setUint(keccak256(abi.encodePacked(_key, "length")), length + 1);
}
function removeUnorderedListItem(bytes32 _key, uint _index) external {
require(_index < es.getUint(keccak256(abi.encodePacked(_key, "length"))), "List index out of bounds");
uint length = es.getUint(keccak256(abi.encodePacked(_key, "length")));
if (_index < length - 1) {
string memory lastItem = es.getString(keccak256(abi.encodePacked(_key, "item", length - 1)));
es.setString(keccak256(abi.encodePacked(_key, "item", _index)), lastItem);
}
es.setUint(keccak256(abi.encodePacked(_key, "length")), length - 1);
}
...
// See complete contract linked below
}
[View StringListStorage_v1.sol]
Now that we have a nice wrapper for string arrays, we can create a helper around our previous idea of a
Domain
struct. DomainModel
shows usage of the StringListStorage
helper and how we are able to maintain friendly method interfaces through structs despite the underlying storage mechanisms having no notion of them.pragma solidity >=0.5.14 <0.6.0;
pragma experimental ABIEncoderV2;
contract DomainModel {
struct DomainRecord {
string recordType; // "A", "CNAME", "MX", etc...
string value;
uint ttl;
}
EternalStorage es;
StringListStorage sls;
constructor(address _es) public {
es = EternalStorage(_es);
sls = new StringListStorage(_es);
}
function getOwner(string memory name) public view
returns (address owner_)
{
address domainOwner = es.getAddress(keccak256(abi.encodePacked("domains", name, "owner")));
owner_ = domainOwner;
}
function setOwner(string memory name, address owner) public {
if (owner == address(0)) {
return;
}
es.setAddress(keccak256(abi.encodePacked("domains", name, "owner")), owner);
sls.pushListItem(keccak256(abi.encodePacked("owners", owner, "domains")), name);
}
...
// See complete contract linked below
}
With
DomainModel
, we now have an idea of how we can organize larger data structures around Eternal Storage. Finally, let's use DomainModel
inside our DNS
contract.pragma solidity >=0.5.14 <0.6.0;
pragma experimental ABIEncoderV2;
contract DNS {
DomainsModel _domains;
constructor(address _es) public {
_domains = new DomainsModel(_es);
}
function claim(string memory name) public {
address domainOwner = _domains.getOwner(name);
if (domainOwner == address(0)) {
_domains.setOwner(name, msg.sender);
return;
}
require(domainOwner == msg.sender, "Domain has already been claimed");
}
function updateRecords(string memory name, DomainsModel.DomainRecord[] memory records) public {
claim(name);
_domains.setRecords(name, records);
}
function release(string memory name) public {
address domainOwner = _domains.getOwner(name);
require(domainOwner == msg.sender, "Sender does not own this domain");
_domains.setRecords(name, []);
_domains.deleteOwner(name);
}
function domains(string memory name) public view
returns (address owner_)
{
address domainOwner = _domains.getOwner(name);
owner_ = domainOwner;
}
function recordsForName(string memory name) public view
returns (DomainsModel.DomainRecord[] memory recordsForName_)
{
DomainsModel.DomainRecord[] memory records = _domains.getRecords(name);
recordsForName_ = records;
}
function domainNamesForOwner(address owner) public view
returns (string[] memory domainNamesForOwner_)
{
string[] memory domainNames = _domains.getDomainNamesForOwner(owner);
domainNamesForOwner_ = domainNames;
}
function domainNamesForSender() public view
returns (string[] memory domainNamesForSender_)
{
string[] memory domainNames = _domains.getDomainNamesForOwner(msg.sender);
domainNamesForSender_ = domainNames;
}
}
This isn't so bad. In fact it reads much more nicely than before. Now that we are happy with our storage, let's wrap it all up and secure our new network of contracts.
We cannot talk about authorization and upgradeability separately as each implementation depends on the other.
Once we set
"contract.storage.intialized"
, only contracts registered at the boolean map with key “access.role”, “contract”, CONTRACT_ADDRESS
will be able to modify storage. Unfortunately, admins will locked out and now unable to add/upgrade/authorize contracts. Additionally with our current logic, any sender can still bypass domain ownership checks in the DNS
contract by calling the methods in DomainModel
and StringListStorage
directly.Upgradeability, and any other actions admins may need to perform can be implemented as... contracts! Admins should never need fully unrestricted access to Eternal Storage. Limiting the power admins hold also reduces risk and improves overall trust in the system.
We need the following for authorization/upgradeability:
“access.role”, “contract”, _SenderAddress
(boolean)“access.role”, “admin”, _SenderAddress
(boolean)“contract”, _contractName, “address”
(address)Let’s jump ahead a little bit and add a
Base
contract that implements these checks.pragma solidity >=0.5.14 <0.6.0;
contract Base {
EternalStorage internal es;
modifier onlyNetworkContracts() {
require(
hasRole("contract") == true,
"Sender is not a contract in the network"
);
_;
}
modifier onlyNetworkContract(string memory _contractName) {
address contractAddress = getContractAddress(_contractName);
require(contractAddress == msg.sender, "Incorrect or outdated contract access used");
_;
}
modifier onlyRole(string memory _role) {
require(
hasRole(_role) == true,
"Sender doesn't have matching role"
);
_;
}
constructor(address _es) public {
es = EternalStorage(_es);
}
function getContractAddress(string memory _contractName) public view returns(address) {
address contractAddress = es.getAddress(
abi.encodePacked(keccak256(abi.encodePacked("contracts", _contractName, "address")))
);
require(address(contractAddress) != address(0x0), "Contract not found");
return contractAddress;
}
function hasRole(string memory _role) public view returns(bool) {
return es.getBool(
abi.encodePacked(keccak256(abi.encodePacked("access.role", _role, msg.sender)))
) ;
}
}
The
Base
contract gives us:pragma solidity >=0.5.14 <0.6.0;
contract ContractAdmin is Base {
constructor(address _es) Base(_es) public {}
function addContract(
string memory _name,
address _contractAddress,
string memory _contractAbi
) public onlyRole("owner") {
require(_contractAddress != address(0x0), "Invalid contract address");
address existingContractName = es.getAddress(keccak256(abi.encodePacked("contracts", _name, "address")));
require(existingContractName == address(0x0), "Contract name is already in use");
bool existingContractAddress = es.getBool(keccak256(abi.encodePacked("access.role", "contract", _contractAddress)));
require(!existingContractAddress, "Contract address is already in use");
es.setAddress(keccak256(abi.encodePacked("contracts", _name, "address")), _contractAddress);
es.setString(keccak256(abi.encodePacked("contracts", _name, "abi")), _contractAbi);
es.setBool(keccak256(abi.encodePacked("access.role", "contract", _contractAddress)), true);
}
...
// See complete contract linked below
}
The
ContractAdmin
contract gives admins (via onlyRole("owner")
) the ability to add and upgrade contracts with required authorization-related entries being updated automatically.Now how will the rest of our contracts look after they inherit the
Base
contract and lock down access to their methods?pragma solidity >=0.5.14 <0.6.0;
contract StringListStorage is Base {
constructor(address _es) Base(_es) public {}
// Getters remain the same
function getListLength(bytes32 _key) external view returns (uint) {...}
function getListItem(bytes32 _key, uint _index) external view returns (string memory) {...}
// Setters have the onlyNetworkContracts modifier applied to them
function pushListItem(bytes32 _key, string memory _value) public onlyNetworkContracts {...}
function removeUnorderedListItem(bytes32 _key, uint _index) external onlyNetworkContracts {...}
...
// See complete contract linked below
}
[View StringListStorage_v2.sol]
The
StringListStorage
contract locks down storage-modifying methods to only contracts in the network.pragma solidity >=0.5.14 <0.6.0;
pragma experimental ABIEncoderV2;
contract DomainModel is Base {
struct DomainRecord { ... }
constructor(address _es) Base(_es) public {}
...
// this is method and the other setters are now only accessible by the most recently deployed DNS contract
function setOwner(string memory name, address owner) onlyNetworkContract("DNS") public {
...
// notice how we are using getContractAddress to discover the address of the StringListStorage contract
StringListStorage sls = StringListStorage(getContractAddress("StringListStorage"));
...
sls.pushListItem(keccak256(abi.encodePacked("owners", owner, "domains")), name);
}
...
// See complete contract linked below
}
The
DomainModel
contract locks down storage modifying methods to only the currently deployed DNS contract. Additionally, it now fetches the most current address of StringListStorage
from Eternal Storage rather than having the address passed to its constructor.pragma solidity >=0.5.14 <0.6.0;
pragma experimental ABIEncoderV2;
contract DNS is Base {
constructor(address _es) public Base(_es) {}
function claim(string memory name) public {
// notice how we are using getContractAddress to discover the address of the DomainModel contract
DomainModel dm = DomainModel(getContractAddress("DomainModel"));
address domainOwner = dm.getOwner(name);
if (domainOwner == address(0)) {
dm.setOwner(name, msg.sender);
return;
}
require(domainOwner == msg.sender, "Domain has already been claimed");
}
function updateRecords(string memory name, DomainModel.DomainRecord[] memory records) public {
claim(name);
// another example of using getContractAddress
DomainModel dm = DomainModel(getContractAddress("DomainModel"));
dm.setRecords(name, records);
}
function release(string memory name) public {
DomainModel dm = DomainModel(getContractAddress("DomainModel"));
address domainOwner = dm.getOwner(name);
require(domainOwner == msg.sender, "Sender does not own this domain");
dm.deleteOwner(name);
}
function domains(string memory name) public view
returns (address owner_)
{
DomainModel dm = DomainModel(getContractAddress("DomainModel"));
address domainOwner = dm.getOwner(name);
owner_ = domainOwner;
}
function recordsForName(string memory name) public view
returns (DomainModel.DomainRecord[] memory recordsForName_)
{
DomainModel dm = DomainModel(getContractAddress("DomainModel"));
DomainModel.DomainRecord[] memory records = dm.getRecords(name);
recordsForName_ = records;
}
function domainNamesForOwner(address owner) public view
returns (string[] memory domainNamesForOwner_)
{
DomainModel dm = DomainModel(getContractAddress("DomainModel"));
string[] memory domainNames = dm.getDomainNamesForOwner(owner);
domainNamesForOwner_ = domainNames;
}
function domainNamesForSender() public view
returns (string[] memory domainNamesForSender_)
{
DomainModel dm = DomainModel(getContractAddress("DomainModel"));
string[] memory domainNames = dm.getDomainNamesForOwner(msg.sender);
domainNamesForSender_ = domainNames;
}
}
The DNS contract does not lock down access to any of its methods since it’s directly user facing. It now fetches the most current address of the
DomainModel
contract from Eternal Storage rather than deploying an instance and storing that address in its constructor.We now have a dapp with the following properties:
You can find all of the contracts and their revisions here.
Finally, we have a strong base for building and maintaining long-lived decentralized applications. This system isolates the trust to admins who can't directly access the eternal storage, and can only modify it by contract upgrades. Furthermore it allows for the implementation of clear and appropriate governance policies to further control the actions of the admins.
Hopefully this exercise has demonstrated why it's a good idea to use Eternal Storage in your future dapp.