From the blockchain of Ethereum through the Digital ocean and to the Google clouds

My setup to push smart contract data to website — scalable, secure and almost free

The problem — how to connect my smart contract and website

One of the key features of The Million Ether Homepage and its underlying smart contract (a decentralized advertising platform) is that all income is shared between referrals and charity. It means that there is a scenario where I get very little or no money at all (as I’m just a referral too). This sets the first requirement to my infrastructure connecting smart contract and the Internet — it has to be very low cost or free.

The second requirement comes from my main goal. Which is to get as much hype as possible in the press. Which in turn means that the platform has to endure very high and inconsistent load. So cheap shouldn’t mean compromised speed or security.

And of course the platform has to be user-friendly and reasonably simple to develop and maintain.

In other words I need a good balance between cheap, scalable, secure, simple and user-friendly.

The solution

I decided I’m not gonna attach any custom GUI to my smart contract (at least for now, at least for MVP). As the threshold of getting into blockchain is already quite high I thought it won’t be much higher if I offer my users a GUI auto-generated by blockchain client (see this entrepreneur’s guide on smart contracts to see how GUIs are attached). This will let me build super-lightweight and super-simple web page. The web page will just represent the state of the smart contract, requiring very little traffic.

The Million Ether Homepage is a 1000 by 1000 pixels field where anybody can buy can buy (or sell) pixels and place ads. Meaning I need to store ads images somewhere. I decided to use Imgur.com for this to minimize traffic even further.

Any smart contract by it’s definition is already cheap and scalable (and I worked hard to make a secure, simple and user-friendly one). In turn Imgur is an infinite everlasting source (free, scalable, secure, etc.). So the real question now is how to make my smart contract interact with the outer world. Or how to push data from Ethereum blockchain to website.

My solution consists of 5 parts:

1. Smart contract provides UI (with no G) in addition to business logic and database
2. Imgur.com hosts images
3. Geth Ethereum node at Digital Ocean droplet provides blockchain access
4. Python app (polyether.py) at the same DO droplet listens to the contract, processes data and pushes it to TheMillionEtherHomepage.com website
5. Google App Engine hosts the website

Pushing smart contract data to www

Next I will show you what happens when a user places new advertising. Or in other words uses placeImage function of the MillionEther smart contract.

Notes in italics like this show my decision making, additional info and alternatives.

How it works

1. User uploads an image to Imgur

placeImage function of the contract requires a link to an image. So the first step for a user is to get it — drag&drop an ad to imgur.com and copy the link.

I stick with Imgur.com, because everybody knows it. It’s free for non-commercial and open-source projects. It has huuuge rate limits “approximately 1,250 uploads per day or approximately 12,500 requests per day”. It has API and python library. All I had to do is register my application, get a token (application password) and start using it.

Alternatively to Imgur I could offer users to encode their images in base 64 and send those bytes directly to the smart contract. But this way the contract will not be able to work with any image hosting or swarm storage (a sentence further). Meaning being too inflexible and too complicated for a user. Another solution is to use Ethereum’s swarm storage. But it is still too beta and too complicated for me. I’m quite sure though my contract is compatible with the swarm and I’ll be able to switch to it later. For now — Imgur.

2. User interacts with the smart contract

User interacts with the smart contract through a GUI auto-generated by a blockchain client (MyEtherWallet.com or EthereumWallet desktop app). If you are not familiar with the process check out this video.

Placing an ad requires sending desired coordinates (within 1000x1000px field), image url (copied from imgur), link to a user’s website and an advertisement text to the placeImage function. Once user submits the transaction the contract assigns an ID to the image and fires an event:

NewImage(numImages, fromX, fromY, toX, toY, imageSourceUrl, adUrl, adText);

After the transaction is mined into a block it’s receipt with event logs becomes available to any synchronized blockchain client. This is how placeImage function receipt looks like at Etherescan.io blockchain explorer (click Convert to Ascii to see something human readable).

There are two ways of getting smart contract information: constant functions (those which don’t modify contract state, i.e. write to contract) and events. The easiest way to access contract data from an external app is to use events. The only way for another contract to access contract data is to use function calls. My polyether.py listens to events but the MillionEther contract has constant functions too to let other developers attach their own extensions (contracts).

3. Geth downloads a block with the transaction

At Digital Ocean droplet Geth is run with this command:

geth --cache=16 --rpc --rpcport 8545 --rpccorsdomain "localhost"

Letting any local app to interact with it through JavaScript API.

Geth is constantly downloading new blocks (syncing). And now it receives a block with the event.

I use the cheapest Digital Ocean droplet with attached 40 GB volume. It lets me run full Ethereum node and my python app simultaneously. Here is how it’s configured.

There are several ways to connect to geth: ipc (through a local file), rpc (through http protocol) and ws (through web-socket). I use rpc. Will show its benefits below.

4. Python app asks geth if there are any new events and processes them if there are any

Polyether.py runs at the same DO droplet as Geth. Every minute it asks Geth if there are any new events.

With EthJsonRpc library listening to evens is a breeze. Here is pretty all of the code needed to get all logs of a particular smart contract event (I’ll publish my ethereum_gateway.py soon — follow me to catch the post).

self.client = EthJsonRpc('127.0.0.1', 8545) params = {       "fromBlock": hex(from_block),       "toBlock": "latest",       "address": 0x15dbdB25f870f21eaf9105e68e249E0426DaE916, # million_ether contract address       "topics": [event_definition["signature"]]      } logs = self.client.eth_getLogs(params)

Both Geth and polyether.py are monitored by monit to prevent them from crashing (monit auto-restarts both if something bad happens).

If a set of blocks has any new NewImage events of the MillionEther smart contract polyether.py receives and decodes them. The app then verifies Imgur links, downloads and stores new images, generates a 1000 by 1000 pixels image and uploads it back to Imgur (the url is received in return).

With Infura.io (remote blockchain client) and pure JavaScript I could generate 1000 by 1000 image right in the user browser. But the image is rendered locally for two reasons: 1. The query to the blockchain node to get all ads images is very heavy and it would be getting heavier with time (as the number of blocks to check for events grows fast). It is cheaper to have all past events and images written to local database and to search for new events within only few blocks every other minute. 2. Generating 1000 by 1000 image out of 10000 small ones is unbearable task for the majority of computers.

Using the link to the 1000x1000 image and ads details polyether.py generates new web page (the whole web page html), hash-sums it and sends it in JSON format through POST method to a secret GAE url with a long password.

I could pass raw data to app engine and generate the web page there, but I’m already paying for the DO droplet. So I’m getting most of it to minimize app engine load.

5. Google App engine receives the web page

The GAE app receives the POST request, checks the hash sum, the password and puts the web page into database.

As far as I understand I could use App Engine API instead of those POST requests and write data to the database directly, securely, reliably and out of the box. But Google documentation… well, you know… it’s complicated.

There are 6 ways (at least) you can store data at Google Cloud Platform: Bigtable, Cloud SQL, Datastore, Storage, Persistence disk and Spanner. I’m using Datastore — “a highly-scalable NoSQL database for non-relational data”. I find it the simplest. In addition every DB query is automatically cached. My web page though stored in database is usually loaded from cache, which has much higher free quotas and is much cheaper beyond them too. Alternatively I could store my web page at Storage, which is (presumingly) designed for static files and which is (probably) cheaper but… again… Google docs.

6. GAE delivers web page to a user

Typically a user arrives through a url like this http://themillionetherhomepage.com/0xF51f08910eC370DB5977Cff3D01dF4DfB06BfBe1. The 42 character string after slash is an Ethereum address of a referral which invited the user.

Referrals addresses get to the GAE app the same way as images do: signIn function of the smart contract fires NewUser event -> Geth -> polyether.py -> GAE and the address is saved in the same database.

The only load then app engine undergoes when somebody visits the link is to check if the address is in the database and load the current version of the web page. Both of these operations are usually performed using cached data, which is fast and cheap.

I use Google engine for its scalability and for its free quotas. I think my super-lightweight app should be able to serve around 1000 requests a day for free and for little money above that.

Conclusion

I wanted cheap, scalable, secure and simple infrastructure to push data from Ethereum to the Internet and keep the platform as user-friendly as possible. Here is what I got.

I paid 20 cents to deploy my smart contract to the blockchain and it will stay there for free forever. I pay $5 a month for DO and $4 a month for additional disk space (which I’ll soon get rid of). Still $0 for app engine. Imgur is free. Docs are for free at ReadTheDocs.com. Video instructions for free on Youtube. Cheap.

The smart contract may experience literally any load (as much as the whole Ethereum blockchain may handle). Imgur is bottomless. Digital ocean setup has no heavy tasks and can be duplicated on any computer if needed (haven’t I already said that?). App engine if paid well can bear more load than the whole Imgur.com. Hopefully cached data will save my bills. Scalable.

Assuming my smart contract is secure (and I worked hard to follow these official safety rules) the only point subject to attacks is the website. But there is almost no code there (and nothing to steal). Plus I’m quite sure (as I’ve seen it somewhere in the docs, but cannot find it any more) my app engine has DOS protection out of the box. Secure.

Simple. I spent a great deal of time to build all this up. I can’t say it was simple, but it surely was fun. And I hope I made it simpler for you.

User-friendly. As user-friendly as blockchain technology currently lets it to be.

Did I meet my requirements? I think I did. Of course there is no limit to perfection…

Bonus. How to make it work even better (and cheaper)

Currently as I’m running full geth node I’m cluttering with chain data 7 GB of disk space every month and have to buy more and more GBs. I can’t make up with it. I want to get rid off the local node and switch to remote node by Infura

Buying additional disk space

At it’s core it is as simple as changing only one line of code:

- self.client = EthJsonRpc('127.0.0.1', 8545) + self.client = EthJsonRpc('mainnet.infura.io/my-secret-token',443,True) # True for ssl

Just change http to https and localhost to infura.io and it works! Beautiful right? And even more beautiful if backed up by another node (e.g. Etherscan) and a local light client.

Thank you for reading. If you find this useful please like, share and follow me to get more articles like this in the future. See you!