The Great Game of Validators or How We Launched Proof-of-Stake DAObet

So, your team has finished the alpha version of your blockchain, and it is time to run the test-net, and then the main-net. You have a real blockchain with independent participants, a good economic model and security, and you have designed governance.

And now it is time to try it all out for real. In an ideal crypto-anarchic world, you upload the genesis block to the network, the final node code and the validators themselves do everything else – they launch all the auxiliary services and everything happens by itself.

However, this is in a fictional world. In the real world, the team should prepare quite a lot of auxiliary software and carry out various manipulations to help the validators launch a stable network. This article will delve into these issues.

Introduction

Launching consensus-based networks of the “proof-of-stake” type, where the validators are determined by the votes of the system’s token holders, is a rather specific event, because even launching traditional, centrally managed systems with dozens and hundreds of servers is not an easy task in itself, and a blockchain needs to be launched by the efforts of loyal but independent participants.

And, if at the startup stage in the corporation, the administrators have full access to all the machines, logs and general monitoring, then the validators will not let anyone into their servers and will most likely prefer to build their own infrastructure, because it controls access to the validator’s main assets – the voting stakes.

It is this behavior that allows you to build distributed, secure networks – the independence of the cloud providers, the virtual and bare metal servers used, and the different operating systems. All this makes attacks on such a network extremely inefficient, as too many different software types are used.

For example, Ethereum uses two main node implementations, Go and Rust, and an attack effective for one implementation does not work for another.

Therefore, all the processes of starting and operating blockchains should be organized so that any validator, or even a small group of validators, could at any time throw their computers out the window and leave, and nothing should break down and the remaining validators should continue to support the work of the network effectively and connect new validators.

When the network is launched, when one validator is in Europe, the second in South America, and the third in Asia, it is rather difficult to achieve coordinated work of several dozen independent groups and interest them as a result.

The Validators

Let us imagine the launch of a hypothetical modern blockchain (most of what is being described is suitable for blockchains based on any modern family of blockchains: Ethereum, EOS, Polkadot, Cosmos and others, which provide proof-of-stake consensus.

The main characters of such blockchains are validator teams engaged in the installation of their own independent servers running blockchain software, producing new blocks, and receiving awards provided by the network for those who participate in the consensus.

There are several dozen validators (that many can now reach consensus in
seconds more or less efficiently), therefore, the project announces a
registration process in which the validators share public information about
themselves with users, convincing them that they are going to provide quality services to the network being launched.

Validation is a business that allows one to accurately assess the validator’s potential income, quickly transfer capacities between projects, and if the network chosen is successful, the validator can develop the project as a full-fledged DAO participant and responsible person, or simply provide excellent technical services for completely transparently and honestly earned money.

When calculating the reward for validators, projects try to take into account the costs of validators and make a reward for blocks so that this business is profitable, but at the same time, it does not allow the validators to bring down the economy by gorging them with funds and depriving the rest of the network users of said funds.

Validation requires ensuring high fault tolerance of services, which means a high level of training for dev-ops and developers, and expensive computing resources. Even without the need to mine hashes in proof-of-work networks, the blockchain node is a large service that consumes a lot of memory, consumes a lot of computation power, validates, writes to disks and transfers large amounts of data to the network.

To store the transaction log and chains of blocks for a blockchain with several thousand small transactions in a block, storage of 50 Gb or more is now required, and for blocks this should be an SSD.

State database blockchains with support for smart contracts can already exceed 64 Gb of RAM. Servers with the required characteristics are quite expensive, as an Ethereum or EOS node can cost from $100 to $200 per month.

Add to this the increased remuneration for the round-the-clock work of developers and dev-ops, who solve problems even at night during the launch period, since some of the validators can easily be on the other side of the globe.

However, in good times, owning a validator node can generate significant revenue (up to $10,000 per day for EOS).

Validation is only one of the new potential IT roles for entrepreneurs and companies, as programmers come up with more sophisticated algorithms that allow rewarding honesty and punishing deception and theft.

Services are appearing that perform the functions of publishing important data (oracles) that conduct supervision (slashing deposits and punishment deceivers by publishing evidence of fraud), dispute resolution services, insurance and options services, even garbage collection is a potentially large market in smart contract systems where one must pay for data storage.

Blockchain Launch Problems

The openness of the blockchain, which makes it possible for computers from any country to freely participate in the work of a network and the ease of connecting any script kiddie to the network according to the instructions on GitHub, is not always an advantage.

The pursuit of a new token often forces validators to “mine a new coin at the start” in the hopes of an appreciation and the ability to quickly sell off what they have earned.

Also, this means that anyone can be a validator, even an anonymous one, and one can vote for them as well as other validators (though it will be difficult for an anonymous person to collect the votes of stakeholders, so we leave the terrible tales about anonymous cryptocurrencies to the politicians).

Still…

The project team has a task of somehow getting into their network those who will be able to ensure stable operation of nodes in the future, understand security issues, quickly solve problems, cooperate with other validators and act together.

The quality of the token depends on these qualities in which the network participants are going to invest their time and resources. Adequate founders, assessing risks, are well aware that when launching software of such a volume, they will definitely have to face errors in the code, configuration of nodes, and that network stability depends on how well the developers and validators solve such problems together.

The team is ready to vote in the mainnet for any validators, but just for the sake of knowing which ones are good? The ones with the biggest portfolio?

Almost no one has one now.

Based on team profiles on LinkedIn? Experienced dev-ops or security specialists will not give you any profiles on LinkedIn.

According to the statements in the chat, posts and help offered to others in the preparation phase? Good, but subjective and inaccurate.

In such circumstances, one thing remains – that, which solves the problems of everyone well enough – a game in which you can choose the best validators, but the main thing is to check the blockchain for stability and conduct a full-scale battle test of the blockchain in conditions of active use, changes in consensus, appearance and correction of errors .

For the first time, this procedure was presented as a game by the guys from the Cosmos project, and this idea is undoubtedly an excellent way to prepare the network for the launch of a reliable and fault-tolerant mainnet.

Game of Validators

I will describe the game of validators as we designed it for the DAObet (ex-DAO.Casino) blockchain based on the EOS fork, which is called Haya and has a close to EOS governance mechanism, where the validators are selected by voting from any account in which part of the balance that votes for the validator is frozen.

Any account with the main BET Token on the balance can vote for the selected validator with any part of its balance. The votes are summed up and the top validators are selected based on the results. In different blockchains, this process is organized in different ways, and usually in this part, the new blockchain differs from the parent.

And I must say that in our case, EOS fully justifies “OS” in its name, as we really use EOS as the base operating system for deploying a modified version of the blockchain for the tasks of DAObet.

I will describe several individual problems and how they can be solved within the framework of the game. Imagine a network in which your server can be openly attacked, where in order to maintain the position of a validator, you need to continuously interact with the network, promoting your validator and making sure that it produces blocks and they are delivered to the other validators in time, otherwise, the validator will be dropped from the list.

How to Choose the Top Winners?

The main technical requirement for the game is that its results are publicly verified. This means that the results of the game – the TOP winners, must be decided strictly on the basis of data that any participant can verify. In a centralized system, we could measure the “uptime” of each validator and reward those who were more time online or had the most network traffic throughput.

You can collect data on CPU utilization and memory, and reward those who have worked with dignity.

However, any such collection of metrics means the existence of a collection center, and the nodes are all independent and can behave as they want and send any data they wish.

Therefore, the natural solution is that the winners should be determined according to the data from the blockchain, since it can be used to see which validator made which block and which transactions were included in it.

We called this number the Validator Points (VP), and earning them is the main goal of the validators in the game.

In our case, the simplest, easily publicly verified and effective metric of the “usefulness” of the validator are the VP = the number of blocks produced by a validator for a given time period.

Such a simple choice is due to the fact that governance in EOS already provides solution for many problems that arise, since EOS is the successor of three generations of really working blockchains (Bitshares, Steem) with extensive experience in complex network management.

Almost any validator problems with a network, processor, or disk lead to only one the problem – if the validator signs fewer blocks, then they receive less pay for their work, which again leads us simply to the number of blocks to be signed.

For EOS, this is a great and easy option.

For other blockchains, the method of calculating Validator Points may differ. For example, for pBFT-based consensus (Tendermint / Cosmos, Aura consensus from Parity Substrate), where each block must be signed by many validators, it makes sense to read individual validator signatures, not blocks.

It may make sense to consider the incomplete consensus rounds that spend the resources of other validators. In general, much depends on the type of consensus.

How to Simulate Real Operating Conditions

The task of the founders is to check the validators in conditions that are close to reality, while not having any centralized control.

This problem can be solved with the help of a contract-faucet, which distributes equal amounts of the main tokens to the validators and everyone else, allowing to claim this fixed amount each hour, minute, or second.

To get the tokens on the balance, one needs to form a transaction, and ensure that the network includes it in the block. Thus, to win, the validator needs to constantly replenish their balance with new tokens and vote for themselves, thus promoting themselves to the top.

These activities create a constant load on the network and the parameters ("tokens per second" for validtors and "tokens per second" for any other account) can be chosen in such a way so that the transactions flow is serious enough for a full scale network test.

Therefore, plan the contract-faucet in advance as an important tool for launching the network, and start determining its parameters in advance.

The request for tokens from the faucet and the voting of validators still do not honestly emulate the work of the blockchain, especially in extremely loaded modes.

Therefore, the blockchain team will still have to write additional benchmarks anyway to load the network. A special role in this is played by smart contracts that are created in advance for this specific purpose, which allow testing a separate subsystem.

For storage testing, the contract stores massive pack of random data in the blockchain. The test contract requires a large amount of input data for checking network resources, thereby inflating the volume of transactions. By launching the flow of such transactions at arbitrary points in time, the team simultaneously tests the stability of the code and the validity of the validators.

Also, team can use "network-hard" and "cpu-hard" transactions (with a lot of data in transaction, many calculations, etc...)

A separate issue is updating the node code and conducting hard forks. It is required in the event of a bug, vulnerability, or collusion of malicious validators.

In such a case, the validators should have an action plan already worked out in the game of validators. Here you can come up with VP accrual schemes for the fast apply of hard forks, for example, fine all the validators who have not yet downloaded and run a new version of the node code.

But this is difficult to implement and complicates the calculation. You can simulate emergency situations of hard forks by artificially “breaking” the blockchain on a given block.

Block production stops, and as a result, those who turn on earlier versions and start signing blocks will benefit, so the VP accrual scheme based on the number of signed blocks is well suited here.

How to Inform Participants About the Network Status and Patch-up Errors

Despite the distrust between the validators, it is beneficial for every participant of game to get up-to-date information on the network status in order to make decisions faster.

The project team can launch a service for collecting and visualizing many metrics from validator servers, which allows one to see the situation simultaneously for the entire network, thus allowing to quickly determine what going on. It is also beneficial for both the validators and the project that the project team quickly corrects any errors that are found.

Therefore, in addition to collecting metrics, it makes sense to immediately start collecting logs and error data from the validator nodes on a machine accessible to the blockchain developers. It is not beneficial for anyone to distort information, therefore these services are launched by the project team and can be trusted.

It makes sense to collect system metrics from the validators, and the most important metrics of the blockchain itself.

For DAObet, ​​it is the time of finalization and the lag of the last finalized block. Thanks to this, the team sees an increase in memory consumption on nodes when the benchmark is launched and some problems of individual validators.

Important Points to Consider for Conducting a Game of Validators

As it turned out, if you want to officially allow validators to attack each other’s machines (unofficially they can do it anyway), you need to formulate this separately legally as security testing, since under the laws of some countries, DDoS or network attacks are punishable.

Also, the good idea is to deny social and physical attacks on validators, but allow other, close to reality attacks using public networks. Another important issue is how to reward the validators. The natural prizes are project tokens, which will be transferred to the mainnet.

But massive distribution of tokens to anyone who was able to launch a node is also not the best option. Most likely, you will have to balance between the two extreme options:

1. Distribute the entire prize pool in accordance with the VP earned:

• It is very democratic and allows anyone who has invested time and resources in the game of validators to earn.
• It attracts random people without an infrastructure to the game.

2. Distribute the prize pool to the top-N validators according to the results of the game:

• The winners will most likely be the validators who sustained their nodes with the most stability during the game, and those who were seriously set on winning.
• Some of the validators will not want to participate, assessing their chances of winning as low, especially if the participants include respected validators.

So, now we have figured out how to give away the prizes. There is one more point – it is not at all a fact that dozens of validators will rush to participate in the game at your call.

And of those who do decide to participate, not all will even install and launch the node.

Usually, at this stage, projects have rather poor documentation, there are errors, and developers working overtime have trouble answering questions quickly.

Therefore, before launching the game it is also necessary to foresee some actions if the necessary number of validators is not reached. In this case, at the start of the game, the missing validators are launched by the project team, they participate in consensus, but cannot win.

Conclusion

In conclusion, I will try to compile list of what was said above about what a team needs to come up with, make and run for an effective game of validators.

What you need to do to run a validator game:

1. Develop your blockchain, of course;
2. Develop and run web-interface and provide CLI for governance functions (voting, delegation, etc...)
3. Make sure that the metrics of the nodes can be sent to a centralized service (for example, Prometheus);
4. Launch the metrics collection server (Prometheus + Grafana) for the game of validators;
5. Figure out how Validator Points (VP) will be calculated;
6. Develop a public script that calculates the validator VP based on data from the blockchain and a web interface to display the top validators;
7. Develop and automate the launch of an arbitrary number of internal nodes, design the process of connecting validators to the game (when and how to disconnect your nodes, submit and remove votes for them);
8. Calculate how much tokens need to be issued and develop a contract-faucet;
9. Make a benchmark script (token transfers, massive use of storage, massive use of the network);
10. Gather all the participants in one chat for fast communication;
11. Launch test network for game before game starts
12. Wait for starting block and start the game
13. Test network with different types of transactions
14. Make a hardfork
15. Change list and/or amount of validators
16. Repeat p.13,14,15 if needed
17. Wait for final block, finish game
18. Profit...

I must say that the game of validators is a new thing and was carried out only a couple of times, so you should not perceive this text as a ready-made guide.

There are no analogues in the modern IT business. Imagine that banks compete among themselves who is better at conducting customer transactions before launching a new payment system :)

Traditional approaches are unlikely to help create large decentralized networks, so you need master new business models, launch your games, identify worthy ones, issue rewards, and let your distributed systems work quickly and in a stable fashion.

The story was made in collaboration with DAObet’s joint partner - MixBytes.