The main network is running, transactions are being sent, the wallet is working. What's next? In this article, we will consider how to maintain a network and solve its problems. Code updates The younger the mainnet is, the more bug fixes will be required; the older - the greater the responsibility for each change. Core project code updates This happens when your blockchain is built on the basis of a ready-made blockchain engine. The engine is developing and changing - bug fixes, optimizations, new logic. Thus, it is necessary to transfer all these changes to your project. The more updates you make, the more you should do every time to transfer them from the core project into yours. When building a blockchain, try not to touch the main mechanisms: network consensus, internal database, transaction types and other basic structures. Any change should be accompanied by additional tests - they will show if the code was broken. When modifying ready-made blockchain engines, developers often turn off tests that stopped working after their manipulations. Fixing such tests is more difficult than making changes. This is the wrong way, as the tempting simplicity of the “now” may return as serious hidden issues of the “after”. Replay chains Sometimes code changes require rebuilding the database or writing new auxiliary data from scratch. This procedure is called “replay” because transactions are “replayed” using the updated code. Replay has many kinds, depending on how blocks and transactions are processed. A complete chain replay is extremely costly and time consuming. You will have to repeat all the operations of the blockchain nodes during project life time. Alternatively, the chain can be processed in the old way before updates, and another way - after a certain block. In this case, the updates will be automatic, without chain rebuilding Bear in mind that the code once used in the blockchain will never be deleted. The nodes of more mature projects are able to quickly “catch up” with the chain and to do a quick replay and snapshots of the database in order to switch between different versions following strict determinism. The quality and validity of such mechanisms determines the quality of the blockchain software. The replay chain script should be tested before mainnet launch and become a routine task for the team. Network design and optimization A peer-to-peer network between blockchain nodes simplifies network connection. It is enough to exchange data with any neighbour node once, and in seconds you will learn about other nodes it has been dealing with recently. For efficient block production, data must first be distributed among network validators - those who produce new blocks - and then between other participants of the p2p network. Sometimes there is a strictly configured network between validators: messages are prioritized, and validators interact with the public network only through intermediary nodes that already support a p2p network and deliver new blocks and transactions. The read access is easy to scale because the nodes are the same and almost everyone uses the well-known JSON REST API and WebSockets standards. As for the “write” access, everything is a bit more complicated - you need to think about which validators will be the first to receive new transactions (to put it in block or transfer to another validator) so that the entire flux could not go through one server. Therefore, client software (for example, a wallet) must contain the addresses of several reserved servers. Infrastructure ownership The total cost of blockchain ownership (not by individual validators) is significantly higher than that of a distributed database. Each complete blockchain node double-checks all data and stores a copy of it. In regular databases, updates are received from trusted neighbors without additional checks. On the whole, blockchain nodes are more picky in terms of resources than distributed database servers. For each blockchain, you can build different types of “non-full” nodes - sometimes they’re called “light nodes” (for example, those sending only block numbers, transactions and balances). Validators and blockchain node types In terms of resource consumption and ownership costs, pay attention to the following: CPU load Checking block and transaction signatures cause a pretty serious CPU load. The processor load for blockchain tasks will always be greater than zero, since most blockchains, even in the absence of transactions, generate empty blocks that also require consensus. Storage Headers of empty blocks also need to be stored, and even an empty blockchain constantly loads the disk. For example, a web project database is usually idle if there are no users and no new messages in the logs. At the same time, the blockchain node receives, writes and validates new blocks. Network Each peer is obliged to receive new blocks and distribute them between neighbors. Continuous block production also adds to the network load. “New” blockchains may use the network inefficiently if the team has postponed optimization of the network stack, focusing on other priorities. Memory If there are few addresses and transactions, and the former are not owners of big data, then memory consumption grows slowly and predictably. Sometimes it takes more memory to build a project than to operate a blockchain. Remember that memory consumption is proportional to the number of active network addresses. If everything works on a simple and cheap VPS server, wait for the bill that includes traffic and disk operations (IOPS). This might be an unpleasant surprise. However, there is good news for devops. For example, there is no need for backups for blockchain nodes. Any full blockchain node can serve as a backup. As there is no need to trust someone in the network, blockchain nodes are easy to restore, replace, disable, duplicate. If validator addresses and governance-procedures are organized correctly, you can fully automate the launch, hot swapping the blockchain nodes in case of hacking and hardware failures, and for scaling the load. Moreover, a lot of useful code is public, and your devops will be able to use many ready-made mechanisms. As for network validators and full nodes, it is more beneficial to use regular servers rather than VPS. They are cheaper, there is no need to pay extra for disk operations, and disk failure doesn’t cause big trouble. In the majority of cases, the disk operation pattern is “append only”, i.e. blockchain always adds new data, and most updates occur in RAM. Such pattern prolongs exploitation of modern SSD disks so that they don’t lose resources due to active rewriting of data (that is the main cause of flash memory degradation on SSD disks). Additional software Block explorer Block explorer is the most loaded service with the largest number of components in the blockchain network. It consists of several complex components: subsystem for importing data about blocks and transactions constantly monitors one or more blockchain nodes, downloads new blocks, and prepares data for transmission to the indexer. subsystem for updating data and indexes takes fresh data, places them in the database, and updates indexes in database. search subsystem quickly provides the user with information about transactions. There is no “full-text” search, but data should be is quickly updated for good users’ experience. external import subsystem. Block explorer often needs information from the outside world, such as cryptocurrency rates or monitoring data. This subsystem requires constant connection to different sites, tracking their availability, query history, etc. additional functionality: multilingualism, personalization, verification of smart-contract sources, etc. Block Explorer is a web application that requires backup with a large and ever-growing database. These are the most demanding project servers, since in addition to complete information about blocks and transactions, the database stores indexes that speed up the search and additional data. The block explorer will require a lot of resources, reserve servers and backup. Wallet and main dapps A wallet is lightweight software with a number of features. First of all, the wallet manages user private keys.Thus, the most stringent security measures must be followed. For example, a private key or a seed-phrase should be in RAM only at the time of signing a transaction, when they are not used, since another process can read them from memory (if it receives the necessary permissions). Lightweight dapps in JavaScript are extremely vulnerable to XSS. Dapps operate with cryptocurrency addresses on web pages using JavaScript, so XSS vulnerabilities can cause serious harm. I recommend auditing this software regularly by external independent companies and adding new functionality carefully. System Smart Contracts System smart contracts mean code that controls the main blockchain logic (lists of validators, distribution of awards, main parameters). In top blockchain engines, all important business logic is implemented using smart contracts similar to the platform’s smart contracts. They are deployed in the network from the very beginning and updated only via a quorum of validators. System smart contracts include: "main token contract": for example, ETH in Ethereum, or DOT in Polkadot. This is exactly the same token as any ERC-20, created by users, but its address is forever hardcoded in the Ethereum blockchain and this contract is always present in blockchain “library” contracts: provide access to important blockchain functions from the very beginning. For example, using a system smart contract, users create multisig addresses. This is the basic primitive that allows validators to vote, protect the infrastructure from hacks, and securely store earned cryptocurrency. “governance” contracts: manage the list of validators, the size of their rewards, change global system parameters (for example, the maximum number of transactions in a block, the time between blocks, etc.). Usually they set up the network economy, so you need to be prepared for changes. "bridge contracts": usually are multisig contracts that will allow you to issue X network tokens when the validator quorum is reached. Validators confirm the operation only if they see a token transfer from the external network. This version of bridges is the most popular, simple for users and requires exactly the same level of trust in validators as in the entire blockchain. When modifying system contracts, ask developers to write detailed tests. Tests should emulate specific scenarios and check whether all parties have received exactly what is meant by the system logic. Writing detailed tests slows down the blockchain development but engineers will be able to reproduce any difficult situations and fix system smart contracts without errors. New developers will quickly catch up, and links to test code with good comments will show any user how the network economy works even if they don’t have any programming knowledge. Notes There is no such thing as a smooth network launch - something always goes wrong. The blockchain is reliable and resistant to attacks, but if you actively keep changing the system core (network layer, consensus, system contracts), be ready to face tricky bugs difficult to fix. Blockchain nodes never rest. Even in idle state, they are continuously writing something to disk, consuming CPU and network. This is what makes them different from services in which the load increases only during user activity. First months of the blockchain launch are spent on chats monitoring, attempts to organize validator activities and project management. Much depends on the community manager, his activity and ability to coordinate actions of completely different people. Developers and devops engineers will have to answer many questions, so do not forget to prepare FAQ and update it on time. * * * We have completed the series of articles “How to launch your blockchain”. We hope it will help you make well-thought decisions and avoid problems when starting blockchain projects. Good luck!
