Telegram Open Network Architecture Explained by A Network Validator [Deep Dive]

The launch of Telegram Open Network mainnet has been delayed till early 2020, meanwhile, the testing of TON blockchain is in progress. Test network has been relaunched on 15 November 2019. To become a validator, you must run TON Blockchain Full Node and have a certain amount of test Gram tokens. In testnet2 the validator stake has been lowered to 10,001 test Gram. Theoretically, everyone who meets these two requirements can become a validator, but in reality, it might be tricky due to lack of experience with TON blockchain. While the basic version is thoroughly documented in , there is almost no information on how to use HowTo tonlib-cli. team who has become the second biggest validator in TON testnet2 shares some insights about working with tonlib-cli. Mercuryo The key component of Telegram Open Network is a flexible system of blockchains further referred to as TON Blockchain, which, according to the developers’ statements, can process millions of transactions per second, support the full TuringComplete Smart Contracts, updatable official blockchain specifications, multi-currency transactions as well as micropayment channels for off-chain payment systems. The architecture of TON Blockchain is unique due to having such specific features as a ‘self-healing’ vertical blockchain mechanism and Instant Hypercube Routing that allow it to be fast, reliable, scalable and resistible at once. As we previously said, TON Blockchain is a code name for the decentralized network (a set of blockchains) or 2D blockchains consisting of three main blockchain types. or – the unique blockchain containing general information about the protocol and the current values of its parameters, a set of validators and their shares, a set of currently active workchains and their shards and the set of hashes of the latest blocks of masterchains and shardshains. Master blockchain Masterchain or are a multitude (up to 232) of blockchains which are workhorses, containing asset-transfer transactions and smart contracts. Herewith, individual workchains can have their own ‘rules’, formats of writing account addresses, formats of transactions, various virtual machines (VM) for smart contracts, various base tokens and cryptocurrencies, etc. But all of them must comply with several rules of functional compatibility to ensure a simple interaction between them. Thus, TON Blockchain is essentially heterogeneous as well as the blockchains of EOS and Polkabot. Working blockchains Workchains or are a subset of blockchains (up to 260) inside a multitude of workchains, ensuring operation of the sharding system and having the same rights and the format of the blocks as the workchains. The shardchains contain just a subset of accounts depending on the first several (the most important) bits of address of any particular account. Because all shardchains have a common format and the rules of building blocks, the TON blockchain is homogeneous and complies with the requirements, described in one of the propositions for the scaling of Ethereum. Shard blockchains Shardchains Every block of the shardchain (as well as one of the masterchain) in reality is not a block but a small blockchain or ‘vertical blockchain’ which consists of a single block; It may be called a standard block of the corresponding ‘normal’ (or ‘horizontal blockchain’). However, if there is a need for correcting the incorrect blocks, into the ‘vertical chain of blocks’ is entered a new block that contains a replacer for the currently active ‘horizontal’ block or the ‘difference of blocks’, containing only the description of the parts of the previous version of the block that need to be replaced. This TON-specific mechanism of replacing the found incorrect blocks without the need for a hardfork has been named 2D blockchain or simply 2-blockchain. Consensus Algorithms and Network Protection Mechanism TON offers a blockchain based on Infinite Sharing Paradigm using a traditional Proof-of-Stake or PoS mechanism. According to the developer’s documentation: ‘Almost all blockchain sharding proposals are “top-down”: one first imagines a single blockchain, and then discusses how to split it into several interacting shardchains to improve performance and achieve scalability. The TON approach to sharding is ‘bottom-up’, explained as follows. Imagine that sharding has been taken to its extreme, so that exactly one account or smart contract remains in each shardchain. Then we have a huge number of “account-chains”, each describing the state and state transitions of only one account, and sending value-bearing messages to each other to transfer value and information. Of course, it is impractical to have hundreds of millions of blockchains, with updates (i.e., new blocks) usually appearing quite rarely in each of them. In order to implement them more efficiently, we group these “account-chains” into “shardchains”, so that each block of the shardchain is essentially a collection of blocks of account-chains that have been assigned to this shard. Thus the “account-chains”have only a purely virtual or logical existence inside the “shardchains”. We call this perspective the Infinite Sharding Paradigm. It explains many of the design decisions for the TON Blockchain.’ The TON consensus network includes nodes of different types: and validators, nominators, fishermen collators. The validators are the PoS nodes and block producers. The fishermen can that the consensus network looking for errors or trying to detect a presumably malicious node and in case the fisherman clearly confirms that there is such node, it will obtain a reward in the form of confiscation of a part of the validator’s stake. The collators are tasked with preparing the shardchain blocks and submitting them to validation by the PoS nodes for which they obtain their fraction of reward for creating a block. In such case, the collators are basically additional participants of the consensus as the validators almost always generate the blocks on their own. The nominators lend their assets (Gram tokens) to the validators in order to get a profit. The nominators do not practically enter the infrastructure of the validators and only distribute their initial large stake in the asset among them in exchange for a proportionate percentage of the entire reward. Thus, the scheme and the amount of reward that the nominators obtain fully depends on the results of the work of the validators, the nominators lending them their Gram tokens while ‘voting’ for the validators. Both the individual token holders and pools, which manage individual TON users’ funds and simultaneously act as validators, acting as delegates through the TON smart contract, can become nominators. In this case, the total reward of such pool is distributed among its participants proportionately to their stakes. The new block generation process itself goes in the following way: some particular number of validators nominate the masterchain blocks permissioned for validation (shards) using a special algorithm, then for every such shard a smaller subset of validators is selected in the order determined by a pseudorandom method with the interval of approximately every 1,024 blocks. Thus, for every block, there is a pseudorandomly chosen set of validators that choose whose block candidate has the highest priority. The validators and other nodes verify the validity of suggested block candidates. In case a validator automatically (unintentionally) signs an invalid block candidate, it is punished by losing a part or the entire reward, or by being temporarily alienated from participation in the election of validators. Next, validators need to reach a consensus based on the BFT (Byzantine fault tolerance protocol) algorithm, similar to the pBFT or Honey Badger BFT protocol. Then, after consensus is reached, a new block is created, while transaction fees are distributed among the validators. It should be noted that each validator can be selected to participate in several subsets of validators; therefore, it is assumed that all validation and consensus algorithms are run in parallel. After all the new blocks of the shards of the chain are generated or the timeout is over, there is a message announcing that a new block of the masterchain has been created, which includes hashes of the last blocks of all shards based on the pBFT consensus of all validators. TON Testnet: hands-on experience in the Telegram Open Network Our team is an active participant in the testnet, which was launched in early September 2019, and during the testing we have gained some experience that we would like to share with you. Ways to access the network Interacting with the TON network one way or another means using the TL specifications which describe the ways of interacting using the API. The specification files are accessible . here There are three types of API: interact with the Full Node validator-engine-console ton_api to work with the lite-client lite_api to contains everything that is related to the wallet, this is the only publicly accessible tonlib-cli API. tonlib Wallet The easiest way of creating a wallet is to use Test Gram Wallet that is available on for such operating systems as Windows, macOS, Linux. the official site There are also several ways of interacting through the interface of command prompt: basic and using the tonlib-cli. Unfortunately, there is currently no compatibility between the two. Here we will be looking at the instruments offered by TON developer themselves. While the basic version is thoroughly documented in , there is almost no information on how to use tonlib-cli. HowTo As was previously mentioned, there are three APIs for different tasks. Tonlib-cli controls the functions relating to the wallet’s operation. To start working with tonlib-cli, besides the interface of command prompt[I1] you need to have a configuration file in order to connect to TON network’s public liteserver, which is available at this here . The connection is done by the command. tonlib-cli -c ton-lite-client-test1.config.json-v 0 where -v 0 is the parameter responsible for the output of debugging information. List of commands: To create a wallet, we use the command and a list of mnemonic phrases that can be necessary for the recovery of access to the address in case of loss of the private key. genkey List of keys The keys command brings up a list of keys. For further operation, when using other commands, it is necessary to use their ordinal number, i.e. for the first key it will be id 0. Address initialization After creating an address, you should register it in the network. To do this, you must first replenish it. Initially, a special smart contract was used for this – testgiver, but now it is easier and more convenient to use a special bot in Telegram @test_ton_bot. Right after the replenishment, the account status is defined as uninited_accountStateand will only change after you send the GRM test tokens from it. If you already have tokens on your balance and you need to activate another wallet, you can use the command transfer, and then along with the wallet being replenished it will be initialized. You can learn the wallet’s status by entering the command . getstate 0 You can acquire the history of transactions using the command gethistory where is the ordinal number of the key. The network base As well as in the majority of existing blockchains, TON is based on servers which store a complete record of all blockchains that have ever been created in the network. For running a full node of the TON testnet, one needs 8 high-performance cores, 4 – 8 GB of Random Access Memory; at the time of writing the data occupied around 50 GB of disc [I2] space, but it is safer to have at least another 50 GB in reserve. Note that it is better to use an SSD drive because there will be a large number of IOPS to record, otherwise the synchronisation with the network will be very slow. As a working OS it is better to use Ubuntu 18.04 as the most of the community’s tests are run in it. https://test.ton.org/README.txt https://test.ton.org/FullNode-HOWTO.txt https://test.ton.org/Validator-HOWTO.txt Validator system It is widely known that TON consists of shardchain and masterchain blocks which are created and verified by special designated nodes called validators. The validators obtain a certain reward for their work: maintaining the functionality of the TON blockchain, the income being proportionately distributed inside the validators’ community. In order to take part in the validation process, a user has to have not just a sufficient amount of coins (GRAM tokens) but a constantly working full node of the network (TON Blockchain Full Node). Theoretically, any user can become a validator just satisfying a condition of possessing a minimal stake in the asset (in Gram coins) on the masterchain, but in practice it raises several questions: Is there an upper limit for a validator’s stake? A limit for one validator’s stake can always be checked using the getconfig 17 command, which will show the relevant sizes for the stakes: In the screenshot, you can see that currently the minimum size of the stake is 10,000 GRAM. However, in case a validator does not collect over 100,000 GRAM in an election round, it cannot take part in the election. Yet, the maximum number of tokens for one validator cannot exceed 10 million GRAM and for an election to take place the minimum size of collective stake should exceed 1 million GRAM. How are validators elected? To apply for participation in the validation process, you must have a minimum of 10,000 GRAM. The algorithm of the election process is described in detail in the smart contract elector-code.fc. Most likely, the contract will be different in the mainnet, so, the current version is applicable only for the testnet. A share of 10,000 GRAM does not mean that you can become a validator, as receiving test tokens could be easily automated by requests to testgiver. At the moment, almost all validators, when participating in the election, set the max-factor at 2.7 and a stake of 120,000 GRAM. Since there are most of these bets, because of their weight the minimum steak rises to 120,000 / 2.7 = 45,000 GRAM (unlike 100,000 according to official documentation). But even with such minimum steak, your chances are practically zero, since the three top validators have a max-factor of 2, which raises the minimum stake, which allows you to become a validator in the testnet, to 60,000 GRAM. If all the current validators increased their max factor or reduced the size of their stake, then it would be possible to become a validator with the minimum stake, providing that the maximum number of validators (1000 nodes) was not be exceeded. If the validator system is centralized, is the entire blockchain centralized? there are no checks, i.e. no one centrally controls the validators, nominators themselves determine the risks when choosing a validator What types of fines are provided for validators? there is no information, most likely there will be a consensus mechanism in the document, because, in the testnet, even the out-of-sync nodes received awards Smart Contracts To create TON smart contracts, two special programming languages are used: Fift and FunC. If there is at least general documentation for Fifth, finding information about Func is almost impossible (even in the terms of the development competition it is written that it can only be found by analyzing its source code). During the testing, it was possible to find out that the FunC code base is not so voluminous (compared to Fift) and allows you to learn it much faster, so, working with FunC is much easier than with Fift. Relevant/pressing/open issues Slow synchronisation https://github.com/ton-blockchain/ton/issues/100 Access rights for validation-engine +0 = usual lite-client queries +1 = full node statistics queries +2 = full code configuration modification queries +4 = potentially dangerous queries (such as private key export or signing arbitrary strings) +8 = reserved for future extensions (does nothing at the moment) How to make PIPE work with lite-client? By default, the lite-client withdrawal is sent to stderr; therefore, to process it you first must redirect the withdrawal from stderr to stdout: $ lite-client 2> >(grep …) What ways of program-based access to the network exist? https://github.com/ton-blockchain/ton/issues/76 What server configuration is required for the validator? The use of a dual-processor server is officially recommended (at least 8 cores per processor). The software is not very demanding on RAM, so, 16 GB is enough. You must use an SSD as the primary drive, the minimum recommended size of which is 512 GB. An 8TB HDD is sufficient for storing archived data. It is imperative that you have a high-speed Internet connection: with a projected average weighted load of 100 Mbit/sec, you must be able to handle peak loads of up to 1Gbit/sec. It is recommended to use XFS as the file system, since information about each block is stored in a separate file. It is known that, for example, ext4 does not work very well with a large number of small files and can lead to a situation where you simply do not have free inodes with enough disc space. How do I know that the node is synced? There will be a ‘completed sync’ message in the log or you can use validator-engine-console -c «getstats» unixtime and masterchainblocktime should be almost the same How many validators can there be in the network? Getconfig 16max_validators:1000 max_main_validators:100 min_validators:5 How do I recognise the currently active validators? Getconfig 34The previo us set of validators getconfig 32 T he time the validators are elected for? The White Paper says that the validators are elected for one month but in the testnet this time is much shorter and you can only find it in the config getconfig 15. After the launch of the testnet, the time intervals for the validators: ConfigParam (15) = (validators_elected_for:65536 elections_start_before:32768 elections_end_before:8192 stake_held_for:32768) From which it follows that a group of validators is selected for 65536 seconds. How are validators elected for the next round? The process of request submission is explained in detail in . The validators are elected with the help of a smart contract, whose relevant address can always be found with the help of command. Validator-HOWTO getconfig 1 If the answer contains ‘result: [ 0 ]’, it means that the voting process is inactive, if the answer contains timestamp, it means you can apply for participation. Even if you do not plan to participate, the requests of other selection participants can always be viewed: > runmethod :A4C2C7C05B093D470DE2316DBA089FA0DD775FD9B1EBFC9DC9D04B498D3A2DDA participant_list -1 Can TON be blocked? Despite the complex design of the network’s stack based on overlaying networks, TON still uses UDP and TCP as transport protocols. It is presently known that blocking Telegram is quite unsuccessful, because it is possible to change IP addresses, use proxies and update settings through the push. However, TON will not have such opportunities: it is not possible to quickly move nodes, while validators will not want to risk their shares. Therefore, most likely, in the near future, Telegram developers will introduce new solutblocks, for example, using ADNL Proxy. Below is presented the traffic from one of the full nodes after processing 10 million packages. The list of 159 IP addresses, on which the full nodes of the testnet are launched, looks like this: 126 DIGITAL OCEAN (supposedly managed by TON) 13 AMAZON 4 GOOGLE 3 HETZNER 3 ALIBABA CLOUD 2 OVH 2 SELECTEL 2 ONLINE.NET 1 LINODE 1 hosteurope.de 1 contabo.de and 1 person possibly hosting it at home in Italy telecomitalia.it Consequently, the regulator will have no trouble getting the list of IP addresses of the mainnet. However, this is not a problem only of TON but of any other Internet service. Essentially, the Telegram Open Network is Pavel Durov's attempt to implement an independent decentralised Internet based on the concept of WEB 3.0 within the existing global network, the interaction with which is carried out through already existing informational infrastructure of Telegram messenger. This attempt looks very ambitious and in case of success will start a new era in the world of blockchain technologies. The user base of the messenger (over 200 million people), the funds raised through the ICO, positive expectations and support of the world’s community are by all means positive factors for a successful launch and implementation of plans for building the largest network which will compete with such giants as Ethereum, EOS and others. From the technical viewpoint TON is a unique project whose technological stack has been created from scratch without using already existing solutions. Nowadays, we can say that: 1) The source code of the project is written by highly professional developers who thoroughly understand all intricacies of blockchain technology. 2) They have created their own low-level languages for writing smart contracts (Fifth and FunC), allowing to effectively use all abilities of the virtual machine and use debugging. 3) A new testnet has been launched in which a community of third-party developers is actively working. 4) Special Telegram bots simplifying the interaction with the TON ecosystem have been launched. Real chances of scaling the TON blockchain The Infinite Sharding Paradigm sounds impressive but it needs thorough practical testing because the growing number of shards will result in the growth of the quantity of crosschains, which will cause a decrease in the speed of transaction processing; thus, there is a probability that there is an effective throughput uplimit. TON plans to use what the developers call Instant Hypercube Routing for crosschain routing. While it is early to say that the sharding approach of TON will fail, we must note that the Infinite Paradigm Sharding is slightly misleading and the quantity of shardchains should stay within reasonable limits. Decentralisation and consensus Practically, to get the right to take part in achieving consensus, the TON nodes must have a certain share of the asset (Gram tokens). A shortcoming of such model is that it can create a situation where the rich will be becoming richer, and, consequently, TON provides rich people with the funds to finance controlling the PoS nodes, which means that only big organisations with sufficient financing will receive rewards from most nodes without much effort and technical know-how. The consensus PoS nodes are responsible for creating blocks for every shardchain in the network. As a result, the number of such mini-blockchains will dynamically grow depending on the transaction load, the PoS nodes being pseudorandomly selected for controlling certain sectors of the shardchains, but the number of the consensus nodes does not grow proportionately. Initially, there will be around 100 PoS nodes carrying responsibility for creating, propagating and synchronising the blocks and shardchains. Thus, this creates a substantial load on the PoS nodes from the viewpoint of required throughput as well as computing power. Grams Wallet– an application for storing Gram tokens In the light of the recently published set of rules on how to use the blockchain service , which will be integrated into the Telegram messenger, for storing the native TON tokens, many participants of the blockchain community have expressed concerns that the board of Telegram FZ-LLC intends to block suspicious accounts and unarguably comply with any directions of regulatory institutions in all jurisdictions (even in authoritarian countries). Gram Wallet However, a closer analysis of the document allows to see that this User agreement does not imply any such measures except that the user must be 18+ years of age and does not plan to use the application in ways that violate the principles of law, terms of settlement and recommendations relevant to particular jurisdictions and in case of violations of these regulations and rules the Telegram FZ-LLC company is not liable for the actions of such users. Thus, this document is essentially a standard disclaimer that you can come across when using any other open-source software (e.g., the Linux distribution). Yet, special attention is given to one very important section (s. 4, cl. 4.3) in which it is said that the company does not control the blockchain of Telegram Open Network: ‘We do not control TON Blockchain and are unable to nullify or change the data of already executed transactions.’ Our open source Describe what we have done and how it will help. Go-binding library As it has already been mentioned, the only way of interacting with the TON network using programming languages other than C++ is using the TONLIB library. As all development at Mercuryo project is implemented using the Go language, we have decided to share our results by publishing a wrapper library that we use ourselves. https://github.com/mercuryoio/tonlib-go The tonlib api specification is far from its final look but with its help we already can perform almost all operations related to the functionality of the wallet: operations with keys (creation, deletion, export, import, change of password), sending messages, Gram, boc files in transactions, receiving the wallet state and information about it, receiving the wallet’s list of transactions, Tongo lightweight utility tool for work with the wallet. Presently, the highest priority in library development is given to: · Network monitoring. The ability to receive information on every transaction from all blocks of the network. Anxiously waiting for support on the side of tonlib itself.· Extention of the functionality of the console helper tongo. We are trying to add something new in every new release. · Generation of interface structures using tl specifications. It will allow us to be more mobile and release updates with minimal delays. References 1. White Paper... https://test.ton.org/ton.pdf 2. Github... https://github.com/ton-blockchain/ton 3. Text docs... https://test.ton.org/ 4. Development docs… https://docs.ton.dev/ 5. TON - Telegram Open Network - BitcoinWiki 6. Grams Wallet - Terms of Service (Last Updated on 8 October 2019) https://telegram.org/tos/wallet