Blockchains are transaction machines. But on-chain data represents value-at-rest — confirmed transactions. While pre-chain, or mempool, data explores value-in motion — transactions in flight. To date, the industry has largely focused on value-at-rest. This is an important piece of the Web3 puzzle. But for any transaction to be included in a block, it must move through the mempool. And unlike the blockchain — which acts as a definitive record — the mempool is chaotic, fragmented, and often unpredictable. In the course of building infrastructure to monitor the mempool, we've learned a lot about how it works. And at ETHDenver 2020 , Blocknative's CEO and co-founder, shared some important lessons in his talk Matt Cutler Mastering the Mempool: We've distilled the session into actionable lessons that we believe everyone building in Web3 should know. Let's dive in! TL;DR Experienced builders have been grappling with the complexity of the mempool for some time. Given the structure of the mempool, a number of interesting problems arise: There's no single source of truth.Instead, there are thousands of independent mempools, each containing different data.Transaction data is constantly in motion.The raw data is sparse and sometimes inconsistent.This makes definitive mempool data difficult to capture.Increasing complexity in transaction architectures compounds these challenges. While blockchains act as a single source of truth, mempools have no intrinsic form of ensuring consensus. Organizing this chaos into developer-friendly pre-chain data requires substantial resources and expertise. Luggage 1. Blockchain Transactions are Like Checked When you transact on Web3, you have to hand your transaction over to the network and hope it arrives in a block. Much like checking luggage at the airport. The node that receives your transaction broadcasts it out — and you wait for it to get confirmed. If your transaction gets stuck or lost, no central authority can tell you what's going on – or help you resolve the situation. 2. Transaction Anxiety is Real Given the state of Web3 — and how transactions get validated — transacting creates anxiety. And this anxiety is not felt just by new users. Experienced users, who are more likely to be transacting with real value, often experience the most pronounced transaction anxiety. This anxiety is problematic for our ecosystem. And end-users generally do not engage with systems that induce anxiety. 3. On-Chain is Value at Rest As a community, we spend most of our time talking about value on-chain. And by definition, all on-chain value is at rest. Conversely, the mempool is always in motion. This is important because the inner workings of the mempool is not widely understood. And a working knowledge of mempool operations is key to understanding how every transaction comes to rest on-chain. 4. Even Base Terminology is Subject to Debate Blocknative defines the mempool as: The shared staging area 'in front' of a blockchain that enables transaction ordering, transaction fee prioritization, and general block construction." Think of this as the “waiting area” for transactions to be accepted into a block. But depending where you sit in the broader blockchain ecosystem, you might use a different label: calls this the Geth TX-POOL calls this the Parity TX-QUEUE calls this the Bitcoin Mempool calls this the Libra Mempool calls this the Tezos Mempool Given this, we prefer to use the generic 'Mempool' label for purposes of clarity and inclusiveness. 5. There is No Such Thing as "The" Mempool There is no canonical, master reference mempool. Instead, there are many mempools. Every node has its own unique mempool. So there are as many mempools as there are nodes. A View of The Global Mempool Landscape. Source: matallo.carto.com The composition of transactions inside of a node's mempool is driven by a number of factors: Configuration settingsPeering relationshipsTopology These factors all play a role in determining which transactions enter and persist within a node's mempool. 6. Defaults Matter The default configuration (slots and memory) settings for and mempools dictate much of their behaviors. Geth Parity If all of the slots in a node are full, when a new pending transaction is accepted to the mempool, the transaction with the lowest gas fee will be dropped. Different nodes with different settings will accept and drop different transactions at different rates. 7. Mempools Miss Transactions A missed transaction is a transaction that was written on-chain that your version of the mempool never accepted. From your perspective, this is a transaction that is immediately Confirmed. In our analysis, a single node's mempool in a single geography will miss ~1% of all transactions during operating conditions. Under heavy transaction volumes, the percentage of missed transactions can vary greatly. normal 8. Everything in the Mempool Can Be Replaced While on-chain transactions are immutable, in-flight transactions can be overwritten with a replacement transaction. These are commonly referred to as and transactions. cancel speed up The percentage of replacement transactions vary over time. While they typically account for ~1% of all transactions, under peak loads replacement transactions can comprise >10% of the mempool. Explore Matt's Talk to Dive Deeper During his talk, Matt dives deep into the messy reality of Ethereum transactions. And builders who want to learn more should or check out the . watch the recording in its entirety Mastering The Mempool Blog Series How to Build with Mempool Data is a powerful framework that enables developers to access a global node network that captures and organizes the mempool in real-time. Developers can access Notify as an , , or library. This real-time notification network is trusted by over 40 teams in production. Notify API SDK JS Build Faster with a Real-Time Data Feed Developers are leveraging mempool data to reduce development time and debug smart contracts in real-time. By , developers can debug Dapps and smart contracts in real-time. This takes about 5 minutes to set up and can empower developers to get instant feedback on their code. setting up a Notify API data feed in slack or discord Improve User Experience with Transaction Alerts Rather than expecting users to pull data and refresh for updates, your application can leverage the Notify API to push information and let them know what is going on in their transactions. You can see an example of live push notifications in and — both of which alert users of both incoming and outgoing transactions. This feedback loop reduces anxiety for users and helps them to take action when something goes wrong with their transaction. Pillar Eidoo Benchmark Protocol Performance DeFi protocols can use mempool data to understand how long transactions take on their protocol as compared to other leading protocols. The reality is that many protocols have no insight into transaction times (either the average or when things start breaking). And performance here matters. An Example Of Building With Mempool Data Aside from Matt's talk at ETH Denver, we were excited to see the developer ecosystem dig deep into using to build next-generation applications. Notify And coming out of the hackathon, we were excited by one project in particular — , a service that monitors pending transactions for unauthorized transfers of ERC20 tokens from wallets and fires off a competing transaction at a higher gas price to divert funds into a rescue wallet. The non-custodial system uses a signed permission token (EIP712) to execute the recovery. OutFront Aside from the obvious excitement of seeing , we were thrilled to see that the conversation around the complexity of the mempool and the opportunities that pre-chain data can provide. a team building with Blocknative present on ETH Denver’s mainstage The complexity of the mempool has previously prevented teams from building on it. But now, with Blocknative’s API and SDK, developers can access the rich world of value in motion. And in that world, they can see all of the possible future states of the blockchain. Final Thoughts At any given time, there are tens of thousands to hundreds of thousands of transactions in the mempool. And the number of can be two-to-five times the volume of the external transactions. In the mempool, data flows continuously in and out. And about every 15 seconds, the top of the mempool gets included in a block. internal transactions This creates a chaotic, ever-changing mempool data environment. As we drive towards , it is crucial for our ecosystem to leverage mempool data to improve end-user experience and create more robust experiences. 1 billion transactions As we drive mainstream adoption of blockchain, we need to create friendly end-user experiences. Transaction confidence matters. Transaction performance matters. And the mempool is an important part of the puzzle towards creating a more robust end-user experiences. This post was originally published on the Blocknative blog at https://blog.blocknative.com/blog/8-lesson-master-the-mempool
![featured image - Mastering the Mempool [A How-To Guide]](https://hackernoon.imgix.net/images/m01z32kc.jpg?auto=format&fit=max&w=3840)
