I was recently asked to generate a list of all token transactions undertaken within my company's wallet. Naturally, I began with Etherscan in hopes that I would be able to download and export the list of transactions — only to find that such a functionality is limited to 5,000 rows. As you can guess, the wallet I was interested in had significantly higher number of transactions. What a bummer. Which meant the only way to get the list I wanted was to write a script that collected the information I needed. I have decided to share my learning via an open source script because I believe this problem might be common enough that other others might benefit from it. At the end of the day, all companies in the blockchain space have similar issues. You can find the script and its source code on Github In this article, I'm using my personal wallet to simplify examples. My personal wallet has only 17 transactions but the script on my Github works well with significantly higher amounts of transactions, so feel free to use it. I will be working on my transactions related DAI stablecoin. Let's check . my wallet history on Etherscan As you can see, I did a bunch of transfers and a few interactions (methods such as , , , and so on). I'm mentioning the smart contract interactions because they are harder to process due to encoding, but that becomes irrelevant if you were to follow my guide. smart contract Deposit Withdraw File Order Swap Exact Token Disclaimer All examples in this article are written in . I am using Python and extensively taking advantage of a library called . This library allows me to interact with Ethereum and you can install it using PyPI by typing the following command in your terminal: Python v3.10.5 web3.py $ pip install web3 If you write in JavaScript only, don't worry. You can translate my snippets to JavaScript and use the library instead. There are some differences in namespaces and naming, but the guide is largely applicable. web3.js Let's get started! The first thing the script needs to do is to establish connection to the blockchain. To do so, we will need to find an Ethereum API. If you don't run your own node, you can get a free API from , or . Choose one, I'm using Infura for my personal projects (As you can see, I pasted it in line 9). Ethereum Infura QuickNode Moralis import asyncio from web3 import Web3, AsyncHTTPProvider from web3.eth import AsyncEth, Eth from web3.net import AsyncNet async def runner(): ethereum_api_url = "https://mainnet.infura.io/v3/MY_SUPER_SECRET_TOKEN" async_web3 = Web3( AsyncHTTPProvider(ethereum_api_url), modules={ "eth": (AsyncEth,), "net": (AsyncNet,), }, middlewares=[], ) web3_modules = get_default_modules() web3_modules["eth"] = Eth block = await async_web3.eth.get_block("latest") print("Current block:", block.number) # All code goes here await asyncio.sleep(0) if __name__ == "__main__": asyncio.run(runner()) I created an asynchronous provider, and I will be using it to interact with the . You might have noticed that on the snippet above I'm fetching the latest block. Even though that isn’t required to complete my task, I'm doing so to verify that my web3 provider is connected to Ethereum. web3 blockchain When you run this script you will see the following result: $ python runner.py Current block: 16436276 Now we are ready to scan the blockchain. Hunting logs Let's look into my wallet transaction history again, especially my very first transaction. As you can see, the first transaction is and it was dispatched in block . At the moment of writing this article, Ethereum mined block 16436276; it means my first transaction was executed more than 3 million blocks ago! Obviously, retrieving all transactions by querying every single block isn't really an option. Don't get me wrong - it's possible, but extremely time consuming. The whole processing would take at least few hours (or days!). It's too slow. 0xe3d38...cefaa2e 13352962 Let's dig a bit deeper and focus on the tab on Etherscan. As you can see, this transaction has one event - a event. Logs Transfer Ethereum indexes logs of every transactions. This indexing means it allows to request particular logs by providing broader block ranges and smart contract addresses. The block range can be broad but the output cannot exceed 10,000 logs per call. Isn't it amazing? Let's return back to our event. You can spot it has three topics - is the event's signature. It tells us that all events have signature . This information is crucial, we will be using it to find all transfer to our wallet but I will return to this later. Let's focus on and for now. Transfer topic 0 Transfer Transfer 0xddf252ad...f523b3ef topic 1 2 Most tokens on Ethereum (excluding NFTs and Ether) are ERC-20 tokens and DAI isn't any different. The easiest way to find out about topics exposed by Transfer event will be reading the ERC-20 token smart contract. Fortunately, some of them are open source and we can find . We will be looking into interfaces and search for the and its definition. their source code on Github Transfer event // SPDX-License-Identifier: MIT // OpenZeppelin Contracts (last updated v4.6.0) (token/ERC20/IERC20.sol) pragma solidity ^0.8.0; /** * @dev Interface of the ERC20 standard as defined in the EIP. */ interface IERC20 { /** * @dev Emitted when `value` tokens are moved from one account (`from`) to * another (`to`). * * Note that `value` may be zero. */ event Transfer(address indexed from, address indexed to, uint256 value); ... } As you can see, the event has two indexed arguments - and - and which isn't indexed. The indexed arguments are exposed as topics, and we will be able to request logs associated with them. Transfer from to value Now we know that and . Let's request all incoming DAI transactions to my wallet from block 13,352,962 (my first DAI transaction) to block 16,436,276 (the current block). is the address topic 1 from is the address topic 2 to incoming_logs = await async_web3.eth.get_logs( { "fromBlock": 13352962, "toBlock": 16436276, "address": "0x6B175474E89094C44Da98b954EedeAC495271d0F", # DAI "topics": [ "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", None, "0x000000000000000000000000d5e73f9199e67b6ff8dface1767a1bdadf1a7242", ], } ) When you look closer into this snippet, you might feel confused by two things. The first one is the field. It my wallet address (smart contract). If you'd provide the wallet address there, you'd find nothing. Wallets don't implement any events with signature. address isn't but DAI address Transfer The second thing, the wallet address is prefixed with 24 zeros. It is related to EVM - the topic user 32 bytes data types and wallet addresses are stored in 20 bytes. It means I need to fill the "free" space with zeros. If you run the snippet code, you will see the array of logs that looks like this one: [ AttributeDict({ 'address': '0x6B175474E89094C44Da98b954EedeAC495271d0F', 'blockHash': HexBytes('0xa67b8ceaeb79ec2592e161ee2efee6fba3fd329c87131d9335ccaa869cc857ec'), 'blockNumber': 13352962, 'data': '0x0000000000000000000000000000000000000000000000356ea11fcb4975c000', 'logIndex': 85, 'removed': False, 'topics': [ HexBytes('0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef'), HexBytes('0x00000000000000000000000021a31ee1afc51d94c2efccaa2092ad1028285549'), HexBytes('0x000000000000000000000000d5e73f9199e67b6ff8dface1767a1bdadf1a7242') ], 'transactionHash': HexBytes('0xe3d380b3647abee2f8e2980d8e3bf6e9a43b00a0f4a388765585df13ecefaa2e'), 'transactionIndex': 41 }), ... ] If you read it carefully, you can spot the sender and receiver of each log ( and . The only missing part is the transaction value, right? Actually, it is stored in the field but it is encoded. It can be decoded by using the method from the web3.py library. topic 1 2) data toInt Let's try it out. from web3 import Web3 value = Web3.toInt(hexstr="0x0000000000000000000000000000000000000000000000356ea11fcb4975c000") print(value) # 985649123680000000000 The script returned and this number feels a bit too high, right? 985649123680000000000 Not really, - it means you need to divide this number by 10 to the power of 18 ( ) to get a "human" readable number. If you do so you will find out that the value of this transaction was DAI. You can confirm this number . DAI has 18 decimals 10^18 985.64912368 with transaction on Etherscan Conclusion Processing Ethereum block-by-block can be tedious and not necessary. Often we can search logs to find events related to transactions we are looking for. As I said at the very beginning, you can find a fully-fledged script on with all instructions on how to run it. The article contains only the juicy parts. Github If you have any question you can start a discussion on Github or . Feel free to follow me - I tweet things related to Ethereum and software engineering. ping me on Twitter It would mean the world to me if you would share this article on your social media. Thank you! Also published . here
