The Best Method for Bulk Fetching ERC20 Token Balances

Fetching token balances for one or more addresses (at a specific point in time) is a super common task, regardless of what it is that you’re building. Despite being such a common task, when I first started working with data I found it surprisingly difficult to figure out how to do this. It isn’t exactly obvious from just scrolling through the list of available endpoints Ethereum JSON-RPC Once you do figure it out, the other issue (even less obvious) is figuring out how to do this for multiple addresses across different time periods in a reasonable amount of time that won’t burn through all your credits. API In this article, I am going to save you the pain I had to go through figuring out the best way to fetch token balances. I’ll go over the different methods available (and their issues), starting with the “naive approach” and ending with the best approach. Overview: The naive approach Batching JSON-RPC requests Using a multicall contract Final remarks (link to code) The naive approach The naive approach is to make a single HTTP request using the “eth_call” JSON-RPC endpoint. Here’s the code: def fetch_token_balance_naive(wallet_address, token_address, block_number, node_provider_url, api_key): balanceof_function = "balanceOf(address)(uint256)" balanceof_signature = Signature(balanceof_function) block_number_hex = Web3.toHex(primitive=block_number) data = balanceof_signature.encode_data([wallet_address]).hex() payload = { "jsonrpc": "2.0", "method": "eth_call", "params": [ { "to": token_address, "data": "0x" + data, }, block_number_hex, ], "id": 1, } headers = {"Content-Type": "application/json", "Accept-Encoding": "gzip"} url = f"{node_provider_url}/{api_key}" res = requests.post(url, headers=headers, json=payload) res_data = res.json() balance_encoded_hex = res_data["result"] balance_encoded_bytes = Web3.toBytes(hexstr=balance_encoded_hex) balance_decoded = Call.decode_output(balance_encoded_bytes, balanceof_signature, returns=None) return balance_decoded There are a few things that are not so obvious if this is your first time using “eth_call” so let’s quickly go over these things. “eth_call” is the JSON-RPC endpoint used to call smart contract functions. “balanceOf” is a function that all ERC20 tokens have. You can call this function to return how much an address holds of the token in question. To use “eth_call” you need to specify two parameters: “to” and “data”. The “to” parameter is the address of the smart contract you want to call a function from. In the “data” parameter you specify the function you want to call and its inputs. Not super user-friendly: But to “data” requires you to encode the function name and inputs into a hexadecimal string. Similarly, the output of our call has to be decoded. To make life easier, I use the Call and Signature classes of the multicall package (see ) to help me with encoding and decoding. https://github.com/banteg/multicall.py The problem with the naive approach is that it’s (in terms of API credits). If you need to fetch balances for multiple addresses, blocks, and/or tokens. For each block, address, and token you need to perform a separate request. super slow and expensive Batching requests Batching requests alleviates some of the problems of the naive approach if you need to fetch balances for multiple blocks, addresses, and/or tokens. In particular, it helps . Instead of making multiple separate requests — batching enables you to do it in a single request. The code for batching “eth_call” requests is as follows: speed things up significantly def fetch_token_balance_batch(wallet_addresses, token_addresses, block_numbers, node_provider_url, api_key): balanceof_function = "balanceOf(address)(uint256)" balanceof_signature = Signature(balanceof_function) payload_list = [] for i, (wallet_address, token_address, block_number) in enumerate( zip( wallet_addresses, token_addresses, block_numbers, ) ): block_number_hex = Web3.toHex(primitive=block_number) data = balanceof_signature.encode_data([wallet_address]).hex() payload = { "jsonrpc": "2.0", "method": "eth_call", "params": [ { "to": token_address, "data": "0x" + data, }, block_number_hex, ], "id": i + 1, } payload_list.append(payload) headers = {"Content-Type": "application/json", "Accept-Encoding": "gzip"} url = f"{node_provider_url}/{api_key}" res = requests.post(url, headers=headers, json=payload_list) res_data_list = res.json() balances = [] for res_data in res_data_list: balance_encoded_hex = res_data["result"] balance_encoded_bytes = Web3.toBytes(hexstr=balance_encoded_hex) balance_decoded = Call.decode_output(balance_encoded_bytes, balanceof_signature, returns=None) balances.append(balance_decoded) return balances Some things to keep in mind: You can’t batch an unlimited number of requests. You’re allowed by your plan. (This depends on the node provider you use.) limited by the maximum response size, and maximum number of requests per second Despite being much faster than the naive approach, the problem with batching is that you will still end up using the same amount of API credits. Depending on your use-case this can be . cost prohibitive Using a multicall contract Multicall contracts are smart contracts that allow multiple function calls to be bundled together and executed as a single function call. Similar to batching requests, using a multicall significantly speeds up bulk fetching balances. Instead of being charged for each separate “eth_call” request, you’ll only be charged for a single request. The other benefit: It’s a lot more cost efficient. The code that uses the multicall contract is a bit long. To make it more readable I have broken the code up into two functions: the main function and the inner function . fetch_token_balance_multicall create_multicall_payload_list def fetch_token_balance_multicall(wallet_addresses, token_addresses, block_numbers, node_provider_url, api_key): block_map = defaultdict(lambda: []) for block_number, token_address, wallet_address in zip(block_numbers, token_addresses, wallet_addresses): block_map[block_number].append((token_address, wallet_address)) aggregate_function = "tryBlockAndAggregate(bool,(address,bytes)[])(uint256,uint256,(bool,bytes)[])" aggregate_signature = Signature(aggregate_function) balanceof_function = "balanceOf(address)(uint256)" balanceof_signature = Signature(balanceof_function) payload_list = create_multicall_payload_list(block_map, aggregate_signature, balanceof_signature) headers = {"Content-Type": "application/json", "Accept-Encoding": "gzip"} url = f"{node_provider_url}/{api_key}" res = requests.post(url, headers=headers, json=payload_list) res_data_list = res.json() balances = [] for res_data in res_data_list: output_hex = res_data["result"] output_bytes = Web3.toBytes(hexstr=output_hex) returns = None decoded_output = Call.decode_output( output_bytes, aggregate_signature, returns, ) output_pairs = decoded_output[2] for flag, balance_encoded in output_pairs: balance_decoded = Call.decode_output(balance_encoded, balanceof_signature, returns) balances.append(balance_decoded) return balances The logic is very similar to what we have already seen in the previous sections. All the interesting logic is contained in . That being said, there is still one thing worth mentioning: fetch_token_balance_multicall create_multicall_payload_list combines both request batching and the use of a multicall contract. The request batching was implemented to enable us to fetch historical balances across multiple blocks in a single call. fetch_token_balance_multicall Now the interesting code: def create_multicall_payload_list(block_map, balanceof_signature, aggregate_signature): multicall3_address = "0xcA11bde05977b3631167028862bE2a173976CA11" state_override_code = load_state_override_code() require_success = False gas_limit = 50000000 payload_list = [] for i, block_number in enumerate(block_map.keys()): block_number_hex = Web3.toHex(primitive=block_number) call_params_list = [] for token_address, wallet_address in block_map[block_number]: call_params_list.append( { "to": token_address, "data": balanceof_signature.encode_data([wallet_address]), }, ) multicall_params = [ { "to": multicall3_address, "data": Web3.toHex( aggregate_signature.encode_data( [ require_success, [[c["to"], c["data"]] for c in call_params_list], ] ) ), }, block_number_hex, ] if gas_limit: multicall_params[0]["gas"] = Web3.toHex(primitive=gas_limit) if state_override_code: multicall_params.append({multicall3_address: {"code": state_override_code}}) payload = { "jsonrpc": "2.0", "method": "eth_call", "params": multicall_params, "id": i + 1, } payload_list.append(payload) The function creates the payload_list for a batch JSON-RPC request. For each block we create a separate payload and append it to the list. create_multicall_payload_list Each payload is an “eth_call” request. The call we are making is to the function, which requires us provide it with the list of calls we want to aggregate into a single call. tryBlockAndAggregate(bool, (address,bytes)[])(uint256, uint256,(bool,bytes)[]) Things to note: If the number of balances you’re fetching is large you should set a high value for “gas_limit”. The value of 50000000 will almost always work. “state_override_code” is a long hexadecimal string that needs to be provided in order for us to be able to fetch historical balances. The multicall contract I am using can be found . But, it’s also possible to use other multicall contracts. here Final remarks All code and test cases can be found on my Github here: . https://github.com/martkir/get-erc20-balance If you have any questions or want to give feedback on anything I have written you can let me know on Twitter @martkiro If you’re working with onchain data you might also be interested in checking out where we are indexing the Ethereum blockchain :) https://www.syve.ai

The Best Method for Bulk Fetching ERC20 Token Balances

Too Long; Didn't Read

@martinkirov

Credibility

RELATED STORIES