There are many code optimizations that can improve the speed of data processing in high frequency trading (HFT), such as switching from Python to C++, using specific libs. In tradFi HFT, it also includes switching to FPGA-based network adapters and writing the trading logic in Verilog.
In crypto trading, there is usually no option to install your own network hardware to a cloud server, even if you have a dedicated/bare metal instance. But there are other optimizations that can be done when it comes to network/telecom service providers who have their own autonomous systems (AS) that are connected to cloud providers. They do it in a way that can help you to get rid of milliseconds or even tens of milliseconds of latency between a cloud provider’s regions. That can allow you to send data from one remote crypto exchange to another crypto exchange faster than your competitors.
I don’t work for any network/telecom service provider, and have no active contracts with the network provider I use in my example below. This article is my initiative and hope it can be helpful for people in crypto trading and for junior network engineers.
I was interested to interconnect Binance and Bitmex locations to get market data from a remote exchange with predictable latency and then to process it by a local trading system. Amazon allows you to do inter-region VPC peering to allow connectivity between your own private IPs in different regions. When I first tried to connect two AWS regions with VPC peering I got some latency fluctuations. Although the volatility was not huge, it was shifting every couple of hours because AWS has multiple channels between regions and balances traffic over those links. However, in the end, my Tokyo VM, located in the same region as Binance, had a 49ms latency advantage for getting Bitmex market data!
AWS VPC Peering gave me 201.5ms latency:
--- 10.10.4.28 ping statistics ---
255 packets transmitted, 255 received, 0% packet loss, time 254274ms
rtt min/avg/max/mdev = 199.858/201.542/301.538/10.553 ms
So I decided to try an evaluation period for a line from Avelacom telecom provider to give me better latency and predictability. In order to interconnect VPCs, I started from scratch:
1. Created new VPCs in the regions you want to interconnect. You don’t want to use default network subnets in regions since the default CIDR used by AWS would be the same for every region.
AWS Console → Services → VPC → Your VPCs → Create VPC
For my example I would be using
10.10.4.0/2410.10.5.0/24
2. I would also create Subnets for each of three availability zones (AZ) in each region:
10.10.5.0/2610.10.5.64/2610.10.5.128/26For Dublin I would use
10.10.4.0/2610.10.4.64/2610.10.4.128/26
AWS Console → Services → VPC → Subnets → Create Subnet
When creating a subnet, choose the VPC you created in the first step, choose AZ in which your subnet will reside and name your subnets somehow.
3. In order to let traffic from future ec2 instances be routed to the internet you have to create Internet Gateway in
AWS Console → Services → VPC → Internet gateways → Create Internet Gateway.
After you named and created new IGW you can immediately press Attach to VPC and choose fresh VPCs we created recently.
4. Add route to the Routing Table that was automatically created for your VPC in
AWS Console → Services → VPC → Route tables
Choose route table that is attached to the new VPC you created and press Edit Routes in Routes section. Add
0.0.0.0/0
5. Create your instances in
AWS Console → Services → EC2 → Instances → Launch Instances
When creating ensure you are using our new VPCs and one of the subnets recently created. It might be handy to use Elastic IP to fix public IP address but you can just enable Auto-assignment of public IP if your connection to the exchange does not involve any IP whitelisting. You might want to experiment with placing your servers in different AZ according to the best latency you can get to the resource (Crypto Exchange) of interest in the region. But in this case I would just use 1a AZ in each region for the demo purposes.
6. Create Virtual Private Gateway
AWS Console → Services → VPC → Virtual private network (VPN) →Virtual Private Gateway → Create virtual private gateway
When creating use default ASN for each
7. Attach newly created Virtual Private Gateways to your VPCs.
8. Enable route propagation in:
AWS Console → Services → VPC →Route Tables (Select route table attached to your VPC)→ Route Propagation → Enable route propagation for created Virtual Private Gateway
9. In order to interconnect your VPGs with telecom provider, in my case it was Avelacom, that I provided with my Amazon Account ID and region names I wanted to interconnect beforehand:
Go to AWS Console → Services → Direct Connect → Virtual interfaces (Press on each) → Accept and choose which VPG you want to attach the line to.
10. You should inform provider after you accepted the requests in the Virtual interfaces section. After that it takes some time for provider to do configuration on their side.
Once you got confirmation from the provider everything is done, you should be able to test new latency between your virtual machines:
--- 10.10.4.28 ping statistics ---
3600 packets transmitted, 3600 received, 0% packet loss, time 3601277ms
rtt min/avg/max/mdev = 150.942/151.120/166.567/0.402 ms
Not bad, 151.1ms average latency! So how can we use that? Lets check if it gives you any benefit. Lets check public IP of the Bitmex in Ireland:
# ping bitmex.com
PING bitmex.com (18.66.171.111) 56(84) bytes of data.
64 bytes from server-18-66-171-111.dub56.r.cloudfront.net (18.66.171.111): icmp_seq=1 ttl=241 time=0.788 ms
Ok, looks like
18.66.171.1110.788 ms“Our servers are located at AWS EU-West-1.”
Checking ping from Tokyo VM to the same address:
# ping 18.66.171.111
PING 18.66.171.111 (18.66.171.111) 56(84) bytes of data.
64 bytes from 18.66.171.111: icmp_seq=1 ttl=217 time=201 ms
So we are getting 201ms to the same Bitmex address over the internet from Tokyo.
Using AWS VPC peering gave us ~201.5ms average latency so it would hardly give us any benefit. So lets check Avelacom’s test line.
I configured Wireguard tunnel between my two VMs in Dublin and Tokyo in order to let them have a directly connected route to each other. I would not go into describing the process of setting wireguard tunnels as a services here, as its out of the scope of this article. But following configuration was used on the endpoints:
Tokyo side:
[Interface]
Address = 192.168.20.1/32
PrivateKey = %specify_Tokyo_PrivateKey_here%
ListenPort = 51820
[Peer]
PublicKey = %specify_Dublin_PublicKey_here%
AllowedIPs = 192.168.20.2/32,18.66.171.111/32
Endpoint = 10.10.4.28:51820
PersistentKeepalive = 25
Dublin side:
[Interface]
Address = 192.168.20.2/32
PrivateKey = %specify_Dublin_PrivateKey_here%
ListenPort = 51820
[Peer]
PublicKey = %specify_Tokyo_PublicKey_here%
AllowedIPs = 192.168.20.1/32
Endpoint = 10.10.5.41:51820
PersistentKeepalive = 25
Adding Bitmex IP (
18.66.171.111/3218.66.171.111
root@ip-10-10-5-41:/home/ubuntu# ip route get 18.66.171.111
18.66.171.111 dev vpn-dublin src 192.168.20.1 uid 0
cache
In order to make Dublin VM to route my traffic outside I should enable IPv4 routing, disable RP filter and enable NAT for outgoing traffic via the default network interface:
sysctl -w net.ipv4.ip_forward=1
for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 1 > "$i"; done
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
So now my Tokyo machine should be able to reach Bitmex via Avelacom’s channel and my Dublin’s VM:
root@ip-10-10-5-41:/home/ubuntu# ping 18.66.171.111PING 18.66.171.111 (18.66.171.111) 56(84) bytes of data.
64 bytes from 18.66.171.111: icmp_seq=1 ttl=240 time=152 ms
64 bytes from 18.66.171.111: icmp_seq=2 ttl=240 time=152 msNice! Almost 25% lower latency!
Crypto trading newcomers might be confused that all the resources they use are hosted in the cloud. And it’s not obvious to them that they can use alternative but more optimal paths to reach their intended destinations. So, if your activity is sensitive to financial risks you might think of using more resilient solutions adopted from the traditional HFT world such as dedicated telecom lines. This would help to interconnect regions in a more efficient way. Latency will be improved not only because of alternative paths, but also because of connectivity optimization with the cloud provider termination points using AWS Direct Connect. Even if AWS uses similar paths, you will still get better latency. You can also be sure that all the data will arrive without packet losses because you can set guaranteed bandwidth on these lines.
