Large Scale Distributed Computing Architect
Traffic replication is a process of making copies of incoming traffic. This could be all traffic coming into a particular host or only traffic coming to a particular port(s) on a host.
Fig 1: Traffic Replication
For any web application it's now a norm to have separate environments for Development, Testing/Staging/Pre-Production and Production. The utilization of the application is each environment is varied and the traffic pattern on each is different.
Fig 2: Application Environments
Fig 3: Environment Utilization
Even after all the performance and automated tests it's still possible for new bugs to pop-up in production environment.
Common causes for the bugs :
Now, what if we had an environment (say pre-production) to which the replicated production traffic can be sent to. This environment would receive a wide variety of requests and also follow a traffic pattern similar to production. This makes it easier to validate any new code before pushing to production and also to troubleshoot any issues that pop up only in production environment and hard to reproduce in non-prod environments.
Fig 4: Replicate Production Traffic
In this section I will explain the different ways to replicate traffic in AWS along with the pros & cons of each approach.
Approach #1 - Traffic Replication Using AWS VPC Mirroring
AWS has a VPC mirroring service which allows one to replicate traffic coming into an ENI (elastic networking interface) to another ENI or to a Load Balancer.
Fig 5: Architecture: VPC Traffic Mirroring
The source for traffic mirroring is an ENI. Even load balancers are backed by multiple (based on incoming scale of traffic) ENI's and it's possible to replicate traffic directly from the Load Balancers instead of hooking in with the target host ENI's.
AWS takes the source traffic packet and wraps that as a VXLAN packet. The original packet is placed in the payload of the VXLAN packet. The target receiving the replicated packet has to unwrap the VXLAN packet and extract the original payload for processing.
Fig 6: VXLAN unwrap & Replay
GoLang Script to Process Replicated Traffic:
Download Script : https://github.com/elang2/AWS-Traffic-Replication/blob/master/aws-traffic-replication.go
Fig 7: GoLang Script - Part 1
Fig 8: GoLang Script - Part 2
Fig 9: GoLang Script - Part 3
Approach #2 - Traffic Replication VIA A Third Party Tool
GoReplay is an excellent tool to replicate traffic. The tool can be installed on an EC2 instance and can be configured to replicate traffic to any target.
Traffic replication is an effective way of maintaining a near production like environment with minimal effort. The benefits of replication goes beyond just monitoring and troubleshooting issues. The replicated information can be used in performing data analysis, analyze traffic pattern and in detecting security vulnerabilities and attacks on the system.
In my opinion and based on personal experience I find the AWS VPC mirroring to be the best approach for replicating traffic for applications deployed on AWS.