TLDR
Amazon Kinesis Data Streams has a free retention period of only 24 hours, beyond which the cost factor kicks in. The team is trying to replicate and sync over 220 tables from DynamoDB to Spanner and Bigtable. We needed to move the streaming data to Pub/Sub because some Data Science jobs needed topic updates. Google has already taken a stab at it, but we won’t cover them here because Google has taken a shot at it. We decided to employ a two-step data transfer approach to transfer data from AWS to GCP. We discussed multiple options, starting with the seemingly straightforward Lambda Lambda-to-AWC transfer.via the TL;DR App
no story
Written by roshanpatil | Product Developer at CloudCover
