How To Successfully Migrate Data From DynamoDB To Bigtable And Spanner

Written by roshanpatil | Published 2020/10/07
Tech Story Tags: cloud-computing | cloud | google-cloud-platform | cloud-native | cloudcover | cloud-migration | aws | amazon-web-services

TLDR Amazon Kinesis Data Streams has a free retention period of only 24 hours, beyond which the cost factor kicks in. The team is trying to replicate and sync over 220 tables from DynamoDB to Spanner and Bigtable. We needed to move the streaming data to Pub/Sub because some Data Science jobs needed topic updates. Google has already taken a stab at it, but we won’t cover them here because Google has taken a shot at it. We decided to employ a two-step data transfer approach to transfer data from AWS to GCP. We discussed multiple options, starting with the seemingly straightforward Lambda Lambda-to-AWC transfer.via the TL;DR App

no story

Written by roshanpatil | Product Developer at CloudCover
Published by HackerNoon on 2020/10/07