Moving an Amazon S3 bucket from one AWS account to another sounds simple… until you try it with 4–5 terabytes of data. 4–5 terabytes At that scale, the usual methods — aws s3 cp or cross-account replication — feel like mailing every single file via carrier pigeon. They’re fine for gigabytes, but you’ll be waiting days (and paying more than you’d like) for terabytes. aws s3 cp That’s where AWS DataSync comes in. In my case, it was 10× faster than the “normal” way, fully automated, and secure. Here’s how I did it. AWS DataSync 10× faster Why DataSync for Large Migrations? AWS DataSync is built for bulk movement of data — think terabytes to petabytes. It: Moves data directly between AWS services (or from on-prem to AWS) without middleman storage.
Parallelizes transfers, making them much faster.
Handles metadata, object tags, and ACLs automatically.
Can be run incrementally so you can do a cutover with minimal downtime. Moves data directly between AWS services (or from on-prem to AWS) without middleman storage. Moves data directly between AWS services Parallelizes transfers, making them much faster. Parallelizes transfers Handles metadata, object tags, and ACLs automatically. Can be run incrementally so you can do a cutover with minimal downtime. When you’re moving 4–5 TB between AWS accounts, those benefits matter. 4–5 TB Step 1: Prepare the Buckets We need two buckets: Source bucket in the original AWS account (the one holding your 5 TB of data).
Destination bucket in the target AWS account. Source bucket in the original AWS account (the one holding your 5 TB of data). Source bucket Destination bucket in the target AWS account. Destination bucket Both must: Exist before starting.
Be in the same AWS region if you want max speed & lower cost.
Have versioning optional (DataSync doesn’t require it like replication does). Exist before starting. Be in the same AWS region if you want max speed & lower cost. Have versioning optional (DataSync doesn’t require it like replication does). Step 2: IAM Roles and Permissions DataSync needs permission to read from the source and write to the destination — across accounts. This is where IAM roles and bucket policies come in. Source Bucket Policy (Allow destination account’s DataSync role and logged-in user to read & list objects) Source Bucket Policy {
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Principal": {
    "AWS": [
     "arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
     "arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
    ]
   },
   "Action": [
    "s3:GetBucketLocation",
    "s3:ListBucket",
    "s3:ListBucketMultipartUploads"
   ],
   "Resource": "arn:aws:s3:::source_bucket"
  },
  {
   "Effect": "Allow",
   "Principal": {
    "AWS": [
     "arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
     "arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
    ]
   },
   "Action": [
    "s3:AbortMultipartUpload",
    "s3:DeleteObject",
    "s3:GetObject",
    "s3:ListMultipartUploadParts",
    "s3:PutObjectTagging",
    "s3:GetObjectTagging",
    "s3:PutObject"
   ],
   "Resource": "arn:aws:s3:::source_bucket/*"
  }
 ]
} {
 "Version": "2012-10-17",
 "Statement": [
  {
   "Effect": "Allow",
   "Principal": {
    "AWS": [
     "arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
     "arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
    ]
   },
   "Action": [
    "s3:GetBucketLocation",
    "s3:ListBucket",
    "s3:ListBucketMultipartUploads"
   ],
   "Resource": "arn:aws:s3:::source_bucket"
  },
  {
   "Effect": "Allow",
   "Principal": {
    "AWS": [
     "arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
     "arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
    ]
   },
   "Action": [
    "s3:AbortMultipartUpload",
    "s3:DeleteObject",
    "s3:GetObject",
    "s3:ListMultipartUploadParts",
    "s3:PutObjectTagging",
    "s3:GetObjectTagging",
    "s3:PutObject"
   ],
   "Resource": "arn:aws:s3:::source_bucket/*"
  }
 ]
} Destination Account Role Policy (This is the role DataSync uses to access the source bucket) Destination Account Role Policy {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::source_bucket"
        },
        {
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:GetObjectTagging",
                "s3:ListBucket",
                "s3:PutObjectTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::source_bucket/*"
        }
    ]
} {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::source_bucket"
        },
        {
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:GetObjectTagging",
                "s3:ListBucket",
                "s3:PutObjectTagging"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::source_bucket/*"
        }
    ]
} Destination Bucket Policy (Allow DataSync role and destination account user to write data) Destination Bucket Policy {
 "Version": "2008-10-17",
 "Statement": [
  {
   "Sid": "DataSyncCreateS3LocationAndTaskAccess",
   "Effect": "Allow",
   "Principal": {
    "AWS": [
     "arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
     "arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
    ]
   },
   "Action": [
    "s3:GetBucketLocation",
    "s3:ListBucket",
    "s3:ListBucketMultipartUploads",
    "s3:AbortMultipartUpload",
    "s3:DeleteObject",
    "s3:GetObject",
    "s3:ListMultipartUploadParts",
    "s3:PutObject",
    "s3:GetObjectTagging",
    "s3:PutObjectTagging"
   ],
   "Resource": [
    "arn:aws:s3:::dist_bucket",
    "arn:aws:s3:::dist_bucket/*"
   ]
  }
 ]
} {
 "Version": "2008-10-17",
 "Statement": [
  {
   "Sid": "DataSyncCreateS3LocationAndTaskAccess",
   "Effect": "Allow",
   "Principal": {
    "AWS": [
     "arn:aws:iam::DIST_ACCOUNT_ID:role/datasync-role",
     "arn:aws:iam::DIST_ACCOUNT_ID:user/distention_account_logged_in_user"
    ]
   },
   "Action": [
    "s3:GetBucketLocation",
    "s3:ListBucket",
    "s3:ListBucketMultipartUploads",
    "s3:AbortMultipartUpload",
    "s3:DeleteObject",
    "s3:GetObject",
    "s3:ListMultipartUploadParts",
    "s3:PutObject",
    "s3:GetObjectTagging",
    "s3:PutObjectTagging"
   ],
   "Resource": [
    "arn:aws:s3:::dist_bucket",
    "arn:aws:s3:::dist_bucket/*"
   ]
  }
 ]
} Step 3: Create the DataSync Locations We need two “locations” — one for the source bucket, one for the destination. Example AWS CLI for source bucket: aws datasync create-location-s3 \
  --s3-bucket-arn arn:aws:s3:::s3-source-01 \
  --s3-storage-class STANDARD \
  --s3-config BucketAccessRoleArn="arn:aws:iam::1234567890:role/datasync-role" \
  --region us-east-1 aws datasync create-location-s3 \
  --s3-bucket-arn arn:aws:s3:::s3-source-01 \
  --s3-storage-class STANDARD \
  --s3-config BucketAccessRoleArn="arn:aws:iam::1234567890:role/datasync-role" \
  --region us-east-1 Repeat for the destination bucket with its own bucket ARN. Step 4: Create the DataSync Task Once both locations are created, set up a DataSync task to copy objects from the source location to the destination location. You can: Enable metadata copy (preserves timestamps, tags).
Run incremental syncs until final cutover.
Use the console to monitor transfer speed and completion. Enable metadata copy (preserves timestamps, tags). metadata copy Run incremental syncs until final cutover. incremental syncs Use the console to monitor transfer speed and completion. Step 5: Run and Monitor the Transfer For 4–5 TB, you’re not finishing in minutes, but you’ll still see a huge improvement over traditional methods. In my migration, the speed difference was dramatic — hours instead of days. 4–5 TB Pro Tips for Large Transfers Same region is cheaper & faster — cross-region transfers can double the bill.
Incremental runs are your friend — run DataSync multiple times before final cutover so you only copy changes on the last run.
Keep IAM tight — you’re opening cross-account access; remove it after the migration.
Tag the migration, so you can track transfer costs in AWS Cost Explorer. Same region is cheaper & faster — cross-region transfers can double the bill. Same region is cheaper & faster Incremental runs are your friend — run DataSync multiple times before final cutover so you only copy changes on the last run. Incremental runs are your friend Keep IAM tight — you’re opening cross-account access; remove it after the migration. Keep IAM tight Tag the migration, so you can track transfer costs in AWS Cost Explorer. Tag the migration Final Thoughts Migrating 5 TB of S3 data between AWS accounts used to mean choosing between slow and free-ish or fast and expensive. DataSync changes that equation — giving you fast, secure, repeatable migrations without manual scripting or downtime. 5 TB of S3 data slow and free-ish fast and expensive fast, secure, repeatable migrations Think of it as upgrading from carrying buckets of water to installing a high-pressure pipeline. Once you’ve used it, you won’t go back.

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Amazon

Node.js Made Easy: Rate Limiting and Throttling for Smooth Traffic Flow

Migrating S3 Buckets Between AWS Accounts Like a Pro (Without Losing Your Sanity)

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

From Junior to Senior Node.js Developer: Lessons Learned

4EVERLAND's Bucket API for Decentralized Object Storage

ALB Controller Logs With Vector and OpenSearch: A Guide

AWS S3 Bucket Policy Gone Wrong

Basics of AWS Tags & Terraform with S3 - Part 1

Bridging Domains:How to Migrate From GoDaddy to AWS S3 in Style!

From Junior to Senior Node.js Developer: Lessons Learned

4EVERLAND's Bucket API for Decentralized Object Storage

ALB Controller Logs With Vector and OpenSearch: A Guide

AWS S3 Bucket Policy Gone Wrong

Basics of AWS Tags & Terraform with S3 - Part 1

Bridging Domains:How to Migrate From GoDaddy to AWS S3 in Style!

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps