One of the strong use cases for MinIO is the fact that it can run anywhere and on everything. As the industry slowly shifts towards repatriating data to a colo or data center, more and more companies want the same object storage capabilities that they had in the cloud, with full control of the infrastructure. Why would you want to have data closer to home? There are a number of reasons, but first and foremost is cost. The public cloud has become very expensive. For example, some time ago I had an ElasticSearch managed cluster running in AWS. I was eager to try out this new managed service, but I was not eager to discuss my surprise $30K bill with my boss. This was a painful, yet familiar wake-up call because at that moment I realized that I had just paid AWS six months of cloud budget to do something that I could have set up myself. The moral of the story is that unless you are very careful and monitor your cloud spend closely, it can get out of control very fast. There is also the matter of security. No matter where your data is located in the public cloud, it's almost always on a node or storage pool that is shared by someone completely unrelated to you; this is the nature of the cloud because it is how virtualization works. The cloud provides a warm feeling of comfort because now someone else must tackle security challenges, but if there are any security-related issues, there will be no insight into the issue (if someone was even able to detect it) and how to resolve it. The feeling of comfort rapidly evaporates when you get stuck securing someone else’s infrastructure in order to protect your data. Many enterprises have enjoyed the return to total control provided by repatriating to MinIO onto hardware they manage. To make the most of your repatriation efforts, MinIO comes with a number of enterprise-ready features such as to ensure data integrity, to siphon off data to a cold storage tier, which saves objects as a collection of data and parity blocks, and reconstructs them on the fly without any additional hardware or software. In addition to these, MinIO supports both at and in . This ensures that data is encrypted in all facets of the transaction from the moment the call is made until the object is placed in the bucket, where it is then protected with IAM S3-style policies and a built-in or external IDP, see for more information. Bitrot Protection Tiering Erasure Coding Encryption rest transit MinIO Best Practices - Security and Access Control Repatriation must be planned thoroughly and carefully. If you are dealing with petabytes of data generally it's more cost-effective to have your own infrastructure and servers to run on, you can even build a private cloud with your own (or leased) hardware. In addition, this also includes managing the real estate (colo space), power/UPS, cooling/HVAC, among other components. Don’t be deterred by these as we will demonstrate to you how you can migrate yet the overall ROI is still better than in the public cloud. A private cloud is like an apartment (as our CEO AB Periasamy likes to say). You are in full control of the costs and expenses that are associated with it, you never wake up to an alert of a surprise bill caused by some recursive loop function that ran overnight. There is, of course, some friction to moving when you are trying to make things better, for instance, when you are trying to expand a highway you inevitably have to close down some lanes so construction can proceed safely, but once it's done you will not only be able to drive on the original lanes but also the newly built ones to handle the capacity. Two of the most important cost considerations we need to make in the public cloud are the amount of storage space you need and egress costs when accessing/moving that data – these can be around 39% and 42% higher respectively in comparison to your own hardware in your data center or colocation facility. In addition to this, some of the other cost factors to consider are software, hardware, networking/switches, real estate/rack space/colocation rental, S3-API calls – everything you can think of and more. Learn more about possible savings that result from moving to your own private cloud in . The Lifecycle of the Cloud Between the public cloud and your data center exists a middle ground where you can have full control over infrastructure hardware, without the high initial cost of investment. , as the name states, provides bare metal servers with the exact specifications requested by the customer. If you want to use NVMe SSDs, then you can add those disks to the bare metal server. Equinix provides a management API to simplify hardware deployment and operations. To the developer/end user, it's as straightforward as launching an instance in the cloud. In fact, there is even a Terraform provider for Equinix Metal (which we’ll show you later). Equinix Metal Let's get started! Deploy the Infrastructure While we can deploy resources manually, the DevOps in me wants to automate at least some of the repetitive portions of this process to save time and effort,  especially when we want to do site-to-site replication among other things. Set Up Equinix Metal Terraform Equinix is one of the few bare metal providers that have an API to fully automate the infrastructure management process. Using their , you can automate deploying physical servers, shutting them down and even terminating them. You can do all this without using your own hardware, switches, routers and other resources. This is as close as you can get to public-cloud-level automation while still guaranteeing that no one else is sharing your hardware. Because Equinix Metal supports a myriad of instance types and storage options and interconnects such as SAS or SATA, and SSD, NVMe SSD, or HDD, in a variety of sizes. You can also configure the hardware that MinIO runs on to your exact specifications – right down to the exact type of drive to house MinIO partitions. API No one expects you to write Python scripts to talk to the Metal API; Equinix Metal has a Terraform Provider that allows us to connect to it and provide the high-level information needed to deploy cluster resources, while abstracting the internal jugglery required to get the networking, hardware, MinIO and other applications set up. provider "metal" {
auth_token = var.auth_token
} If you don’t have Terraform installed already, you can download it from their . downloads page Clone the GitHub repo to your local workstation. equinix/terraform-metal-distributed-minio git clone https://github.com/equinix/terraform-metal-distributed-minio.git Go into the repo and initialize Terraform so that it can download all the required modules and plugins from upstream. $ cd terraform-metal-distributed-minio
$ terraform init This will ensure all the modules required are downloaded automatically. Now, let's make sure a couple of mandatory variables are set. You can either set them as or there is a file in the repo cloned above called , which you can copy as . environment variables vars.template cp vars.template terraform.tfvars Ultimately, whichever method you choose, you need to set the following two variables auth_token project_id You can find more information on these in the docs. API There are several other variables that you can modify in , and we will modify the following later when we do site-to-site replication. terraform.tfvars facility: https://deploy.equinix.com/developers/docs/metal/locations/metros/ minio_region_name: The region where the MinIO cluster is deployed. Once you have your preferred configuration set, apply the Terraform plan. If the plan looks okay, run the command. approve $ terraform plan
$ terraform apply --auto-approve If the resources have been applied properly with the right configuration then the resulting output should look something like this Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:

minio_access_key = Xe245QheQ7Nwi20dxsuF
minio_access_secret = 9g4LKJlXqpe7Us4MIwTPluNyTUJv4A5T9xVwwcZh
minio_endpoints = [
  "minio-storage-node1 minio endpoint is http://147.75.65.29:9000",
  "minio-storage-node2 minio endpoint is http://147.75.39.227:9000",
  "minio-storage-node3 minio endpoint is http://147.75.66.53:9000",
  "minio-storage-node4 minio endpoint is http://147.75.194.101:9000",
]
minio_region_name = us-east-1 This is the whole shebang. When you see this output, not only have your physical servers been provisioned, but also MinIO has been deployed to these nodes, and the nodes have been configured as a cluster of distributed storage. Access MinIO Cluster We used Terraform to automate most of the process, so now all that is left is to access the MinIO cluster. Our recommended tool is to use . Use the following command to download the binary mc curl https://dl.min.io/client/mc/release/linux-amd64/mc \
  --create-dirs \
  -o $HOME/minio-binaries/mc

chmod +x $HOME/minio-binaries/mc
export PATH=$PATH:$HOME/minio-binaries/ Create an alias that points to the MinIO cluster we deployed mc config host add minio1 $MINIO_ENDPOINT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY You can replace the variables above with the values that you set while launching the MinIO cluster via Terraform, but make sure to set the alias name to . This will make sense later when we show you how to do site-to-site replication. minio1 Check to see if you are able to connect successfully by fetching some metadata from the cluster $ mc admin info minio1 --json | jq .info.backend

{
  "backendType": "Erasure",
  "onlineDisks": 48,
  "rrSCData": 6,
  "rrSCParity": 2,
  "standardSCData": 6,
  "standardSCParity": 2
} If you see an output similar to the above then you are able to successfully access the MinIO cluster via the command. So what’s next? When should we migrate the data from S3? mc Load Balancing the MinIO Cluster We can migrate the data from S3, or even add some of our own data, and start using the cluster. But let’s take it one step further. We want to achieve the same level of redundancy as AWS S3, meaning if one site goes down we want to make sure our data is accessible on another site. AWS accomplished this with regions, but how do we accomplish this with MinIO? Now, we can see the beauty of the little automation we did with Terraform earlier. Let me show you how simple it is to get another MinIO region up in Equinix Metal. Lets our source repo again, but this time in a new directory git clone terraform-metal-distributed-minio-site-2 git clone https://github.com/equinix/terraform-metal-distributed-minio.git terraform-metal-distributed-minio-site-2 Go into the repo and initialize Terraform so that it can download all the required modules and plugins from upstream similar to the original MinIO deployment. terraform-metal-distributed-minio-site-2 $ cd terraform-metal-distributed-minio-site-2
$ terraform init Once all the modules have been downloaded, copy the variables file and set the two variables cp vars.template terraform.tfvars auth_token project_id So far the process should look very similar to how we launched the first cluster, but this is where things will differ. Let's set the variables that differentiate the second site from the first site. First, let's set the to or pick one from this list of . Next set the to or anything that differentiates it from the other cluster. facility sv16 facilities minio_region_name us-west-1 Run the plan to ensure the changes you made get reflected in the output. $ terraform plan
$ terraform apply --auto-approve If the resources have been applied properly, with the right configuration, then the resulting output should look something like this Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:

minio_access_key = Xe245QheQ7Nwi20dxsuF
minio_access_secret = 9g4LKJlXqpe7Us4MIwTPluNyTUJv4A5T9xVwwcZh
minio_endpoints = [
  "minio-storage-node1 minio endpoint is http://144.45.65.29:9000",
  "minio-storage-node2 minio endpoint is http://144.45.39.227:9000",
  "minio-storage-node3 minio endpoint is http://144.45.66.53:9000",
  "minio-storage-node4 minio endpoint is http://144.45.194.101:9000",
]
minio_region_name = us-west-1 If you see as then you have successfully launched the second cluster. Let's get that added to . minio_region_name us-west-1 mc mc config host add minio2 $MINIO_ENDPOINT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY Set the alias name to and check to see if you are able to connect successfully by fetching some metadata from the cluster minio2 $ mc admin info minio2 --json | jq .info.backend

{
  "backendType": "Erasure",
  "onlineDisks": 48,
  "rrSCData": 6,
  "rrSCParity": 2,
  "standardSCData": 6,
  "standardSCParity": 2
} At this point, you should have 2 sites: and . minio1 minio2 Let's set up replication across both clusters $ mc admin replicate add minio1 minio2
Requested sites were configured for replication successfully. Verify the both sites are configured correctly mc admin replicate info minio1

SiteReplication enabled for:

Deployment ID                    	| Site Name   	| Endpoint
f96a6675-ddc3-4c6e-907d-edccd9eae7a4 | minio1      	| http://<site1_public_ip>
0dfce53f-e85b-48d0-91de-4d7564d5456f | minio2      	| http://<site2_public_ip> Check to make sure replication is working properly mc admin replicate status minio1

Bucket replication status:
No Buckets present

Policy replication status:
●  5/5 Policies in sync

User replication status:
No Users present

Group replication status:
No Groups present Test by creating a bucket in minio1 /opt/minio-binaries/mc mb minio1/testbucket Add any object into the bucket /opt/minio-binaries/mc cp my_object  minio1/testbucket List the objects in the other sites, in this case on minio2 /opt/minio-binaries/mc ls minio2/testbucket
[2023-07-20 18:52:09 UTC] 3.0KiB STANDARD my_object As you can see it's almost instantaneous to replicate data to other MinIO deployments, even though they are geographically disparate. Let’s do a quick test to see if this is really as simple as it appears. Remember that MinIO is a drop-in replacement for AWS S3, so everything that is supposed to work with S3 will work with MinIO too. In this case, we will use a Terraform to upload an object to a MinIO bucket. In Terraform this is done through the AWS provider which is essentially a module that connects to the AWS API to perform various operations in the AWS ecosystem, but in this case, we will use the Terraform AWS S3 resource to access MinIO bucket. Create an AWS provider in Terraform like below. Make sure you update the details to match the Equinix Metal cluster we just deployed. minio1 provider "aws" {
    region = "us-east-1"
    access_key = "Xe245QheQ7Nwi20dxsuF"
    secret_key = "9g4LKJlXqpe7Us4MIwTPluNyTUJv4A5T9xVwwcZh"
    skip_credentials_validation = true
    skip_metadata_api_check = true
    skip_requesting_account_id = true
    s3_force_path_style = true
    endpoints {
        s3 = "http://147.75.65.29:9000"
    }   
} Upload a file using the terraform resource aws_s3_bucket_object resource "aws_s3_bucket_object" "object" {
    bucket = "public"
    key = "my_file_name.txt"
    source = "path/to/my_file_name.txt"
    etag = filemd5("path/to/my_file_name.txt")
} As you can see above, we haven’t used any MinIO-specific Terraform resource, we are using AWS provider resource. Even though we are using the existing AWS S3 Terraform resource, the object store is completely powered by production enterprise-grade MinIO. aws_s3_bucket_object Migrating Data from AWS S3 We now have all the building blocks ready to go for you to have production-grade object storage and total control of the entire infrastructure. Next, we’ll migrate data that’s already in S3. There are a number of ways you can migrate your data from AWS S3 to MinIO, but the one we recommend is using . mc is a Swiss army knife of data synchronization. It can copy objects from S3 or S3-API-compatible object stores and mirror them to MinIO. One of the more popular use cases of this is mirroring an Amazon S3 bucket to MinIO in order to expose data to non-AWS applications and services. mc mirror Create a new with an access key and secret key, allowing access only to our bucket. Save the generated credentials for the next step. IAM policy /opt/minio-binaries/mc alias set s3 https://s3.amazonaws.com BKIKJAA5BMMU2RHO6IBB V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12 --api S3v4 Use to copy the data from S3 to MinIO mc mirror mc mirror s3/mybucket minio1/testbucket Depending on the amount of data, network speeds and the physical distance from the region where the bucket data is stored, it might take a few minutes or more for you to mirror all the data. You will see a message when is done copying all the objects. mc Once the data is copied or while the data is being copied, list the contents of the bucket in site. You will notice that some of the data is already there from . minio2 minio1 /opt/minio-binaries/mc ls minio2/testbucket
[2022-12-19 18:52:09 UTC] 3.0KiB STANDARD all_object s Ultimately the laptop where you are running from is a bottleneck because the data has to traverse through the system where command is executed. This could be several petabytes of data which could take days if not weeks to migrate depending on the network speed. To migrate data from S3 to MinIO, there is a more efficient method called Batch Replication, please see to learn more about Batch Replication and other migration best practices. mc mirror mc mirror How to Repatriate from AWS S3 to MinIO Put the Pedal to the Metal This blog post demonstrated that MinIO running on Equinix Metal NVMe SSDs, in a site-to-site replication configuration, will give you the same level, if not more, of performance, data durability and resilience at a fraction of the cost of S3 while retaining full control over your cloud. Do you have 100% control of all the infrastructure? Not quite. The switches, routers and other networking gear are managed by Equinix, but the advantages of being on their network exceed the disadvantages. You can get point-to-point, WAN, or other dedicated circuits to virtually connect to any other provider out there. In essence, you can have a private circuit connected directly to AWS (via Equinix Connect) and then you can move your data 10 times as fast, all the while being secure because it's not traversing the open public internet, and only your data goes through that circuit. Moreover, MinIO repeatedly show very little (<1%) throughput performance degradation with encryption turned on, therefore we recommend that all MinIO deployments use encryption at rest and all MinIO deployments should also using TLS. Congratulations, now your data is on a more secure,  yet transparent, system where you have full control and accountability. benchmarks secure network communications If you have any questions on migrating from AWS S3 to MinIO on Equinix Metal, be sure to reach out to us on ! Slack Also published . here

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

How to Migrate from AWS S3 to MinIO on Equinix Metal

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Closer Look Into the MinIO Enterprise Object Store Firewall

$DEFI Token Hits 7 Major Exchanges: A Milestone Achievement

$JTC Network To List On BitMart Exchange

$500k Presale: TG.Casino Passes Milestone with Upcoming Telegram-Powered Platform

$3 Million in Seed Funding for Web3 Founders Announced By Necto Labs

$2M Backing and a Vision: How GAM3S.GG is Reshaping Web3 Gaming

A Closer Look Into the MinIO Enterprise Object Store Firewall

$DEFI Token Hits 7 Major Exchanges: A Milestone Achievement

$JTC Network To List On BitMart Exchange

$500k Presale: TG.Casino Passes Milestone with Upcoming Telegram-Powered Platform

$3 Million in Seed Funding for Web3 Founders Announced By Necto Labs

$2M Backing and a Vision: How GAM3S.GG is Reshaping Web3 Gaming

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps