Migrating Container Images and Helm Charts from Harbor to ECR

The rationale behind this project stems from the various stability issues encountered with Harbor, other registries, prompting a transition away from it.

Project Plan

1. Set up ECR infrastructure using Terraform, This includes private repos creation.

   
   

provider "aws" 
{region = "us-east-1" # Change to your AWS region# 
Optionally, specify a profile name if you're using AWS named profiles
# profile = "your-profile-name"}resource "aws_ecr_repository" "my_ecr_repo" {name = "my-ecr-repo" # Change this to your desired repo nameimage_tag_mutability = "MUTABLE"image_scanning_configuration {scan_on_push = true}
# If you need to create more repositories, you can either duplicate this block# and change the name, or use a more dynamic approach with `count` or `for_each`.# Optional: Enable lifecycle policy to manage image count or untagged imageslifecycle_policy {policy = jsonencode({rules = [{rulePriority = 1description = "Expire untagged images"selection = {==tagStatus = "untagged"countType = "imageCountMoreThan"countNumber = 100}action = {type = "expire"} }, ] })}}

2. Implement replication rules for cross-region and cross-account functionality.

mutable-repos = ["apps/signatures",]
ecr_replication_destinations_rules = 
{lab1 = {region = "us-west-2"id = "<>"}
lab2 = {region = "us-east-1"id = "<>"}
stage1 = {region = "ap-southeast-1"id = "<>"}
stage2 = {region = "us-east-1"id = "<>"}
prod = {region = "us-west-2"id = "<>"}}

3. Once the Infrastructure is available, adapt application pipelines to push images and Helm charts to ECR along with Harbor.

Activities To be Performed

1. Prepare User “ecr_role” with necessary below permissions, facilitating automated image/chart pushes to ECR in the CI/CD section.

2. Established ECR private repositories for all applications, replicating the setup across remaining environments to manage replication costs.
3. Updated and executed pipelines for all application components to ensure visibility of images in both locations (Harbor and ECR).
4. Provided implementation guidelines to the Application team, detailing necessary changes and their locations.
5. Transitioned to ECR from Harbor across all environments (Lab, Stage, Production).

Moving Forward with Helm

Our idea is to maintain the whole CD through ArgoCD. So for that, we need to add this as a repository in Argo. A new Challenge emerges with OCI calls in Helm implementation as it will not use HTTPs calls.

Resolution

1. Install External secrets in the cluster where we have Argo helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace \
--set installCRDs=true

   
   

2. Create AWS credentials in K8 secrets, or if you are using gitlab for pulling the data to argo, then you can configure the same under

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
echo -n AWS_SECRET_ACCESS_KEY > ./secret-access-key
echo -n AWS_ACCESS_KEY_ID > ./access-key
kubectl create secret generic awssm-secret --from-file=./access-key --from-file=./secret-access-key

3. The secret created in previous step, swasm-secret used as below

apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
name: ecr-gen
spec:
# specify aws region (mandatory)
region: us-west-1
auth:
# 1: static credentials
# point to a secret that contains static credentials
# like AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
secretRef:
accessKeyIDSecretRef:
name: "awssm-secret"
key: "access-key"
secretAccessKeySecretRef:
name: "awssm-secret"
key: "secret-access-key"

Challenges

The predominant challenge involved handling Helm charts from ArgoCD when storing an OCI Helm repo in ECR, necessitating token rotation using “External Secrets Operator”. Achievements: Achieved zero downtime across all environments despite numerous redeployments and adherence to application team timelines. Progress made on the migration from Harbor to ECR, with no disruptions in operations.

If this post was helpful, please do follow and click the clap 👏 button below to show your support.

Follow me on Linked-in for further updates- https://www.linkedin.com/in/chaitanya-kanth-tummalachervu-296956121/