Guide to Integrating With Non-AWS Cloud Storage Services Supporting the S3 Protocol using Golang

The Simple Storage Service (S3) protocol is a standard interface developed by Amazon Web Services (AWS) for storing and retrieving data in the cloud. It’s a key-value store, meaning that each piece of data (an object) is associated with a unique identifier (a key) that you use to access that data. The S3 protocol is crucial in cloud storage services for several reasons:

1. Scalability
   S3 allows for virtually unlimited storage, making it ideal for applications that need to scale rapidly or handle large amounts of data.
2. Durability and Availability
   S3 is designed for 99.999999999% (11 9’s) of durability, meaning the chances of losing data are extremely low. It also guarantees 99.99% availability, ensuring that your data is accessible when you need it.
3. Security
   S3 provides robust security features, including encryption for data at rest and in transit, access control policies, and logging capabilities.
4. Interoperability
   Many cloud storage providers, not just AWS, support the S3 protocol, making it a versatile choice for developers.
5. Cost-Effective
   With S3, you only pay for the storage you use, making it a cost-effective solution for businesses of all sizes.

In summary, the S3 protocol is a reliable, secure, and scalable solution for storing data in the cloud, making it a cornerstone of modern cloud-based applications.

Setting Up Go Environment

install AWS SDK into our project

go get github.com/aws/aws-sdk-go-v2/aws
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/credentials
go get github.com/aws/aws-sdk-go-v2/feature/s3/manager
go get github.com/aws/aws-sdk-go-v2/service/s3

Simple project to upload file

AWS SDK Wrapper

package s3client

import (
 "context"
 "github.com/aws/aws-sdk-go-v2/aws"
 "github.com/aws/aws-sdk-go-v2/config"
 "github.com/aws/aws-sdk-go-v2/credentials"
 "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
 "github.com/aws/aws-sdk-go-v2/service/s3"
 "io"
 "log"
 "sync"
)

var (
 uploader *manager.Uploader
 once     sync.Once
)

func setup() {
 endpoint := "http://s3-test-endpoint.com"
 accessKey := "XVXQ8Q3QGQ9QXQ8QXQ8Q"
 secretKey := "/XQ8Q3QGQ9QXQ8QXQ8QXQ/Q3QGQ9QXQ8QXQ8QXQ8"

 resolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
  return aws.Endpoint{
   URL:               endpoint,
   HostnameImmutable: true,
   Source:            aws.EndpointSourceCustom,
  }, nil
 })
 awsConfig, err := config.LoadDefaultConfig(context.TODO(),
  config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(accessKey, secretKey, "")),
  config.WithEndpointResolverWithOptions(resolver),
  config.WithRegion("auto"),
 )
 if err != nil {
  log.Fatalf("failed to load config, %v", err)
 }

 uploader = manager.NewUploader(s3.NewFromConfig(awsConfig)) // Initialize S3 client
}

func UploadFile(ctx context.Context, bucket, filename string, file io.Reader) (string, error) {
 once.Do(setup)
 result, err := uploader.Upload(ctx, &s3.PutObjectInput{
  Bucket: aws.String(bucket),
  Key:    aws.String(filename),
  Body:   file,
  ACL:    "public-read",
 })
 if err != nil {
  log.Fatalf("failed to upload file, %v", err)
 }

 return result.Location, nil
}

Why we need to wrap third-party library?

The purpose wrapper is to simplify the library’s interface, adapt it to fit our application’s specific needs, or add additional functionality. In this case we simplifies the process of uploading a file to S3 by providing a single UploadFile function that handles the setup and upload process. This makes it easier for the rest of our application to upload files to S3, as it only needs to call s3client.UploadFile with the appropriate parameters.

Anyway, let’s explain the code a little bit

We need to setup our project to connect with the cloud storage services, in the setupfunction there is 2 key steps that we need to highlight.

First, in resolver parameter we need to pass aws.EndpointSourceCustomto indicate we didn’t connect to AWS endpoint.

Secondly,config.WithRegion("auto")when load default config.

Then in UploadFile function we use once.Do(setup) to make sure setup function only called once.

Main Function

We will create simple server using chi, then create POST API to upload file to our S3 Cloud Storage. Also Hello World endpoint to make sure our server is working 😆.

package main

import (
 "github.com/go-chi/chi/v5"
 "github.com/yudaph/s3-integration/s3client"
 "net/http"
)

func main() {
 r := chi.NewRouter()

 r.Get("/", func(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte("Hello, World!"))
 })

 r.Post("/", HandleUpload)
 http.ListenAndServe(":8080", r)
}

func HandleUpload(w http.ResponseWriter, r *http.Request) {
 file, header, err := r.FormFile("image")
 if err != nil {
  http.Error(w, err.Error(), http.StatusBadRequest)
  return
 }
 defer file.Close()

 // TODO: Do Validation

 // Upload file to S3
 filename := "suffix_" + header.Filename
 url, err := s3client.UploadFile(r.Context(), "bucket-name", filename, file)
 if err != nil {
  http.Error(w, err.Error(), http.StatusInternalServerError)
  return
 }

 w.Write([]byte(url))
}

Let’s explain the HandleUploadfunction:

We will use r.FormFile("image") to get file from request, in other framework like mux and fiber we only get the header, so if we need to use header.Open() to get the file from header(CMIIW).

Then do some validation, after that we simply call our wrapper function s3client.UploadFile to upload to our cloud storage.

Conclusion

In this article we learn about S3 protocol, how to Integrate with Non-AWS Cloud Storage Service that supports S3 protocol, and a little bit about wrapper.

Hopefully, this article can be helpful. Don’t hesitate to give criticism and suggestions if you find any mistakes in this article.

Let’s connect on https://www.linkedin.com/in/yudaph/ and https://github.com/yudaph

Note:

artikel berbahasa indonesia dapat ditemukan di https://medium.com/@yudaph/panduan-untuk-mengintegrasikan-dengan-layanan-cloud-storage-non-aws-yang-mendukung-protokol-s3-20624638911a