Hello! Today we will deploy a serverless infrastructure based on AWS Lambda for uploading images (and any other files) with private storage in an AWS S3 bucket. We will be using terraform scripts that are uploaded and available at my kompotkot/hatchery GitHub repository.
This approach has the following advantages:
As an example, we’ll be using an abstract app for journal entries with an API.
We can upload an image in each entry, and the structure is similar to a file directory:
- journal_1
- entry_1
- image_1
- entry_2
- image_1
- image_n
- entry_n
- journal_n
Our hypothetical API has an endpoint for receiving an entry in a journal:
curl \
--request GET \
--url 'https://api.example.com/journals/{journal_id}/entries/{entries_id}'
--header 'Authorization: {token_id}'
If in response to this endpoint status_code
is equal to 200, it means the user is authorized and has access to the journal. Accordingly, we will let them store images for this entry.
To avoid adding an extra table to the database, which we would need for storing which image belongs to which entry, we will use resources
from Bugout.dev. This approach is used to simplify our infrastructure, but, if required, this step can be substituted for creating a new table in your database and writing an API for creating, modifying, and deleting data about the stored images.
Bugout.dev is open source and you can review the API documentation at the GitHub repository.
We will need an account and a team called myapp
(you can use any name in relation to your project) at the Bugout.dev Teams page, you should save this team’s ID for the next step (in our case it’s e6006d97-0551-4ec9-aabd-da51ee437909
):
Next, let’s create a Bugout.dev Application for our team myapp through a curl request (the token can be generated at the Bugout.dev Tokens page) and save it in the BUGOUT_ACCESS_TOKEN
variable:
curl \
--request POST \
--url 'https://auth.bugout.dev/applications' \
--header "Authorization: Bearer $BUGOUT_ACCESS_TOKEN" \
--form 'group_id=e6006d97-0551-4ec9-aabd-da51ee437909' \
--form 'name=myapp-images' \
--form 'description=Image uploader for myapp notes' \
| jq .
In response we will get confirmation of a successfully created app:
{
"id": "f0a1672d-4659-49f6-bc51-8a0aad17e979",
"group_id": "e6006d97-0551-4ec9-aabd-da51ee437909",
"name": "myapp-images",
"description": "Image uploader for myapp notes"
}
The ID f0a1672d-4659-49f6-bc51-8a0aad17e979
will be used for storing resources
, where every resource is the uploaded image’s metadata. The structure is set in any form depending on the required keys, in our case, it will look as follows:
{
"id": "a6423cd1-317b-4f71-a756-dc92eead185c",
"application_id": "f0a1672d-4659-49f6-bc51-8a0aad17e979",
"resource_data": {
"id": "d573fab2-beb1-4915-91ce-c356236768a4",
"name": "random-image-name",
"entry_id": "51113e7d-39eb-4f68-bf99-54de5892314b",
"extension": "png",
"created_at": "2021-09-19 15:15:00.437163",
"journal_id": "2821951d-70a4-419b-a968-14e056b49b71"
},
"created_at": "2021-09-19T15:15:00.957809+00:00",
"updated_at": "2021-09-19T15:15:00.957809+00:00"
}
As a result, we have a remote database of sorts, where every time we upload an image to an S3 bucket, we’ll be writing which journal(journal_id
) and which entry(entry_id
) the image was added to under which ID, name, and extension.
AWS will store images in an S3 bucket and function as a server on Lambda for image manipulation. We will need an AWS account and a configured IAM user for terraform. It is an account with Programmatic access
to all resources without having access to the web console:
To get the access keys, add these variables to your environment:
export AWS_ACCESS_KEY_ID=<your_aws_terraform_account_access_key>
export AWS_SECRET_ACCESS_KEY=<your_aws_terraform_account_secret_key>
Let’s also deploy a VPC with the subnets:
variables.tf
file and launch the script:
terraform apply
From the output, add to your environment variables values for AWS_HATCHERY_VPC_ID
, AWS_HATCHERY_SUBNET_PUBLIC_A_ID
и AWS_HATCHERY_SUBNET_PUBL IC_B_ID
.
In our project, we’ll be using a simple AWS Lambda function. In my experience, I’ve noticed that as the packet with code surpasses 10MB, the upload speed to AWS drops dramatically. Even if we upload it to the S3 bucket in advance and then make a Lambda from it, AWS can lag for a long time. Therefore, if you are using third-party libraries it can make sense to use Lambda layers, whereas if you aren’t planning to use any libraries with lightweight code on CloudFront, consider looking into lambda@edge.
The full code can be found in the lambda_function.py file in the repository. In my opinion, it’s more effective to work on nodejs, but to facilitate in-depth file processing we’ll use python. The code consists of main blocks:
MY_APP_JOURNALS_URL = "https://api.example.com" # API эндпоинт для доступа к нашему приложению с заметками
BUGOUT_AUTH_URL = "https://auth.bugout.dev" # Bugout.dev эндпоинт для записи ресурсов (метаданных картинок)
FILES_S3_BUCKET_NAME = "hatchery-files" # Название S3 bucket, где мы будем хранить картинки
FILES_S3_BUCKET_PREFIX = "dev" # Префикс S3 bucket, где мы будем хранить картинки
BUGOUT_APPLICATION_ID = os.environ.get("BUGOUT_FILES_APPLICATION_ID") # Bugout.dev application ID созданный ранее
Let’s expand the default exception to proxy the response from Bugout.dev Resources. E.g., if the image does not exist, when we request the resource, we’ll receive error 404, which we’ll in turn return to the client as a reply to the request for the missing image.
class BugoutResponseException(Exception):
def __init__(self, message, status_code, detail=None) -> None:
super().__init__(message)
self.status_code = status_code
if detail is not None:
self.detail = detail
To save an image in an S3 bucket we’ll use the cgi
standard library that’ll let us parse the request’s body that was sent in multipart/<image_type>
format. We’ll save images under the path {journal_id}/entries/{entry_id}/images/{image_id}
without specifying the file’s name and extension.
def put_image_to_bucket(
journal_id: str,
entry_id: str,
image_id: UUID,
content_type: str,
content_length: int,
decoded_body: bytes,
) -> None:
_, c_data = parse_header(content_type)
c_data["boundary"] = bytes(c_data["boundary"], "utf-8")
c_data["CONTENT-LENGTH"] = content_length
form_data = parse_multipart(BytesIO(decoded_body), c_data)
for image_str in form_data["file"]:
image_path = f"{FILES_S3_BUCKET_PREFIX}/{journal_id}/entries/{entry_id}/images/{str(image_id)}"
s3.put_object(
Body=image_str, Bucket=FILES_S3_BUCKET_NAME, Key=image_path
)
When we extract an image from the S3 bucket we’ll need to encode it into base64 for correct transmission.
def get_image_from_bucket(journal_id: str, entry_id: str, image_id: str) -> bytes:
image_path = f"{FILES_S3_BUCKET_PREFIX}/{journal_id}/entries/{entry_id}/images/{image_id}"
response = s3.get_object(Bucket=FILES_S3_BUCKET_NAME, Key=image_path)
image = response["Body"].read()
encoded_image = base64.b64encode(image)
return encoded_image
The lambda_handler(event,context)
function’s implementation is available at this GitHub link, to sum up:
journal_id
and entry_id
https://api.example.com/journals/{journal_id}/entries/{entry_id}
GET
, POST
or DELETE
we read, upload or delete an image from the journal’s entry
Next, we’ll need to package requests
into the Lambda library. Luckily, boto3
work with AWS functionality is ready out of the box. Let’s create an empty python environment, install the library and package the contents of site-packages
:
python3 -m venv .venv
source .venv/bin/activate
pip install requests
cd .venv/lib/python3.8/site-packages
zip -r9 "lambda_function.zip" .
Place the created archive lambda_function.zip
into the files_distributor/bucket/modules/s3_bucket/files
directory and add the Lambda function itself:
zip -g lambda_function.zip -r lambda_function.py
Our server is ready, now we can upload code to AWS and deploy Lambda server, to do so use the script in files_distributor/bucket
:
terraform apply
We’re left with:
hatchery-sources
that stores the Lambda function codehatchery-files
that we’ll store our images into with the prefix dev
The IAM role rules are in files_distributor/bucket/modules/iam/files/iam_role_lambda_inline_policy.j son
. The other file iam_role_lambda_policy.json
is needed for the Lambda to function correctly.
To debug Lambda you can just print
the required values or use the standard logging
module for python. The output for every Lambda function call is available at AWS CloudWatch:
After creating the function, add a variable BUGOUT_FILES_APPLICATION_ID
from our code to the Lambda environment, which you can do in the tab Configuration/Environment variables
.
As the last step, save the AWS Lambda arn into the variable AWS_HATCHERY_LAMBDA_ARN
.
The only step left now is to create an AWS Security Group where we’ll set a port the AWS Load Balancer will listen to for subsequent data transmit into the Lambda function (in our case it’s 80 and 443).
terraform apply \
-var hatchery_vpc_id=$AWS_HATCHERY_VPC_ID \
-var hatchery_sbn_public_a_id=$AWS_HATCHERY_SUBNET_PUBLIC_A_ID \
-var hatchery_sbn_public_b_id=$AWS_HATCHERY_SUBNET_PUBLIC_B_ID \
-var hatchery_lambda_arn=$AWS_HATCHERY_LAMBDA_ARN
Congratulations, our AWS Lambda function is open to the world and ready to upload and return images for our journal entries app!
This article was also published on Medium