Hello! Today we will deploy a serverless infrastructure based on AWS Lambda for uploading images (and any other files) with private storage in an AWS S3 bucket. We will be using terraform scripts that are uploaded and available at my GitHub repository. kompotkot/hatchery This approach has the following advantages: lambda is called on request and therefore allows to save on server maintenance costs if this functionality is not key for your app lambda functions have an isolated runtime environment, which is ideal for processing uploaded files. Should malicious code be uploaded, the attacker will not be able to leave the sandbox, and the sandbox session will be forcibly terminated after some time storing files in an S3 bucket is very cheap Project structure As an example, we’ll be using an abstract app for journal entries with an API. We can upload an image in each entry, and the structure is similar to a file directory: - journal_1 - entry_1 - image_1 - entry_2 - image_1 - image_n - entry_n - journal_n Our hypothetical API has an endpoint for receiving an entry in a journal: curl \ --request GET \ --url 'https://api.example.com/journals/{journal_id}/entries/{entries_id}' --header 'Authorization: {token_id}' If in response to this endpoint is equal to 200, it means the user is authorized and has access to the journal. Accordingly, we will let them store images for this entry. status_code Registering the app on Bugout.dev To avoid adding an extra table to the database, which we would need for storing which image belongs to which entry, we will use from . This approach is used to simplify our infrastructure, but, if required, this step can be substituted for creating a new table in your database and writing an API for creating, modifying, and deleting data about the stored images. resources Bugout.dev Bugout.dev is open source and you can review the API documentation at the GitHub repository. We will need an account and a team called (you can use any name in relation to your project) at the page, you should save this team’s ID for the next step (in our case it’s ): myapp Bugout.dev Teams e6006d97-0551-4ec9-aabd-da51ee437909 Next, let’s create a Bugout.dev Application for our team myapp through a curl request (the token can be generated at the page) and save it in the variable: Bugout.dev Tokens BUGOUT_ACCESS_TOKEN curl \ --request POST \ --url 'https://auth.bugout.dev/applications' \ --header "Authorization: Bearer $BUGOUT_ACCESS_TOKEN" \ --form 'group_id=e6006d97-0551-4ec9-aabd-da51ee437909' \ --form 'name=myapp-images' \ --form 'description=Image uploader for myapp notes' \ | jq . In response we will get confirmation of a successfully created app: { "id": "f0a1672d-4659-49f6-bc51-8a0aad17e979", "group_id": "e6006d97-0551-4ec9-aabd-da51ee437909", "name": "myapp-images", "description": "Image uploader for myapp notes" } The ID will be used for storing , where every resource is the uploaded image’s metadata. The structure is set in any form depending on the required keys, in our case, it will look as follows: f0a1672d-4659-49f6-bc51-8a0aad17e979 resources { "id": "a6423cd1-317b-4f71-a756-dc92eead185c", "application_id": "f0a1672d-4659-49f6-bc51-8a0aad17e979", "resource_data": { "id": "d573fab2-beb1-4915-91ce-c356236768a4", "name": "random-image-name", "entry_id": "51113e7d-39eb-4f68-bf99-54de5892314b", "extension": "png", "created_at": "2021-09-19 15:15:00.437163", "journal_id": "2821951d-70a4-419b-a968-14e056b49b71" }, "created_at": "2021-09-19T15:15:00.957809+00:00", "updated_at": "2021-09-19T15:15:00.957809+00:00" } As a result, we have a remote database of sorts, where every time we upload an image to an S3 bucket, we’ll be writing which journal( ) and which entry( ) the image was added to under which ID, name, and extension. journal_id entry_id Preparing the AWS project environment AWS will store images in an S3 bucket and function as a server on Lambda for image manipulation. We will need an AWS account and a configured IAM user for terraform. It is an account with to all resources without having access to the web console: Programmatic access To get the access keys, add these variables to your environment: export AWS_ACCESS_KEY_ID=<your_aws_terraform_account_access_key> export AWS_SECRET_ACCESS_KEY=<your_aws_terraform_account_secret_key> Let’s also deploy a VPC with the subnets: 2 private access 2 public access They will be useful for configuring the AWS Load Balancer. The code for this module can be found under . Let’s edit the variables in the file and launch the script: files_distributor/network variables.tf terraform apply From the output, add to your environment variables values for , и . AWS_HATCHERY_VPC_ID AWS_HATCHERY_SUBNET_PUBLIC_A_ID AWS_HATCHERY_SUBNET_PUBL IC_B_ID Server code In our project, we’ll be using . In my experience, I’ve noticed that as the packet with code surpasses 10MB, the upload speed to AWS drops dramatically. Even if we upload it to the S3 bucket in advance and then make a Lambda from it, AWS can lag for a long time. Therefore, if you are using third-party libraries it can make sense to use Lambda layers, whereas if you aren’t planning to use any libraries with lightweight code on CloudFront, consider looking into lambda@edge. a simple AWS Lambda function The full code can be found in the file in the repository. In my opinion, it’s more effective to work on nodejs, but to facilitate in-depth file processing we’ll use python. The code consists of main blocks: lambda_function.py MY_APP_JOURNALS_URL = "https://api.example.com" # API эндпоинт для доступа к нашему приложению с заметками BUGOUT_AUTH_URL = "https://auth.bugout.dev" # Bugout.dev эндпоинт для записи ресурсов (метаданных картинок) FILES_S3_BUCKET_NAME = "hatchery-files" # Название S3 bucket, где мы будем хранить картинки FILES_S3_BUCKET_PREFIX = "dev" # Префикс S3 bucket, где мы будем хранить картинки BUGOUT_APPLICATION_ID = os.environ.get("BUGOUT_FILES_APPLICATION_ID") # Bugout.dev application ID созданный ранее Let’s expand the default exception to proxy the response from Bugout.dev Resources. E.g., if the image does not exist, when we request the resource, we’ll receive error 404, which we’ll in turn return to the client as a reply to the request for the missing image. class BugoutResponseException(Exception): def __init__(self, message, status_code, detail=None) -> None: super().__init__(message) self.status_code = status_code if detail is not None: self.detail = detail To save an image in an S3 bucket we’ll use the standard library that’ll let us parse the request’s body that was sent in format. We’ll save images under the path without specifying the file’s name and extension. cgi multipart/<image_type> {journal_id}/entries/{entry_id}/images/{image_id} def put_image_to_bucket( journal_id: str, entry_id: str, image_id: UUID, content_type: str, content_length: int, decoded_body: bytes, ) -> None: _, c_data = parse_header(content_type) c_data["boundary"] = bytes(c_data["boundary"], "utf-8") c_data["CONTENT-LENGTH"] = content_length form_data = parse_multipart(BytesIO(decoded_body), c_data) for image_str in form_data["file"]: image_path = f"{FILES_S3_BUCKET_PREFIX}/{journal_id}/entries/{entry_id}/images/{str(image_id)}" s3.put_object( Body=image_str, Bucket=FILES_S3_BUCKET_NAME, Key=image_path ) When we extract an image from the S3 bucket we’ll need to encode it into base64 for correct transmission. def get_image_from_bucket(journal_id: str, entry_id: str, image_id: str) -> bytes: image_path = f"{FILES_S3_BUCKET_PREFIX}/{journal_id}/entries/{entry_id}/images/{image_id}" response = s3.get_object(Bucket=FILES_S3_BUCKET_NAME, Key=image_path) image = response["Body"].read() encoded_image = base64.b64encode(image) return encoded_image The function’s implementation is available at this GitHub , to sum up: lambda_handler(event,context) link Firstly, we assert that the request is formatted correctly and contains and journal_id entry_id Then we call our hypothetical app’s API https://api.example.com/journals/{journal_id}/entries/{entry_id} Depending on the request method: , or we read, upload or delete an image from the journal’s entry GET POST DELETE When we’re uploading to the S3 bucket, we check the extension and the file’s size. This can be expanded into hash verification to avoid uploading the same file, etc. Next, we’ll need to package into the Lambda library. Luckily, work with AWS functionality is ready out of the box. Let’s create an empty python environment, install the library and package the contents of : requests boto3 site-packages python3 -m venv .venv source .venv/bin/activate pip install requests cd .venv/lib/python3.8/site-packages zip -r9 "lambda_function.zip" . Place the created archive into the directory and add the Lambda function itself: lambda_function.zip files_distributor/bucket/modules/s3_bucket/files zip -g lambda_function.zip -r lambda_function.py Our server is ready, now we can upload code to AWS and deploy Lambda server, to do so use the script in : files_distributor/bucket terraform apply We’re left with: A private AWS S3 bucket that stores the Lambda function code hatchery-sources A private AWS S3 bucket that we’ll store our images into with the prefix hatchery-files dev AWS Lambda function with working server code An IAM role for the Lambda that allows writing into a specific S3 bucket and logs The IAM role rules are in . The other file is needed for the Lambda to function correctly. files_distributor/bucket/modules/iam/files/iam_role_lambda_inline_policy.j son iam_role_lambda_policy.json To debug Lambda you can just the required values or use the standard module for python. The output for every Lambda function call is available at AWS CloudWatch: print logging After creating the function, add a variable from our code to the Lambda environment, which you can do in the tab . BUGOUT_FILES_APPLICATION_ID Configuration/Environment variables As the last step, save the AWS Lambda arn into the variable . AWS_HATCHERY_LAMBDA_ARN Configuring the AWS Load Balancer and open ports The only step left now is to create an AWS Security Group where we’ll set a port the AWS Load Balancer will listen to for subsequent data transmit into the Lambda function (in our case it’s 80 and 443). terraform apply \ -var hatchery_vpc_id=$AWS_HATCHERY_VPC_ID \ -var hatchery_sbn_public_a_id=$AWS_HATCHERY_SUBNET_PUBLIC_A_ID \ -var hatchery_sbn_public_b_id=$AWS_HATCHERY_SUBNET_PUBLIC_B_ID \ -var hatchery_lambda_arn=$AWS_HATCHERY_LAMBDA_ARN Congratulations, our AWS Lambda function is open to the world and ready to upload and return images for our journal entries app! This article was also published on Medium
