Managing Django Media & Static Files on Heroku with Bucketeer

This article will walk through how we correctly persist static & media files for a Django application hosted on Heroku. As a bonus, it will also explain how we can satisfy the additional constraint of specifying private versus public media files based on model definitions. Before I begin, this post extends from that was written a while back. I frequent it often when setting up my projects, and have built some extra functionality on top of it over the years. I decided to create a more focused post that references Heroku & Bucketeer with these extra features after . Michael Herman’s TestDriven.io article helping an individual on StackOverflow So without further ado, let's first dive into what static & media files are and how Heroku dynos manage their filesystem? What are Media & Static Files If you are working with a project, then you inevitably have all of your Python application code written around a bunch of files. These are the code paths of your application, and the end-user - hopefully - never actually sees these files or their contents. Django .py Outside of these business-logic files, it is common to serve users directly from your server's file system. For these , Django doesn't need to run any code for them; the framework looks up the file and returns the contents for the requesting user to view. static files Some examples of static files include: Non-templated HTML CSS & JavaScript files to make your page look nice User profile pictures Generated PDFs in Django are a particular variant of static files. Media files are read from the server's file system as well. Unlike static files, though, they are usually generated files uploaded by users or generated by your application and are associated with a model's or . In the examples above, user profile pictures and generated PDFs are typical examples of media files. Media files FileField ImageField Django with Media & Static Files When a new media file is uploaded to a Django web application, the framework looks at the settings configuration to determine how to store that file. , it uses the class, which is what most projects start off as having configured. This implementation looks at the configuration that is defined in the file and copies the uploaded file contents to under that given . DEFAULT_FILE_STORAGE By default django.core.files.storage.FileSystemStorage MEDIA_ROOT settings.py a deterministically-created file path MEDIA_ROOT For example, if the is set as , all uploaded files will be copied and written to a location under . MEDIA_ROOT /var/www/media /var/www/media/ Heroku with Media & Static Files Storing these static files on your server's disk file system is okay until you start to work with a containerization platform such as . To explain why this is the case, it helps to take a step back. Heroku When downloading files on your personal computer, it's okay that these get written to the file system - usually under or somewhere similar. This download is because you your computer's file system to persist across restarts and shutdowns; if you download a file and restart your computer, that downloaded file should still be there once the laptop is finished restarting. ~/Downloads expect uses to execute customer workloads. One fact of this environment is that the associated file systems do not persist across restarts and reschedules. are ephemeral, and they can be destroyed, restarted, and moved without any warning, which . This situation means that any uploaded files referenced by ImageField's are just deleted without a trace every time the dyno is restarted, moved, or scaled. Heroku containerization Heroku dynos replaces the associated filesystem FileField's and Complete Example Codebase I will be stepping through the process of configuring the Django application for Heroku & S3-compatible storage, but feel free to reference the repository below for the complete code to browse through. https://github.com/dstarner/django-heroku-static-file-example Bootstrapping Django on Heroku This tutorial aims to help you retrofit an existing Django project with S3-compatible storage, but I'll quickly go through the steps I used to set up the example Django application. It may help those new to Django & Heroku or those who encounter bugs following the rest of the setup process. You can view the tagged project before the storage change . at commit 299bbe2 Bootstrapped a Django project example Uses for dependency management poetry All of the Django code is under the package, and the file is in the root. I've always found this structure cleaner than the defined in the project root. example manage.py Django apps Configured the project for Heroku package to automatically configure , , and more. This reduces the headache of deploying Django on Heroku considerably django-heroku ALLOWED_HOSTS DATABASE_URL A that runs a process for managing the WSGI application Procfile gunicorn An is defined with some fundamental configuration values and resources defined for the project to work app.json A process definition in the and an associated script that runs staticfile collection and database migrations release Procfile scripts/release.sh Introducing Heroku's Bucketeer Add-On Before we can start managing static and media files, the Django application needs a persistent place to store the files. Again, we can look to for s3-compatible storage. Ours of choice will be one called . Heroku's extensive list of Add-Ons Bucketeer Heroku's provides an bucket to upload and download files for our application. The Django application will use this configured bucket to store files uploaded by the server and download them from the S3 when a user requests the files. Bucketeer add-on AWS S3 storage If you'd like to learn more about AWS S3, the widely-popular data storage solution that Bucketeer is built upon, you can read the . S3 user documentation - is $5 per month. If you plan on spinning up the one-click example posted above, it should only cost a few cents if you when you are done using it. It is worth mentioning that the base plan for Bucketeer - Hobbyist proactively destroy the application Including the Bucketeer Add-On To include the in our application, we can configure it through the Heroku CLI, web dashboard, or via the project's file. We will use the third method of including the add-on in an file. Bucketeer add-on app.json app.json If the project does not have one already, we can create the basic structure listed below, with the critical part being the addition of the configuration. This array defines the resource that our application will use, and Heroku will install the add-on into our application if it does not already exist. We also include the keyword, which will preface the associated configuration variables with the term . This prefacing is helpful to keep the generated configuration value names deterministic because, by default, Heroku will generate the prefix as a random color. "add-ons" "bucketeer:hobbyist" " as" BUCKETEER { // ... rest above "addons": [ // ...other addons... { "plan": "bucketeer:hobbyist", "as": "BUCKETEER" } ] } With the required resources being defined, we can start integrating with our storage add-on. Implementing Our Storage Solution The package is a collection of custom, reuseable storage backends for Django. It aids immensely in saving static and media files to different cloud & storage provider options. , which our Bucketeer add-on is built on. We will leverage the S3 backend to handle different file types. django-storages One of the supported storage providers is S3 django-storages Installing django-storages Begin by installing the package and the related package used to interface with AWS's S3. We will also lock our dependencies to ensure and our Heroku deployment continue to work as expected. django-storages boto3 poetry poetry add django-storages boto3 && poetry lock Then, just like most Django-related packages, will need to be added to the project's in the projects file. This will allow Django to load the appropriate code flows as the application starts up. django-storages INSTALLED_APPS settings.py # example/config/settings.py INSTALLED_APPS = [ # ... django.X.Y apps above 'storages', # ... custom project apps below ] Implementing Static, Public & Private Storage Backends We will return to the file later to configure the usage of , but before that can be done, we will implement three custom : settings.py django-storages storage backends A storage backend for static files - CSS, Javascript, and publicly accessible images - that will be stored in version control - aka - and shipped with the application git A storage backend for dynamic media files that are not stored in version control, such as uploaded files and attachments public A storage backend for dynamic media files that are not stored in the version control that require extra access to be viewed, such as per-user reports and potentially profile images. Files managed by this backend require an access key and will block access to those without a valid key. private We can extend from 's storage backend to create these. The following code can be directly "copy and paste "'d into your project. The different attributes read in the module will be written shortly, so . django-storages S3Boto3Storage settings do not expect this code to work if you import it right now # FILE: example/utils/storage_backends.py from django.conf import settings from storages.backends.s3boto3 import S3Boto3Storage class StaticStorage(S3Boto3Storage): """Used to manage static files for the web server""" location = settings.STATIC_LOCATION default_acl = settings.STATIC_DEFAULT_ACL class PublicMediaStorage(S3Boto3Storage): """Used to store & serve dynamic media files with no access expiration""" location = settings.PUBLIC_MEDIA_LOCATION default_acl = settings.PUBLIC_MEDIA_DEFAULT_ACL file_overwrite = False class PrivateMediaStorage(S3Boto3Storage): """ Used to store & serve dynamic media files using access keys and short-lived expirations to ensure more privacy control """ location = settings.PRIVATE_MEDIA_LOCATION default_acl = settings.PRIVATE_MEDIA_DEFAULT_ACL file_overwrite = False custom_domain = False The attributes listed in each storage backend class perform the following: : This dictates the parent directory used in the S3 bucket for associated files. This is concatenated with the generated path provided by a or 's method. location FileField ImageField upload_to : This dictates the access policy required for reading the files. This dictates the storage backend's access control through values of , , and . and the parent class with translate these into object policies. default_acl None public-read private django-storages S3Boto3Storage : In most cases, it's better not to overwrite existing files if we update a specific path. With this set to , a unique suffix will be appended to the path to prevent naming collisions. file_overwrite False : Disabled here, but you can enable it if you want to use to serve from it. custom_domain AWS's CloudFront and django-storage Configure Settings to Use the Storage Backends With our storage backends defined, we can configure them to be used in different situations via the file. However, it is challenging to use S3 and these different cloud storage backends while in development, and I've always been a proponent of keeping all resources and files "local" to the development machine, so we will create a logic path that will: settings.py Use the local filesystem to store static and media files for convenience. The Django server will be responsible for serving these files directly. Use the custom S3 storage backends when an environment variable is enabled. We will use the variable to control this, enabling it in our Heroku configuration variables. S3_ENABLED First, we will assume that you have a relatively vanilla file concerning the static- & media-related variables. For reference, a new project should have a block that looks similar to the following: settings.py # Static files (CSS, JavaScript, Images) # https://docs.djangoproject.com/en/4.0/howto/static-files/ STATIC_URL = 'static/' STATIC_ROOT = BASE_DIR / 'collected-static' We will design a slightly advanced control flow that will seamlessly handle the two cases defined above. In addition, it will provide enough control to override each part of the configuration as needed. Since there are already default values for the static file usage, we can add default values for media file usage. These will be used when serving files locally from the server while in development mode. STATIC_URL = '/static/' STATIC_ROOT = BASE_DIR / 'collected-static' MEDIA_URL = '/media/' MEDIA_ROOT = BASE_DIR / 'collected-media' To begin the process of including S3, let's create the controls to manage if we should serve static & media files from the local server or through the S3 storage backend. We will create three variables : controls whether media & static files should use S3 storage by default S3_ENABLED : controls whether media files should use S3 storage. Defaults to the negated value LOCAL_SERVE_MEDIA_FILES S3_ENABLED : controls whether static files should use S3 storage. Defaults to the negated value LOCAL_SERVE_STATIC_FILES S3_ENABLED from decouple import config # import explained below # ...STATIC and MEDIA settings here... # The following configs determine if files get served from the server or an S3 storage S3_ENABLED = config('S3_ENABLED', cast=bool, default=False) LOCAL_SERVE_MEDIA_FILES = config('LOCAL_SERVE_MEDIA_FILES', cast=bool, default=not S3_ENABLED) LOCAL_SERVE_STATIC_FILES = config('LOCAL_SERVE_STATIC_FILES', cast=bool, default=not S3_ENABLED) if (not LOCAL_SERVE_MEDIA_FILES or not LOCAL_SERVE_STATIC_FILES) and not S3_ENABLED: raise ValueError('S3_ENABLED must be true if either media or static files are not served locally') In the example above, we are using the package to make it easier to read and cast environment variables to Python variables. I highly recommend this package when working with configurations. We also include a value check to ensure consistency across these three variables. If all three variables are defined in the environment but conflict with one another, the program will throw an error. python-decouple settings.py We can now start configuring the different configuration variables required by our file storage backends based on those control variables' value(s). We begin by including some S3 configurations required whether we are serving static, media, or both types of files. if S3_ENABLED: AWS_ACCESS_KEY_ID = config('BUCKETEER_AWS_ACCESS_KEY_ID') AWS_SECRET_ACCESS_KEY = config('BUCKETEER_AWS_SECRET_ACCESS_KEY') AWS_STORAGE_BUCKET_NAME = config('BUCKETEER_BUCKET_NAME') AWS_S3_REGION_NAME = config('BUCKETEER_AWS_REGION') AWS_DEFAULT_ACL = None AWS_S3_SIGNATURE_VERSION = config('S3_SIGNATURE_VERSION', default='s3v4') AWS_S3_ENDPOINT_URL = f'https://{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com' AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'} The above defines some of the variables required by the S3 backend and sets the values to environment configurations that are provided by the Bucketeer add-on. As previously mentioned, all of the add-on environment variables are prefixed with . The environment variable is not required and does not need to be included. django-storages BUCKETEER_ S3_SIGNATURE_VERSION most likely With the S3 configuration together, we can reference the and control variables to override the default static and media file settings if they are desired to be served via S3. LOCAL_SERVE_MEDIA_FILES LOCAL_SERVE_STATIC_FILES if not LOCAL_SERVE_STATIC_FILES: STATIC_DEFAULT_ACL = 'public-read' STATIC_LOCATION = 'static' STATIC_URL = f'{AWS_S3_ENDPOINT_URL}/{STATIC_LOCATION}/' STATICFILES_STORAGE = 'example.utils.storage_backends.StaticStorage' Notice the last line where is set to the custom Backend we created. That ensures it follows the location & ACL (Access Control List) policies that we configured initially. With this configuration, all static files will be placed under in the bucket, but feel free to update if desired. STATICFILES_STORAGE /static/ STATIC_LOCATION We can configure a very similar situation for media files. if not LOCAL_SERVE_MEDIA_FILES: PUBLIC_MEDIA_DEFAULT_ACL = 'public-read' PUBLIC_MEDIA_LOCATION = 'media/public' MEDIA_URL = f'{AWS_S3_ENDPOINT_URL}/{PUBLIC_MEDIA_LOCATION}/' DEFAULT_FILE_STORAGE = 'rn_api.utils.storage_backends.PublicMediaStorage' PRIVATE_MEDIA_DEFAULT_ACL = 'private' PRIVATE_MEDIA_LOCATION = 'media/private' PRIVATE_FILE_STORAGE = 'rn_api.utils.storage_backends.PrivateMediaStorage' The big difference here is that we have configured different storage backends for media files; one for publicly accessible objects and one for objects that require an access token. When the file is requested, this token will be generated internally by so you do not have to worry about anonymous public access. two django-storages Local Development Serving Since we will have set to in our local development environment, it will serve static and media files locally through the Django server instead of from S3. We will need to configure the URL routing to handle this scenario. We can configure our file to serve the appropriate files like so: S3_ENABLED False urls.py from django.conf import settings from django.conf.urls.static import static from django.contrib import admin from django.urls import path urlpatterns = [ path('admin/', admin.site.urls), ] if settings.LOCAL_SERVE_STATIC_FILES: urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT) if settings.LOCAL_SERVE_MEDIA_FILES: urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) This will locally serve the static or media files based on the values of the and settings variables we defined. LOCAL_SERVE_STATIC_FILES LOCAL_SERVE_MEDIA_FILES Enabling S3 Storage We can enable these storages and our add-on in the file to start using these storage backends. This will effectively disable and to start serving both via S3 when deployed to Heroku. app.json LOCAL_SERVE_STATIC_FILES LOCAL_SERVE_MEDIA_FILES { // ...rest of configs... "env": { // ...rest of envs... "S3_ENABLED": { "description": "Enable to upload & serve static and media files from S3", "value": "True" }, } } Using the Private Storage By default, Django will use the class for uploading media files, meaning the contents will be publicly accessible to anyone with the link. However, a model can utilize the backend when desired, which will create short-lived access tokens that prevent the public from viewing the associated object. PublicMediaStorage PrivateMediaStorage The below is an example of using public and private media files on the same model. from django.db import models from example.utils.storage_backends import PrivateMediaStorage class Organization(models.Model): """A sample Organization model with public and private file field usage """ logo = models.ImageField(help_text='A publicly accessible company logo') expense_report = models.FileField( help_text='The private expense report requires a short-lived access token' storage=PrivateMediaStorage() # will create private files ) You can see the code for this complete example . This configuration will allow your project to scale efficiently using on using . at commit 265becc Django Heroku Bucketeer In a future post, we will discuss how to upload and set these files using vanilla Django & . Django REST Framework As always, if you find any bugs, issues, or unclear explanations, please reach out to me so I can improve the tutorial & experience for future readers. Take care, everyone! Also Published Here