How to Optimize Your Image Storage

At the moment when one of the most important components of your product becomes quickly, reliably, and optimally displaying text and images on your website, you start to wonder: "how can I make this more efficient?". This is the problem we faced as well. This article will discuss the optimization we developed and implemented.

CDN for optimized content delivery and distribution

At the top level, we use a CDN (Content Delivery Network) - a geographically distributed network infrastructure that allows optimizing the delivery and distribution of content to end users on the Internet. In the initial stage, the user requests an image from the domain cdn1.yourhost.com, and the request goes to the CDN. If this request is already stored in the cache, it is immediately served to the client; if not, the request goes to our servers.

Server-Side cache management

The request that comes to our servers (MediaPRX) is also checked for the presence of a ready cache; if it exists, it is served immediately; if not, the request is passed to the next level (MediaResize) and the response is cached.

Image transformation and resizing with imgproxy

At the resizer, we transform the request by determining the required format and size, and send it to the locally deployed imgproxy, adding the original image URL to the request. Imgproxy, upon receiving the request, fetches the original image from the same server where the request came from.

How does image resizing work?

All images served through media.yourhost.com and cdn1.yourhost.com support dynamic resizing by specifying the necessary parameters in the request URL. We use imgproxy as the resizer, and the nginx in front of it converts the requested URL into a format understood by imgproxy.

The file must have an extension of jpg, jpeg, png, gif, or webp. The primary regex for checking the match (for images from s3) is

~^/s3/(?<a>[^/]+/(.?/)?)(a|w)?(c|b|d|s|w|h|f)(?<size>[0-9]{2,4})(_q[0-9]{2})?/(?<b>[^/]+.(jpg|jpeg|png|gif|webp))$

• When setting the size, there are 3 options:

1. If only size is present - crop the image to a square while maintaining proportions.
2. If 2 sizes are specified (/size1 x size2/ or /size1 / size2/) - fit within a rectangle while maintaining proportions.
3. Resize by the larger side (w|a)?(w|h)

• When resizing, there are several modifiers - (w|a)?(c|b|d|s|w|h|f):

The first optional modifier:

w - indicates that the requested image needs to be converted to webp. If this modifier is absent, the image will be served without format conversion.

a - on stg/prod, there is already a modifier - avif, but it is not recommended to use it, as the compression operation takes a long time online.

The second mandatory modifier:

c - fit within a square while maintaining proportions without image extension. b - fit within a square while maintaining proportions, filling the empty areas with white color, and positioning the image at the bottom. d - fit within a square while maintaining proportions, overlaying a watermark with filling over the entire image.

s - fit within a square while maintaining proportions, filling the empty areas with white color, and positioning the image at the center.

w - resize by width, maintaining proportions.

h - resize by height, maintaining proportions.

f - resize and crop according to the specified "square" dimensions.

• Quality control:

Depending on the extension, different default quality values are set.

".*" is 95, ".webp" is 80, ".avif" is 50.

After the resize modifiers, you can redefine the desired quality level (w|a)?(c|b|d|s|w|h|f){SIZE}_q({quality}).

{quality} - numbers from 01 to 99.

Caching and serving optimized images

Upon receiving a request from imgproxy, nginx checks if the image is in its local cache (this is done to avoid fetching the original image from S3/MediaOrigin multiple times when requesting different sizes of the same image). If the image is not in the cache, nginx fetches it from S3/MediaOrigin, caching the response.

The result obtained from imgproxy is then passed back through the chain to the user and cached at each level to speed up subsequent requests to the same URL.

Benefits of image storage optimization

In conclusion, the optimization process we have implemented for image display and resizing greatly enhances the performance and efficiency of our system. By leveraging a CDN, caching, and dynamic resizing, we ensure that images are served quickly, reliably, and in an optimal manner. These improvements not only lead to a better end-user experience but also result in reduced load on our servers and storage systems. Furthermore, the use of imgproxy and customizable resize options allows for a high degree of flexibility and control over image quality and dimensions.

By adopting this comprehensive approach to image optimization, you can successfully addressed the challenge of efficiently handling images alongside text information, ultimately contributing to a more streamlined and effective product.