Building a Seamless Image and Video Experience for the Web

At Genius, our unique approach to contextualizing real-world events and actions with academic concepts makes Maths, Science and English highly engaging for children! Our engaging explanations and fun assessments, delivered through Progressive Web Apps (PWA), Android and iOS, make heavy use of static images, animated images, and videos.

A Genius way of learning

Need of the Hour

There was a time when the users were patient enough to wait for the media to load. Image loading time of 0.5 sec or more is quite unacceptable now. A jagged experience doesn’t keep users happy, defeating the purpose of great products. Though the internet connectivity across the world is improving significantly, latency and bandwidth fluctuations are still quite common.

A typical Joe, not impressed with your service

This article will help you prepare for the worst and get really close to making your media available as fast as possible. A great fluid experience is provided by optimizing across 3 main factors:

1. User Experience during asset download.
2. Reducing download time.
3. Browser Caching for near instant loading.

There are many articles which help with one or the other factors. This article gives an almost complete summary of ideas with a solution to run smooth media-rich web apps on spotty networks. It is intended for users who understand fundamentals of web development — HTML, CSS, HTTP requests and common browser behaviour.

At Genius, we heavily use Typescript, React and Node.js. Our PWAs, internal dashboards, tools and server apps are all built with them. Naturally, the solutions are also presented in our field of expertise.

User Experience

Keep users engaged. Show them something which blends into what is about to come. Here are few ways we deploy at Genius to make our experience great.

Animated loading area

A simple rectangle, animating with 2 colours, goes a long way in providing a psychological feeling of “something is happening”. It creates an event which registers subconsciously, and tricks it into believing, “Let me wait and see what shows up.”

An animating box of subconscious trickery

A simple feature like this is quite easy to pull off using plain CSS.

First, define a set of keyframes for the background colour.

@keyframes loading {0% { background-color: #e0e0e0; }50% { background-color: #eeeeee; }100% { background-color: #e0e0e0; }}

Animate them in a CSS class.

.img-container {background-color: #ffffff;animation: loading 1.3s ease-in-out infinite;-webkit-animation: loading 1.3s ease-in-out infinite;}

Use the class in the image container which will have the <img> tag.

<div class="img-container" style="height: 500px; width: 650px;"><img src="stove.jpg"></div>

Progressive Loading

There are 2 types of JPEG — Baseline and Progressive. Progressive shows the complete, low-quality image first, and improving the precision as the data arrives. It also leads to smaller images (detail below).

There are many online tools to build progressive images. At Genius, we have built a Node.js package (written in Typescript) for parsing and packaging our content and media. It uses Lovell Fuller’s Sharp to resize and make all JPEG progressive.

Does progressive JPEG result in a better experience? The answer is not definitive. It probably leads to more engagement but research suggests that it leads to decreased user happiness because of the increased cognitive load.

At Genius, we ended up using progressive to keep the users engaged and to keep the size of the JPEG minimum (more on this below).

Poster Image for Videos

The poster image of a video is the first chance to engage a user while the video downloads. From user experience point of view, it is a bad idea to not put a poster. It helps in keeping the user engaged subconsciously.

A poster image is provided through the attribute poster on the video element <video>.

<video poster="poster.jpg" controls><source src="movie.mp4" type="video/mp4"></video>

Reducing Download Time

To improve the download time of any resource, we can optimize on 3 main factors:

1. Size of the asset: downsize the resolution of images and videos, use correct encoding format to bring down the size even more.
2. HTTP protocol: HTTP/2 leaves HTTP/1.x in dust.
3. Request queue: unfinished entries in the request queue, which are no longer needed, increase the loading time of the assets which are needed.

Downsized Assets

The best case scenario is to deliver the smallest resolution image/video with the best possible format and compression while maintaining acceptable quality. It is commonly understood to NOT deliver high-resolution, high-quality images unless you are a “photograph shop” where quality matters. There are primarily two ways of doing it:

1. Client-side HTML to let the browser decide from a set of images/videos
2. Server sends the exact resized image with resolution specified in the URL

Client-Side HTML

Using CSS media queries with HTML [<picture>](https://www.w3schools.com/tags/tag_picture.asp) and [<source>](https://www.w3schools.com/tags/tag_source.asp) tags makes it quite simple to let the browser decide which image to load. All you have to do is create multiple version of images and save on the server. Here is an example:

<picture><source media="(min-width: 451px)" srcset="picture-660x414.jpg"><source media="(max-width: 450px)" srcset="picture-330x207.jpg"><img src="picture-330x207.jpg"></picture>

Since, the <picture> and <source> tags are relatively new, we also include the <img> tag for compatibility with old browsers.

If you open the developer console, you will see different requests.

900 x 521 resolution is loaded on desktop

450 x 261 resolution is loaded on mobile devices

For video, replacing <picture> element with the <video> element will load the corresponding resolution of video. You will have to encode video in different resolution and store it on your system. At Genius, we use FFmpeg to automate all our video tasks.

Server-Side Resizing and Caching

This works by sending the expected width/height of the image in the URL. The server resizes the target file on each request or serves it from the cache, if present. It is a relatively more complex solution because (i) the expected width/height needs to be calculated on the client to create the URLs for the src attribute, and (ii) the parameters need to be understood by the server to, (iii) resize and cache the image.

Many CDN (Content Delivery Network) services like Cloudflare, Akamai, etc. come with image resizing capabilities. Even software optimized for static content delivery like Nginx support it. For Node.js, we experimented with image-resizer (based on Sharp) and found it quite fast.

WebP instead of JPG and PNG

WebP leads to 20–35% smaller images than JPG and PNG for the same quality. The only caveat being, it is supported only on Chrome, Opera and Chrome’s variants. So care needs to be taken to serve the correct file (outlined excellently here).

WebP gives huge savings over JPG and PNG

Our internal tools use Sharp to create WebP variants of all images. For instance, for picture.jpg we create an optimized JPEG, picture.jpg and a WebP, picture.jpg.webp. We have a Nginx reverse proxy which detects webp support through the Accept header of the HTTP request (looks for webp in the string) and serves the correct web image.

JPEG, PNG and SVG optimizations

We follow a simple 2 step rule to decide the image type and optimize them using Sharp.

1. If it is a real-world photo or close to it, use JPEG.
2. If there are limited number of colours and curves, use PNG/SVG (logos and icons)

**Progressive JPEG**Besides being better for user experience, encoding JPEG as progressive also results in smaller image size. This mostly true only when the source image is over 10 KB. For less than 10 KB, prefer Baseline JPEG. Check out this article for a basic experiment.

No GIFs. Use MP4.

Graphics Interchange Format (GIF) was created in the stone age of internet and was not even meant for animation. The GIF spec clearly states:

The Graphics Interchange Format is not intended as a platform for animation, even though it can be done in a limited way.

According to Google,

Delivering the same file as an MP4 video can often shave 80% or more off your file-size. Not only do GIFs often waste significant bandwidth, but they take longer to load, include fewer colours and generally offer sub-par user experiences.

At Genius, we call FFmpeg from our tools to convert GIF to MP4. We have seen 700 KB GIF becoming 170 KB MP4. Here is a command to convert animated.gif to video.mp4 (inspired by this blog).

ffmpeg -i animated.gif -movflags faststart -pix_fmt yuv420p -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" video.mp4

Use HTTP/2

HTTP/2 is a new protocol for doing HTTP request. It is a binary, fully multiplexed protocol with support for header compression. In other words, it can use 1 TCP connection to load multiple files in parallel. That makes loading assets on HTTP/2 super fast, and it consumes less bandwidth because it compresses headers. This css-tricks.com blog has a detailed performance analysis.

At Genius, we use AWS Elastic Load Balancer and CloudFront as our endpoints. Both the services use HTTP/2 by default.

Clean unfinished entries in request queue

We experienced an undesirable behaviour with our Single Page Application (SPAs) when the network becomes intermittent. The assets requested through <img> and <video> elements on previous screens remained in the network download queue even after the corresponding elements were removed from the HTML. This, in turn, delayed downloading of assets that needed to be displayed in the current view.

Clearing the src attribute on the <img> element cancels the request instantly. Here is a React component in Typescript implementing this feature.

For <video> element, remove the src attribute from source element, and call load on the mounted <video> element.

With this, the browser cancels the download requests once the elements are removed from the HTML and sets (canceled) as the status of these requests.

Canceled network requests

Browser Caching

Caching a resource correctly can help the browser avoid the network call altogether when the same resource is fetched again. There are two types of caches — one which persists the assets on the permanent storage (persistent) and ones which are memory based (non-persistent), hence application session specific.

There are 4 types of cache in advanced browsers. They are encountered by a request in the following order.

1. Memory Cache — as the name suggests, it caches assets in the memory.
2. Service Worker Cache — a persistent cache used by service workers (uses Cache API).
3. HTTP Cache — persists the response in accordance with the cache-control header set in the HTTP request.
4. HTTP/2 Cache — caches assets in memory which are pushed by HTTP/2.

A fun article, “A Tale of Four Caches” by Yoav Weiss explains how all 4 work together.

HTML Preload

HTML preload is a declarative fetch, forcing the browser to request a resource before it is used. It can be used to preload assets into the HTTP and Memory Cache that will be needed in the immediate future. For instance, when someone is reading page 1 of a blog, we can preload images of page 2.

<link rel="preload" as="image" href="sunlight-trees.jpg">

There is also prefetch declaration which works differently from preload. Here is a great article from Addy Osmani, describing prefetch and preload in depth.

There is one caveat that we didn’t realize. Unlike preload which can load assets as image, font, script, the prefetch declaration cannot load assets with a resource type. Simply put, the preload declaration in the block above will set the Accept header as if the request is being made by the <img> element. The prefetch doesn’t support as="image" attribute and will always set the Accept header to */*. This makes prefetch undesirable because the browser will anyway re-fetch the resource when it is used in the HTML.

HTTP Caching

Browser HTTP cache largely has 3 modes, controlled by the [Cache-Control](https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching/#cache-control) HTTP header.

1. No caching — Do not store the response. Achieved by setting the response HTTP header Cache-Control: no-store.
2. Re-validate every time — Store the response but validate with the server before using the response. Set the response HTTP header Cache-Control: no-cache and the [ETag](https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching/#validating_cached_responses_with_etags) header, uniquely identifying the delivered resource. If the ETag doesn’t match with the latest file on the server, the server returns the new file else returns with HTTP status 304.
3. Don’t re-validate for a specified time — Store the response and don’t bother asking the server until a specified time is expired. Achieved by setting the Cache-Control: public, max-age:<seconds>. The seconds specify the time (max value of 31536000 (1 year)) until the client is going to serve the cached response instead of asking the server for new response.

Use the 2nd mode when an asset corresponding to a URL may change in the future. For instance, if the image URL contains is just filename [https://example.com/pic.jpg](https://example.com/pic.jpg).

Use 3rd mode, when you can generate unique URLs for each resource. For instance, by having image content hash in the name [https://example.com/pic-6a8sd6d82.jpg](https://example.com/pic.jpg).

At Genius, all our image names contain hash. Hence, we always use the 3rd mode. Once we deliver the image, the client doesn’t ask the server unless the cache is deleted or a year has passed. This saves user as well as server bandwidth while delivering a seamless experience.

Service Worker Caching

Service Workers are the new rage of web development for building rich offline experiences and much more. A service worker runs a script in the background, separate from the web page. The feature we are interested in is the ability to intercept the requests coming from the web app and provide the response from the cache that, as a developer, we manage for the app.

Asset caching strategy. Taken from the offline cookbook.

Building a service worker can get quite complex. To make life easy, the good folks at Google have developed Workbox which only needs a configuration to install the service worker and manage the cache. For instance, to cache all the requests to [https://example.com/image/](https://example.com/image/,), such that we don’t cache more than 20 images, for a maximum of 1 week, all we have to do is write a config like this.

workboxSW.router.registerRoute('https://example.com/image/(.*)',workboxSW.strategies.cacheFirst({cacheName: 'images',cacheExpiration: {maxEntries: 50,maxAgeSeconds: 7 * 24 * 60 * 60,},cacheableResponse: {statuses: [0, 200]},}));

Check out this detailed example.

The End

Phew! That was a lot of information. Congratulations on reading until the end. If you find any aspect missing, please do mention in the comments below so that this document can be improved and made more useful. Thanks!