Essential Guide to Image Processing with WebAssembly

In the previous part of this series, we already learned how to write Wasm modules in AssemblyScript. In this part, we will use this knowledge in a practical scenario: image manipulations with WebAssembly.

We will demonstrate a typical use-case by a simple function for converting an image to grayscale.

Albeit the calculation of the gray color is not very demanding, it clearly demonstrates the real-world usage of WebAssembly on the web: computation-intensive tasks.

You can find the full discussed source code on my GitHub.

Browser Runtime

Unlike our previous experiments with AssemblyScript, this time we will run our Wasm module in a browser. You might first recall how to use a web browser as a Wasm runtime, but the code is quite straight-forward:

WebAssembly
  .instantiateStreaming(fetch('grayscale.wasm'), {})
  .then(({ instance }) => {
    ...
  }

To make the Fetch API work you must serve the web page via HTTP(S).

The second argument of the 

instantiateStreaming

env

When running in the browser, we must import an error-callback function

abort

WebAssembly
  .instantiateStreaming(fetch('grayscale.wasm'), {
    env: {
      abort: (_msg, _file, line, column) =>
        console.error(`Error at ${line}:${column}`)
    } 
  })

In Node.js those are already provided by the AssemblyScript loader out of the box.

Canvas

To show the image and its transformation in a browser, we will use the HTML canvas element:

<canvas id="canvas" width="500" height="500"></canvas>

And its 2D context:

const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
const [width, height] = [canvas.width, canvas.height];

First, we will draw the original image on the canvas:

const img = new Image();
img.src = './my-pretty-picture.png';
img.crossOrigin = 'anonymous';
img.onload = () =>
  ctx.drawImage(img, 0, 0, width, height);

Then, we will get the image data that we will work with:

const imageData = ctx.getImageData(0, 0, width, height);
const data = imageData.data;

The image data is represented as a 

Uint8ClampedArray

Memory

Because we will work extensively with memory, we should recall the fundamentals.

We can either create a memory instance from JavaScript and import it into the Wasm module or let the module initialize memory and export its instance into JavaScript.

One benefit of the former approach is the possibility of initializing memory to the needed size.

When exporting memory from Wasm, the initial size is one page (64 KB). To increase its size, we have to call 

memory.grow()

Our image might be bigger than 64 KB, so we had better create a big-enough memory instance:

const arraySize = (width * height * 4) >>> 0;
const nPages = ((arraySize + 0xffff) & ~0xffff) >>> 16;
const memory = new WebAssembly.Memory({ initial: nPages });

Here comes the catch. To be able to import memory into a Wasm module, we have to compile our AssemblyScript code with the 

--importMemory

$ npm run asbuild:optimized -- --importMemory

This will generate the following line in the compiled Wasm module:

(import "env" "memory" (memory $0 1))

Now, we can import our memory object from JavaScript into Wasm:

WebAssembly
  .instantiateStreaming(fetch('grayscale.wasm'), {
    env:{ memory }
  })

The next step is to initialize the memory with the image data:

const bytes = new Uint8ClampedArray(memory.buffer);
       
for (let i = 0; i < data.length; i++)
  bytes[i] = data[i];

As you might have noticed, we use the same typed array —

Uint8ClampedArray

When the memory bytes are filled, we can execute our Wasm function:

instance.exports
  .convertToGrayscale(width, height);

And store the result from memory back into the image:

for (let i = 0; i < bytes.length; i++)
  data[i] = bytes[i];
 
ctx.putImageData(imageData, 0, 0);

Image Manipulation

Now, when we have all we need to run an image-manipulation function in a browser, we shall actually write one. As previously mentioned, we will create a function that converts an image to grayscale:

export function convertToGrayscale(
    width: i32, height: i32): void {
 
  const len = width * height * 4;
 
  for (let i = 0; i < len; i += 4 /*rgba*/) {
    const r = load<u8>(i);
    const g = load<u8>(i + 1);
    const b = load<u8>(i + 2);
     
    const gray = u8(
      r * 0.2126 + g * 0.7152 + b * 0.0722);
 
    store<u8>(i,     gray);
    store<u8>(i + 1, gray);
    store<u8>(i + 2, gray);
  }
}

As the image data contains linear RGBA quadruples of 8-bit signed integers (clamped to the range of values from 0 to 255), we iterate the array of data in 4-incremental steps.

We load the RGB values from memory, compute the gray color, and store the color values back into the memory.

Demo Time

That’s it! Now we have all the know-how needed to develop awesome image processing Wasm functions in AssemblyScript.

Here are some more examples:

You can see demos live on my blog.

The source code is on my GitHub.

Previously published at https://blog.ttulka.com/learning-webassembly-10-image-processing-in-assemblyscript