Creating WebAssembly-powered library for modern web

This article tells about my first practical experience with WebAssembly and few useful technics which I’ve obtained while creating vmsg library . I’ve got some free time recently so I decided to try the new WebAssembly standard and implement simple but useful library with it. As stated in one of the main goals was to create format that can be parsed fast and has compact code representation. So I was really curious to leverage that main advantage. I’ve had experience with asm.js in past, making ports of pretty complicated C software such as FFmpeg and video/audio encoders. The build sizes were terrible — about 15 megabytes of minified JavaScript for ffmpeg CLI with few basic filters and encoders. Given that parsing of code is computationally expensive operation, especially on mobiles, it didn’t look so practical, more like targeted for proof of concepts little demos: , . WebAssembly docs webm.js deviceframe.es On the other hand, WebAssembly binary can be parsed and compiled making it a perfect build target for libraries written in C which you need to use inside a web page. as fast as it comes over the network The idea of library came pretty quickly: I’m spending quite a lot of time on web forums, discussing various things via text messages and images. Recently with the raise of HTML5 video and WebM/VPx formats, it’s became quite common to attach small videos to the posts, increasing the possibilities of self-expression even more. What about voice? What if you can literally tell your message and send it as part of the post? Sounds great, let’s try it! Deciding on high-level architecture So first we need to grab audio samples from microphone, then encode it, then return file to the library user. Looks pretty simple. In 2018 is widely supported, no real headaches here. with conjuction of are capable of the first step, WebAssembly module would be responsible for the second. Because callback of node is being executed in the main thread and also to keep interface of web page responsive, WebAssembly module will be instantiated in the Web Worker, communicating with the main thread via messages. Web Audio API getUserMedia ScriptProcessorNode onaudioprocess ScriptProcessor Side note: has been deprecated and is soon to be replaced by but it’s only implemented in Chrome 64+ behind a flag for the moment and for compatibility we have to use old API in near future anyway. Moreover, since we process samples in a worker, don’t output them to speakers and can use large buffer, Worklet isn’t required in our particular case. should work just fine, all it needs to do is send samples to Web Worker which is very fast and lightweight operation. ScriptProcessNode Audio Workers ScriptProcessNode We also going to create simple interface which would ask for permission to use mic and display recording form with start/stop/close buttons. Below you can see schematical overview of components of the library: Library architecture Choosing the format Now we need to decide which audio format we will use to encode received samples to. Prerequisites: it should work in all browsers that support WebAssembly, it should give sane compression, it should be widespread across all platforms. My initial desire was to grab Opus because it’s the best that you can use for speech compression. Unfortunately it’s not supported by element in Safari and Edge. Of course there’re various workarounds for this. For example in Edge you can manually fetch Opus file and play it via . It’s also seems to be possible to package for the full support. In Safari you can use Opus decoders ported to JavaScript such as . MediaSource API install Web Media Extension ogv.js Even though it’s possible it’s too impractical for real use in my opinion. It’s ok to say: “If you want to support for voice messages add this library to your project”. But now you dictate which player to use to listen for resulting audios or require some non-trivial code to handle playback. I don’t like it so I had to abandon Opus. Maybe in few years from now the choice would be much easier. Side note: Chrome and Firefox support and can encode data with Opus codec right out of the box. Not in Safari and Edge though and I really want to make my library work in all 4 of them so no luck here again. MediaStream Recording API MediaStream Next one, there is WAV/PCM format available in all browsers. Creating WAV file from raw samples is a dead-simple process, there is already for that. It has one drawback though: there is no compression . So whether you sing some beatiful song in your mic or keep silent, 30 seconds of record (48KHz/mono) will always weight exactly 2.7 megabytes. This is way too wasteful. library little at all What about MP3? It’s supported everywhere, has decent compression and great . Historially FOSS projects stepped aside from using it because of software patents but all of them . So seems like we have a winner. LAME encoder have expired in last year There are also AAC and Vorbis but neither of them fit. Former which our WebAssembly module will effectively be. (Also it’s questionable whether free implementations as good as proprietary.) Latter doesn’t suitable for the speech compression. forbids distribution of codec implementation in binary form Compiling So we need to grab LAME encoder, compile to WebAssembly module and make possible to use it from JavaScript. There are tons of asm.js ports of LAME and maybe even WASM ports but I decided to make a new one from scratch in order to focus on build size optimizations. At first I with git-svn because the was deprecated for some reason and don’t contain the latest 3.100 release, which might have some helpful bugfixes. mirrored SVN repo previous semi-official mirror For compiling we use de-facto standard , nothing new here. It’s been actively developed for many years and designed to port C/C++ libraries to the web, exactly what we need. I won’t go into details, you can read more about Emscripten at . Emscripten toolchain official site Emscripten’s asm.js compiler is powered by LLVM backend called fastcomp. For WebAssembly you have two options: compile to asm.js first and translate to WASM with . Or use LLVM’s in-tree WebAssembly backend which is capable of producing WebAssembly binaries by itself (almost, you still need to use Binaryen for the final step). I chose the second because it seems to be the preferred one in near future. Also Emscripten’s got support for recently which is again going to be preferred soon. Binaryen standard LLVM linker Side note: I’m not going to describe the process of compiling LLVM with WASM backend. It’s generally recommended to use the latest SVN version. You can check out for the starting point. It’s also possible to compile WASM backend with emsdk by providing flag but it uses pretty old LLVM (the base for fastcomp patches) so the resulting module might be bigger/slower than with SVN LLVM. It also doesn’t build LLD. this gist --enable-wasm Let’s create stub of our library. I will use Linux shell commands, YMMV. $ cd ~$ git init vmsg && cd vmsg$ npm init -y Now we need sources of LAME encoder, git submodules are really handy for that: $ git submodule add https://github.com/Kagami/lame-svn.git$ cd lame-svn && git checkout RELEASE__3_100 && cd .. So far so good. Let’s compile (shared LAME library) so that we can later call its functions from WebAssembly module. I use GNU Makefile even though modern builders like webpack and parcel are getting support for WASM, because it’s not mature yet and I want to experiment with compiler flags and other optimizations. And builders will only stand in the way here. libmp3lame.so Create with the following text (make sure to use tabs for indentation): Makefile export EMCC_WASM_BACKEND = 1export EMCC_EXPERIMENTAL_USE_LLD = 1 lame-svn/lame/dist/lib/libmp3lame.so:cd lame-svn/lame && \git reset --hard && \patch -p2 < ../../lame-svn.patch && \emconfigure ./configure \CFLAGS="-DNDEBUG -Oz" \--prefix="$$(pwd)/dist" \--host=x86-none-linux \--disable-static \--disable-gtktest \--disable-analyzer-hooks \--disable-decoder \--disable-frontend \&& \emmake make -j8 && \emmake make install I told Emscripten to use WASM backend and LLD, enabled advanced shrinking size optimizations, disabled asserts and disabled some extra stuff in LAME we don’t need. The fixes strtol check in configure script and disables default LAME’s reporters to shrink the build size (otherwise Emscripten will include implementation of function and other stuff). patch printf $ source /path/to/emsdk/emsdk_env.sh$ make This activates Emscripten environment and creates LAME library at directory. lame-svn/lame/dist/lib/ Now we need to use LAME library functions in WebAssembly module and export MP3 creation routines so they can be called from JavaScript. I won’t go much into details here, you can check out the resulting . It has 4 methods: , , and which are self-descriptive and call one or few corresponding LAME functions inside. To communicate back and forth with JavaScript simple structure is being used which stores the current state of encoding. It’s also possible to encode multiple files in parallel because we don’t have global variables. vmsg.c init encode flush free vmsg Let’s finally compile our WebAssembly module. Add this to : Makefile vmsg.wasm: lame-svn/lame/dist/lib/libmp3lame.so vmsg.cemcc $^ \-DNDEBUG -Oz --llvm-lto 3 \-Ilame-svn/lame/dist/include \-s WASM=1 \-s "EXPORTED_FUNCTIONS=['_vmsg_init','_vmsg_encode','_vmsg_flush','_vmsg_free']" \-o _vmsg.jscp _vmsg.wasm $@ Nothing really puzzled here, we ask Emscripten to compile our C wrapper, combine it with LAME shared library and export functions we would need on JavaScript side. Type and that’s it. We’ve ported fully-functional MP3 encoder to web which weights only about 70kb gzipped: make vmsg.wasm $ wc -c < vmsg.wasm152799$ gzip -6 -c vmsg.wasm | wc -c74152 Note that 70kb of gzipped WebAssembly is not even similar to 70kb of gzipped JavaScript in terms of parsing complexity. It’s like a small image: WASM module will be compiled and ready to use roughly just after it has been downloaded. It might be possible to shrink size of the module even more but for now I’m satisfied with the numbers. Runtime for loading and calling WebAssembly module is pretty simple. The tricky part here is to provide functions that module needs to correctly operate on a web platform. WebAssembly spec doesn’t define any standard library like in C which is in charge of memory allocations, math operations, input/output API and so on. And LAME won’t work without some of them. Emscripten uses patched lightweight C standard library on WASM side (so the libraries being ported don’t need to be rewritten) and generates wrapper module in JS which would work in tandem with musl and make calls to e.g. in-browser object so the date/time functions of musl can work properly. Unfortunately it comes with a cost: even minified by Closure library it would weight about 10kb, so I was curious if I can do a better job than Emscripten in my particular case. JavaScript API musl Date Let’s first look what module actually needs with from Binaryen toolchain: wasm-dis $ wasm-dis vmsg.wasm | grep '(import'(import "env" "memory" (memory $0 3))(import "env" "pow" (func $import$1 (param f64 f64) (result f64)))(import "env" "exit" (func $import$2 (param i32)))(import "env" "powf" (func $import$3 (param f32 f32) (result f32)))(import "env" "exp" (func $import$4 (param f64) (result f64)))(import "env" "sqrtf" (func $import$5 (param f32) (result f32)))(import "env" "cos" (func $import$6 (param f64) (result f64)))(import "env" "log" (func $import$7 (param f64) (result f64)))(import "env" "sin" (func $import$8 (param f64) (result f64)))(import "env" "sbrk" (func $import$9 (param i32) (result i32))) Only 10 functions and most of them can be mapped directly to object! There is also which is called when module decided to, well, exit, which is virtual memory and created with and called by musl when it needs to allocate more memory. you can see my implementation of all that functions which takes only 30 lines and works perfectly fine. Math exit memory new WebAssembly.Memory sbrk Here Polyfill It’s a great thing that WebAssembly is supported by all 4 major browsers (Chrome/Firefox/Safari/Edge) but not all users on the web have access to latest versions of browsers. So it’s reasonable to make your application support as many versions as you can if it doesn’t hurt readability/performance/maintenance/etc much. For example I intentionally use XHR on browsers without because it makes code only 5 lines longer and allows to support browsers without Fetch API e.g. Edge 12–14. WebAssembly.instantiateStreaming Right now the recommended way of “polyfilling” WebAssembly is to make separate asm.js build of the same code. It works pretty well with Emscripten’s runtime because it abstracts out differences between these two techs and provides single interface to interact with compiled code. Because we use our own runtime and because it feels more natural to use WebAssembly API if available and emulate it if not, I decided to put foot at “true polyfill” path. Module Quick search for “WebAssembly polyfill” returned several projects, most promising of them was . It works by emulating WebAssembly browser APIs such as and , parsing binary module and generating asm.js-alike code on the fly. Exactly what I wanted! Unfortunately it was no longer maintained so I had to fork it, slightly refactor, fix obvious issues and tests and publish to NPM. The most horrible bug was in code generation of instruction but eventually I kinda fixed it. is my fork, I think it might be useful for other projects too. by Ryan Kelly WebAssembly.Memory WebAssembly.Table i64.store Here I’ve also found but it was too huge (2.5mb vs 95kb in case of wasm-polyfill) and not complete: it doesn’t emulate the WebAssembly browser API. Finally the looks abandoned so wasm-polyfill is probably the best option that we have now. It’s not ideal though: generated code is not as effecient as it could be, there’re lot of extra bound checkings created to be fully semantically correct. See last section for the possible improvements in that area. polyfill in Binaryen repo official polyfill prototype Usage of polyfill is straight-forward: include minified build with tag or call in case of Worker. A piece of cake. wasm-polyfill.js importScripts Putting everything together We already have WebAssembly module, can load it from JavaScript and call its functions, what else? We need to spawn it in Worker, define communication protocol and feed it some real data from mic. Also we need to build some UI so users won’t have to reimplement it all the time. At first I leant towards to React because it’s so extremely popular and powerful library for creating composable UI components. It comes with a cost though: not everyone in the world uses React e.g. Angular and Vue.js are widespread too, and by sticking to React-only you leave lot of potential users of your library outside. Given that I planned to make interface pretty simple, React won’t help much here, so better to utilize standard DOM API. Moreover it’s always possible to include such library into site powered by any framework but not the other way around. I won’t annotate all code I’ve got, most of it is self-descriptive. Check out resulting . Interaction with Web Audio and Web Workers is already documented quite good on the web. The only interesting part is that I don’t use separate file for worker source but create a instead. This makes library a bit more pleasant to use: you don’t have to care about extra file. vmsg.js Blob URL The full demo is available . here Future ideas What now? Library works pretty well, I already use it at my forum for voice messages support. But there are few more interesting areas worth trying: Emscripten is going to get support for soon which should decrease build size without any efforts, just by enabling it via option. It comes with a cost of making malloc operations less effecient but for LAME it shouldn’t be a problem. emmalloc It might be interesting to implement some audio filters such as noise supression and pitch shifting as part of WebAssembly module. Because it runs in separate Worker thread, we can use advanced algorithms without the fear of introducing latency and freezing the main thread. It’s worth to compare performance of wasm-polyfill with and without memory bound checkings. Because LAME shouldn’t generally trap it’s rather safe to just disable all checks for better performance. Even in case of bug in C code it won’t affect JavaScript side because of sandboxing. Pretty complex but doable task is to make code translated by wasm-polyfill asm.js-compatible. Right now there’re a lot of spec violations so it can’t be AOT compiled. It runs as normal JavaScript therefore not as fast and performant (though JIT should optimize such code pretty hard anyway).