Hacking Facebook with Powster Labs: Audio Focus for 360 Video

“Inspiration struck in the form of noise cancelling headphones.”

For a recent project at Powster Labs, we were presented with a seemingly impossible challenge — a 360 video for Facebook with fully interactive audio. The concept was a 360 scene with four pieces of John Williams’ music playing at the same time. Depending on which wall you look at, you can hear a different track and see corresponding animations playing out over the walls.

This is easy enough in a custom app or website, but the challenge here was to get this feature inside of Facebook — a platform with a potentially huge audience who aren’t likely to download a special app or even click out to a custom website.

Facebook provides a very advanced set of 360 audio tools, but they are modeled firmly on the real world where we can’t focus our ears precisely and we always hear a bit of everything going on around us. Their toolkit lets us reduce out of focus audio by -24db, which was a good start, but not nearly enough for clean isolation of audio.

Through experimentation and creative thinking we were able to come up with a great solution which, we believe, is a world-first. Check it out here.

Inspiration

Putting these four audio tracks into Facebook’s player sounded messy — it was like they were all playing at once. What we needed to do was find a way to further turn down the audio in the scene that we weren’t looking at.

Inspiration struck in the form of noise cancelling headphones.

They work by monitoring the ambient noise around you then playing back the same audio but inverted, cancelling out the background noise. It’s a clever trick called Destructive Interference.

When two identical waves are played in sync, they create a larger wave. When identical waves are played inverted (out of phase), the result is no wave — silence

Powster Labs tried replicating this effect inside of the Facebook player, creating inverted recordings of the background audio in all four directions of the scene. We were surprised at just how well it worked!

This kind of isolated audio brings so many creative possibilities to 360 video, going beyond the realistic approach favored by most 360 platforms. It allows us to engineer 360 stories that don’t have a singular point of focus and to pack several concurrent stories into one scene, as I did in my 2012 app ‘In Limbo’.

I’m excited to share the breakthrough technique we developed at Powster which allows for this new way of filmmaking on the world’s largest 360 video platform.

Tutorial

Here are step by step instructions if you would like to try for yourself.

• Create a new Reaper spatial project using the tools available from Facebook. We found the best way is to duplicate their sample project and work from there. Facebook have good video tutorials here that cover the basic workflow of using their tools and Reaper.

• For simplicity, we delete all but one of the orange, spatial, tracks. Check that it’s working, then duplicate until you have 4 working spatial tracks

Basic layout with only one spatial track

Layout with 4 duplicated spatial tracks

• Open the 360 Video player app from the Facebook tools and load your video file. The video playback is synced to the playhead in reaper, so you can play your mix and then switch to the video app and hear how it works spatially.

Monitoring a 360 video linked to Reaper with Facebook’s Video Player tool

• Layout your main mix. Import your audio tracks for each angle and use the ‘fx’ button on each layer to position them in space. We found it best to use whole numbers for this to help lay things out symmetrically and neatly, and also to keep things on the horizon line.

Click the green FX button to open the spatial options for that track

The spatial layout plugin

Our final layout looks like the above, with all the spatial tracks evenly spread out around the room

• Enable mix focus at maximum. Open the ‘fx’ window on the CONTROL PLUGIN track and turn mix focus on, with a focus area of 40º and volume of -24db. This feature gets us half the way there — turning down the volume of audio we aren’t looking at. The problem is that -24db isn’t very much, so it’s a way too subtle effect for what we want.

Very important to enable mix focus like this

• Play the video through and check with the interactive video. It will sound messy, but you should be able to tell if the audio roughly focussing on the right tracks, and that everything else is good in terms of syncing with the video etc.

• Make sure you are happy — At this point we have to manually create the inverted layers using the current mix. No going back now!

Our mix is done and ready to create the noise cancellation tracks

• Go through each track, one by one, and create the noise cancellation track for each.
• Mute the target track.

Note ‘M’ muted layer toggle

• In the video player, point the camera as close as possible to the centre of that track’s position.

I’ve muted the Jurassic Park audio, and have positioned the video player to look directly at the centre of the Jurassic park scene.

• In reaper render a mix down as stereo. Select the Master Track then go file > render

• Duplicate target track (to copy position data precisely).
• Import exported audio track to this channel.

• Click the button ‘ø’ on the track to invert the phasing.

• Try it out with the video player. Play around with audio level for the new inverted track in the mixer until the sound disappears. There seems to be a magic point for each track which is slightly different.

• Repeat this process for each audio track! Mute, turn the camera to the centre of the muted track, then render the audio as a stereo track.

The final layout with all noise cancellation tracks in place

• We kept the inverted tracks audible too when we rendered files out. We want the background audio to sound as close as possible to how it will sound in the final mix.
• Export your ambisonic mix from reaper. Remember to turn off ‘decode binaural’ in the export settings, and specify 9 channels when you output. This will create the correct audio format to add to your video and upload.

• Mix the audio with the video using Facebook’s tool. Make sure to choose ‘2nd order ambisonics’ as the audio type. And to enable the ‘Focus’ feature — this effect relies heavily on Focus.

• Lastly, check the results on Facebook! Upload your file, perhaps setting the privacy to private, and check how it sounds. We found that iOS has better results than Android but perhaps this will improve with updates to the Facebook app.

Check out the final thing here

At Powster labs we do the impossible for our clients. We are an innovation partner for the movie studios and music labels, delivering world firsts and ground breaking projects. From 3D image galleries, to multichannel video in the web browser, plus AR and VR on the web.

Want to work on innovative projects like this?? Luckily for you, we have an open creative developer position in the LA office. Please apply here, we look forward to seeing your work!

Follow Powster on Twitter, Instagram, Facebook, and LinkedIn for more helpful hacks.