How Full Body Tracking Is Revolutionizing Social VR

Image credit: HaritoraX trackers by Shiftall

Immersion is one of the most critical components of virtual reality experiences. With companies hyping up their VR products by emphasizing their immersion, consumer expectations are higher than ever. Virtual reality, a $16.8 billion dollar industry, can create unreal experiences that can’t be reproduced in any other medium. However, the less natural it feels, the more that it takes them out of that experience. Among the most important aspects of immersion in VR are our own bodies. That’s where full body tracking comes in. Let’s talk about how full body tracking and estimation are changing how we experience social VR and what lies on the horizon.

Where Are My Legs?

Back during Meta’s hype cycle a few years ago, immersion was a commonly discussed topic. The marketing push for Meta’s leap into virtual reality promised a world that consumers could be fully immersed in. After subsidizing the cost of the Quest 2 to get it into the hands of 14.8 million people by June of 2022, the trough of disillusionment of immersion quickly set in.

Steam’s VR Hardware Survey from December 2023 reveals that the Oculus Quest 2 is the most used headset for their PC VR platform.

Aside from the cartoonish graphics needed to support standalone Quest 2 hardware, the most important detail for many people was their lack of legs. With only information from headset cameras and controller positions, pose estimation can only accomplish so much. However, consumers naturally yearned for more.

Google Trends data reveals that full body tracking search interest has increased a great deal over the past five years. It has repeatedly spiked and stabilized since 2022, showing that consumers are interested in the topic. With IMU tracking like SlimeVR just now beginning to ship to backers, now is a pivotal moment for startups to get ahead of the competition by developing and releasing trackers to support social VR users. Yet, what is full body tracking, and why do people want it so badly?

What Is Full Body Tracking?

If VR enthusiasts want accurately tracked legs, they might be surprised to learn that this has been accessible for consumers for many years. Vive released their first base-station enabled trackers in 2017. Initially marketed to allow for tracking of real-world objects in VR, it soon was adopted by players to track parts of their bodies like their legs, knees, elbows, and chest.

Full Body Tracking (FBT) enables VR users and motion capture artists to track as much of their bodies as possible. In both cases, users translate their body’s real-world movements into the movements of an avatar. FBT enables much more expression and body language than VR users might have otherwise. Full body tracking is extremely popular in social VR platforms like VRChat.

Full body tracking makes activities like dancing much more dynamic in VRChat. Image source.

If the goal is to have greater freedom of movement and immersion in VR, there are three approaches:

• Lighthouse-based tracking: Vive, Tundra trackers, measured by external base stations.

• Inertial Measurement Unit (IMU) tracking: all measurements are done internally by onboard sensors.

• Full-body estimation: an alternative to full body tracking that calculates the most probable positions of the body based on limited information.

  
  

Let’s look deeper into each example to see the pros and cons and their potential.

Lighthouse Tracking

The most popular and accurate consumer-level full body tracking solution for VR is lighthouse tracking. Powered by base stations, the position of trackers strapped to your body are measured by line of sight. Vive and Tundra are the main manufacturers in this space, and they are known for being reliable and easy to use.

However, lighthouse-based tracking is expensive. Not only do consumers need to buy trackers for hundreds of dollars each, they also need expensive base stations and shelves to mount them on. Lighthouse-based tracking can fall victim to occlusion issues, so trackers cannot be obscured by clothing or other objects like blankets.

Inertial Measurement Unit (IMU) Tracking

A technology rising in popularity for VR is IMU tracking. Powered by individual sensors, IMU trackers don’t require base stations to function. They can be completely wireless and battery operated. However, they are more difficult to install and operate. IMU trackers are more difficult to set up and are prone to drift, requiring recalibration from time to time during sessions.

IMU trackers have the advantage of not needing any line of sight. This means that they can be occluded by clothing or other objects. This tracking technology is cheaper but there aren’t many options out there. Sony’s Mocopi trackers are available now but are pretty expensive. Despite this, Mocopi trackers are still cheaper than lighthouse-based tracking solutions.

SlimeVR has firmware available for the ESP-32/8266 and a variety of different IMU hardware.

The rising star of IMU tracking is SlimeVR, an open-source full body tracking project. With SlimeVR, it’s possible for anyone to make their own IMU tracker if they have the right hardware. Thanks to a successful crowdfunding campaign, customers can order prebuilt SlimeVR trackers right now. However, shipping times are long and will take months to fulfill. There are some alternatives that use the SlimeVR protocol. owoTrack is an Android app that uses your Android phone’s sensors as a full body tracker.

Full Body Estimation (FBE)

Although not truly ‘full body tracking’, full body estimation is an alternative solution to the previous two options. Instead of using dedicated trackers for each part of your body, FBE calculates your body’s position based on the data the program already has. Generally, this is done using the position of your VR headset and controllers. However, different approaches might use something like an Xbox Kinect sensor or a webcam.

Standable is one of the most popular projects for full body estimation compatible with SteamVR. Although not an open-source project, its compatibility and ease of use are popular among users, with 74% of its reviews on Steam being positive. It’s a great option for those who can’t afford more expensive tracking solutions or just want to try out something more accurate for measuring their body in VR.

Meta has also taken this approach with the Quest 3 with its ‘inside out full body tracking’ feature, which is actually just FBE.

What Is the Future of Full Body Tracking?

With consumer interest in full body tracking rising and stabilizing, it’s clear that consumer demand in full body tracking is rising. What’s less clear is how the market will respond. There are three variables that will impact customer buying choices with FBT:

• Setup difficulty

• Cost

• Accuracy and performance

  
  

VR as an industry is beginning to recover from a trough of disillusionment. However, for VR enthusiasts, the technology for full body tracking has been around for a long time, just not in a form that has been the most accessible to consumers. The easiest and highest performing trackers require a significant investment.

However, the rise of open source IMU solutions like SlimeVR suggest future market disruption. If more businesses were to enter the market and sell competitively priced IMU tracking solutions with high performance and convenience, there’s a lot of money to be made. The same can be said if a company manages to figure out how to reduce the cost of lighthouse-based tracking.

Aside from IMU solutions, full body estimation methods are improving. Perhaps the simplest and best option for most people may be no trackers at all. Although imperfect, FBE is a great starting point to improve the quality and immersion of VR experiences for most users.

Time will tell how the market shifts, but with SlimeVR’s crowdfunded trackers finally starting to ship, I am definitely curious how businesses will respond to increasing demand for the ability to kick somebody in the face or dance in VR.