Researchers at Northwestern University have created an app that turns your smartphone into a movie-quality, full-body motion capture system.
Unlike competing systems that require large numbers of fixed cameras in fixed environments, the new app, called MobilePoser, is completely mobile and enables real-time motion capture in any environment.
According to the team behind MobilePoser, its technology leverages a variety of sensors already built into mobile devices such as smartphones, smartwatches, tablets, and wireless earbuds. The app captures and transforms this data using customized psychology-based algorithms to achieve full-body motion capture without the need for additional cost or equipment.
Once commercially available, MobilePoser could lead to fully immersive virtual reality experiences and hands-free gaming applications, the team says. The app could also enable indoor navigation where GPS systems may fail, as well as advanced biometric tracking options for medical applications.
“Running in real-time on mobile devices, MobilePoser delivers cutting-edge accuracy through advanced machine learning and physics-based optimization, unlocking new possibilities for gaming, fitness, and indoor navigation without the need for specialized equipment. It unleashes sexuality,” says Karan Ahuja. , assistant professor of computer science and research leader in Northwestern University’s McCormick School of Engineering. “This technology represents a major leap forward into mobile motion capture, making immersive experiences more accessible and opening the door to innovative applications across a variety of industries.”
Invention of full body motion capture using a smartphone
Cutting-edge motion capture technology has helped Hollywood bring fictional characters to life on screen. Unfortunately, capturing the movements of an actor like Andy Serkis and turning him into Gollum from The Lord of the Rings is extremely complex and expensive.
The main limitations of film-quality motion capture technology are the expensive cameras and sensors used to capture the full range of motion used to make actors look and move as realistically as possible. It’s a body suit. These costs and complexities only increase for movies like Avatar or big-budget superhero series involving large numbers of actors playing digital characters.
“It’s the gold standard in motion capture, but it costs upwards of $100,000 to run that setup,” Ahuja said of the technology employed by film studios. “We wanted to develop an accessible and democratized version that could be used with equipment that basically everyone already has.”
To achieve their goal, the team evaluated the sensors that already exist on most mobile devices. Called inertial measurement units (IMUs), these include gyroscopes, magnetometers, and accelerometers to determine whether the devices are rotating or count the wearer’s individual steps. It can also alert emergency services if it detects a fall. Unfortunately, these devices are not accurate enough to achieve accurate motion capture without some help.
Ahuja’s team tackled this problem by building a customized, multi-step artificial intelligence algorithm to refine IMU data. The team then trained the algorithm using an extensive database of “synthetic” IMU measurements generated from high-quality motion capture data.
The team then used a physically-based optimizer to refine the algorithm so that the predicted movements matched the smartphone carrier’s actual body movements. This means they are unable to bend their joints backwards or fully rotate their head.
After testing and training, the team said the device accurately captured the subjects’ movements and body posture to within 8 to 10 centimeters. For comparison, the now-defunct Microsoft Kinect motion capture system, which uses multiple cameras and requires a fixed environment, had a tracking error of 4 to 5 centimeters.
App accuracy improves with increasing number of devices
The MobilePoser app is not as accurate as the expensive motion capture technology used by large movie studios, but the inventor says wearing or carrying multiple devices can further reduce tracking error margins. they say. However, it also says it is robust enough to provide high-quality motion capture from a single device.
“Accuracy improves when you wear multiple devices, such as a smartwatch on your wrist and a smartphone in your pocket,” Ahuja said. “But the important part of this system is that it’s adaptive. Even if you don’t have a sundial and only have your phone, it will adapt to figure out the pose of your whole body. You can.”
In addition to bringing cinematic-quality motion capture technology to aspiring filmmakers and everyday users with the potential to revolutionize immersive gaming experiences, the team is confident that the app will deliver more accurate motion capture of patients. The team says there is potential for technology upgrades that can also be made available to medical professionals who benefit from this.
“Currently, doctors use pedometers to track patients’ movements,” Ahuja said. “That’s kind of sad, isn’t it? Our phones can calculate the temperature in Rome. They know more about the outside world than our own bodies.”
“We want mobile phones to become more than just intelligent pedometers,” the researchers added. “Smartphones need to detect different activities, determine poses, and act as more proactive assistants.”
The team has made the pre-trained model, data preprocessing scripts, and all model training code publicly available as open source in hopes of encouraging other researchers and engineers to take the app to the next level. did.
After a successful presentation of MobilePoser and its high-quality motion capture smartphone technology at the 2024 ACM Symposium on User Interface Software and Technology in Pittsburgh on October 15, Ahuja also announced that the app will be available on iPhones, AirPods, and Apple But owners of the watch said it will be available soon.
Christopher Plain is a science fiction and fantasy novelist and head science writer at The Debrief. Follow and connect with him ×, Check out his books at plainfiction.com or email him directly at christopher@thedebrief.org.
