Researchers from Cornell University have developed a first-of-its-kind wristband that tracks the entire body posture in 3D, thanks to a miniature camera and a customized deep neural network. Called BodyTrak, the device is an intelligent sensing technology that could be a game-changer in monitoring user body mechanics in physical activities where precision is critical.
According to Cornell researchers, BodyTrak is the first wearable to track the full body pose with a single camera.
“Since smartwatches already have a camera, technology like BodyTrak could understand the user’s pose and give real-time feedback,” said Cheng Zhang, assistant professor of information science and the paper’s senior author. “That’s handy, affordable, and does not limit the user’s moving area.”
BodyTrak is the latest body-sensing system from the SciFiLab at Cornell. Previously, the group had developed deep learning models to track hand and finger movements, facial expressions, and even silent-speech recognition.
BodyTrak comprises a dime-sized camera on the wrist, along with a customized deep neural network behind it. The deep neural network reads the camera’s rudimentary images or “silhouettes” of the user’s body in motion and virtually re-creates 14 body poses in 3D and in real-time.
To test their wristband, researchers conducted a user study with 9 participants in which each participant performed 12 daily activities, such as walking, sitting, or exercising, in varying scenarios (wearing different clothes, outdoors/indoors) with a different number of camera settings on the wrist. The results show that the system can infer the full body pose (3D positions of 14 joints) with an average error of 6.9 cm using only one miniature RGB camera (11.5mm x 9.5mm) on the wrist pointing towards the body.
“Our research shows that we don’t need our body frames to be fully within camera view for body sensing,” said Hyunchul Lim, a doctoral student in the field of information science and the paper’s lead author. “If we are able to capture just a part of our bodies, that is a lot of information to infer to reconstruct the full body.”
Maintaining privacy for bystanders near someone wearing such a sensing device is a legitimate concern when developing these technologies. Researchers said their device mitigates privacy concerns for bystanders since the camera is pointed toward the user’s body and collects only partial body images of the user. They also recognize that today’s smartwatches don’t yet have small or powerful enough cameras and adequate battery life to integrate full-body sensing, but they could in the future.
Journal reference:
- Hyunchul Lim, Yaxuan Li, Matthew Dressa, Fang Hu, Jae Hoon Kim, Ruidong Zhang, and Cheng Zhang. BodyTrak: Inferring Full-body Poses from Body Silhouettes Using a Miniature Camera on a Wristband. Proceedings of the Association for Computing Machinery (ACM). DOI: 10.1145/3552312
