Researchers at the Massachusetts Institute of Technology (MIT) have invented an augmented reality (AR) headset that gives humans X-ray vision.
The invention, dubbed X-AR, enables users to see things that are hidden, perhaps inside closed boxes, under piles, or behind occlusions. The headset can help users find missing items and guide them toward these items for retrieval.
X-AR combines wireless sensing with computer vision to enable users to perceive things that are invisible to the human eye. It utilizes radio-frequency (RF) signals, which can pass through common materials like cardboard boxes, plastic containers, or wooden dividers, to find hidden items that have been labeled with RFID tags, which reflect signals sent by an RF antenna.
The AR-conformal wide-band antenna tightly matches the shape of the augmented reality headset visor and provides the headset with Radio Frequency (RF) sensing capabilities. The antenna is flexible, lightweight, and fits on existing headsets without obstructing any of their cameras or the user’s field of view. The antenna could communicate with RFID-tagged items.
The team leverages a technique known as synthetic aperture radar (SAR), which is similar to how airplanes image objects on the ground. The AR-based SAR localization algorithm fuses RF sensing from the antenna and visual sensing from the headset’s cameras to localize RF-tagged objects, even when they are hidden from view.
To localize the hidden object, the X-AR utilizes visual data from the headset’s self-tracking capability to build a map of the environment and determine its location within that environment. As the user walks, it computes the probability of the RFID tag at each location. The probability will be highest at the tag’s exact location, so it uses this information to zero in on the hidden object.
Additionally, the researchers utilized the holographic visualization capabilities of the headset to guide users toward desired items and verify when they have picked them up.
When researchers tested X-AR in a warehouse-like environment, the headset could localize hidden items to within 9.8 centimeters, on average. They also found that it verified that users picked up the correct item with 96% accuracy. These results show that X-AR is successful in extending AR systems to non-line-of-sight perception, with important implications for warehousing, retail, e-commerce fulfillment, and manufacturing applications.
Next, the team plans to explore how different sensing modalities, like WiFi, mmWave technology, or terahertz waves, could enhance its visualization and interaction capabilities. They could also enhance the antenna so its range can go beyond 3 meters and extend the system for use by multiple coordinated headsets.
“Our whole goal with this project was to build an augmented reality system that allows you to see things that are invisible – things that are in boxes or around corners – and in doing so, it can guide you toward them and truly allow you to see the physical world in ways that were not possible before,” says Fadel Adib, the senior author of a paper on X-AR that will be presented at the USENIX Symposium on Networked Systems Design and Implementation.
