Lidar can be used to see objects hidden around corners. However, until now, such a feat required lab-grade devices. A new study reveals off-the-shelf smartphone-grade lidar, which costs less than US $100, can also help see around corners.

The advance may have a host of potential applications. “In autonomous driving, around-the-corner sensing could help self-driving cars detect other vehicles, cyclists, or pedestrians before they come into direct view, improving safety at blind intersections or obstructed roads,” says Siddharth Somasundaram, a doctoral student at MIT’s Media Lab. “In robotics, it could help robots navigate cluttered or partially hidden environments.”

More broadly, “we think the most important implication is the democratization of the technology,” Somasundaram says. “When technologies like this become accessible, people often discover applications far beyond what the original researchers imagined.” The scientists have publicly released the code required to perform such work.

How lidar sees around corners

Lidar is increasingly giving 3D scanning capabilities to autonomous vehicles, drones, robots, and smartphones. A lidar sensor uses light much as radar uses radio waves—it shines a laser onto a location and analyzes how long reflected pulses take to return in order to calculate distances and generate a 3D map of a place.

By analyzing laser pulses that bounce off reflective surfaces, lidar can see items obscured from their direct line of sight, such as something hidden behind a corner. However, the first examples of such non-line-of-sight imaging “relied on extremely specialized scientific equipment often costing [US] $0.5 million to $1 million,” Somasundaram says. “These systems were large, expensive, and confined to laboratory environments.”

Still, over the years, higher-grade sensors—such as single-photon detectors—began appearing in consumer hardware, Somasundaram says. “Once we started experimenting with these sensors, we realized that even off-the-shelf devices were actually capturing faint around-the-corner signals,” he explains.

But such findings did not necessarily mean these sensors could actually prove useful in non-line-of-sight imaging. Consumer lidar typically captures images that are full of noise, due in part to the low-power lasers it uses because of eye safety concerns. In addition, the consumer nature of these devices meant the sensors are often relatively low resolution. Moreover, movements of the camera and target objects could result in blurry pictures.

“There were definitely moments where we weren’t sure whether meaningful imaging would even be possible,” Somasundaram says.

To overcome these challenges, instead of attempting non-line-of-sight imaging based on the data within individual pictures, the researchers analyze multiple images at once. They note they were inspired by how smartphone cameras often take bursts of photos in quick succession to improve their merged quality and how synthetic aperture radar in satellite imaging mixes signals from multiple antennas to capture images with the quality of a single large antenna.

“Once we developed algorithms that could combine information across those measurements, the hidden signals started to emerge much more clearly,” Somasundaram says.

The researchers demonstrate their lidar system in action. MIT Media Lab/YouTube

Improving consumer lidar signals

The scientists experimented with a portable smartphone lidar system that has about 100 pixels, each consisting of a laser emitter combined with a single-photon detector. They were able to use it to reconstruct 3D images of static hidden objects and track 3D motions of hidden items of known shape. In addition, they could also use hidden objects as landmarks to pinpoint the location of the lidar sensor, a capability that could help robots stay oriented in textureless spaces in which they often struggle to find their bearings. All this was done without specialized calibration.

“What I found most surprising is that these consumer lidar systems are able to capture any useful signal from around a corner at all,” Somasundaram says. “The amount of light that reaches the sensor after multiple bounces is incredibly small, and these devices were never designed with this kind of imaging in mind. Seeing that there is enough information present to reconstruct and track hidden objects was very exciting for us.”

However, Somasundaram cautions against imagining this system “as producing full photographic images of hidden scenes. Right now, the system is recovering sparse geometric and motion information from extremely weak signals.” He notes “there is still a large gap between that and the kind of detailed imagery people may associate with megapixel images from their phones.”

The new system does assume the shapes and motions of objects stay relatively consistent from frame to frame, “which allows us to combine many extremely weak measurements into a stronger signal,” Somasundaram says. However, he adds, “there are situations where those assumptions could break down.” For instance, humans may change how they are positioned. Objects may also become obscured, and so their shapes and motions could appear to alter. And the lidar sensor itself could move abruptly, he notes.

“An important direction for future work is reducing how much the system depends on these assumptions,” Somasundaram says. Better physical models, signal processing, and machine learning methods “could help the system adapt to more complex motion and scene dynamics,” he says. ”Over time, we hope these approaches will make around-the-corner sensing more robust in realistic, unconstrained environments.”

Another future direction lies in tinkering with the lidar hardware. “Right now, consumer lidar systems are primarily designed for conventional line-of-sight imaging and depth sensing,” Somasundaram says. “What would happen if future lidar hardware were designed with both visible and hidden scene understanding in mind? Improved sensor sensitivity, resolution, scanning strategies, or optical designs could dramatically improve non-line-of-sight performance as well.”

The scientists detailed their findings online 20 May in the journal Nature.