A new method for reconstructing images in 3D has been developed that up until now has only worked under controlled lab environments.
By moving outside of the lab, the Technical University of Munich (TUM) technique could help advance autonomous driving and the preservation of cultural artefacts.
Neural, photometric stereo approaches have limitations
Neural approaches have shown significant progress on camera-based reconstruction. But they require either a fairly dense sampling of the viewing sphere or pre-training on an existing dataset, which limits their ability to make predictions based on past observations.
In contrast, photometric stereo (PS) approaches have shown great potential for achieving high-quality reconstruction under sparse viewpoints. Yet they are impractical because they typically require tedious laboratory conditions that are restricted to dark rooms and often multi-staged making them subject to accumulated errors.
The TUM technique addresses these shortcomings through an end-to-end uncalibrated multi-view PS framework for reconstructing high-resolution shapes acquired from sparse viewpoints in a real-world environment.
The method reduces the dark room assumption, allowing a combination of static ambient lighting and dynamic near LED lighting to enable easy data capture outside of the lab.
The team’s experimental validation confirms that it outperforms existing baseline approaches in the regime of sparse viewpoints by a large margin. This allows high accuracy 3D reconstruction from the dark room to the real world while maintaining a reasonable data capture complexity.
“We can use the natural surroundings and can reconstruct relatively textureless objects for our reconstructions,” said Daniel Cremers, professor of Computer Vision and Artificial Intelligence at TUM and leader of MCML and a director of the Munich Data Science Institute (MDSI).
Applications in autonomous driving and preservation of historical artefacts
The method can be used to preserve historical monuments or digitise museum exhibits. If these are destroyed or decay over time, photographic images can be used to reconstruct the originals and create authentic replicas.
Cremers’ team also develops neural camera-based reconstruction methods for autonomous driving, where a camera films the vehicle's surroundings. The autonomous car can model its surroundings in real-time, develop a three-dimensional representation of the scene, and use it to make decisions. The process is based on neural networks that predict 3D point clouds for individual video images that are then merged into a large-scale model of the roads travelled.
Credit for main image: Marko Aliaksandr/shutterstock
