Single shot 3D engineered from existing digital camera functions

Share this on social media:

A team of engineers from Duke University has unlocked a previously unrecognised 3D imaging capability of modern cameras by repurposing existing components.

The research demonstrates how image stabilisation and focus modules in modern digital cameras could be harnessed to achieve 3D imaging capabilities without additional hardware.

The team demonstrated the concept in a laboratory experiment using a small deformable mirror. The purpose of the experiment was to extract depth-of-field information from a single shot image without suffering any trade-offs in image quality.

When integrated into commercial cameras and other optical technologies, this visualisation technique could improve core functions, like image stabilisation, and increase the speed of autofocus, which would enhance the quality of photographs.

‘Real scenes are in three dimensions and they’re normally captured by taking multiple images focused at various distances,’ said Patrick Llull, Duke Imaging and Spectroscopy Program (DISP), Duke University. ‘A variety of single-shot approaches to improve the speed and quality of 3D image capture has been proposed over the past decades. Each approach, however, suffers from permanent degradations in 2D image quality and/or hardware complexity.’

The research team, led by David Brady, a professor at Duke, was able to overcome these hurdles, developing an adaptive system that can accurately extract 3D data while maintaining the ability to capture a full-resolution 2D image without a dramatic system change, such as switching out a lens.

Brady and his team have presented their findings in Optica, the open access journal from The Optical Society.

Modern digital cameras, especially those with video capabilities, are frequently equipped with modules that take the jitter out of recordings. They do this by measuring the inertia or motion of the camera and compensate by rapidly moving the lens – making multiple adjustments per second – in the module. This same hardware can also change the image capture process, recording additional information about the scene. With proper software and processing, this additional information can unlock the third dimension.

The camera is programmed to perform three functions simultaneously: sweeping through the focus range with the sensor, collecting light over a set period of time in a process called integration, and activating the stabilisation module.

As the optical stabilisation is engaged, it wobbles the lens to move the image relative to a fixed point. This, in conjunction with a focal sweep of the sensor, integrates that information into a single measurement in a way that preserves image details while granting each focus position a different optical response. The images that would have otherwise been acquired at various focal settings are directly encoded into this measurement based on where they reside in the depth of field.

For the paper, the researchers used a comparatively long exposure time to compensate for the set-up of the equipment. To emulate the workings of a camera, a beam splitter was necessary to control the deformable lens. This extra step sacrifices about 75 per cent of the light received. ‘When translated to a fully integrated camera without a beam splitter, this light loss will not be an issue and much faster exposure times will be possible,’ noted Llull.

The researchers then process a single exposure taken with the camera and obtain a data cube, including the all-focused 2D image as well as a depth map. This depth map data, in effect, describes the focus position of each pixel of the image. Since this information is already encoded into the single measurement, it’s possible to construct a depth map for the entire scene.

The final step is to process the image and depth map with a commercial 3D graphics engine. The resulting image can be used to determine the optimal focal setting for subsequent full-resolution 2D shots, as an autofocus algorithm does, but from only one image. Additionally, synthetic refocusing may be used on the resulting 3D imagery to display the scene as viewed at different depths by a human.

Though only performed in laboratory settings with surrogate technologies, the researchers believe the techniques they employed could be applied to basic consumer products. The result would be a more efficient auto-focusing process, as well as the added third dimension to traditional photography.

Further information:

Duke Imaging and Spectroscopy Program (DISP), Duke University

‘Image translation for single-shot focal tomography’, Patrick Llull et al., Optica, 2, 9, 822 (2015)

Recent News

25 May 2021

The face recognition imager consumes 10,000 times less energy than a typical camera and processor. CEA-Leti is working with STMicroelectronics on the imager

06 May 2021

The GTOF0503 sensor features a 5µm three-tap iToF pixel, incorporating an array with a resolution of 640 x 480 pixels

30 April 2021

The algorithm can deduce the shape, size and layout of a room by measuring the time it takes for sound from speakers to return to the phone's microphone

20 April 2021

The Kria K26 SOM is built on top of the Zynq UltraScale+ MPSoC architecture. It has 4GB of DDR4 memory and 245 IOs for connecting sensors