Rethinking 3D scans of reflective surfaces
Isak du Preez first had the idea for a 3D scanner for reflective surfaces in South Africa after talking to a friend who had spent the summer counting hail dents on cars. Insurance companies pay by the dent to repair hail damage; they want to know the number of dents, their size and their location. Du Preez thought it might be possible to scan a car's surface in 3D to find the dents.
In 2016 he built a miniature prototype of the scanner. A rudimentary draw machine moved a patterned light arch over a toy car. He used his cell phone camera to capture reflections from the shiny car body and developed software to calculate the surface profile in 3D.
Since then six large scanning machines have been built, mostly for scanning cars, and du Preez is working on a seventh for BMW. Counting hail dents is no longer the goal; now it's end-of-line inspection of cars, which is usually a manual task or uses a robot arm to scan the surface from different angles. Scanning car windscreens is another potential application, or large panes of curved architectural glass. Du Preez has also had interest from aerospace firms for inspecting some of their components. Any reflective surface – typically a large smooth surface – could benefit from this technology.
The technology is a type of deflectometry, although when du Preez started out he'd never heard of the term. The strength of du Preez's approach is to bring the precision of 3D deflectometry to large curved surfaces, previously not addressable by deflectometry, in a practical and economical way. Objects several metres wide can now be scanned with sufficient precision to detect micrometre-level local shape aberrations.
Isak du Preez’s first prototype scanner, built in 2016 from hardware store materials. Credit: Isak du Preez
Deflectometry works, traditionally, with a camera and a flat panel display, with each pixel of the camera mapped to a scene point on the display, normally the display pixels. When a reflective surface is introduced, what each pixel of the camera sees can be traced back, via specular reflection from the surface, to points in the surrounding scene. The mapping is based entirely on geometric information; it depends only on the surface shape and the positions of the camera and display.
The technique relies on specular reflection, so the surface does have to be shiny or at least glossy, but any reflection is extremely sensitive to shape – hence, hail dents only micrometres deep can be detected using regular cameras. It is much more sensitive to depth than structured light scanning, which is based on triangulation between camera and projector, and disregards specular reflection as noise.
The disadvantage of deflectometry is that the reflection visible in an object – especially a convex object – is typically small. So, with standard methods, only a tiny patch of the surface can be mapped at one time, with the process repeated and all the patches stitched together to capture the entire 3D shape. A deflectometry sensor mounted on a robot would take hundreds of different images for each small neighbourhood when inspecting paintwork on a car body, for instance.
‘The contribution of this technology is to reinvent the scene encoder,’ he said. ‘An electronic display is very convenient hardware for generating a flat scene, but far less practical if you need to comprehensively surround an object like a car. It is also over-engineered, as we don’t need to display arbitrary images in order to encode the scene.’
Isak du Preez integrating Axiscan's software with the first full-scale machine. Credit: Isak du Preez/Axiscan
Instead of a display, du Preez's method relies on a static pattern printed on the inside of an arch. Moving the arch over the car encodes a scene that is uniquely decodable. The pattern consists of a set of latitude codewords that when translated with longitudinal motion encode each scene point’s latitude and longitude. To differentiate it from other varieties of deflectometry, du Preez calls this approach Latitude Code Scanning Deflectometry.
The pattern’s design is optimised using a technique analogous to channel coding in telecommunications. The codewords are packed together inside a codespace as efficiently as possible. ‘But where discrete communications codes are optimised by sphere packing, what we do could more accurately be described as spaghetti packing, since we are encoding a continuous variable,’ du Preez explained.
At one stage, du Preez said the easiest way to design the pattern was to work with a physical analogy. He fixed up a jig shaped like a pyramid to represent the codespace, and experimented with different ways of fitting a length of flexible pipe into the volume. ‘An efficient encoding ultimately allows for shorter codewords, reducing the depth and mass of the required arch and the machine as a whole,’ he explained.
Du Preez is a software engineer and the software is where the complexity lies; the machines themselves are relatively simple. Through his company, Axiscan, du Preez provides the pattern and the software to decode it, while the large scanning machines are built or commissioned by his customers according to the requirements of the application. Ultimately, Axiscan’s software accepts camera video as input and produces a 3D mesh of the surface as output.
The 3D mesh contains one vertex per camera pixel; its lateral density is limited only by the camera resolution. In addition, the system is self-calibrating, with the software automatically calculating each camera’s position from its direct view of the encoded scene.
The 3D data from a scan of a Mini. Credit: Isak du Preez/Axiscan
The early scanners built for assessing hail damage on cars used security cameras, which are less than ideal as they compress video. 'This is the last thing you want,' du Preez said. 'You want the pixels as independent as possible for this.
'But it still worked,' he added, saying dent reconstruction from different cameras was consistent with each other down to approximately 1µm. ‘Considering the low camera specifications, the video compression, and the generally uncontrolled environmental conditions, this level of repeatability from the software is encouraging.’
He continued: ‘The hardware specifications in this project are extremely relaxed. I don’t think we’re anywhere near the ceiling of attainable accuracy yet.’
Precision in the infrared
Most metal sheets that go into building a car scatter light a lot more than exhibiting specular reflection. It's normally only after the car has been painted that it becomes reflective. But rougher surface types can still resemble a mirror when imaged in the longwave infrared. If automakers could detect defects before the expensive step of painting the car it would add a lot of value.
Attempts at infrared deflectometry in the past have usually involved moving a pattern made up of heated elements in a complex motion with many degrees of freedom, and this still only encodes a planar scene. The technology has been difficult to engineer because of the lack of off-the-shelf infrared displays and because thermal elements only change temperature slowly.
A 3D scan showing the curvature of a car windscreen. Credit: Isak du Preez/Axiscan
Du Preez thinks Latitude Code Scanning Deflectometry might be ideal for working in the infrared because it already relies on a moving scene generator, and encodes large curved surfaces. The camera just needs to be changed to an infrared camera, and the pattern switched to a thermally emissive pattern painted on a heated arch. Testing this potential is one of the aims of a collaboration with a research lab at BMW.
At the moment, du Preez's software takes about seven minutes to produce the 3D result, but that’s running on a single threaded CPU. He expects with parallel processing to cut that down to 15 seconds or less, which would fit within the natural cycle time of the movement of the arch.
Du Preez hopes that Latitude Code Scanning Deflectometry will bring the precision of deflectometry to many new applications, and the interest from BMW and others will only help refine the technique. He said Latitude Code Scanning Deflectometry is not quite as accurate as interferometry, nor as versatile as structured light scanning, but that the combination of precision and versatility, while also being cost effective, suggests the approach has promise. ‘There’s a lot to explore,’ he said.