3D image processing has seen a steady increase in usage over the past 10 years. It is hoped that the formation of a new standards committee, announced this July by ASTM International, one of the largest voluntary standards development organisations (SDO) in the world, will further this progress.
3D imaging systems are used to rapidly capture three-dimensional information of a scene or object. It is an exciting area of the machine vision industry, with a huge variety of applications, ranging from its life-saving employment in the automotive industry, to modelling the rock group U2 in the video for Original Of The Species. 3D processing can also be used in the industrial and medical sectors, as Nick Morris briefly touched on in the last issue.
There are three main methods of 3D image processing, each with their own applications: time-of-flight measurement, laser triangulation, and photogrammetry. Time-of-flight 3D measuring technology times how long it takes for a beam of light (‘shot’ from the device) to reach an object and return to a sensor. Using the known speed of light, it is then possible to calculate the distance from this. PMD Technologies has developed a chip that can measure this distance, and the intensity of light, in each pixel, creating what is effectively a 3D camera. They are low-cost, high-resolution, robust and very compact.
However, this technique is not without its challenges. Sunlight, accuracy of time measurement and sensitivity to the intensity of light all pose problems. PMD Technologies has combated these difficulties in a number of ways. It has employed SBI circuitry, which removes the charge packets of background illumination; decreased the time resolution to 6.67ps, giving a 1mm accuracy of distance measurements; and developed scalable sensors, providing a higher sensitivity.
It is hoped that this technology could have key applications in the automotive industry. A 3D camera installed in a car could save lives by warning the driver about passing pedestrians, and give the driver accurate distances when backing into difficult parking places. The ‘Smart Airbag’, which uses time-of-flight technology rather than stereovision to detect when the passenger is ‘out of position’, is cheaper, more compact, and uses 80 per cent less computational power. On a more frivolous note, it could also provide a quicker, easier way of controlling the in-car CD player and telephone, using hand signals rather than buttons.
The laser triangulation technique has much greater precision – down to 0.1mm or better, but cannot cover such a large area. This type of sensor projects one or more spots, or lines, of laser light, onto the object under consideration. A lens is then used to form an image of this line or spot onto the detector. The position of the spot on the detector can then be used to determine the distance of the object.
Tyre sidewall inspection using scanners from LMI Technologies
LMI Technologies specialises in the laser triangulation technique, and has recently combined 2D photography with 3D sensing into the same sensor head, giving a colour image with the distances integrated into the one picture. According to Dr Walt Pastorius, technical and marketing advisor for LMI: ‘The advantages of this are that it puts 2D and 3D information into a single data stream, meaning it is automatically synchronised. It also simplifies mechanical arrangements.’
An application of this new development is in lumber manufacturing. Traditionally the industry used a lot of 3D measuring for optimisation processes. However, the added 2D element allows users to find knots or splits, and other discolorations in the wood. It is now similarly used in the food industry – when processing raw chickens, for example. It can not only guarantee uniformity in size of portions, but can also look for blemishes that would reduce the quality of the meat.
‘The combination of 2D and 3D technologies is a relatively new, exciting development, and we believe it can provide added value to applications in many other industries,’ says Dr Pastorius.
Sometimes, the geometry of the object being considered may cause optical interference, where the laser line is shadowed and cannot be seen by the camera. This is a particular problem for tyre sidewall inspection, where tyres have large amounts of raised lettering. To overcome this, the Selcom EyeCON-2 has two cameras, either side of the laser, so only one camera will be in the shadow at any one time. Tyre sidewall inspection looks for dents and bulges on the sidewalls. With tyre-quality control becoming more stringent, this accuracy could not have come at a better time.
STT has developed a non-contact 3D surface scanning technique for a number of different fields such as: quality control in Volkswagen cars;, to customise the design of prostheses and orthoses for patients in medicine; and to reconstruct the surfaces of statues, sculptures, altarpieces and other valuable artistic objects, which are then entered into a database and used for replication.
The 3D image of a scanned sculture, using an STT 3D scanner
‘The most important things to be considered are the characteristics of the surface,’ says José Manuel Jiménez, co-director of STT. ‘The STT 3D scanner automatically calculates the optimal exposure for the camera and the optimal brightness for the project depending on the characteristics of the actual object being scanned.’
To achieve this, STT developed the STT 3D scanner from scratch. At least one camera is calibrated with a lighting device – which could be either a projector or a laser beam. When scanning with a projector, the scanner projects a pattern of vertical lines on the object being considered. The scanner can reconstruct a large rectangle on the object in a single shot. ‘The advantage of scanning with a projector is that by taking multiple shots, the user can reconstruct a large object very quickly.’
3D motion capture is another important area of 3D imaging systems. Using a method called photogrammetry, STT has been successful in this field too, with equally varied applications including medicine, virtual reality, video gaming and animation. The CLIMA optical motion-capture system uses reflective markers placed on various parts of the patient’s body. As the patient performs a simple movement the system, involving two or more cameras, detects the markers, and performs an analysis on the movement, thus helping the doctor to diagnose any back or posture problems. Video images are also stored, for a more comprehensive diagnosis of neurological diseases such as Parkinson’s and Alzheimer’s, or those with limb injuries. This method is also used in sports science, where it would, for example, be used to analyse a golfer’s swing.
The STT 3D scanner captures a man's face
Once again, this tool has an important industrial application. Volkswagen uses STT’s motion capture tool in quality control, to track the movement of mobile parts in the body of the car, to calculate the kinematics and geometry of the motion, and to measure the vibrations of mechanical parts.
‘An outstanding characteristic of both technologies developed by STT is that the products are based on the use of off-the-shelf components. In particular, both product lines make use of frame grabbers from Matrox and cameras from Sony or JAI. This dramatically simplifies maintainability and customer support,’ says Jiménez.
3D image processing has already proven itself to be a versatile, multifunctional tool that is relevant in a huge range of different fields. However, some feel that it is still undervalued. For the future, industry experts hope that the new standards initiative will encourage the wider acceptance of the technology in the machine vision industry that it deserves, by providing a basis for fair comparison of products and a clear and consistent terminology, and by encouraging user confidence.
As John W. Palmateer, technical fellow at Boeing Commercial Airlines, says: ‘Consensus standards for 3D imaging systems will aid commerce and improve the quality of products and services. ASTM International is the organisation uniquely capable of bringing together the community of developers and manufacturers, technical and professional societies, and users with an interest in creating these important standards.’
3D imaging in entertainment
Away from industrial vision, XYZ RGB, a Canadian 3D scanning company, has a list of film credits that include The Matrix – Reloaded, Lord of the Rings – The Return of the King, and King Kong. XYZ RGB uses laser triangulation, with a patented method involving a galvanometer-driven mirror, to ultimately give a full 3D colour image, with nearly microscopic resolution.
XYZ RGB claims that these datasets are airtight and without error. It may be this accuracy that led Matt Aitken, digital models supervisor at Weta Digital, to comment: ‘...and when I pulled the model up in Maya, they didn’t actually fall off their chairs – but they came very close. We were able to zoom right in to almost artefact level, seeing the sculptor’s fingerprints in the clay.’
For visual effects, the film company sends sculptures or head casts to be scanned. The scan data is then used to animate characters and put actors in physically impossible positions and places. An example of this would be the punch scenes in The Matrix, and the dinosaurs in King Kong.
Of course, the benefits of this are not just limited to the entertainment industry. 3D scan data of production parts can be compared to the original CAD designs, to check that critical tolerances and quality standards are met. Sometimes, entire parts may even be directly reengineered from their datasets.
According to Troy Robinson, CTO at XYZ RGB, the difference between the two applications lies in the accuracy required. ‘We use the same scanner for all of our jobs, developed by the Canadian government 20 years ago. In films, you can get away with a much bigger tolerance for the accuracy, since they are more of an organic shape. With industrial applications you need to be spot-on. For example, one of the highest precision jobs involved moulding tiny
Weld seam inspection
BMW is using combined 2D and 3D image processing to inspect laser-brazed seams on the bodywork of its 3-Series model, using technology from Photonfocus.
In the car body manufacturing plants, conventional spot welding is giving way to laser welding and laser brazing techniques. If faults occur in the welded or brazed seam, these degrade both the strength and the sealing properties of the joints and make strict quality control indispensable. For instance, the laser-brazed seams of vehicle tailgates must be inspected with particular care, since the recessed area for the mounting of the vehicle licence plate requires the assembly of several individual components. The laser-brazed seams themselves form part of the body shell of the car, which must bear critical appraisal in terms of welding seam profile and tightness. Breaks in the weld seam are not permissible. Even small pores of diameter 0.2mm and above require subsequent finishing. These high quality standards make a visual inspection by humans for faults in the laser seam too expensive and insufficiently reliable.
In order to meet the high quality standards of welded seam inspection in vehicle production, a high dynamic range CMOS camera with a linear-logarithmic sensor response curve (LinLog technology) was taken as the basis for the development of an image processing system, which can be universally deployed for fully automated seam monitoring in laser welding or laser brazing. Immediately after the welding or brazing process, a robot-controlled measuring head is guided along the seam lines. This captures the complex seam contour in terms of surface quality and tightness and passes the data on to a computer for image processing.
The combination of specially developed image processing algorithms and a high dynamic range, fast CMOS camera allows the fully automated online inspection of welded seams. Investments in a 100 per cent control system allow a reduction of process costs in laser welding and laser brazing.
