Go to the Vision show in Stuttgart in November or any number of automation shows and you’ll see 3D vision technologies in action. Robotic bin picking is a popular demo, whereby a box of randomly orientated parts is rapidly emptied by an industrious robot arm controlled by stereovision or a triangulation approach of some kind. 3D vision is often quoted as one of the stronger growth areas in machine vision, but is it really such a significant technology?
The first thing to say, as Arnaud Lina, processing team manager for analysis tools at Matrox Imaging, rightly points out, is that 3D imaging solves problems that cannot be solved by 2D imaging. Anything where a volume is required would need 3D and there are a number of areas where 3D systems are installed and operating well. Portioning food is a classic example of where 3D excels, such as filleting chicken into equal sized pieces. In the timber industry, 3D measurements are made of planks of wood for quality control; in automotive, stereocameras guide robots fitting windshields in vehicles, or laser triangulation is used to generate a 3D profile of car body panels; in semiconductor, the 3D structure of silicon wafers can be measured at a micron scale; and the list goes on.
It is, however, quite a young technology relative to 2D machine vision and still needs to develop. It also isn’t the answer to every vision-related problem. Mark Williamson, director of corporate market development at Stemmer Imaging, comments: ‘Acquiring height information in some applications can be very important, but I would suggest 3D imaging is still very much a niche capability and not needed for many applications.’
Stemmer Imaging conducted a market survey when it took on LMI’s Gocator 3D sensor, which, according to Williamson, estimates that 3D is currently about 10 per cent of the 2D market. ‘You need to have a reason to go to 3D,’ he says, but adding that, in some cases, 3D can deliver the easiest, most robust solution and that it does solve some key issues.
One of the main difficulties with 3D imaging is its complexity. Williamson states: ‘3D might give more information than 2D, but it also becomes a lot more complicated.’ These complications lie in setting up and calibrating the system, as well as acquiring and making sense of the larger datasets 3D imaging produces.
‘We are, in a sense, reductionists in machine vision because most of the time we try and make the problem into a planar one,’ comments Ben Dawson, director of strategic development at Teledyne Dalsa. ‘We love things like semiconductors and machine parts that are essentially flat that we can dimension. This is both because the depth of field of most cameras is quite limited and that image processing is more difficult for 3D data.
‘The problem in 3D is essentially calibration, getting measurements to some calibrated standard, which requires in general a lot more information than a planar calibration. In 3D, you’ve got to compensate for optics and geometry, i.e. the position of the camera and lighting with respect to the object,’ he says.
There are efforts to make the technology easier to use; LMI’s Gocator 3D sensor, for instance, is pre-calibrated and compensates for temperature variation. With changes in temperature, the mechanics and the laser projection can move slightly which causes inaccuracies. Companies like LMI are building in compensation for temperature variation so that it delivers real, calibrated measurements.
With laser profiling, height increases cause the magnification to change because of the optical angles and consequently it’s more complicated to calibrate the system. ‘There’s a push to make 3D simple and put it into a smart camera for measurement and inspection and we are now starting to see a choice in this market,’ states Williamson.
The constraints tend to be in terms of the environment and the object under analysis, according to Lina of Matrox Imaging. For example, occlusions can be a problem with laser scanning, stereoscopy requires features in the scene to generate the data, while shape from focus requires a textured object, and so on.
Even camera manufacturers are having to engineer higher performance models to accommodate 3D vision. ‘A lot of our customers have been struggling with the kind of requirements for 3D imaging,’ says Jochem Herrmann, chief scientist at Adimec.
Generating 3D data with a 2D camera, which generally involves combining several 2D images to create one 3D image, needs more accurate illumination compared to taking a single 2D snapshot. It also requires a very stable and accurate camera to ensure each image is the same in terms of gain and other parameters. ‘It’s not only machine builders that have to overcome the complexities of 3D imaging when designing their systems and algorithms,’ Herrmann continues. ‘Camera manufacturers have to understand the requirements for using their products for making 3D measurements. There are more requirements for the camera in 3D imaging; to achieve the same 2D accuracy in 3D is hard to do.’
Adimec’s high-resolution 2D cameras are used for 3D imaging. One of the company’s larger markets for machine vision is semiconductor and electronics manufacturing, in which, Herrmann says, all of its customers are moving from 2D to 3D, or already have done so.
Many high-end semiconductor metrology equipment now incorporates some sort of 3D imaging capability and checks are made on the volume of solder deposited on PCBs or the thickness and flatness of silicon wafers. Chromasens’ tri-linear colour line scan system is particularly suitable for scanning large areas in 3D and provides the accuracy required for semiconductor inspection. The system generates a 3D image based on the surface texture of the object, correlating a 3D image from two separate line scan images. GPUs are used for image processing, making it fast at up to 200 million measurements per second. Markus Schnitzlein, CEO of Chromasens, says the system has a resolution of 10μm per pixel, depending on the optics and the width of the object, and can measure to an accuracy of 1μm, which can be required in the semiconductor industry.
Apart from ease of setup and calibration, inspection speed is also a problem in some circumstances, according to Dr Wolfgang Eckstein, managing director of MVTec Software. ‘In terms of speed, you’re always somewhat at a limit when working in 3D; data acquisition can be a couple of seconds, depending on the 3D imaging method. This isn’t so critical in robotics, but can be in inspection of wood or food. But this will change in the near future due to significant improvement in 3D sensor technologies,’ he says.
Software tools for 3D image processing are becoming more complete. Here, a whole three dimensional area can be subtracted out of the model in Halcon 11. Credit: MVTec Software
The other side of improvements in the technology is in the software. ‘The difficulty is that whatever tools are available in 2D have to be made available in 3D,’ states Eckstein. Tools like blob analysis, edge inspection, feature extraction, filters, classification, object recognition, alignment, identification – all are common software tools in 2D and similar tools are needed in 3D. ‘You have to acquire the 3D information, you have to do pre-processing, use blob analysis, determine the relation between objects, identify them, determine the position, and so on. In 3D it’s a little more complex, but it generally corresponds to 2D machine vision.’ The latest release of Halcon, version 11, offers the full range of these operators allowing the user to program a 3D application in a well know process chain.
The exhibit most often shown at trade fairs to demonstrate the power of 3D vision is robotic bin picking. This, however, is still in its infancy as far as real-world implementations go. Eckstein comments: ‘Vision for robotics is an important area for 3D. However, bin picking is still a challenge and there’s not a 100 per cent solution to any kind of object yet.’ He says the 3D alignment is possible in terms of vision, but typically there isn’t full 360° of freedom in the robot movement. He adds that it is becoming feasible now though.
Techniques such as stereovision and time-of-flight, which are not necessarily highly accurate, but are good for recognising the shape and orientation of an object, are often used for bin picking, palletisation and robot guidance. The vision system has to tell the robot not just where the item is, but its angle and surface profile so the gripper can pick it up in the right place. The challenge here is engineering a robust 3D system, as the material being handled and the lighting can all vary dramatically, much more than in classical machine vision applications. Robot bin picking is a newer area for 3D imaging than triangulation and inspection.
Williamson states that robotic random part bin picking and palletising is a growing area and only recently have the techniques and tools started to become more reliable. ‘Bin picking is an area that potentially could grow quite a lot in the future,’ he says. ‘There are a lot of people trying it, but I don’t know many companies that have actually implemented it successfully.’
The cost of 3D vision is another barrier for companies to contend with. According to Eckstein, a high-quality 3D sensor can cost between €4,000-15,000, which is a lot of money for most companies. It is slowly coming down in price though and a device like Microsoft’s Kinect sensor, while not a machine vision sensor per se and won’t give high accuracy or resolution, makes 3D imaging available at an extremely low price. ‘There’s now a time-of-flight camera for the mass market for €500 with a decent resolution, which would typically cost ten times that,’ states Eckstein. ‘We can expect in industry in the next two or three years there will be sensors available below €1,000. This will have a big impact on 3D imaging.’
The Kinect sensor is an example of a consumer product that can be adapted to machine vision tasks. Vladimir Tucakov, director of business development at Point Grey, comments that Kinect and the ecosystem around it and the value it has created probably surpasses everything else that’s been done with 3D in all spaces of computer vision and machine vision. ‘That’s the kind of opportunity there is for 3D in my opinion,’ he states. ‘There are all kinds of non-industrial applications in life science, traffic, surveillance, entertainment, and many other areas.’
Microsoft’s Kinect sensor operates by projecting a pattern of dots onto a scene with the returning light providing depth information via triangulation. It’s not very exact and it only operates over relatively short distances of a couple of metres. Dawson of Teledyne Dalsa comments: ‘Kinect is an example of a non-traditional machine vision system and I think we’re going to see more of these.’
There are many opportunities for 3D vision outside of the factory floor. For an application like face recognition, for example, 3D imaging can add information to 2D imaging data already acquired, thereby reducing false positives. ‘There are all sorts of applications like that where 3D is going to be, if not the core data used for analytics, the supporting data,’ explains Tucakov.
Of course, 3D imaging has it’s place just as 2D techniques have theirs. ‘3D is just another tool in the toolbox,’ comments Pierantonio Boriero, product line manager at Matrox Imaging, adding that 3D shouldn’t detract from the work that needs to be done within 2D imaging. ‘Many might get the perception that it’s all about 3D now because the 2D problem has been solved. That’s far from true. There are still applications that don’t have any vision at all. The traceability market is exploding at the moment and that’s all 2D imaging.’
This is all to say that 3D imaging won’t solve everything, but it is a powerful tool and one where a lot of development is taking place. Williamson feels the 3D market is growing at a higher rate than 2D. ‘It’s definitely a key growth area in machine vision and that’s why everyone is so excited about it.’
‘I’m arguing that 3D is necessary in the situations where it’s necessary,’ Dawson sums up. ‘We’re seeing more and more places where it is necessary and things that you wouldn’t normally think of like collision avoidance in automobiles, gaming and package sorting. These are huge markets for machine vision. Particularly automotive, with both Mercedes and Volvo integrating collision avoidance systems on their newest models, based on various forms of 3D imaging. Automotive is a market with potentially millions of orders, although the technology has to be cheap in the first place.’
Overall, 3D vision is an exciting area for the vision community and while it needs to develop further and won’t solve everything, it does add valuable capabilities to what can be achieved with machine vision.