Start small and start quickly with AI: advice from Automate Forward

Share this on social media:


Greg Blackman reports from the Association for Advancing Automation's latest digital show on the advances being made using AI for industrial inspection

Andrew Ng positioned his phone above the cracked screen of an old mobile phone, took a picture, and an AI model running in the cloud picked out the crack. He did the same with a scratch on the back of the phone.

The results popped up instantaneously, and while the demonstration was a simple one – and Ng an expert in AI, previously leading the Google Brain research team, a former vice president of Baidu, a professor at Stanford University, and now CEO of industrial AI firm Landing AI – the point Ng was making is that a simple proof-of-concept study with just a mobile phone camera is a good place to start experimenting with AI in industrial inspection.

Start small and start quickly was one of his messages in his presentation during the Association for Advancing Automation's Automate Forward digital show last week. He said a common mistake with AI was making a huge investment in hardware without planning for the software, and spending years building an IT infrastructure before even looking at a neural network. Start with small projects and get AI running on one production line before scaling up.

He trained the neural network for his demo with 40 images of cracks and 30 images of scratches. The model was running in the cloud, but he said that all practical deployments Landing AI deals with are edge-based.

The other central message of his talk was to take care of the data. He said a lot of emphasis is put on the machine learning model, but that building the model accounts for only five to ten per cent of the workload. The majority – Ng said around 80 per cent of time invested – is in dealing with data.

He gave an example of a project Landing AI had worked on using an AI model to spot defects in steel. The model achieved 76 per cent accuracy in finding defects; the goal was 90 per cent accuracy. Using Landing AI's analysis tool, the steel producer was able to improve the accuracy of its model by 16 per cent, by identifying inconsistencies in data labelling.

The consistency of how the images are labelled will have a big effect on the accuracy of the model, Ng said. This is particularly important in manufacturing, where images of defects are relatively scarce.

The key to having clean data is to use the expertise of the quality control production staff to label the data, Ng advised. Then, after training the model, a tool like Landing AI's LandingLens can be used to identify any classes of data where the algorithm falls short in terms of accuracy. These images can then be revisited and relabelled to iron out ambiguities in how the defects are labelled.

Ng added that deploying a model in production is really only the halfway point in working with AI. The model needs to be retrained regularly, and it's critical to flow data back for continuous learning. He said that AI models can drift far from reality if left to their own devices, and that a QC inspector is needed to keep the model on track.

Don't expect consumer neural networks to work in manufacturing, Ng said. He also advised that, in terms of camera choice, a basic test is whether a QC inspector can look at a picture and pick out the defect. If the answer is yes, then it's reasonable to assume an AI system could do the same. 

Getting a grip on simulation

Automate Forward also heard from Dieter Fox, senior director of robotics research at Nvidia, who spoke about Nvidia's Isaac Sim robotics and AI simulator. Nvidia is using physics simulation through Isaac Sim to assess the stability of grasping points of a robotic gripper. It can randomly generate thousands of grasp points of a 3D object in parallel to see which ones are successful.

This can be done for many objects – Nvidia did this for 8,872 objects from ShapeNet with 2,000 grasps per object based on the Franka Panda gripper, a total of 17.7m grasps to assess gripper stability. The input to the neural network is an observed point cloud and the output is a possible way of grasping the object, determined through physics simulation

The neural network, trained on a database of 17.7m simulated grasps, can then be used on real-world depth data to instruct a robot arm to pick up real objects it has not seen before – the model had a 88 per cent success rate.

Nvidia is working with BMW on a neural network that will learn to pick up boxes of unknown size and from random locations to stack on a pallet. The idea is to minimise the amount of manual design and tuning of such a robot palletiser, because the robot doesn't need specific CAD models of the objects it is grasping.

In simulation, many random scenes can be generated of different size boxes and positions. The observed point cloud is simulated and used to train a neural network to segment the point cloud into different boxes, and to suggest a grasp point for each box. This is then deployed in the real world.

The task of placing the box on a target pallet is simulated too. The system can therefore stack boxes onto pallets without knowing the dimensions of the box beforehand.

The palletising work has only been done in simulation so far, but Fox said the team is close to trialling it in the real world.

He added that there is a lot of work being done by the research community investigating how to reduce the labelling effort for training neural networks, and one way to do this is to train in simulation.

Pent-up demand

Demand for automation couldn't be higher at the moment, after the disruption of last year – Mike Cicco, president and CEO of Fanuc America, described the levels of incoming orders as 'incredible' during Automate Forward, while Patrick McDermott, president North America for B&R Industrial Automation, noted that it's now a logistical and planning challenge to deliver on pent-up demand.

Areas where improvements could be made to increase uptake of automation include things like remote access for equipment, making it easier for robots and other machines to communicate, and, as noted by Christine Boles, VP of the Internet of Things group at Intel, de-risking automation and proving return on investment.

There's also the question of workforce development, which Fanuc is heavily involved in, working with 1,300 education providers – schools, colleges, universities – to develop the skills of young people. Cicco also said that the existing labour force needs an injection of new skills to tackle IoT projects.

Specifically on the automotive sector, one of the hardest hit by disruption caused by the pandemic, Cicco said he is 'bullish' about the state of that industry. Automotive production is cyclical, but Cicco expects it to remain strong over the coming years because there is a shift to new electric vehicles and new concept vehicles.

He said that automotive makers are now not just making combustion engines, but also batteries and electric drivetrains, and that with new production processes comes a need for automation. This also trickles down into the tier community, he noted.

Overall, Cicco added that automation is now propagating to smaller users, and that there is significant growth in new – and smaller – users of automation equipment because the technology has become a lot more accessible.

Next year's Automate show will be held in Detroit, USA from 6 to 9 June 2022.

Related analysis & opinion

Panelists from left: Christian Vollrath, Wenglor Sensoric; Rainer Schönhaar, Balluff; Peter Keppler, Stemmer Imaging; Andreas Waldl, B&R; and Dr Klaus-Henning Noffz, Basler

11 February 2020

Anne Wendel, director of VDMA Machine Vision, reports from SPS 2019 in Nuremberg, where vision system simplicity was discussed

05 March 2021

Greg Blackman reports from the Embedded World show, where industry experts gave insights into vision processing at the edge

14 December 2020

Greg Blackman reports on the views of panellists from AIT, MVTec, Irida Labs, and Xilinx discussing AI and machine vision

27 March 2020

Newly elected EMVA president Chris Yates considers what the future might hold for machine vision

27 January 2020

Prior to speaking at the Embedded World trade fair, The Khronos Group’s president, Neil Trevett, discusses the open API standards available for applications using machine learning and embedded vision

Related features and analysis & opinion

05 March 2021

Greg Blackman reports from the Embedded World show, where industry experts gave insights into vision processing at the edge

Vision Components MIPI modules can be connected to various embedded processors, including Nvidia Jetson boards

17 February 2021

Greg Blackman examines the effort that goes into creating an embedded vision system

14 December 2020

Greg Blackman reports on the views of panellists from AIT, MVTec, Irida Labs, and Xilinx discussing AI and machine vision

MVTec’s deep learning tool provides means to label data simply and efficiently

10 November 2020

Matthew Dale explores the software to simplify the management and labelling of deep learning data

08 June 2021

With logistics experiencing huge growth, Keely Portway considers business opportunities for vision companies in warehouse automation

Outfield analyses aerial images of orchards using machine learning to predict crop yield

01 April 2021

Farmers are starting to reap the rewards of robotics and machine vision, as Keely Portway finds out

The highlighted objects on the left show scaling errors, rotation errors or translation errors, while the objects on the right are a truer representation. Credit: Zivid

14 December 2020

Matthew Dale explores the new 3D vision tools that are enabling automated bin picking

09 December 2020

Imaging and Machine Vision Europe gathered a panel of experts to discuss uptake of 3D vision in robot automation. Greg Blackman reports on what was said