Robot uses vision to infer human actions to work together better

Share this on social media:

Engineers at the Politecnico di Milano in Italy are developing a vision system that can infer what a human is going to do in order for a robot to operate more effectively with its human co-workers.

Professor Paolo Rocco at the Politecnico di Milano gave a presentation at the European Machine Vision Forum in Bologna, Italy about the research his group is conducting on cobots, robots designed to work with humans. The conference took place from 5 to 7 September.

One of the projects Rocco spoke about is a vision system that can predict a human’s intention by looking at how they're working. In an assembly process, for instance, once the robot knows what part the human will pick up, it can move to another task or potentially aid the human.

The system uses a deep learning algorithm trained, in the example Rocco showed, by tracking the motion of the human's right hand. In this example the engineer was assembling a box by adding a lid to it. The human and robot are in front of one another and between them are two trays: one with the box, the other with the lid. Whenever the human goes to take a box or pick up a lid, the robot understands and does the complementary action. So, when the human takes the lid, the robot picks up the box and positions it for the human to attach the lid, and vice versa. 

‘Humans normally do a sequence of actions,’ Rocco explained during his presentation. ‘The most recent thing we’ve done is to predict the sequence of actions the human will do. So, whenever the human wants to make a collaborative action, the robot will be ready, so there will be no waste of time.’

In Rocco’s setup, the robot constantly monitors the human’s action so that the cycle of the human and that of the robot synchronises. ‘The benefit was a 20 per cent reduction in cycle time just using the machine learning algorithm to infer what the human is doing,’ Rocco said.

Rocco’s group has recently founded a spin-off company called Smart Robots. The company makes a smart hardware and software solution, which optimises the space where human operators and robots work side by side. The device can be linked to robots from various manufacturers, and can be used to ensure collision avoidance, or reschedule the robot’s tasks if it interferes with what the human is doing. 

Seaweed is grown on ropes, the moorings and position of which can be tracked by a vision system. Credit: PEBL

11 February 2022

Robot solutions at the Manufacturing Technology Centre. Credit: Manufacturing Technology Centre

13 December 2021

Seaweed is grown on ropes, the moorings and position of which can be tracked by a vision system. Credit: PEBL

11 February 2022

Imec researchers on-board the parabolic flight. Credit Imec

12 January 2022

Dr Alireza Rastegarpanah working with a cobot on the task of unbolting a lithium-ion battery. Credit: Dr Alireza Rastegarpanah

25 April 2022

Lab setup demonstrating the MWIR 3D measurement principle. Credit: Fraunhofer IOF

23 March 2022