Thanks for visiting Imaging and Machine Vision Europe.

You're trying to access an editorial feature that is only available to logged in, registered users of Imaging and Machine Vision Europe. Registering is completely free, so why not sign up with us?

By registering, as well as being able to browse all content on the site without further interruption, you'll also have the option to receive our magazine (multiple times a year) and our email newsletters.

Robot uses vision to infer human actions to work together better

Share this on social media:

Engineers at the Politecnico di Milano in Italy are developing a vision system that can infer what a human is going to do in order for a robot to operate more effectively with its human co-workers.

Professor Paolo Rocco at the Politecnico di Milano gave a presentation at the European Machine Vision Forum in Bologna, Italy about the research his group is conducting on cobots, robots designed to work with humans. The conference took place from 5 to 7 September.

One of the projects Rocco spoke about is a vision system that can predict a human’s intention by looking at how they're working. In an assembly process, for instance, once the robot knows what part the human will pick up, it can move to another task or potentially aid the human.

The system uses a deep learning algorithm trained, in the example Rocco showed, by tracking the motion of the human's right hand. In this example the engineer was assembling a box by adding a lid to it. The human and robot are in front of one another and between them are two trays: one with the box, the other with the lid. Whenever the human goes to take a box or pick up a lid, the robot understands and does the complementary action. So, when the human takes the lid, the robot picks up the box and positions it for the human to attach the lid, and vice versa. 

‘Humans normally do a sequence of actions,’ Rocco explained during his presentation. ‘The most recent thing we’ve done is to predict the sequence of actions the human will do. So, whenever the human wants to make a collaborative action, the robot will be ready, so there will be no waste of time.’

In Rocco’s setup, the robot constantly monitors the human’s action so that the cycle of the human and that of the robot synchronises. ‘The benefit was a 20 per cent reduction in cycle time just using the machine learning algorithm to infer what the human is doing,’ Rocco said.

Rocco’s group has recently founded a spin-off company called Smart Robots. The company makes a smart hardware and software solution, which optimises the space where human operators and robots work side by side. The device can be linked to robots from various manufacturers, and can be used to ensure collision avoidance, or reschedule the robot’s tasks if it interferes with what the human is doing. 

Related news

Reconstruction of traffic signs with high-resolution colour non-line-of-sight imaging using conventional CMOS cameras sensors

21 June 2019

Dr Johannes Meyer (left) and EMVA president Jochem Herrmann

20 May 2019

Recent News

04 October 2019

Each pixel in Prophesee’s Metavision sensor only activates if it detects a change in the scene – an event – which means low power, latency and data processing requirements

18 September 2019

3D sensing company, Outsight, has introduced a 3D semantic camera that combines lidar ranging with hyperspectral material analysis. The camera was introduced at the Autosens conference in Brussels

16 September 2019

OmniVision Technologies will be showing an automotive camera module at the AutoSens conference in Brussels from 17 to 19 September, built using OmniVision’s OX03A1Y image sensor with an Arm Mali-C71 image signal processor

09 September 2019

Hamamatsu Photonics claims it is the first company to mass produce a mid-infrared detector that doesn’t use mercury and cadmium, which are restricted under the European Commission’s RoHS directive