Thanks for visiting Imaging and Machine Vision Europe.

You're trying to access an editorial feature that is only available to logged in, registered users of Imaging and Machine Vision Europe. Registering is completely free, so why not sign up with us?

By registering, as well as being able to browse all content on the site without further interruption, you'll also have the option to receive our magazine (multiple times a year) and our email newsletters.

Researchers teach computer to recognise body language

Share this on social media:

Researchers have used a facility featuring hundreds of video cameras to teach a computer to recognise and understand the body language of multiple people in real time, including the movements and position of their individual fingers.

The work could open up new ways for human’s and machines to interact with each other, such as being able to direct computers by simply pointing. The corresponding computer code has been released by the researchers to encourage further research and applications.

The researchers have trained a laptop to recognise body language down to the movement of individual fingers using a single camera. (Credit: Carnegie Mellon University’s Robotics Institute)

The researchers from Carnegie Mellon University’s Robotics Institute in Pennsylvania, USA, used a two-story dome embedded with 500 video cameras to teach a laptop computer to detect the exact poses of a group of people in 2D using a single camera.

Yaser Sheikh, associate professor of robotics at the university, explained that the work could enable new, more natural ways for machines to interact with humans, as understanding the nonverbal communication between individuals would allow robots to serve in social spaces, perceiving what people around them are doing, what moods they are in and whether they can be interrupted.

‘We communicate almost as much with the movement of our bodies as we do with our voice, but computers are more or less blind to it,he said.

The work could also have applications in sports analytics, where real-time pose detection would make it possible for computers to know exactly what players are doing with their arms, legs and heads at each point in time. Autonomous vehicles could also benefit from the technolgy, monitoring the body language of pedestrians to determine whether they are about to step into the street.

Further potential applications have also been identified in teaching machines to recognise conditions such as autism, dyslexia and depression, which could enable new approaches to behavioural diagnoses and rehabilitation.

Tracking multiple interacting people in real time has presented a number of challenges in the past, therefore Sheikh and his colleagues took a bottom-up approach that first localises the types of body parts in a scene, and then associates those parts with particular individuals.

The researchers took a bottom-up approach that first localises the types of body parts in a scene, and then associates those parts with particular individuals. (Credit: Carnegie Mellon University’s Robotics Institute)

The challenges for hand and finger detection were even greater however, as a camera is unlikely to see all the parts of the hand at the same time when objects are being held or gestures are made. The researchers therefore used the university’s Panoptic Studio containing 500 video cameras to obtain multiple images of different hand positions, enabling them to train a computer to predict the position of hidden appendages in an image taken by a single frontal camera.

‘The Panoptic Studio supercharges our research,’ Sheikh said. ‘It now is being used to improve body, face and hand detectors by jointly training them. Also, as work progresses to move from the 2D models of humans to 3D models, the facility’s ability to automatically generate annotated images will be crucial.’

To encourage further research and application, the researchers have released their computer code for both multi-person and hand pose estimation. It is already being widely used by research groups and more than 20 commercial groups, including automotive companies, which have expressed interest in licensing the technology.

Sheikh and his colleagues will present reports on their multiperson and hand-pose detection methods at the Computer Vision and Pattern Recognition Conference at the end of July in Honolulu.

Related news

Michał Czardybon (left), Adaptive Vision CEO, and Zbigniew Kawalec, QZ Solutions CEO

06 March 2020

The Moon landing was 50 years old in 2019. Zeiss lenses were used to capture this image

23 December 2019

Recent News

23 March 2020

Infervision's coronavirus AI system has now been installed at the Campus Bio-Medico University Hospital in Rome to screen and diagnose Covid-19 patients

19 February 2020

Sony and Prophesee have developed a stacked event-based vision sensor, it was announced at the International Solid-State Circuits Conference in San Francisco

17 January 2020

The lens-free technology from CEA-Leti, which digitally reconstructs microscope images and will be presented at SPIE Photonics West, opens up fast, automated cell screening

24 October 2019

Imec says the new production method promises an order of magnitude gain in fabrication throughput and cost compared to processing conventional infrared imagers