Thanks for visiting Imaging and Machine Vision Europe.

You're trying to access an editorial feature that is only available to logged in, registered users of Imaging and Machine Vision Europe. Registering is completely free, so why not sign up with us?

By registering, as well as being able to browse all content on the site without further interruption, you'll also have the option to receive our magazine (multiple times a year) and our email newsletters.

Augmented reality visor to assist doctors during surgery

Share this on social media:

European scientists are developing an augmented reality (AR) surgical visor capable of superimposing a patient’s x-ray images and other medical information in perfect 3D unison with their anatomy.

The visor prevents surgeons from having to look away during delicate procedures to view patient data, which could lead to reductions in surgery times and improvements in surgical accuracy, according to the researchers.

The VOSTARS will combine virutal and augmented reality elements to display important patient information to surgeons during delicate operations. (Credit: Medical Realities/PA)

The Video Optical See-Through Augmented Reality surgical System (VOSTARS) will display a patient’s anaesthetic data, heart rate, body temperature, blood pressure, and breathing rates conveniently into the surgeon’s field of vision. The project, which is funded under Horizon 2020, forecasts significant improvement in surgical accuracy and reductions in the time spent under anaesthetic by at least 11 per cent.

‘With this state-of-the-art, highly ergonomic visor, we intend to provide all the information required to improve surgery,’ explained project coordinator Dr Vincenzo Ferrari, biomedical engineering researcher at the department of information engineering at the University of Pisa. ‘The primary goal is to reduce not just surgery times, but also the time spent under anaesthetic and the cost involved in any operation. For the patient, this means saving 20 minutes of every three hours of surgery.’

The visor incorporates small, highly luminous micro displays and LED optical waveguides to project 2D x-ray images into the vision of the wearer. The system will work by capturing a surgeon’s perspective using a head-mounted camera and then merging this footage with real-time patient data. A central processor using advanced registration techniques for surgical navigation will then present a real-time hybrid image on the visor. ‘A clinician can move freely while still seeing the patient, the hybrid x-ray image and all of critical data all at once in a surgical dashboard inside the screen,’ said Ferrari.

The visor will combine two existing AR technologies: video see-through (VST) and optical see-through (OST). OST systems such as Microsoft Hololens grant the user a direct view of the natural environment with computer-generated images superimposed on the user’s field of vision using a semi-transparent mirror. VST systems such as the Occulus Rift submerge the user in the virtual world, seeing through a closed head-mounted display (HMD) together with stereo cameras and screens. 

Fabrizio Cutolo, engineer and expert in wearable augmented reality systems, believes neither VST nor OST alone are suitable for operating on a live patient. ‘When operating on a real person, it was clear to us that the benefits of OST and VST systems could be combined to make a hybrid device,’ he said.

‘For something as critical as an operation, we had to have the naturalistic “feel” of OST, while having the fluid interaction of the VST,’ Cutolo continued. ‘Therefore a brand new device had to be made from scratch, rather than extend an existing technology. The VST-OST augmented reality hybrid was born.'

Already three months into the three-year project, VOSTARS aims to have a working prototype of the device ready for May 2018. Initially being trialled on a number of procedures to the head, including maxillofacial (jaws and face) surgery, neurosurgery, ENT (ear, nose and throat) and orthopaedic surgery, the project hopes to be available to end users in three years, with mass production underway by 2022.

Related stories

Heightened reality: VR experts give optics wish list at Photonics West

Related news

Jeffrey Chew, CEO of ADE, and Jean-François Delepau, chairman of Lynred

09 September 2019

Recent News

24 October 2019

Imec says the new production method promises an order of magnitude gain in fabrication throughput and cost compared to processing conventional infrared imagers

04 October 2019

Each pixel in Prophesee’s Metavision sensor only activates if it detects a change in the scene – an event – which means low power, latency and data processing requirements

18 September 2019

3D sensing company, Outsight, has introduced a 3D semantic camera that combines lidar ranging with hyperspectral material analysis. The camera was introduced at the Autosens conference in Brussels

16 September 2019

OmniVision Technologies will be showing an automotive camera module at the AutoSens conference in Brussels from 17 to 19 September, built using OmniVision’s OX03A1Y image sensor with an Arm Mali-C71 image signal processor