A touch-screen on the arm of your sofa

Share this on social media:

Researchers at Carnegie Mellon University (CMU) have demonstrated how touch-screen interfaces incorporating depth cameras and projectors can be created 'almost at will, with the wave of a hand'.

CMU’s WorldKit system enables someone to rub the arm of a sofa to 'paint' a remote control for a TV or swipe a hand across an office door to post a calendar – from which subsequent users can 'pull down' an extended version. These ad hoc interfaces can be moved, modified or deleted with similar gestures, making them highly personalised.

Researchers at the university’s Human-Computer Interaction Institute (HCII) used a ceiling-mounted camera and projector to record room geometries, sense hand gestures and project images on desired surfaces.

Robert Xiao, an HCII doctoral student, explained that WorldKit does not require such an elaborate installation: 'Depth sensors are getting better and projectors just keep getting smaller. We envision an interactive "light bulb" – a miniaturised device that could be screwed into an ordinary light fixture and pointed or moved to wherever an interface is needed.'

The system does not require prior calibration, automatically adjusting its sensing and image projection to the orientation of the chosen surface. Users can summon switches, message boards, indicator lights and a variety of other interface designs from a menu. Ultimately, the WorldKit team anticipates that users will be able to custom design interfaces with gestures.

Though WorldKit now focuses on interacting with surfaces, the researchers anticipate future work may enable users to interact with the system in free space. Likewise, higher-resolution depth cameras may someday enable the system to sense detailed finger gestures. In addition to gestures, the system also could be designed to respond to voice commands.

Recent News

06 May 2021

The GTOF0503 sensor features a 5µm three-tap iToF pixel, incorporating an array with a resolution of 640 x 480 pixels

30 April 2021

The algorithm can deduce the shape, size and layout of a room by measuring the time it takes for sound from speakers to return to the phone's microphone

20 April 2021

The Kria K26 SOM is built on top of the Zynq UltraScale+ MPSoC architecture. It has 4GB of DDR4 memory and 245 IOs for connecting sensors

18 March 2021

CEA-Leti scientists have developed a lensless, infrared spectral imaging system for medical diagnostics. It plans to commercialise the technology through a start-up