Researchers at Penn State University in the USA are leading an effort to help visually impaired people shop independently. The scientists are creating a wearable vision device that uses computer vision to interpret a complex visual scene, and aid visually impaired people do their grocery shopping.
This work is part of Visual Cortex on Silicon, a project that spans fields of inquiry ranging from materials design to brain circuitry and includes nearly 50 researchers. The wearable vision system is being developed as part of the Third Eye-VI section of the project.
Vijay Narayanan, Distinguished Professor of Computer Science and Engineering at Penn State who led the project, said the team's goal is to develop a computer vision system that will recognise that an object it sees is new to it, and store that object in memory. If it encounters the same or similar items enough times, that category will take on more importance. At some point, the system may prompt its human operator to give the item a name and tell the system where it fits in its collection of all known items.
A major challenge is to create a system that will know what to pay attention to within a crowded visual field. The human visual cortex has two general modes of attention, said Narayanan. The bottom-up mode is akin to browsing, where the scene is taken in without looking for a particular item – until something stands out from its surroundings, like a face the person recognises in a crowd. In top-down mode, the person is looking for a specific item and the eyes are drawn to things or qualities (size, colour, shape) that the individual knows resemble that item.
Third Eye scientists are trying to devise a computer vision system that can operate in either mode or combine the two, depending on the situation. Their major challenge is how to get the system to deal with a complex scene.
The device operates by reading grocery labels using recognition skills such as reading and interpreting text, and identifying logos and images. The team also incorporated barcode recognition into its Third Eye prototype to give shoppers a way to make sure they had the right item.
Narayanan said that eventually, the Third Eye system will be so good at recognising products that shoppers will be able to fine-tune the degree of match between an object it sees on the shelf and an object in the system's memory.
With a low degree of match, Third Eye might consider Corn Flakes and Sugar Frosted Flakes similar enough to be the same; with greater stringency, the system would not judge them to match, or might offer them as a potential match the shopper might want to consider.
As of December 2014, the Third Eye: VI system could recognise 87 grocery products. Precision is necessary if the system is to be useful, said Narayanan; most shoppers have strong preferences as to brand and variety.
‘If it just says “cereal” or “dairy”, it's not going to help anyone,’ he says. ‘If you want tomato sauce, we need to know if it's Prego tomato sauce. Is it organic Prego tomato sauce? That's the fine level of detail we need, and that's part of the challenge we face.’
Further information:
