Computer vision library uses physics-inspired algorithms

Share this on social media:

PhyCV can perform real-time edge detection at 35 frames per second on a 4K video (Image: Zhou et al.)

Researchers at the Jalali-Lab at the University of California, Los Angeles, have developed what they say is the first physics-inspired computer vision library.

Dubbed PhyCV, the new Python library uses a new class of algorithms derived directly from the equations of physics governing physical phenomena. The algorithms emulate the propagation of light through a physical medium with natural and engineered diffractive properties followed by coherent detection.

Unlike traditional algorithms that are a sequence of hand-crafted empirical rules, physics-inspired algorithms leverage physical laws of nature as blueprints.

In addition, these algorithms can, in principle, be implemented in real physical devices for fast and efficient computation in the form of analogue computing.

Currently PhyCV has three algorithms, Phase-Stretch Transform (PST), Phase-Stretch Adaptive Gradient-Field Extractor (PAGE), and Vision Enhancement via Virtual diffraction and coherent Detection (VEViD). Each algorithm has CPU and GPU versions and are inspired by the physics of the photonic time stretch, a hardware technique for ultrafast and single-shot data acquisition.

Two of the main highlights of PhyCV, according to the researchers, are its modular code architecture and its support of GPU acceleration, which is beneficial when applying the algorithms in real-time image video processing and other deep learning tasks.

In their detailed wiki describing the new computer vision library, the researchers give the running time per frame of PhyCV algorithms on a CPU (Intel i9-9900K) and GPU (NVIDIA TITAN RTX) for videos at different resolutions.

As explained in the wiki, the three current algorithms of PhyCV are shown below:

Phase-Stretch Transform (PST)

PST is a computationally efficient edge and texture detection algorithm with exceptional performance in visually impaired images. The algorithm transforms the image by emulating propagation of light through a device with engineered diffractive property followed by coherent detection. It has been applied in improving the resolution of MRI image, extracting blood vessels in retina images, dolphin identification, and wastewater treatment, single molecule biological imaging, and classification of UAV using micro Doppler imaging.

Retina vessel detection using PST in PhyCV

Phase-Stretch Adaptive Gradient-Field Extractor (PAGE)

PAGE is designed to detect edges and their orientations in digital images at various scales. The algorithm is based on the diffraction equations of optics. Metaphorically speaking, PAGE emulates the physics of birefringent (orientation-dependent) diffractive propagation through a physical device with a specific diffractive structure. The propagation converts a real-valued image into a complex function. Related information is contained in the real and imaginary components of the output. The output represents the phase of the complex function.

Directional edge detection of a sunflower using PAGE in PhyCV

Vision Enhancement via Virtual diffraction and coherent Detection (VEViD)

The newest of the three physics-inspired algorithms, VEViD, is an efficient and interpretable low-light and colour enhancement algorithm that reimagines a digital image as a spatially varying metaphoric light field and then subjects the field to the physical processes akin to diffraction and coherent detection. The term “Virtual” captures the deviation from the physical world. The light field is pixelated and the propagation imparts a phase with an arbitrary dependence on frequency which can be different from the quadratic behaviour of physical diffraction. VEViD can be further accelerated through mathematical approximations that reduce the computation time without appreciable sacrifice in image quality. A closed-form approximation for VEViD, called VEViD-lite, can achieve up to 200fps for 4K video enhancement.

Left: Colour enhancement using VEViD in PhyCV. Right: Low-light enhancement using VEViD in PhyCV

PhyCV is now available on GitHub and can be installed from pip.

Seaweed is grown on ropes, the moorings and position of which can be tracked by a vision system. Credit: PEBL

11 February 2022

Credit: Skycolors/shutterstock.com

17 October 2022

Reconstructed 3D images rendered in different perspective for the letter X. Credit: Intelligent Optics Laboratory, Liang Gao/UCLA

18 August 2022

Credit: Bartlomiej K. Wroblewski/shutterstock.com

16 August 2022

Seaweed is grown on ropes, the moorings and position of which can be tracked by a vision system. Credit: PEBL

11 February 2022