Deep Learning Enabled Real Time Speckle Recognition and Hyperspectral Imaging using a Multimode Fiber Array

We demonstrate the use of deep learning for fast spectral deconstruction of speckle patterns. The artificial neural network can be effectively trained using numerically constructed multispectral datasets taken from a measured spectral transmission matrix. Optimized neural networks trained on these datasets achieve reliable reconstruction of both discrete and continuous spectra from a monochromatic camera image. Deep learning is compared to analytical inversion methods as well as to a compressive sensing algorithm and shows favourable characteristics both in the oversampling and in the sparse undersampling (compressive) regimes. The deep learning approach offers significant advantages in robustness to drift or noise and in reconstruction speed. In a proof-of-principle demonstrator we achieve real time recovery of hyperspectral information using a multi-core, multi-mode fiber array as a random scattering medium.Comment: 12 pages, 6 figures + Appendix of 5 pages and 5 figure

ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks using Angle Sensitive Pixels

Deep learning using convolutional neural networks (CNNs) is quickly becoming the state-of-the-art for challenging computer vision applications. However, deep learning's power consumption and bandwidth requirements currently limit its application in embedded and mobile systems with tight energy budgets. In this paper, we explore the energy savings of optically computing the first layer of CNNs. To do so, we utilize bio-inspired Angle Sensitive Pixels (ASPs), custom CMOS diffractive image sensors which act similar to Gabor filter banks in the V1 layer of the human visual cortex. ASPs replace both image sensing and the first layer of a conventional CNN by directly performing optical edge filtering, saving sensing energy, data bandwidth, and CNN FLOPS to compute. Our experimental results (both on synthetic data and a hardware prototype) for a variety of vision tasks such as digit recognition, object recognition, and face identification demonstrate using ASPs while achieving similar performance compared to traditional deep learning pipelines.Comment: Presented in CVPR 2016 (oral), 10 pages, 12 figures. This new version corrects the comparison between imaging power for ASPs and a regular image senso

ReconNet: Non-Iterative Reconstruction of Images from Compressively Sensed Random Measurements

The goal of this paper is to present a non-iterative and more importantly an extremely fast algorithm to reconstruct images from compressively sensed (CS) random measurements. To this end, we propose a novel convolutional neural network (CNN) architecture which takes in CS measurements of an image as input and outputs an intermediate reconstruction. We call this network, ReconNet. The intermediate reconstruction is fed into an off-the-shelf denoiser to obtain the final reconstructed image. On a standard dataset of images we show significant improvements in reconstruction results (both in terms of PSNR and time complexity) over state-of-the-art iterative CS reconstruction algorithms at various measurement rates. Further, through qualitative experiments on real data collected using our block single pixel camera (SPC), we show that our network is highly robust to sensor noise and can recover visually better quality images than competitive algorithms at extremely low sensing rates of 0.1 and 0.04. To demonstrate that our algorithm can recover semantically informative images even at a low measurement rate of 0.01, we present a very robust proof of concept real-time visual tracking application.Comment: Accepted at IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 201

Deep Ptych: Subsampled Fourier Ptychography using Generative Priors

This paper proposes a novel framework to regularize the highly ill-posed and non-linear Fourier ptychography problem using generative models. We demonstrate experimentally that our proposed algorithm, Deep Ptych, outperforms the existing Fourier ptychography techniques, in terms of quality of reconstruction and robustness against noise, using far fewer samples. We further modify the proposed approach to allow the generative model to explore solutions outside the range, leading to improved performance

On some common compressive sensing recovery algorithms and applications - Review paper

Compressive Sensing, as an emerging technique in signal processing is reviewed in this paper together with its common applications. As an alternative to the traditional signal sampling, Compressive Sensing allows a new acquisition strategy with significantly reduced number of samples needed for accurate signal reconstruction. The basic ideas and motivation behind this approach are provided in the theoretical part of the paper. The commonly used algorithms for missing data reconstruction are presented. The Compressive Sensing applications have gained significant attention leading to an intensive growth of signal processing possibilities. Hence, some of the existing practical applications assuming different types of signals in real-world scenarios are described and analyzed as well.Comment: submitted to Facta Universitatis Scientific Journal, Series: Electronics and Energetics, March 201

Robust event-stream pattern tracking based on correlative filter

Object tracking based on retina-inspired and event-based dynamic vision sensor (DVS) is challenging for the noise events, rapid change of event-stream shape, chaos of complex background textures, and occlusion. To address these challenges, this paper presents a robust event-stream pattern tracking method based on correlative filter mechanism. In the proposed method, rate coding is used to encode the event-stream object in each segment. Feature representations from hierarchical convolutional layers of a deep convolutional neural network (CNN) are used to represent the appearance of the rate encoded event-stream object. The results prove that our method not only achieves good tracking performance in many complicated scenes with noise events, complex background textures, occlusion, and intersected trajectories, but also is robust to variable scale, variable pose, and non-rigid deformations. In addition, this correlative filter based event-stream tracking has the advantage of high speed. The proposed approach will promote the potential applications of these event-based vision sensors in self-driving, robots and many other high-speed scenes

Rank Minimization for Snapshot Compressive Imaging

Snapshot compressive imaging (SCI) refers to compressive imaging systems where multiple frames are mapped into a single measurement, with video compressive imaging and hyperspectral compressive imaging as two representative applications. Though exciting results of high-speed videos and hyperspectral images have been demonstrated, the poor reconstruction quality precludes SCI from wide applications.This paper aims to boost the reconstruction quality of SCI via exploiting the high-dimensional structure in the desired signal. We build a joint model to integrate the nonlocal self-similarity of video/hyperspectral frames and the rank minimization approach with the SCI sensing process. Following this, an alternating minimization algorithm is developed to solve this non-convex problem. We further investigate the special structure of the sampling process in SCI to tackle the computational workload and memory issues in SCI reconstruction. Both simulation and real data (captured by four different SCI cameras) results demonstrate that our proposed algorithm leads to significant improvements compared with current state-of-the-art algorithms. We hope our results will encourage the researchers and engineers to pursue further in compressive imaging for real applications.Comment: 18 pages, 21 figures, and 2 tables. Code available at https://github.com/liuyang12/DeSC

Cells exploit a phase transition to mechanically remodel the fibrous extracellular matrix

Through mechanical forces, biological cells remodel the surrounding collagen network, generating striking deformation patterns. Tethers—tracts of high densification and fibre alignment—form between cells, thinner bands emanate from cell clusters. While tethers facilitate cell migration and communication, how they form is unclear. Combining modelling, simulation and experiment, we show that tether formation is a densification phase transition of the extracellular matrix, caused by buckling instability of network fibres under cell-induced compression, featuring unexpected similarities with martensitic microstructures. Multiscale averaging yields a two-phase, bistable continuum energy landscape for fibrous collagen, with a densified/aligned second phase. Simulations predict strain discontinuities between the undensified and densified phase, which localizes within tethers as experimentally observed. In our experiments, active particles induce similar localized patterns as cells. This shows how cells exploit an instability to mechanically remodel the extracellular matrix simply by contracting, thereby facilitating mechanosensing, invasion and metastasis

Affect Sensing on Smartphone - Possibilities of Understanding Cognitive Decline in Aging Population

Due to increasing sensing capacity, smartphones offer unprecedented opportunity to monitor human health. Affect sensing is one such essential monitoring that can be achieved on smartphones. Information about affect can be useful for many modern applications. In particular, it can be potentially used for understanding cognitive decline in aging population. In this paper we present an overview of the existing literature that offer affect sensing on smartphone platform. Most importantly, we present the challenges that need to be addressed to make affect sensing on smartphone a reality.Comment: This paper has been withdrawn due to some conceptual erro

Flexible and tunable silicon photonic circuits on plastic substrates

Flexible microelectronics has shown tremendous promise in a broad spectrum of applications, especially those that cannot be addressed by conventional microelectronics in rigid materials and constructions1-3. These unconventional yet important applications range from flexible consumer electronics to conformal sensor arrays and biomedical devices. A recent successful paradigm shift in implementing flexible electronics is to physically transfer and bond highly integrated devices made in high-quality, crystalline semiconductor materials on to plastic materials4-8. Here we demonstrate a flexible form of silicon photonics on plastic substrates using the transfer-and-bond fabrication method. Photonic circuits including interferometers and resonators have been transferred onto flexible plastic substrates with preserved functionalities and performance. By mechanically deforming the flexible substrates, the optical characteristics of the devices can be tuned reversibly over a remarkably large range. The demonstration of the new flexible photonic system based on the silicon-on-plastic (SOP) material platform could open the door to a plethora of novel applications, including tunable photonics, optomechanical sensors and bio-mechanical and bio-photonic probes.Comment: Part of this work was presented at 2012 CLEO conference on May 8th, 201