Applications of Silicon Retinas: from Neuroscience to Computer Vision

Traditional visual sensor technology is firmly rooted in the concept of sequences of image frames. The sequence of stroboscopic images in these "frame cameras" is very different compared to the information running from the retina to the visual cortex. While conventional cameras have improved in the direction of smaller pixels and higher frame rates, the basics of image acquisition have remained the same. Event-based vision sensors were originally known as "silicon retinas" but are now widely called "event cameras." They are a new type of vision sensors that take inspiration from the mechanisms developed by nature for the mammalian retina and suggest a different way of perceiving the world. As in the neural system, the sensed information is encoded in a train of spikes, or so-called events, comparable to the action potential generated in the nerve. Event-based sensors produce sparse and asynchronous output that represents in- formative changes in the scene. These sensors have advantages in terms of fast response, low latency, high dynamic range, and sparse output. All these char- acteristics are appealing for computer vision and robotic applications, increasing the interest in this kind of sensor. However, since the sensor’s output is very dif- ferent, algorithms applied for frames need to be rethought and re-adapted. This thesis focuses on several applications of event cameras in scientific scenarios. It aims to identify where they can make the difference compared to frame cam- eras. The presented applications use the Dynamic Vision Sensor (event camera developed by the Sensors Group of the Institute of Neuroinformatics, University of Zurich and ETH). To explore some applications in more extreme situations, the first chapters of the thesis focus on the characterization of several advanced versions of the standard DVS. The low light condition represents a challenging situation for every vision sensor. Taking inspiration from standard Complementary Metal Oxide Semiconductor (CMOS) technology, the DVS pixel performances in a low light scenario can be improved, increasing sensitivity and quantum efficiency, by using back-side illumination. This thesis characterizes the so-called Back Side Illumination DAVIS (BSI DAVIS) camera and shows results from its application in calcium imaging of neural activity. The BSI DAVIS has shown better performance in the low light scene due to its high Quantum Efficiency (QE) of 93% and proved to be the best type of technology for microscopy application. The BSI DAVIS allows detecting fast dynamic changes in neural fluorescent imaging using the green fluorescent calcium indicator GCaMP6f. Event camera advances have pushed the exploration of event-based cameras in computer vision tasks. Chapters of this thesis focus on two of the most active research areas in computer vision: human pose estimation and hand gesture classification. Both chapters report the datasets collected to achieve the task, fulfilling the continuous need for data for this kind of new technology. The Dynamic Vision Sensor Human Pose dataset (DHP19) is an extensive collection of 33 whole-body human actions from 17 subjects. The chapter presents the first benchmark neural network model for 3D pose estimation using DHP19. The network archives a mean error of less than 8 mm in the 3D space, which is comparable with frame-based Human Pose Estimation (HPE) methods using frames. The gesture classification chapter reports an application running on a mobile device and explores future developments in the direction of embedded portable low power devices for online processing. The sparse output from the sensor suggests using a small model with a reduced number of parameters and low power consumption. The thesis also describes pilot results from two other scientific imaging applica- tions for raindrop size measurement and laser speckle analysis presented in the appendices

The Boston University Photonics Center annual report 2013-2014

This repository item contains an annual report that summarizes activities of the Boston University Photonics Center in the 2013-2014 academic year. The report provides quantitative and descriptive information regarding photonics programs in education, interdisciplinary research, business innovation, and technology development. The Boston University Photonics Center (BUPC) is an interdisciplinary hub for education, research, scholarship, innovation, and technology development associated with practical uses of light.This annual report summarizes activities of the Boston University Photonics Center in the 2013–2014 academic year.This has been a good year for the Photonics Center. In the following pages, you will see that the center’s faculty received prodigious honors and awards, generated more than 100 notable scholarly publications in the leading journals in our field, and attracted $14.5M in new research grants and contracts this year. Faculty and staff also expanded their efforts in education and training, through National Science Foundation–sponsored sites for Research Experiences for Undergraduates and for Teachers. As a community, we hosted a compelling series of distinguished invited speakers, and emphasized the theme of Innovations at the Intersections of Micro/Nanofabrication Technology, Biology, and Biomedicine at our annual Future of Light Symposium. We took a leadership role in running national workshops on emerging photonic fields, including an OSA Incubator on Controlled Light Propagation through Complex Media, and an NSF Workshop on Noninvasive Imaging of Brain Function. Highlights of our research achievements for the year include a distinctive Presidential Early Career Award for Scientists and Engineers (PECASE) for Assistant Professor Xue Han, an ambitious new DoD-sponsored grant for Multi-Scale Multi-Disciplinary Modeling of Electronic Materials led by Professor Enrico Bellotti, launch of our NIH-sponsored Center for Innovation in Point of Care Technologies for the Future of Cancer Care led by Professor Cathy Klapperich, and successful completion of the ambitious IARPA-funded contract for Next Generation Solid Immersion Microscopy for Fault Isolation in Back-Side Circuit Analysis led by Professor Bennett Goldberg. These three programs, which represent more than$20M in research funding for the University, are indicative of the breadth of Photonics Center research interests: from fundamental modeling of optoelectronic materials to practical development of cancer diagnostics, from exciting new discoveries in optogenetics for understanding brain function to the achievement of world-record resolution in semiconductor circuit microscopy. Our community welcomed an auspicious cohort of new faculty members, including a newly hired assistant professor and a newly hired professor (and Chair of the Mechanical Engineering Department). The Industry/University Cooperative Research Center—the centerpiece of our translational biophotonics program—continues to focus on advancing the health care and medical device industries, and has entered its fourth year of operation with a strong record of achievement and with the support of an enthusiastic industrial membership base

Mapping the spatiotemporal dynamics of calcium signaling in cellular neural networks using optical flow

An optical flow gradient algorithm was applied to spontaneously forming net- works of neurons and glia in culture imaged by fluorescence optical microscopy in order to map functional calcium signaling with single pixel resolution. Optical flow estimates the direction and speed of motion of objects in an image between subsequent frames in a recorded digital sequence of images (i.e. a movie). Computed vector field outputs by the algorithm were able to track the spatiotemporal dynamics of calcium signaling pat- terns. We begin by briefly reviewing the mathematics of the optical flow algorithm, and then describe how to solve for the displacement vectors and how to measure their reliability. We then compare computed flow vectors with manually estimated vectors for the progression of a calcium signal recorded from representative astrocyte cultures. Finally, we applied the algorithm to preparations of primary astrocytes and hippocampal neurons and to the rMC-1 Muller glial cell line in order to illustrate the capability of the algorithm for capturing different types of spatiotemporal calcium activity. We discuss the imaging requirements, parameter selection and threshold selection for reliable measurements, and offer perspectives on uses of the vector data.Comment: 23 pages, 5 figures. Peer reviewed accepted version in press in Annals of Biomedical Engineerin

The Boston University Photonics Center annual report 2013-2014

This repository item contains an annual report that summarizes activities of the Boston University Photonics Center in the 2013-2014 academic year. The report provides quantitative and descriptive information regarding photonics programs in education, interdisciplinary research, business innovation, and technology development. The Boston University Photonics Center (BUPC) is an interdisciplinary hub for education, research, scholarship, innovation, and technology development associated with practical uses of light.This annual report summarizes activities of the Boston University Photonics Center in the 2013–2014 academic year.This has been a good year for the Photonics Center. In the following pages, you will see that the center’s faculty received prodigious honors and awards, generated more than 100 notable scholarly publications in the leading journals in our field, and attracted $14.5M in new research grants and contracts this year. Faculty and staff also expanded their efforts in education and training, through National Science Foundation–sponsored sites for Research Experiences for Undergraduates and for Teachers. As a community, we hosted a compelling series of distinguished invited speakers, and emphasized the theme of Innovations at the Intersections of Micro/Nanofabrication Technology, Biology, and Biomedicine at our annual Future of Light Symposium. We took a leadership role in running national workshops on emerging photonic fields, including an OSA Incubator on Controlled Light Propagation through Complex Media, and an NSF Workshop on Noninvasive Imaging of Brain Function. Highlights of our research achievements for the year include a distinctive Presidential Early Career Award for Scientists and Engineers (PECASE) for Assistant Professor Xue Han, an ambitious new DoD-sponsored grant for Multi-Scale Multi-Disciplinary Modeling of Electronic Materials led by Professor Enrico Bellotti, launch of our NIH-sponsored Center for Innovation in Point of Care Technologies for the Future of Cancer Care led by Professor Cathy Klapperich, and successful completion of the ambitious IARPA-funded contract for Next Generation Solid Immersion Microscopy for Fault Isolation in Back-Side Circuit Analysis led by Professor Bennett Goldberg. These three programs, which represent more than$20M in research funding for the University, are indicative of the breadth of Photonics Center research interests: from fundamental modeling of optoelectronic materials to practical development of cancer diagnostics, from exciting new discoveries in optogenetics for understanding brain function to the achievement of world-record resolution in semiconductor circuit microscopy. Our community welcomed an auspicious cohort of new faculty members, including a newly hired assistant professor and a newly hired professor (and Chair of the Mechanical Engineering Department). The Industry/University Cooperative Research Center—the centerpiece of our translational biophotonics program—continues to focus on advancing the health care and medical device industries, and has entered its fourth year of operation with a strong record of achievement and with the support of an enthusiastic industrial membership base

In vivo measurement of afferent activity with axon-specific calcium imaging.

In vivo calcium imaging from axons provides direct interrogation of afferent neural activity, informing the neural representations that a local circuit receives. Unlike in somata and dendrites, axonal recording of neural activity-both electrically and optically-has been difficult to achieve, thus preventing comprehensive understanding of neuronal circuit function. Here we developed an active transportation strategy to enrich GCaMP6, a genetically encoded calcium indicator, uniformly in axons with sufficient brightness, signal-to-noise ratio, and photostability to allow robust, structure-specific imaging of presynaptic activity in awake mice. Axon-targeted GCaMP6 enables frame-to-frame correlation for motion correction in axons and permits subcellular-resolution recording of axonal activity in previously inaccessible deep-brain areas. We used axon-targeted GCaMP6 to record layer-specific local afferents without contamination from somata or from intermingled dendrites in the cortex. We expect that axon-targeted GCaMP6 will facilitate new applications in investigating afferent signals relayed by genetically defined neuronal populations within and across specific brain regions

Polarization Sensor Design for Biomedical Applications

Advances in fabrication technology have enabled the development of compact, rigid polarization image sensors by integrating pixelated polarization filters with standard image sensing arrays. These compact sensors have the capability for allowing new applications across a variety of disciplines, however their design and use may be influenced by many factors. The underlying image sensor, the pixelated polarization filters, and the incident lighting conditions all directly impact how the sensor performs. In this research endeavor, I illustrate how a complete understanding of these factors can lead to both new technologies and applications in polarization sensing. To investigate the performance of the underlying image sensor, I present a new CMOS image sensor architecture with a pixel capable of operation using either measured voltages or currents. I show a detailed noise analysis of both modes, and that, as designed, voltage mode operates with lower noise than current mode. Further, I integrated aluminum nanowires with this sensor post fabrication, realizing the design of a compact CMOS sensor with polarization sensitivity. I describe a full set of experiments designed as a benchmark to evaluate the performance of compact, integrated polarization sensors. I use these tests to evaluate for incident intensity, wavelength, focus, and polarization state, demonstrating the accuracy and limitations of polarization measurements with such a compact sensor. Using these as guides, I present two novel biomedical applications that rely on the compact, real-time nature of compact integrated polarimeters. I first demonstrate how these sensors can be used to measure the dynamics of soft tissue in real-time, with no moving parts or complex optical alignment. I used a 2 megapixel integrated polarization sensor to measure the direction and strength of alignment in a bovine flexor tendon at over 20 frames per second, with results that match the current method of rotating polarizers. Secondly, I present a new technique for optical neural recording that uses intrinsic polarization reflectance and requires no fluorescent dyes or electrodes. Exposing the antennal lobe of the locust Schistocerca americana, I was able to measure a change in the polarization reflectance during the introduction of the odors hexanol and octanol with the integrated CMOS polarization sensor

Tools for interfacing, extracting, and analyzing neural signals using wide-field fluorescence imaging and optogenetics in awake behaving mice

Imaging of multiple cells has rapidly multiplied the rate of data acquisition as well as our knowledge of the complex dynamics within the mammalian brain. The process of data acquisition has been dramatically enhanced with highly affordable, sensitive image sensors enable high-throughput detection of neural activity in intact animals. Genetically encoded calcium sensors deliver a substantial boost in signal strength and in combination with equally critical advances in the size, speed, and sensitivity of image sensors available in scientific cameras enables high-throughput detection of neural activity in behaving animals using traditional wide-field fluorescence microscopy. However, the tremendous increase in data flow presents challenges to processing, analysis, and storage of captured video, and prompts a reexamination of traditional routines used to process data in neuroscience and now demand improvements in both our hardware and software applications for processing, analyzing, and storing captured video. This project demonstrates the ease with which a dependable and affordable wide-field fluorescence imaging system can be assembled and integrated with behavior control and monitoring system such as found in a typical neuroscience laboratory. An Open-source MATLAB toolbox is employed to efficiently analyze and visualize large imaging data sets in a manner that is both interactive and fully automated. This software package provides a library of image pre-processing routines optimized for batch-processing of continuous functional fluorescence video, and additionally automates a fast unsupervised ROI detection and signal extraction routine. Further, an extension of this toolbox that uses GPU programming to process streaming video, enabling the identification, segmentation and extraction of neural activity signals on-line is described in which specific algorithms improve signal specificity and image quality at the single cell level in a behaving animal. This project describes the strategic ingredients for transforming a large bulk flow of raw continuous video into proportionally informative images and knowledge

The Boston University Photonics Center annual report 2016-2017

This repository item contains an annual report that summarizes activities of the Boston University Photonics Center in the 2016-2017 academic year. The report provides quantitative and descriptive information regarding photonics programs in education, interdisciplinary research, business innovation, and technology development. The Boston University Photonics Center (BUPC) is an interdisciplinary hub for education, research, scholarship, innovation, and technology development associated with practical uses of light.This has undoubtedly been the Photonics Center’s best year since I became Director 10 years ago. In the following pages, you will see highlights of the Center’s activities in the past year, including more than 100 notable scholarly publications in the leading journals in our field, and the attraction of more than 22 million dollars in new research grants/contracts. Last year I had the honor to lead an international search for the first recipient of the Moustakas Endowed Professorship in Optics and Photonics, in collaboration with ECE Department Chair Clem Karl. This professorship honors the Center’s most impactful scholar and one of the Center’s founding visionaries, Professor Theodore Moustakas. We are delighted to haveawarded this professorship to Professor Ji-Xin Cheng, who joined our faculty this year.The past year also marked the launch of Boston University’s Neurophotonics Center, which will be allied closely with the Photonics Center. Leading that Center will be a distinguished new faculty member, Professor David Boas. David and I are together leading a new Neurophotonics NSF Research Traineeship Program that will provide $3M to promote graduate traineeships in this emerging new field. We had a busy summer hosting NSF Sites for Research Experiences for Undergraduates, Research Experiences for Teachers, and the BU Student Satellite Program. As a community, we emphasized the theme of “Optics of Cancer Imaging” at our annual symposium, hosted by Darren Roblyer. We entered a five-year second phase of NSF funding in our Industry/University Collaborative Research Center on Biophotonic Sensors and Systems, which has become the centerpiece of our translational biophotonics program. That I/UCRC continues to focus on advancing the health care and medical device industries