Digital neuromorphic auditory systems

This dissertation presents several digital neuromorphic auditory systems. Neuromorphic systems are capable of running in real-time at a smaller computing cost and consume lower power than on widely available general computers. These auditory systems are considered neuromorphic as they are modelled after computational models of the mammalian auditory pathway and are capable of running on digital hardware, or more specifically on a field-programmable gate array (FPGA). The models introduced are categorised into three parts: a cochlear model, an auditory pitch model, and a functional primary auditory cortical (A1) model. The cochlear model is the primary interface of an input sound signal and transmits the 2D time-frequency representation of the sound to the pitch models as well as to the A1 model. In the pitch model, pitch information is extracted from the sound signal in the form of a fundamental frequency. From the A1 model, timbre information in the form of time-frequency envelope information of the sound signal is extracted. Since the computational auditory models mentioned above are required to be implemented on FPGAs that possess fewer computational resources than general-purpose computers, the algorithms in the models are optimised so that they fit on a single FPGA. The optimisation includes using simplified hardware-implementable signal processing algorithms. Computational resource information of each model on FPGA is extracted to understand the minimum computational resources required to run each model. This information includes the quantity of logic modules, register quantity utilised, and power consumption. Similarity comparisons are also made between the output responses of the computational auditory models on software and hardware using pure tones, chirp signals, frequency-modulated signal, moving ripple signals, and musical signals as input. The limitation of the responses of the models to musical signals at multiple intensity levels is also presented along with the use of an automatic gain control algorithm to alleviate such limitations. With real-world musical signals as their inputs, the responses of the models are also tested using classifiers – the response of the auditory pitch model is used for the classification of monophonic musical notes, and the response of the A1 model is used for the classification of musical instruments with their respective monophonic signals. Classification accuracy results are shown for model output responses on both software and hardware. With the hardware implementable auditory pitch model, the classification score stands at 100% accuracy for musical notes from the 4th and 5th octaves containing 24 classes of notes. With the hardware implementation auditory timbre model, the classification score is 92% accuracy for 12 classes musical instruments. Also presented is the difference in memory requirements of the model output responses on both software and hardware – pitch and timbre responses used for the classification exercises use 24 and 2 times less memory space for hardware than software

Auf einem menschlichen Gehörmodell basierende Elektrodenstimulationsstrategie für Cochleaimplantate

Cochleaimplantate (CI), verbunden mit einer professionellen Rehabilitation, haben mehreren hunderttausenden Hörgeschädigten die verbale Kommunikation wieder ermöglicht. Betrachtet man jedoch die Rehabilitationserfolge, so haben CI-Systeme inzwischen ihre Grenzen erreicht. Die Tatsache, dass die meisten CI-Träger nicht in der Lage sind, Musik zu genießen oder einer Konversation in geräuschvoller Umgebung zu folgen, zeigt, dass es noch Raum für Verbesserungen gibt.Diese Dissertation stellt die neue CI-Signalverarbeitungsstrategie Stimulation based on Auditory Modeling (SAM) vor, die vollständig auf einem Computermodell des menschlichen peripheren Hörsystems beruht.Im Rahmen der vorliegenden Arbeit wurde die SAM Strategie dreifach evaluiert: mit vereinfachten Wahrnehmungsmodellen von CI-Nutzern, mit fünf CI-Nutzern, und mit 27 Normalhörenden mittels eines akustischen Modells der CI-Wahrnehmung. Die Evaluationsergebnisse wurden stets mit Ergebnissen, die durch die Verwendung der Advanced Combination Encoder (ACE) Strategie ermittelt wurden, verglichen. ACE stellt die zurzeit verbreitetste Strategie dar. Erste Simulationen zeigten, dass die Sprachverständlichkeit mit SAM genauso gut wie mit ACE ist. Weiterhin lieferte SAM genauere binaurale Merkmale, was potentiell zu einer Verbesserung der Schallquellenlokalisierungfähigkeit führen kann. Die Simulationen zeigten ebenfalls einen erhöhten Anteil an zeitlichen Pitchinformationen, welche von SAM bereitgestellt wurden. Die Ergebnisse der nachfolgenden Pilotstudie mit fünf CI-Nutzern zeigten mehrere Vorteile von SAM auf. Erstens war eine signifikante Verbesserung der Tonhöhenunterscheidung bei Sinustönen und gesungenen Vokalen zu erkennen. Zweitens bestätigten CI-Nutzer, die kontralateral mit einem Hörgerät versorgt waren, eine natürlicheren Klangeindruck. Als ein sehr bedeutender Vorteil stellte sich drittens heraus, dass sich alle Testpersonen in sehr kurzer Zeit (ca. 10 bis 30 Minuten) an SAM gewöhnen konnten. Dies ist besonders wichtig, da typischerweise Wochen oder Monate nötig sind. Tests mit Normalhörenden lieferten weitere Nachweise für die verbesserte Tonhöhenunterscheidung mit SAM.Obwohl SAM noch keine marktreife Alternative ist, versucht sie den Weg für zukünftige Strategien, die auf Gehörmodellen beruhen, zu ebnen und ist somit ein erfolgversprechender Kandidat für weitere Forschungsarbeiten.Cochlear implants (CIs) combined with professional rehabilitation have enabled several hundreds of thousands of hearing-impaired individuals to re-enter the world of verbal communication. Though very successful, current CI systems seem to have reached their peak potential. The fact that most recipients claim not to enjoy listening to music and are not capable of carrying on a conversation in noisy or reverberative environments shows that there is still room for improvement.This dissertation presents a new cochlear implant signal processing strategy called Stimulation based on Auditory Modeling (SAM), which is completely based on a computational model of the human peripheral auditory system.SAM has been evaluated through simplified models of CI listeners, with five cochlear implant users, and with 27 normal-hearing subjects using an acoustic model of CI perception. Results have always been compared to those acquired using Advanced Combination Encoder (ACE), which is today’s most prevalent CI strategy. First simulations showed that speech intelligibility of CI users fitted with SAM should be just as good as that of CI listeners fitted with ACE. Furthermore, it has been shown that SAM provides more accurate binaural cues, which can potentially enhance the sound source localization ability of bilaterally fitted implantees. Simulations have also revealed an increased amount of temporal pitch information provided by SAM. The subsequent pilot study, which ran smoothly, revealed several benefits of using SAM. First, there was a significant improvement in pitch discrimination of pure tones and sung vowels. Second, CI users fitted with a contralateral hearing aid reported a more natural sound of both speech and music. Third, all subjects were accustomed to SAM in a very short period of time (in the order of 10 to 30 minutes), which is particularly important given that a successful CI strategy change typically takes weeks to months. An additional test with 27 normal-hearing listeners using an acoustic model of CI perception delivered further evidence for improved pitch discrimination ability with SAM as compared to ACE.Although SAM is not yet a market-ready alternative, it strives to pave the way for future strategies based on auditory models and it is a promising candidate for further research and investigation

Neuromorphic audio processing through real-time embedded spiking neural networks.

In this work novel speech recognition and audio processing systems based on a spiking artificial cochlea and neural networks are proposed and implemented. First, the biological behavior of the animal’s auditory system is analyzed and studied, along with the classical mechanisms of audio signal processing for sound classification, including Deep Learning techniques. Based on these studies, novel audio processing and automatic audio signal recognition systems are proposed, using a bio-inspired auditory sensor as input. A desktop software tool called NAVIS (Neuromorphic Auditory VIsualizer) for post-processing the information obtained from spiking cochleae was implemented, allowing to analyze these data for further research. Next, using a 4-chip SpiNNaker hardware platform and Spiking Neural Networks, a system is proposed for classifying different time-independent audio signals, making use of a Neuromorphic Auditory Sensor and frequency studies obtained with NAVIS. To prove the robustness and analyze the limitations of the system, the input audios were disturbed, simulating extreme noisy environments. Deep Learning mechanisms, particularly Convolutional Neural Networks, are trained and used to differentiate between healthy persons and pathological patients by detecting murmurs from heart recordings after integrating the spike information from the signals using a neuromorphic auditory sensor. Finally, a similar approach is used to train Spiking Convolutional Neural Networks for speech recognition tasks. A novel SCNN architecture for timedependent signals classification is proposed, using a buffered layer that adapts the information from a real-time input domain to a static domain. The system was deployed on a 48-chip SpiNNaker platform. Finally, the performance and efficiency of these systems were evaluated, obtaining conclusions and proposing improvements for future works.Premio Extraordinario de Doctorado U

Biophysical modeling of a cochlear implant system: progress on closed-loop design using a novel patient-specific evaluation platform

The modern cochlear implant is one of the most successful neural stimulation devices, which partially mimics the workings of the auditory periphery. In the last few decades it has created a paradigm shift in hearing restoration of the deaf population, which has led to more than 324,000 cochlear implant users today. Despite its great success there is great disparity in patient outcomes without clear understanding of the aetiology of this variance in implant performance. Furthermore speech recognition in adverse conditions or music appreciation is still not attainable with today's commercial technology. This motivates the research for the next generation of cochlear implants that takes advantage of recent developments in electronics, neuroscience, nanotechnology, micro-mechanics, polymer chemistry and molecular biology to deliver high fidelity sound. The main difficulties in determining the root of the problem in the cases where the cochlear implant does not perform well are two fold: first there is not a clear paradigm on how the electrical stimulation is perceived as sound by the brain, and second there is limited understanding on the plasticity effects, or learning, of the brain in response to electrical stimulation. These significant knowledge limitations impede the design of novel cochlear implant technologies, as the technical specifications that can lead to better performing implants remain undefined. The motivation of the work presented in this thesis is to compare and contrast the cochlear implant neural stimulation with the operation of the physiological healthy auditory periphery up to the level of the auditory nerve. As such design of novel cochlear implant systems can become feasible by gaining insight on the question `how well does a specific cochlear implant system approximate the healthy auditory periphery?' circumventing the necessity of complete understanding of the brain's comprehension of patterned electrical stimulation delivered from a generic cochlear implant device. A computational model, termed Digital Cochlea Stimulation and Evaluation Tool (‘DiCoStET’) has been developed to provide an objective estimate of cochlear implant performance based on neuronal activation measures, such as vector strength and average activation. A patient-specific cochlea 3D geometry is generated using a model derived by a single anatomical measurement from a patient, using non-invasive high resolution computed tomography (HRCT), and anatomically invariant human metrics and relations. Human measurements of the neuron route within the inner ear enable an innervation pattern to be modelled which joins the space from the organ of Corti to the spiral ganglion subsequently descending into the auditory nerve bundle. An electrode is inserted in the cochlea at a depth that is determined by the user of the tool. The geometric relation between the stimulation sites on the electrode and the spiral ganglion are used to estimate an activating function that will be unique for the specific patient's cochlear shape and electrode placement. This `transfer function', so to speak, between electrode and spiral ganglion serves as a `digital patient' for validating novel cochlear implant systems. The novel computational tool is intended for use by bioengineers, surgeons, audiologists and neuroscientists alike. In addition to ‘DiCoStET’ a second computational model is presented in this thesis aiming at enhancing the understanding of the physiological mechanisms of hearing, specifically the workings of the auditory synapse. The purpose of this model is to provide insight on the sound encoding mechanisms of the synapse. A hypothetical mechanism is suggested in the release of neurotransmitter vesicles that permits the auditory synapse to encode temporal patterns of sound separately from sound intensity. DiCoStET was used to examine the performance of two different types of filters used for spectral analysis in the cochlear implant system, the Gammatone type filter and the Butterworth type filter. The model outputs suggest that the Gammatone type filter performs better than the Butterworth type filter. Furthermore two stimulation strategies, the Continuous Interleaved Stimulation (CIS) and Asynchronous Interleaved Stimulation (AIS) have been compared. The estimated neuronal stimulation spatiotemporal patterns for each strategy suggest that the overall stimulation pattern is not greatly affected by the temporal sequence change. However the finer detail of neuronal activation is different between the two strategies, and when compared to healthy neuronal activation patterns the conjecture is made that the sequential stimulation of CIS hinders the transmission of sound fine structure information to the brain. The effect of the two models developed is the feasibility of collaborative work emanating from various disciplines; especially electrical engineering, auditory physiology and neuroscience for the development of novel cochlear implant systems. This is achieved by using the concept of a `digital patient' whose artificial neuronal activation is compared to a healthy scenario in a computationally efficient manner to allow practical simulation times.Open Acces

Biomimetic cochlea filters : from modelling, design to analogue VLSI implementation

This thesis presents a novel biomimetic cochlea filter which closely resembles the biological cochlea behaviour. The filter is highly feasible for analogue very-large-scale integration (VLSI) circuits, which leads to a micro-watt-power and millimetre-sized hardware implementation. By virtue of such features, the presented filter contributes to a solid foundation for future biologically-inspired audio signal processors. Unlike existing works, the presented filter is developed by taking direct inspirations from the physiologically measured results of the biological cochlea. Since the biological cochlea has prominently different characteristics of frequency response from low to high frequencies, the biomimetic cochlea filter is built by cascading three sub-filters accordingly: a 2nd-order bandpass filter for the constant gentle low-frequency response, a 2nd-order tunable low-pass filter for the variable and selective centre frequency response and a 5th-order elliptic filter for the ultra-steep roll-off at stop-band. As a proof of concept, a biomimetic cochlea filter bank is built to process audio signals, which demonstrates the highly discriminative spectral decomposition and high-resolution time-frequency analysis capabilities similar to the biological cochlea. The filter has simple representation in the Laplace domain which leads to a convenient analogue circuit realisation. A floating-active-inductor circuit cell is developed to build the corresponding RLC ladder for each of the three sub-filters. The circuits are designed based on complementary metal-oxide-semiconductor (CMOS) transistors for VLSI implementation. Non-ideal factors of CMOS transistors including parasitics, noise and mismatches are extensively analysed and consciously considered in the circuit design. An analogue VLSI chip is successfully fabricated using 0.35μ m CMOS process. The chip measurements demonstrate that the centre frequency response of the filter has about 20 dB wide gain tuning range and a high quality factor reaching maximally over 19. The filter has a 20 dB/decade constant gentle low-frequency tail and an over 300 dB/decade sharp stop-band roll-off slope. The measured results agree with the filter model expectations and are comparable with the biological cochlea characteristics. Each filter channel consumes as low as 59.5 ~90μ Wpower and occupies only 0.9 mm2 area. Besides, the biomimetic cochlea filter chip is characterised from a wide range of angles and the experimental results cover not only the auditory filter specifications but also the integrated circuit design considerations. Furthermore, following the progressive development of the acoustic resonator based on microelectro- mechanical systems (MEMS) technology, a MEMS-CMOS implementation of the proposed filter becomes possible in the future. A key challenge for such implementation is the low sensing capacitance of the MEMS resonator which suffers significantly from sensitivity degradation due to the parasitic capacitance. A novel MEMS capacitive interface circuit chip is additionally developed to solve this issue. As shown in the chip results, the interface circuit is able to cancel the parasitic capacitance and increase the sensitivity of capacitive sensors by 35 dB without consuming any extra power. Besides, the chopper-stabilisation technique is employed which effectively reduces the circuit flicker noise and offsets. Due to these features, the interface circuit chip is capable of converting a 7.5 fF capacitance change of a 1-Volt-biased 0.5 pF capacitive sensor pair into a 0.745 V signal-conditioned output while consuming only 165.2μ W power

A Low-Power DSP Architecture for a Fully Implantable Cochlear Implant System-on-a-Chip.

The National Science Foundation Wireless Integrated Microsystems (WIMS) Engineering Research Center at the University of Michigan developed Systems-on-a-Chip to achieve biomedical implant and environmental monitoring functionality in low-milliwatt power consumption and 1-2 cm3 volume. The focus of this work is implantable electronics for cochlear implants (CIs), surgically implanted devices that utilize existing nerve connections between the brain and inner-ear in cases where degradation of the sensory hair cells in the cochlea has occurred. In the absence of functioning hair cells, a CI processes sound information and stimulates the nderlying nerve cells with currents from implanted electrodes, enabling the patient to understand speech. As the brain of the WIMS CI, the WIMS microcontroller unit (MCU) delivers the communication, signal processing, and storage capabilities required to satisfy the aggressive goals set forth. The 16-bit MCU implements a custom instruction set architecture focusing on power-efficient execution by providing separate data and address register windows, multi-word arithmetic, eight addressing modes, and interrupt and subroutine support. Along with 32KB of on-chip SRAM, a low-power 512-byte scratchpad memory is utilized by the WIMS custom compiler to obtain an average of 18% energy savings across benchmarks. A synthesizable dynamic frequency scaling circuit allows the chip to select a precision on-chip LC or ring oscillator, and perform clock scaling to minimize power dissipation; it provides glitch-free, software-controlled frequency shifting in 100ns, and dissipates only 480μW. A highly flexible and expandable 16-channel Continuous Interleaved Sampling Digital Signal Processor (DSP) is included as an MCU peripheral component. Modes are included to process data, stimulate through electrodes, and allow experimental stimulation or processing. The entire WIMS MCU occupies 9.18mm2 and consumes only 1.79mW from 1.2V in DSP mode. This is the lowest reported consumption for a cochlear DSP. Design methodologies were analyzed and a new top-down design flow is presented that encourages hardware and software co-design as well as cross-domain verification early in the design process. An O(n) technique for energy-per-instruction estimations both pre- and post-silicon is presented that achieves less than 4% error across benchmarks. This dissertation advances low-power system design while providing an improvement in hearing recovery devices.Ph.D.Electrical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/91488/1/emarsman_1.pd

Technology 2000, volume 1

The purpose of the conference was to increase awareness of existing NASA developed technologies that are available for immediate use in the development of new products and processes, and to lay the groundwork for the effective utilization of emerging technologies. There were sessions on the following: Computer technology and software engineering; Human factors engineering and life sciences; Information and data management; Material sciences; Manufacturing and fabrication technology; Power, energy, and control systems; Robotics; Sensors and measurement technology; Artificial intelligence; Environmental technology; Optics and communications; and Superconductivity

The Boston University Photonics Center annual report 2013-2014

This repository item contains an annual report that summarizes activities of the Boston University Photonics Center in the 2013-2014 academic year. The report provides quantitative and descriptive information regarding photonics programs in education, interdisciplinary research, business innovation, and technology development. The Boston University Photonics Center (BUPC) is an interdisciplinary hub for education, research, scholarship, innovation, and technology development associated with practical uses of light.This annual report summarizes activities of the Boston University Photonics Center in the 2013–2014 academic year.This has been a good year for the Photonics Center. In the following pages, you will see that the center’s faculty received prodigious honors and awards, generated more than 100 notable scholarly publications in the leading journals in our field, and attracted $14.5M in new research grants and contracts this year. Faculty and staff also expanded their efforts in education and training, through National Science Foundation–sponsored sites for Research Experiences for Undergraduates and for Teachers. As a community, we hosted a compelling series of distinguished invited speakers, and emphasized the theme of Innovations at the Intersections of Micro/Nanofabrication Technology, Biology, and Biomedicine at our annual Future of Light Symposium. We took a leadership role in running national workshops on emerging photonic fields, including an OSA Incubator on Controlled Light Propagation through Complex Media, and an NSF Workshop on Noninvasive Imaging of Brain Function. Highlights of our research achievements for the year include a distinctive Presidential Early Career Award for Scientists and Engineers (PECASE) for Assistant Professor Xue Han, an ambitious new DoD-sponsored grant for Multi-Scale Multi-Disciplinary Modeling of Electronic Materials led by Professor Enrico Bellotti, launch of our NIH-sponsored Center for Innovation in Point of Care Technologies for the Future of Cancer Care led by Professor Cathy Klapperich, and successful completion of the ambitious IARPA-funded contract for Next Generation Solid Immersion Microscopy for Fault Isolation in Back-Side Circuit Analysis led by Professor Bennett Goldberg. These three programs, which represent more than$20M in research funding for the University, are indicative of the breadth of Photonics Center research interests: from fundamental modeling of optoelectronic materials to practical development of cancer diagnostics, from exciting new discoveries in optogenetics for understanding brain function to the achievement of world-record resolution in semiconductor circuit microscopy. Our community welcomed an auspicious cohort of new faculty members, including a newly hired assistant professor and a newly hired professor (and Chair of the Mechanical Engineering Department). The Industry/University Cooperative Research Center—the centerpiece of our translational biophotonics program—continues to focus on advancing the health care and medical device industries, and has entered its fourth year of operation with a strong record of achievement and with the support of an enthusiastic industrial membership base

The Boston University Photonics Center annual report 2013-2014

This repository item contains an annual report that summarizes activities of the Boston University Photonics Center in the 2013-2014 academic year. The report provides quantitative and descriptive information regarding photonics programs in education, interdisciplinary research, business innovation, and technology development. The Boston University Photonics Center (BUPC) is an interdisciplinary hub for education, research, scholarship, innovation, and technology development associated with practical uses of light.This annual report summarizes activities of the Boston University Photonics Center in the 2013–2014 academic year.This has been a good year for the Photonics Center. In the following pages, you will see that the center’s faculty received prodigious honors and awards, generated more than 100 notable scholarly publications in the leading journals in our field, and attracted $14.5M in new research grants and contracts this year. Faculty and staff also expanded their efforts in education and training, through National Science Foundation–sponsored sites for Research Experiences for Undergraduates and for Teachers. As a community, we hosted a compelling series of distinguished invited speakers, and emphasized the theme of Innovations at the Intersections of Micro/Nanofabrication Technology, Biology, and Biomedicine at our annual Future of Light Symposium. We took a leadership role in running national workshops on emerging photonic fields, including an OSA Incubator on Controlled Light Propagation through Complex Media, and an NSF Workshop on Noninvasive Imaging of Brain Function. Highlights of our research achievements for the year include a distinctive Presidential Early Career Award for Scientists and Engineers (PECASE) for Assistant Professor Xue Han, an ambitious new DoD-sponsored grant for Multi-Scale Multi-Disciplinary Modeling of Electronic Materials led by Professor Enrico Bellotti, launch of our NIH-sponsored Center for Innovation in Point of Care Technologies for the Future of Cancer Care led by Professor Cathy Klapperich, and successful completion of the ambitious IARPA-funded contract for Next Generation Solid Immersion Microscopy for Fault Isolation in Back-Side Circuit Analysis led by Professor Bennett Goldberg. These three programs, which represent more than$20M in research funding for the University, are indicative of the breadth of Photonics Center research interests: from fundamental modeling of optoelectronic materials to practical development of cancer diagnostics, from exciting new discoveries in optogenetics for understanding brain function to the achievement of world-record resolution in semiconductor circuit microscopy. Our community welcomed an auspicious cohort of new faculty members, including a newly hired assistant professor and a newly hired professor (and Chair of the Mechanical Engineering Department). The Industry/University Cooperative Research Center—the centerpiece of our translational biophotonics program—continues to focus on advancing the health care and medical device industries, and has entered its fourth year of operation with a strong record of achievement and with the support of an enthusiastic industrial membership base