Real-time FGPA implementation of a neuromorphic pitch detection system

This thesis explores the real-time implementation of a biologically inspired pitch detection system in digital electronics. Pitch detection is well understood and has been shown to occur in the initial stages of the auditory brainstem. By building such a system in digital hardware we can prove the feasibility of implementing neuromorphic systems using digital technology. This research not only aims to prove that such an implementation is possible but to investigate ways of achieving efficient and effective designs. We aim to achieve this complexity reduction while maintaining the fine granularity of the signal processing inherent in neural systems. By producing an efficient design we present the possibility of implementing the system within the available resources, thus producing a demonstrable system. This thesis presents a review of computational models of all the components within the pitch detection system. The review also identifies key issues relating to the efficient implementation and development of the pitch detection system. Four investigations are presented to address these issues for optimal neuromorphic designs of neuromorphic systems. The first investigation aims to produce the first-ever digital hardware implementation of the inner hair cell. The second investigation develops simplified models of the auditory nerve and the coincidence cell. The third investigation aims to reduce the most complex stage of the system, the stellate chopper cell array. Finally, we investigate implementing a large portion of the pitch detection system in hardware. The results contained in this thesis enable us to understand the feasibility of implementing such systems in real-time digital hardware. This knowledge may help researchers to make design decisions within the field of digital neuromorphic systems

Processing of signals in the peripheral auditory system in relation to aural perception

Imperial Users onl

Auf einem menschlichen Gehörmodell basierende Elektrodenstimulationsstrategie für Cochleaimplantate

Cochleaimplantate (CI), verbunden mit einer professionellen Rehabilitation, haben mehreren hunderttausenden Hörgeschädigten die verbale Kommunikation wieder ermöglicht. Betrachtet man jedoch die Rehabilitationserfolge, so haben CI-Systeme inzwischen ihre Grenzen erreicht. Die Tatsache, dass die meisten CI-Träger nicht in der Lage sind, Musik zu genießen oder einer Konversation in geräuschvoller Umgebung zu folgen, zeigt, dass es noch Raum für Verbesserungen gibt.Diese Dissertation stellt die neue CI-Signalverarbeitungsstrategie Stimulation based on Auditory Modeling (SAM) vor, die vollständig auf einem Computermodell des menschlichen peripheren Hörsystems beruht.Im Rahmen der vorliegenden Arbeit wurde die SAM Strategie dreifach evaluiert: mit vereinfachten Wahrnehmungsmodellen von CI-Nutzern, mit fünf CI-Nutzern, und mit 27 Normalhörenden mittels eines akustischen Modells der CI-Wahrnehmung. Die Evaluationsergebnisse wurden stets mit Ergebnissen, die durch die Verwendung der Advanced Combination Encoder (ACE) Strategie ermittelt wurden, verglichen. ACE stellt die zurzeit verbreitetste Strategie dar. Erste Simulationen zeigten, dass die Sprachverständlichkeit mit SAM genauso gut wie mit ACE ist. Weiterhin lieferte SAM genauere binaurale Merkmale, was potentiell zu einer Verbesserung der Schallquellenlokalisierungfähigkeit führen kann. Die Simulationen zeigten ebenfalls einen erhöhten Anteil an zeitlichen Pitchinformationen, welche von SAM bereitgestellt wurden. Die Ergebnisse der nachfolgenden Pilotstudie mit fünf CI-Nutzern zeigten mehrere Vorteile von SAM auf. Erstens war eine signifikante Verbesserung der Tonhöhenunterscheidung bei Sinustönen und gesungenen Vokalen zu erkennen. Zweitens bestätigten CI-Nutzer, die kontralateral mit einem Hörgerät versorgt waren, eine natürlicheren Klangeindruck. Als ein sehr bedeutender Vorteil stellte sich drittens heraus, dass sich alle Testpersonen in sehr kurzer Zeit (ca. 10 bis 30 Minuten) an SAM gewöhnen konnten. Dies ist besonders wichtig, da typischerweise Wochen oder Monate nötig sind. Tests mit Normalhörenden lieferten weitere Nachweise für die verbesserte Tonhöhenunterscheidung mit SAM.Obwohl SAM noch keine marktreife Alternative ist, versucht sie den Weg für zukünftige Strategien, die auf Gehörmodellen beruhen, zu ebnen und ist somit ein erfolgversprechender Kandidat für weitere Forschungsarbeiten.Cochlear implants (CIs) combined with professional rehabilitation have enabled several hundreds of thousands of hearing-impaired individuals to re-enter the world of verbal communication. Though very successful, current CI systems seem to have reached their peak potential. The fact that most recipients claim not to enjoy listening to music and are not capable of carrying on a conversation in noisy or reverberative environments shows that there is still room for improvement.This dissertation presents a new cochlear implant signal processing strategy called Stimulation based on Auditory Modeling (SAM), which is completely based on a computational model of the human peripheral auditory system.SAM has been evaluated through simplified models of CI listeners, with five cochlear implant users, and with 27 normal-hearing subjects using an acoustic model of CI perception. Results have always been compared to those acquired using Advanced Combination Encoder (ACE), which is today’s most prevalent CI strategy. First simulations showed that speech intelligibility of CI users fitted with SAM should be just as good as that of CI listeners fitted with ACE. Furthermore, it has been shown that SAM provides more accurate binaural cues, which can potentially enhance the sound source localization ability of bilaterally fitted implantees. Simulations have also revealed an increased amount of temporal pitch information provided by SAM. The subsequent pilot study, which ran smoothly, revealed several benefits of using SAM. First, there was a significant improvement in pitch discrimination of pure tones and sung vowels. Second, CI users fitted with a contralateral hearing aid reported a more natural sound of both speech and music. Third, all subjects were accustomed to SAM in a very short period of time (in the order of 10 to 30 minutes), which is particularly important given that a successful CI strategy change typically takes weeks to months. An additional test with 27 normal-hearing listeners using an acoustic model of CI perception delivered further evidence for improved pitch discrimination ability with SAM as compared to ACE.Although SAM is not yet a market-ready alternative, it strives to pave the way for future strategies based on auditory models and it is a promising candidate for further research and investigation

On the mechanism of response latencies in auditory nerve fibers

Despite the structural differences of the middle and inner ears, the latency pattern in auditory nerve fibers to an identical sound has been found similar across numerous species. Studies have shown the similarity in remarkable species with distinct cochleae or even without a basilar membrane. This stimulus-, neuron-, and species- independent similarity of latency cannot be simply explained by the concept of cochlear traveling waves that is generally accepted as the main cause of the neural latency pattern. An original concept of Fourier pattern is defined, intended to characterize a feature of temporal processing—specifically phase encoding—that is not readily apparent in more conventional analyses. The pattern is created by marking the first amplitude maximum for each sinusoid component of the stimulus, to encode phase information. The hypothesis is that the hearing organ serves as a running analyzer whose output reflects synchronization of auditory neural activity consistent with the Fourier pattern. A combined research of experimental, correlational and meta-analysis approaches is used to test the hypothesis. Manipulations included phase encoding and stimuli to test their effects on the predicted latency pattern. Animal studies in the literature using the same stimulus were then compared to determine the degree of relationship. The results show that each marking accounts for a large percentage of a corresponding peak latency in the peristimulus-time histogram. For each of the stimuli considered, the latency predicted by the Fourier pattern is highly correlated with the observed latency in the auditory nerve fiber of representative species. The results suggest that the hearing organ analyzes not only amplitude spectrum but also phase information in Fourier analysis, to distribute the specific spikes among auditory nerve fibers and within a single unit. This phase-encoding mechanism in Fourier analysis is proposed to be the common mechanism that, in the face of species differences in peripheral auditory hardware, accounts for the considerable similarities across species in their latency-by-frequency functions, in turn assuring optimal phase encoding across species. Also, the mechanism has the potential to improve phase encoding of cochlear implants

Hair cell loss and repair processes in mammalian vestibular sensory epithelia

The sensory hair cells of the inner ear transduce the mechanical stimuli involved in hearing and balance. Loss of hair cells has been thought to be irreversible in mammals and is a major cause of human deafness and vestibular disturbances. In this thesis, scanning electron microscopy (SEM), transmission electron microscopy (TEM), fluorescence microscopy and immunohistochemistry have been used to examine the progression of hair cell loss and subsequent repair processes in the vestibular sensory epithelia of guinea pigs following injury induced by ototoxic aminoglycoside gentamicin. It is found that hair cell recovery may occur in the mammalian vestibular tissues. Hair cell loss in the utricular maculae and cristae was apparent after chronic, systemic gentamicin treatment. With topical application of drug, the saccular macula was also affected. Damaged hair cells undergoing degeneration showed morphological features characteristic of apoptosis. The lost hair cells were replaced by expansion of supporting cells. In longer survival animals, SEM showed there were many cells with immature hair bundles within the areas where hair cells were originally lost. Thin sections of equivalent areas showed the presence of immature hair cells. This suggested that replacement of hair cells occurred spontaneously after gentamicin induced hair cell loss. Studies of this phenomenon over a period of 33 weeks after gentamicin treatment demonstrated that the immature hair cells continued to develop towards structural maturity. The number of hair bundles assessed by SEM of utricles and saccules showed that an initial decrease in number was followed by an increase, confirming recovery of hair cells. However, the hair cell density was lower than in controls, suggesting recovery was still incomplete. A technique for the maintenance of explants of the utricles and saccules from adult guinea pigs and gerbils in organotypic cultures was developed. Exposure of cultures to gentamicin resulted in progressive hair cell loss in a pattern similar to that seen in vivo. The degenerated hair cells again showed morphological features of apoptosis. In situ end labelling (ISEL) method was applied to detect apoptotic cells in the cultured utricles. Immature like hair cells were occasionally identified in cultures exposed to gentamicin. The results suggest that organotypic culture system of the mature mammalian vestibular epithelia is a useful model for examination of ototoxicity and recovery processes. Proliferative activity was identified in the damaged vestibular sensory tissues by immunohistochemical bromodeoxyuridine (BrdU) labelling, but the extent was insufficient to account for the recovery of hair cell numbers. Mechanisms other than proliferation may also be invoked after gentamicin induced hair cell loss in the vestibular sensory epithelia of the mammalian inner ear. Nevertheless, these results show that the potential for replacement of hair cells after loss may exist in the vestibular sensory organs of mature mammals

The endocannabinoid/cannabinoid receptor 2 system protects against cisplatin-induced hearing loss

Previous studies have demonstrated the presence of cannabinoid 2 receptor (CB2R) in the rat cochlea which was induced by cisplatin. In an organ of Corti-derived cell culture model, it was also shown that an agonist of the CB2R protected these cells against cisplatin-induced apoptosis. In the current study, we determined the distribution of CB2R in the mouse and rat cochleae and examined whether these receptors provide protection against cisplatin-induced hearing loss. In a knock-in mouse model expressing the CB2R tagged with green fluorescent protein, we show distribution of CB2R in the organ of Corti, stria vascularis, spiral ligament and spiral ganglion cells. A similar distribution of CB2R was observed in the rat cochlea using a polyclonal antibody against CB2R. Trans-tympanic administration of (2-methyl-1-propyl-1H-indol-3-yl)-1-naphthalenylmethanone (JWH015), a selective agonist of the CB2R, protected against cisplatin-induced hearing loss which was reversed by blockade of this receptor with 6-iodo-2-methyl-1-[2-(4-morpholinyl)ethyl]-1H-indol-3-yl](4-methoxyphenyl)methanone (AM630), an antagonist of CB2R. JWH015 also reduced the loss of outer hair cells (OHCs) in the organ of Corti, loss of inner hair cell (IHC) ribbon synapses and loss of Na+/K+-ATPase immunoreactivity in the stria vascularis. Administration of AM630 alone produced significant hearing loss (measured by auditory brainstem responses) which was not associated with loss of OHCs, but led to reductions in the levels of IHC ribbon synapses and strial Na+/K+-ATPase immunoreactivity. Furthermore, knock-down of CB2R by trans-tympanic administration of siRNA sensitized the cochlea to cisplatin-induced hearing loss at the low and middle frequencies. Hearing loss induced by cisplatin and AM630 in the rat was associated with increased expression of genes for oxidative stress and inflammatory proteins in the rat cochlea. In vitro studies indicate that JWH015 did not alter cisplatin-induced killing of cancer cells suggesting this agent could be safely used during cisplatin chemotherapy. These data unmask a protective role of the cochlear endocannabinoid/CB2R system which appears tonically active under normal conditions to preserve normal hearing. However, an exogenous agonist is needed to boost the activity of endocannabinoid/CB2R system for protection against a more traumatic cochlear insult, as observed with cisplatin administration.</p

Sound Encoding in the Mouse Cochlea: Molecular Physiology and Optogenetic Stimulation

Afferent synapses between inner hair cells (IHCs) and spiral ganglion neurons in the cochlea translate sound information into a discrete spike code, providing us the opportunity to directly observe the output of the cochlea. The availability of mutant strains with genetic hearing impairment makes the mouse a valuable species to investigate the molecular mechanisms of cochlear function. In this thesis, mouse was used as a model species to study cochlear sound encoding by recording single unit activities from auditory nerve fibers (ANFs) in vivo. First, developmental changes of ANF responses before and after hearing onset were characterized as an introduction on how normal ANF responses mature during the early postnatal age. Spontaneous bursting activity from ANFs/cochlear nucleus neurons was observed before hearing onset. After hearing onset, the average spontaneous and evoked spike rates of single ANFs increased, while tuning threshold and frequency selectivity improved between p14-15 to p20-21. To gain insight into the role of synaptic organization in cochlear and ANF function, mice carrying targeted mutations of presynaptic scaffold protein Bassoon were analyzed. IHCs of mice that are deficient of the central portion of the presynaptic scaffold protein Bassoon (BSNΔEx4/5) were previously shown to mostly lack synaptic ribbons and to have a smaller readily releasable pool of synaptic vesicles and reduced exocytosis, resulting in lower firing rates of ANFs. To distinguish better between the effects of the Bassoon mutation and those of the loss of the synaptic ribbon, the BSNΔEx4/5 phenotype was compared with that of a newly generated gene trap mutant (BSNgt), which has an intermediate phenotype in terms of the fraction of ribbon occupied active zones, presumably due to leaky expression of a small amount of Bassoon protein. The mean distance between the remaining ribbons and the active zone was greater in BSNgt than in wildtype and the synaptic calcium channel clusters had reduced immunostaining reactivity. The BSNgt IHCs showed a slightly less severe reduction of peak Ca2+ currents and sustained exocytosis compared to BSNΔEx4/5. However, IHC fast exocytosis and single unit responses of ANFs showed almost identical response properties between the two mutants. These data suggest that it is not the physical presence or absence of a synaptic ribbon but rather the disruption of presynaptic ultrastructure (e.g. abnormal calcium channel clustering, looser ribbon anchorage) that mainly determines the synaptic phenotype of Bassoon mutants.  Next, the ANF responses of Black Swiss mice (BLSW) were characterized. BLSW mice have inherited early onset sensorineural hearing loss and susceptibility to audiogenic seizures due to a mutation in the Gipc3 gene. BLSW ANFs showed higher tuning thresholds and broader frequency selectivity, which is consistent with a previous report of OHC dysfunction. Interestingly, BLSW ANFs had elevated spontaneous discharge activity, indicating that Gipc3 is a key molecular player not only for normal OHC but also for IHC function.  Upon hearing loss due to HC dysfunction, the remaining ANFs can be electrically stimulated to restore the sense of hearing. The number of useful frequency channels using electrical stimulation is limited by the spread of current. Focused optical stimulation may allow for more selective activation of ANFs compared to electrical stimulation. In the last part of the thesis, spiking activity was measured in response to laser light stimulation in ANFs and cochlear nucleus neurons of mice with constitutive expression of the light-gated ion channel Channelrhodopsin-2 and virus-mediated expression of the faster ChR2 variant CatCh.

An application of an auditory periphery model in speaker identification

The number of applications of automatic Speaker Identification (SID) is growing due to the advanced technologies for secure access and authentication in services and devices. In 2016, in a study, the Cascade of Asymmetric Resonators with Fast Acting Compression (CAR FAC) cochlear model achieved the best performance among seven recent cochlear models to fit a set of human auditory physiological data. Motivated by the performance of the CAR-FAC, I apply this cochlear model in an SID task for the first time to produce a similar performance to a human auditory system. This thesis investigates the potential of the CAR-FAC model in an SID task. I investigate the capability of the CAR-FAC in text-dependent and text-independent SID tasks. This thesis also investigates contributions of different parameters, nonlinearities, and stages of the CAR-FAC that enhance SID accuracy. The performance of the CAR-FAC is compared with another recent cochlear model called the Auditory Nerve (AN) model. In addition, three FFT-based auditory features – Mel frequency Cepstral Coefficient (MFCC), Frequency Domain Linear Prediction (FDLP), and Gammatone Frequency Cepstral Coefficient (GFCC), are also included to compare their performance with cochlear features. This comparison allows me to investigate a better front-end for a noise-robust SID system. Three different statistical classifiers: a Gaussian Mixture Model with Universal Background Model (GMM-UBM), a Support Vector Machine (SVM), and an I-vector were used to evaluate the performance. These statistical classifiers allow me to investigate nonlinearities in the cochlear front-ends. The performance is evaluated under clean and noisy conditions for a wide range of noise levels. Techniques to improve the performance of a cochlear algorithm are also investigated in this thesis. It was found that the application of a cube root and DCT on cochlear output enhances the SID accuracy substantially