Botteldooren, Dick

De Baets, Bernard

De Coensel, Bert

Oldoni, Damiano

Rademaker, Michaël

Van Renterghem, Timothy

English

Ghent University Academic Bibliography

Computational soundscape analysis based ona human-like auditory processing modelDamiano OLDONIa, Bert DE COENSELa, Michael RADEMAKERb,Timothy VAN RENTERGHEMa, Dick BOTTELDOORENa and Bernard DE BAETSbaDepartment of Information Technology, Ghent University,St.-Pietersnieuwstraat 41, B-9000 Ghent, BELGIUM, e-mail: damiano.oldoni@intec.ugent.bebDepartment of Applied Mathematics, Biometrics and Process Control, Ghent University,Coupure Links 653, B-9000 Ghent, BELGIUMABSTRACTIn the context of computational soundscape analysis, few attempts have been made to imitate thehuman approach to soundscape perception. This paper introduces a biologically plausible andhuman-like auditory processing model for general soundscape analysis. The model is based ontwo types of neural networks. The first is a Self-Organizing Map (SOM) that models the plas-ticity during learning and the complex morphology of the human auditory cortex and allows torecognize co-occurring sound features. The second is a Locally-Excitatory-Globally-Inhibitory-Oscillator-Network (LEGION) simulating the oscillatory correlation property of neurons of theauditory cortex. The model is shown to be highly context dependent. The potential of SOMin soundscape analysis and its use as a sound classifier are extensively studied. The proposedmethodology can easily distinguish between common and rare sounds of a given soundscape,thus drawing up its acoustic summary.1. INTRODUCTIONSoundscape analysis is a quite new resesarch field whose computational counterpart has sofar remained largely unexplored. The reasons are manifold, but most of them stem from thedifficulty to reproduce the human psychological mechanisms that allow labelling a sound astypical or atypical for the given context. Some models that do not draw on humans mimickingtechniques for sound recognition tasks, aim to provide generic environmental sound recognition,but they do not attempt to provide soundscape description. The psychoacoustical processesunderlying the detection or the recognition of a sound are governed by numerous psychologicalvariables, i.e. sound context perception, attention level and habituation.In this paper a biologically and psychologically plausible model for computational soundscapeanalysis is proposed. It is based on a combination of two types of neural networks that imitatedifferent stages of human auditory processing. Factors like context dependency and learningfrom experience are taken into account.2. METHODOLOGYOverviewThe model starts by extracting sound features encoding loudness and spectro-temporal irreg-ularities from standard 1/3-octave band levels. Contrary to other methods focused merely onsound recognition and deploying complicated tone detectors by means of harmonic analysis,such sound features are designed to extract only the information useful for potentially trigger-ing the so-called bottom-up attention [1]. Subsequently, the sounds forming the soundscapeare discerned using a combination of two types of neural networks: a Self-Organizing Map(SOM) [2] that allows—after extensive training—to identify co-occurring sound features anda Locally Excitatory Globally Inhibitory Oscillator Network (LEGION) [3] for grouping andsegregation of corresponding sound feature clusters. Contrary to [4], here we dynamically traina SOM in order to imitate human continuous learning. After an initial static learning phase, anew learning phase is triggered whenever a bad matching between the SOM and the new soundfeature vectors occurs.The structure of the SOM resembles for certain aspects the complex morphology of the hu-man auditory cortex. The combination of SOM and LEGION models two essential features ofthe brain: the SOM mimics the plasticity and the learning abilities of the network of neuronsforming the auditory cortex, while the LEGION approximates the dynamic oscillations betweenconnected neurons. This combines part of the auditory experience into “a sound”.Sound feature extractionThe model starts from the 1/3-octave band spectrum of the sound pressure level with temporalresolutions of 1 s. The energetic masking is the first psychoacoustical phenomenon to be takeninto account: it is simulated calculating a simplified cochleagram s( f , t) via the Zwicker loud-ness model [5] with a frequency range from 0 to 24 Bark and a frequency resolution of 0.5 Barkfor a total of 48 frequency values.The second step highlights the loudness changes and spectro-temporal irregularities embeddedin the sound signal. This is done by convolving the cochleagram with various 2D gaussian anddifference-of-gaussian filters: the gaussian filters are suitable for intensity encoding, the latterfor spectral and temporal contrast. In particular 16 different scales are used: 4 for intensity, 6for spectral and temporal contrast each. By means of convolution, 16× 48 = 768 values areextracted at each timestep; such values can be seen as a vector called sound feature vector [4].From now on we will work only with such vectors and not with the sound spectrogram.SOM: sound context and learningThe Self-Organizing Map (SOM), called also Kohonen map [2], is a neural network often usedas a nonlinear dimension reduction technique. The network is composed of several nodes placedin a 2D grid, usually forming an hexagonal lattice. Each node has a corresponding referencevector which represents the node position in the high-dimensional sound feature space. Af-ter initialization, the training can modify the position of the reference vectors by means of thesound feature vectors calculated from the initial 1/3-octave band spectrogram as previously ex-plained. The training is a vast number of iterations of a 2-step algorithm. The first step of thealgorithm is to feed the SOM with an input sound feature vector and determine its nearest refer-ence vector, whose corresponding node is called best matching unit (BMU). In the second stepthe reference vector of the BMU and, to a lesser extent the reference vectors of its neighbours,are moved closer to the position of the input sound feature vector. For more details about theSOM mathematical model in the current context see Oldoni [4].After this training, the reference vectors of the SOM units provide a nonlinear projection of theprobability density function of the input data. Furthermore feature vectors occurring very oftenduring the training phase will be very close to the corresponding BMUs and are said to be wellrecognized. However, sound feature vectors that occur rarely are usually not well representedby the trained SOM, although they are also important for a comprehensive and more realisticsoundscape representation. To address this problem, dynamic learning can be adopted: duringSOM operation, a new learning phase is initiated when the distance of the input sound featurevector is greater than an activation threshold, T1, and it continues until the distance is less thana disactivation threshold T2, with T2 < T1. After such training the updated SOM is used until anext dynamic training session is triggered.The learning parameters and the initial number of degrees of freedom (number of nodes) inthe SOM have to be chosen carefully in order to not lose information from previous trainingsessions. This kind of dynamic learning aims to reproduce the continuous human learningbased on experience. An alternative method, not yet tested, could be the insertion of new nodesto enhance learning and avoid any loss of previously acquired information.Usually a SOM trained on a certain soundscape cannot match typical sounds of other sound-scapes, due to the innate context dependency of the training phase. However, applying thedynamic learning strategy, a SOM can improve its matching power to new sounds without los-ing information about what was previously learned.LEGION: synchronization and segregationDuring the last thirty years oscillatory correlation properties of sensory cortical neurons were in-tensively studied [6] and in particular evidence of synchronous oscillation was discovered in theauditory cortical cortex [7, 8]. Based on the work of von der Malsburg and Schneider [9], Wangdeveloped a computational model that is called shifting synchronization theory [10]. The mainpoint of the oscillatory correlation findings and Wang’s model is to represent specific sound fea-tures as a synchronized group of neurons. If sounds with different features are contemporarilypresent, the two groups of neurons are internally synchronized but desynchronization betweenthe two groups occurs. This idea is at the base of LEGION [3], the network of oscillators usedby Wang.LEGION is usually composed of a 2D grid of oscillators: in our coupled SOM-LEGION modelthere is a one-to-one correspondence between the nodes of SOM and the oscillators of LEGION.The dynamics of the single oscillator are simple: if there is no external excitation, the oscillatorfalls in a fixed stable point, whereas, if the external excitation is greater than zero, the oscillatormoves in a stable limit cycle, alternating between a so-called active phase and a silent phase.There are also 2 different coupling terms: the local coupling between each oscillator and itsneighbours and the coupling with an external global inhibitory source that is active when atleast one oscillator is in the active phase. These two couplings are responsible for the oscillatorysynchronization within each group of stimulated oscillators and of the desynchronization amongdifferent groups (for more details see the exhaustive description of the LEGION model [11]).Some features of the SOM-LEGION coupling should be kept in mind: there is a one-to-one cor-respondence between the nodes of SOM and the oscillators of LEGION; a LEGION oscillatorreceives a positive external stimulation only if the distance of the corresponding SOM node tothe input sound feature vector is less than a fixed threshold; each time step a new sound featurevector is provided resulting in a time-dependent external stimulation.3. RESULTSThis model was tested on two different soundscapes: an urban environment with a mixture ofquiet, light and occasionally heavy traffic noise, labelled as T, and a botanic garden, with anonly limited human presence, labelled as P. The soundscape for each location was monitoredby two measurements stations, at 55 m from each other, which recorded standard 1/3-octaveband levels at time interval of 1 s. Two SOMs, one for each soundscape, were trained with setsof sound feature vectors corresponding to an entire day, that is 86400 samples. It is clear thatsuch training produces sound context-dependent maps. This can be seen in Fig. 1 in which theSOM trained in the botanic garden does not recognize a 1 s sample extracted from the passageof a car in T. Like humans learn from experience, dynamic learning based on data from T can543210(d)(c)(b)(a)Figure 1. Distance of the sound feature vector related to a typical sample from P (quiteness) and theunits in the SOM trained in (a) T, (b) P. Distance of the sound feature vector related to a typicalsample from T (during the passage of a car) and the units in the SOM trained in (c) T, (d) P.improve the ability of the SOM previously trained in P to match urban sounds. A comparisonbetween the distance to the BMU over one hour of input vectors (3600 samples) from T withoutor with dynamic learning shows such improvement (see Fig. 2 and Tab. 1). Learning new soundfeatures could introduce a sort of forgetting term reducing the performance on samples from theprevious soundscape. If the parameters controlling the dynamic learning are chosen correctlysuch phenomenon does not produce significant collateral effects, as normally occurs in humans.The LEGION oscillators corresponding to the SOM nodes, whose distance to an input featurevector is less than a certain threshold, receive an external excitation. The (de)synchronizationproperties of LEGION are showed in Fig. 3. The change of the external stimulation in timecan result in a transient in which the oscillators recombine their dynamic connection weightsto adapt themselves to the new external input (see Fig. 3 at t = 4.2 s). However, such phasedisruption is not present if the external stimulation is a quasistatic function. In such casesstream formation can be observed.0 500 1000 1500 2000 2500 3000 350001020304050Distance to the BMU  0 500 1000 1500 2000 2500 3000 350005101520SamplesDistance to the BMU  with dynamic learningwithout dynamic learningwithout dynamic learningwith dynamic learningFigure 2. Top: distance before (blue) and after (green) dynamic learning of the BMU of SOM initiallytrained on P to 3600 samples (1 hour) input sound feature vectors taken from T. Bottom:distance before (blue) and after (green) dynamic learning of the BMU of SOM initially trainedon P to 3600 samples (1 hour) input sound feature vectors taken from P. In both cases dynamiclearning is carried out using 86400 samples (1 day) from T.Table 1. Effects of dynamic learning on matching. Dynamic learning is carried out using 86400 samples(1 day) from T: it can improve significantly the matching of the SOM previously trained in Pwithout strongly affecting the ability to match sounds from the same location.Initial traininglocationDynamic learning Average distance to the BMUinput samples from T input samples from PP no 5.76 1.19yes 2.60 1.53T no 1.94 1.87t = 3 t = 3.05 t = 3.20 t = 3.24t = 4 t = 4.20 t = 4.24 t = 4.36 t = 4.44Figure 3. Left (2 columns): similarity (inverse of the distance) of two input samples at t = 3 s (top) andt = 4 s (bottom), before (1st column) and after (2nd column) binarization (λ = 0.92, movingaverage order: h = 3, see [4]). Right (4 columns): some snapshots of LEGION taken atdifferent times. The samples used here are extracted from test input data recorded in scenarioP previously used for SOM testing.4. CONCLUSIONS AND PERSPECTIVESA computational model for soundscape analysis is constructed based on human auditory pro-cessing and pyschological aspects as context dependency and learning from experience. Soundfeatures related to the degree of novelty of the sound are calcuated at each time step. Basedon such sound features, a self-organizing map, SOM, can learn which combinations of featuresare typical for the soundscape of a particular location by means of extensive training. In thispaper a new dynamic learning algorhithm is described and applied to enhance the ability of theSOM to match more sounds from the same location as time goes on or to adapt its operationto other locations. Next, LEGION imitates the oscillatory (de)synchronization properties ofexcited neurons. Conceptually the SOM unit and the LEGION oscillator can be considered thesame formal neural unit. The coupling model represents two different functionalities of idealneurons: the memory formation, modeled by SOM training and the dynamic oscillatory corre-lation of sensory cortex neurons excited by an auditory stimulus, schematized by LEGION.The presence of a transient in LEGION prohibits continuous stream formation. However, thetransient is sensibly reduced or even removed if the region of oscillators that are externallystimulated changes only slowly in time. This could be achieved using a shorter time step in thesound feature extraction and a lifelong dynamic learning of the SOM. Furthermore, attentionalmechanisms can be simulated at the LEGION level: attention to a particular type of soundcould be simulated by a mask enabling only the oscillators in a precise region of the network tooscillate.ACKNOWLEDGMENTSBert De Coensel is a postdoctoral fellow of the Research Foundation – Flanders; the supportof this organisation is gratefully acknowledged. This work was supported in part by the IWTVlaanderen Project IDEA (IWT-080054) and FWO.REFERENCES[1] E. I. Knudsen, “Fundamental components of attention,” Annu. Rev. Neurosci., vol. 30, pp. 57–78, 2007.[2] T. Kohonen, Self-Organizing Maps. Heidelberg, Germany: Springer-Verlag, 3rd ed., 2001.[3] D. Wang and D. Terman, “Locally excitatory globally inhibitory oscillator networks,” IEEE Transactions onNeural Networks, vol. 6, no. 1, pp. 283–286, 1995.[4] D. Oldoni, B. De Coensel, M. Rademaker, T. Van Renterghem, B. De Baets and D. Botteldooren, “Context-dependent environmental sound monitoring using som coupled with legion,” in Proceedings of the IEEEWorld Congress on Computational Intelligence, (Barcelona, Spain), July 2010.[5] E. Zwicker and H. Fastl, Psychoacoustics. Facts and Models. No. 22 in Springer Series in InformationSciences, Berlin, Germany: Springer-Verlag, 2nd ed., 1999.[6] C. M. Gray and W. Singer, “Stimulus-specific neuronal oscillations in orientation columns of cat visualcortex,” Proc. Natl. Acad. Sci. USA, vol. 86, no. 5, pp. 1698–702, 1989.[7] S. M. R. Galambos and P. J. Talmachoff, “A 40-hz auditory potential recorded from human scalp,” in Proc.Natl. Acad. Sci. USA, vol. 78, pp. 2643–2647, 1981.[8] E. B. M. Brosch and H. Scheich, “Stimulus-related gamma oscillations in primate auditory cortex,” Journalof neurophysiology, vol. 87, no. 6, p. 2715, 2002.[9] C. von der Malsburg, “The correlation theory of the brain function,” Internal Report 81-2, Max-Planck-Institute for Biophysical Chemistry, 1981.[10] D. Wang, “Primitive auditory segregation based on oscillatory correlation,” Cognit. Sci., vol. 20, no. 3,pp. 409–456, 1996.[11] D. Wang, “Auditory stream segregation based on oscillatory correlation,” in Neural Networks for SignalProcessing [1994] IV. Proceedings of the 1994 IEEE Workshop, (Ermioni, Greece), pp. 624–632, Sept. 1994.     

Computational soundscape analysis based on a human-like auditory processing model

https://biblio.ugent.be/publication/1180461/file/1180464.pdf

Computational soundscape analysis based on a human-like auditory processing model

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography