Philippsen A, Wrede B. Towards Multimodal Perception and Semantic Understanding in a Developmental Model of Speech Acquisition. Presented at the 2nd Workshop on Language Learning at Intern. Conf. on Development and Learning (ICDL-Epirob) 2017, Lisbon

Philippsen, Anja 

Wrede, Britta

Publications at Bielefeld University

1Towards Multimodal Perception and SemanticUnderstanding in a Developmental Model ofSpeech AcquisitionAnja Kristina Philippsen and Britta WredeApplied Informatics Group, Cognitive Interaction Technology Center (CITEC), Bielefeld University{aphilipp, bwrede}@techfak.uni-bielefeld.deAbstract—Babbling is a crucial process in young infants foracquiring articulatory control. By constantly trying out motorcommands and observing the consequences that they causein the environment, they develop an understanding of theirown body and learn to control their articulators in order toproduce meaningful speech. In earlier works, we proposed adevelopmental model of speech acquisition that learns to controla 3-d vocal tract simulation for producing vowel or syllablesounds. The system self-organizes learning according to speechit perceives from its environment, so that ambient speech shapesthe learning process. Here, we discuss how the proposed modelcould be extended to form a bridge between perception, on theone end, and semantics, on the other end. The idea is to connectacoustic perception with other perceptual modalities. As anexample, we discuss how the system could integrate visual inputin its learning loop. By learning associations between acoustictargets and simultaneous visual perceptions, such an enhancedmodel could produce speech not only in reaction to acousticinput, but also triggered by visual input. Vision, thus, couldhelp to establish a common ground between learner and tutorfor interactive articulatory learning.I. A DEVELOPMENTAL MODEL OF SPEECH ACQUISITIONInfants explore in a goal-directed manner from the verybeginning. Studies show that even neonates orient themselvestowards more interesting targets [1] and prenatal exposureto their native language seems to influence infants’ earlybabbling behavior (e.g. [2]).We combined these ideas in a developmental model ofspeech acquisition in which we model the influence ofambient speech on the learning process [3]. We provide a setof speech sounds to the system from which it extracts the im-portant components from a high-dimensional acoustic spacerepresentation via linear discriminant analysis. The resulting2-d goal space forms a low-dimensional representation ofspeech that the system is exposed to in its environment.Through this dimension reduction, full syllables are projectedonto a single point in goal space. Fig. 1 depicts such a goalspace trained from ambient speech sound sets consisting ofthe three syllables /a/, /ba/ and /ma/.This research has been supported by the Cluster of Excellence CognitiveInteraction Technology ’CITEC’ (EXC 277) at Bielefeld University, whichis funded by the German Research Foundation (DFG), and is related to theEuropean Project CODEFROR (FP7-PIRSES-2013-612555)´1 0 1´1´0.500.51 abamaFig. 1. A goal space for the three syllables /a/, /ba/ and /ma/.ArticulatoryParametersGoalSpaceForward ModelInverseModelSpeech Sounds GenerationPerceptionFig. 2. Forward and inverse model map between articulatory parametersand points in the low-dimensional goal spaceUsing this goal space, we employ goal babbling, a develop-mental approach to learning motor coordination [4], [5]. Goalbabbling learns by drawing targets from the goal space andtrying to achieve them. Collected action–outcome pairs areused to learn an inverse model, mapping from target positionsin goal space to motor commands (see Fig. 2). Whereasthe inverse model is updated with a radial basis functionneural network in an online fashion, the forward model isavailable to the system before babbling starts. Executing theforward model includes two steps. In the generation step, thearticulatory parameters are fed into the speech synthesizerand a speech sound is produced. In the perception step,this speech sound is “perceived”, i.e. it is mapped to thepreviously trained goal space. This perception step modelsthat learning is shaped by ambient language exposure.II. BABBLING A SET OF SYLLABLESMotor commands are either static vocal tract shapes (forvowels), or articulatory trajectories modelled with dynamic20 100 200 300 400 5000.20.40.60.81competence abamaFig. 3. Competence increase during babbling /a/, /ba/ and /ma/ for 500iterations (using GBFB features).movement primitives (for syllables). Because the dimensionreduction step extracts a low-dimensional representation,arbitrary high-dimensional acoustic features can be applied,e.g. MFCCs or Gabor Filterbank (GBFB) Features [6].In the beginning, the inverse model only knows a singlespeech sound (in the example in Fig. 3 this is /a/). Thenbabbling proceeds: the system draws targets randomly fromthe distribution of ambient speech sounds, estimates therequired motor command by executing the inverse model,adds exploratory noise and observes the actual outcomes byexecuting the forward model. Then, the inverse model getsupdated with the new action–outcome pairs (see [3]).In this way, the system gradually increases its competence.We measure the system’s competence by testing how wellit can produce the three target sounds that are present inambient speech. Competence is computed as the exponentialof the negative distance between the desired and the actuallyachieved point in goal space.Fig. 3 shows how the competence increases during 500babbling iterations for learning /a/, /ba/ and /ma/.III. LINKING ACOUSTIC AND VISUAL GOAL SPACEAfter learning, our model is capable to produce speechsounds in two ways, namely via imitation or via self-production. Imitation means that the system perceives an(external) speech sound, maps it to its goal space and triesto achieve this point in goal space by executing its inverseand forward model in a row. Self-production means that thesystem selects its own target in goal space. Computationally,this target could be drawn randomly from the distribution ofambient speech. Such spontaneous productions might occurin young infants from time to time, however, human speechproduction is rarely purely random.The reason is that we learn how to speak in order to interactwith and learn about our environment. Speech sounds are as-sociated with visual impressions, tactile feedback, emotionalstates and various other sensations. Semantic emerges froman association of speech with multimodal experiences [7].How could multimodal perception be integrated into ourspeech acquisition model? Analogously to how the acousticgoal space forms a low-dimensional representation of ambi-ent speech, a dimension reduction on the visual perceptionscould generate a visual goal space. As visual and acousticperception occur in parallel, points in both goal spacesare activated simultaneously, so Hebbian learning, whichAmbientspeechVisualworldAcousticgoal spaceVisualgoal spaceembedembedHebbianlearningInverse modelForward modelArticulation... ...Fig. 4. Associations between visual and acoustic perceptions learned viaHebbian learning could extend the model towards multimodel perceptionand, eventually, towards a better representation of semantics.strengthens connections between co-occurring events, couldbe applied to learn associations between these two modalities(similar to applications in e.g. [8], [9]).In consequence, the resulting system could produce speechsounds not only imitatively or randomly, but also in reactionto a visual percept. Either the visual percept activates associ-ated acoustics which then can serve as an imitation target. Orthe visual percept is directly inputted into the inverse model.Furthermore, such multimodal information could alleviatethe correspondence problem, i.e. the problem of mappingbetween speech sounds produced by different individuals.Currently, this problem is avoided by generating ambientspeech via the same vocal tract model that the learner uses forbabbling. Goal spaces in additional modalities such as visionare not affected by speaker variations and could help to builda common ground between learner and tutor. Multimodalassociations, thus, could extend our system towards theintegration of semantic information, eventually connectingarticulatory learning to interactive language learning.REFERENCES[1] C. Von Hofsten, “An action perspective on motor development,” Trendsin cognitive sciences, vol. 8, no. 6, pp. 266–272, 2004.[2] A. J. DeCasper and M. J. Spence, “Prenatal maternal speech influencesnewborns’ perception of speech sounds,” Infant behavior and Develop-ment, vol. 9, no. 2, pp. 133–150, 1986.[3] A. K. Philippsen, R. F. Reinhart, and B. Wrede, “Goal babbling ofacoustic-articulatory models with adaptive exploration noise,” in IEEEIntern. Conf. on Development and Learning, 2016.[4] M. Rolf, J. J. Steil, and M. Gienger, “Goal babbling permits directlearning of inverse kinematics,” IEEE Transactions on AutonomousMental Development, vol. 2, no. 3, pp. 216–229, 2010.[5] A. Baranes and P.-Y. Oudeyer, “Active learning of inverse modelswith intrinsically motivated goal exploration in robots,” Robotics andAutonomous Systems, vol. 61, no. 1, pp. 49–73, 2013.[6] M. R. Scha¨dler, B. T. Meyer, and B. Kollmeier, “Spectro-temporalmodulation subspace-spanning filter bank features for robust automaticspeech recognition,” The Journal of the Acoustical Society of America,vol. 131, no. 5, pp. 4134–4151, 2012.[7] L. Fernandino, J. R. Binder, R. H. Desai, S. L. Pendl, C. J. Humphries,W. L. Gross, L. L. Conant, and M. S. Seidenberg, “Concept repre-sentation reflects multimodal abstraction: A framework for embodiedsemantics,” Cerebral Cortex, vol. 26, no. 5, pp. 2018–2034, 2015.[8] V. R. De Sa and D. H. Ballard, “Category learning through multimodal-ity sensing,” Neural Computation, vol. 10, no. 5, pp. 1097–1117, 1998.[9] B. Golosio, A. Cangelosi, O. Gamotina, and G. L. Masala, “A cognitiveneural architecture able to learn and communicate through naturallanguage,” PloS one, vol. 10, no. 11, p. e0140866, 2015.

Towards Multimodal Perception and Semantic Understanding in a Developmental Model of Speech Acquisition

https://pub.uni-bielefeld.de/download/2914268/2914269

Towards Multimodal Perception and Semantic Understanding in a Developmental Model of Speech Acquisition

Abstract

Similar works

Full text

Available Versions

Publications at Bielefeld University