Gesture-speech temporal integration in language development

In everyday interactions, speakers integrate gestures and speech sounds at a temporal level. One of the linguistic functions of temporally synchronous gesture-speech combinations is to provide prominence to specific parts of a discourse. While a bulk of evidence has explored the gesture-speech co-expressiveness at a semantic level, little is known about the children’s ability to use synchronized gestural and prosodic prominences in the benefit of language. This PhD thesis investigates gesture-speech temporal integration abilities in development and its beneficial impact for children’s language. The dissertation includes three independent studies at different time points in development, each one described in one chapter. The first two studies aim at investigating the role of perceiving gesture-speech temporal synchronizations functioning as markers of prominence, and its linkage to language abilities. First, a study investigated whether three- to- five- year- old children responded better to a word recall task when the word was presented with a contrast of prominence expressed with a synchronous beat gesture (i.e., a hand gesture synchronized with prominence in speech). The results indicated a beneficial local effect of the beat gesture on the recall of the temporally synchronous word. Second, a study examined whether six- to- eight- year-old children processed pragmatic inferences online more rapidly when the relevant information was presented together with a beat gesture. Additionally, this study investigated whether these potential benefits were due to the prominence expressed in the gesture or to its concomitant prosodic prominence. Results showed that children’s processing of a pragmatic inference was improved by both prosodic and beat gesture prominence contributions to the discourse. The last study focused on the predictive role of the first infant’s uses of temporally synchronous gesture-speech combinations on later language development. To do so, a longitudinal study correlated the infants’ production of synchronous pointing gesture-speech combinations during controlled socio-communicative interactions at 12 months with linguistic measures at 18 months. Results demonstrated that synchronous productions positively correlated with lexical and grammatical development at 18 months of age. Overall, the three studies show evidence that infant's synchronous gesture-speech abilities (a) function as multimodal markers of prominence; (b) when perceived in a discourse context synchronies have positive impact on children’s word recall (Study 1) and pragmatic inference resolution (Study 2); and (c) infants’ first productions of synchronous gesture-speech combinations serve a communicative strategy which is correlated to later language abilities (Study 3). The findings of the studies presented in this thesis point out the importance of synchronous gesture-speech combinations in highlighting information, as well as their beneficial effects in language acquisition

Multimodal Strategies in Development./nPredictive Value of early simultaneos/ngesture-speech comination

The present study investigates the predictive value of the early appearance of simultaneous pointing-speech combinations. An experimental task was used to obtain a communicative productive sample from nineteen children at 1;0 and 1;3. Infant’s communicative productions, in combination with gaze joint engagement patterns, were analyzed in relation to different social conditions. The results show a significant effect of age and social condition on infants’ communicative productions. Gesture-speech combinations seem to work as a strong communicative resource to attract the adult’s attention in social demanding communicative contexts. Gaze joint engagement was used in combination with simultaneous pointing-speech combinations to attract adults’ attention during social demanding conditions. Finally, the use of simultaneous pointing-speech combinations at 1;0 in demanding conditions predicted greater expressive vocabulary acquisition at 1;3 and 1;6. These results indicate that the use of gesture-speech combinations may be considered a significant step towards the early integration of language components

Multimodal Strategies in Development./nPredictive Value of early simultaneos/ngesture-speech comination

The present study investigates the predictive value of the early appearance of simultaneous pointing-speech combinations. An experimental task was used to obtain a communicative productive sample from nineteen children at 1;0 and 1;3. Infant’s communicative productions, in combination with gaze joint engagement patterns, were analyzed in relation to different social conditions. The results show a significant effect of age and social condition on infants’ communicative productions. Gesture-speech combinations seem to work as a strong communicative resource to attract the adult’s attention in social demanding communicative contexts. Gaze joint engagement was used in combination with simultaneous pointing-speech combinations to attract adults’ attention during social demanding conditions. Finally, the use of simultaneous pointing-speech combinations at 1;0 in demanding conditions predicted greater expressive vocabulary acquisition at 1;3 and 1;6. These results indicate that the use of gesture-speech combinations may be considered a significant step towards the early integration of language components

Language development at 18 months is related to communicative strategies at 12 months

The present study investigated the degree to which an infants’ use of simultaneous gesture–speech combinations during controlled social interactions predicts later language development. Nineteen infants participated in a declarative pointing task involving three different social conditions: two experimental conditions (a) available, when the adult was visually attending to the infant but did not attend to the object of reference jointly with the child, and (b) unavailable, when the adult was not visually attending to neither the infant nor the object; and (c) a baseline condition, when the adult jointly engaged with the infant's object of reference. At 12 months of age measures related to infants’ speech-only productions, pointing-only gestures, and simultaneous pointing–speech combinations were obtained in each of the three social conditions. Each child's lexical and grammatical output was assessed at 18 months of age through parental report. Results revealed a significant interaction between social condition and type of communicative production. Specifically, only simultaneous pointing–speech combinations increased in frequency during the available condition compared to baseline, while no differences were found for speech-only and pointing-only productions. Moreover, simultaneous pointing–speech combinations in the available condition at 12 months positively correlated with lexical and grammatical development at 18 months of age. The ability to selectively use this multimodal communicative strategy to engage the adult in joint attention by drawing his attention toward an unseen event or object reveals 12-month-olds’ clear understanding of referential cues that are relevant for language development. This strategy to successfully initiate and maintain joint attention is related to language development as it increases learning opportunities from social interactions.This research was funded by a Recercaixa 2013–2015 project to Pilar Prieto and two projects from the Spanish MINECO (BFU2012-31995 to Pilar Prieto and PSI-2011-25376 to Laura Bosch). We would like to thank all GrEP and APAL Lab members for their support and comments to earlier versions of this paper, and especially Jorgina Solé for help in sample recruitment and testing assistance, and Joan Borràs-Comes for his help with the statistical analysis of the data. We are grateful to all the subjects and their families for having participated in this longitudinal study

Observing storytellers who use rhythmic beat gestures improves children’s narrative discourse performance

Iconic and pointing gestures are important precursors of children’s early language and cognitive development. While beat gestures seem to have positive effects on the recall of information by preschoolers, little is known about the potential beneficial effects of observing beat gestures on the development of children’s narrative performance. We tested forty-four 5- and 6-year-old children in a between-subject study with a pretest–posttest design. Following a pretest in which they were asked to retell the story of an animated cartoon they had watched, the children were exposed to a training session in which they observed an adult telling a total of six one-minute stories under two between-subject experimental conditions: 1) a no-beat condition, where focal elements in the narratives were not highlighted by means of beat gestures; and 2) a beat condition, in which focal elements were highlighted by beat gestures. Following the training session, a posttest was administered following the same procedure as the pretest. Narrative structure scores were independently coded from recordings of the pretest and posttest and subjected to statistical comparisons. The results revealed that children who were exposed to the beat condition showed a higher gain in narrative structure scores. This study thus shows for the first time that a brief training session with beat gestures has immediate benefits for children’s narrative discourse performance.The study “Observing storytellers who use rhythmic beat gestures improves children’s narrative discourse performance” obtained ethics approval from the ethics committee at the Universitat Pompeu Fabra, as part of the approval of the Spanish Ministry of Economy and Competitiveness project FFI2015-66533-P “Intonational and gestural meaning in language”

Gesture as a facilitator and precursor of language development

Els estudis sobre desenvolupament del llenguatge típicament s'han centrat en l'anàlisi de la parla. Tanmateix, hi ha altres elements comunicatius que són una peça fonamental en la comunicació i cognició humanes: els gestos. Aquests elements visuals estan íntimament integrats amb la parla des del punt de vista temporal i des del punt de vista semanticopragmàtic, i fan que es pugui parlar del llenguatge i la comunicació com uns fets multimodal. En aquest article posem a l'abast dels lectors un resum dels estudis més recents sobre adquisició del llenguatge des d'una perspectiva multimodal, tot i emfasitzant el rol dels gestos com a facilitadors i precursors del llenguatge no només en les etapes més primerenques sinó també en etapes de desenvolupament pragmàtic més tardà. Els estudis demostren que el gest és un component central de l'adquisició del llenguatge i que actua com a precursor i predictor de l’aprenentatge del vocabulari, de la sintaxi, així com del desenvolupament pragmàtic i discursiu. Finalment apuntem les àrees de recerca més actuals i innovadores i el possible impacte d'aquesta recerca en l'àmbit de la rehabilitació del llenguatge.Los estudios sobre desarrollo del lenguaje típicamente se han centrado en el análisis del habla. Sin embargo, hay otros elementos comunicativos que son una pieza fundamental en la comunicación y cognición humanas: los gestos. Estos elementos visuales están íntimamente integrados con el habla desde el punto de vista temporal y desde el punto de vista semanticopragmàtic, y hacen que se pueda hablar del lenguaje y la comunicación como unos hechos multimodales. En este artículo ponemos al alcance de los lectores un resumen de los estudios más recientes sobre adquisición del lenguaje desde una perspectiva multimodal, aunque enfatizando el rol de los gestos como facilitadores y precursores del lenguaje no sólo en las etapas más tempranas sino también en etapas de desarrollo pragmático más tardío. Los estudios demuestran que el gesto es un componente central de la adquisición del lenguaje y que actúa como precursor y predictor del aprendizaje del vocabulario, de la sintaxis, así como del desarrollo pragmático y discursivo. Finalmente apuntamos las áreas de investigación más actuales e innovadoras y el posible impacto de esta investigación en el ámbito de la rehabilitación del lenguaje.Language development studies have typically focused on speech analysis. However, there are other communicative elements that are a fundamental piece in human communication and cognition: gestures. These visual elements are intimately integrated with speech from a temporal point of view and from a semanticopragmatic point of view, and make it possible to speak of language and communication as multimodal facts. In this article we provide readers with a summary of the most recent studies on language acquisition from a multimodal perspective, although emphasizing the role of gestures as facilitators and precursors of language not only in the earliest stages but also in later stages of pragmatic development. Studies show that gesture is a central component of language acquisition and that it acts as a precursor and predictor of vocabulary learning, syntax, pragmatic and discursive development. Finally, we point out the most current and innovative areas of research and the possible impact of this research in the field of language rehabilitation

Language development at 18 months is related to communicative strategies at 12 months

The present study investigated the degree to which an infants’ use of simultaneous gesture–speech combinations during controlled social interactions predicts later language development. Nineteen infants participated in a declarative pointing task involving three different social conditions: two experimental conditions (a) available, when the adult was visually attending to the infant but did not attend to the object of reference jointly with the child, and (b) unavailable, when the adult was not visually attending to neither the infant nor the object; and (c) a baseline condition, when the adult jointly engaged with the infant's object of reference. At 12 months of age measures related to infants’ speech-only productions, pointing-only gestures, and simultaneous pointing–speech combinations were obtained in each of the three social conditions. Each child's lexical and grammatical output was assessed at 18 months of age through parental report. Results revealed a significant interaction between social condition and type of communicative production. Specifically, only simultaneous pointing–speech combinations increased in frequency during the available condition compared to baseline, while no differences were found for speech-only and pointing-only productions. Moreover, simultaneous pointing–speech combinations in the available condition at 12 months positively correlated with lexical and grammatical development at 18 months of age. The ability to selectively use this multimodal communicative strategy to engage the adult in joint attention by drawing his attention toward an unseen event or object reveals 12-month-olds’ clear understanding of referential cues that are relevant for language development. This strategy to successfully initiate and maintain joint attention is related to language development as it increases learning opportunities from social interactions.This research was funded by a Recercaixa 2013–2015 project to Pilar Prieto and two projects from the Spanish MINECO (BFU2012-31995 to Pilar Prieto and PSI-2011-25376 to Laura Bosch). We would like to thank all GrEP and APAL Lab members for their support and comments to earlier versions of this paper, and especially Jorgina Solé for help in sample recruitment and testing assistance, and Joan Borràs-Comes for his help with the statistical analysis of the data. We are grateful to all the subjects and their families for having participated in this longitudinal study

Beat gestures improve word recall in 3- to 5-year-old children

Although research has shown that adults can benefit from the presence of beat gestures in word recall tasks, studies have failed to conclusively generalize these findings to preschool children. This study investigated whether the presence of beat gestures helps children to recall information when these gestures have the function of singling out a linguistic element in its discourse context. A total of 106 3- to 5-year-old children were asked to recall a list of words within a pragmatically child-relevant context (i.e., a storytelling activity) in which the target word was or was not accompanied by a beat gesture. Results showed that children recalled the target word significantly better when it was accompanied by a beat gesture than when it was not, indicating a local recall effect. Moreover, the recall of adjacent non-target words did not differ depending on the condition, revealing that beat gestures seem to have a strictly local highlighting function (i.e., no global recall effect). These results demonstrate that preschoolers benefit from the pragmatic contribution offered by beat gestures when they function as multimodal markers of prominence.This research was funded by grants awarded by Recercaixa 2013–2015 and the Spanish Ministry of Science and Innovation (FFI2015-66533 BFU2012-31995 “Intonational and gestural meaning in language”), by a Labex Brain and Language Research Institute (BLRI) grant (ANR-11-LABX-0036), and by a grant awarded by the Generalitat de Catalunya (2014SGR-925) to the Prosodic Studies Group

Developmental and cognitive aspects of children's disbelief comprehension through intonation and facial gesture

We investigate how children leverage intonational and gestural cues to an individual's belief state through unimodal (intonation-only or facial gestureonly) and multimodal (intonation + facial gesture) cues. A total of 187 preschoolers (ages 3-5) participated in a disbelief comprehension task and were assessed for Theory of Mind (ToM) ability using a false belief task. Significant predictors included Age, Condition and success on the ToM task. Performance improved with age, and was significantly better for the multimodal condition compared to both unimodal conditions, suggesting that even though unimodal cues were useful to children, the presence of reinforcing information for the multimodal condition was more effective for detecting disbelief. However, results also point to the development of intonational and gestural comprehension in tandem. Children that passed the ToM task significantly outperformed those that failed it for all conditions, showing that children who can attribute a false belief to another individual may more readily access these intonational and gestural cues

Intonation and gesture as bootstrapping devices in speaker uncertainty

This study investigates 3- to 5-year-old children’s sensitivity to lexical, intonational, and gestural information in the comprehension of speaker uncertainty. Most previous studies on children’s understanding of speaker certainty and uncertainty across languages have focused on the comprehension of lexical markers, and little is known about the potential facilitation effects of intonational and gestural features in this process. A total of 102 3- to 5-year-old Catalan-speaking children participated in a comprehension task which involved the detection of uncertainty in materials that combined lexical, intonational, and gestural markers. In a between-subjects design, the children were either administered the lexical condition (where they were exposed to lexical and gestural cues to uncertainty) or the intonation condition (where they were exposed to intonational and gestural cues to uncertainty. Within each condition, three different presentation formats were used (audio-only, visual-only and audio-visual) in a within-subjects design. Our results indicated that all the children performed better overall when they had gestural cues present. Furthermore, in comparison with the older group, the younger group was more sensitive to intonational marking of speaker uncertainty than to lexical marking. This evidence suggests that the intonational and gestural features of communicative interactions may act as bootstrapping mechanisms in early pragmatic development.This research has been funded by a grant awarded by the Spanish Ministry of Science and Innovation (FFI2015-66533 BFU2012-31995 “Intonational and gestural meaning in language”), and by a grant awarded by the Generalitat de Catalunya (2014SGR-925) to the Prosodic Studies Group