Multimodal Data Analysis of Dyadic Interactions for an Automated Feedback System Supporting Parent Implementation of Pivotal Response Treatment

abstract: Parents fulfill a pivotal role in early childhood development of social and communication skills. In children with autism, the development of these skills can be delayed. Applied behavioral analysis (ABA) techniques have been created to aid in skill acquisition. Among these, pivotal response treatment (PRT) has been empirically shown to foster improvements. Research into PRT implementation has also shown that parents can be trained to be effective interventionists for their children. The current difficulty in PRT training is how to disseminate training to parents who need it, and how to support and motivate practitioners after training. Evaluation of the parents’ fidelity to implementation is often undertaken using video probes that depict the dyadic interaction occurring between the parent and the child during PRT sessions. These videos are time consuming for clinicians to process, and often result in only minimal feedback for the parents. Current trends in technology could be utilized to alleviate the manual cost of extracting data from the videos, affording greater opportunities for providing clinician created feedback as well as automated assessments. The naturalistic context of the video probes along with the dependence on ubiquitous recording devices creates a difficult scenario for classification tasks. The domain of the PRT video probes can be expected to have high levels of both aleatory and epistemic uncertainty. Addressing these challenges requires examination of the multimodal data along with implementation and evaluation of classification algorithms. This is explored through the use of a new dataset of PRT videos. The relationship between the parent and the clinician is important. The clinician can provide support and help build self-efficacy in addition to providing knowledge and modeling of treatment procedures. Facilitating this relationship along with automated feedback not only provides the opportunity to present expert feedback to the parent, but also allows the clinician to aid in personalizing the classification models. By utilizing a human-in-the-loop framework, clinicians can aid in addressing the uncertainty in the classification models by providing additional labeled samples. This will allow the system to improve classification and provides a person-centered approach to extracting multimodal data from PRT video probes.Dissertation/ThesisDoctoral Dissertation Computer Science 201

Understanding Spoken Language Development of Children with ASD Using Pre-trained Speech Embeddings

Speech processing techniques are useful for analyzing speech and language development in children with Autism Spectrum Disorder (ASD), who are often varied and delayed in acquiring these skills. Early identification and intervention are crucial, but traditional assessment methodologies such as caregiver reports are not adequate for the requisite behavioral phenotyping. Natural Language Sample (NLS) analysis has gained attention as a promising complement. Researchers have developed benchmarks for spoken language capabilities in children with ASD, obtainable through the analysis of NLS. This paper proposes applications of speech processing technologies in support of automated assessment of children's spoken language development by classification between child and adult speech and between speech and nonverbal vocalization in NLS, with respective F1 macro scores of 82.6% and 67.8%, underscoring the potential for accurate and scalable tools for ASD research and clinical use.Comment: Accepted to Interspeech 2023, 5 page

Couple relationship quality and the infant home language environment: Gender-specific findings.

Couple relationship quality is known to drop significantly across the transition to parenthood (Ahlborg & Strandmark, 2001; Doss, Rhoades, Stanley, & Markman, 2009), yet individual differences in the amount of parent-to-infant talk have rarely been studied in relation to variation in couple relationship quality. Addressing this gap, the current study of 93 first-time parents with 4-month-old infants included multimeasure reports of couple relationship quality from both mothers and fathers and examined associations between couple relationship quality and the home language environment, assessed via the Language Environment Analysis (LENA), when infants were approximately 7 months old. LENA consists of a wearable talk pedometer that records a full day of naturalistic parent-infant talk and is coupled to software that provides automated analysis. Given the covariation between depression and both couple relationship quality and parental infant-directed talk, both maternal and paternal depression were controlled for in all analyses. Results showed that, for mothers of sons, frequency of infant-directed talk was inversely related to couple relationship quality. Consistent with family systems theory, this finding provides partial support for the compensation hypothesis. However, variation in couple relationship quality was unrelated to infant-directed speech in fathers or in mothers of daughters. Together, these findings demonstrate that the gender composition of the parent-infant dyads plays a moderating role on the association between couple relationship quality and parent-infant talk. (PsycINFO Database Record (c) 2020 APA, all rights reserved)

Vocal development in a large‐scale crosslinguistic corpus

This study evaluates whether early vocalizations develop in similar ways in children across diverse cultural contexts. We analyze data from daylong audio recordings of 49 children (1–36 months) from five different language/cultural backgrounds. Citizen scientists annotated these recordings to determine if child vocalizations contained canonical transitions or not (e.g., “ba” vs. “ee”). Results revealed that the proportion of clips reported to contain canonical transitions increased with age. Furthermore, this proportion exceeded 0.15 by around 7 months, replicating and extending previous findings on canonical vocalization development but using data from the natural environments of a culturally and linguistically diverse sample. This work explores how crowdsourcing can be used to annotate corpora, helping establish developmental milestones relevant to multiple languages and cultures. Lower inter‐annotator reliability on the crowdsourcing platform, relative to more traditional in‐lab expert annotators, means that a larger number of unique annotators and/or annotations are required, and that crowdsourcing may not be a suitable method for more fine‐grained annotation decisions. Audio clips used for this project are compiled into a large‐scale infant vocalization corpus that is available for other researchers to use in future work

Child-Directed Speech and the Developing Brain: An Investigation of Adult Verbal Warmth and Negative Affect

This dissertation examines the association between the quality of children’s language experiences – as operationalized by adult verbal warmth – and their cognitive developmental outcomes. A socioeconomically diverse sample of 43 parents and their 5-to-9-year-old children participated in this study. A digital audio recording of the home environment was obtained, and children completed a high-resolution, structural MRI scan as well as direct assessments of their language and reading skills. The audio recordings were transcribed and coded using a coding scheme newly developed by the candidate in consultation with leading experts, in order to identify and quantify psycholinguistic elements of adult-child communication. Primary hypotheses included that adult verbal warmth is associated with (1) language and reading outcomes (2) the neural regions associated with each. To date, no studies have combined a transcription-based, fine-grained analysis of naturalistic home recordings with neuroimaging data. As such, this study represents a new line of inquiry at the nexus of developmental psychology, neuroscience, and education. The findings shed light on the impact of psychosocial language experiences on child development and on which forms of adult-child communication are most conducive to learning. Such information can inform programs that aim to teach parents ways to nurture their children’s development through high-quality child-directed speech. Social, educational, and clinical implications for mitigating risk factors and bolstering protective factors in order to, ultimately, foster healthy development for all children, are discussed

Validation of the Language ENvironment Analysis (LENA) system for Dutch

The validity of the Language ENvironment Analysis (LENA) System was evaluatedfor Dutch. 216 5-min samples (six samples per age per child) were selected from daylong recordings at 5, 10 and 14 months of age of native Dutch-speaking younger siblings of children with autism spectrum disorder (N=6) and of typically developing children (N=6). Two native Dutch-speaking coders counted the amount of adult words (AWC),child vocalisations(CVC)and conversational turns (CT). Consequently, correlations between LENA and human estimates were explored. Correlations were high for AWC at all ages (r= .73 to .81). Regarding CVC, estimates weremoderately correlated at 5 months (r= .57) but the correlation decreased at 10 (r= .37) and 14 months (r= .14). Correlations for CT were low at all ages (r= .19 to .28). Lastly, correlations were not influenced by the risk status of the children

Vocal Patterns in Infants with Autism Spectrum Disorder: Canonical Babbling Status and Vocalization Frequency

Canonical babbling is a critical milestone for speech development and is usually well in place by 10 months. The possibility that infants with autism spectrum disorder (ASD) show late onset of canonical babbling has so far eluded evaluation. Rate of vocalization or “volubility” has also been suggested as possibly aberrant in infants with ASD. We conducted a retrospective video study examining vocalizations of 37 infants at 9–12 and 15–18 months. Twenty-three of the 37 infants were later diagnosed with ASD and indeed produced low rates of canonical babbling and low volubility by comparison with the 14 typically developing infants. The study thus supports suggestions that very early vocal patterns may prove to be a useful component of early screening and diagnosis of ASD