Learning to recognize novel words and novel objects

Reading seems as easy and natural as listening. It is still not clear how we acquire this skill, and how visual word identification mechanisms are refined through reading experience. Theoretical models of word recognition describe general principles of skilled reading behaviour. However, these models have been based on averaged data from relatively small samples of skilled readers, mainly English native speakers, and are based on the assumption that skilled reading involves a specialized system of word identification. In this thesis it is proposed that expert reading requires the development and refinement of basic visual processing mechanisms originally employed to identify everyday objects, and then adapted to reading. To test this hypothesis, I carried out three experiments investigating: (i) how L2 visual word recognition changes with growing proficiency; (ii) how novel lexical memories are integrated into the lexicon, i.e., how they interact with previously existing words; and (iii) how sensitivity to the lexicon statistics plays out in the process of learning a novel set of visual stimuli, either in the language and non--language domain

Parametric synthesis of sign language

The isolation of the deaf community from mainstream society is in part due to the lack of knowledge most hearing people have of sign language. To most, there seems to be little need to learn a language that is spoken by such a small minority unless perhaps a relative is unable to hear. Even with a desire to learn, the task may seem insurmountable due to the unique formational and grammatical rules of the language. This linguistic rift has led to the call for an automatic translation system with the ability to take voice or written text as input and produce a comprehensive sequence of signed gestures through computing. This thesis focused on the development of the foundation of a system that would receive English language input and generate a sequence of related signed gestures each synthesized from their basic kinematic parameters. A technique of sign specification for a computer-based translation system was developed through the use of Python objects and functions. Sign definitions, written as Python algorithms, were used to drive the simulation engine of a human-modeling software known as Jack. This research suggests that 3-dimensional computer graphics can be utilized in the production of sign representations that are intelligible and natural in appearance

The Role of Socially-Mediated Alignment in the Development of Second Language Grammar and Vocabulary: Comparing Face-to-Face and Synchronous Mobile-Mediated Communication

Decades of research has shown that speakers mutually adapt to each other’s linguistic behaviors at different levels of language during dialogue. Recent second language (L2) research has suggested that alignment occurring while L2 learners carry out collaborative activities may lead to L2 development, highlighting the benefits of using alignment activities for L2 learning. However, despite the notion that speakers linguistically align in interactions happening in socially-situated contexts, little is known about the role of social factors in the magnitude and learning outcomes of alignment occurring in L2 interaction. The purpose of the study was to examine the pedagogical benefits of alignment activities for the development of L2 grammar and vocabulary during peer interaction across two different interactional contexts: Face-to-Face (FTF) and synchronous mobile-mediated communication (SMMC; mobile text-chat). The target vocabulary items included 32 words and the target structure was a stranded preposition construction embedded in an English relative clause. Furthermore, this study investigated whether social factors (i.e., L2 learners’ perceptions of their interlocutor’s proficiency, comprehensibility of the interlocutor’s language production, and task experience with the interlocutor) and cognitive factors (i.e., individual differences in language aptitude, cognitive style, and proficiency) would modulate alignment effects. Ninety-eight Korean university students were assigned to either the FTF or SMMC group. They completed two alignment activities in pairs, three measurement tests (pre-, post-, and delayed post-test), various cognitive ability tests, and perception questionnaires over four weeks. Results indicated that alignment occurred at the structural and lexical levels in FTF and SMMC modes, but also that structural alignment was facilitated significantly more in the SMMC mode when compared to FTF. However, there was no significant modality effect on the degree of lexical alignment. Findings also demonstrated beneficial role of alignment activities in L2 grammar and vocabulary learning, irrespective of the modality. Furthermore, results suggested that language proficiency and explicit language aptitude were significantly associated with structural alignment driven learning. Learners’ perceptions did not show a significant impact on the degree of alignment and learning outcomes. Implications for the benefits of interactive alignment activities for L2 development and the effect of modality, social factors, and cognitive factors are discussed

Acoustic-Prosodic Entrainment in Human-Human and Human-Computer Dialogue

Entrainment (sometimes called adaptation or alignment) is the tendency of human speakers to adapt to or imitate characteristics of their interlocutors' behavior. This work focuses on entrainment on acoustic-prosodic features. Acoustic-prosodic entrainment has been extensively studied but is not well understood. In particular, it is difficult to compare the results of different studies, since entrainment is usually measured in different ways, reflect- ing disparate conceptualizations of the phenomenon. In the first part of this thesis, we look for evidence of entrainment on a variety of acoustic-prosodic features according to various conceptualizations, and show that human speakers of both Standard American English and Mandarin Chinese entrain to each other globally and locally, in synchrony, and that this entrainment can be constant or convergent. We explore the relationship between entrainment and gender and show that entrainment on some acoustic-prosodic features is related to social behavior and dialogue coordination. In addition, we show that humans entrain in a novel domain, backchannel-inviting cues, and propose and test a novel hypothesis: that entrainment will be stronger in the case of an outlier feature value. In the second part of the thesis, we describe a method for flexibly and dynamically entraining a TTS voice to multiple acoustic-prosodic features of a user's input utterances, and show in an exploratory study that users prefer an entraining avatar to one that does not entrain, are more likely to ask its advice, and choose more positive adjectives to describe its voice. This work introduces a coherent view of entrainment in both familiar and novel domains. Our results add to the body of knowledge of entrainment in human-human conversations and propose new directions for making use of that knowledge to enhance human-computer interactions

Students´ language in computer-assisted tutoring of mathematical proofs

Truth and proof are central to mathematics. Proving (or disproving) seemingly simple statements often turns out to be one of the hardest mathematical tasks. Yet, doing proofs is rarely taught in the classroom. Studies on cognitive difficulties in learning to do proofs have shown that pupils and students not only often do not understand or cannot apply basic formal reasoning techniques and do not know how to use formal mathematical language, but, at a far more fundamental level, they also do not understand what it means to prove a statement or even do not see the purpose of proof at all. Since insight into the importance of proof and doing proofs as such cannot be learnt other than by practice, learning support through individualised tutoring is in demand. This volume presents a part of an interdisciplinary project, set at the intersection of pedagogical science, artificial intelligence, and (computational) linguistics, which investigated issues involved in provisioning computer-based tutoring of mathematical proofs through dialogue in natural language. The ultimate goal in this context, addressing the above-mentioned need for learning support, is to build intelligent automated tutoring systems for mathematical proofs. The research presented here has been focused on the language that students use while interacting with such a system: its linguistic propeties and computational modelling. Contribution is made at three levels: first, an analysis of language phenomena found in students´ input to a (simulated) proof tutoring system is conducted and the variety of students´ verbalisations is quantitatively assessed, second, a general computational processing strategy for informal mathematical language and methods of modelling prominent language phenomena are proposed, and third, the prospects for natural language as an input modality for proof tutoring systems is evaluated based on collected corpora

A Study of Accomodation of Prosodic and Temporal Features in Spoken Dialogues in View of Speech Technology Applications

Inter-speaker accommodation is a well-known property of human speech and human interaction in general. Broadly it refers to the behavioural patterns of two (or more) interactants and the effect of the (verbal and non-verbal) behaviour of each to that of the other(s). Implementation of thisbehavior in spoken dialogue systems is desirable as an improvement on the naturalness of humanmachine interaction. However, traditional qualitative descriptions of accommodation phenomena do not provide sufficient information for such an implementation. Therefore, a quantitativedescription of inter-speaker accommodation is required. This thesis proposes a methodology of monitoring accommodation during a human or humancomputer dialogue, which utilizes a moving average filter over sequential frames for each speaker. These frames are time-aligned across the speakers, hence the name Time Aligned Moving Average (TAMA). Analysis of spontaneous human dialogue recordings by means of the TAMA methodology reveals ubiquitous accommodation of prosodic features (pitch, intensity and speech rate) across interlocutors, and allows for statistical (time series) modeling of the behaviour, in a way which is meaningful for implementation in spoken dialogue system (SDS) environments.In addition, a novel dialogue representation is proposed that provides an additional point of view to that of TAMA in monitoring accommodation of temporal features (inter-speaker pause length and overlap frequency). This representation is a percentage turn distribution of individual speakercontributions in a dialogue frame which circumvents strict attribution of speaker-turns, by considering both interlocutors as synchronously active. Both TAMA and turn distribution metrics indicate that correlation of average pause length and overlap frequency between speakers can be attributed to accommodation (a debated issue), and point to possible improvements in SDS “turntaking” behaviour. Although the findings of the prosodic and temporal analyses can directly inform SDS implementations, further work is required in order to describe inter-speaker accommodation sufficiently, as well as to develop an adequate testing platform for evaluating the magnitude ofperceived improvement in human-machine interaction. Therefore, this thesis constitutes a first step towards a convincingly useful implementation of accommodation in spoken dialogue systems

The Processing of Emotional Sentences by Young and Older Adults: A Visual World Eye-movement Study

Carminati MN, Knoeferle P. The Processing of Emotional Sentences by Young and Older Adults: A Visual World Eye-movement Study. Presented at the Architectures and Mechanisms of Language and Processing (AMLaP), Riva del Garda, Italy