Proficiency-aware systems

In an increasingly digital world, technological developments such as data-driven algorithms and context-aware applications create opportunities for novel human-computer interaction (HCI). We argue that these systems have the latent potential to stimulate users and encourage personal growth. However, users increasingly rely on the intelligence of interactive systems. Thus, it remains a challenge to design for proficiency awareness, essentially demanding increased user attention whilst preserving user engagement. Designing and implementing systems that allow users to become aware of their own proficiency and encourage them to recognize learning benefits is the primary goal of this research. In this thesis, we introduce the concept of proficiency-aware systems as one solution. In our definition, proficiency-aware systems use estimates of the user's proficiency to tailor the interaction in a domain and facilitate a reflective understanding for this proficiency. We envision that proficiency-aware systems leverage collected data for learning benefit. Here, we see self-reflection as a key for users to become aware of necessary efforts to advance their proficiency. A key challenge for proficiency-aware systems is the fact that users often have a different self-perception of their proficiency. The benefits of personal growth and advancing one's repertoire might not necessarily be apparent to users, alienating them, and possibly leading to abandoning the system. To tackle this challenge, this work does not rely on learning strategies but rather focuses on the capabilities of interactive systems to provide users with the necessary means to reflect on their proficiency, such as showing calculated text difficulty to a newspaper editor or visualizing muscle activity to a passionate sportsperson. We first elaborate on how proficiency can be detected and quantified in the context of interactive systems using physiological sensing technologies. Through developing interaction scenarios, we demonstrate the feasibility of gaze- and electromyography-based proficiency-aware systems by utilizing machine learning algorithms that can estimate users' proficiency levels for stationary vision-dominant tasks (reading, information intake) and dynamic manual tasks (playing instruments, fitness exercises). Secondly, we show how to facilitate proficiency awareness for users, including design challenges on when and how to communicate proficiency. We complement this second part by highlighting the necessity of toolkits for sensing modalities to enable the implementation of proficiency-aware systems for a wide audience. In this thesis, we contribute a definition of proficiency-aware systems, which we illustrate by designing and implementing interactive systems. We derive technical requirements for real-time, objective proficiency assessment and identify design qualities of communicating proficiency through user reflection. We summarize our findings in a set of design and engineering guidelines for proficiency awareness in interactive systems, highlighting that proficiency feedback makes performance interpretable for the user.In einer zunehmend digitalen Welt schaffen technologische Entwicklungen - wie datengesteuerte Algorithmen und kontextabhängige Anwendungen - neuartige Interaktionsmöglichkeiten mit digitalen Geräten. Jedoch verlassen sich Nutzer oftmals auf die Intelligenz dieser Systeme, ohne dabei selbst auf eine persönliche Weiterentwicklung hinzuwirken. Wird ein solches Vorgehen angestrebt, verlangt dies seitens der Anwender eine erhöhte Aufmerksamkeit. Es ist daher herausfordernd, ein entsprechendes Design für Kompetenzbewusstsein (Proficiency Awareness) zu etablieren. Das primäre Ziel dieser Arbeit ist es, eine Methodik für das Design und die Implementierung von interaktiven Systemen aufzustellen, die Nutzer dabei unterstützen über ihre eigene Kompetenz zu reflektieren, um dadurch Lerneffekte implizit wahrnehmen können. Diese Arbeit stellt ein Konzept für fähigkeitsbewusste Systeme (proficiency-aware systems) vor, welche die Fähigkeiten von Nutzern abschätzen, die Interaktion entsprechend anpassen sowie das Bewusstsein der Nutzer über deren Fähigkeiten fördern. Hierzu sollten die Systeme gesammelte Daten von Nutzern einsetzen, um Lerneffekte sichtbar zu machen. Die Möglichkeit der Anwender zur Selbstreflexion ist hierbei als entscheidend anzusehen, um als Motivation zur Verbesserung der eigenen Fähigkeiten zu dienen. Eine zentrale Herausforderung solcher Systeme ist die Tatsache, dass Nutzer - im Vergleich zur Abschätzung des Systems - oft eine divergierende Selbstwahrnehmung ihrer Kompetenz haben. Im ersten Moment sind daher die Vorteile einer persönlichen Weiterentwicklung nicht unbedingt ersichtlich. Daher baut diese Forschungsarbeit nicht darauf auf, Nutzer über vorgegebene Lernstrategien zu unterrichten, sondern sie bedient sich der Möglichkeiten interaktiver Systeme, die Anwendern die notwendigen Hilfsmittel zur Verfügung stellen, damit diese selbst über ihre Fähigkeiten reflektieren können. Einem Zeitungseditor könnte beispielsweise die aktuelle Textschwierigkeit angezeigt werden, während einem passionierten Sportler dessen Muskelaktivität veranschaulicht wird. Zunächst wird herausgearbeitet, wie sich die Fähigkeiten der Nutzer mittels physiologischer Sensortechnologien erkennen und quantifizieren lassen. Die Evaluation von Interaktionsszenarien demonstriert die Umsetzbarkeit fähigkeitsbewusster Systeme, basierend auf der Analyse von Blickbewegungen und Muskelaktivität. Hierbei kommen Algorithmen des maschinellen Lernens zum Einsatz, die das Leistungsniveau der Anwender für verschiedene Tätigkeiten berechnen. Im Besonderen analysieren wir stationäre Aktivitäten, die hauptsächlich den Sehsinn ansprechen (Lesen, Aufnahme von Informationen), sowie dynamische Betätigungen, die die Motorik der Nutzer fordern (Spielen von Instrumenten, Fitnessübungen). Der zweite Teil zeigt auf, wie Systeme das Bewusstsein der Anwender für deren eigene Fähigkeiten fördern können, einschließlich der Designherausforderungen , wann und wie das System erkannte Fähigkeiten kommunizieren sollte. Abschließend wird die Notwendigkeit von Toolkits für Sensortechnologien hervorgehoben, um die Implementierung derartiger Systeme für ein breites Publikum zu ermöglichen. Die Forschungsarbeit beinhaltet eine Definition für fähigkeitsbewusste Systeme und veranschaulicht dieses Konzept durch den Entwurf und die Implementierung interaktiver Systeme. Ferner werden technische Anforderungen objektiver Echtzeitabschätzung von Nutzerfähigkeiten erforscht und Designqualitäten für die Kommunikation dieser Abschätzungen mittels Selbstreflexion identifiziert. Zusammengefasst sind die Erkenntnisse in einer Reihe von Design- und Entwicklungsrichtlinien für derartige Systeme. Insbesondere die Kommunikation, der vom System erkannten Kompetenz, hilft Anwendern, die eigene Leistung zu interpretieren

Language Factors Modulate Audiovisual Speech Perception. A Developmental Perspective

[eng] In most natural situations, adults look at the eyes of faces in seek of social information (Yarbus, 1967). However, when the auditory information becomes unclear (e.g. speech-in- noise) they switch their attention towards the mouth of a talking face and rely on the audiovisual redundant cues to help them process the speech signal (Barenholtz, Mavica, & Lewkowicz, 2016; Buchan, Paré, & Munhall, 2007; Lansing & McConkie, 2003; Vatikiotis- Bateson, Eigsti, Yano, & Munhall, 1998). Likewise, young infants are sensitive to the correspondence between acoustic and visual speech (Bahrick & Lickliter, 2012), and they also rely on the talker’s mouth during the second half of the first year of life, putatively to help them acquire language by the time they start babbling (Lewkowicz & Hansen-Tift, 2012), and also to aid language differentiation in the case of bilingual infants (Pons, Bosch & Lewkowicz, 2015). The current set of studies provides a detailed examination of the audiovisual (AV) speech cues contribution to speech processing at different language development stages, through the analysis of selective attention patterns when processing speech from talking faces. To do so, I compared different linguistic experience factors (i.e. types of bilingualism – distance between bilinguals’ two languages –, language familiarity and language proficiency) that modulate audiovisual speech perception in first language acquisition during infancy (Studies 1 and 2), early childhood (Studies 3 and 4), and in second language (L2) learning during adulthood (Studies 5, 6 and 7). The findings of the present work demonstrate that (1) perceiving speech audiovisually hampers close bilingual infants’ ability to discriminate their languages, that (2) 15-month-old and 5 year-old close language bilinguals rely more on the mouth cues of a talking face than do their distant bilingual peers, that (3) children’s attention to the mouth follows a clear temporal pattern: it is maximal in the beginning of the presentation and it diminishes gradually as speech continues, and that (4) adults also rely more on the mouth speech cues when they perceive fluent non-native vs. native speech, regardless of their L2 expertise. All in all, these studies shed new light into the field of audiovisual speech perception and language processing by showing that selective attention to a talker’s eyes and mouth is a dynamic, information-seeking process, which is largely modulated by perceivers’ early linguistic experience and the tasks’ demands. These results suggest that selectively attending the redundant speech cues of a talker’s mouth at the adequate moment enhances speech perception and is crucial for normal language development and speech processing, not only in infancy – during first language acquisition – but also in more advanced language stages in childhood, as well as in L2 learning during adulthood. Ultimately, they confirm that mouth reliance is greater in close bilingual environments, where the presence of two related languages increases the necessity for disambiguation and keeping separate language systems.[cat] Atendre selectivament a la boca d’un parlant ens ajuda a beneficiar-nos de la informació audiovisual i processar millor el senyal de la parla, quan el senyal auditiu es torna confús. Paral·lelament, els infants també atenen a la boca durant la segona meitat del primer any de vida, la qual cosa els ajuda en l'adquisició del llenguatge/s. Aquesta tesi examina la contribució del senyal audiovisual al processament de la parla, a través de les anàlisis d'atenció selectiva a una cara parlant. Es comparen diferents factors lingüístics (tipologies de bilingüisme, la familiaritat i la competència amb l'idioma) que modulen la percepció audiovisual de la parla en l'adquisició del llenguatge durant la primera infància (Estudis 1 i 2), en nens d’edat escolar (Estudis 3 i 4) i l’aprenentatge d'una segona llengua durant l'edat adulta (Estudis 5, 6 i 7). Els resultats demostren que (1) la percepció audiovisual de la parla dificulta la capacitat dels infants bilingües de discriminar les seves llengües properes, que (2) els bilingües de llengües properes de 15 mesos i de 5 anys d’edat posen més atenció a les pistes audiovisuals de la boca que els bilingües de llengües distants, que (3) l’atenció dels nens a la boca del parlant és màxima al començament i disminueix gradualment a mesura que continua la parla, i que (4) els adults també es recolzen més en els senyals audiovisuals de la boca quan perceben una llengua no nativa (L2), independentment de la seva competència en aquesta. Aquests estudis demostren que l'atenció selectiva a la cara d'un parlant és un procés dinàmic i de cerca d'informació, i que aquest és modulat per l'experiència lingüística primerenca i les exigències que comporten les situacions comunicatives. Aquests resultats suggereixen que atendre a les pistes audiovisuals de la boca en els moments adequats és crucial per al desenvolupament normal del llenguatge, tan durant la primera infància com en les etapes més avançades del llenguatge, així com en l’aprenentatge de segones llengües. Per últim, aquests resultats confirmen que l’estratègia de recolzar-se en les pistes audiovisuals s’utilitza en major mesura en entorns bilingües propers, on la presència de dues llengües relacionades augmenta la necessitat de desambiguació

Figuring Out How Verb-Particle Constructions Are Understood During L1 and L2 Reading

The aim of this paper was to investigate first-language (L1) and second-language (L2) reading of verb particle constructions (VPCs) among English–French bilingual adults. VPCs, or phrasal verbs, are highly common collocations of a verb paired with a particle, such as eat up or chew out, that often convey a figurative meaning. VPCs vary in form (eat up the candy vs. eat the candy up) and in other factors, such as the semantic contribution of the constituent words to the overall meaning (semantic transparency) and frequency. Much like classic forms of idioms, VPCs are difficult for L2 users. Here, we present two experiments that use eye-tracking to discover factors that influence the ease with which VPCs are processed by bilingual readers. In Experiment 1, we compared L1 reading of adjacent vs. split VPCs, and then explored whether the general pattern was driven by item-level factors. L1 readers did not generally find adjacent VPCs (eat up the candy) easier to process than split VPCs (eat the candy up); however, VPCs low in co-occurrence strength (i.e., low semantic transparency) and high in frequency were easiest to process in the adjacent form during first pass reading. In Experiment 2, we compared L2 reading of adjacent vs. split VPCs, and then explored whether the general pattern varied with item-level or participant-level factors. L2 readers generally allotted more second pass reading time to split vs. adjacent forms, and there was some evidence that this pattern was greater for L2 English readers who had less English experience. In contrast with L1 reading, there was no influence of item differences on L2 reading behavior. These data suggest that L1 readers may have lexicalized VPC representations that are directly retrieved during comprehension, whereas L2 readers are more likely to compositionally process VPCs given their more general preference for adjacent particles, as demonstrated by longer second pass reading time for all split items

The reliability of eyetracking to assess attentional bias to threatening words in healthy individuals

Eyetracking is commonly used to investigate attentional bias. Although some studies have investigated the internal consistency of eyetracking, data are scarce on the test–retest reliability and agreement of eyetracking to investigate attentional bias. This study reports the test–retest reliability, measurement error, and internal consistency of 12 commonly used outcome measures thought to reflect the different components of attentional bias: overall attention, early attention, and late attention. Healthy participants completed a preferential-looking eyetracking task that involved the presentation of threatening (sensory words, general threat words, and affective words) and nonthreatening words. We used intraclass correlation coefficients (ICCs) to measure test–retest reliability (ICC \u3e .70 indicates adequate reliability). The ICCs(2, 1) ranged from –.31to.71. Reliability varied according to the outcome measure and threat word category. Sensory words had a lower mean ICC (.08) than either affective words (.32) or general threat words (.29). A longer exposure time was associated with higher test–retest reliability. All of the outcome measures, except second-run dwell time, demonstrated low measurement error (\u3c6%). Most of the outcome measures reported high internal consistency (α \u3e.93). Recommendations are discussed for improving the reliability of eyetracking tasks in future research

Hyper-learning from hyper-teaching: what might the future hold for learning mathematics from & with electronic screens?

Traditionally, mathematics has been learned through learners manipulating the content of their mental screens (mental imagery). When the situation became complicated, devices were found to augment and support the mental screen, holding some aspects invariant while transformations and anticipations were experienced and performed mentally. Now electronic escreens further augment, but threaten to overwhelm, our mental screens. In this paper different modes of interaction with e-screens are outlined, and attention is drawn to the discipline required to use them effectively. Questions are raised about how e-screens could alter what we mean by learning mathematics, at least for the majority.Traditionally, mathematics has been learned through learners manipulating the content of their mental screens (mental imagery). When the situation became complicated, devices were found to augment and support the mental screen, holding some aspects invariant while transformations and anticipations were experienced and performed mentally. Now electronic escreens further augment, but threaten to overwhelm, our mental screens. In this paper different modes of interaction with e-screens are outlined, and attention is drawn to the discipline required to use them effectively. Questions are raised about how e-screens could alter what we mean by learning mathematics, at least for the majority

Hyper-learning from hyper-teaching: what might the future hold for learning mathematics from & with electronic screens?

Traditionally, mathematics has been learned through learners manipulating the content of their mental screens (mental imagery). When the situation became complicated, devices were found to augment and support the mental screen, holding some aspects invariant while transformations and anticipations were experienced and performed mentally. Now electronic escreens further augment, but threaten to overwhelm, our mental screens. In this paper different modes of interaction with e-screens are outlined, and attention is drawn to the discipline required to use them effectively. Questions are raised about how e-screens could alter what we mean by learning mathematics, at least for the majority.Traditionally, mathematics has been learned through learners manipulating the content of their mental screens (mental imagery). When the situation became complicated, devices were found to augment and support the mental screen, holding some aspects invariant while transformations and anticipations were experienced and performed mentally. Now electronic escreens further augment, but threaten to overwhelm, our mental screens. In this paper different modes of interaction with e-screens are outlined, and attention is drawn to the discipline required to use them effectively. Questions are raised about how e-screens could alter what we mean by learning mathematics, at least for the majority

Is semantic preview benefit due to relatedness or plausibility?

There is increasing evidence that skilled readers of English benefit from processing a parafoveal preview of a semantically related word. However, in previous investigations of semantic preview benefit using the gaze-contingent boundary paradigm the semantic relatedness between the preview and target has been confounded with the plausibility of the preview word in the sentence. In the present study, preview relatedness and plausibility were independently manipulated in neutral sentences read by a large sample of skilled adult readers. Participants were assessed on measures of reading and spelling ability to identify possible sources of individual differences in preview effects. The results showed that readers benefited from a preview of a plausible word, regardless of the semantic relatedness of the preview and the target. However, there was limited evidence of a semantic relatedness benefit when the plausibility of the preview was controlled. The plausibility preview benefit was strongest for low proficiency readers, suggesting that poorer readers were more likely to program a forward saccade based on information extracted from the preview. High proficiency readers showed equivalent disruption from all non-identical previews suggesting that they were more likely to suffer interference from the orthographic mismatch between preview and target.Australian Research Counci

Conversational Movement Dynamics and Nonverbal Indicators of Second Language Development: A Microgenetic Approach

This dissertation study extends on current understandings of gesture and embodied interaction with the eco-social environment in second language development (SLD) while introducing new aspects of movement analysis through dynamical modeling. To understand the role of embodiment during learning activities, a second language learning task has been selected. Dyads consisting of a non-native English-speaking student and a native English-speaking tutor were video recorded during writing consultations centered on class assignments provided by the student. Cross-recurrence quantification analysis was used to measure interactional movement synchrony between the members of each dyad. Results indicate that students with varied English proficiency levels synchronize movements with their tutors over brief, frequent periods of time. Synchronous movement pattern complexity is highly variable across and within the dyads. Additionally, co-speech gesture and gesture independent of speech were analyzed qualitatively to identify the role of gesture as related to SLD events. A range of movement types were used during developmental events by the students and tutors to interact with their partner. The results indicated that language development occurs within a movement rich context through negotiated interaction which depends on a combination of synchronized and synergistic movements. Synchronized movements exhibited complex, dynamical behaviors including variability, self-organization, and emergent properties. Synergistic movement emergence revealed how the dualistic presence of the self/other in each dyad creates a functioning intersubjective space. Overall, the dyads demonstrated that movement is a salient factor in the writing consultation activity

Language processing in bilingual children and adults: evidence from filler-gap dependencies and garden path sentences

The present thesis examines morphosyntactic processing in bilingual children and adults. It bridges gaps in the existing literature in three ways. Firstly, previous work on bilingual children has focused on inflectional morphology but has not examined the timecourse of processing in terms of misinterpretation and real time reanalysis or use of information of different sources to aid disambiguation. Second, it extends the use of the visual world eye-tracking paradigm to research in morphosyntactic processing in bilingual children. Third, for the adults, it compares early/native bilinguals to monolinguals and late bilinguals / L2 learners. Two linguistic phenomena were investigated with adults and children; which-questions and garden-path sentences in English. Overall, both bilingual children and adults showed qualitatively similar patterns of processing to their monolingual counterparts. All groups experienced greater difficulty with structures where there was ambiguity and a need for syntactic reanalysis, i.e. object which-questions and garden-path sentences, suggesting incremental processing. The main difference between monolinguals and bilinguals is that of speed; bilinguals appeared to process sentences slower than monolinguals even when their comprehension accuracy was equally as high. This difference was found for both children and adults, as evidenced by the reaction times or changes in the gaze data, and was generally not more pronounced in sentences where reanalysis is required. With regards to the bilingual adults, the early/native bilinguals clustered with the L2 learners in terms of processing speed but more so with the monolingual adults in terms of accuracy. The bilingual groups showed a reduced utilisation of information from various sources to aid processing. Bilingual adults and children made use of number mismatch between the two noun phrases in the study on which-questions to facilitate disambiguation; however, they showed this effect just for off-line comprehension accuracy and not for real time processing, i.e. in the gaze data. The bilinguals also did not show consistent use of referential context to disambiguate in the study on garden path sentences, although this was also the case for the monolingual adults and children. In sum, the results from the studies in this thesis suggest the both bilingual adults and children were equally as able as their monolingual counterparts at an end stage but differed to the monolinguals on more fine-grained measures of real time processing. These measures point to qualitatively similar but more protracted over time processing for bilinguals and with more limited use of facilitatory information to disambiguate