Psychophysiological analysis of a pedagogical agent and robotic peer for individuals with autism spectrum disorders.

Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by ongoing problems in social interaction and communication, and engagement in repetitive behaviors. According to Centers for Disease Control and Prevention, an estimated 1 in 68 children in the United States has ASD. Mounting evidence shows that many of these individuals display an interest in social interaction with computers and robots and, in general, feel comfortable spending time in such environments. It is known that the subtlety and unpredictability of people’s social behavior are intimidating and confusing for many individuals with ASD. Computerized learning environments and robots, however, prepare a predictable, dependable, and less complicated environment, where the interaction complexity can be adjusted so as to account for these individuals’ needs. The first phase of this dissertation presents an artificial-intelligence-based tutoring system which uses an interactive computer character as a pedagogical agent (PA) that simulates a human tutor teaching sight word reading to individuals with ASD. This phase examines the efficacy of an instructional package comprised of an autonomous pedagogical agent, automatic speech recognition, and an evidence-based instructional procedure referred to as constant time delay (CTD). A concurrent multiple-baseline across-participants design is used to evaluate the efficacy of intervention. Additionally, post-treatment probes are conducted to assess maintenance and generalization. The results suggest that all three participants acquired and maintained new sight words and demonstrated generalized responding. The second phase of this dissertation describes the augmentation of the tutoring system developed in the first phase with an autonomous humanoid robot which serves the instructional role of a peer for the student. In this tutoring paradigm, the robot adopts a peer metaphor, where its function is to act as a peer. With the introduction of the robotic peer (RP), the traditional dyadic interaction in tutoring systems is augmented to a novel triadic interaction in order to enhance the social richness of the tutoring system, and to facilitate learning through peer observation. This phase evaluates the feasibility and effects of using PA-delivered sight word instruction, based on a CTD procedure, within a small-group arrangement including a student with ASD and the robotic peer. A multiple-probe design across word sets, replicated across three participants, is used to evaluate the efficacy of intervention. The findings illustrate that all three participants acquired, maintained, and generalized all the words targeted for instruction. Furthermore, they learned a high percentage (94.44% on average) of the non-target words exclusively instructed to the RP. The data show that not only did the participants learn nontargeted words by observing the instruction to the RP but they also acquired their target words more efficiently and with less errors by the addition of an observational component to the direct instruction. The third and fourth phases of this dissertation focus on physiology-based modeling of the participants’ affective experiences during naturalistic interaction with the developed tutoring system. While computers and robots have begun to co-exist with humans and cooperatively share various tasks; they are still deficient in interpreting and responding to humans as emotional beings. Wearable biosensors that can be used for computerized emotion recognition offer great potential for addressing this issue. The third phase presents a Bluetooth-enabled eyewear – EmotiGO – for unobtrusive acquisition of a set of physiological signals, i.e., skin conductivity, photoplethysmography, and skin temperature, which can be used as autonomic readouts of emotions. EmotiGO is unobtrusive and sufficiently lightweight to be worn comfortably without interfering with the users’ usual activities. This phase presents the architecture of the device and results from testing that verify its effectiveness against an FDA-approved system for physiological measurement. The fourth and final phase attempts to model the students’ engagement levels using their physiological signals collected with EmotiGO during naturalistic interaction with the tutoring system developed in the second phase. Several physiological indices are extracted from each of the signals. The students’ engagement levels during the interaction with the tutoring system are rated by two trained coders using the video recordings of the instructional sessions. Supervised pattern recognition algorithms are subsequently used to map the physiological indices to the engagement scores. The results indicate that the trained models are successful at classifying participants’ engagement levels with the mean classification accuracy of 86.50%. These models are an important step toward an intelligent tutoring system that can dynamically adapt its pedagogical strategies to the affective needs of learners with ASD

Proceedings of the SITIE2020 Symposium

Complete Summary of Proceedings

Brain Waves, A Cultural History: Oscillations of Neuroscience, Technology, Telepathy, and Transcendence

This project proceeds from a narrow question: What, if anything, is a brain wave? Beguiling in its simplicity, this question prompts a cultural-historical investigation that spans over 150 years of science, technology, and society. Proposed in 1869, the original theory of brain waves cites etheric undulations to explain reports of apparent thought transference. Though most modern thinkers no longer believe in outright telepathy, I argue that dreams of thought transmission and other mental miracles subtly persist—not in obscure and occult circles, but at the forefront of technoscience. A hybrid of science and fiction, brain waves represent an ideal subject through which to explore the ways in which technical language shrouds spiritual dreams. Today, the phrase “brain waves” often function as shorthand for electrical changes in the brain, particularly in the context of technologies that purport to “read” some aspect of mental function, or to transmit neural data to a digital device. While such technologies appear uniquely modern, the history of brain waves reveals that they are merely the millennial incarnation of a much older hope—a hope for transmission and transcendence via the brain’s emanations

Ctrl Shift: How Crip Alt Ctrl Designers Change the Game and Reimagine Access

My journey as a disabled arts practitioner has been one of invention, hacking, and re-imagining what input systems could be. I have created my own modalities for creating work, rather than relying on commercially available options. This is a common practice within the disabled community, as individuals often modify and hack their surroundings to make them more usable. For example, ADAPT activists took sledgehammers to smash curb cuts and poured curb ramps with cement bags, ultimately leading to the widespread adoption of curb cuts as a standard architectural feature. As Yergeau notes, this type of "criptastic hacking" represents a creative resistance.(Yergeau, 2012) My interfaces and art projects are a combination of science fiction world-building, technology prototyping, and experimentation with novel ways of experiencing the world that work for my ability. I have been building interactive objects for over 20 years, and my bespoke controller games are both pieces I find comfortable to play and conceptual proposals that I share with the games community to spark consideration for alternative ways of interacting with games culture. This interdisciplinary design research herein crosses a range of disciplines, drawing inspiration from radical forms of cognitive science, games studies, feminist studies, HCI, crip technoscience, radical science fiction, disability studies, and making practices. What has emerged through studying my own practice and the practices of others during this research is a criptastic design framework for creating playful experiences. My research aims to gain a deeper understanding of the ways that hacking and remaking the world manifests as modifications to the design process itself. I created four versions of a physical alt ctrl game and conducted a design study with disabled artists and alt ctrl game creators. The game, Bot Party, was developed through a series of public exhibitions and explored my relationship between criptastic bespoke interface design and embodied experiences of group play. Bot Party involves physical interaction among players in groups to understand my own ways of designing, while the study looks three other disabled designers to understand the ways in which their process is similar or different to my own. By conducting this work, I aim to contribute to the larger conversation within the games studies community about the importance of accessibility and inclusivity in game design. The results highlight the need for continued exploration and development in this area, specifically in design methods. The study’s findings as they relate to my own practice revealed the importance of considering a set of values and design processes in relation to disability when creating games and playful experiences. With this perspective, I propose an initial framework that outlines possible key themes for disabled game designers. Using values as a starting point for creating deeply accessible games, this framework serves as a starting point for future research into accessible game design. This framework seeks to subvert the notion that accessibility is a list of UX best practices, audio descriptions, captions, and haptic additions and moves towards embedding within game design the values and practices used by disabled designers from the outset of the creative process. Access can be a creative framework. An important point to make is that my efforts to do a PhD resist the academic ableism limiting the participation of people who are not from a normative background. The act of creating this PhD has eaten at the edge of my ability, and the research here was often conducted in pain under extremely trying circumstances. This perspective is relevant because it often informed my design choices and thinking. Additionally, it was conducted at a university where I experienced active discrimination from members of staff who simply refused to believe in disabilities they could not see, and in one case writing down my disability was, “self-ascribed.” To work, I had to move outside the academy and seek out workshops which gave me accessible, ergonomic equipment as is discussed in the Bot Party section. This bears mentioning because it reflects on how threatening disabilities can be within academic settings and how even providing basic levels of accessibility remains a challenge for academic institutions. The above framework could benefit academia if used to redesign postgraduate academic research practices within the academy from a place of Crip-informed pedagogy. This is future work that this academic researcher hopes to explore in depth within their academic journey. It is important to note, much of the most relevant research to this thesis around disability studies and technology has emerged in recent years and as a result, was included iteratively in the literature review. It has informed the third study and my iterative design practice as part of the journey; however, I began this work before much of the writing in the literature review existed, including the creation of Bot Party’s first iterations. Finding this scholarship and these authors has been a kinning. Kinship, according to Gavin Van Horn, “can be considered a noun…shared and storied relations and memories that inhere in people and places; or more metaphorical imaginings that unite us to faith traditions, cultures, countries, or the planet…Perhaps this kinship-in-action should be called kinning.” (Horn et al., 2021) Kinning happened throughout this work and this thesis served me as a place for discovery, contemplation, and empowerment. It is my hope sections of it will serve this function for others within my community. I found kinship with other authors working in the field of disability studies and technology, particularly with Alison Kafer, who offers a critique of Donna Haraway's cyborg in her book "Feminist Queer Crip." (Kafer, 2013) Kafer's work highlights the limitations of Haraway's cyborg as a figure of empowerment for marginalized bodies and identities, and instead advocates for a crip-queer-feminist perspective on technology and embodiment. Additionally, the author has also found resonance in the work of Aimi Hamraie and Kelly Fritsh, whose work in disability studies and HCI has been instrumental in shaping this research. Specifically, their concept of "crip technoscience" has been a key framework for understanding technology creation by disabled technologists. (Hamraie and Fritsch, 2019) Overall, it is my hope that this thesis will serve as a generative resource for others within the community on this journey, particularly for those who are working towards a more inclusive and intersectional understanding of technology and embodiment

Sensored: The Quantified Self, Self-Tracking, and the Limits of Digital Transparency

The idea that daily life overflows with data has entered our common sense. Digital sensors placed in phones, clothing, or household appliances to track how we walk, how much we sleep, or where we travel have heightened the sense that everything about our lives is rapidly being translated into data. Theorists writing about data overload have largely converged around questions of privacy and agency, focusing on the feelings of impotence produced by large quantities of data that now let corporations effortlessly monitor and regulate people’s lives. By contrast, I am interested in moments of friction. Scholars point to real issues, but they overstate the efficacy of data gathering and discount the professional dynamics that motivate the proliferation of data. As I evaluate how data discourse operates and builds, I concentrate on the experiences of those involved in the business of self-tracking, and mainly on the work of U.S.-based developers of wearable computing and the technology professionals who participate in the international forum for data enthusiasts called the Quantified Self. As I analyze how digital entrepreneurialism configures notions of data and transforms digital self-monitoring into meaningful work, I examine how the relationship of technology professionals to data opens onto wider debates about the politics of digital representation. Ultimately, by applying an anthropological lens to explore how the practices, beliefs, and views of marketers, engineers, and developers of self-tracking tools shape digital knowledge, this research challenges accounts of data based purely on transparency, anxiety, and fear and reveals just how precarious the control exerted by digital companies and self-monitoring tools really is

Bit Bang 8: Digitalization

This book is the 8th in the Bit Bang series of books produced as multidisciplinary teamwork exercises by doctoral students participating in the course Bit Bang 8: Digitalization at Aalto University during the academic year 2015–2016. Digitalization has brought great opportunities for economic growth, productivity gain and job creation in our societies, and will change the way industry will operate. Bit Bang 8 addressed the topic of digitalization from the perspective of its economic, environmental and social sustainability. The course elaborated on the interconnectedness of these phenomena, and linked them to possible future scenarios, global megatrends and ethical considerations. How will digitalization shape our future? How can we prepare can prepare our societies to respond to these changes? Working in teams, the students set out to answer questions related to the digitalization and to brainstorm radical scenarios of what the future could hold. This joint publication contains articles produced as teamwork assignments for the course, in which the students were encouraged to take novel and radical views on digitalization. The Bit Bang series of courses is supported by the Multidisciplinary Institute of Digitalisation and Energy (MIDE). Previous Bit Bang publications are available from http:/mide.aalto.fi