Workload-aware systems and interfaces for cognitive augmentation

In today's society, our cognition is constantly influenced by information intake, attention switching, and task interruptions. This increases the difficulty of a given task, adding to the existing workload and leading to compromised cognitive performances. The human body expresses the use of cognitive resources through physiological responses when confronted with a plethora of cognitive workload. This temporarily mobilizes additional resources to deal with the workload at the cost of accelerated mental exhaustion. We predict that recent developments in physiological sensing will increasingly create user interfaces that are aware of the user’s cognitive capacities, hence able to intervene when high or low states of cognitive workload are detected. In this thesis, we initially focus on determining opportune moments for cognitive assistance. Subsequently, we investigate suitable feedback modalities in a user-centric design process which are desirable for cognitive assistance. We present design requirements for how cognitive augmentation can be achieved using interfaces that sense cognitive workload. We then investigate different physiological sensing modalities to enable suitable real-time assessments of cognitive workload. We provide empirical evidence that the human brain is sensitive to fluctuations in cognitive resting states, hence making cognitive effort measurable. Firstly, we show that electroencephalography is a reliable modality to assess the mental workload generated during the user interface operation. Secondly, we use eye tracking to evaluate changes in eye movements and pupil dilation to quantify different workload states. The combination of machine learning and physiological sensing resulted in suitable real-time assessments of cognitive workload. The use of physiological sensing enables us to derive when cognitive augmentation is suitable. Based on our inquiries, we present applications that regulate cognitive workload in home and work settings. We deployed an assistive system in a field study to investigate the validity of our derived design requirements. Finding that workload is mitigated, we investigated how cognitive workload can be visualized to the user. We present an implementation of a biofeedback visualization that helps to improve the understanding of brain activity. A final study shows how cognitive workload measurements can be used to predict the efficiency of information intake through reading interfaces. Here, we conclude with use cases and applications which benefit from cognitive augmentation. This thesis investigates how assistive systems can be designed to implicitly sense and utilize cognitive workload for input and output. To do so, we measure cognitive workload in real-time by collecting behavioral and physiological data from users and analyze this data to support users through assistive systems that adapt their interface according to the currently measured workload. Our overall goal is to extend new and existing context-aware applications by the factor cognitive workload. We envision Workload-Aware Systems and Workload-Aware Interfaces as an extension in the context-aware paradigm. To this end, we conducted eight research inquiries during this thesis to investigate how to design and create workload-aware systems. Finally, we present our vision of future workload-aware systems and workload-aware interfaces. Due to the scarce availability of open physiological data sets, reference implementations, and methods, previous context-aware systems were limited in their ability to utilize cognitive workload for user interaction. Together with the collected data sets, we expect this thesis to pave the way for methodical and technical tools that integrate workload-awareness as a factor for context-aware systems.Tagtäglich werden unsere kognitiven Fähigkeiten durch die Verarbeitung von unzähligen Informationen in Anspruch genommen. Dies kann die Schwierigkeit einer Aufgabe durch mehr oder weniger Arbeitslast beeinflussen. Der menschliche Körper drückt die Nutzung kognitiver Ressourcen durch physiologische Reaktionen aus, wenn dieser mit kognitiver Arbeitsbelastung konfrontiert oder überfordert wird. Dadurch werden weitere Ressourcen mobilisiert, um die Arbeitsbelastung vorübergehend zu bewältigen. Wir prognostizieren, dass die derzeitige Entwicklung physiologischer Messverfahren kognitive Leistungsmessungen stets möglich machen wird, um die kognitive Arbeitslast des Nutzers jederzeit zu messen. Diese sind in der Lage, einzugreifen wenn eine zu hohe oder zu niedrige kognitive Belastung erkannt wird. Wir konzentrieren uns zunächst auf die Erkennung passender Momente für kognitive Unterstützung welche sich der gegenwärtigen kognitiven Arbeitslast bewusst sind. Anschließend untersuchen wir in einem nutzerzentrierten Designprozess geeignete Feedbackmechanismen, die zur kognitiven Assistenz beitragen. Wir präsentieren Designanforderungen, welche zeigen wie Schnittstellen eine kognitive Augmentierung durch die Messung kognitiver Arbeitslast erreichen können. Anschließend untersuchen wir verschiedene physiologische Messmodalitäten, welche Bewertungen der kognitiven Arbeitsbelastung in Realzeit ermöglichen. Zunächst validieren wir empirisch, dass das menschliche Gehirn auf kognitive Arbeitslast reagiert. Es zeigt sich, dass die Ableitung der kognitiven Arbeitsbelastung über Elektroenzephalographie eine geeignete Methode ist, um den kognitiven Anspruch neuartiger Assistenzsysteme zu evaluieren. Anschließend verwenden wir Eye-Tracking, um Veränderungen in den Augenbewegungen und dem Durchmesser der Pupille unter verschiedenen Intensitäten kognitiver Arbeitslast zu bewerten. Das Anwenden von maschinellem Lernen führt zu zuverlässigen Echtzeit-Bewertungen kognitiver Arbeitsbelastung. Auf der Grundlage der bisherigen Forschungsarbeiten stellen wir Anwendungen vor, welche die Kognition im häuslichen und beruflichen Umfeld unterstützen. Die physiologischen Messungen stellen fest, wann eine kognitive Augmentierung sich als günstig erweist. In einer Feldstudie setzen wir ein Assistenzsystem ein, um die erhobenen Designanforderungen zur Reduktion kognitiver Arbeitslast zu validieren. Unsere Ergebnisse zeigen, dass die Arbeitsbelastung durch den Einsatz von Assistenzsystemen reduziert wird. Im Anschluss untersuchen wir, wie kognitive Arbeitsbelastung visualisiert werden kann. Wir stellen eine Implementierung einer Biofeedback-Visualisierung vor, die das Nutzerverständnis zum Verlauf und zur Entstehung von kognitiver Arbeitslast unterstützt. Eine abschließende Studie zeigt, wie Messungen kognitiver Arbeitslast zur Vorhersage der aktuellen Leseeffizienz benutzt werden können. Wir schließen hierbei mit einer Reihe von Applikationen ab, welche sich kognitive Arbeitslast als Eingabe zunutze machen. Die vorliegende wissenschaftliche Arbeit befasst sich mit dem Design von Assistenzsystemen, welche die kognitive Arbeitslast der Nutzer implizit erfasst und diese bei der Durchführung alltäglicher Aufgaben unterstützt. Dabei werden physiologische Daten erfasst, um Rückschlüsse in Realzeit auf die derzeitige kognitive Arbeitsbelastung zu erlauben. Anschließend werden diese Daten analysiert, um dem Nutzer strategisch zu assistieren. Das Ziel dieser Arbeit ist die Erweiterung neuartiger und bestehender kontextbewusster Benutzerschnittstellen um den Faktor kognitive Arbeitslast. Daher werden in dieser Arbeit arbeitslastbewusste Systeme und arbeitslastbewusste Benutzerschnittstellen als eine zusätzliche Dimension innerhalb des Paradigmas kontextbewusster Systeme präsentiert. Wir stellen acht Forschungsstudien vor, um die Designanforderungen und die Implementierung von kognitiv arbeitslastbewussten Systemen zu untersuchen. Schließlich stellen wir unsere Vision von zukünftigen kognitiven arbeitslastbewussten Systemen und Benutzerschnittstellen vor. Durch die knappe Verfügbarkeit öffentlich zugänglicher Datensätze, Referenzimplementierungen, und Methoden, waren Kontextbewusste Systeme in der Auswertung kognitiver Arbeitslast bezüglich der Nutzerinteraktion limitiert. Ergänzt durch die in dieser Arbeit gesammelten Datensätze erwarten wir, dass diese Arbeit den Weg für methodische und technische Werkzeuge ebnet, welche kognitive Arbeitslast als Faktor in das Kontextbewusstsein von Computersystemen integriert

Designing Novel Approaches to Personalise Behaviour Change in Intelligent Systems

AI personalisation presents a promising source of innovation for improving the quality of behaviour change technologies. Current approaches are limited in their success, and a proposed solution is the inclusion of intelligent tailoring to best align users with their interventions. This thesis presents three key contributions that explore this promise: A classification system and accompanying survey to examine the current research landscape of intelligent personalisation; The E˙ect-Led Design process which combines high-eÿcacy, limitless expert design concepts with focused user discussion and refinement to best explore how to implement high eÿcacy AI that is acceptable to users; and a conceptual framework, the principles of which are tested in real-world situations to examine whether the intelligent algorithms are able to learn human behaviour and whether proposed systems of personalisation encourage motivation in users. The survey paper identified current trends in the contemporary personalised technology space and explored where the scope for innovation sits. E˙ect-Led Design showed promise in developing significantly di˙erent design concepts to those seen in contemporary applications, and both experts and users commented positively on the process. The studies testing the principles of the experimental platform showed the approaches were positively received by users in terms of motivation and engagement. However, initial implementation issues meant that algorithms did not return any significant evidence of learning. Further explorations into the algorithm through simulated studies using real-world data uncovered alterations that enabled learning. These combined outcomes provided a means to better explore the inclusion of AI in the digital intervention space, with a dedicated design process and investigation of the feasibility of a conceptual framework in this domain showing both the current potential of such a system and where future work can push these ideas to provoke e˙ective behaviour change

Data autonomy in the age of AI: designing autonomy-supportive data tools for children & families

The age of AI is a rapidly evolving and complex space for children. As children increasingly interact with AI-based apps, services and platforms, their data is being increasingly tracked, harvested, aggregated, analysed and exploited in multiple ways that include behavioural engineering and monetisation. Central to such datafication is online service providers' ability to analyse user data to infer personal attributes, subtly manipulating interests and beliefs through micro-targeting and opinion shaping. This can alter the way children perceive and interact with the world, undermining their autonomy. Yet, this datafication often unfolds behind the scenes in apps and services, remaining less noticed and discussed compared to the more straightforward data privacy issues like direct data collection or disclosure. On the other hand, children are often seen as less capable of navigating the intricacies of online life, with parents and guardians presumed to possess greater expertise to steer their children through the digital world. However, the rapid evolution of AI technology and online trends has outpaced parents' ability to keep up. As they adapt to platforms like Snapchat or YouTube, children may already move to the next trend, a shift accelerated by rapid datafication that heightens the challenge of effectively guiding children online. Consequently, there's a mounting call for a child-centred approach, which shifts from just protecting or limiting children with parents in charge, to actively guiding and empowering children to take a leading role. In this shift towards a child-centred approach, there's growing consensus on fostering children's autonomy in the digital space, encompassing the development of their understanding, values, self-determination, and self-identity. Given that data is the cornerstone of AI-based platforms' vast influence, this thesis uniquely focuses on the key concept of data autonomy for children. This exploration follows a structured four-step methodology: 1) Landscape analysis to comprehend the present scope of AI-based platforms for children and the prevalent challenges they encounter; 2) Conceptual review to elucidate the meaning of autonomy for children in the digital realm; 3) Empirical investigation focusing on children's perceptions, needs, and obstacles concerning data autonomy; and 4) Technical evaluation to assess the impact of technical interventions on children's sense of data autonomy. Synthesising the research presented in this thesis, we propose the pivotal concept of data autonomy for children in the age of AI, aiming to address their online wellbeing from a unique data perspective. This work not only lays the foundation for future research on data autonomy as a novel research agenda, but also prompts a rethinking of existing data governance structures towards a more ethical data landscape

The Keck Interferometer

The Keck Interferometer (KI) combined the two 10 m W. M. Keck Observatory telescopes on Mauna Kea, Hawaii, as a long-baseline near- and mid-infrared interferometer. Funded by NASA, it operated from 2001 until 2012. KI used adaptive optics on the two Keck telescopes to correct the individual wavefronts, as well as active fringe tracking in all modes for path-length control, including the implementation of cophasing to provide long coherent integration times. KI implemented high sensitivity fringe-visibility measurements at H (1.6 μm), K (2.2 μm), and L (3.8 μm) bands, and nulling measurements at N band (10 μm), which were used to address a broad range of science topics. Supporting these capabilities was an extensive interferometer infrastructure and unique instrumentation, including some additional functionality added as part of the NSF-funded ASTRA program. This paper provides an overview of the instrument architecture and some of the key design and implementation decisions, as well as a description of all of the key elements and their configuration at the end of the project. The objective is to provide a view of KI as an integrated system, and to provide adequate technical detail to assess the implementation. Included is a discussion of the operational aspects of the system, as well as of the achieved system performance. Finally, details on V^2 calibration in the presence of detector nonlinearities as applied in the data pipeline are provided

Development of Immersive and Interactive Virtual Reality Environment for Two-Player Table Tennis

Although the history of Virtual Reality (VR) is only about half a century old, all kinds of technologies in the VR field are developing rapidly. VR is a computer generated simulation that replaces or augments the real world by various media. In a VR environment, participants have a perception of “presence”, which can be described by the sense of immersion and intuitive interaction. One of the major VR applications is in the field of sports, in which a life-like sports environment is simulated, and the body actions of players can be tracked and represented by using VR tracking and visualisation technology. In the entertainment field, exergaming that merges video game with physical exercise activities by employing tracking or even 3D display technology can be considered as a small scale VR. For the research presented in this thesis, a novel realistic real-time table tennis game combining immersive, interactive and competitive features is developed. The implemented system integrates the InterSense tracking system, SwissRanger 3D camera and a three-wall rear projection stereoscopic screen. The Intersense tracking system is based on ultrasonic and inertia sensing techniques which provide fast and accurate 6-DOF (i.e. six degrees of freedom) tracking information of four trackers. Two trackers are placed on the two players’ heads to provide the players’ viewing positions. The other two trackers are held by players as the racquets. The SwissRanger 3D camera is mounted on top of the screen to capture the player’

In Pursuit of Aviation Cybersecurity: Experiences and Lessons From a Competitive Approach

The passive and independent localization of aircraft has been the subject of much cyberphysical security research. We designed a multistage open competition focusing on the offline batch localization problem using opportunistic data sources. We discuss setup, results, and lessons learned

Understanding expressive action

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2000.Also available online at the MIT Theses Online homepage Includes bibliographical references (p. 117-120).This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.We strain our eyes, cramp our necks, and destroy our hands trying to interact with computer on their terms. At the extreme, we strap on devices and weigh ourselves down with cables trying to re-create a sense of place inside the machine, while cutting ourselves off from the world and people around us. The alternative is to make the real environment responsive to our actions. It is not enough for environments to respond simply to the presence of people or objects: they must also be aware of the subtleties of changing situations. If all the spaces we inhabit are to be responsive, they must not require encumbering devices to be worn and they must be adaptive to changes in the environment and changes of context. This dissertation examines a body of sophisticated perceptual mechanisms developed in response to these needs as well as a selection of human-computer interface sketches designed to push the technology forward and explore the possibilities of this novel interface idiom. Specifically, the formulation of a fully recursive framework for computer vision called DYNA that improves performance of human motion tracking will be examined in depth. The improvement in tracking performance is accomplished with the combination of a three-dimensional, physics-based model of the human body with modifications to the pixel classification algorithms that enable them to take advantage of this high-level knowledge. The result is a novel vision framework that has no completely bottom-up processes, and is therefore significantly faster and more stable than other approaches.by Christopher R. Wren.Ph.D

A white paper: NASA virtual environment research, applications, and technology

Research support for Virtual Environment technology development has been a part of NASA's human factors research program since 1985. Under the auspices of the Office of Aeronautics and Space Technology (OAST), initial funding was provided to the Aerospace Human Factors Research Division, Ames Research Center, which resulted in the origination of this technology. Since 1985, other Centers have begun using and developing this technology. At each research and space flight center, NASA missions have been major drivers of the technology. This White Paper was the joint effort of all the Centers which have been involved in the development of technology and its applications to their unique missions. Appendix A is the list of those who have worked to prepare the document, directed by Dr. Cynthia H. Null, Ames Research Center, and Dr. James P. Jenkins, NASA Headquarters. This White Paper describes the technology and its applications in NASA Centers (Chapters 1, 2 and 3), the potential roles it can take in NASA (Chapters 4 and 5), and a roadmap of the next 5 years (FY 1994-1998). The audience for this White Paper consists of managers, engineers, scientists and the general public with an interest in Virtual Environment technology. Those who read the paper will determine whether this roadmap, or others, are to be followed