Workload-aware systems and interfaces for cognitive augmentation

In today's society, our cognition is constantly influenced by information intake, attention switching, and task interruptions. This increases the difficulty of a given task, adding to the existing workload and leading to compromised cognitive performances. The human body expresses the use of cognitive resources through physiological responses when confronted with a plethora of cognitive workload. This temporarily mobilizes additional resources to deal with the workload at the cost of accelerated mental exhaustion. We predict that recent developments in physiological sensing will increasingly create user interfaces that are aware of the user’s cognitive capacities, hence able to intervene when high or low states of cognitive workload are detected. In this thesis, we initially focus on determining opportune moments for cognitive assistance. Subsequently, we investigate suitable feedback modalities in a user-centric design process which are desirable for cognitive assistance. We present design requirements for how cognitive augmentation can be achieved using interfaces that sense cognitive workload. We then investigate different physiological sensing modalities to enable suitable real-time assessments of cognitive workload. We provide empirical evidence that the human brain is sensitive to fluctuations in cognitive resting states, hence making cognitive effort measurable. Firstly, we show that electroencephalography is a reliable modality to assess the mental workload generated during the user interface operation. Secondly, we use eye tracking to evaluate changes in eye movements and pupil dilation to quantify different workload states. The combination of machine learning and physiological sensing resulted in suitable real-time assessments of cognitive workload. The use of physiological sensing enables us to derive when cognitive augmentation is suitable. Based on our inquiries, we present applications that regulate cognitive workload in home and work settings. We deployed an assistive system in a field study to investigate the validity of our derived design requirements. Finding that workload is mitigated, we investigated how cognitive workload can be visualized to the user. We present an implementation of a biofeedback visualization that helps to improve the understanding of brain activity. A final study shows how cognitive workload measurements can be used to predict the efficiency of information intake through reading interfaces. Here, we conclude with use cases and applications which benefit from cognitive augmentation. This thesis investigates how assistive systems can be designed to implicitly sense and utilize cognitive workload for input and output. To do so, we measure cognitive workload in real-time by collecting behavioral and physiological data from users and analyze this data to support users through assistive systems that adapt their interface according to the currently measured workload. Our overall goal is to extend new and existing context-aware applications by the factor cognitive workload. We envision Workload-Aware Systems and Workload-Aware Interfaces as an extension in the context-aware paradigm. To this end, we conducted eight research inquiries during this thesis to investigate how to design and create workload-aware systems. Finally, we present our vision of future workload-aware systems and workload-aware interfaces. Due to the scarce availability of open physiological data sets, reference implementations, and methods, previous context-aware systems were limited in their ability to utilize cognitive workload for user interaction. Together with the collected data sets, we expect this thesis to pave the way for methodical and technical tools that integrate workload-awareness as a factor for context-aware systems.Tagtäglich werden unsere kognitiven Fähigkeiten durch die Verarbeitung von unzähligen Informationen in Anspruch genommen. Dies kann die Schwierigkeit einer Aufgabe durch mehr oder weniger Arbeitslast beeinflussen. Der menschliche Körper drückt die Nutzung kognitiver Ressourcen durch physiologische Reaktionen aus, wenn dieser mit kognitiver Arbeitsbelastung konfrontiert oder überfordert wird. Dadurch werden weitere Ressourcen mobilisiert, um die Arbeitsbelastung vorübergehend zu bewältigen. Wir prognostizieren, dass die derzeitige Entwicklung physiologischer Messverfahren kognitive Leistungsmessungen stets möglich machen wird, um die kognitive Arbeitslast des Nutzers jederzeit zu messen. Diese sind in der Lage, einzugreifen wenn eine zu hohe oder zu niedrige kognitive Belastung erkannt wird. Wir konzentrieren uns zunächst auf die Erkennung passender Momente für kognitive Unterstützung welche sich der gegenwärtigen kognitiven Arbeitslast bewusst sind. Anschließend untersuchen wir in einem nutzerzentrierten Designprozess geeignete Feedbackmechanismen, die zur kognitiven Assistenz beitragen. Wir präsentieren Designanforderungen, welche zeigen wie Schnittstellen eine kognitive Augmentierung durch die Messung kognitiver Arbeitslast erreichen können. Anschließend untersuchen wir verschiedene physiologische Messmodalitäten, welche Bewertungen der kognitiven Arbeitsbelastung in Realzeit ermöglichen. Zunächst validieren wir empirisch, dass das menschliche Gehirn auf kognitive Arbeitslast reagiert. Es zeigt sich, dass die Ableitung der kognitiven Arbeitsbelastung über Elektroenzephalographie eine geeignete Methode ist, um den kognitiven Anspruch neuartiger Assistenzsysteme zu evaluieren. Anschließend verwenden wir Eye-Tracking, um Veränderungen in den Augenbewegungen und dem Durchmesser der Pupille unter verschiedenen Intensitäten kognitiver Arbeitslast zu bewerten. Das Anwenden von maschinellem Lernen führt zu zuverlässigen Echtzeit-Bewertungen kognitiver Arbeitsbelastung. Auf der Grundlage der bisherigen Forschungsarbeiten stellen wir Anwendungen vor, welche die Kognition im häuslichen und beruflichen Umfeld unterstützen. Die physiologischen Messungen stellen fest, wann eine kognitive Augmentierung sich als günstig erweist. In einer Feldstudie setzen wir ein Assistenzsystem ein, um die erhobenen Designanforderungen zur Reduktion kognitiver Arbeitslast zu validieren. Unsere Ergebnisse zeigen, dass die Arbeitsbelastung durch den Einsatz von Assistenzsystemen reduziert wird. Im Anschluss untersuchen wir, wie kognitive Arbeitsbelastung visualisiert werden kann. Wir stellen eine Implementierung einer Biofeedback-Visualisierung vor, die das Nutzerverständnis zum Verlauf und zur Entstehung von kognitiver Arbeitslast unterstützt. Eine abschließende Studie zeigt, wie Messungen kognitiver Arbeitslast zur Vorhersage der aktuellen Leseeffizienz benutzt werden können. Wir schließen hierbei mit einer Reihe von Applikationen ab, welche sich kognitive Arbeitslast als Eingabe zunutze machen. Die vorliegende wissenschaftliche Arbeit befasst sich mit dem Design von Assistenzsystemen, welche die kognitive Arbeitslast der Nutzer implizit erfasst und diese bei der Durchführung alltäglicher Aufgaben unterstützt. Dabei werden physiologische Daten erfasst, um Rückschlüsse in Realzeit auf die derzeitige kognitive Arbeitsbelastung zu erlauben. Anschließend werden diese Daten analysiert, um dem Nutzer strategisch zu assistieren. Das Ziel dieser Arbeit ist die Erweiterung neuartiger und bestehender kontextbewusster Benutzerschnittstellen um den Faktor kognitive Arbeitslast. Daher werden in dieser Arbeit arbeitslastbewusste Systeme und arbeitslastbewusste Benutzerschnittstellen als eine zusätzliche Dimension innerhalb des Paradigmas kontextbewusster Systeme präsentiert. Wir stellen acht Forschungsstudien vor, um die Designanforderungen und die Implementierung von kognitiv arbeitslastbewussten Systemen zu untersuchen. Schließlich stellen wir unsere Vision von zukünftigen kognitiven arbeitslastbewussten Systemen und Benutzerschnittstellen vor. Durch die knappe Verfügbarkeit öffentlich zugänglicher Datensätze, Referenzimplementierungen, und Methoden, waren Kontextbewusste Systeme in der Auswertung kognitiver Arbeitslast bezüglich der Nutzerinteraktion limitiert. Ergänzt durch die in dieser Arbeit gesammelten Datensätze erwarten wir, dass diese Arbeit den Weg für methodische und technische Werkzeuge ebnet, welche kognitive Arbeitslast als Faktor in das Kontextbewusstsein von Computersystemen integriert

Quantifying Quality of Life

Describes technological methods and tools for objective and quantitative assessment of QoL Appraises technology-enabled methods for incorporating QoL measurements in medicine Highlights the success factors for adoption and scaling of technology-enabled methods This open access book presents the rise of technology-enabled methods and tools for objective, quantitative assessment of Quality of Life (QoL), while following the WHOQOL model. It is an in-depth resource describing and examining state-of-the-art, minimally obtrusive, ubiquitous technologies. Highlighting the required factors for adoption and scaling of technology-enabled methods and tools for QoL assessment, it also describes how these technologies can be leveraged for behavior change, disease prevention, health management and long-term QoL enhancement in populations at large. Quantifying Quality of Life: Incorporating Daily Life into Medicine fills a gap in the field of QoL by providing assessment methods, techniques and tools. These assessments differ from the current methods that are now mostly infrequent, subjective, qualitative, memory-based, context-poor and sparse. Therefore, it is an ideal resource for physicians, physicians in training, software and hardware developers, computer scientists, data scientists, behavioural scientists, entrepreneurs, healthcare leaders and administrators who are seeking an up-to-date resource on this subject

Crossmodal Load and Selective Attention

This thesis explores a current dominant theory of attention - the load theory of selective attention and cognitive control (Lavie et al., 2004). Load theory has been posited as a potential resolution to the long-running debate over the locus of selection in attention. Numerous studies confirm that high visual perceptual load in a relevant task leads to reduced interference from task-relevant distractors; whereas high working memory load leads to increased interference from task-irrelevant distractors in a relevant task. However, very few studies have directly tested perceptual and working memory load effects on the processing of task-relevant stimuli, and even fewer studies have tested the impact of load on processing both within and between different sensory modalities. This thesis details several novel experiments that test both visual and auditory perceptual and working memory load effects on task-relevant change detection in a change-blindness “flicker” task. Results indicate that both high visual and auditory perceptual load can impact on change detection, which implies that the perceptual load model can account for load effects on change detection, both within and between different sensory modalities. Results also indicate that high visual working memory load can impact on change detection. By contrast, high auditory working memory load did not appear to impact change detection. These findings do not directly challenge load theory per-se, but instead highlight how working memory load can have markedly different effects in different experimental paradigms. The final part of this thesis explores whether high perceptual load can attenuate distraction from highly emotionally salient stimuli. The findings suggest that potent emotional stimuli can “breakthrough” and override the effects of high perceptual load - a result that presents a challenge to load theory. All findings are discussed with reference to new challenges to load theory, particularly the “dilution” argument

The Impact of Trait Anxiety on Slow-Wave Sleep Processes.

The sleep of high anxiety/worry and low anxiety/worry subjects was studied by means of nocturnal polysomnography. Fifteen high anxiety adult subjects were chosen to represent an analog Generalized Anxiety Disorder (GAD) population, and were selected from among individuals scoring within the upper 20th percentile on the trait portion of the State-Trait Anxiety Inventory (STAI) and the Penn State Worry Questionnaire (PSWQ). Fifteen low anxiety subjects were selected from among individuals scoring within the lower 50th percentile on those measures, and were matched with high anxiety subjects for age, sex, race, and Beck Depression Inventory (BDI) scores. Analysis of polysomnography variables revealed that high anxiety subjects took longer to fall asleep, had a smaller percentage of deep (slow-wave) sleep, and a larger percentage of light transitional sleep (stage 1 NREM). High anxiety subjects also had more stage 1 NREM periods, more micro-arousals during the first half of the sleep period, and a lower REM density relative to low anxiety subjects. In addition, high anxiety subjects had more electrodermal storming when SWS and REM sleep variables were co-varied. Results indicated that the pattern of disruption of sleep depth and continuity was very similar to that documented in clinical anxiety disorder patients, and was distinct from that typically shown in depressed patients. A major implication is that anxiety and worry are associated with disrupted sleep, independent of whether or not a diagnosed anxiety disorder is present. Current results indicate that these factors are associated with the sleep disruption observed in anxiety disorders, independent of the presence of panic, previous exposure to trauma, or a disease process per se

Quantifying Quality of Life

Describes technological methods and tools for objective and quantitative assessment of QoL Appraises technology-enabled methods for incorporating QoL measurements in medicine Highlights the success factors for adoption and scaling of technology-enabled methods This open access book presents the rise of technology-enabled methods and tools for objective, quantitative assessment of Quality of Life (QoL), while following the WHOQOL model. It is an in-depth resource describing and examining state-of-the-art, minimally obtrusive, ubiquitous technologies. Highlighting the required factors for adoption and scaling of technology-enabled methods and tools for QoL assessment, it also describes how these technologies can be leveraged for behavior change, disease prevention, health management and long-term QoL enhancement in populations at large. Quantifying Quality of Life: Incorporating Daily Life into Medicine fills a gap in the field of QoL by providing assessment methods, techniques and tools. These assessments differ from the current methods that are now mostly infrequent, subjective, qualitative, memory-based, context-poor and sparse. Therefore, it is an ideal resource for physicians, physicians in training, software and hardware developers, computer scientists, data scientists, behavioural scientists, entrepreneurs, healthcare leaders and administrators who are seeking an up-to-date resource on this subject

Exploring cognition in visual search and vigilance tasks with eye tracking and pupillometry

Recent findings in experimental psychology suggest that pupillometry, the measurement of pupil size, can provide insight into cognitive processes associated with effort and target detection in visual search tasks and monitoring performance in vigilance tasks. With the increasing availability, affordability and flexibility of video-based eye tracking hardware, these experimental findings point to lucrative practical applications such as real-time biobehavioural monitoring systems to assist with socially important tasks in operational settings. The aim of the current thesis was to explore this potential with further experimental work paying close attention to methodological issues which complicate cognitive interpretations of pupillary responses, such as physical stimulus confounds and eye movement-related measurement error in video-based systems. Six original experiments were designed to specifically explore the relationship between pupil size, cognition and behavioural performance in classic visual search and vigilance paradigms. Experiments 1-2 examined the pupillometric effects of effort and target detection in visual search with briefly presented stimuli. Pupil responses showed small variability with respect to manipulations of set size and target presence but were influenced substantially by the requirement for a motor response. Experiments 3-4 explored the cognitive pupil dynamics of free-viewing visual search with data-driven correction for eye movement artefacts. Group-level averages revealed small transient pupil dilations following fixations on targets but not distractors, an effect which was not contingent on a motor response or correction for gaze position artefacts. Experiments 5-6 looked at the relationship between pupil size and detection performance measures in two types of vigilance task. Changes in baseline and stimulus-evoked pupil responses loosely mirrored changes in performance, but the relationships were neither linear nor consistent. Overall, the thesis affirms the practical potential for using cognitive pupillometry in research and applied settings, but emphasises the constraints arising from methodological and theoretical limitations