Using video games to study the acquisition and performance of psychomotor skills

Understanding how humans learn complex skills is a fundamental aim of cognitive science. Digital games offer promising opportunities to study cognitive factors associated with skill acquisition and performance, as they motivate longitudinal engagement and produce rich, multivariate data sets. By applying mutlivariate analysis techniques to data arising from gameplay, this thesis extended the literature on cognition as it pertains to psychomotor skill. We describe three studies that were conducted in this regard. In the first study, we analyzed the relationship between the temporal distribution of play instances and performance in a commercial digital game (League of Legends). Using clustering techniques and big data, we demonstrated that players who cram gameplay into short time frames ultimately perform worse than those who space the same number of games over longer periods. In the second study, we examined an experimental data set of participants who played Meta-T, a laboratory version of Tetris. Using Principal Components Analysis and regression techniques, we identified cognitive-behavioural markers of performance, such as action-latency and motor coordination. We also applied Hidden Markov models (HMM) to time series of these markers, showing that moment-to-moment dynamics in performance can be segmented into behavioural states related to latent psychological states. In the third study, we investigated the neural correlates of behavioural states during performance. Using simultaneous MEG and behavioural recordings of participants playing Tetris, we segmented time series datasets of neural activity based on time stamps of behavioural epochs derived from HMMs. We compared behavioural epochs based on neural markers, showing that cognitive states derived from multivariate behavioural data correlate with neural activity in the alpha band power. Taken together, this thesis advances our understanding of using digital game data to study cognition and learning. It demonstrates the feasibility of recording high-density neuroimaging data during complex behavioural tasks and obtaining reliable measures of internal neuronal states during complex behaviour

Évaluation et modulation des fonctions exécutives en neuroergonomie - Continuums cognitifs et expérimentaux

Des études en neuroergonomie ont montré que le pilote d’avion pouvait commettre des erreurs en raison d’une incapacité transitoire à faire preuve de flexibilité mentale. Il apparait que certains facteurs, tels qu’une forte charge mentale ou une pression temporelle importante, un niveau de stress trop élevé, la survenue de conflits, ou une perte de conscience de la situation, peuvent altérer temporairement l’efficience des fonctions exécutives permettant cette flexibilité. Depuis mes travaux initiaux, dans lesquels je me suis intéressé aux conditions qui conduisent à une négligence auditive, j’ai souhaité développer une approche scientifique visant à quantifier et limiter les effets délétères de ces différents facteurs. Ceci a été fait à travers l’étude des fonctions exécutives chez l’être humain selon le continuum cognitif (du cerveau lésé au cerveau en parfait état de fonctionnement) et le continuum expérimental (de l’ordinateur au monde réel). L’approche fondamentale de l’étude des fonctions exécutives en neurosciences combinée à l’approche neuroergonomique graduelle avec des pilotes et des patients cérébro-lésés, a permis de mieux comprendre la manière dont ces fonctions sont mises en jeu et altérées. Cette connaissance à contribuer par la suite à la mise en place de solutions pour préserver leur efficacité en situation complexe. Après avoir rappelé mon parcours académique, je présente dans ce manuscrit une sélection de travaux répartis sur trois thématiques de recherche. La première concerne l’étude des fonctions exécutives impliquées dans l’attention et en particulier la façon dont la charge perceptive et la charge mentale peuvent altérer ces fonctions. La deuxième correspond à un aspect plus appliqué de ces travaux avec l’évaluation de l’état du pilote. Il a été question d’analyser cet état selon l’activité de pilotage elle-même ou à travers la gestion et la supervision d’un système en particulier. La troisième et dernière thématique concerne la recherche de marqueurs prédictifs de la performance cognitive et l’élaboration d’entraînements cognitifs pour limiter les troubles dysexécutifs, qu’ils soient d’origine contextuelle ou lésionnelle. Ces travaux ont contribué à une meilleure compréhension des troubles cognitifs transitoires ou chroniques, mais ils ont aussi soulevé des questions auxquelles je souhaite répondre aujourd’hui. Pour illustrer cette réflexion, je présente en dernière partie de ce document mon projet de recherche qui vise à développer une approche multifactorielle de l’efficience cognitive, éthique et en science ouverte

Virtual Reality Games for Motor Rehabilitation

This paper presents a fuzzy logic based method to track user satisfaction without the need for devices to monitor users physiological conditions. User satisfaction is the key to any product’s acceptance; computer applications and video games provide a unique opportunity to provide a tailored environment for each user to better suit their needs. We have implemented a non-adaptive fuzzy logic model of emotion, based on the emotional component of the Fuzzy Logic Adaptive Model of Emotion (FLAME) proposed by El-Nasr, to estimate player emotion in UnrealTournament 2004. In this paper we describe the implementation of this system and present the results of one of several play tests. Our research contradicts the current literature that suggests physiological measurements are needed. We show that it is possible to use a software only method to estimate user emotion

Pedestrian velocity obstacles: pedestrian simulation through reasoning in velocity space

We live in a populous world. Furthermore, as social animals, we participate in activities which draw us together into shared spaces -- office buildings, city sidewalks, parks, events (e.g., religious, sporting, or political), etc. Models that can predict how crowds of humans behave in such settings would be valuable in allowing us to analyze the designs for novel environments and anticipate issues with space utility and safety. They would also better enable robots to safely work in a common environment with humans. Furthermore, credible simulation of crowds of humans would allow us to populate virtual worlds, helping to increase the immersive properties of virtual reality or entertainment applications. We propose a new model for pedestrian crowd simulation: Pedestrian Velocity Obstacles (PedVO). PedVO is based on Optimal Reciprocal Collision Avoidance (ORCA), a local navigation algorithm for computing optimal feasible velocities which simultaneously avoid collisions while still allowing the agents to progress toward their individual goals. PedVO extends ORCA by introducing new models of pedestrian behavior and relationships in conjunction with a modified geometric optimization planning technique to efficiently simulate agents with improved human-like behaviors. PedVO introduces asymmetric relationships between agents through two complementary techniques: Composite Agents and Right of Way. The former exploits the underlying collision avoidance mechanism to encode abstract factors and the latter modifies the optimization algorithm's constraint definition to enforce asymmetric coordination. PedVO further changes the optimization algorithm to more fully encode the agent's knowledge of its environment, allowing the agent to make more intelligent decisions, leading to a better utilization of space and improved flow. PedVO incorporates a new model, which works in conjunction with the local planning algorithm, to introduce a ubiquitous density-sensitive behavior observed in human crowds -- the so-called "fundamental diagram." We also provide a physically-plausible, interactive model for simulating walking motion to support the computed agent trajectories. We evaluate these techniques by simulating various scenarios, such as pedestrian experiments and a challenging real-world scenario: simulating the performance of the Tawaf, an aspect of the Muslim Hajj.Doctor of Philosoph

Temporally scattered brain: neural mechanism apprehending the paradox of the discrete and continuous flow of consciousness

In dieser Dissertation werden offene Fragen über „Bewußtsein“ aus neurowissenschaftlicher, psychologischer und auch philosophischer Sicht erörtert. Es geht insbesondere darum, wie in traditionellen und modernen Ansätzen die Beziehung von zeitlicher Informations-Verarbeitung und Repräsentationen im Bewußtsein konzipiert werden. In einer historischen Aufarbeitung wird gezeigt, dass Zeit als mentale Kategorie in der psychologischen und neurowissenschaftlichen Forschung lange eher vernachlässigt worden ist, was sich erst jetzt zu ändern scheint. Wie wichtig Informations-Verarbeitung im Zeitbereich für ein besseres Verständnis von Bewußtseinsprozessen ist, wird herausgearbeitet, denn alle Verhaltensweisen und alles bewußte Erleben haben notwendigerweise eine zeitliche Charakteristik. Die grundsätzliche aber bisher nicht hinreichend beantwortete Frage ist, ob diese Charakteristik, die Informations-Verarbeitung in der Zeit, als kontinuierlich oder als diskret zu verstehen ist. Neue Befunde legen nahe, dass so genannte „Zeitfenster“ bestimmter Dauer notwendig sind, um neuronale Informations-Verarbeitung zu ermöglichen, damit Bewußtseinsprozesse überhaupt entstehen können. Wenn aber diskrete Zeitfenster notwendig sind, dann stellt sich die weitere Frage, wie es paradoxerweise zum subjektiven Eindruck einer zeitlichen Kontinuität zum Beispiel in der Wahrnehmung kommen kann. Das Erleben zeitlicher Kontinuität bezieht sich nicht nur auf Prozesse der Wahrnehmng und des Erlebens, sondern auch auf den viel weiteren Rahmen der Kontinuität personaler Identität. Es wird darauf hingewiesen, dass diese Kontinuität bei bestimmten Erkrankungen oder veränderten Bewußtseinszuständen verloren gehen kann. Ein derartiger Verlust legt nahe, dass es aktive Mechanismen auf neuronaler Ebene geben sollte, die personale Identität über die Zeit hinweg erzeugen. Es mußt gleichsam ein „Klebstoff“ vermutet werden, der zeitlich Diskretes in anschauliche Kontinuität verwandelt. Mechanismen, die hierfür in Frage kommen könnten, sind bisher nicht bekannt. Ein erster Versuch zur Aufklärung dieser Frage wird mit einem fMRT-Experiment gemacht, in dem visuelle und auditive Vorstellungen untersucht werden. Hier zeigt sich, dass in beiden Modalitäten gemeinsame neuronale Aktivierungen in bestimmten Hirnstrukturen zu beobachten sind, was möglicherweise einen ersten Hinweis auf die Erzeugung von anschaulicher Kontinuität geben könnte. Des weiteren wird am Ende der Arbeit auf die Bedeutung des Reafferenzprinzips hingewiesen, das vielleicht einen neuen Ansatz zum besseren Verständnis mancher Phänomene wie des Déjà Vu geben könnte, wenn man in dieses klassische Prinzip einen Zeitfaktor integriert, was bisher theoretisch nicht geschehen ist. Während der Fokus der Arbeit auf theoretischen Konzepten zu Zeit und Bewußtsein im psychologischen, neurowissenschaftlichen und philosophischen Kontext liegt, werden auch Bezüge zu den Künsten, der Dichtkunst und der Musik, offen gelegt. Dies soll darauf hinweisen, dass die Beziehung zwischen der „Zeit des Menschen“ und dem bewußten Erleben ein Menschheitsthema ist, das über den wissenschhaftlichen Rahmen hinaus weist.In this dissertation open questions about “consciousness” are discussed from a psychological, neuroscientific and also philosophical perspective. In particular, it is described how the relationship between temporal information processing and conscious representations is conceived in traditional and modern approaches. In a historical review it is shown that time as a mental category has been neglected for a long time in psychological and neuroscientific research, although this is changing recently. The importance of temporal information processing for a better understanding of conscious processes is analyzed because any behavior and all subjective experiences necessarily have a temporal characteristic. There is, however, a basic and still open question whether this characteristic, i.e., temporal information processing, is continuous or discrete. New research suggests that “time windows” of specific durations are necessary in neural information processing being the basis for conscious representations. If, however, discrete time windows are necessary the question arises how paradoxically the subjective impression of temporal continuity for instance in perception is possible. The impression of temporal continuity refers, however, not only to processes of perception and experience, but in a broader context also to the continuity of personal identity. It is indicated that this continuity can break down in certain diseases or in altered states of consciousness. Such losses suggest the existence of active mechanisms on the neural level which allow for the creation of personal identity across time. Some kind of “glue” has to be suspected that transforms what is temporally discrete into apparent continuity. Potential mechanisms for this transformation are still not known. A first attempt is made to answer this question with an fMRI-experiment in which visual and auditory images are analyzed. For both modalities’ common activations in certain brain regions are observed which possibly might be a first indication about a mechanism creating apparent continuity. At the end of the dissertation the importance of the reafference principle is stressed as it may provide a new perspective towards a better understanding of some phenomena like Déja Vu, if one includes in this classical principle a temporal factor which theoretically has not been done yet. Although the focus of the dissertation lies on theoretical concepts of time and consciousness within a psychological, neuroscientific and philosophical context, some links to the arts like poetry and music are made transparent. This shall indicate that the relationship between the “time of humans” and conscious experiences is a topic of humankind that goes beyond the scientific frame

Investigating the modulation of cognition and event-related potentials relating to visual attention, working memory, and executive control in habitual videogame players

The overall objective of this thesis was to produce a document that investigated whether habitual videogame playing modulated cognitive processes related to visual processing and where in the processing stream these modulations occur. In this thesis, the term ‘cognitive modulation’ refers to any neurological differences (as identified through ERP) between videogame players and non-videogame players that theoretically may have been a result of videogame playing. Using this method, I am able to ascertain whether differences between the two groups are observed in early sensory ERPs, in which case VGPs might possess an advantage in bottom-up visual processing, later selective attention which might indicate alterations in top-down attentional processing, motor-response waveforms that may indicate difference in stimulus response mappings, and finally any differences in working memory capacity that might be the underlying cause of supposed attentional differences. An example of cognitive modulation was observed by Wu et al., (2012) and discussed in more detail in the introduction of this thesis. Indeed these modulations should also be accompanied by a behavioural difference between the two groups. As ERP was the primary source of neurophysiological recordings in this thesis, modulations could occur in the amplitude, mean activity, or peak latency of ERP waveforms.The paradigms employed in this thesis were chosen and designed so that in combination they provide a measure of potential cognitive modulation across the entire processing stream. That is, from early sensory ERPs, through selective attentional ERPs, including executive control ERPs and concluding at ERPs related to motor response priming. As these studies primarily focused on attentional processes, an ERP chapter towards the end of this thesis was included to identify whether any modulations in attentional ERPs were an indirect result of modulated working memory.Chapters 2 and 3 in this thesis focus on attentional control, resources, and the inhibitory processes of attention. Specifically, these chapters related to the attentional control each group employed when being presented with distracting items. Indeed, I observed modulated cognitive processes in chapter 3 related inhibitory processing in both attention and executive control related processes. In addition to this, the flanker task in Chapter 3 also allowed me to measure and modulation in motor priming between videogame players and non-videogame players.Chapters 4 and 5 looked more closely at ERPs related to selective attention such as the N2pc and P3, alongside early sensory ERPs (N1, P1, etc.). In response to observing differences in how each group processed distractors (related to the N2pc in Chapter 4), Chapter 5 employed a very specific test in order to split the N2pc into its component parts to further investigate whether any cognitive modulation between groups was a result of altered priority on processing targets or inhibiting distractors.Chapter 6 in this thesis sought to identify whether any differences observed in the attentional processing stream was actually the result of modulations in working memory, a cognitive process theoretically closely related to selective attention. Chapter 6 measured the contralateral delay activity, a neurological waveform that correlates with items held in visual working memory.Chapter 7 provided further exploratory psychophysical testing to identify whether any potential behavioural between-group differences extend beyond the usual visual field our groups would play videogames in. This involved testing the crowding phenomenon whereby participants are unable to identify a stimulus when closely flanked by distractors.This these concludes with Chapter 8, an overall discussion of each chapters results and how these theoretically synthesise with one another in relation to the two objectives of this thesis; does videogame playing modulate cognitive, and where in the attentional processing stream does this occur

Switch choice in applied multi-task management

2014 Summer.Little to date is known concerning how operators make choices in environments where cognitive load is high and they are faced with multiple different tasks to choose from. This dissertation reviewed a large body of voluntary task switching literature concerning basic research into choice in task switching, as well as what literature was available for applied task switching. From this and a prior model, a revised model of task switching choice that takes into account specific task attributes of difficulty, priority, interest and salience, was developed. In the first experiment, it was shown that task difficulty and priority influenced switching behavior. While task attributes were hypothesized to influence switching, a second major influence is time on task. In the second experiment, it was shown that tasks indeed vary in their interruptability over time, and this was related in part to what task was competing for attention as well as the cognitive processing required for the ongoing task performance. In a third experiment, a new methodology was developed to experimentally assess the role of diminishing rate of returns for performing a task. This declining rate was expected (and did result in) a general increase of switching away from an ongoing task over time. In conclusion, while task attributes and time on task play a major role in task switching in the current studies, defining the time period for theorized effects appears to be the next major step toward understanding switching choice behavior. Additionally, though the experiments are novel and certainly make a major contribution, to the extent that behavior is only represented in them, the methodology may miss some amount of `other' task behavior, such as visual sampling

Architectures for ubiquitous 3D on heterogeneous computing platforms

Today, a wide scope for 3D graphics applications exists, including domains such as scientific visualization, 3D-enabled web pages, and entertainment. At the same time, the devices and platforms that run and display the applications are more heterogeneous than ever. Display environments range from mobile devices to desktop systems and ultimately to distributed displays that facilitate collaborative interaction. While the capability of the client devices may vary considerably, the visualization experiences running on them should be consistent. The field of application should dictate how and on what devices users access the application, not the technical requirements to realize the 3D output. The goal of this thesis is to examine the diverse challenges involved in providing consistent and scalable visualization experiences to heterogeneous computing platforms and display setups. While we could not address the myriad of possible use cases, we developed a comprehensive set of rendering architectures in the major domains of scientific and medical visualization, web-based 3D applications, and movie virtual production. To provide the required service quality, performance, and scalability for different client devices and displays, our architectures focus on the efficient utilization and combination of the available client, server, and network resources. We present innovative solutions that incorporate methods for hybrid and distributed rendering as well as means to manage data sets and stream rendering results. We establish the browser as a promising platform for accessible and portable visualization services. We collaborated with experts from the medical field and the movie industry to evaluate the usability of our technology in real-world scenarios. The presented architectures achieve a wide coverage of display and rendering setups and at the same time share major components and concepts. Thus, they build a strong foundation for a unified system that supports a variety of use cases.Heutzutage existiert ein großer Anwendungsbereich für 3D-Grafikapplikationen wie wissenschaftliche Visualisierungen, 3D-Inhalte in Webseiten, und Unterhaltungssoftware. Gleichzeitig sind die Geräte und Plattformen, welche die Anwendungen ausführen und anzeigen, heterogener als je zuvor. Anzeigegeräte reichen von mobilen Geräten zu Desktop-Systemen bis hin zu verteilten Bildschirmumgebungen, die eine kollaborative Anwendung begünstigen. Während die Leistungsfähigkeit der Geräte stark schwanken kann, sollten die dort laufenden Visualisierungen konsistent sein. Das Anwendungsfeld sollte bestimmen, wie und auf welchem Gerät Benutzer auf die Anwendung zugreifen, nicht die technischen Voraussetzungen zur Erzeugung der 3D-Grafik. Das Ziel dieser Thesis ist es, die diversen Herausforderungen zu untersuchen, die bei der Bereitstellung von konsistenten und skalierbaren Visualisierungsanwendungen auf heterogenen Plattformen eine Rolle spielen. Während wir nicht die Vielzahl an möglichen Anwendungsfällen abdecken konnten, haben wir eine repräsentative Auswahl an Rendering-Architekturen in den Kernbereichen wissenschaftliche Visualisierung, web-basierte 3D-Anwendungen, und virtuelle Filmproduktion entwickelt. Um die geforderte Qualität, Leistung, und Skalierbarkeit für verschiedene Client-Geräte und -Anzeigen zu gewährleisten, fokussieren sich unsere Architekturen auf die effiziente Nutzung und Kombination der verfügbaren Client-, Server-, und Netzwerkressourcen. Wir präsentieren innovative Lösungen, die hybrides und verteiltes Rendering als auch das Verwalten der Datensätze und Streaming der 3D-Ausgabe umfassen. Wir etablieren den Web-Browser als vielversprechende Plattform für zugängliche und portierbare Visualisierungsdienste. Um die Verwendbarkeit unserer Technologie in realitätsnahen Szenarien zu testen, haben wir mit Experten aus der Medizin und Filmindustrie zusammengearbeitet. Unsere Architekturen erreichen eine umfassende Abdeckung von Anzeige- und Rendering-Szenarien und teilen sich gleichzeitig wesentliche Komponenten und Konzepte. Sie bilden daher eine starke Grundlage für ein einheitliches System, das eine Vielzahl an Anwendungsfällen unterstützt