Modality switching and performance in a thought and speech controlled computer game

Providing multiple modalities to users is known to improve the overall performance of an interface. Weakness of one modality can be overcome by the strength of another one. Moreover, with respect to their abilities, users can choose between the modalities to use the one that is the best for them. In this paper we explored whether this holds for direct control of a computer game which can be played using a brain-computer interface (BCI) and an automatic speech recogniser (ASR). Participants played the games in unimodal mode (i.e. ASR-only and BCI-only) and multimodal mode where they could switch between the two modalities. The majority of the participants switched modality during the multimodal game but for the most of the time they stayed in ASR control. Therefore multimodality did not provide a significant performance improvement over unimodal control in our particular setup. We also investigated the factors which influence modality switching. We found that performance and peformance-related factors were prominently effective in modality switching

Say-it: Design of a Multimodal Game Interface for Children Based on CMU Sphinx 4 Framework

Nowadays, computer games are involved in a child’s education as tools for learning or practicing some academic skills. However, most educational games are designed without any considerations about children with motor system difficulties. Thus, the main benefit of multimodal interfaces is to allow for inclusive design, which will enable children with motor disabilities to use the same applications other children use. Since focus and concentration are major skills for children in the learning process, many physical games can be performed with assist from parents or teachers to practice these skills. This project aims to explore the effectiveness and the use of multimodal applications as computer-based exercises by implementing a multimodal system that offers an inclusive design for three interactive games to practice the skill to focus. In this project, Say-it is implemented as a functional multimodal prototype to demonstrate the value of multimodal interfaces in the education of young children. It is designed to provide the advantages of both current physical games and computer exercises and make these exercises available for children with wide rang of abilities. Also, Say-it can be considered as an experimental prototype to explore the performance of using CMU Sphinx 4 framework as an underlying speech recognition tool for the application. The results of using Say-it show that multimodal games can be designed using the existing speech recognition technologies, such as CMU Sphinx 4 framework. Also, using different input modalities in the proposed prototype makes the games more enjoyable and challenging for children

Brain activations related to attention and working memory and their association with technology-mediated activities

Executive functions are pivotal in our everyday lives, as they form the basis for complex and goal-directed behavior. For example, the ability to maintain information in memory while making a decision requires executive processes. Whether or not executive functions can exhibit experience-dependent changes is still a topic of debate, but generally accepted principles of brain plasticity suggest that environmental factors can have an impact on cognitive processes and the activity and structure of their respective brain networks. One such environmental factor is the increasingly ubiquitous daily interaction with technology, which has been suggested to affect mental faculties such as the ability to maintain focus on a single task or to actively maintain information in short-term memory. The aim of the present thesis was to study activity in cortical networks of attention and working memory. In addition, we investigated whether any associations could be found between the recruitment of these networks or performance speed and accuracy in working memory and attention tasks, and the extent of daily technology-mediated activities reported by adolescent and young adult participants. In all studies, functional magnetic resonance imaging (fMRI) was used to record brain activity during task performance. By using novel experimental paradigms, the present results shed more light on the specific cortical networks recruited by different executive functions by showing that both common and specific brain regions are recruited by auditory and visual selective attention, divided attention and working memory processes. Furthermore, they demonstrate that during division of attention between two concurrent tasks (listening to speech and reading text), competition for neural resources in regions shared by the component tasks is a major contributor to performance limitations observed during multitasking. Importantly, the results of the present thesis also demonstrate that detectable associations exist between different types of daily technology use and cognitive functioning already in adolescence. More specifically, the results demonstrate that a tendency to use several media simultaneously (i.e., media multitasking) is related to increased distractibility. The extent of computer gaming in daily life, in turn, is associated with enhanced working memory functioning. These findings are of great importance, since it is vital to understand how the increasing amount of on-screen time might affect or interact with the cognitive and brain functioning of the current youth.Toiminnanohjaus käsittää joukon toimintoja, jotka mahdollistavat tavoitteellinen ja monimutkaisen toiminnan jokapäiväisissä tilanteissa. Esimerkki toiminnanohjauksesta on kyky ylläpitää tietoa muistissa samalla kun tekee päätöksen. On edelleen epäselvää, kuinka suuri vaikutus kokemuksella voi olla toiminnanohjaukseen lukeutuviin toimintoihin, mutta yleisesti hyväksytyt aivojen muovautuvuuteen liittyvät periaatteet antavat syyn olettaa, että ympäristötekijöiden on mahdollista vaikuttaa kognitiiviseen suoriutumiseen ja niihin liittyviin aivoverkostoihin. Jatkuvasti lisääntyvä teknologian parissa vietetty aika on yksi niistä ympäristötekijöistä, joiden on ehdotettu vaikuttavan kognitiivisiin toimintoihin kuten kykyyn keskittyä yhteen tehtävään samanaikaisesti, tai kykyyn ylläpitää tietoa lyhytkestoisessa muistissa. Tässä esitellyn väitöskirjatyön tavoite oli tutkia tarkkaavaisuuteen ja työmuistiin liittyviä aivoverkostoja. Lisäksi selvitettiin sitä, onko näiden aivoverkostojen toiminnalla ja tarkkaavaisuus- ja työmuistitehtävissä suoriutumisella yhteyksiä nuorten ja nuorten aikuisten itseraportoituihin teknologiankäyttötapoihin. Kaikissa väitöskirjan osatutkimuksissa käytettiin toiminnallista magneettiresonanssikuvantamista (fMRI) mittaamaan aivojen aktivoitumista tehtäväsuorituksen aikana. Käyttämällä uusia ja innovatiivisia koeasetelmia, tutkimuksemme tulokset tuottivat lisää tietoa eri toiminnanohjaukseen liittyvien toimintojen aktivoimista aivoverkostoista näyttämällä, että valikoiva tarkkaavaisuus, jaettu tarkkaavaisuus ja työmuistiprosessit aktivoivat sekä yhteisiä että erillisiä aivoalueita. Lisäksi tuloksemme osoittivat, että jaettaessa tarkkaavaisuutta kahden samanaikaisen tehtävän kesken, kilpailu hermostollisista resursseista näiden kahden samaa aivoaluetta kuormittavan tehtävän välillä vaikuttaa oleellisesti ihmisen rajalliseen monisuorittamiskykyyn. Tuloksemme osoittavat myös, että päivittäisten teknologian käyttötapojen ja kognitiivisen suoriutumisen välillä on havaittavia yhteyksiä jo nuoruusiässä. Taipumus käyttää montaa mediaa samanaikaisesti (nk. media multitasking) oli yhteydessä suurempaan häiriintyvyyteen, kun taas tietokonepelien pelaaminen oli yhteydessä parempaan työmuistisuoriutumiseen. Nämä tulokset ovat erittäin merkityksellisiä, sillä on tärkeää ymmärtää, minkälaisia mahdollisia vaikutuksia nuorten alati kasvavalla ruutuajalla on heidän kognitiivisiin toimintoihinsa ja aivojen toimintaan

Heavy Hero or Digital Dummy? Multimodal Player–Avatar Relations in Final Fantasy 7

This article analyses the player-avatar relation in Final Fantasy 7, drawing on multimodality theory to analyse textual structures both in the game and in the discourse of player-interviews and fan writing. It argues that the avatar is a two-part structure, partly designed in conventional narrative terms as a protagonist of popular narrative, and partly as a vehicle for interactive game-play. The former structure is replete with the traditions and designs of Japanese popular narrative, oral formulaic narrative and contemporary superhero narratives; and is presented to the player as an offer act – a declarative narrative statement. The latter is a construct of evolving attributes and economies characteristic of roleplaying games; and is presented to the player as a demand act – a rule-based command. Though these two functions separate out in the grammar of player and fan discourse, it is their integration which provides the pleasure of gameplay and narrative engagement

Combining brain-computer interfaces and assistive technologies: state-of-the-art and challenges

In recent years, new research has brought the field of EEG-based Brain-Computer Interfacing (BCI) out of its infancy and into a phase of relative maturity through many demonstrated prototypes such as brain-controlled wheelchairs, keyboards, and computer games. With this proof-of-concept phase in the past, the time is now ripe to focus on the development of practical BCI technologies that can be brought out of the lab and into real-world applications. In particular, we focus on the prospect of improving the lives of countless disabled individuals through a combination of BCI technology with existing assistive technologies (AT). In pursuit of more practical BCIs for use outside of the lab, in this paper, we identify four application areas where disabled individuals could greatly benefit from advancements in BCI technology, namely,“Communication and Control”, “Motor Substitution”, “Entertainment”, and “Motor Recovery”. We review the current state of the art and possible future developments, while discussing the main research issues in these four areas. In particular, we expect the most progress in the development of technologies such as hybrid BCI architectures, user-machine adaptation algorithms, the exploitation of users’ mental states for BCI reliability and confidence measures, the incorporation of principles in human-computer interaction (HCI) to improve BCI usability, and the development of novel BCI technology including better EEG devices

Telephone conversation impairs sustained visual attention via a central bottleneck

Recent research has shown that holding telephone conversations disrupts one's driving ability. We asked whether this effect could be attributed to a visual attention impairment. In Experiment 1, participants conversed on a telephone or listened to a narrative while engaged in multiple object tracking (MOT), a task requiring sustained visual attention. We found that MOT was disrupted in the telephone conversation condition, relative to single-task MOT performance, but that listening to a narrative had no effect. In Experiment 2, we asked which component of conversation might be interfering with MOT performance. We replicated the conversation and single-task conditions of Experiment 1 and added two conditions in which participants heard a sequence of words over a telephone. In the shadowing condition, participants simply repeated each word in the sequence. In the generation condition, participants were asked to generate a new word based on each word in the sequence. Word generation interfered with MOT performance, but shadowing did not. The data indicate that telephone conversation disrupts attention at a central stage, the act of generating verbal stimuli, rather than at a peripheral stage, such as listening or speaking

Human-human multi-threaded spoken dialogs in the presence of driving

The problem addressed in this research is that engineers looking for interface designs do not have enough data about the interaction between multi-threaded dialogs and manual-visual tasks. Our goal was to investigate this interaction. We proposed to analyze how humans handle multi-threaded dialogs while engaged in a manual-visual task. More specifically, we looked at the interaction between performance on two spoken tasks and driving. The novelty of this dissertation is in its focus on the intersection between a manual-visual task and a multi-threaded speech communication between two humans. We proposed an experiment setup that is suitable for investigating multi-threaded spoken dialogs while subjects are involved in a manual-visual task. In our experiments one participant drove a simulated vehicle while talking with another participant located in a different room. The participants communicated using headphones and microphones. Both participants performed an ongoing task, which was interrupted by an interrupting task. Both tasks, the ongoing task and the interrupting task, were done using speech. We collected corpora of annotated data from our experiments and analyzed the data to verify the suitability of the proposed experiment setup. We found that, as expected, driving and our spoken tasks influenced each other. We also found that the timing of interruption influenced the spoken tasks. Unexpectedly, the data indicate that the ongoing task was more influenced by driving than the interrupting task. On the other hand, the interrupting task influenced driving more than the ongoing task. This suggests that the multiple resource model [1] does not capture the complexity of the interactions between the manual-visual and spoken tasks. We proposed that the perceived urgency or the perceived task difficulty plays a role in how the tasks influence each other