Lisätty todellisuus oppimisen tukena : Arttu-sovelluksen käyttäjien kokemuksia lisätyn todellisuuden hyödyllisyydestä

Tieto- ja viestintätekniikan kehittyminen on muuttanut oppimisen ja opetuksen luonnetta sekä tuonut uusia pedagogisia käytäntöjä ja erilaisia oppimisympäristön mahdollisuuksia. Perinteisten opetus- ja oppimisvälineiden rinnalle on tullut erilaisia opetusteknologioita. Yksi opetukseen mukaan otetuista mobiiliteknologioista on lisätty todellisuus, joka on tämän tutkimuksen keskiössä. Tämän pro gradu -tutkimuksen tavoitteena on selvittää, kokevatko opiskelijat lisätyn todellisuuden olevan hyödyllinen teknologia oppimisen tukena. Aineisto kerätään haastattelemalla Arttu-sovelluksen käyttäjiä K12-oppijaryhmästä harkinnanvaraisen otannan keinoin. Harkinnanvaraisen otannan ansiosta tutkimukseen saatiin osallistumaan myös henkilöita, joilla oli todettu lukemisen ja kirjoittamisen erityisvaikeus. Näin tutkimukseen saatiin näkökulma myös lisätyn todellisuuden käytöstä avustavana teknologiana. Haastattelut toteutetaan puolistrukturoidulla teemahaastattelulla, jossa haastattelu etenee etukäteen valittujen teemojen johdolla. Aineiston analyysi painottuu lisätyn todellisuuden hyödyllisyyttä tutkivaan lukuun. Tutkimusta ohjaa laadullinen tutkimusote, jota syvennetään osittain vertailevalla tutkimusotteella. Aineiston analyysi on toteutettu laadullisen sisällönanalyysin keinoin induktiivisella analyysillä, jonka avulla aineistosta on muodostettu yhdistäviä teemoja. Analyysi lähtee liikkeelle lisätyn todellisuuden havaitsemisesta, jonka jälkeen perehdytään lisätyn todellisuuden tuomiin hyötyihin oppimisympäristössä. Tämän jälkeen perehdytään tarkemmin vielä lisätyn todellisuuden tuomiin hyötyihin oppimisessa, jossa oppimisen arviointia peilataan Bloomin taksonomian kautta. Aineistosta nousi esiin yhteensä viisi teemaa, joiden alle edelleen sijoittui alateemoja. Teemat lisätyn todellisuuden vaikutus motivaatioon, lisätyn todellisuuden vaikutus suoriutumiseen ja lisätty todellisuus avustavana teknologiana vastasivat lisätyn todellisuuden hyötyihin oppimisympäristössä. Teemat lisätty todellisuus oppimisprosessin tukena ja lisätty todellisuus hahmottamisen tukena vastasivat taas lisätyn todellisuuden tuomiin hyötyihin opiskelijan oppimisprosessiin. Näiden teemojen alla löytyi alateemoja, jotka vahvistivat teemojen olemassaoloa. Tutkimuksen tulokset osoittivat, että opiskelijat, jotka käyttivät Arttu-sovellusta, kokivat sen tukevan opiskelua. Tätä vahvistaa se, että ainakin osa lisätyn todellisuuden toiminnoista toimi avustavan teknologian tavoin. Lisäksi opiskelijat kokivat lisätyn todellisuuden tuovan oppikirjoihin sellaisia toimintoja, jotka tukivat heidän kognitiivisia oppimisprosessajaan

Enhancing the Decoding Performance of Steady-State Visual Evoked Potentials based Brain-Computer Interface

Non-invasive Brain-Computer Interfaces (BCIs) based on steady-state visual evoked potential (SSVEP) responses are the most widely used BCI. SSVEP are responses elicited in the visual cortex when a user gazes at an object flickering at a certain frequency. In this thesis, we investigate different BCI system design parameters for enhancing the detection of SSVEP such as change in inter-stimulus distance (ISD), EEG channels, detection algorithms and training methodologies. Closely placed SSVEP stimuli compete for neural representations. This influences the performance and limits the flexibility of the stimulus interface. In this thesis, we study the influence of changing ISD on the decoding performance of an SSVEP BCI. We propose: (i) a user-specific channel selection method and (ii) using complex spectrum features as input to a convolutional neural network (C-CNN) to overcome this challenge. We also evaluate the proposed C-CNN method in a user-independent (UI) training scenario as this will lead to a minimal calibration system and provide the ability to run inference in a plug-and-play mode. The proposed methods were evaluated on a 7-class SSVEP dataset collected on 21 healthy participants (Dataset 1). The UI method was also assessed on a publicly available 12-class dataset collected on 10 healthy participants (Dataset 2). We compared the proposed methods with canonical correlation analysis (CCA) and CNN classification using magnitude spectrum features (M-CNN). We demonstrated that the user-specific channel set (UC) is robust to change in ISD (viewing angles of 5.24ᵒ, 8.53ᵒ, and 12.23ᵒ) compared to the classic 3-channel set (3C - O1, O2, Oz) and 6-channel set (6C - PO3, PO4, POz, O1, O2, Oz). A significant improvement in accuracy of over 5% (p=0.001) and a reduction in variation of 56% (p=0.035) was achieved across ISDs using the UC set compared to the 3C set and 6C set. Secondly, the proposed C-CNN method obtained a significantly higher classification accuracy across ISDs and window lengths compared to M-CNN and CCA. The average accuracy of the C-CNN increased by over 12.8% compared to CCA and an increase of over 6.5% compared to the M-CNN for the closest ISD across all window lengths was achieved. Thirdly, the C-CNN method achieved the highest accuracy in both UD and UI training scenarios on both 7-class and 12-class SSVEP Datasets. The overall accuracies of the different methods for 1 s window length for Dataset 1 were: CCA: 69.1±10.8%, UI-M-CNN: 73.5±16.1%, UI-C-CNN: 81.6±12.3%, UD-M-CNN: 87.8±7.6% and UD-C-CNN: 92.5±5%. And for Dataset 2 were: CCA: 62.7±21.5%, UI-M-CNN: 70.5±22%, UI-C-CNN: 81.6±18%, UD-M-CNN: 82.8±16.7%, and UD-C-CNN: 92.3±11.1%. In summary, using the complex spectrum features, the C-CNN likely learned to use both frequency and phase related information to classify SSVEP responses. Therefore, the CNN can be trained independent of the ISD resulting in a model that generalizes to other ISDs. This suggests that the proposed methods are robust to changes in inter-stimulus distance for SSVEP detection and provides increased flexibility for user interface design of SSVEP BCIs for commercial applications. Finally, the UI method provides a virtually calibration free approach to SSVEP BCI

A Hybrid Brain-Computer Interface Based on Electroencephalography and Functional Transcranial Doppler Ultrasound

Hybrid brain computer interfaces (BCIs) combining multiple brain imaging modalities have been proposed recently to boost the performance of single modality BCIs. We advance the state of hybrid BCIs by introducing a novel system that measures electrical brain activity as well as cerebral blood flow velocity using Electroencephalography (EEG) and functional transcranial Doppler ultrasound (fTCD), respectively. The system we developed employs two different paradigms to induce changes simultaneously in EEG and fTCD and to infer user intent. One of these paradigms includes visual stimuli to simultaneously induce steady state visually evoked potentials (SSVEPs) and instructs users to perform word generation (WG) and mental rotation (MR) tasks, while the other paradigm instructs users to perform left and right arm motor imagery (MI) tasks through visual stimuli. To improve accuracy and information transfer rate (ITR) of the proposed system compared to those obtained through our preliminary analysis, using classical feature extraction approaches, we mainly contribute to multi-modal fusion of EEG and fTCD features. Specifically, we proposed a probabilistic fusion of EEG and fTCD evidences instead of simple concatenation of EEG and fTCD feature vectors that we performed in our preliminary analysis. Experimental results showed that the MI paradigm outperformed the MR/WG one in terms of both accuracy and ITR. In particular, 93.85%, 93.71%, and 100% average accuracies and 19.89, 26.55, and 40.83 bits/min v average ITRs were achieved for right MI vs baseline, left MI versus baseline, and right MI versus left MI, respectively. Moreover, for both paradigms, the EEG-fTCD BCI with the proposed analysis techniques outperformed all EEG- fNIRS BCIs in terms of accuracy and ITR. In addition, to investigate the feasibility of increasing the possible number of BCI commands, we extended our approaches to solve the 3-class problems for both paradigms. It was found that the MI paradigm outperformed the MR/WG paradigm and achieved 96.58% average accuracy and 45 bits/min average ITR. Finally, we introduced a transfer learning approach to reduce the calibration requirements of the proposed BCI. This approach was found to be very efficient especially with the MI paradigm as it reduced the calibration requirements by at least 60.43%

Hacking as a playful strategy for designing the artistic and experimental BCI-VR game: ride your mind

Hacking is an ambiguous term. Over the past 50 years, its meaning has been constantly expanded and refined, filtered through several disciplines from divergent fields of application such as, for example, technology, computers, media, art, design, games and more. First used to describe what can be called a playful strategy employed to (creatively) solve a problem (Levy 1986), in public discourse the term hacking now often connotes a form of illicit behaviour in cyberspace. Today, the common perception is that hackers are rule-breakers and system-intruders who seek to do damage or even commit acts of war. In the 1950s, hackers helped transform computers from military devices into entertainment devices. This context swap (military to entertainment) forms the cradle of digital games and functions as the starting point of my research, which will seek to trace the history of hacking as a design strategy and to discover artistic strategies contained within the act of hacking itself. Hacking is, in fact, directly and historically related to computers and particularly to digital games. The first hacks were algorithmic visualisations and interactive programs, specifically interactive games; Spacewar! (1962) is the most famous example. To understand hacking as a strategy for designing games, I will explore historical and artistic approaches that have been used by hackers. In the late 1950s and early 1960s, hackers at the Massachusetts Institute of Technology (MIT) – more specifically, the students of the Tech Model Railroad Club (TMRC) – were introduced to MIT’s early digital computing machines. The members of TMRC launched a creative examination of this emerging computing technology; they were equipped with neither instructions nor experience, but they were driven by their goal of using the computer to create “art and beauty” (Levy 1986, p.31). Hackers do not have to be passionate computer users. As Eric S. Raymond (2013) asserts in The Jargon File, anyone can become a hacker. Hacking combines several creative strategies that are related to art, design and other creative disciplines. Like a hacker, I will create my own tools for conducting the proposed research, especially methodological ones. My basic methodological approach is to look at hacking in a chronological and historical way, in order to identify recurring design principles that are representative of hacking understood as a form of design practice. Many other fields intersect with the history of hacking, such as, for example, the history of computers and the history of computer games. Each of these three fields is also linked to fields such as philology, art history, the history of science, telecommunications engineering, economics and communication studies. The result is an as-of-yet undefined field of enquiry. This multi-dimensional context in which hacking exists poses a challenge: it is tempting to take detours in all manner of fascinating thematic and historical directions. To avoid straying from my topic, I will concentrate on the origins of hacking by investigating historical records like Steven Levy’s (1986) and Raymond’s (2013) and by comparing these to the broader context of hacking in the spirit of, for example, Claus Pias (2002b; 2002a; 2013) and Stephan Schwingeler (2012; 2014) and others. I will then synthesise the design elements I have identified and use them to outline a strategy for designing artistic games that will serve as the theoretical backbone for my own work as a media artist and, hopefully, for others’ work as well. I will combine the main strategic elements with an artistic approach into a game, which will constitute the practical element of this research. The outcome is based on the concept of an interactive, hackish neurofeedback real-time virtual-reality game art installation, or in brief, an artistic BCI-VR Game, titled Ride Your Mind (RYM). In the spirit of the early hackers, my research project Ride Your Mind (RYM) playfully explores, examines and hacks the possibilities of an emerging consumer technology, Brain-Computer Interfacing (BCI), from the perspective of a game artist and a game designer who is seeking to potentially create a BCI-VR Game. Initially, Brain-Computer Interfacing (BCI) games (games controlled/influenced by BCI) were used in medical and BCI research to successfully treat diseases such as ADHD (Nijholt et al. 2009, p.88). In the last years, hardware manufacturers such as Emotiv or Neurosky have begun producing consumer BCI technology, and the focus group for BCI has shifted to healthy users (Nijholt et al. 2008; Nijholt et al. 2009; Tan & Nijholt 2010; Loup-Escande et al. 2015; Martišius & Damaševičius 2016; Kerous et al. 2017; Chavarriaga et al. 2017; Vourvopoulos et al. 2017). The concept of RYM (Stober 2013) was developed in 2012 and presented in 2013 at the FROG Games Conference (Mitgutsch et al. 2013). RYM integrates various methods from academic and artistic disciplines, such as experimental and artistic game design, artistic practice, game design research, HCI and BCI research, computer science and neuroscience. Therefore, the approach in this PhD by project is quintessentially transdisciplinary. My work primarily links art and science as creative and artistic research and as an approach of research-through-design (Zimmerman et al. 2007; Zimmerman et al. 2010; Batty & Berry 2015; Gaver 2012; Ylirisku et al. 2016; Barab & Squire 2004; Bateson & Martin 2013; Hjelm 2003; Klein 2010; Balkema & Slager 2004; Mäkelä et al. 2011; Busch 2009; Hellström 2010; Lesage 2009; Ladd 1979; Borgdorff 2007). In summary, the aim of RYM as a research project is to (1) expand traditional digital game design with new knowledge on how to design future BCI games with more sophisticated consumer BCI technology; and (2) to test the possibility of designing an experimental BCI-VR game with existing consumer grade BCI hardware based on hacking as a creative and artistic design strategy. Research with respect to gaming and playful characteristics has previously been done in cognitive sciences and in particular human-centred computing; however, research from a game design point of view is limited. Thus far there are no available guidelines or strategies for BCI game design from a game design research perspective. Apart from its merit as a research-practical exercise in hacking, the work on RYM has revealed current and future possibilities and issues related to consumer BCI technology in the gaming context, and as such contributes knowledge to the chosen field of application. Since there is practically no material on BCI game design, I hope that the insights provided by this game art and game design-centred creative research project will be game-changing for an arising research field within game design research (Gürkök et al. 2015; Loup-Escande et al. 2015; Bos et al. 2010; Nijholt 2016)