Cross-Reality Re-Rendering: Manipulating between Digital and Physical Realities

The advent of personalized reality has arrived. Rapid development in AR/MR/VR enables users to augment or diminish their perception of the physical world. Robust tooling for digital interface modification enables users to change how their software operates. As digital realities become an increasingly-impactful aspect of human lives, we investigate the design of a system that enables users to manipulate the perception of both their physical realities and digital realities. Users can inspect their view history from either reality, and generate interventions that can be interoperably rendered cross-reality in real-time. Personalized interventions can be generated with mask, text, and model hooks. Collaboration between users scales the availability of interventions. We verify our implementation against our design requirements with cognitive walkthroughs, personas, and scalability tests.Comment: updated. arXiv admin note: text overlap with arXiv:2204.0373

Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

Metaphor and Senses

The book deals with the synesthetic metaphors in Synamet – a semantically and grammatically annotated corpus. The texts included in the corpus are excerpted from blogs devoted to, among others, perfume, wine, beer, music, art, massage and wellness. The thesis presents a Conceptual Metaphor Theory (CMT) and frame-based analysis of synesthetic metaphors in Polish. Using data from the corpus, the book provides ample empirical support for embodiment in metaphor and internal logic of mappings between frames. The study proposes new models of verbal synesthesia in the corpus and calls into question a universality of hierarchy of senses. This book should be of interest to researchers working within cognitive linguistics, in particular metaphor theory, frame semantics, corpus linguistics, and sensory science

Learners' perceptions of teachers' non-verbal behaviours in the foreign language class

This study explores the meanings that participants in a British ELT setting give to teachers' non-verbal behaviours. It is a qualitative, descriptive study of the perceived functions that gestures and other non-verbal behaviours perform in the foreign language classroom, viewed mainly from the language learners' perspective. The thesis presents the stages of the research process, from the initial development of the research questions to the discussion of the research findings that summarise and discuss the participants' views. There are two distinct research phases presented in the thesis. The pilot study explores the perceptions of 18 experienced language learners of teachers' non-verbal behaviours. The data is collected in interviews based on videotaped extracts of classroom interaction, presented to the participants in two experimental conditions, with and without sound. The findings of this initial study justify the later change of method from the experimental design to a more exploratory framework. In the main study, 22 learners explain, in interviews based on stimulated recall, their perceptions on their teachers' verbal and non-verbal behaviours as occurring within the immediate classroom context. Finally, learners' views are complemented by 20 trainee teachers' written reports of classroom observation and their opinions expressed in focus group interviews. The data for the main study were thus collected through a combination of methods, ranging from classroom direct observations and videotaped recordings, to semi-structured interviews with language learners. The research findings indicate that participants generally believe that gestures and other non-verbal behaviours playa key role in the language learning and teaching process. Learners identify three types of functions that non-verbal behaviours play in the classroom interaction: (i) cognitive, i.e. non-verbal behaviours which work as enhancers of the learning processes, (ii) emotional, i.e. non-verbal behaviours that function as reliable communicative devices of teachers' emotions and attitudes and (iii) organisational, i.e. non-verbal behaviours which serve as tools of classroom management and control. The findings suggest that learners interpret teachers' non-verbal behaviours in a functional manner and use these messages and cues in their learning and social interaction with the teacher. The trainee teachers value in a similar manner the roles that non-verbal behaviours play in the language teaching and learning. However, they seem to prioritise the cognitive and managerial functions of teachers' non-verbal behaviours over the emotional ones and do not consider the latter as important as the learners did. This study is original in relation to previous studies of language classroom interaction in that it: • describes the kinds of teachers' behaviours which all teachers and learners are familiar with, but which have seldom been foregrounded in classroom-based research; • unlike previous studies of non-verbal behaviour, investigates the perceiver's view of the others' non-verbal behaviour rather than its production; • documents these processes of perception through an innovative methodology of data collection and analysis; • explores the teachers' non-verbal behaviours as perceived by the learners themselves, suggesting that their viewpoint can be one window on the reality of language classrooms; • provides explanations and functional interpretations for the many spontaneous and apparently unimportant actions that teachers use on a routine basis; • identifies a new area which needs consideration in any future research and pedagogy of language teaching and learning

The self which enacts learning, a research with/through the Bay Area Artists for Women's Art (BAAWA)

This thesis reveals and clarifies through creative research practices and forms a movement towards, and a recognition of, embodied arts learning through and as a result of an association with the Bay Area Artists for Women's Art (BAAWA), a particular feminist art educational community. Interpretive inquiry on a present learning process both locates and frames this research. The components which inform this interpretation include: a study of the associated literature on feminist art communities, a participant research study with the BAAWA community presented as a notated theatrical performance script, and an exhibition. They relate to and through each other in a complementary relationship and record the various values, perspectives and orientations I have taken on at different occasions in my learning. It is the relationship and specificity of these occasions or events which point to naming my learning process as curriculum as I have understood it from, and applied it beyond, BAAWA. This research is multi-layered, multi-textured, dialogic, open and aesthetic and points to the particular complexities, needs, and practices of contemporary North American women artists in reaching to name, affirm and expand their practices as artists

Tingles, tea boxes and sticky sounds – ASMR read diffractively with agential realism

In this thesis, I contemplate a phenomenon called ‘ASMR’ through and with a few of the most prominent concepts of a new material feminist scholar and theoretical physicist Karen Barad. The chosen theory, agential realism, merges physics and metaphysics and proposes a fresh and comprehensive contribution on how matter matters, and how agency is produced in material discursive practices involving ”human” and ”nonhuman” participants. Agential realism is formed in critical conversations with contemporary physics, science studies, (intersectional) feminist, posthumanist, and post-structuralist theories. Intra-action, matter, and agency are core concepts of agential realism, which is not realism toward entities (things), but realism toward phenomena. ASMR (’Autonomous Sensory Meridian Response’) is still a somewhat inscrutable topic, although it has become more and more recognized in public conversations. A bodily characteristic of ASMR is reported to be a pleasurable (non-sexual) tingly sensation, experienced in relaxed situations and triggered by certain multisensory stimuli. Private vloggers, playfully calling themselves as ASMRtists, are uploading millions of ASMR videos to spark tingles in the spectator. A YouTube community has formed around the phenomenon. Themes of ASMR videos vary from imaginary medical appointments to seemingly absurd play with household items. Peculiar to the videos is that they feature extraordinarily intent and durational engagements with ”human,” ”nonhuman,” ”animate” and ”inanimate” objects and materials (for example with human hair, light bulbs, foreign words, and velvet). At its simplest, ASMR can be defined as an example of visual subculture: ASMR videos are user-created content on YouTube, which is a site of participatory culture online. ASMR is a topical example of the emergence of phenomena in this era. Advertisements, opinions, politics, consumerism, resistance, trends, idolization and ideals are all entangled on YouTube channels. Making a living as a professional YouTuber has become an increasingly common wish for one’s occupation, alongside the more traditional aspirations to become an actor or a teacher. Research in art education, then, is a critical prospect towards and within (visual) culture; it is and should be attentive to the practices and relations that constitute matter and meaning. Posthumanist discourses, emerging with new formulations of feminisms, as well as theories that account for materiality again, are influencing today’s art education. My challenge is to expand the instantaneous perception of the ASMR phenomenon as a mere example of human culture’s production, a bunch of YouTube videos, and a human-centric physical-emotional sensorial experience. In this thesis, I contemplate how intra-active entanglements and the production of agency materializes and manifests in the ASMR phenomenon. I’m not only interested in what is produced as part of ASMR, but how it is produced. My method is diffractive reading, which can also be phrased as ”thinking the theory” (agential realism) ”while reading the data” (ASMR phenomenon, with two exemplary videos serving as concrete examples of the genre). As I get entangled with ASMR, I strive to think with and within the box, as much as from outside it.Opinnäytteessäni tarkastelen teoreettisen fyysikon ja uusmaterialistisen feministitutkijan Karen Baradin käsitteiden kautta ilmiötä nimeltä ASMR. Baradin teoria, agential realism (suom. agentiaalinen realismi), on kokonaisvaltainen ontologia, joka on muotoiltu kriittisessä vuoropuhelussa nykyfysiikan, tieteentutkimuksen, (intersektionaalisen) feminismin, sekä posthumanististen ja jälkistrukturalististen teorioiden kanssa. Agentiaalinen realismi sulauttaa kvanttifysiikan tulkinnat ja metafysiikan peruskysymykset yhteen, ja ottaa kantaa todellisuuden materiaalisuuteen ja materian merkitykseen. Teoria pyrkii selittämään toimijuuden (agency) materiaalis-diskursiivista toteutumista ilmiöissä, joihin kuuluu sekä ”inhimillisiä” (human) että ”muita kuin inhimillisiä” (nonhuman / other-than-human) osallistumisia. Agentiaalinen realismi ei ole entiteettien realismia, vaan sen sijaan se käsittää ilmiöiden olevan todellisuuden perusyksiköitä. ASMR (sanoista ’Autonomous Sensory Meridian Response’) on melko tutkimaton, suhteellisen hiljattain tunnistettu ja tunnustettu ilmiö, joka on viime vuosina alkanut herättää kiinnostusta julkisessa keskustelussa. ASMR-ilmiön ytimessä on spesifi, kihelmöivä hyvänolontunne (tyypillisesti ei seksuaalinen), joka koetaan rentouttavissa tilanteissa tiettyjen moniaististen ärsykkeiden yhteydessä.Yksityiset videoblokkaajat, jotka ovat leikkisästi nimenneet itsensä ASMRtisteiksi, lataavat miljoonia videoita YouTubeen, tarkoituksenaan synnyttää ASMR-kokemus katsojassa. Ilmiön ympärille onkin syntynyt nettiyhteisö, jonka keskiössä ovat ASMR-videot. Videoiden teemat vaihtelevat kuvitteellisista lääkärikäynneistä näennäisen absurdeihin leikkeihin tyypillisesti taloustavaroiden, tai vaikkapa ruoan kanssa. Erityistä videoiden sisällöissä onkin intensiivinen ja tarkoituksellisen pitkäkestoinen antautuminen kanssaoloon ”inhimillisten”, ”ei-inhimillisten”, ”elävien” ja ”elottomien” objektien ja materian kanssa (video saattaa esimerkiksi kulua hiusten, hehkulamppujen, vieraskielisten sanojen tai sametin parissa). Ykskantaan ASMR-ilmiön voisi lukea visuaalisen alakulttuurin muodoksi: ASMR-videot ovat itsetuotettua sisältöä YouTubessa, joka on eräs osallistavan online-kulttuurin areenoista. ASMR on kiinnostava esimerkki ilmiöiden toteutumisesta ja leviämisestä tässä ajassa. Mainonta, mielipiteet, politiikka, kuluttaminen, vastustaminen, trendit, ihanteet ja idolisointi ovat kietoutuneet YouTube-kanavilla. Itsensä elättäminen ”tubettajana” on yhä tyypillisempi toiveammatti perinteisen näyttelijän tai opettajantyön ohella. Taidekasvatuksella tehdyn tutkimuksen pitäisi ottaa kriittinen tarkastelukulma (visuaaliseen) kulttuuriin, jonka erottamattomana osana niin taide, kuin taidekasvatus toteutuvat. Tämä kriittinen suhtautuminen tarkoittaa virittäytymistä (visuaalisen) kulttuurin käytäntöihin ja suhteellisuuksiin, joissa materia ja merkitys toteutuvat ja tapahtuvat. Posthumanistiset ja feministiset diskurssit, sekä uusmaterialistiset teoriat ovat osa myös tämän päivän taidekasvatusfilosofiaa. Tässä tutkielmassa laajennan ihmiskeskeistä käsitystä ASMR-ilmiöstä silkkana kulttuurin tuotteena, fyysis-emotionaalisena aistikokemuksena, tai joukkona erikoisia YouTube-videoita.Tutkin Baradin käsitteillä miten, ja millaisissa intra-aktioissa (ei vuoro-, vaan kanssavaikutuksissa) toimijuus materialisoituu ja manifestoituu ASMR-ilmiön kontekstissa. Metodini on diffraktiivinen: olennaista on, että oivallukset eivät synny pelkästä aineistosta, ja sitten selity teorialla, eikä teoria määritä huomionarvoisia havaintoja, vaan teoria, aineisto (keskeisimpänä kaksi ASMR-ilmiöön kuuluvaa videota), ja minä toimimme kanssavaikutuksessa

Stimulating motivation and awareness through sensory-oriented art therapy with multiply handicapped preschool children.

Children with cerebral palsy or other neuromotor problems are often multiply handicapped and delayed in physical and psychosexual development. Because of the severity and multiplicity of the disabilitiesof these children the need for effective, documented intervention at an early age is imperative. This study proposes thatsensory-oriented art therapy is a useful means of therapy for these children and can help them in many areas, particularly in terms of motivation and awareness.Six children in an experimental group receiving art therapy; six children in a control group receiving an equal amount of individual attention; and six children receiving no supplementary sessions, wererated at the beginning and end of this five month study, on scales measuring motivation and awareness. It was expected that the children in the art therapy group would show the greatest amount of advancement on these scales. Following statistical analysis this prediction was substantially supported. To determine behavioral and emotional changesin the children over the duration of the study, teachers were asked to complete a behavioral evaluation form for each child, before and after the study. This study explores concepts such as sensory-orientedart therapy and the use of art with multiply handicapped preschool children.M.A., Art Therapy -- Hahnemann Medical College, 198