Multisensory Perception and Learning: Linking Pedagogy, Psychophysics, and Human–Computer Interaction

In this review, we discuss how specific sensory channels can mediate the learning of properties of the environment. In recent years, schools have increasingly been using multisensory technology for teaching. However, it still needs to be sufficiently grounded in neuroscientific and pedagogical evidence. Researchers have recently renewed understanding around the role of communication between sensory modalities during development. In the current review, we outline four principles that will aid technological development based on theoretical models of multisensory development and embodiment to foster in-depth, perceptual, and conceptual learning of mathematics. We also discuss how a multidisciplinary approach offers a unique contribution to development of new practical solutions for learning in school. Scientists, engineers, and pedagogical experts offer their interdisciplinary points of view on this topic. At the end of the review, we present our results, showing that one can use multiple sensory inputs and sensorimotor associations in multisensory technology to improve the discrimination of angles, but also possibly for educational purposes. Finally, we present an application, the ‘RobotAngle’ developed for primary (i.e., elementary) school children, which uses sounds and body movements to learn about angles

Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

The Problem of Mental Action

In mental action there is no motor output to be controlled and no sensory input vector that could be manipulated by bodily movement. It is therefore unclear whether this specific target phenomenon can be accommodated under the predictive processing framework at all, or if the concept of “active inference” can be adapted to this highly relevant explanatory domain. This contribution puts the phenomenon of mental action into explicit focus by introducing a set of novel conceptual instruments and developing a first positive model, concentrating on epistemic mental actions and epistemic self-control. Action initiation is a functionally adequate form of self-deception; mental actions are a specific form of predictive control of effective connectivity, accompanied and possibly even functionally mediated by a conscious “epistemic agent model”. The overall process is aimed at increasing the epistemic value of pre-existing states in the conscious self-model, without causally looping through sensory sheets or using the non-neural body as an instrument for active inference

Are abstract concepts like dinosaur feathers?

Ewolucja ludzkiego układu nerwowego pozwoliła nam na wykonywanie niezwykle skomplikowanych czynności takich jak obliczenia matematyczne, analizy gospodarcze czy choćby napisanie tej książki. Mimo to wciąż nie jesteśmy pewni jak i dlaczego człowiek nabył zdolność abstrakcyjnego myślenia. Jedna z teorii sugeruje, że myślenie abstrakcyjne i konkretne opierają się na tym samym mechanizmie: doświadczeniu. Według tej teorii, nazwanej teorią ucieleśnionego poznania, świat rozumiemy dzięki doświadczeniom fizycznym. Kiedy opisujemy jakiś argument jako "chwiejny" albo pogląd jako "bezpodstawny" to korzystamy z doświadczeń, które zdobyliśmy bawiąc się kolckami jako dzieci. W tej książce zadaję postawione przez psychologa Daniela Casasanto pytanie: „czy pojęcia abstrakcyjne są jak pióra dinozaurów”. Jakie procesy ewolucyjne doprowadziły do tego, że jesteśmy w stanie opisać nawet bardzo abstrakcyjne zagadnienia w odniesieniu do konkretnych zjawisk? Przedstawiając wyniki badań nad mową i gestem osób widzących, słabowidzących oraz niewidomych, staram się pokazać, że podstawy zrozumienia wielu pojęć abstrakcyjnych szukać można w geście

Exploring the Pulse of Design and Music: The Impact of Visual Rhythm in Design for Auditorily Impaired Music Students

Auditorily impaired music students struggle to understand musical rhythms and patterns which can lead to frustration, low self-esteem, lack of developing musical skills, and reduced music appreciation. Often, the needs of those with hearing challenges are overlooked in the curriculum planning process. This research aims to show the importance of providing appropriate learning methods and materials for music students who are hearing impaired. Specifically, this study focuses on the elements of patterns, repetition of patterns, and rhythm within the visual designs used in the instruction of musical elements. The methods used in this research are based upon observation, investigation, and analysis of prior research and case studies using visual artifacts as learning tools. Moreover, the results of this research involve the revelation and importance of using adequate visual materials as supplemental learning materials for music students of all levels of abilities, specifically designed for those with special needs, hearing disorders, and auditory challenges

A Proposed Theoretical Model of Literacy Learning Using Multisensory Structured Language Instruction (MSLI)

Multisensory Structured Language Instruction has been used for decades by clinicians and practitioners as an intervention for teaching students with dyslexia. Multisensory Structured Language Instruction uses the integration of multiple senses (visual, auditory, and kinesthetic/tactile) simultaneously to teach literacy. Although the anecdotal evidence for Multisensory Structured Language Instruction is strong, there is a lack of empirical evidence to support its effectiveness. In addition, Multisensory Structured Language Instruction includes the foundational skills recommended by the National Reading Panel (2000), but the use of multiple senses to teach these skills has not been thoroughly studied. This theoretical dissertation focused on one element of Multisensory Structured Language Instruction that has not been adequately explored in the literature. A vast amount of brain imaging research demonstrates how the brain reads and writes and how a brain with dyslexia works A vast amount of brain imaging research demonstrates how the brain reads and writes and how a brain with dyslexia works differently from a typically developing brain. However, this research has mainly focused on the visual and auditory elements of learning to read. The kinesthetic modality has not been explored with respect to language learning disorders. This theoretical dissertation specifically examines the kinesthetic modality and offers a hypothesis as to why incorporating this modality into intervention may help some students with dyslexia. A literature review in the areas of dyslexia, Multisensory Structured Language Instruction, executive functioning, phonological awareness, attention, and learning disabilities was used to construct a theoretical model to explain the use of the kinesthetic modality for dyslexia intervention. Results are twofold: that Multisensory Structured Language Instruction is effective in teaching students with dyslexia because its pedagogy is grounded in methods supported by learning theory; and that use of the kinesthetic modality is useful for students who exhibit specific deficits in rapid automatic naming, a processing deficit underlying many students’ reading difficulties. The use of the kinesthetic modality improves rapid naming via attention and uses the teacher as a surrogate central executive. An intervention model was also constructed to triage students who would benefit from this intervention. Students with a single deficit in phonological processing only are treated with linguistically based interventions, while students with double-deficits in both phonological processing and rapid naming benefit from the addition of the kinesthetic modality. The electronic version of this dissertation is accessible at the OhioLink ETD Center, http://www.ohiolink.edu/etd

A Proposed Theoretical Model of Literacy Learning Using Multisensory Structured Language Instruction (MSLI)

Multisensory Structured Language Instruction has been used for decades by clinicians and practitioners as an intervention for teaching students with dyslexia. Multisensory Structured Language Instruction uses the integration of multiple senses (visual, auditory, and kinesthetic/tactile) simultaneously to teach literacy. Although the anecdotal evidence for Multisensory Structured Language Instruction is strong, there is a lack of empirical evidence to support its effectiveness. In addition, Multisensory Structured Language Instruction includes the foundational skills recommended by the National Reading Panel (2000), but the use of multiple senses to teach these skills has not been thoroughly studied. This theoretical dissertation focused on one element of Multisensory Structured Language Instruction that has not been adequately explored in the literature. A vast amount of brain imaging research demonstrates how the brain reads and writes and how a brain with dyslexia works A vast amount of brain imaging research demonstrates how the brain reads and writes and how a brain with dyslexia works differently from a typically developing brain. However, this research has mainly focused on the visual and auditory elements of learning to read. The kinesthetic modality has not been explored with respect to language learning disorders. This theoretical dissertation specifically examines the kinesthetic modality and offers a hypothesis as to why incorporating this modality into intervention may help some students with dyslexia. A literature review in the areas of dyslexia, Multisensory Structured Language Instruction, executive functioning, phonological awareness, attention, and learning disabilities was used to construct a theoretical model to explain the use of the kinesthetic modality for dyslexia intervention. Results are twofold: that Multisensory Structured Language Instruction is effective in teaching students with dyslexia because its pedagogy is grounded in methods supported by learning theory; and that use of the kinesthetic modality is useful for students who exhibit specific deficits in rapid automatic naming, a processing deficit underlying many students’ reading difficulties. The use of the kinesthetic modality improves rapid naming via attention and uses the teacher as a surrogate central executive. An intervention model was also constructed to triage students who would benefit from this intervention. Students with a single deficit in phonological processing only are treated with linguistically based interventions, while students with double-deficits in both phonological processing and rapid naming benefit from the addition of the kinesthetic modality. The electronic version of this dissertation is accessible at the OhioLink ETD Center, http://www.ohiolink.edu/etd