Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

Informing bowing and violin learning using movement analysis and machine learning

Violin performance is characterized by an intimate connection between the player and her instrument that allows her a continuous control of sound through a sophisticated bowing technique. A great importance in violin pedagogy is, then, given to techniques of the right hand, responsible of most of the sound produced. This study analyses the bowing trajectory in three different classical violin exercises from audio and motion capture recordings to classify, using machine learning techniques, the different kinds of bowing techniques used. Our results show that a clustering algorithm is able to appropriately group together the different shapes produced by the bow trajectories

MIE 2017: 1st international workshop on multimodal interaction for education (workshop summary)

The International Workshop on Multimodal Interaction for Education aims at investigating how multimodal interactive systems, firmly grounded on psychophysical, psychological, and pedagogical bases, can be designed, developed, and exploited for enhancing teaching and learning processes in different learning environments, with a special focus on children in the classroom. Whilst the usage of multisensory technologies in the education area is rapidly expanding, the need for solid scientific bases, design guidelines, and appropriate procedures for evaluation is emerging. Moreover, the introduction of multimodal interactive systems in the learning environment needs to develop at the same time suitable pedagogical paradigms. This workshop aims at bringing together researchers and practitioners from different disciplines, including pedagogy, psychology, psychophysics, and computer science - with a particular focus on human-computer interaction, affective computing, and social signal processing - to discuss such challenges under a multidisciplinary perspective. The workshop is partially supported by the EU-H2020-ICT Project weDRAW (http://www.wedraw.eu)

Celiac disease, rare symptoms, autoimmune patology

We report a case of a 42-years-old woman with constipation, anemia and recurrent itch. After several investigations, celiac disease was diagnosed and a treatment with a gluten-free diet was applied with beneficial effects. Recognizing celiac disease can be difficult because some of its symptoms are similar to those of other diseases. In fact, sometimes it is confused with irritable bowel syndrome or iron-deficiency anemia or intestinal infections: as a result, celiac disease is commonly underdiagnosed or misdiagnosed. This case report is described to address the physician to a correct diagnosis of celiac disease

Automatic Detection of Reflective Thinking in Mathematical Problem Solving based on Unconstrained Bodily Exploration

For technology (like serious games) that aims to deliver interactive learning, it is important to address relevant mental experiences such as reflective thinking during problem solving. To facilitate research in this direction, we present the weDraw-1 Movement Dataset of body movement sensor data and reflective thinking labels for 26 children solving mathematical problems in unconstrained settings where the body (full or parts) was required to explore these problems. Further, we provide qualitative analysis of behaviours that observers used in identifying reflective thinking moments in these sessions. The body movement cues from our compilation informed features that lead to average F1 score of 0.73 for binary classification of problem-solving episodes by reflective thinking based on Long Short-Term Memory neural networks. We further obtained 0.79 average F1 score for end-to-end classification, i.e. based on raw sensor data. Finally, the algorithms resulted in 0.64 average F1 score for subsegments of these episodes as short as 4 seconds. Overall, our results show the possibility of detecting reflective thinking moments from body movement behaviours of a child exploring mathematical concepts bodily, such as within serious game play

Early Occupational Therapy Intervention: Patients’ Occupational Needs

The occupational therapy management involves the assessment of the individual’s specific needs. This kind of assessment facilitates the therapeutic relationship and boosts the person’s motivation, as he or she feels valued and heard. Early-stage collection of information about meaningful activities for the individual helps them project themselves outside the context of illness. Collecting occupational need at an early stage, permits “Engagement”, which means participating in activities even without actually doing them. An occupational therapy model called “Personal Environment Occupation Model” suggests that already at an early stage we should make the environment and occupations meaningful to the person in order to maximise the patient’s performance. An observational study on stroke patients shows how people have personal occupational needs beyond simple self-care, including productive life and leisure time, already in the subacute phase. A further study is underway to demonstrate the effectiveness of early occupational therapy intervention, including complex patients regardless of diagnosis and taking into account their need for care and disability in order to promote their participation and maximise their autonomy

Automated analysis of postural and movement qualities of violin players

Learning to play a music instrument is a complex task, requiring continuous practice and the development of sophisticated motor control techniques. The traditional model of music learning is based on a master-apprentice relationship, leading often to a solitary learning process, in which the time spent with the teacher is usually limited to weekly lessons and a long period of self-study is needed. Moreover, a large amount of time passes from the teacher's feedback and the student's proprioceptive perception while studying, requiring a big effort in developing an efficient and healthy technique. In this paper, we present our recent developments concerning an assistive and adaptive technology to help violin students overcoming all these difficulties, and developing their technique and repertoire properly and safely. In particular, we focus on the multimodal corpus of violin performances which was collected for the purpose, and on the analysis of such data to compute postural and gestural features characterizing the performance under a biomechanical perspective and in terms of movement quality. Analysis is expected to provide students with feedback for reaching a physically accurate performance, maximizing efficiency and minimizing injuries

Gerontechnology: Definitions and Classification

Technology is now part of our lives and the current perception by patients and their caregivers, healthcare professionals, managers and policy makers is that a modern view of medicine includes the technologies as essential instruments both in clinical practice, i.e. assessment, management and follow-up of patients and more in general in many healthcare processes as diagnosis, prognosis, treatments, rehabilitation and prevention both at individual and at general population level. In this chapter we present an overview of the different definitions and classifications of technologies potentially useful for older people including: (i) Information and communication technologies, i.e. tools aimed at informing and communicating, including digital technologies, telehealth programs and the application of artificial intelligence in healthcare; (ii) assistive technologies designed to maintain older people’s independence and increasing safety including ambient and personal sensors and domotic devices; and (iii) human-computer interaction technologies to support treatment and rehabilitation of older people with mobility and cognitive impairments such as robots (humanoid, rehabilitation, service and companion-type robots) but also exergames and virtual reality activities and their clinical and social components

Designing a Multisensory Social Serious-Game for Primary School Mathematics Learning

Psychophysical and developmental psychology evidence shows that children have a preferential sensory channel to learn specific concepts, highlighting the need of a multisensory approach to education. Despite such a potential for a positive impact, in education technology research multisensory learning has been often penalised and limited by design choices. In this work, we present the requirements and the design of an implementable prototype of serious-game aimed at teaching children a precise mathematical concept: Angles. The game is based on the interaction between different sensory modalities, in particular movement, vision, and sound

Digital vs. Hybrid: Comparing Two Versions of a Board Game for Teacher Training

This study compares two versions (one digital, one hybrid) of a serious board game for teacher training called the “4Ts game”. Teachers play the game in groups to learn about—and directly engage in—the joint design of collaborative learning activities for their students by choosing the Tasks to be proposed, the Timing of activities, the Technologies to be used, and the Team composition, in an iterative process of decision making. The game comes in three versions: fully tangible, digital, and hybrid. This paper focuses on the interaction design of the digital and hybrid versions. In both cases, teachers pick cards up from four decks, read the prompts provided in the cards, and place them on a board to design a learning activity together. Their decision-making process is scaffolded by the digital or hybrid game versions, which provide feedback and suggestions and guide teachers toward the creation of a coherent design. The user experience is quite different in the two formats. In the hybrid game, teachers physically manipulate tangible cards on a tabletop board, and the board status is replicated on a laptop application that displays automatic feedback and guidance. By contrast, the digital version is played using an Interactive Whiteboard with touch-screen capabilities, thereby allowing teachers to manipulate digital cards on a digital board. The game was used in the context of two training initiatives targeting in-service school teachers (N = 42). Data were collected on acceptance of the model upon which the game is built, acceptance of the game itself, overall user satisfaction, and knowledge gains. Results show that teachers generally liked both versions of the game, especially the opportunity provided for receiving guidance in the design process. Additionally, teachers’ knowledge about learning design and collaborative learning increased significantly between the pre- and post-test for both the digital and the hybrid game groups. However, few significant differences were found between the groups that used the digital and hybrid versions of the game: the digital version was perceived as being slightly easier to use (p < 0.001). Overall, the study suggests that both versions of the game have the potential for teacher training, while the user interface of the hybrid version should be further refined to fully harness the game’s potential