Investigating Precise Control in Spatial Interactions: Proxemics, Kinesthetics, and Analytics

Augmented and Virtual Reality (AR/VR) technologies have reshaped the way in which we perceive the virtual world. In fact, recent technological advancements provide experiences that make the physical and virtual worlds almost indistinguishable. However, the physical world affords subtle sensorimotor cues which we subconsciously utilize to perform simple and complex tasks in our daily lives. The lack of this affordance in existing AR/VR systems makes it difficult for their mainstream adoption over conventional $2D$ user interfaces. As a case in point, existing spatial user interfaces (SUI) lack the intuition to perform tasks in a manner that is perceptually familiar to the physical world. The broader goal of this dissertation lies in facilitating an intuitive spatial manipulation experience, specifically for motor control. We begin by investigating the role of proximity to an action on precise motor control in spatial tasks. We do so by introducing a new SUI called the Clock-Maker's Work-Space (CMWS), with the goal of enabling precise actions close to the body, akin to the physical world. On evaluating our setup in comparison to conventional mixed-reality interfaces, we find CMWS to afford precise actions for bi-manual spatial tasks. We further compare our SUI with a physical manipulation task and observe similarities in user behavior across both tasks. We subsequently narrow our focus on studying precise spatial rotation. We utilize haptics, specifically force-feedback (kinesthetics) for augmenting fine motor control in spatial rotational task. By designing three kinesthetic rotation metaphors, we evaluate precise rotational control with and without haptic feedback for 3D shape manipulation. Our results show that haptics-based rotation algorithms allow for precise motor control in 3D space, also, help reduce hand fatigue. In order to understand precise control in its truest form, we investigate orthopedic surgery training from the point of analyzing bone-drilling tasks. We designed a hybrid physical-virtual simulator for bone-drilling training and collected physical data for analyzing precise drilling action. We also developed a Laplacian based performance metric to help expert surgeons evaluate the resident training progress across successive years of orthopedic residency

Spatial representation and visual impairement - Developmental trends and new technological tools for assessment and rehabilitation

It is well known that perception is mediated by the five sensory modalities (sight, hearing, touch, smell and taste), which allows us to explore the world and build a coherent spatio-temporal representation of the surrounding environment. Typically, our brain collects and integrates coherent information from all the senses to build a reliable spatial representation of the world. In this sense, perception emerges from the individual activity of distinct sensory modalities, operating as separate modules, but rather from multisensory integration processes. The interaction occurs whenever inputs from the senses are coherent in time and space (Eimer, 2004). Therefore, spatial perception emerges from the contribution of unisensory and multisensory information, with a predominant role of visual information for space processing during the first years of life. Despite a growing body of research indicates that visual experience is essential to develop spatial abilities, to date very little is known about the mechanisms underpinning spatial development when the visual input is impoverished (low vision) or missing (blindness). The thesis's main aim is to increase knowledge about the impact of visual deprivation on spatial development and consolidation and to evaluate the effects of novel technological systems to quantitatively improve perceptual and cognitive spatial abilities in case of visual impairments. Chapter 1 summarizes the main research findings related to the role of vision and multisensory experience on spatial development. Overall, such findings indicate that visual experience facilitates the acquisition of allocentric spatial capabilities, namely perceiving space according to a perspective different from our body. Therefore, it might be stated that the sense of sight allows a more comprehensive representation of spatial information since it is based on environmental landmarks that are independent of body perspective. Chapter 2 presents original studies carried out by me as a Ph.D. student to investigate the developmental mechanisms underpinning spatial development and compare the spatial performance of individuals with affected and typical visual experience, respectively visually impaired and sighted. Overall, these studies suggest that vision facilitates the spatial representation of the environment by conveying the most reliable spatial reference, i.e., allocentric coordinates. However, when visual feedback is permanently or temporarily absent, as in the case of congenital blindness or blindfolded individuals, respectively, compensatory mechanisms might support the refinement of haptic and auditory spatial coding abilities. The studies presented in this chapter will validate novel experimental paradigms to assess the role of haptic and auditory experience on spatial representation based on external (i.e., allocentric) frames of reference. Chapter 3 describes the validation process of new technological systems based on unisensory and multisensory stimulation, designed to rehabilitate spatial capabilities in case of visual impairment. Overall, the technological validation of new devices will provide the opportunity to develop an interactive platform to rehabilitate spatial impairments following visual deprivation. Finally, Chapter 4 summarizes the findings reported in the previous Chapters, focusing the attention on the consequences of visual impairment on the developmental of unisensory and multisensory spatial experience in visually impaired children and adults compared to sighted peers. It also wants to highlight the potential role of novel experimental tools to validate the use to assess spatial competencies in response to unisensory and multisensory events and train residual sensory modalities under a multisensory rehabilitation

How do humans mediate with the external physical world? From perception to control of articulated objects

Many actions in our daily life involve operation with articulated tools. Despite the ubiquity of articulated objects in daily life, human ability in perceiving the properties and control of articulated objects has been merely studied. Articulated objects are composed of links and revolute or prismatic joints. Moving one part of the linkage results in the movement of the other ones. Reaching a position with the tip of a tool requires adapting the motor commands to the change of position of the endeffector different from the action of reaching the same position with the hand. The dynamic properties are complex and variant in the movement of articulated bodies. For instance, apparent mass, a quantity that measures the dynamic interaction of the articulated object, varies as a function of the changes in configuration. An actuated articulated system can generate a static, but position-dependent force field with constant torques about joints. There are evidences that internal models are involved in the perception and control of tools. In the present work, we aim to investigate several aspects of the perception and control of articulated objects and address two questions, The first question is how people perceive the kinematic and dynamic properties in the haptic interaction with articulated objects? And the second question is what effect has seeing the tool on the planning and execution of reaching movements with a complex tool? Does the visual representation of mechanism structures help in the reaching movement and how? To address these questions, 3D printed physical articulated objects and robotic systems have been designed and developed for the psychophysical studies. The present work involves three studies in different aspects of perception and control of articulated objects. We first did haptic size discrimination tasks using three different types of objects, namely, wooden boxes, actuated apparatus with two movable flat surfaces, and large-size pliers, in unimanual, bimanual grounded and bimanual free conditions. We found bimanual integration occurred in particular in the free manipulation of objects. The second study was on the visuo-motor reaching with complex tools. We found that seeing the mechanism of the tool, even briefly at the beginning of the trial, improved the reaching performance. The last study was about force perception, evidences showed that people could take use of the force field at the end-effector to induce the torque about the joints generated by the articulated system

Research on real-time physics-based deformation for haptic-enabled medical simulation

This study developed a multiple effective visuo-haptic surgical engine to handle a variety of surgical manipulations in real-time. Soft tissue models are based on biomechanical experiment and continuum mechanics for greater accuracy. Such models will increase the realism of future training systems and the VR/AR/MR implementations for the operating room

Getting the point: tracing worked examples enhances learning

Embodied cognition perspectives suggest that pointing and tracing with the index finger may support learning, with basic laboratory research indicating such gestures have considerable effects on information processing in working memory. The present thesis examined whether tracing worked examples could enhance learning through decreased intrinsic cognitive load. In Experiment 1, 56 Year 6 students (mean age = 11.20, SD = .44) were presented with either tracing or no-tracing instructions on parallel lines relationships. The tracing group solved more acquisition phase practice questions and made fewer test phase errors, but otherwise test results were limited by ceiling effects. 42 Year 5 students (mean age = 10.50, SD = .51) were recruited in Experiment 2 to better align the materials with students’ knowledge levels. The tracing group outperformed the non-tracing group at the test and reported lower levels of test difficulty, interpreted as lower levels of intrinsic cognitive load. Experiment 3 recruited 52 Year 6 and Year 7 students (mean age = 12.04, SD = .59) presented with materials on angle relationships of a triangle; the tracing effect was replicated on test scores and errors, but not test difficulty. Experiment 4 used the parallel lines materials to test hypothesized gradients across experimental conditions with 72 Year 5 students (mean age = 9.94, SD = .33), predicting the tracing on the paper group would outperform the tracing above the paper group, who in turn would outperform the non-tracing group. The hypothesized gradient was established across practice questions correctly answered, practice question errors, test questions correctly answered, test question time to solution, and test difficulty self-reports. The results establish that incorporating the haptic input into worked example-based instruction design enhances the worked example effect and that tracing worked examples is a natural, simple yet effective way to enhance novices’ mathematics learning

Visuospatial Integration: Paleoanthropological and Archaeological Perspectives

The visuospatial system integrates inner and outer functional processes, organizing spatial, temporal, and social interactions between the brain, body, and environment. These processes involve sensorimotor networks like the eye–hand circuit, which is especially important to primates, given their reliance on vision and touch as primary sensory modalities and the use of the hands in social and environmental interactions. At the same time, visuospatial cognition is intimately connected with memory, self-awareness, and simulation capacity. In the present article, we review issues associated with investigating visuospatial integration in extinct human groups through the use of anatomical and behavioral data gleaned from the paleontological and archaeological records. In modern humans, paleoneurological analyses have demonstrated noticeable and unique morphological changes in the parietal cortex, a region crucial to visuospatial management. Archaeological data provides information on hand–tool interaction, the spatial behavior of past populations, and their interaction with the environment. Visuospatial integration may represent a critical bridge between extended cognition, self-awareness, and social perception. As such, visuospatial functions are relevant to the hypothesis that human evolution is characterized by changes in brain–body–environment interactions and relations, which enhance integration between internal and external cognitive components through neural plasticity and the development of a specialized embodiment capacity. We therefore advocate the investigation of visuospatial functions in past populations through the paleoneurological study of anatomical elements and archaeological analysis of visuospatial behaviors