A multisensory monitoring and interpretation framework based on the model-view-controller paradigm

This paper proposes a monitoring and interpretation framework inspired in the Model?View?Controller (MVC) paradigm. Indeed, the paper proposes the extension of the traditional MVC paradigm to make it more flexible in incorporating the functionalities of a monitoring and interpretation system. The proposed model is defined as a hybrid distributed system where remote nodes perform lower level processing as well as data acquisition, while a central node is in charge of collecting the information and of its fusion. Firstly, the framework levels as well as their functionalities are described. Then, a fundamental part of the proposed framework, namely the common model, is introduced

A MECHANISTIC APPROACH TO POSTURAL DEVELOPMENT IN CHILDREN

Upright standing is intrinsically unstable and requires active control. The central nervous system's feedback process is the active control that integrates multi-sensory information to generate appropriate motor commands to control the plant (the body with its musculotendon actuators). Maintaining standing balance is not trivial for a developing child because the feedback and the plant are both developing and the sensory inputs used for feedback are continually changing. Knowledge gaps exist in characterizing the critical ability of adaptive multi-sensory reweighting for standing balance control in children. Furthermore, the separate contributions of the plant and feedback and their relationship are poorly understood in children, especially when considering that the body is multi-jointed and feedback is multi-sensory. The purposes of this dissertation are to use a mechanistic approach to study multi-sensory abilities of typically developing (TD) children and children with Developmental Coordination Disorder (DCD). The specific aims are: 1) to characterize postural control under different multi-sensory conditions in TD children and children with DCD; 2) to characterize the development of adaptive multi-sensory reweighting in TD children and children with DCD; and, 3) to identify the plant and feedback for postural control in TD children and how they change in response to visual reweighting. In the first experiment (Aim 1), TD children, adults, and 7-year-old children with DCD are tested under four sensory conditions (no touch/no vision, with touch/no vision, no touch/with vision, and with touch/with vision). We found that touch robustly attenuated standing sway in all age groups. Children with DCD used touch less effectively than their TD peers and they also benefited from using vision to reduce sway. In the second experiment (Aim 2), TD children (4- to 10-year-old) and children with DCD (6- to 11-year-old) were presented with simultaneous small-amplitude touch bar and visual scene movement at 0.28 and 0.2 Hz, respectively, within five conditions that independently varied the amplitude of the stimuli. We found that TD children can reweight to both touch and vision from 4 years on and the amount of reweighting increased with age. However, multi-sensory fusion (i.e., inter-modal reweighting) was only observed in the older children. Children with DCD reweight to both touch and vision at a later age (10.8 years) than their TD peers. Even older children with DCD do not show advanced multisensory fusion. Two signature deficits of multisensory reweighting are a weak vision reweighting and a general phase lag to both sensory modalities. The final aim involves closed-loop system identification of the plant and feedback using electromyography (EMG) and kinematic responses to a high- or low-amplitude visual perturbation and two mechanical perturbations in children ages six and ten years and adults. We found that the plant is different between children and adults. Children demonstrate a smaller phase difference between trunk and leg than adults at higher frequencies. Feedback in children is qualitatively similar to adults. Quantitatively, children show less phase advance at the peak of the feedback curve which may be due to a longer time delay. Under the high and low visual amplitude conditions, children show less gain change (interpreted as reweighting) than adults in the kinematic and EMG responses. The observed kinematic and EMG reweighting are mainly due to the different use of visual information by the central nervous system as measured by the open-loop mapping from visual scene angle to EMG activity. The plant and the feedback do not contribute to reweighting

Multisensory System for Fruit Harvesting Robots. Experimental Testing in Natural Scenarios and with Different Kinds of Crops

The motivation of this research was to explore the feasibility of detecting and locating fruits from different kinds of crops in natural scenarios. To this end, a unique, modular and easily adaptable multisensory system and a set of associated pre-processing algorithms are proposed. The offered multisensory rig combines a high resolution colour camera and a multispectral system for the detection of fruits, as well as for the discrimination of the different elements of the plants, and a Time-Of-Flight (TOF) camera that provides fast acquisition of distances enabling the localisation of the targets in the coordinate space. A controlled lighting system completes the set-up, increasing its flexibility for being used in different working conditions. The pre-processing algorithms designed for the proposed multisensory system include a pixel-based classification algorithm that labels areas of interest that belong to fruits and a registration algorithm that combines the results of the aforementioned classification algorithm with the data provided by the TOF camera for the 3D reconstruction of the desired regions. Several experimental tests have been carried out in outdoors conditions in order to validate the capabilities of the proposed system.The motivation of this research was to explore the feasibility of detecting and locating fruits from different kinds of crops in natural scenarios. To this end, a unique, modular and easily adaptable multisensory system and a set of associated pre-processing algorithms are proposed. The offered multisensory rig combines a high resolution colour camera and a multispectral system for the detection of fruits, as well as for the discrimination of the different elements of the plants, and a Time-Of-Flight (TOF) camera that provides fast acquisition of distances enabling the localisation of the targets in the coordinate space. A controlled lighting system completes the set-up, increasing its flexibility for being used in different working conditions. The pre-processing algorithms designed for the proposed multisensory system include a pixel-based classification algorithm that labels areas of interest that belong to fruits and a registration algorithm that combines the results of the aforementioned classification algorithm with the data provided by the TOF camera for the 3D reconstruction of the desired regions. Several experimental tests have been carried out in outdoors conditions in order to validate the capabilities of the proposed system

The Effects of Age-related Differences in State Estimation on Sensorimotor Control of the Arm in School-age Children

Previous research examining sensorimotor control of arm movements in school-age children has demonstrated age-related improvements in performance. A unifying, mechanistic explanation of these improvements is currently lacking. This dissertation systematically examined the processes involved in sensorimotor control of the arm to investigate the hypothesis that improvements in performance can be attributed, in part, to developmental changes in state estimation, defined as estimates computed by the central nervous system (CNS) that specify current and future hand positions and velocities (i.e., hand `state'). A series of behavioral experiments were employed in which 5- to 12-year-old children and adults executed goal-directed arm movements. Experiment 1 demonstrated that improvements in proprioceptive functioning resulted in an increased contribution of proprioception to the multisensory estimate of hand position, suggesting that the CNS of children flexibly integrates redundant sensorimotor feedback based on the accuracy of the individual inputs. Experiment 2 demonstrated that improvements in proprioceptive functioning for localizing initial hand position reduced the directional variability of goal-directed reaching, suggesting that improvements in static state estimation contribute to the age-related improvements in performance. Relying on sensory feedback to provide estimates of hand state during movement execution can result in erroneous movement trajectories due to delays in sensory processing. Research in adults has suggested that the CNS circumvents these delays by integrating sensory feedback with predictions of future hand states (i.e., dynamic state estimation), a finding that has not been investigated in children. Experiment 3 demonstrated that young children utilized delayed and unreliable state estimates to make on-line trajectory modifications, resulting in poor sensorimotor performance. Last, Experiment 4 hypothesized that if improvements in state estimation drive improvements in sensorimotor performance, then exposure to a perturbation that simulated the delayed and unreliable dynamic state estimation in young children would cause the adults to perform similarly to the young children (i.e., eliminating age-related improvements in performance). Results from this study were equivocal. Collectively, the results from these experiments: 1) characterized a developmental trajectory of state estimation across 5- to 12-year-old children; and, 2) demonstrated that the development of state estimation is one mechanism underlying the age-related improvements in sensorimotor performance

Stress monitoring in conflict resolution situations

Online Dispute Resolution is steadily growing to become the major alternative to litigation in court. In fact, given the characteristics of current disputes, technology-based conflict resolution may be a quite efficient approach. However, in this shift of paradigm, there are also threats that should be considered. Specifically, in this paper we deal with the problem of the lack of important context information when parties communicate in a virtual setting. In that sense, we propose the addition of a monitoring framework capable of measuring the level of stress of the parties in a non-invasive way. This information will be used by the platform and the mediator throughout the complete conflict resolution process to adapt strategies in real-time, resulting in a context-aware and more efficient approach.The work described in this paper is included in TIARAC - Telematics and Artificial Intelligence in Alternative Conflict Resolution Project (PTDC/JUR/71354/2006), which is a research project supported by FCT (Science & Technology Foundation), Portugal. The work of Davide Carneiro is supported by a doctoral grant by FCT (SFRH/BD/64890/2009). This work is also partially supported by the Spanish Ministerio de Ciencia e Innovacion under project TIN2010-20845-C03-01 and by the Spanish Junta de Comunidades de Castilla-La Mancha under projects PII2I09-0069-0994 and PEII09-0054-9581

Principles of sensorimotor control and learning in complex motor tasks

The brain coordinates a continuous coupling between perception and action in the presence of uncertainty and incomplete knowledge about the world. This mapping is enabled by control policies and motor learning can be perceived as the update of such policies on the basis of improving performance given some task objectives. Despite substantial progress in computational sensorimotor control and empirical approaches to motor adaptation, to date it remains unclear how the brain learns motor control policies while updating its internal model of the world. In light of this challenge, we propose here a computational framework, which employs error-based learning and exploits the brain’s inherent link between forward models and feedback control to compute dynamically updated policies. The framework merges optimal feedback control (OFC) policy learning with a steady system identification of task dynamics so as to explain behavior in complex object manipulation tasks. Its formalization encompasses our empirical findings that action is learned and generalised both with regard to a body-based and an object-based frame of reference. Importantly, our approach predicts successfully how the brain makes continuous decisions for the generation of complex trajectories in an experimental paradigm of unfamiliar task conditions. A complementary method proposes an expansion of the motor learning perspective at the level of policy optimisation to the level of policy exploration. It employs computational analysis to reverse engineer and subsequently assess the control process in a whole body manipulation paradigm. Another contribution of this thesis is to associate motor psychophysics and computational motor control to their underlying neural foundation; a link which calls for further advancement in motor neuroscience and can inform our theoretical insight to sensorimotor processes in a context of physiological constraints. To this end, we design, build and test an fMRI-compatible haptic object manipulation system to relate closed-loop motor control studies to neurophysiology. The system is clinically adjusted and employed to host a naturalistic object manipulation paradigm on healthy human subjects and Friedreich’s ataxia patients. We present methodology that elicits neuroimaging correlates of sensorimotor control and learning and extracts longitudinal neurobehavioral markers of disease progression (i.e. neurodegeneration). Our findings enhance the understanding of sensorimotor control and learning mechanisms that underlie complex motor tasks. They furthermore provide a unified methodological platform to bridge the divide between behavior, computation and neural implementation with promising clinical and technological implications (e.g. diagnostics, robotics, BMI).Open Acces

Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

Humanoid robots for contract visualisation

This paper challenges two assumptions made by most lawyers: first, that contracts should consist of words alone; second, that only human beings are capable of designing the “look and feel” of contracts. These assumptions amount to taboos – even in today’s digital world. Humanoid robots for contract visualisation would or rather will break these taboos. Contract visualisation constitutes a fledgling subject concerning various fields of law (e.g. visual law, legal design, contract law, legal theory and EU law). This topic needs to be explored from different perspectives. Although humanoid robots are being increasingly implemented in the legal context, their potential for contract visualisation has not yet been investigated. This paper therefore discusses contract visualisation and how humanoid robots might use visuals of the Contract Design Pattern Library presented by the International Association for Contract & Commercial Management (IACCM). The findings prompt discussion about whether and, if so, how to communicate legally with those anthropomorphic machines. Or even more specifically, about whether and, if so, how humanoid robots might best represent contracts visually and communicate these both to humans and to other humanoid robots