24 research outputs found
Autoencoding sensory substitution
Tens of millions of people live blind, and their number is ever increasing. Visual-to-auditory sensory substitution (SS) encompasses a family of cheap, generic solutions to assist the visually impaired by conveying visual information through sound. The required SS training is lengthy: months of effort is necessary to reach a practical level of adaptation. There are two reasons for the tedious training process: the elongated substituting audio signal, and the disregard for the compressive characteristics of the human hearing system.
To overcome these obstacles, we developed a novel class of SS methods, by training deep recurrent autoencoders for image-to-sound conversion. We successfully trained deep learning models on different datasets to execute visual-to-auditory stimulus conversion. By constraining the visual space, we demonstrated the viability of shortened substituting audio signals, while proposing mechanisms, such as the integration of computational hearing models, to optimally convey visual features in the substituting stimulus as perceptually discernible auditory components. We tested our approach in two separate cases. In the first experiment, the author went blindfolded for 5 days, while performing SS training on hand posture discrimination. The second experiment assessed the accuracy of reaching movements towards objects on a table. In both test cases, above-chance-level accuracy was attained after a few hours of training.
Our novel SS architecture broadens the horizon of rehabilitation methods engineered for the visually impaired. Further improvements on the proposed model shall yield hastened rehabilitation of the blind and a wider adaptation of SS devices as a consequence
Physics-Guided Deep Learning for Dynamical Systems: A survey
Modeling complex physical dynamics is a fundamental task in science and
engineering. Traditional physics-based models are interpretable but rely on
rigid assumptions. And the direct numerical approximation is usually
computationally intensive, requiring significant computational resources and
expertise. While deep learning (DL) provides novel alternatives for efficiently
recognizing complex patterns and emulating nonlinear dynamics, it does not
necessarily obey the governing laws of physical systems, nor do they generalize
well across different systems. Thus, the study of physics-guided DL emerged and
has gained great progress. It aims to take the best from both physics-based
modeling and state-of-the-art DL models to better solve scientific problems. In
this paper, we provide a structured overview of existing methodologies of
integrating prior physical knowledge or physics-based modeling into DL and
discuss the emerging opportunities
From Demonstrations to Task-Space Specifications:Using Causal Analysis to Extract Rule Parameterization from Demonstrations
Learning models of user behaviour is an important problem that is broadly
applicable across many application domains requiring human-robot interaction.
In this work, we show that it is possible to learn generative models for
distinct user behavioural types, extracted from human demonstrations, by
enforcing clustering of preferred task solutions within the latent space. We
use these models to differentiate between user types and to find cases with
overlapping solutions. Moreover, we can alter an initially guessed solution to
satisfy the preferences that constitute a particular user type by
backpropagating through the learned differentiable models. An advantage of
structuring generative models in this way is that we can extract causal
relationships between symbols that might form part of the user's specification
of the task, as manifested in the demonstrations. We further parameterize these
specifications through constraint optimization in order to find a safety
envelope under which motion planning can be performed. We show that the
proposed method is capable of correctly distinguishing between three user
types, who differ in degrees of cautiousness in their motion, while performing
the task of moving objects with a kinesthetically driven robot in a tabletop
environment. Our method successfully identifies the correct type, within the
specified time, in 99% [97.8 - 99.8] of the cases, which outperforms an IRL
baseline. We also show that our proposed method correctly changes a default
trajectory to one satisfying a particular user specification even with unseen
objects. The resulting trajectory is shown to be directly implementable on a
PR2 humanoid robot completing the same task.Comment: arXiv admin note: substantial text overlap with arXiv:1903.0126