Viia-hand: a Reach-and-grasp Restoration System Integrating Voice interaction, Computer vision and Auditory feedback for Blind Amputees

Visual feedback plays a crucial role in the process of amputation patients completing grasping in the field of prosthesis control. However, for blind and visually impaired (BVI) amputees, the loss of both visual and grasping abilities makes the "easy" reach-and-grasp task a feasible challenge. In this paper, we propose a novel multi-sensory prosthesis system helping BVI amputees with sensing, navigation and grasp operations. It combines modules of voice interaction, environmental perception, grasp guidance, collaborative control, and auditory/tactile feedback. In particular, the voice interaction module receives user instructions and invokes other functional modules according to the instructions. The environmental perception and grasp guidance module obtains environmental information through computer vision, and feedbacks the information to the user through auditory feedback modules (voice prompts and spatial sound sources) and tactile feedback modules (vibration stimulation). The prosthesis collaborative control module obtains the context information of the grasp guidance process and completes the collaborative control of grasp gestures and wrist angles of prosthesis in conjunction with the user's control intention in order to achieve stable grasp of various objects. This paper details a prototyping design (named viia-hand) and presents its preliminary experimental verification on healthy subjects completing specific reach-and-grasp tasks. Our results showed that, with the help of our new design, the subjects were able to achieve a precise reach and reliable grasp of the target objects in a relatively cluttered environment. Additionally, the system is extremely user-friendly, as users can quickly adapt to it with minimal training

A Highly Accurate And Reliable Data Fusion Framework For Guiding The Visually Impaired

The world has approximately 285 million visually impaired (VI) people according to a report by the World Health Organization. Thirty-nine million people are estimated to be blind, whereas 246 million people are estimated to have impaired vision. An important factor that motivated this research is the fact that 90% of VI people live in developing countries. Several systems have been designed to improve the quality of the life of VI people and support the mobility of VI people. Unfortunately, none of these systems provides a complete solution for VI people, and the systems are very expensive. Therefore, this work presents an intelligent framework that includes several types of sensors embedded in a wearable device to support the visually impaired (VI) community. The proposed work is based on an integration of sensor-based and computer vision-based techniques in order to introduce an efficient and economical visual device. The designed algorithm is divided to two components: obstacle detection and collision avoidance. The system has been implemented and tested in real-time scenarios. A video dataset of 30 videos and an average of 700 frames per video was fed to the system for the testing purpose. The achieved 96.53% accuracy rate of the proposed sequence of techniques that are used for real-time detection component is based on a wide detection view that used two camera modules and a detection range of approximately 9 meters. The 98% accuracy rate was obtained for a larger dataset. However, the main contribution in this work is the proposed novel collision avoidance approach that is based on the image depth and fuzzy control rules. Through the use of x-y coordinate system, we were able to map the input frames, whereas each frame was divided into three areas vertically and further 1/3 of the height of that frame horizontally in order to specify the urgency of any existing obstacles within that frame. In addition, we were able to provide precise information to help the VI user in avoiding front obstacles using the fuzzy logic. The strength of this proposed approach is that it aids the VI users in avoiding 100% of all detected objects. Once the device is initialized, the VI user can confidently enter unfamiliar surroundings. Therefore, this implemented device can be described as accurate, reliable, friendly, light, and economically accessible that facilitates the mobility of VI people and does not require any previous knowledge of the surrounding environment. Finally, our proposed approach was compared with most efficient introduced techniques and proved to outperform them

Multimodal Computational Attention for Scene Understanding

Robotic systems have limited computational capacities. Hence, computational attention models are important to focus on specific stimuli and allow for complex cognitive processing. For this purpose, we developed auditory and visual attention models that enable robotic platforms to efficiently explore and analyze natural scenes. To allow for attention guidance in human-robot interaction, we use machine learning to integrate the influence of verbal and non-verbal social signals into our models

Apport de la vision par ordinateur dans l'utilisabilité des neuroprothèses visuelles

L'OMS estime que 45 millions de personnes dans le monde sont aveugles. Avec le vieillissement de la population, ce chiffre ne cesse de progresser car la cécité touche majoritairement les personnes âgées. Les neuroprothèses visuelles ont pour objectif de restaurer une forme de vision. Ces systèmes convertissent les informations de la scène visuelle en percepts lumineux via des microstimulations électriques du système visuel. La perception visuelle ainsi générée consiste en un ensemble restreint de phosphènes. Ces systèmes sont, à ce jour, inutilisables dans un environnement naturel : l'information visuelle restituée est insuffisante pour que les personnes implantées puissent se déplacer, localiser des objets et les reconnaître. Au cours des dernières décennies, la vision par ordinateur a connu d'énormes avancées, grâce aux améliorations apportées aux algorithmes de traitement d'images et à l'augmentation de la puissance de calcul disponible. Il est désormais possible de localiser de manière fiable des objets, des visages ou du texte dans un environnement naturel. Or, la plupart des neuroprothèses visuelles intègrent une caméra facilement associable à un module de traitement d'images. Partant de ces constatations, nous avons montré qu'il est possible d'améliorer l'utilisabilité de ces systèmes, en utilisant des algorithmes de traitement d'images performants. En détectant des zones d'intérêt dans une scène naturelle et en les restituant à l'utilisateur par le biais d'un nombre limité de phosphènes, nos résultats indiquent qu'il est possible de restaurer des comportements visuo-moteurs adaptés : localisation d'objets, de visages ou encore de textes.The WHO estimates that 45 million people worldwide are blind. This figure is rapidly increasing because of the ageing of the world population, as blindness primarily affects elderly people. Visual neuroprostheses aim at restoring a sort of vision. These systems convert visual information captured by a camera into dots-like percepts via electrical microstimulation of the visual system. The evoked visual perception corresponds to a black and white image with a few dozen of pixels with gaps separating them. Although these systems give great hope to blind people, they are still inefficient in a natural environment: the restored visual information is too coarse to allow complex functions such as navigation, object localization and recognition, or reading at a convenient speed. Over the last decades, computer vision has been steadily improving, thanks to the development of new image processing algorithms and the increase of processing power. For instance, this is now possible to localize objects, faces or texts in real outdoor conditions. Interestingly, most of the current visual neuroprostheses include an external camera making it possible to process the input images in order to adapt the phosphenes display. In the current work, we showed that real-time image processing can improve the usability of low resolution visual neuroprostheses relying on the extraction of high-level information from the input images. Indeed, our results showed that the augmentation of the phosphene display with a limited number of phosphenes allows restoring visuomotor behaviors, such as localizing pertinent objects, faces or texts within a natural scene