Helping the Blind to Get through COVID-19: Social Distancing Assistant Using Real-Time Semantic Segmentation on RGB-D Video

The current COVID-19 pandemic is having a major impact on our daily lives. Social distancing is one of the measures that has been implemented with the aim of slowing the spread of the disease, but it is difficult for blind people to comply with this. In this paper, we present a system that helps blind people to maintain physical distance to other persons using a combination of RGB and depth cameras. We use a real-time semantic segmentation algorithm on the RGB camera to detect where persons are and use the depth camera to assess the distance to them; then, we provide audio feedback through bone-conducting headphones if a person is closer than 1.5 m. Our system warns the user only if persons are nearby but does not react to non-person objects such as walls, trees or doors; thus, it is not intrusive, and it is possible to use it in combination with other assistive devices. We have tested our prototype system on one blind and four blindfolded persons, and found that the system is precise, easy to use, and amounts to low cognitive load

A comparative study in real-time scene sonification for visually impaired people

In recent years, with the development of depth cameras and scene detection algorithms, a wide variety of electronic travel aids for visually impaired people have been proposed. However, it is still challenging to convey scene information to visually impaired people efficiently. In this paper, we propose three different auditory-based interaction methods, i.e., depth image sonification, obstacle sonification as well as path sonification, which convey raw depth images, obstacle information and path information respectively to visually impaired people. Three sonification methods are compared comprehensively through a field experiment attended by twelve visually impaired participants. The results show that the sonification of high-level scene information, such as the direction of pathway, is easier to learn and adapt, and is more suitable for point-to-point navigation. In contrast, through the sonification of low-level scene information, such as raw depth images, visually impaired people can understand the surrounding environment more comprehensively. Furthermore, there is no interaction method that is best suited for all participants in the experiment, and visually impaired individuals need a period of time to find the most suitable interaction method. Our findings highlight the features and the differences of three scene detection algorithms and the corresponding sonification methods. The results provide insights into the design of electronic travel aids, and the conclusions can also be applied in other fields, such as the sound feedback of virtual reality applications

Interactive AR-based tool for gamification of smart touristic places

Augmented Reality is a computer-generated image technology that transcends the user’sview of the real world, thereby providing complex vision, it adds to the real world digitalelements depending where the user is looking and how he is interacting with the realworld.One of the main goal is to produce an AR app in ”Biblioteca Museu Victor Balaguer”, atouristic site based in Vilanova i La Geltru (Barcelona,Spain). An AR application has beenbuilt using Android studio and Unity3D platforms evaluated and tested in the Museumgoing through many 3D modules, videos and images rendered and augmented in the realworld of the museum.Unity, which is the main platform used to build this AR app has different levels of renderingsover the real world. It varies from photos to videos rendered upon the real environmentpassing through 3D modules and animations and 360 degree scenes.AR applications canbe built on many devices other than a mobile phone. In this report we will see an implemen-tation of another application on Magic Leap glasses using the Lumin platform integratedwith Unity 3D. The output of the same Lumin application was visualized using the Oculusdevices to test the result in a virtual reality world.In this report we will take a look on some current state-of-the-art in AR, describing thework performed in many other touristic places all around the world passing by enlighteningthe main differences between their project and the project explained in this document. Al-though the AR field has entered into medical, visualization, military and other technologicalprograms, we will only touch the tourism part of the field. As any other touristic project, thisapp aims to encourage the touristic domain in some places that are not alive like it shouldbe, which will end by turning back more money and benefits than these sites were earningbefore. For that we implemented a plan and a business canvas model that explains howthese applications will make these changes.The results shows that the combination of many framework together can lead to a newkind of AR gamification. The interaction between the user and the AR environment isaccomplished from one side and between users from another. The mobile game appdescribes the site of ́Biblioteca Museu Victor Balaguer ́adding some fun for the users inthe way of interacting with the real world of the Museum. This app is already programmedand tested on the field. The last stage of our game show an app developed on MagicLeap One that contributes and transmits Point clouds from one site of the Mediterraneanto another, providing the user the ability to see and talk with another user at the same tim

A Systematic Review of Urban Navigation Systems for Visually Impaired People

Blind and Visually impaired people (BVIP) face a range of practical difficulties when undertaking outdoor journeys as pedestrians. Over the past decade, a variety of assistive devices have been researched and developed to help BVIP navigate more safely and independently. In~addition, research in overlapping domains are addressing the problem of automatic environment interpretation using computer vision and machine learning, particularly deep learning, approaches. Our aim in this article is to present a comprehensive review of research directly in, or relevant to, assistive outdoor navigation for BVIP. We breakdown the navigation area into a series of navigation phases and tasks. We then use this structure for our systematic review of research, analysing articles, methods, datasets and current limitations by task. We also provide an overview of commercial and non-commercial navigation applications targeted at BVIP. Our review contributes to the body of knowledge by providing a comprehensive, structured analysis of work in the domain, including the state of the art, and guidance on future directions. It will support both researchers and other stakeholders in the domain to establish an informed view of research progress

Review of substitutive assistive tools and technologies for people with visual impairments: recent advancements and prospects

YesThe development of many tools and technologies for people with visual impairment has become a major priority in the field of assistive technology research. However, many of these technology advancements have limitations in terms of the human aspects of the user experience (e.g., usability, learnability, and time to user adaptation) as well as difficulties in translating research prototypes into production. Also, there was no clear distinction between the assistive aids of adults and children, as well as between “partial impairment” and “total blindness”. As a result of these limitations, the produced aids have not gained much popularity and the intended users are still hesitant to utilise them. This paper presents a comprehensive review of substitutive interventions that aid in adapting to vision loss, centred on laboratory research studies to assess user-system interaction and system validation. Depending on the primary cueing feedback signal offered to the user, these technology aids are categorized as visual, haptics, or auditory-based aids. The context of use, cueing feedback signals, and participation of visually impaired people in the evaluation are all considered while discussing these aids. Based on the findings, a set of recommendations is suggested to assist the scientific community in addressing persisting challenges and restrictions faced by both the totally blind and partially sighted people

Expanding the Detection of Traversable Area with RealSense for the Visually Impaired

The introduction of RGB-Depth (RGB-D) sensors into the visually impaired people (VIP)-assisting area has stirred great interest of many researchers. However, the detection range of RGB-D sensors is limited by narrow depth field angle and sparse depth map in the distance, which hampers broader and longer traversability awareness. This paper proposes an effective approach to expand the detection of traversable area based on a RGB-D sensor, the Intel RealSense R200, which is compatible with both indoor and outdoor environments. The depth image of RealSense is enhanced with IR image large-scale matching and RGB image-guided filtering. Traversable area is obtained with RANdom SAmple Consensus (RANSAC) segmentation and surface normal vector estimation, preliminarily. A seeded growing region algorithm, combining the depth image and RGB image, enlarges the preliminary traversable area greatly. This is critical not only for avoiding close obstacles, but also for allowing superior path planning on navigation. The proposed approach has been tested on a score of indoor and outdoor scenarios. Moreover, the approach has been integrated into an assistance system, which consists of a wearable prototype and an audio interface. Furthermore, the presented approach has been proved to be useful and reliable by a field test with eight visually impaired volunteers

Affective Computing for Emotion Detection using Vision and Wearable Sensors

The research explores the opportunities, challenges, limitations, and presents advancements in computing that relates to, arises from, or deliberately influences emotions (Picard, 1997). The field is referred to as Affective Computing (AC) and is expected to play a major role in the engineering and development of computationally and cognitively intelligent systems, processors and applications in the future. Today the field of AC is bolstered by the emergence of multiple sources of affective data and is fuelled on by developments under various Internet of Things (IoTs) projects and the fusion potential of multiple sensory affective data streams. The core focus of this thesis involves investigation into whether the sensitivity and specificity (predictive performance) of AC, based on the fusion of multi-sensor data streams, is fit for purpose? Can such AC powered technologies and techniques truly deliver increasingly accurate emotion predictions of subjects in the real world? The thesis begins by presenting a number of research justifications and AC research questions that are used to formulate the original thesis hypothesis and thesis objectives. As part of the research conducted, a detailed state of the art investigations explored many aspects of AC from both a scientific and technological perspective. The complexity of AC as a multi-sensor, multi-modality, data fusion problem unfolded during the state of the art research and this ultimately led to novel thinking and origination in the form of the creation of an AC conceptualised architecture that will act as a practical and theoretical foundation for the engineering of future AC platforms and solutions. The AC conceptual architecture developed as a result of this research, was applied to the engineering of a series of software artifacts that were combined to create a prototypical AC multi-sensor platform known as the Emotion Fusion Server (EFS) to be used in the thesis hypothesis AC experimentation phases of the research. The thesis research used the EFS platform to conduct a detailed series of AC experiments to investigate if the fusion of multiple sensory sources of affective data from sensory devices can significantly increase the accuracy of emotion prediction by computationally intelligent means. The research involved conducting numerous controlled experiments along with the statistical analysis of the performance of sensors for the purposes of AC, the findings of which serve to assess the feasibility of AC in various domains and points to future directions for the AC field. The AC experiments data investigations conducted in relation to the thesis hypothesis used applied statistical methods and techniques, and the results, analytics and evaluations are presented throughout the two thesis research volumes. The thesis concludes by providing a detailed set of formal findings, conclusions and decisions in relation to the overarching research hypothesis on the sensitivity and specificity of the fusion of vision and wearables sensor modalities and offers foresights and guidance into the many problems, challenges and projections for the AC field into the future