Object Detection and Recognition for Visually Impaired People

Object detection plays a very important role in many applications such as image retrieval, surveillance, robot navigation, wayfinding, etc. In this thesis, we propose different approaches to detect indoor signage, stairs and pedestrians. In the first chapter we introduce some related work in this field. In the second chapter, we introduced a new method to detect the indoor signage to help blind people find their destination in unfamiliar environments. Our method first extracts the attended areas by using a saliency map. Then the signage is detected in the attended areas by using bipartite graph matching. The proposed method can handle multiple signage detection. Experimental results on our collected indoor signage dataset demonstrate the effectiveness and efficiency of our proposed method. Furthermore, saliency maps could eliminate the interference information and improve the accuracy of the detection results. In the third chapter, we present a novel camera-based approach to automatically detect and recognize restroom signage from surrounding environments. Our method first extracts the attended areas which may content signage based on shape detection. Then, Scale-Invariant Feature Transform (SIFT) is applied to extract local features in the detected attended areas. Finally, signage is detected and recognized as the regions with the SIFT matching scores larger than a threshold. The proposed method can handle multiple signage detection. Experimental results on our collected restroom signage dataset demonstrate the effectiveness and efficiency of our proposed method. In the fourth chapter, we develop a new framework to detect and recognize stairs and pedestrian crosswalks using a RGBD camera. Since both stairs and pedestrian crosswalks are featured by a group of parallel lines, we first apply Hough transform to extract the concurrent parallel lines based on the RGB channels. Then, the Depth channel is employed to further recognize pedestrian crosswalks, upstairs, and downstairs using support vector machine (SVM) classifiers. Furthermore, we estimate the distance between the camera and stairs for the blind users. The detection and recognition results on our collected dataset demonstrate that the effectiveness and efficiency of our proposed framework Keywords: Blind people, Navigation and wayfinding, Camera, Signage detection and recognition, Independent trave

Review of Machine Vision-Based Electronic Travel Aids

Visual impaired people have navigation and mobility problems on the road. Up to now, many approaches have been conducted to help them navigate around using different sensing techniques. This paper reviews several machine vision- based Electronic Travel Aids (ETAs) and compares them with those using other sensing techniques. The functionalities of machine vision-based ETAs are classified from low-level image processing such as detecting the road regions and obstacles to high-level functionalities such as recognizing the digital tags and texts. In addition, the characteristics of the ETA systems for blind people are particularly discussed

Detection and modelling of staircases using a wearable depth sensor

In this paper we deal with the perception task of a wearable navigation assistant. Specifically, we have focused on the detection of staircases because of the important role they play in indoor navigation due to the multi-floor reaching possibilities they bring and the lack of security they cause, specially for those who suffer from visual deficiencies. We use the depth sensing capacities of the modern RGB-D cameras to segment and classify the different elements that integrate the scene and then carry out the stair detection and modelling algorithm to retrieve all the information that might interest the user, i.e. the location and orientation of the staircase, the number of steps and the step dimensions. Experiments prove that the system is able to perform in real-time and works even under partial occlusions of the stairway

RGB-D-based Stair Detection using Deep Learning for Autonomous Stair Climbing

Stairs are common building structures in urban environments, and stair detection is an important part of environment perception for autonomous mobile robots. Most existing algorithms have difficulty combining the visual information from binocular sensors effectively and ensuring reliable detection at night and in the case of extremely fuzzy visual clues. To solve these problems, we propose a neural network architecture with RGB and depth map inputs. Specifically, we design a selective module, which can make the network learn the complementary relationship between the RGB map and the depth map and effectively combine the information from the RGB map and the depth map in different scenes. In addition, we design a line clustering algorithm for the postprocessing of detection results, which can make full use of the detection results to obtain the geometric stair parameters. Experiments on our dataset show that our method can achieve better accuracy and recall compared with existing state-of-the-art deep learning methods, which are 5.64% and 7.97%, respectively, and our method also has extremely fast detection speed. A lightweight version can achieve 300 + frames per second with the same resolution, which can meet the needs of most real-time detection scenes

Stairs detection with odometry-aided traversal from a wearable RGB-D camera

Stairs are one of the most common structures present in human-made scenarios, but also one of the most dangerous for those with vision problems. In this work we propose a complete method to detect, locate and parametrise stairs with a wearable RGB-D camera. Our algorithm uses the depth data to determine if the horizontal planes in the scene are valid steps of a staircase judging their dimensions and relative positions. As a result we obtain a scaled model of the staircase with the spatial location and orientation with respect to the subject. The visual odometry is also estimated to continuously recover the current position and orientation of the user while moving. This enhances the system giving the ability to come back to previously detected features and providing location awareness of the user during the climb. Simultaneously, the detection of the staircase during the traversal is used to correct the drift of the visual odometry. A comparison of results of the stair detection with other state-of-the-art algorithms was performed using public dataset. Additional experiments have also been carried out, recording our own natural scenes with a chest-mounted RGB-D camera in indoor scenarios. The algorithm is robust enough to work in real-time and even under partial occlusions of the stair

Detección y modelado de escaleras con sensor RGB-D para asistencia personal

La habilidad de avanzar y moverse de manera efectiva por el entorno resulta natural para la mayoría de la gente, pero no resulta fácil de realizar bajo algunas circunstancias, como es el caso de las personas con problemas visuales o cuando nos movemos en entornos especialmente complejos o desconocidos. Lo que pretendemos conseguir a largo plazo es crear un sistema portable de asistencia aumentada para ayudar a quienes se enfrentan a esas circunstancias. Para ello nos podemos ayudar de cámaras, que se integran en el asistente. En este trabajo nos hemos centrado en el módulo de detección, dejando para otros trabajos el resto de módulos, como podría ser la interfaz entre la detección y el usuario. Un sistema de guiado de personas debe mantener al sujeto que lo utiliza apartado de peligros, pero también debería ser capaz de reconocer ciertas características del entorno para interactuar con ellas. En este trabajo resolvemos la detección de uno de los recursos más comunes que una persona puede tener que utilizar a lo largo de su vida diaria: las escaleras. Encontrar escaleras es doblemente beneficioso, puesto que no sólo permite evitar posibles caídas sino que ayuda a indicar al usuario la posibilidad de alcanzar otro piso en el edificio. Para conseguir esto hemos hecho uso de un sensor RGB-D, que irá situado en el pecho del sujeto, y que permite captar de manera simultánea y sincronizada información de color y profundidad de la escena. El algoritmo usa de manera ventajosa la captación de profundidad para encontrar el suelo y así orientar la escena de la manera que aparece ante el usuario. Posteriormente hay un proceso de segmentación y clasificación de la escena de la que obtenemos aquellos segmentos que se corresponden con "suelo", "paredes", "planos horizontales" y una clase residual, de la que todos los miembros son considerados "obstáculos". A continuación, el algoritmo de detección de escaleras determina si los planos horizontales son escalones que forman una escalera y los ordena jerárquicamente. En el caso de que se haya encontrado una escalera, el algoritmo de modelado nos proporciona toda la información de utilidad para el usuario: cómo esta posicionada con respecto a él, cuántos escalones se ven y cuáles son sus medidas aproximadas. En definitiva, lo que se presenta en este trabajo es un nuevo algoritmo de ayuda a la navegación humana en entornos de interior cuya mayor contribución es un algoritmo de detección y modelado de escaleras que determina toda la información de mayor relevancia para el sujeto. Se han realizado experimentos con grabaciones de vídeo en distintos entornos, consiguiendo buenos resultados tanto en precisión como en tiempo de respuesta. Además se ha realizado una comparación de nuestros resultados con los extraídos de otras publicaciones, demostrando que no sólo se consigue una eciencia que iguala al estado de la materia sino que también se aportan una serie de mejoras. Especialmente, nuestro algoritmo es el primero capaz de obtener las dimensiones de las escaleras incluso con obstáculos obstruyendo parcialmente la vista, como puede ser gente subiendo o bajando. Como resultado de este trabajo se ha elaborado una publicación aceptada en el Second Workshop on Assitive Computer Vision and Robotics del ECCV, cuya presentación tiene lugar el 12 de Septiembre de 2014 en Zúrich, Suiza

Stairs and Pedestrian Crosswalks Detection Using Morphological Image Processing and Analysis in Order to Guide Visually Impaired Persons

Većina slijepih i slabovidnih osoba još ne koristi napredne sustave za pomoć pri kretanju i orijentaciji. Iako još nije vrijeme za potpuno izbacivanje ustaljenih metoda poput bijelog štapa, napredak tehnologije sada omogućava razvijanje i postupno uvođenje digitalnih mobilnih sustava za pomoć slijepima i slabovidnima. U ovoj disertaciji opisana je problematika koju mora riješiti takav sustav s naglaskom na metode navođenja prilikom kretanja korištenjem kamere i računalnom obradom slike. Ovo istraživanje usmjereno je na specifične situacije kada se osoba nalazi ispred ili na stepenicama i pješačkim prijelazima kao potencijalnim kritičnim točkama prilikom kretanja. Osim pregleda postojećih metoda detaljno su opisane tri novorazvijene metode zajedno s njihovom evaluacijom. Razvijene metode uključuju: metodu za detekciju stepenica zasnovanu na vertikalnoj i horizontalnoj analizi, multirezolucijsku metodu za detekciju pješačkih prijelaza zasnovanu na morfološkoj analizi i energiji linija, metodu za zvučno usmjeravanje slijepih i slabovidnih određivanjem prostora za sigurno kretanje. Dodatno je razvijen okvir za evaluaciju metoda usmjeravanja slijepih i slabovidnih osoba na stepenicama i pješačkim prijelazima. Testiranjem razvijenih metoda pokazane su određene prednosti u odnosu na postojeće metode po pitanju uspješnosti detekcije, mogućnosti korištenja širokokutnih ulaznih slika i robusnosti u slučajevima zaklonjenosti traženih objekata. Testiranjem brzine izvođenja razvijenih metoda pokazana je mogućnost izvođenja u realnom vremenu što je iznimno važno za pomoćne sustave koji bi se trebali koristiti u pokretu.Most of the blind and visually impaired persons are still not using advanced navigation and orientation assistance systems. Though it is not yet time to fully expel standard methods such as a white cane, advances in technology now enable the development and gradual introduction of digital mobile systems for helping the blind and visually impaired people. This dissertation describes the issues that need to be solved by such a system, focusing on navigation methods using camera and digital image processing. This research is focused on specific situations when a person is in front of or on stairs and pedestrian crosswalks as potential critical points when walking. In addition to an overview of the existing methods, three newly developed methods are described in detail along with their evaluation. Developed methods include: method for stairs detection using vertical and horizontal analysis, multiresolution method for pedestrian crosswalks detection based on morphological analysis and line energy, method for sound guidance of the blind and visually impaired by determining space for safe movement. There is also an additionally developed framework for evaluating the methods for guidance of the blind and visually impaired on stairs and pedestrian crosswalks. Testing of the developed methods has shown some advantages over existing methods regarding the accuracy, the ability to use with wide-angle input images and the robustness in cases of concealed objects. By testing the processing speed for developed methods, possibility to perform in real-time is proven, which is extremely important for the assistance systems that should be used in the movement