Qualitative localization using vision and odometry for path following in topo-metric maps

International audienceWe address the problem of navigation in topo- metric maps created by using odometry data and visual loop- closure detection. Based on our previous work [6], we present an optimized version of our loop-closure detection algorithm that makes it possible to create consistent topo-metric maps in real-time while the robot is teleoperated. Using such a map, the proposed navigation algorithm performs qualitative localization using the same loop-closure detection framework and the odometry data. This qualitative position is used to support robot guidance to follow a predicted path in the topo-metric map compensating the odometry drift. Compared to purely visual servoing approaches for similar tasks, our path-following algorithm is real-time, light (not more than two images per seconds are processed), and robust as odometry is still available to navigate even if vision information is absent for a short time. The approach has been validated experimentally with a Pioneer P3DX robot in indoor environments with embedded and remote computations

Cartographie hybride pour des environnements de grande taille

In this thesis, a novel vision based hybrid mapping framework which exploits metric, topological and semantic information is presented. We aim to obtain better computational efficiency than pure metrical mapping techniques, better accuracy as well as usability for robot guidance compared to the topological mapping. A crucial step of any mapping system is the loop closure detection which is the ability of knowing if the robot is revisiting a previously mapped area. Therefore, we first propose a hierarchical loop closure detection framework which also constructs the global topological structure of our hybrid map. Using this loop closure detection module, a hybrid mapping framework is proposed in two step. The first step can be understood as a topo-metric map with nodes corresponding to certain regions in the environment. Each node in turn is made up of a set of images acquired in that region. These maps are further augmented with metric information at those nodes which correspond to image sub-sequences acquired while the robot is revisiting the previously mapped area. The second step augments this model by using road semantics. A Conditional Random Field based classification on the metric reconstruction is used to semantically label the local robot path (road in our case) as straight, curved or junctions. Metric information of regions with curved roads and junctions is retained while that of other regions is discarded in the final map. Loop closure is performed only on junctions thereby increasing the efficiency and also accuracy of the map. By incorporating all of these new algorithms, the hybrid framework presented can perform as a robust, scalable SLAM approach, or act as a main part of a navigation tool which could be used on a mobile robot or an autonomous car in outdoor urban environments. Experimental results obtained on public datasets acquired in challenging urban environments are provided to demonstrate our approach.Dans cette thèse, nous présentons une nouvelle méthode de cartographie visuelle hybride qui exploite des informations métriques, topologiques et sémantiques. Notre but est de réduire le coût calculatoire par rapport à des techniques de cartographie purement métriques. Comparé à de la cartographie topologique, nous voulons plus de précision ainsi que la possibilité d’utiliser la carte pour le guidage de robots. Cette méthode hybride de construction de carte comprend deux étapes. La première étape peut être vue comme une carte topo-métrique avec des nœuds correspondants à certaines régions de l’environnement. Ces cartes sont ensuite complétées avec des données métriques aux nœuds correspondant à des sous-séquences d’images acquises quand le robot revenait dans des zones préalablement visitées. La deuxième étape augmente ce modèle en ajoutant des informations sémantiques. Une classification est effectuée sur la base des informations métriques en utilisant des champs de Markov conditionnels (CRF) pour donner un label sémantique à la trajectoire locale du robot (la route dans notre cas) qui peut être "doit", "virage" ou "intersection". L’information métrique des secteurs de route en virage ou en intersection est conservée alors que la métrique des lignes droites est effacée de la carte finale. La fermeture de boucle n’est réalisée que dans les intersections ce qui accroît l’efficacité du calcul et la précision de la carte. En intégrant tous ces nouveaux algorithmes, cette méthode hybride est robuste et peut être étendue à des environnements de grande taille. Elle peut être utilisée pour la navigation d’un robot mobile ou d’un véhicule autonome en environnement urbain. Nous présentons des résultats expérimentaux obtenus sur des jeux de données publics acquis en milieu urbain pour démontrer l’efficacité de l’approche proposée

Loop closure for topological mapping and navigation with omnidirectional images

Dans le cadre de la robotique mobile, des progrès significatifs ont été obtenus au cours des trois dernières décennies pour la cartographie et la localisation. La plupart des projets de recherche traitent du problème de SLAM métrique. Les techniques alors développées sont sensibles aux erreurs liées à la dérive ce qui restreint leur utilisation à des environnements de petite échelle. Dans des environnements de grande taille, l utilisation de cartes topologiques, qui sont indépendantes de l information métrique, se présentent comme une alternative aux approches métriques.Cette thèse porte principalement sur le problème de la construction de cartes topologiques pour la navigation de robots mobiles dans des environnements urbains de grande taille, en utilisant des caméras omnidirectionnelles. La principale contribution de cette thèse est la résolution efficace et avec précision du problème de fermeture de boucles, problème qui est au coeur de tout algorithme de cartographie topologique. Le cadre de cartographie topologique éparse / hiérarchique proposé allie une approche de partionnement de séquence d images (ISP) par regroupement des images visuellement similaires dans un noeud avec une approche de détection de fermeture de boucles permettant de connecter ces noeux. Le graphe topologique alors obtenu représente l environnement du robot. L algorithme de fermeture de boucle hiérarchique développé permet d extraire dans un premier temps les noeuds semblables puis, dans un second temps, l image la plus similaire. Cette détection de fermeture de boucles hiérarchique est rendue efficace par le stockage du contenu des cartes éparses sous la forme d une structure de données d indexation appelée fichier inversé hiérarchique (HIF). Nous proposons de combiner le score de pondération TFIDF avec des contraintes spatiales et la fréquence des amers détectés pour obtenir une meilleur robustesse de la fermeture de boucles. Les résultats en terme de densité et précision des cartes obtenues et d efficacité sont évaluées et comparées aux résultats obtenus avec des approches de l état de l art sur des séquences d images omnidirectionnelles acquises en milieu extérieur. Au niveau de la précision des détections de boucles, des résultats similaires ont été observés vis-à-vis des autres approches mais sans étape de vérification utilisant la géométrie épipolaire. Bien qu efficace, l approche basée sur HIF présente des inconvénients comme la faible densité des cartes et le faible taux de détection des boucles. Une seconde technique de fermeture de boucle a alors été développée pour combler ces lacunes. Le problème de la faible densité des cartes est causé par un sur-partionnement de la séquence d images. Celui-ci est résolu en utilisant des vecteurs de descripteurs agrégés localement (VLAD) lors de l étape de ISP. Une mesure de similarité basée sur une contrainte spatiale spécifique à la structure des images omnidirectionnelles a également été développée. Des résultats plus précis sont obtenus, même en présence de peu d appariements. Les taux de réussite sont meilleurs qu avec FABMAP 2.0, la méthode la plus utilisée actuellement, sans étape supplémentaire de vérification géométrique.L environnement est souvent supposé invariant au cours du temps : la carte de l environnement est construite lors d une phase d apprentissage puis n est pas modifiée ensuite. Une gestion de la mémoire à long terme est nécessaire pour prendre en compte les modifications dans l environnement au cours du temps. La deuxième contribution de cette thèse est la formulation d une approche de gestion de la mémoire visuelle à long terme qui peut être utilisée dans le cadre de cartes visuelles topologiques et métriques. Les premiers résultats obtenus sont encourageants. (...)Over the last three decades, research in mobile robotic mapping and localization has seen significant progress. However, most of the research projects these problems into the SLAM framework while trying to map and localize metrically. As metrical mapping techniques are vulnerable to errors caused by drift, their ability to produce consistent maps is limited to small scale environments. Consequently, topological mapping approaches which are independent of metrical information stand as an alternative to metrical approaches in large scale environments. This thesis mainly deals with the loop closure problem which is the crux of any topological mapping algorithm. Our main aim is to solve the loop closure problem efficiently and accurately using an omnidirectional imaging sensor. Sparse topological maps can be built by representing groups of visually similar images of a sequence as nodes of a topological graph. We propose a sparse / hierarchical topological mapping framework which uses Image Sequence Partitioning (ISP) to group visually similar images of a sequence as nodes which are then connected on occurrence of loop closures to form a topological graph. A hierarchical loop closure algorithm that can first retrieve the similar nodes and then perform an image similarity analysis on the retrieved nodes is used. An indexing data structure called Hierarchical Inverted File (HIF) is proposed to store the sparse maps to facilitate an efficient hierarchical loop closure. TFIDF weighting is combined with spatial and frequency constraints on the detected features for improved loop closure robustness. Sparsity, efficiency and accuracy of the resulting maps are evaluated and compared to that of the other two existing techniques on publicly available outdoor omni-directional image sequences. Modest loop closure recall rates have been observed without using the epi-polar geometry verification step common in other approaches. Although efficient, the HIF based approach has certain disadvantages like low sparsity of maps and low recall rate of loop closure. To address these shortcomings, another loop closure technique using spatial constraint based similarity measure on omnidirectional images has been proposed. The low sparsity of maps caused by over-partitioning of the input sequence has been overcome by using Vector of Locally Aggregated Descriptors (VLAD) for ISP. Poor resolution of the omnidirectional images causes fewer feature matches in image pairs resulting in reduced recall rates. A spatial constraint exploiting the omnidirectional image structure is used for feature matching which gives accurate results even with fewer feature matches. Recall rates better than the contemporary FABMAP 2.0 approach have been observed without the additional geometric verification. The second contribution of this thesis is the formulation of a visual memory management approach suitable for long term operability of mobile robots. The formulated approach is suitable for both topological and metrical visual maps. Initial results which demonstrate the capabilities of this approach have been provided. Finally, a detailed description of the acquisition and construction of our multi-sensor dataset is provided. The aim of this dataset is to serve the researchers working in the mobile robotics and vision communities for evaluating applications like visual SLAM, mapping and visual odometry. This is the first dataset with omnidirectional images acquired on a car-like vehicle driven along a trajectory with multiple loops. The dataset consists of 6 sequences with data from 11 sensors including 7 cameras, stretching 18 kilometers in a semi-urban environmental setting with complete and precise ground-truth.CLERMONT FD-Bib.électronique (631139902) / SudocSudocFranceF

Topological Mapping and Navigation in Real-World Environments

We introduce the Hierarchical Hybrid Spatial Semantic Hierarchy (H2SSH), a hybrid topological-metric map representation. The H2SSH provides a more scalable representation of both small and large structures in the world than existing topological map representations, providing natural descriptions of a hallway lined with offices as well as a cluster of buildings on a college campus. By considering the affordances in the environment, we identify a division of space into three distinct classes: path segments afford travel between places at their ends, decision points present a choice amongst incident path segments, and destinations typically exist at the start and end of routes. Constructing an H2SSH map of the environment requires understanding both its local and global structure. We present a place detection and classification algorithm to create a semantic map representation that parses the free space in the local environment into a set of discrete areas representing features like corridors, intersections, and offices. Using these areas, we introduce a new probabilistic topological simultaneous localization and mapping algorithm based on lazy evaluation to estimate a probability distribution over possible topological maps of the global environment. After construction, an H2SSH map provides the necessary representations for navigation through large-scale environments. The local semantic map provides a high-fidelity metric map suitable for motion planning in dynamic environments, while the global topological map is a graph-like map that allows for route planning using simple graph search algorithms. For navigation, we have integrated the H2SSH with Model Predictive Equilibrium Point Control (MPEPC) to provide safe and efficient motion planning for our robotic wheelchair, Vulcan. However, navigation in human environments entails more than safety and efficiency, as human behavior is further influenced by complex cultural and social norms. We show how social norms for moving along corridors and through intersections can be learned by observing how pedestrians around the robot behave. We then integrate these learned norms with MPEPC to create a socially-aware navigation algorithm, SA-MPEPC. Through real-world experiments, we show how SA-MPEPC improves not only Vulcan’s adherence to social norms, but the adherence of pedestrians interacting with Vulcan as well.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/144014/1/collinej_1.pd

Body-relative navigation guidance using uncalibrated cameras

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2010.Cataloged from PDF version of thesis.Includes bibliographical references (p. 89-97) and index.The ability to navigate through the world is an essential capability to humans. In a variety of situations, people do not have the time, the opportunity or the capability to learn the layout of the environment before visiting an area. Examples include soldiers in the field entering an unknown building, firefighters responding to an emergency, or a visually impaired person walking through the city. In absence of external source of localization (such as GPS), the system must rely on internal sensing to provide navigation guidance to the user. In order to address real-world situations, the method must provide spatially extended, temporally consistent navigation guidance, through cluttered and dynamic environments. While recent research has largely focused on metric methods based on calibrated cameras, the work presented in this thesis demonstrates a novel approach to navigation using uncalibrated cameras. During the first visit of the environment, the method builds a topological representation of the user's exploration path, which we refer to as the place graph. The method then provides navigation guidance from any place to any other in the explored environment. On one hand, a localization algorithm determines the location of the user in the graph. On the other hand, a rotation guidance algorithm provides a directional cue towards the next graph node in the user's body frame. Our method makes little assumption about the environment except that it contains descriptive visual features. It requires no intrinsic or extrinsic camera calibration, and relies instead on a method that learns the correlation between user rotation and feature correspondence across cameras. We validate our approach using several ground truth datasets. In addition, we show that our approach is capable of guiding a robot equipped with a local obstacle avoidance capability through real, cluttered environments. Finally, we validate our system with nine untrained users through several kilometers of indoor environments.by Olivier Koch.Ph.D

Learning cognitive maps: Finding useful structure in an uncertain world

In this chapter we will describe the central mechanisms that influence how people learn about large-scale space. We will focus particularly on how these mechanisms enable people to effectively cope with both the uncertainty inherent in a constantly changing world and also with the high information content of natural environments. The major lessons are that humans get by with a less is more approach to building structure, and that they are able to quickly adapt to environmental changes thanks to a range of general purpose mechanisms. By looking at abstract principles, instead of concrete implementation details, it is shown that the study of human learning can provide valuable lessons for robotics. Finally, these issues are discussed in the context of an implementation on a mobile robot. © 2007 Springer-Verlag Berlin Heidelberg

Path Planning, Mapping & Learning for Mobile Robot Navigation

Ph.DDOCTOR OF PHILOSOPH

Hybrid mapping for static and non-static indoor environments

Mención Internacional en el título de doctorIndoor environments populated by humans, such as houses, offices or universities, involve a great complexity due to the diversity of geometries and situations that they may present. Apart from the size of the environment, they can contain multiple rooms distributed into floors and corridors, repetitive structures and loops, and they can get as complicated as one can imagine. In addition, the structure and situations that the environment present may vary over time as objects could be moved, doors can be frequently opened or closed and places can be used for different purposes. Mobile robots need to solve these challenging situations in order to successfully operate in the environment. The main tools that a mobile robot has for dealing with these situations relate to navigation and perception and comprise mapping, localization, path planning and map adaptation. In this thesis, we try to address some of the open problems in robot navigation in non-static indoor environments. We focus on house-like environments as the work is framed into the HEROITEA research project that aims attention at helping elderly people with their everyday-life activities at their homes. This thesis contributes to HEROITEA with a complete robotic mapping system and map adaptation that grants safe navigation and understanding of the environment. Moreover, we provide localization and path planning strategies within the resulting map to further operate in the environment. The first problem tackled in this thesis is robot mapping in static indoor environments. We propose a hybrid mapping method that structures the information gathered from the environment into several maps. The hybrid map contains diverse knowledge of the environment such as its structure, the navigable and blocked paths, and semantic knowledge, such as the objects or scenes in the environment. All this information is separated into different components of the hybrid map that are interconnected so the system can, at any time, benefit from the information contained in every component. In addition to the conceptual conception of the hybrid map, we have also developed building procedures and an exploration algorithm to autonomous build the hybrid map. However, indoor environments populated by humans are far from being static as the environment may change over time. For this reason, the second problem tackled in this thesis is the adaptation of the map to non-static environments. We propose an object-based probabilistic map adaptation that calculates the likelihood of moving or remaining in its place for the different objects in the environment. Finally, a map is just a description of the environment whose importance is mostly related to how the map is used. In addition, map representations are more valuable as long as they offer a wider range of applications. Therefore, the third problem that we approach in this thesis is exploiting the intrinsic characteristics of the hybrid map in order to enhance the performance of localization and path planning methods. The particular objectives of these approaches are precision for robot localization and efficiency for path planning in terms of execution time and traveled distance. We evaluate our proposed methods in a diversity of simulated and real-world indoor environments. In this extensive evaluation, we show that hybrid maps can be efficiently built and maintained over time and they open up for new possibilities for localization and path planning. In this thesis, we show an increase in localization precision and robustness and an improvement in path planning performance. In sum, this thesis makes several contributions in the context of robot navigation in indoor environments, and especially in hybrid mapping. Hybrid maps offer higher efficiency during map building and other applications such as localization and path planning. In addition, we highlight the necessity of dealing with the dynamics of indoor environments and the benefits of combining topological, semantic and metric information to the autonomy of a mobile robot.Los entornos de interiores habitados por personas, como casas, oficinas o universidades, entrañan una gran complejidad por la diversidad de geometrías y situaciones que pueden ocurrir. Aparte de las diferencias en tamaño, estos entornos pueden contener muchas habitaciones organizadas en diferentes plantas o pasillos, pueden presentar estructuras repetitivas o bucles de tal forma que los entornos pueden llegar a ser tan complejos como uno se pueda imaginar. Además, la estructura y el estado del entorno pueden variar con el tiempo, ya que los objetos pueden moverse, las puertas pueden estar cerradas o abiertas y diferentes espacios pueden ser usados para diferentes propósitos. Los robots móviles necesitan resolver estas situaciones difíciles para poder funcionar de una forma satisfactoria. Las principales herramientas que tiene un robot móvil para manejar estas situaciones están relacionadas con la navegación y la percepción y comprenden el mapeado, la localización, la planificación de trayectorias y la adaptación del mapa. En esta tesis, abordamos algunos de los problemas sin resolver de la navegación de robots móviles en entornos de interiores no estáticos. Nos centramos en entornos tipo casa ya que este trabajo se enmarca en el proyecto de investigación HEROITEA que se enfoca en ayudar a personas ancianas en tareas cotidianas del hogar. Esta tesis contribuye al proyecto HEROITEA con un sistema completo de mapeado y adaptación del mapa que asegura una navegación segura y la comprensión del entorno. Además, aportamos métodos de localización y planificación de trayectorias usando el mapa construido para realizar nuevas tareas en el entorno. El primer problema que se aborda en esta tesis es el mapeado de entornos de interiores estáticos por parte de un robot. Proponemos un método de mapeado híbrido que estructura la información capturada en varios mapas. El mapa híbrido contiene información sobre la estructura del entorno, las trayectorias libres y bloqueadas y también incluye información semántica, como los objetos y escenas en el entorno. Toda esta información está separada en diferentes componentes del mapa híbrido que están interconectados de tal forma que el sistema puede beneficiarse en cualquier momento de la información contenida en cada componente. Además de la definición conceptual del mapa híbrido, hemos desarrollado unos procedimientos para construir el mapa y un algoritmo de exploración que permite que esta construcción se realice autónomamente. Sin embargo, los entornos de interiores habitados por personas están lejos de ser estáticos ya que pueden cambiar a lo largo del tiempo. Por esta razón, el segundo problema que intentamos solucionar en esta tesis es la adaptación del mapa para entornos no estáticos. Proponemos un método probabilístico de adaptación del mapa basado en objetos que calcula la probabilidad de que cada objeto en el entorno haya sido movido o permanezca en su posición anterior. Para terminar, un mapa es simplemente una descripción del entorno cuya importancia está principalmente relacionada con su uso. Por ello, los mapas más valiosos serán los que ofrezcan un rango mayor de aplicaciones. Para abordar este asunto, el tercer problema que intentamos solucionar es explotar las características intrínsecas del mapa híbrido para mejorar el desempeño de métodos de localización y de planificación de trayectorias usando el mapa híbrido. El objetivo principal de estos métodos es aumentar la precisión en la localización del robot y la eficiencia en la planificación de trayectorias en relación al tiempo de ejecución y la distancia recorrida. Hemos evaluado los métodos propuestos en una variedad de entornos de interiores simulados y reales. En esta extensa evaluación, mostramos que los mapas híbridos pueden construirse y mantenerse en el tiempo de forma eficiente y que dan lugar a nuevas posibilidades en cuanto a localización y planificación de trayectorias. En esta tesis, mostramos un aumento en la precisión y robustez en la localización y una mejora en el desempeño de la planificación de trayectorias. En resumen, esta tesis lleva a cabo diversas contribuciones en el ámbito de la navegación de robots móviles en entornos de interiores, y especialmente en mapeado híbrido. Los mapas híbridos ofrecen más eficiencia durante la construcción del mapa y en otras tareas como la localización y la planificación de trayectorias. Además, resaltamos la necesidad de tratar los cambios en entornos de interiores y los beneficios de combinar información topológica, semántica y métrica para la autonomía del robot.Programa de Doctorado en Ingeniería Eléctrica, Electrónica y Automática por la Universidad Carlos III de MadridPresidente: Carlos Balaguer Bernaldo de Quirós.- Secretario: Javier González Jiménez.- Vocal: Nancy Marie Amat