StairNetV3: Depth-aware Stair Modeling using Deep Learning

Vision-based stair perception can help autonomous mobile robots deal with the challenge of climbing stairs, especially in unfamiliar environments. To address the problem that current monocular vision methods are difficult to model stairs accurately without depth information, this paper proposes a depth-aware stair modeling method for monocular vision. Specifically, we take the extraction of stair geometric features and the prediction of depth images as joint tasks in a convolutional neural network (CNN), with the designed information propagation architecture, we can achieve effective supervision for stair geometric feature learning by depth information. In addition, to complete the stair modeling, we take the convex lines, concave lines, tread surfaces and riser surfaces as stair geometric features and apply Gaussian kernels to enable the network to predict contextual information within the stair lines. Combined with the depth information obtained by depth sensors, we propose a stair point cloud reconstruction method that can quickly get point clouds belonging to the stair step surfaces. Experiments on our dataset show that our method has a significant improvement over the previous best monocular vision method, with an intersection over union (IOU) increase of 3.4 %, and the lightweight version has a fast detection speed and can meet the requirements of most real-time applications. Our dataset is available at https://data.mendeley.com/datasets/6kffmjt7g2/1

Robotic system for garment perception and manipulation

Mención Internacional en el título de doctorGarments are a key element of people’s daily lives, as many domestic tasks -such as laundry-, revolve around them. Performing such tasks, generally dull and repetitive, implies devoting many hours of unpaid labor to them, that could be freed through automation. But automation of such tasks has been traditionally hard due to the deformable nature of garments, that creates additional challenges to the already existing when performing object perception and manipulation. This thesis presents a Robotic System for Garment Perception and Manipulation that intends to address these challenges. The laundry pipeline as defined in this work is composed by four independent -but sequential- tasks: hanging, unfolding, ironing and folding. The aim of this work is the automation of this pipeline through a robotic system able to work on domestic environments as a robot household companion. Laundry starts by washing the garments, that then need to be dried, frequently by hanging them. As hanging is a complex task requiring bimanipulation skills and dexterity, a simplified approach is followed in this work as a starting point, by using a deep convolutional neural network and a custom synthetic dataset to study if a robot can predict whether a garment will hang or not when dropped over a hanger, as a first step towards a more complex controller. After the garment is dry, it has to be unfolded to ease recognition of its garment category for the next steps. The presented model-less unfolding method uses only color and depth information from the garment to determine the grasp and release points of an unfolding action, that is repeated iteratively until the garment is fully spread. Before storage, wrinkles have to be removed from the garment. For that purpose, a novel ironing method is proposed, that uses a custom wrinkle descriptor to locate the most prominent wrinkles and generate a suitable ironing plan. The method does not require a precise control of the light conditions of the scene, and is able to iron using unmodified ironing tools through a force-feedback-based controller. Finally, the last step is to fold the garment to store it. One key aspect when folding is to perform the folding operation in a precise manner, as errors will accumulate when several folds are required. A neural folding controller is proposed that uses visual feedback of the current garment shape, extracted through a deep neural network trained with synthetic data, to accurately perform a fold. All the methods presented to solve each of the laundry pipeline tasks have been validated experimentally on different robotic platforms, including a full-body humanoid robot.La ropa es un elemento clave en la vida diaria de las personas, no sólo a la hora de vestir, sino debido también a que muchas de las tareas domésticas que una persona debe realizar diariamente, como hacer la colada, requieren interactuar con ellas. Estas tareas, a menudo tediosas y repetitivas, obligan a invertir una gran cantidad de horas de trabajo no remunerado en su realización, las cuales podrían reducirse a través de su automatización. Sin embargo, automatizar dichas tareas ha sido tradicionalmente un reto, debido a la naturaleza deformable de las prendas, que supone una dificultad añadida a las ya existentes al llevar a cabo percepción y manipulación de objetos a través de robots. Esta tesis presenta un sistema robótico orientado a la percepción y manipulación de prendas, que pretende resolver dichos retos. La colada es una tarea doméstica compuesta de varias subtareas que se llevan a cabo de manera secuencial. En este trabajo, se definen dichas subtareas como: tender, desdoblar, planchar y doblar. El objetivo de este trabajo es automatizar estas tareas a través de un sistema robótico capaz de trabajar en entornos domésticos, convirtiéndose en un asistente robótico doméstico. La colada comienza lavando las prendas, las cuales han de ser posteriormente secadas, generalmente tendiéndolas al aire libre, para poder realizar el resto de subtareas con ellas. Tender la ropa es una tarea compleja, que requiere de bimanipulación y una gran destreza al manipular la prenda. Por ello, en este trabajo se ha optado por abordar una versión simplicada de la tarea de tendido, como punto de partida para llevar a cabo investigaciones más avanzadas en el futuro. A través de una red neuronal convolucional profunda y un conjunto de datos de entrenamiento sintéticos, se ha llevado a cabo un estudio sobre la capacidad de predecir el resultado de dejar caer una prenda sobre un tendedero por parte de un robot. Este estudio, que sirve como primer paso hacia un controlador más avanzado, ha resultado en un modelo capaz de predecir si la prenda se quedará tendida o no a partir de una imagen de profundidad de la misma en la posición en la que se dejará caer. Una vez las prendas están secas, y para facilitar su reconocimiento por parte del robot de cara a realizar las siguientes tareas, la prenda debe ser desdoblada. El método propuesto en este trabajo para realizar el desdoble no requiere de un modelo previo de la prenda, y utiliza únicamente información de profundidad y color, obtenida mediante un sensor RGB-D, para calcular los puntos de agarre y soltado de una acción de desdoble. Este proceso es iterativo, y se repite hasta que la prenda se encuentra totalmente desdoblada. Antes de almacenar la prenda, se deben eliminar las posibles arrugas que hayan surgido en el proceso de lavado y secado. Para ello, se propone un nuevo algoritmo de planchado, que utiliza un descriptor de arrugas desarrollado en este trabajo para localizar las arrugas más prominentes y generar un plan de planchado acorde a las condiciones de la prenda. A diferencia de otros métodos existentes, este método puede aplicarse en un entorno doméstico, ya que no requiere de un contol preciso de las condiciones de iluminación. Además, es capaz de usar las mismas herramientas de planchado que usaría una persona sin necesidad de realizar modificaciones a las mismas, a través de un controlador que usa realimentación de fuerza para aplicar una presión constante durante el planchado. El último paso al hacer la colada es doblar la prenda para almacenarla. Un aspecto importante al doblar prendas es ejecutar cada uno de los dobleces necesarios con precisión, ya que cada error o desfase cometido en un doblez se acumula cuando la secuencia de doblado está formada por varios dobleces consecutivos. Para llevar a cabo estos dobleces con la precisión requerida, se propone un controlador basado en una red neuronal, que utiliza realimentación visual de la forma de la prenda durante cada operación de doblado. Esta realimentación es obtenida a través de una red neuronal profunda entrenada con un conjunto de entrenamiento sintético, que permite estimar la forma en 3D de la parte a doblar a través de una imagen monocular de la misma. Todos los métodos descritos en esta tesis han sido validados experimentalmente con éxito en diversas plataformas robóticas, incluyendo un robot humanoide.Programa de Doctorado en Ingeniería Eléctrica, Electrónica y Automática por la Universidad Carlos III de MadridPresidente: Abderrahmane Kheddar.- Secretario: Ramón Ignacio Barber Castaño.- Vocal: Karinne Ramírez-Amar

Procedural content generation for games

Virtual worlds play an increasingly important role in game development today. Whether in the entertainment industry, education, collaboration or data visualization - virtual space offers a freely definable environment that can be adapted to any purpose. Nevertheless, the creation of complex worlds is time-consuming and cost-intensive. A classic example for the use of a virtual world is a driving simulator where learner drivers can test their skills. The goal of the generation process is to model a realistic city that is large enough to move around for a long time without constantly passing places that have already been seen. Streets must be realistically modeled, have intersections, represent highways and country roads and create an image through buildings that create the greatest possible immersion in the virtual world. But there is still a lack of life. Pedestrians have to populate the streets in large numbers, other cars have to take part in the traffic, and a driving instructor has to sit next to the learner driver, commenting on the actions and chatting away on long journeys. In short, the effort to model such a world by hand would be immense. This thesis deals with different approaches to generate digital content for virtual worlds procedurally i.e., algorithmically. In the first part of this thesis, virtual, three-dimensional road networks are generated using a pre-defined network graph. The nodes in the graph can be generated procedurally or randomly or can be imported from open data platforms, e.g., from OpenStreetMaps (OSM). The automatic detection of intersections makes the generation flexible. The textures used for roads and intersections are constructed from prefabricated sprites whenever possible, or, in the case of a very individual construction, are newly generated during generation. The ability to create multi-lane roads gives the virtual cities a higher degree of realism. The interstices of the road network usually contain buildings, industrial areas, common areas or agricultural land. Once these so-called parcels have been identified, they can be populated with precisely these contents. In this dissertation we focus on accessible residential buildings. The second part of this thesis discusses a novel method of building generation that allows to procedurally create walk-in, multi-storey buildings. The proceeding of simple mesh generation as shown in the road network generation is extended by rules and constraints that allow a flexible floor planning and guarantee a connection of all rooms by a common corridor per floor and a staircase. Since a cityscape is usually characterised by different building shapes, the generation can be parameterized with regard to texturing, roof design, number of floors, and window and door layout. In order to ensure performance when rendering the city, each building is generated in three levels of detail. The lowest level only shows the outer walls, the highest level shows the interior rooms including stairs, doors and window frames. Once the environment is created in a way that allows the player a certain immersion, the game world has to be filled with life. Thus, the third part of this thesis discusses the procedural creation of stories for games based on pre-trained language models. The focus here is on an interactive, controlled way of playing, in which the player can interact with the objects, persons and places of the story and influence the plot. Actions generated from the entities of the previous section of the story should give a feeling of a prepared story, but always ensure the greatest possible flexibility of course. The controlled use of places, people and objects in the player's inventory allows a porting to a three-dimensional game world as well as the gameplay in the form of a text adventure. All methods for creating digital content presented in this thesis were fully implemented and evaluated with respect to usability and performance

Monitoring and Self-diagnosis of Civil Engineering Structures: Classical and Innovative Applications

Eventi estremi come esplosioni o terremoti possono avere un profondo impatto nella sicurezza degli edifici. Le zone sismiche devono convivere con questi tragici eventi, per questo monitorare in maniera continua le condizioni di salute di una struttura è necessario e auspicabile in molti casi. Il monitoraggio strutturale (Structural Health Monitoring – SHM) rappresenta un potente strumento per la valutazione del comportamento dinamico della struttura monitorata. Fino a pochi anni fa queste tecniche erano impiegate prevalentemente in ambito meccanico, aeronautico e nell’ingegneria aerospaziale. Al giorno d’oggi, la riduzione dei costi della strumentazione, sistemi di acquisizione dati di nuova generazione e l’incremento continuo dell’efficienta nelle analisi numeriche hanno reso possibile l’applicazione di queste tecniche anche a strutture civili ordinarie. Le tecniche di monitoraggio strutturale vengono applicate non solo in grandi infrastrutture come ponti, dighe o grattacieli, ma anche in strutture storiche o edifici residenziali. In questo contesto questa tesi tenta di esaminare differenti aspetti del monitoraggio strutturale, in particolar modo riferite a edifici ordinari. Attraverso tecniche Output-Only (Operational Modal Analysis – OMA) sono state monitorate diverse strutture civili con reti di sensori cablate, al fine di ottenere il comportamento dinamico strutturale nelle reali condizioni opertive. Particolare attenzione è stata focalizzata in un altra importante tematica dell’ingegneria strutturale: il danneggiamento strutturale. Attraverso un approccio numerico viene presentato un nuovo metodo per la localizzazione e quantificazione del danno a seguito di un evento sismico. In alternativa alla classica rete cablata, è stato sviluppato un sistema di acquisizione con sensori wireless (Wireless Sensor Network – WSN). I principali risultati ottenuti con questa applicazione vengono riportati nella presente tesi, unitamente al design dei sensori low-cost. Con l’ausilio della sensoristica sviluppata è stato monitorato un edificio storico in muratura, mostrando i risultati positivi ottenuti a seguito della campagna di acquisizione di rumore ambientale (Ambient Vibration Survey -AVS).Extreme events like explosions and earthquakes may have a deep impact on building safety. Seismic regions must live with these tragic events, so that continuous monitoring of structure health conditions is necessary in many cases. Structural Health Monitoring (SHM) represents a powerful tool for the evaluation of dynamic behavior of monitored structures. Until a few years ago these techniques were widely employed especially in mechanical, aeronautical and aerospace engineering. Nowadays, the reduction of equipment costs, the new generation of data acquisition systems, together with the continuous improvement of computational analysis have made it possible to apply SHM also to civil structures without strategic importance. SHM has moved from large infrastructures like bridges, dams and skyscrapers to historical heritage and residential buildings. In this background, the present work tries to examine different aspects of SHM applications, especially referred to ordinary buildings. Using Operational Modal Analysis (OMA) techniques, several civil structures have been monitored through a wired network sensor, in order to obtain the dynamic behavior in operating conditions. The relevant data collection provides a useful tool for calibrating the accuracy and sensitivity of similar SHM case studies. Specific attention is focused in another important issue in civil and in mechanical engineering: detection of structural damages. Through a numerical approach, a new method for damage localization and quantification is proposed. Besides the traditional wired acquisition system a Wireless Sensor Network (WSN) has been developed. The issues related to the usage of low-cost sensors and new generation data acquisition tools for non-destructive structural testing are discussed. Using the WSN an historical masonry building has been monitored, showing the positive results obtained following the Ambient Vibration Survey (AVS)

The Evaluation of a Performance-Based Design Process for a Hotel Building: The Comparison of Two Egress Models

This work emphasizes the importance for egress model users to choose a model for each project with the appropriate input features and simulation capabilities. This report also gives model users a mechanism for choosing the appropriate model by providing a detailed egress model review (Chapter 2). Specifically this report focuses on the ability of two egress models, EXIT89 and Simulex, to simulate a high-rise hotel building evacuation. When EXIT89 and Simulex are used to 1) simulate the same design scenarios and 2) perform a bounding analysis of the hotel building, significant differences in egress times were identified. EXIT89's evacuation times were found to be 25-40% lower than Simulex for the design scenarios, attributed to differences in unimpeded speeds, movement algorithms, methods of simulating slow occupants, density in the stairs, and stair configuration input between the models. For the bounding analysis, EXIT89 produced maximum evacuation times 30-40% lower than Simulex

Positioning algorithms for RFID-based multi-sensor indoor/outdoor positioning techniques

Position information has been very important. People need this information almost everywhere all the time. However, it is a challenging task to provide precise positions indoor/outdoor seamlessly. Outdoor positioning has been widely studied and accurate positions can usually be achieved by well developed GPS techniques. However, these techniques are difficult to be used indoor since GPS signals are too weak to be received. The alternative techniques, such as inertial sensors and radio-based pseudolites, can be used for indoor positioning but have limitations. For example, the inertial sensors suffer from drifting problems caused by the accumulating errors of measured acceleration and velocity and the radio-based techniques are prone to the obstructions and multipath effects of the transmitted signals. It is therefore necessary to develop improved methods for minimising the limitations of the current indoor positioning techniques and providing an adequately precise solution of the indoor positioning and seamless indoor/outdoor positioning. The main objectives of this research are to investigate and develop algorithms for the low-cost and portable indoor personal positioning system using Radio Frequency Identification (RFID) based multi-sensor techniques, such as integrating with Micro-Electro-Mechanical Systems (MEMS) Inertial Navigation System (INS) and/or GPS. A RFID probabilistic Cell of Origin (CoO) algorithm is developed, which is superior to the conventional CoO positioning algorithm in its positioning accuracy and continuity. Integration algorithms are also developed for RFID-based multi-sensor positioning techniques, which can provide metre-level positioning accuracy for dynamic personal positioning indoors. In addition, indoor/outdoor seamless positioning algorithms are investigated based on the iterated Reduced Sigma Point Kalman Filter (RSPKF) for RFID/MEMS INS/low-cost GPS integrated technique, which can provide metre-level positioning accuracy for personal positioning. 3-D GIS assisted personal positioning algorithms are also developed, including the map matching algorithm based on the probabilistic maps for personal positioning and the Site Specific (SISP) propagation model for efficiently generating the RFID signal strength distributions in location fingerprinting algorithms. Both static and dynamic indoor positioning experiments have been conducted using the RFID and RFID/MEMS INS integrated techniques. Metre-level positioning accuracy is achieved (e.g. 3.5m in rooms and 1.5m in stairways for static position, 4m for dynamic positioning and 1.7m using the GIS assisted positioning algorithms). Various indoor/outdoor experiments have been conducted using the RFID/MEMS INS/low-cost GPS integrated technique. It indicates that the techniques selected in this study, integrated with the low-cost GPS, can be used to provide continuous indoor/outdoor positions in approximately 4m accuracy with the iterated RSPKF. The results from the above experiments have demonstrated the improvements of integrating multiple sensors with RFID and utilizing the 3-D GIS data for personal positioning. The algorithms developed can be used in a portable RFID based multi-sensor positioning system to achieve metre-level accuracy in the indoor/outdoor environments. The proposed system has potential applications, such as tracking miners underground, monitoring athletes, locating first responders, guiding the disabled and providing other general location based services (LBS)

Minimal Infrastructure Radio Frequency Home Localisation Systems

The ability to track the location of a subject in their home allows the provision of a number of location based services, such as remote activity monitoring, context sensitive prompts and detection of safety critical situations such as falls. Such pervasive monitoring functionality offers the potential for elders to live at home for longer periods of their lives with minimal human supervision. The focus of this thesis is on the investigation and development of a home roomlevel localisation technique which can be readily deployed in a realistic home environment with minimal hardware requirements. A conveniently deployed Bluetooth ® localisation platform is designed and experimentally validated throughout the thesis. The platform adopts the convenience of a mobile phone and the processing power of a remote location calculation computer. The use of Bluetooth ® also ensures the extensibility of the platform to other home health supervision scenarios such as wireless body sensor monitoring. Central contributions of this work include the comparison of probabilistic and nonprobabilistic classifiers for location prediction accuracy and the extension of probabilistic classifiers to a Hidden Markov Model Bayesian filtering framework. New location prediction performance metrics are developed and signicant performance improvements are demonstrated with the novel extension of Hidden Markov Models to higher-order Markov movement models. With the simple probabilistic classifiers, location is correctly predicted 80% of the time. This increases to 86% with the application of the Hidden Markov Models and 88% when high-order Hidden Markov Models are employed. Further novelty is exhibited in the derivation of a real-time Hidden Markov Model Viterbi decoding algorithm which presents all the advantages of the original algorithm, while producing location estimates in real-time. Significant contributions are also made to the field of human gait-recognition by applying Bayesian filtering to the task of motion detection from accelerometers which are already present in many mobile phones. Bayesian filtering is demonstrated to enable a 35% improvement in motion recognition rate and even enables a floor recognition rate of 68% using only accelerometers. The unique application of time-varying Hidden Markov Models demonstrates the effect of integrating these freely available motion predictions on long-term location predictions

SYCOPHANT WIRELESS SENSOR NETWORKS TRACKED BY SPARSE MOBILE WIRELESS SENSOR NETWORKS WHILE COOPERATIVELY MAPPING AN AREA

Documentos apresentados no âmbito do reconhecimento de graus e diplomas estrangeiro