Automatic manhole extraction from MMS data to update basemaps

Basemaps are the main resource used in urban planning, building and infrastructure asset management. Therefore, they must be accurate and up to date to better serve citizens, contractors, property owners and town planning departments. Traditionally, they have been updated by aerial photogrammetry, but this is not always possible and alternatives need to be sought. In such cases, a useful option for large scales is the mobile mapping system (MMS). However, automatic extraction from MMS point clouds is limited by the complexity of the urban environment. Therefore, the influence of the urban pattern is analysed in three zones with varied urban characteristics: areas with high buildings, open areas, and areas with a low level of urbanization. In these areas, the capture and automatic extraction of 3D urban elements is performed using commercial software, which is useful for some elements but not for manholes. The objective of this study is to establish a methodology for extracting manholes automatically and completing hidden buildings' corners, in order to update urban basemaps. Shape and intensity are the main detection parameters for manholes, whereas additional information from satellite image Quickbird is used to complete the buildings. The worst rate of detection for all the extracted urban elements was found in areas of high buildings. Finally, the article analyses the computing cost for manhole extraction, and the economic cost and time consume of the entire process, including the proposed methodolgy using an MMS point cloud and the traditional survey in this case.Peer ReviewedPostprint (updated version

Revisión de métodos para la clasificación de fallas superficiales en pavimentos flexibles

The status of the road infrastructure affects the social, economic, and political environment of a nation. Evaluation of the pavement surface condition is essential to plan timely and effective interventions. Timely actions avoid operating cost overruns, prevent uncontrolled deterioration and reduce operational and safety inconveniences. The problem raises the concern of studying alternatives to evaluate the status of pavement, for which a large number of investigations on automatic detection of surface flaws in flexible pavements through image processing techniques have been developed. The objective of this article is to review and analyze these contributions. Based on the review, it was concluded that the performance of this type of systems is determined by two factors: data collection and processing. The analysis presented herein unfolds based on these factors. The development of systems that take advantage of the qualities of different sensors in data acquisition and that integrate the detection and classification of a variety of faults including severity data is considered opportune.El estado de la infraestructura vial impacta el entorno social, económico y político de una nación. La evaluación de la condición superficial del pavimento es esencial para planificar intervenciones oportunas y eficaces. Las acciones oportunas evitan sobrecostos de operación, impiden el deterioro no controlado y disminuyen los inconvenientes operacionales y de seguridad. El problema expuesto plantea la inquietud de estudiar alternativas para evaluar el estado del pavimento, por lo cual un gran número de investigaciones sobre detección automática de fallas superficiales en pavimentos flexibles a través de técnicas de procesamiento de imágenes han sido desarrolladas. El objetivo de este artículo es revisar y analizar estos aportes. Sobre la base de la revisión, se concluyó que el rendimiento de este tipo de sistemas está determinado por dos factores: la recopilación de los datos y su procesamiento. El análisis presentado se despliega en función de estos factores. Se considera oportuno el desarrollo de sistemas que aprovechen las cualidades de diferentes sensores en la adquisición de datos y que integren la detección y clasificación de variedad de fallas incluyendo datos de severidad

Semi-automated Generation of Road Transition Lines Using Mobile Laser Scanning Data

Recent advances in autonomous vehicles (AVs) are exponential. Prominent car manufacturers, academic institutions, and corresponding governmental departments around the world are taking active roles in the AV industry. Although the attempts to integrate AV technology into smart roads and smart cities have been in the works for more than half a century, the High Definition Road Maps (HDRMs) that assists full self-driving autonomous vehicles did not yet exist. Mobile Laser Scanning (MLS) has enormous potential in the construction of HDRMs due to its flexibility in collecting wide coverage of street scenes and 3D information on scanned targets. However, without proper and efficient execution, it is difficult to generate HDRMs from MLS point clouds. This study recognizes the research gaps and difficulties in generating transition lines (the paths that pass through a road intersection) in road intersections from MLS point clouds. The proposed method contains three modules: road surface detection, lane marking extraction, and transition line generation. Firstly, the points covering road surface are extracted using the voxel- based upward-growing and the improved region growing. Then, lane markings are extracted and identified according to the multi-thresholding and the geometric filtering. Finally, transition lines are generated through a combination of the lane node structure generation algorithm and the cubic Catmull-Rom spline algorithm. The experimental results demonstrate that transition lines can be successfully generated for both T- and cross-intersections with promising accuracy. In the validation of lane marking extraction using the manually interpreted lane marking points, the method can achieve 90.80% precision, 92.07% recall, and 91.43% F1-score, respectively. The success rate of transition line generation is 96.5%. Furthermore, the Buffer-overlay-statistics (BOS) method validates that the proposed method can generate lane centerlines and transition lines within 20 cm-level localization accuracy from MLS point clouds. In addition, a comparative study is conducted to indicate the better performance of the proposed road marking extraction method than that of three other existing methods. In conclusion, this study makes a considerable contribution to the research on generating transition lines for HDRMs, which further contributes to the research of AVs

Semi-automated Generation of High-accuracy Digital Terrain Models along Roads Using Mobile Laser Scanning Data

Transportation agencies in many countries require high-accuracy (2-20 cm) digital terrain models (DTMs) along roads for various transportation related applications. Compared to traditional ground surveys and aerial photogrammetry, mobile laser scanning (MLS) has great potential for rapid acquisition of high-density and high-accuracy three-dimensional (3D) point clouds covering roadways. Such MLS point clouds can be used to generate high-accuracy DTMs in a cost-effective fashion. However, the large-volume, mixed-density and irregular-distribution of MLS points, as well as the complexity of the roadway environment, make DTM generation a very challenging task. In addition, most available software packages were originally developed for handling airborne laser scanning (ALS) point clouds, which cannot be directly used to process MLS point clouds. Therefore, methods and software tools to automatically generate DTMs along roads are urgently needed for transportation users. This thesis presents an applicable workflow to generate DTM from MLS point clouds. The entire strategy of DTM generation was divided into two main parts: removing non-ground points and interpolating ground points into gridded DTMs. First, a voxel-based upward growing algorithm was developed to effectively and accurately remove non-ground points. Then through a comparative study on four interpolation algorithms, namely Inverse Distance Weighted (IDW), Nearest Neighbour, Linear, and Natural Neighbours interpolation algorithms, the IDW interpolation algorithm was finally used to generate gridded DTMs due to its higher accuracy and higher computational efficiency. The obtained results demonstrated that the voxel-based upward growing algorithm is suitable for areas without steep terrain features. The average overall accuracy, correctness, and completeness values of this algorithm were 0.975, 0.980, and 0.986, respectively. In some cases, the overall accuracy can exceed 0.990. The results demonstrated that the semi-automated DTM generation method developed in this thesis was able to create DTMs with a centimetre-level grid size and 10 cm vertical accuracy using the MLS point clouds

Update urban basemap by using the LiDAR mobile mapping system : the case of Abu Dhabi municipal system

Basemaps are the main resource used in urban planning and in building and infrastructure asset management. These maps are used by citizens and by private and public stakeholders. Therefore, accurate, up-to-date geoinformation of reference are needed to provide a good service. In general, basemaps have been updated by aerial photogrammetry or field surveying, but these methods are not always possible and alternatives need to be sought. Current limitations and challenges that face traditional field surveys include areas with extreme weather, deserts or artic environments, and flight restrictions due to proximity with other countries if there is not an agreement. In such cases, alternatives for large-scale are required. This thesis proposes the use of a mobile mapping system (MMS) to update urban basemaps. Most urban features can be extracted from point cloud using commercial software or open libraries. However, there are some exceptions: manhole covers, or hidden elements even with captures from defferent perspective, the most common building corners. Therefore, the main objective of this study was to establish a methodology for extracting manholes automatically and for completing hidden corners of buildings, so that urban basemaps can be updated. The algorithm developed to extract manholes is based on time, intensity and shape detection parameters, whereas additional information from satellite images is used to complete buildings. Each municipality knows the materials and dimensions of its manholes. Taking advantage of this knowledge, the point cloud is filtered to classify points according to the set of intensity values associated with the manhole material. From the classified points, the minimum bounding rectangles (MBR) are obtained and finally the shape is adjusted and drawn. We use satellite imagery to automatically digitize the layout of building footprints with automated software tools. Then, the visible corners of buildings from the LiDAR point cloud are imported and a fitting process is performed by comparing them with the corners of the building from the satellite image. Two methods are evaluated to establish which is the most suitable for adjustment in these conditions. In the first method, the differences in X and Y directions are measured in the corners, where LiDAR and satellite data are available, and is often computed as the average of the offsets. In the second method, a Helmert 2D transformation is applied. MMS involves Global Navigation Satellite Systems (GNSS) and Inertial Measurement Units (IMU) to georeference point clouds. Their accuracy depends on the acquisition environment. In this study, the influence of the urban pattern is analysed in three zones with varied urban characteristics: different height buildings, open areas, and areas with a low and high level of urbanization. To evaluate the efficiency of the proposed algorithms, three areas were chosen with varying urban patterns in Abu Dhabi. In these areas, 3D urban elements (light poles, street signs, etc) were automatically extracted using commercial software. The proposed algorithms were applied to the manholes and buildings. The completeness and correctness ratio, and geometric accuracy were calculated for all urban elements in the three areas. The best success rates (>70%) were for light poles, street signs and road curbs, regardless of the height of the buildings. The worst rate was obtained for the same features in peri-urban areas, due to high vegetation. In contrast, the best results for trees were found in theses areas. Our methodology demonstrates the great potential and efficiency of mobile LiDAR technology in updating basemaps; a process that is required to achieve standard accuracy in large scale maps. The cost of the entire process and the time required for the proposed methodology was calculated and compared with the traditional method. It was found that mobile LiDAR could be a standard cost-efficient procedure for updating maps.La cartografía de referencia es la principal herramienta en planificación urbanística, y gestión de infraestructuras y edificios, al servicio de ciudadanos, empresas y administración. Por esta razón, debe estar actualizada y ser lo más precisa posible. Tradicionalmente, la cartografía se actualiza mediante fotogrametría aérea o levantamientos terrestres. No obstante, deben buscarse alternativas válidas para escalas grandes, porque no siempre es posible emplear estas técnicas debido a las limitaciones y retos actuales a los que se enfrenta la medición tradicional en algunas zonas del planeta, con meteorología extrema o restricciones de vuelo por la proximidad a la frontera con otros países. Esta tesis propone el uso del sistema Mobile Mapping System (MMS) para actualizar la cartografía urbana de referencia. La mayoría de los elementos pueden extraerse empleando software comercial o librerías abiertas, excepto los registros de servicios. Los elementos ocultos son otro de los inconvenientes encontrados en el proceso de creación o actualización de la cartografía, incluso si se dispone de capturas desde diferentes puntos de vista. El caso más común es el de las esquinas de edificios. Por ello, el principal objetivo de este estudio es establecer una metodología de extracción automática de los registros y completar las esquinas ocultas de los edificios para actualizar cartografía urbana. El algoritmo desarrollado para la detección y extracción de registros se basa en parámetros como el tiempo, la intensidad de la señal laser y la forma de los registros, mientras que para completar los edificios se emplea información adicional de imágenes satélite. Aprovechando el conocimiento del material y dimensión de los registros, en disposición de los gestores municipales, el algoritmo propuesto filtra y clasifica los puntos de acuerdo a los valores de intensidad. De aquellos clasificados como registros se calcula el mínimo rectángulo que los contiene (Minimum Bounding Rectangle) y finalmente se ajusta la forma y se dibuja. Las imágenes de satélite son empleadas para obtener automáticamente la huella de los edificios. Posteriormente, se importan las esquinas visibles de los edificios obtenidas desde la nube de puntos y se realiza el ajuste comparándolas con las obtenidas desde satélite. Para llevar a cabo este ajuste se han evaluado dos métodos, el primero de ellos considera las diferencias entre las coordenadas XY, desplazándose el promedio. En el segundo, se aplica una transformación Helmert2D. MMS emplea sistemas de navegación global por satélite (Global Navigation Satellite Systems, GNSS) e inerciales (Inertial Measurement Unit, IMU) para georreferenciar la nube de puntos. La precisión de estos sistemas de posicionamiento depende del entorno de adquisición. Por ello, en este estudio se han seleccionado tres áreas con distintas características urbanas (altura de edificios, nivel de urbanización y áreas abiertas) de Abu Dhabi con el fin de analizar su influencia, tanto en la captura, como en la extracción de los elementos. En el caso de farolas, señales viales, árboles y aceras se ha realizado con software comercial, y para registros y edificios con los algoritmos propuestos. Las ratios de corrección y completitud, y la precisión geométrica se han calculado en las diferentes áreas urbanas. Los mejores resultados se han conseguido para las farolas, señales y bordillos, independientemente de la altura de los edificios. La peor ratio se obtuvo para los mismos elementos en áreas peri-urbanas, debido a la vegetación. Resultados opuestos se han conseguido en la detección de árboles. El coste económico y en tiempo de la metodología propuesta resulta inferior al de métodos tradicionales. Lo cual demuestra el gran potencial y eficiencia de la tecnología LiDAR móvil para la actualización cartografía de referenciaPostprint (published version

Lidar-based scene understanding for autonomous driving using deep learning

With over 1.35 million fatalities related to traffic accidents worldwide, autonomous driving was foreseen at the beginning of this century as a feasible solution to improve security in our roads. Nevertheless, it is meant to disrupt our transportation paradigm, allowing to reduce congestion, pollution, and costs, while increasing the accessibility, efficiency, and reliability of the transportation for both people and goods. Although some advances have gradually been transferred into commercial vehicles in the way of Advanced Driving Assistance Systems (ADAS) such as adaptive cruise control, blind spot detection or automatic parking, however, the technology is far from mature. A full understanding of the scene is actually needed so that allowing the vehicles to be aware of the surroundings, knowing the existing elements of the scene, as well as their motion, intentions and interactions. In this PhD dissertation, we explore new approaches for understanding driving scenes from 3D LiDAR point clouds by using Deep Learning methods. To this end, in Part I we analyze the scene from a static perspective using independent frames to detect the neighboring vehicles. Next, in Part II we develop new ways for understanding the dynamics of the scene. Finally, in Part III we apply all the developed methods to accomplish higher level challenges such as segmenting moving obstacles while obtaining their rigid motion vector over the ground. More specifically, in Chapter 2 we develop a 3D vehicle detection pipeline based on a multi-branch deep-learning architecture and propose a Front (FR-V) and a Bird’s Eye view (BE-V) as 2D representations of the 3D point cloud to serve as input for training our models. Later on, in Chapter 3 we apply and further test this method on two real uses-cases, for pre-filtering moving obstacles while creating maps to better localize ourselves on subsequent days, as well as for vehicle tracking. From the dynamic perspective, in Chapter 4 we learn from the 3D point cloud a novel dynamic feature that resembles optical flow from RGB images. For that, we develop a new approach to leverage RGB optical flow as pseudo ground truth for training purposes but allowing the use of only 3D LiDAR data at inference time. Additionally, in Chapter 5 we explore the benefits of combining classification and regression learning problems to face the optical flow estimation task in a joint coarse-and-fine manner. Lastly, in Chapter 6 we gather the previous methods and demonstrate that with these independent tasks we can guide the learning of higher challenging problems such as segmentation and motion estimation of moving vehicles from our own moving perspective.Con más de 1,35 millones de muertes por accidentes de tráfico en el mundo, a principios de siglo se predijo que la conducción autónoma sería una solución viable para mejorar la seguridad en nuestras carreteras. Además la conducción autónoma está destinada a cambiar nuestros paradigmas de transporte, permitiendo reducir la congestión del tráfico, la contaminación y el coste, a la vez que aumentando la accesibilidad, la eficiencia y confiabilidad del transporte tanto de personas como de mercancías. Aunque algunos avances, como el control de crucero adaptativo, la detección de puntos ciegos o el estacionamiento automático, se han transferido gradualmente a vehículos comerciales en la forma de los Sistemas Avanzados de Asistencia a la Conducción (ADAS), la tecnología aún no ha alcanzado el suficiente grado de madurez. Se necesita una comprensión completa de la escena para que los vehículos puedan entender el entorno, detectando los elementos presentes, así como su movimiento, intenciones e interacciones. En la presente tesis doctoral, exploramos nuevos enfoques para comprender escenarios de conducción utilizando nubes de puntos en 3D capturadas con sensores LiDAR, para lo cual empleamos métodos de aprendizaje profundo. Con este fin, en la Parte I analizamos la escena desde una perspectiva estática para detectar vehículos. A continuación, en la Parte II, desarrollamos nuevas formas de entender las dinámicas del entorno. Finalmente, en la Parte III aplicamos los métodos previamente desarrollados para lograr desafíos de nivel superior, como segmentar obstáculos dinámicos a la vez que estimamos su vector de movimiento sobre el suelo. Específicamente, en el Capítulo 2 detectamos vehículos en 3D creando una arquitectura de aprendizaje profundo de dos ramas y proponemos una vista frontal (FR-V) y una vista de pájaro (BE-V) como representaciones 2D de la nube de puntos 3D que sirven como entrada para entrenar nuestros modelos. Más adelante, en el Capítulo 3 aplicamos y probamos aún más este método en dos casos de uso reales, tanto para filtrar obstáculos en movimiento previamente a la creación de mapas sobre los que poder localizarnos mejor en los días posteriores, como para el seguimiento de vehículos. Desde la perspectiva dinámica, en el Capítulo 4 aprendemos de la nube de puntos en 3D una característica dinámica novedosa que se asemeja al flujo óptico sobre imágenes RGB. Para ello, desarrollamos un nuevo enfoque que aprovecha el flujo óptico RGB como pseudo muestras reales para entrenamiento, usando solo information 3D durante la inferencia. Además, en el Capítulo 5 exploramos los beneficios de combinar los aprendizajes de problemas de clasificación y regresión para la tarea de estimación de flujo óptico de manera conjunta. Por último, en el Capítulo 6 reunimos los métodos anteriores y demostramos que con estas tareas independientes podemos guiar el aprendizaje de problemas de más alto nivel, como la segmentación y estimación del movimiento de vehículos desde nuestra propia perspectivaAmb més d’1,35 milions de morts per accidents de trànsit al món, a principis de segle es va predir que la conducció autònoma es convertiria en una solució viable per millorar la seguretat a les nostres carreteres. D’altra banda, la conducció autònoma està destinada a canviar els paradigmes del transport, fent possible així reduir la densitat del trànsit, la contaminació i el cost, alhora que augmentant l’accessibilitat, l’eficiència i la confiança del transport tant de persones com de mercaderies. Encara que alguns avenços, com el control de creuer adaptatiu, la detecció de punts cecs o l’estacionament automàtic, s’han transferit gradualment a vehicles comercials en forma de Sistemes Avançats d’Assistència a la Conducció (ADAS), la tecnologia encara no ha arribat a aconseguir el grau suficient de maduresa. És necessària, doncs, una total comprensió de l’escena de manera que els vehicles puguin entendre l’entorn, detectant els elements presents, així com el seu moviment, intencions i interaccions. A la present tesi doctoral, explorem nous enfocaments per tal de comprendre les diferents escenes de conducció utilitzant núvols de punts en 3D capturats amb sensors LiDAR, mitjançant l’ús de mètodes d’aprenentatge profund. Amb aquest objectiu, a la Part I analitzem l’escena des d’una perspectiva estàtica per a detectar vehicles. A continuació, a la Part II, desenvolupem noves formes d’entendre les dinàmiques de l’entorn. Finalment, a la Part III apliquem els mètodes prèviament desenvolupats per a aconseguir desafiaments d’un nivell superior, com, per exemple, segmentar obstacles dinàmics al mateix temps que estimem el seu vector de moviment respecte al terra. Concretament, al Capítol 2 detectem vehicles en 3D creant una arquitectura d’aprenentatge profund amb dues branques, i proposem una vista frontal (FR-V) i una vista d’ocell (BE-V) com a representacions 2D del núvol de punts 3D que serveixen com a punt de partida per entrenar els nostres models. Més endavant, al Capítol 3 apliquem i provem de nou aquest mètode en dos casos d’ús reals, tant per filtrar obstacles en moviment prèviament a la creació de mapes en els quals poder localitzar-nos millor en dies posteriors, com per dur a terme el seguiment de vehicles. Des de la perspectiva dinàmica, al Capítol 4 aprenem una nova característica dinàmica del núvol de punts en 3D que s’assembla al flux òptic sobre imatges RGB. Per a fer-ho, desenvolupem un nou enfocament que aprofita el flux òptic RGB com pseudo mostres reals per a entrenament, utilitzant només informació 3D durant la inferència. Després, al Capítol 5 explorem els beneficis que s’obtenen de combinar els aprenentatges de problemes de classificació i regressió per la tasca d’estimació de flux òptic de manera conjunta. Finalment, al Capítol 6 posem en comú els mètodes anteriors i demostrem que mitjançant aquests processos independents podem abordar l’aprenentatge de problemes més complexos, com la segmentació i estimació del moviment de vehicles des de la nostra pròpia perspectiva

Low Power IoT based Automated Manhole Cover Monitoring System as a Smart City application

With the increased population in the big cities, Internet of Things (IoT) devices to be used as automated monitoring systems are required in many of the Smart city’s applications. Monitoring road infrastructure such as a manhole cover (MC) is one of these applications. Automating monitoring manhole cover structure has become more demanding, especially when the number of MC failure increases rapidly: it affects the safety, security and the economy of the society. Only 30% of the current MC monitoring systems are automated with short lifetime in comparison to the lifetime of the MC, without monitoring all the MC issues and without discussing the challenges of the design from IoT device design point of view. Extending the lifetime of a fully automated IoT-based MC monitoring system from circuit design point of view was studied and addressed in this research. The main circuit that consumes more power in the IoT-based MC monitoring system is the analogue to digital converter (ADC) found at the data acquisition module (DAQ). In several applications, the compressive sensing (CS) technique proved its capability to reduce the power consumption for ADC. In this research, CS has been investigated and studied deeply to reach the aim of the research. CS based ADC is named analogue to information converter (AIC). Because the heart of the AIC is the pseudorandom number generator (PRNG), several researchers have used it as a key to secure the data, which makes AIC more suitable for IoT device design. Most of these PRNG designs for AIC are hardware implemented in the digital circuit design. The presence of digital PRNG at the AIC analogue front end requires: a) isolating digital and analogue parts, and b) using two different power supplies and grounds for analogue and digital parts. On the other hand, analogue circuit design becomes more demanding for the sake of the power consumption, especially after merging the analogue circuit design with other fields such as neural networks and neuroscience. This has motivated the researcher to propose two low-power analogue chaotic oscillators to replace digital PRNG using opamp Schmitt Trigger. The proposed systems are based on a coupling oscillator concept. The design of the proposed systems is based on: First, two new modifications for the well-known astable multivibrator using opamp Schmitt trigger. Second, the waveshaping design technique is presented to design analogue chaotic oscillators instead of starting with complex differential equations as it is the case for most of the chaotic oscillator designs. This technique helps to find easy steps and understanding of building analogue chaotic oscillators for electronic circuit designers. The proposed systems used off the shelf components as a proof of concept. The proposed systems were validated based on: a) the range of the temperature found beneath a manhole cover, and b) the signal reconstruction under the presence and the absence of noise. The results show decent performance of the proposed system from the power consumption point of view, as it can exceed the lifetime of similar two opamps based Jerk chaotic oscillators by almost one year for long lifetime applications such as monitoring MC using Li-Ion battery. Furthermore, in comparison to PRNG output sequence generated by a software algorithm used in AIC framework in the presence of the noise, the first proposed system output sequence improved the signal reconstruction by 6.94%, while the second system improved the signal reconstruction by 17.83