Automatic manhole extraction from MMS data to update basemaps

Basemaps are the main resource used in urban planning, building and infrastructure asset management. Therefore, they must be accurate and up to date to better serve citizens, contractors, property owners and town planning departments. Traditionally, they have been updated by aerial photogrammetry, but this is not always possible and alternatives need to be sought. In such cases, a useful option for large scales is the mobile mapping system (MMS). However, automatic extraction from MMS point clouds is limited by the complexity of the urban environment. Therefore, the influence of the urban pattern is analysed in three zones with varied urban characteristics: areas with high buildings, open areas, and areas with a low level of urbanization. In these areas, the capture and automatic extraction of 3D urban elements is performed using commercial software, which is useful for some elements but not for manholes. The objective of this study is to establish a methodology for extracting manholes automatically and completing hidden buildings' corners, in order to update urban basemaps. Shape and intensity are the main detection parameters for manholes, whereas additional information from satellite image Quickbird is used to complete the buildings. The worst rate of detection for all the extracted urban elements was found in areas of high buildings. Finally, the article analyses the computing cost for manhole extraction, and the economic cost and time consume of the entire process, including the proposed methodolgy using an MMS point cloud and the traditional survey in this case.Peer ReviewedPostprint (updated version

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

KemptenCity - Semantic Segmentation of Urban Areas for Simulation

Autonomous driving and traffic flow simulation requires a realistic and accurate representation of the environment. Therefore, this research focuses on the semantic segmentation of aerial images for simulation purposes. Initially, a dataset was created based on true orthophotos from 2019 and Kempten’s street cadaster, with true orthophotos being fully rectified aerial images. The chosen classes were oriented towards the subsequent conversion and usage in simulation. The proposed labeling workflow used cadaster data and demonstrated significant time efficiency compared to state-of-the-art datasets. Subsequently, a neural network was implemented that was trained and tested on the dataset. In addition, the network was also trained only on the lane markings to compare the network’s performance. Both cases demonstrated excellent segmentation results. The generalizability was then tested on true orthophotos from 2021. The results indicated a solid generalizability, but still needs to be improved. Finally, the aerial information was converted into a 3D environment, that can be used in simulations. Our results confirm the usage of aerial imagery and street cadaster data as a basis for the simulations

Update urban basemap by using the LiDAR mobile mapping system : the case of Abu Dhabi municipal system

Basemaps are the main resource used in urban planning and in building and infrastructure asset management. These maps are used by citizens and by private and public stakeholders. Therefore, accurate, up-to-date geoinformation of reference are needed to provide a good service. In general, basemaps have been updated by aerial photogrammetry or field surveying, but these methods are not always possible and alternatives need to be sought. Current limitations and challenges that face traditional field surveys include areas with extreme weather, deserts or artic environments, and flight restrictions due to proximity with other countries if there is not an agreement. In such cases, alternatives for large-scale are required. This thesis proposes the use of a mobile mapping system (MMS) to update urban basemaps. Most urban features can be extracted from point cloud using commercial software or open libraries. However, there are some exceptions: manhole covers, or hidden elements even with captures from defferent perspective, the most common building corners. Therefore, the main objective of this study was to establish a methodology for extracting manholes automatically and for completing hidden corners of buildings, so that urban basemaps can be updated. The algorithm developed to extract manholes is based on time, intensity and shape detection parameters, whereas additional information from satellite images is used to complete buildings. Each municipality knows the materials and dimensions of its manholes. Taking advantage of this knowledge, the point cloud is filtered to classify points according to the set of intensity values associated with the manhole material. From the classified points, the minimum bounding rectangles (MBR) are obtained and finally the shape is adjusted and drawn. We use satellite imagery to automatically digitize the layout of building footprints with automated software tools. Then, the visible corners of buildings from the LiDAR point cloud are imported and a fitting process is performed by comparing them with the corners of the building from the satellite image. Two methods are evaluated to establish which is the most suitable for adjustment in these conditions. In the first method, the differences in X and Y directions are measured in the corners, where LiDAR and satellite data are available, and is often computed as the average of the offsets. In the second method, a Helmert 2D transformation is applied. MMS involves Global Navigation Satellite Systems (GNSS) and Inertial Measurement Units (IMU) to georeference point clouds. Their accuracy depends on the acquisition environment. In this study, the influence of the urban pattern is analysed in three zones with varied urban characteristics: different height buildings, open areas, and areas with a low and high level of urbanization. To evaluate the efficiency of the proposed algorithms, three areas were chosen with varying urban patterns in Abu Dhabi. In these areas, 3D urban elements (light poles, street signs, etc) were automatically extracted using commercial software. The proposed algorithms were applied to the manholes and buildings. The completeness and correctness ratio, and geometric accuracy were calculated for all urban elements in the three areas. The best success rates (>70%) were for light poles, street signs and road curbs, regardless of the height of the buildings. The worst rate was obtained for the same features in peri-urban areas, due to high vegetation. In contrast, the best results for trees were found in theses areas. Our methodology demonstrates the great potential and efficiency of mobile LiDAR technology in updating basemaps; a process that is required to achieve standard accuracy in large scale maps. The cost of the entire process and the time required for the proposed methodology was calculated and compared with the traditional method. It was found that mobile LiDAR could be a standard cost-efficient procedure for updating maps.La cartografía de referencia es la principal herramienta en planificación urbanística, y gestión de infraestructuras y edificios, al servicio de ciudadanos, empresas y administración. Por esta razón, debe estar actualizada y ser lo más precisa posible. Tradicionalmente, la cartografía se actualiza mediante fotogrametría aérea o levantamientos terrestres. No obstante, deben buscarse alternativas válidas para escalas grandes, porque no siempre es posible emplear estas técnicas debido a las limitaciones y retos actuales a los que se enfrenta la medición tradicional en algunas zonas del planeta, con meteorología extrema o restricciones de vuelo por la proximidad a la frontera con otros países. Esta tesis propone el uso del sistema Mobile Mapping System (MMS) para actualizar la cartografía urbana de referencia. La mayoría de los elementos pueden extraerse empleando software comercial o librerías abiertas, excepto los registros de servicios. Los elementos ocultos son otro de los inconvenientes encontrados en el proceso de creación o actualización de la cartografía, incluso si se dispone de capturas desde diferentes puntos de vista. El caso más común es el de las esquinas de edificios. Por ello, el principal objetivo de este estudio es establecer una metodología de extracción automática de los registros y completar las esquinas ocultas de los edificios para actualizar cartografía urbana. El algoritmo desarrollado para la detección y extracción de registros se basa en parámetros como el tiempo, la intensidad de la señal laser y la forma de los registros, mientras que para completar los edificios se emplea información adicional de imágenes satélite. Aprovechando el conocimiento del material y dimensión de los registros, en disposición de los gestores municipales, el algoritmo propuesto filtra y clasifica los puntos de acuerdo a los valores de intensidad. De aquellos clasificados como registros se calcula el mínimo rectángulo que los contiene (Minimum Bounding Rectangle) y finalmente se ajusta la forma y se dibuja. Las imágenes de satélite son empleadas para obtener automáticamente la huella de los edificios. Posteriormente, se importan las esquinas visibles de los edificios obtenidas desde la nube de puntos y se realiza el ajuste comparándolas con las obtenidas desde satélite. Para llevar a cabo este ajuste se han evaluado dos métodos, el primero de ellos considera las diferencias entre las coordenadas XY, desplazándose el promedio. En el segundo, se aplica una transformación Helmert2D. MMS emplea sistemas de navegación global por satélite (Global Navigation Satellite Systems, GNSS) e inerciales (Inertial Measurement Unit, IMU) para georreferenciar la nube de puntos. La precisión de estos sistemas de posicionamiento depende del entorno de adquisición. Por ello, en este estudio se han seleccionado tres áreas con distintas características urbanas (altura de edificios, nivel de urbanización y áreas abiertas) de Abu Dhabi con el fin de analizar su influencia, tanto en la captura, como en la extracción de los elementos. En el caso de farolas, señales viales, árboles y aceras se ha realizado con software comercial, y para registros y edificios con los algoritmos propuestos. Las ratios de corrección y completitud, y la precisión geométrica se han calculado en las diferentes áreas urbanas. Los mejores resultados se han conseguido para las farolas, señales y bordillos, independientemente de la altura de los edificios. La peor ratio se obtuvo para los mismos elementos en áreas peri-urbanas, debido a la vegetación. Resultados opuestos se han conseguido en la detección de árboles. El coste económico y en tiempo de la metodología propuesta resulta inferior al de métodos tradicionales. Lo cual demuestra el gran potencial y eficiencia de la tecnología LiDAR móvil para la actualización cartografía de referenciaPostprint (published version

An Exploration of Recent Intelligent Image Analysis Techniques for Visual Pavement Surface Condition Assessment.

Road pavement condition assessment is essential for maintenance, asset management, and budgeting for pavement infrastructure. Countries allocate a substantial annual budget to maintain and improve local, regional, and national highways. Pavement condition is assessed by measuring several pavement characteristics such as roughness, surface skid resistance, pavement strength, deflection, and visual surface distresses. Visual inspection identifies and quantifies surface distresses, and the condition is assessed using standard rating scales. This paper critically analyzes the research trends in the academic literature, professional practices and current commercial solutions for surface condition ratings by civil authorities. We observe that various surface condition rating systems exist, and each uses its own defined subset of pavement characteristics to evaluate pavement conditions. It is noted that automated visual sensing systems using intelligent algorithms can help reduce the cost and time required for assessing the condition of pavement infrastructure, especially for local and regional road networks. However, environmental factors, pavement types, and image collection devices are significant in this domain and lead to challenging variations. Commercial solutions for automatic pavement assessment with certain limitations exist. The topic is also a focus of academic research. More recently, academic research has pivoted toward deep learning, given that image data is now available in some form. However, research to automate pavement distress assessment often focuses on the regional pavement condition assessment standard that a country or state follows. We observe that the criteria a region adopts to make the evaluation depends on factors such as pavement construction type, type of road network in the area, flow and traffic, environmental conditions, and region\u27s economic situation. We summarized a list of publicly available datasets for distress detection and pavement condition assessment. We listed approaches focusing on crack segmentation and methods concentrating on distress detection and identification using object detection and classification. We segregated the recent academic literature in terms of the camera\u27s view and the dataset used, the year and country in which the work was published, the F1 score, and the architecture type. It is observed that the literature tends to focus more on distress identification ( presence/absence detection) but less on distress quantification, which is essential for developing approaches for automated pavement rating

Simultaneous localization and mapping for inspection robots in water and sewer pipe networks: a review

At the present time, water and sewer pipe networks are predominantly inspected manually. In the near future, smart cities will perform intelligent autonomous monitoring of buried pipe networks, using teams of small robots. These robots, equipped with all necessary computational facilities and sensors (optical, acoustic, inertial, thermal, pressure and others) will be able to inspect pipes whilst navigating, selflocalising and communicating information about the pipe condition and faults such as leaks or blockages to human operators for monitoring and decision support. The predominantly manual inspection of pipe networks will be replaced with teams of autonomous inspection robots that can operate for long periods of time over a large spatial scale. Reliable autonomous navigation and reporting of faults at this scale requires effective localization and mapping, which is the estimation of the robot’s position and its surrounding environment. This survey presents an overview of state-of-the-art works on robot simultaneous localization and mapping (SLAM) with a focus on water and sewer pipe networks. It considers various aspects of the SLAM problem in pipes, from the motivation, to the water industry requirements, modern SLAM methods, map-types and sensors suited to pipes. Future challenges such as robustness for long term robot operation in pipes are discussed, including how making use of prior knowledge, e.g. geographic information systems (GIS) can be used to build map estimates, and improve the multi-robot SLAM in the pipe environmen

Benchmarking Anchor-Based and Anchor-Free State-of-the-Art Deep Learning Methods for Individual Tree Detection in RGB High-Resolution Images

rban forests contribute to maintaining livability and increase the resilience of cities in the face of population growth and climate change. Information about the geographical distribution of individual trees is essential for the proper management of these systems. RGB high-resolution aerial images have emerged as a cheap and efficient source of data, although detecting and mapping single trees in an urban environment is a challenging task. Thus, we propose the evaluation of novel methods for single tree crown detection, as most of these methods have not been investigated in remote sensing applications. A total of 21 methods were investigated, including anchor-based (one and two-stage) and anchor-free state-of-the-art deep-learning methods. We used two orthoimages divided into 220 non-overlapping patches of 512 × 512 pixels with a ground sample distance (GSD) of 10 cm. The orthoimages were manually annotated, and 3382 single tree crowns were identified as the ground-truth. Our findings show that the anchor-free detectors achieved the best average performance with an AP50 of 0.686. We observed that the two-stage anchor-based and anchor-free methods showed better performance for this task, emphasizing the FSAF, Double Heads, CARAFE, ATSS, and FoveaBox models. RetinaNet, which is currently commonly applied in remote sensing, did not show satisfactory performance, and Faster R-CNN had lower results than the best methods but with no statistically significant difference. Our findings contribute to a better understanding of the performance of novel deep-learning methods in remote sensing applications and could be used as an indicator of the most suitable methods in such applications

Toulouse Hyperspectral Data Set: a benchmark data set to assess semi-supervised spectral representation learning and pixel-wise classification techniques

Airborne hyperspectral images can be used to map the land cover in large urban areas, thanks to their very high spatial and spectral resolutions on a wide spectral domain. While the spectral dimension of hyperspectral images is highly informative of the chemical composition of the land surface, the use of state-of-the-art machine learning algorithms to map the land cover has been dramatically limited by the availability of training data. To cope with the scarcity of annotations, semi-supervised and self-supervised techniques have lately raised a lot of interest in the community. Yet, the publicly available hyperspectral data sets commonly used to benchmark machine learning models are not totally suited to evaluate their generalization performances due to one or several of the following properties: a limited geographical coverage (which does not reflect the spectral diversity in metropolitan areas), a small number of land cover classes and a lack of appropriate standard train / test splits for semi-supervised and self-supervised learning. Therefore, we release in this paper the Toulouse Hyperspectral Data Set that stands out from other data sets in the above-mentioned respects in order to meet key issues in spectral representation learning and classification over large-scale hyperspectral images with very few labeled pixels. Besides, we discuss and experiment the self-supervised task of Masked Autoencoders and establish a baseline for pixel-wise classification based on a conventional autoencoder combined with a Random Forest classifier achieving 82% overall accuracy and 74% F1 score. The Toulouse Hyperspectral Data Set and our code are publicly available at https://www.toulouse-hyperspectral-data-set.com and https://www.github.com/Romain3Ch216/tlse-experiments, respectively.Comment: 17 pages, 13 figure