A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

An approach for real world data modelling with the 3D terrestrial laser scanner for built environment

Capturing and modelling 3D information of the built environment is a big challenge. A number of techniques and technologies are now in use. These include EDM, GPS, and photogrammetric application, remote sensing and traditional building surveying applications. However, use of these technologies cannot be practical and efficient in regard to time, cost and accuracy. Furthermore, a multi disciplinary knowledge base, created from the studies and research about the regeneration aspects is fundamental: historical, architectural, archeologically, environmental, social, economic, etc. In order to have an adequate diagnosis of regeneration, it is necessary to describe buildings and surroundings by means of documentation and plans. However, at this point in time the foregoing is considerably far removed from the real situation, since more often than not it is extremely difficult to obtain full documentation and cartography, of an acceptable quality, since the material, constructive pathologies and systems are often insufficient or deficient (flat that simply reflects levels, isolated photographs,..). Sometimes the information in reality exists, but this fact is not known, or it is not easily accessible, leading to the unnecessary duplication of efforts and resources. In this paper, we discussed 3D laser scanning technology, which can acquire high density point data in an accurate, fast way. Besides, the scanner can digitize all the 3D information concerned with a real world object such as buildings, trees and terrain down to millimetre detail Therefore, it can provide benefits for refurbishment process in regeneration in the Built Environment and it can be the potential solution to overcome the challenges above. The paper introduce an approach for scanning buildings, processing the point cloud raw data, and a modelling approach for CAD extraction and building objects classification by a pattern matching approach in IFC (Industry Foundation Classes) format. The approach presented in this paper from an undertaken research can lead to parametric design and Building Information Modelling (BIM) for existing structures. Two case studies are introduced to demonstrate the use of laser scanner technology in the Built Environment. These case studies are the Jactin House Building in East Manchester and the Peel building in the campus of University Salford. Through these case studies, while use of laser scanners are explained, the integration of it with various technologies and systems are also explored for professionals in Built Environmen

Airborne photogrammetry and LIDAR for DSM extraction and 3D change detection over an urban area : a comparative study

A digital surface model (DSM) extracted from stereoscopic aerial images, acquired in March 2000, is compared with a DSM derived from airborne light detection and ranging (lidar) data collected in July 2009. Three densely built-up study areas in the city centre of Ghent, Belgium, are selected, each covering approximately 0.4 km(2). The surface models, generated from the two different 3D acquisition methods, are compared qualitatively and quantitatively as to what extent they are suitable in modelling an urban environment, in particular for the 3D reconstruction of buildings. Then the data sets, which are acquired at two different epochs t(1) and t(2), are investigated as to what extent 3D (building) changes can be detected and modelled over the time interval. A difference model, generated by pixel-wise subtracting of both DSMs, indicates changes in elevation. Filters are proposed to differentiate 'real' building changes from false alarms provoked by model noise, outliers, vegetation, etc. A final 3D building change model maps all destructed and newly constructed buildings within the time interval t(2) - t(1). Based on the change model, the surface and volume of the building changes can be quantified

Update urban basemap by using the LiDAR mobile mapping system : the case of Abu Dhabi municipal system

Basemaps are the main resource used in urban planning and in building and infrastructure asset management. These maps are used by citizens and by private and public stakeholders. Therefore, accurate, up-to-date geoinformation of reference are needed to provide a good service. In general, basemaps have been updated by aerial photogrammetry or field surveying, but these methods are not always possible and alternatives need to be sought. Current limitations and challenges that face traditional field surveys include areas with extreme weather, deserts or artic environments, and flight restrictions due to proximity with other countries if there is not an agreement. In such cases, alternatives for large-scale are required. This thesis proposes the use of a mobile mapping system (MMS) to update urban basemaps. Most urban features can be extracted from point cloud using commercial software or open libraries. However, there are some exceptions: manhole covers, or hidden elements even with captures from defferent perspective, the most common building corners. Therefore, the main objective of this study was to establish a methodology for extracting manholes automatically and for completing hidden corners of buildings, so that urban basemaps can be updated. The algorithm developed to extract manholes is based on time, intensity and shape detection parameters, whereas additional information from satellite images is used to complete buildings. Each municipality knows the materials and dimensions of its manholes. Taking advantage of this knowledge, the point cloud is filtered to classify points according to the set of intensity values associated with the manhole material. From the classified points, the minimum bounding rectangles (MBR) are obtained and finally the shape is adjusted and drawn. We use satellite imagery to automatically digitize the layout of building footprints with automated software tools. Then, the visible corners of buildings from the LiDAR point cloud are imported and a fitting process is performed by comparing them with the corners of the building from the satellite image. Two methods are evaluated to establish which is the most suitable for adjustment in these conditions. In the first method, the differences in X and Y directions are measured in the corners, where LiDAR and satellite data are available, and is often computed as the average of the offsets. In the second method, a Helmert 2D transformation is applied. MMS involves Global Navigation Satellite Systems (GNSS) and Inertial Measurement Units (IMU) to georeference point clouds. Their accuracy depends on the acquisition environment. In this study, the influence of the urban pattern is analysed in three zones with varied urban characteristics: different height buildings, open areas, and areas with a low and high level of urbanization. To evaluate the efficiency of the proposed algorithms, three areas were chosen with varying urban patterns in Abu Dhabi. In these areas, 3D urban elements (light poles, street signs, etc) were automatically extracted using commercial software. The proposed algorithms were applied to the manholes and buildings. The completeness and correctness ratio, and geometric accuracy were calculated for all urban elements in the three areas. The best success rates (>70%) were for light poles, street signs and road curbs, regardless of the height of the buildings. The worst rate was obtained for the same features in peri-urban areas, due to high vegetation. In contrast, the best results for trees were found in theses areas. Our methodology demonstrates the great potential and efficiency of mobile LiDAR technology in updating basemaps; a process that is required to achieve standard accuracy in large scale maps. The cost of the entire process and the time required for the proposed methodology was calculated and compared with the traditional method. It was found that mobile LiDAR could be a standard cost-efficient procedure for updating maps.La cartografía de referencia es la principal herramienta en planificación urbanística, y gestión de infraestructuras y edificios, al servicio de ciudadanos, empresas y administración. Por esta razón, debe estar actualizada y ser lo más precisa posible. Tradicionalmente, la cartografía se actualiza mediante fotogrametría aérea o levantamientos terrestres. No obstante, deben buscarse alternativas válidas para escalas grandes, porque no siempre es posible emplear estas técnicas debido a las limitaciones y retos actuales a los que se enfrenta la medición tradicional en algunas zonas del planeta, con meteorología extrema o restricciones de vuelo por la proximidad a la frontera con otros países. Esta tesis propone el uso del sistema Mobile Mapping System (MMS) para actualizar la cartografía urbana de referencia. La mayoría de los elementos pueden extraerse empleando software comercial o librerías abiertas, excepto los registros de servicios. Los elementos ocultos son otro de los inconvenientes encontrados en el proceso de creación o actualización de la cartografía, incluso si se dispone de capturas desde diferentes puntos de vista. El caso más común es el de las esquinas de edificios. Por ello, el principal objetivo de este estudio es establecer una metodología de extracción automática de los registros y completar las esquinas ocultas de los edificios para actualizar cartografía urbana. El algoritmo desarrollado para la detección y extracción de registros se basa en parámetros como el tiempo, la intensidad de la señal laser y la forma de los registros, mientras que para completar los edificios se emplea información adicional de imágenes satélite. Aprovechando el conocimiento del material y dimensión de los registros, en disposición de los gestores municipales, el algoritmo propuesto filtra y clasifica los puntos de acuerdo a los valores de intensidad. De aquellos clasificados como registros se calcula el mínimo rectángulo que los contiene (Minimum Bounding Rectangle) y finalmente se ajusta la forma y se dibuja. Las imágenes de satélite son empleadas para obtener automáticamente la huella de los edificios. Posteriormente, se importan las esquinas visibles de los edificios obtenidas desde la nube de puntos y se realiza el ajuste comparándolas con las obtenidas desde satélite. Para llevar a cabo este ajuste se han evaluado dos métodos, el primero de ellos considera las diferencias entre las coordenadas XY, desplazándose el promedio. En el segundo, se aplica una transformación Helmert2D. MMS emplea sistemas de navegación global por satélite (Global Navigation Satellite Systems, GNSS) e inerciales (Inertial Measurement Unit, IMU) para georreferenciar la nube de puntos. La precisión de estos sistemas de posicionamiento depende del entorno de adquisición. Por ello, en este estudio se han seleccionado tres áreas con distintas características urbanas (altura de edificios, nivel de urbanización y áreas abiertas) de Abu Dhabi con el fin de analizar su influencia, tanto en la captura, como en la extracción de los elementos. En el caso de farolas, señales viales, árboles y aceras se ha realizado con software comercial, y para registros y edificios con los algoritmos propuestos. Las ratios de corrección y completitud, y la precisión geométrica se han calculado en las diferentes áreas urbanas. Los mejores resultados se han conseguido para las farolas, señales y bordillos, independientemente de la altura de los edificios. La peor ratio se obtuvo para los mismos elementos en áreas peri-urbanas, debido a la vegetación. Resultados opuestos se han conseguido en la detección de árboles. El coste económico y en tiempo de la metodología propuesta resulta inferior al de métodos tradicionales. Lo cual demuestra el gran potencial y eficiencia de la tecnología LiDAR móvil para la actualización cartografía de referenciaPostprint (published version

Forestry and Arboriculture Applications Using High-Resolution Imagery from Unmanned Aerial Vehicles (UAV)

Forests cover over one-third of the planet and provide unmeasurable benefits to the ecosystem. Forest managers have collected and processed countless amounts of data for use in studying, planning, and management of these forests. Data collection has evolved from completely manual operations to the incorporation of technology that has increased the efficiency of data collection and decreased overall costs. Many technological advances have been made that can be incorporated into natural resources disciplines. Laser measuring devices, handheld data collectors and more recently, unmanned aerial vehicles, are just a few items that are playing a major role in the way data is managed and collected. Field hardware has also been aided with new and improved mobile and computer software. Over the course of this study, field technology along with computer advancements have been utilized to aid in forestry and arboricultural applications. Three-dimensional point cloud data that represent tree shape and height were extracted and examined for accuracy. Traditional fieldwork collection (tree height, tree diameter and canopy metrics) was derived from remotely sensed data by using new modeling techniques which will result in time and cost savings. Using high resolution aerial photography, individual tree species are classified to support tree inventory development. Point clouds were used to create digital elevation models (DEM) which can further be used in hydrology analysis, slope, aspect, and hillshades. Digital terrain models (DTM) are in geographic information system (GIS), and along with DEMs, used to create canopy height models (CHM). The results of this study can enhance how the data are utilized and prompt further research and new initiatives that will improve and garner new insight for the use of remotely sensed data in forest management