Dense semantic labeling of sub-decimeter resolution images with convolutional neural networks

Semantic labeling (or pixel-level land-cover classification) in ultra-high resolution imagery (< 10cm) requires statistical models able to learn high level concepts from spatial data, with large appearance variations. Convolutional Neural Networks (CNNs) achieve this goal by learning discriminatively a hierarchy of representations of increasing abstraction. In this paper we present a CNN-based system relying on an downsample-then-upsample architecture. Specifically, it first learns a rough spatial map of high-level representations by means of convolutions and then learns to upsample them back to the original resolution by deconvolutions. By doing so, the CNN learns to densely label every pixel at the original resolution of the image. This results in many advantages, including i) state-of-the-art numerical accuracy, ii) improved geometric accuracy of predictions and iii) high efficiency at inference time. We test the proposed system on the Vaihingen and Potsdam sub-decimeter resolution datasets, involving semantic labeling of aerial images of 9cm and 5cm resolution, respectively. These datasets are composed by many large and fully annotated tiles allowing an unbiased evaluation of models making use of spatial information. We do so by comparing two standard CNN architectures to the proposed one: standard patch classification, prediction of local label patches by employing only convolutions and full patch labeling by employing deconvolutions. All the systems compare favorably or outperform a state-of-the-art baseline relying on superpixels and powerful appearance descriptors. The proposed full patch labeling CNN outperforms these models by a large margin, also showing a very appealing inference time.Comment: Accepted in IEEE Transactions on Geoscience and Remote Sensing, 201

A Markov Chain Random Field Cosimulation-Based Approach for Land Cover Post-classification and Urban Growth Detection

The recently proposed Markov chain random field (MCRF) approach has great potential to significantly improve land cover classification accuracy when used as a post-classification method by taking advantage of expert-interpreted data and pre-classified image data. This doctoral dissertation explores the effectiveness of the MCRF cosimulation (coMCRF) model in land cover post-classification and further improves it for land cover post-classification and urban growth detection. The intellectual merits of this research include the following aspects: First, by examining the coMCRF method in different conditions, this study provides land cover classification researchers with a solid reference regarding the performance of the coMCRF method for land cover post-classification. Second, this study provides a creative idea to reduce the smoothing effect in land cover post-classification by incorporating spectral similarity into the coMCRF method, which should be also applicable to other geostatistical models. Third, developing an integrated framework by integrating multisource data, spatial statistical models, and morphological operator reasoning for large area urban vertical and horizontal growth detection from medium resolution remotely sensed images enables us to detect and study the footprint of vertical and horizontal urbanization so that we can understand global urbanization from a new angle. Such a new technology can be transformative to urban growth study. The broader impacts of this research are concentrated on several points: The first point is that the coMCRF method and the integrated approach will be turned into open access user-friendly software with a graphical user interface (GUI) and an ArcGIS tool. Researchers and other users will be able to use them to produce high-quality land cover maps or improve the quality of existing land cover maps. The second point is that these research results will lead to a better insight of urban growth in terms of horizontal and vertical dimensions, as well as the spatial and temporal relationships between urban horizontal and vertical growth and changes in socioeconomic variables. The third point is that all products will be archived and shared on the Internet

Tensor-based Hyperspectral Image Processing Methodology and its Applications in Impervious Surface and Land Cover Mapping

The emergence of hyperspectral imaging provides a new perspective for Earth observation, in addition to previously available orthophoto and multispectral imagery. This thesis focused on both the new data and new methodology in the field of hyperspectral imaging. First, the application of the future hyperspectral satellite EnMAP in impervious surface area (ISA) mapping was studied. During the search for the appropriate ISA mapping procedure for the new data, the subpixel classification based on nonnegative matrix factorization (NMF) achieved the best success. The simulated EnMAP image shows great potential in urban ISA mapping with over 85% accuracy. Unfortunately, the NMF based on the linear algebra only considers the spectral information and neglects the spatial information in the original image. The recent wide interest of applying the multilinear algebra in computer vision sheds light on this problem and raised the idea of nonnegative tensor factorization (NTF). This thesis found that the NTF has more advantages over the NMF when work with medium- rather than the high-spatial-resolution hyperspectral image. Furthermore, this thesis proposed to equip the NTF-based subpixel classification methods with the variations adopted from the NMF. By adopting the variations from the NMF, the urban ISA mapping results from the NTF were improved by ~2%. Lastly, the problem known as the curse of dimensionality is an obstacle in hyperspectral image applications. The majority of current dimension reduction (DR) methods are restricted to using only the spectral information, when the spatial information is neglected. To overcome this defect, two spectral-spatial methods: patch-based and tensor-patch-based, were thoroughly studied and compared in this thesis. To date, the popularity of the two solutions remains in computer vision studies and their applications in hyperspectral DR are limited. The patch-based and tensor-patch-based variations greatly improved the quality of dimension-reduced hyperspectral images, which then improved the land cover mapping results from them. In addition, this thesis proposed to use an improved method to produce an important intermediate result in the patch-based and tensor-patch-based DR process, which further improved the land cover mapping results

Geoscience-aware deep learning:A new paradigm for remote sensing

Information extraction is a key activity for remote sensing images. A common distinction exists between knowledge-driven and data-driven methods. Knowledge-driven methods have advanced reasoning ability and interpretability, but have difficulty in handling complicated tasks since prior knowledge is usually limited when facing the highly complex spatial patterns and geoscience phenomena found in reality. Data-driven models, especially those emerging in machine learning (ML) and deep learning (DL), have achieved substantial progress in geoscience and remote sensing applications. Although DL models have powerful feature learning and representation capabilities, traditional DL has inherent problems including working as a black box and generally requiring a large number of labeled training data. The focus of this paper is on methods that integrate domain knowledge, such as geoscience knowledge and geoscience features (GK/GFs), into the design of DL models. The paper introduces the new paradigm of geoscience-aware deep learning (GADL), in which GK/GFs and DL models are combined deeply to extract information from remote sensing data. It first provides a comprehensive summary of GK/GFs used in GADL, which forms the basis for subsequent integration of GK/GFs with DL models. This is followed by a taxonomy of approaches for integrating GK/GFs with DL models. Several approaches are detailed using illustrative examples. Challenges and research prospects in GADL are then discussed. Developing more novel and advanced methods in GADL is expected to become the prevailing trend in advancing remotely sensed information extraction in the future.</p

An integrative approach using remote sensing and social analysis to identify different settlement types and the specific living conditions of its inhabitants

Someday in 2007, the world population reached a historical landmark: for the first time in human history, more than half of the world´s population was urban. A stagnation of this urbanization process is not in sight, so that by 2050, already 70 percent of humankind is projected to live in urban settlements. Over the last few decades, enormous migrations from rural hinterlands to steadily growing cities could be witnessed coming along with a dramatic growth of the world’s urban population. The speed and the scale of this growth, particularly in the so called less developed regions, are posing tremendous challenges to the countries concerned as well as to the world community. Within mega cities the strongest trends and the most extreme dimensions of the urbanization process can be observed. Their rapid growth results in uncontrolled processes of fragmentation which is often associated with pronounced poverty, social inequality, socio-spatial and political fragmentation, environmental degradation as well as population demands that outstrip environmental service capacity. For the majority of the mega cities a tremendous increase of informal structures and processes has to be observed. Consequentially informal settlements are growing, which represent those characteristic municipal areas being subject to particularly high population density, dynamics as well as marginalization. They have quickly become the most visible expression of urban poverty in developing world cities. Due to the extreme dynamics, the high complexity and huge spatial dimension of mega cities, urban administrations often only have an obsolete or not even existing data basis available to be at all informed about developments, trends and dimensions of urban growth and change. The knowledge about the living conditions of the residents is correspondingly very limited, incomplete and not up to date. Traditional methods such as statistical and regional analyses or fieldwork are no longer capable to capture such urban process. New data sources and monitoring methodologies are required in order to provide an up to date information basis as well as planning strate¬gies to enable sustainable developments and to simplify planning processes in complex urban structures. This research shall seize the described problem and aims to make a contribution to the requirements of monitoring fast developing mega cities. Against this background a methodology is developed to compensate the lack of socio-economic data and to deduce meaningful information on the living conditions of the inhabitants of mega cities. Neither social science methods alone nor the exclusive analysis of remote sensing data can solve the problem of the poor quality and outdated data base. Conventional social science methods cannot cope with the enormous developments and the tremendous growth as they are too labor-, as well as too time- and too cost-intensive. On the other hand, the physical discipline of remote sensing does not allow for direct conclusions on social parameters out of remote sensing images. The prime objective of this research is therefore the development of an integrative approach − bridging remote sensing and social analysis – in order to derive useful information about the living conditions in this specific case of the mega city Delhi and its inhabitants. Hence, this work is established in the overlapping range of the research topics remote sensing, urban areas and social science. Delhi, as India’s fast growing capital, meanwhile with almost 25 million residents the second largest city of the world, represents a prime example of a mega city. Since the second half of the 20th century, Delhi has been transformed from a modest town with mainly administrative and trade-related functions to a complex metropolis with a steep socio-economic gradient. The quality and amount of administrative and socio-economic data are poor and the knowledge about the circumstances of Delhi’s residents is correspondingly insufficient and outdated. Delhi represents therefore a perfectly suited study area for this research. In order to gather information about the living conditions within the different settlement types a methodology was developed and conducted to analyze the urban environment of the mega city Delhi. To identify different settlement types within the urban area, regarding the complex and heterogeneous appearance of the Delhi area, a semi-automated, object-oriented classification approach, based on segmentation derived image objects, was implemented. As the complete conceptual framework of this research, the classification methodology was developed based on a smaller representative training area at first and applied to larger test sites within Delhi afterwards. The object-oriented classification of VHR satellite imagery of the QuickBird sensor allowed for the identification of five different urban land cover classes within the municipal area of Delhi. In the focus of the image analysis is yet the identification of different settlement types and amongst these of informal settlements in particular. The results presented within this study demonstrate, that, based on density classes, the developed methodology is suitable to identify different settlement types and to detect informal settlements which are mega urban risk areas and thus potential residential zones of vulnerable population groups. The remote sensing derived land cover maps form the foundation for the integrative analysis concept and deliver there¬fore the general basis for the derivation of social attributes out of remote sensing data. For this purpose settlement characteristics (e.g., area of the settlement, average building size, and number of houses) are estimated from the classified QuickBird data and used to derive spatial information about the population distribution. In a next step, the derived information is combined with in-situ information on socio-economic conditions (e.g., family size, mean water consumption per capita/family) extracted from georeferenced questionnaires conducted during two field trips in Delhi. This combined data is used to characterize a given settlement type in terms of specific population and water related variables (e.g., population density, total water consumption). With this integrative methodology a catalogue can be compiled, comprising the living conditions of Delhi’s inhabitants living in specific settlement structures – and this in a quick, large-scaled, cost effective, by random or regularly repeatable way with a relatively small required data basis.The combined application of remotely sensed imagery and socio-economic data allows for the mapping, capturing and characterizing the socio-economic structures and dynamics within the mega city of Delhi, as well as it establishes a basis for the monitoring of the mega city of Delhi or certain areas within the city respectively by remote sensing. The opportunity to capture the condition of a mega city and to monitor its development in general enables the persons in charge to identify unbeneficial trends and to intervene accordingly from an urban planning perspective and to countersteer against a non-adequate supply of the inhabitants of different urban districts, primarily of those of informal settlements. This study is understood to be a first step to the development of methods which will help to identify and understand the different forms, actors and processes of urbanization in mega cities. It could support a more proactive and sustainable urban planning and land management – which in turn will increase the importance of urban remote sensing techniques. In this regard, the most obvious and direct beneficiaries are on the one hand the governmental agencies and urban planners and on the other hand, and which is possibly the most important goal, the inhabitants of the affected areas, whose living conditions can be monitored and improved as required. Only if the urban monitoring is quickly, inexpensively and easily available, it will be accepted and applied by the authorities, which in turn enables for the poorest to get the support they need. All in all, the listed benefits are very convincing and corroborate the combined use of remotely sensed and socio-economic data in mega city research

Deep Learning for Remote Sensing Image Processing

Remote sensing images have many applications such as ground object detection, environmental change monitoring, urban growth monitoring and natural disaster damage assessment. As of 2019, there were roughly 700 satellites listing “earth observation” as their primary application. Both spatial and temporal resolutions of satellite images have improved consistently in recent years and provided opportunities in resolving fine details on the Earth\u27s surface. In the past decade, deep learning techniques have revolutionized many applications in the field of computer vision but have not fully been explored in remote sensing image processing. In this dissertation, several state-of-the-art deep learning models have been investigated and customized for satellite image processing in the applications of landcover classification and ground object detection. First, a simple and effective Convolutional Neural Network (CNN) model is developed to detect fresh soil from tunnel digging activities near the U.S. and Mexico border by using pansharpened synthetic hyperspectral images. These tunnels’ exits are usually hidden under warehouses and are used for illegal activities, for example, by drug dealers. Detecting fresh soil nearby is an indirect way to search for these tunnels. While multispectral images have been used widely and regularly in remote sensing since the 1970s, with the fast advances in hyperspectral sensors, hyperspectral imagery is becoming popular. A combination of 80 synthetic hyperspectral channels with the original eight multispectral channels collected by the WorldView-2 satellite are used by CNN to detect fresh soil. Experimental results show that detection performance can be significantly improved by the combination of synthetic hyperspectral images with those original multispectral channels. Second, an end-to-end, pixel-level Fully Convolutional Network (FCN) model is implemented to estimate the number of refugee tents in the Rukban area near the Syrian-Jordan border using high-resolution multispectral satellite images collected by WordView-2. Rukban is a desert area crossing the border between Syria and Jordan, and thousands of Syrian refugees have fled into this area since the Syrian civil war in 2014. In the past few years, the number of refugee shelters for the forcibly displaced Syrian refugees in this area has increased rapidly. Estimating the location and number of refugee tents has become a key factor in maintaining the sustainability of the refugee shelter camps. Manually counting the shelters is labor-intensive and sometimes prohibitive given the large quantities. In addition, these shelters/tents are usually small in size, irregular in shape, and sparsely distributed in a very large area and could be easily missed by the traditional image-analysis techniques, making the image-based approaches also challenging. The FCN model is also boosted by transfer learning with the knowledge in the pre-trained VGG-16 model. Experimental results show that the FCN model is very accurate and has less than 2% of error. Last, we investigate the Generative Adversarial Networks (GAN) to augment training data to improve the training of FCN model for refugee tent detection. Segmentation based methods like FCN require a large amount of finely labeled images for training. In practice, this is labor-intensive, time consuming, and tedious. The data-hungry problem is currently a big hurdle for this application. Experimental results show that the GAN model is a better tool as compared to traditional methods for data augmentation. Overall, our research made a significant contribution to remote sensing image processin

Ash Tree Identification Based on the Integration of Hyperspectral Imagery and High-density Lidar Data

Monitoring and management of ash trees has become particularly important in recent years due to the heightened risk of attack from the invasive pest, the emerald ash borer (EAB). However, distinguishing ash from other deciduous trees can be challenging. Both hyperspectral imagery and Light detection and ranging (LiDAR) data are two valuable data sources that are often used for tree species classification. Hyperspectral imagery measures detailed spectral reflectance related to the biochemical properties of vegetation, while LiDAR data measures the three-dimensional structure of tree crowns related to morphological characteristics. Thus, the accuracy of vegetation classification may be improved by combining both techniques. Therefore, the objective of this research is to integrate hyperspectral imagery and LiDAR data for improving ash tree identification. Specifically, the research aims include: 1) using LiDAR data for individual tree crowns segmentation; 2) using hyperspectral imagery for extraction of relative pure crown spectra; 3) fusing hyperspectral and LiDAR data for ash tree identification. It is expected that the classification accuracy of ash trees will be significantly improved with the integration of hyperspectral and LiDAR techniques. Analysis results suggest that, first, 3D crown structures of individual trees can be reconstructed using a set of generalized geometric models which optimally matched LiDAR-derived raster image, and crown widths can be further estimated using tree height and shape-related parameters as independent variables and ground measurement of crown widths as dependent variables. Second, with constrained linear spectral mixture analysis method, the fractions of all materials within a pixel can be extracted, and relative pure crown-scale spectra can be further calculated using illuminated-leaf fraction as weighting factors for tree species classification. Third, both crown shape index (SI) and coefficient of variation (CV) can be extracted from LiDAR data as invariant variables in tree’s life cycle, and improve ash tree identification by integrating with pixel-weighted crown spectra. Therefore, three major contributions of this research have been made in the field of tree species classification:1) the automatic estimation of individual tree crown width from LiDAR data by combining a generalized geometric model and a regression model, 2) the computation of relative pure crown-scale spectral reflectance using a pixel-weighting algorithm for tree species classification, 3) the fusion of shape-related structural features and pixel-weighted crown-scale spectral features for improving of ash tree identification

Recent Advances in Image Restoration with Applications to Real World Problems

In the past few decades, imaging hardware has improved tremendously in terms of resolution, making widespread usage of images in many diverse applications on Earth and planetary missions. However, practical issues associated with image acquisition are still affecting image quality. Some of these issues such as blurring, measurement noise, mosaicing artifacts, low spatial or spectral resolution, etc. can seriously affect the accuracy of the aforementioned applications. This book intends to provide the reader with a glimpse of the latest developments and recent advances in image restoration, which includes image super-resolution, image fusion to enhance spatial, spectral resolution, and temporal resolutions, and the generation of synthetic images using deep learning techniques. Some practical applications are also included

Very High Resolution (VHR) Satellite Imagery: Processing and Applications

Recently, growing interest in the use of remote sensing imagery has appeared to provide synoptic maps of water quality parameters in coastal and inner water ecosystems;, monitoring of complex land ecosystems for biodiversity conservation; precision agriculture for the management of soils, crops, and pests; urban planning; disaster monitoring, etc. However, for these maps to achieve their full potential, it is important to engage in periodic monitoring and analysis of multi-temporal changes. In this context, very high resolution (VHR) satellite-based optical, infrared, and radar imaging instruments provide reliable information to implement spatially-based conservation actions. Moreover, they enable observations of parameters of our environment at greater broader spatial and finer temporal scales than those allowed through field observation alone. In this sense, recent very high resolution satellite technologies and image processing algorithms present the opportunity to develop quantitative techniques that have the potential to improve upon traditional techniques in terms of cost, mapping fidelity, and objectivity. Typical applications include multi-temporal classification, recognition and tracking of specific patterns, multisensor data fusion, analysis of land/marine ecosystem processes and environment monitoring, etc. This book aims to collect new developments, methodologies, and applications of very high resolution satellite data for remote sensing. The works selected provide to the research community the most recent advances on all aspects of VHR satellite remote sensing

FINE SCALE MAPPING OF LAURENTIAN MIXED FOREST NATURAL HABITAT COMMUNITIES USING MULTISPECTRAL NAIP AND UAV DATASETS COMBINED WITH MACHINE LEARNING METHODS

Natural habitat communities are an important element of any forest ecosystem. Mapping and monitoring Laurentian Mixed Forest natural communities using high spatial resolution imagery is vital for management and conservation purposes. This study developed integrated spatial, spectral and Machine Learning (ML) approaches for mapping complex vegetation communities. The study utilized ultra-high and high spatial resolution National Agriculture Imagery Program (NAIP) and Unmanned Aerial Vehicle (UAV) datasets, and Digital Elevation Model (DEM). Complex natural vegetation community habitats in the Laurentian Mixed Forest of the Upper Midwest. A detailed workflow is presented to effectively process UAV imageries in a dense forest environment where the acquisition of ground control points (GCPs) is extremely difficult. Statistical feature selection methods such as Joint Mutual Information Maximization (JMIM) which is not that widely used in the natural resource field and variable importance (varImp) were used to discriminate spectrally similar habitat communities. A comprehensive approach to training set delineation was implemented including the use of Principal Components Analysis (PCA), Independent Components Analysis (ICA), soils data, and expert image interpretation. The developed approach resulted in robust training sets to delineate and accurately map natural community habitats. Three ML algorithms were implemented Random Forest (RF), Support Vector Machine (SVM), and Averaged Neural Network (avNNet). RF outperformed SVM and avNNet. Overall RF accuracies across the three study sites ranged from 79.45-87.74% for NAIP and 87.31-93.74% for the UAV datasets. Different ancillary datasets including spectral enhancement and image transformation techniques (PCA and ICA), GLCM-Texture, spectral indices, and topography features (elevation, slope, and aspect) were evaluated using the JMIM and varImp feature selection methods, overall accuracy assessment, and kappa calculations. The robustness of the workflow was evaluated with three study sites which are geomorphologically unique and contain different natural habitat communities. This integrated approach is recommended for accurate natural habitat community classification in ecologically complex landscapes