Fine-Grained Object Recognition and Zero-Shot Learning in Remote Sensing Imagery

Fine-grained object recognition that aims to identify the type of an object among a large number of subcategories is an emerging application with the increasing resolution that exposes new details in image data. Traditional fully supervised algorithms fail to handle this problem where there is low between-class variance and high within-class variance for the classes of interest with small sample sizes. We study an even more extreme scenario named zero-shot learning (ZSL) in which no training example exists for some of the classes. ZSL aims to build a recognition model for new unseen categories by relating them to seen classes that were previously learned. We establish this relation by learning a compatibility function between image features extracted via a convolutional neural network and auxiliary information that describes the semantics of the classes of interest by using training samples from the seen classes. Then, we show how knowledge transfer can be performed for the unseen classes by maximizing this function during inference. We introduce a new data set that contains 40 different types of street trees in 1-ft spatial resolution aerial data, and evaluate the performance of this model with manually annotated attributes, a natural language model, and a scientific taxonomy as auxiliary information. The experiments show that the proposed model achieves 14.3% recognition accuracy for the classes with no training examples, which is significantly better than a random guess accuracy of 6.3% for 16 test classes, and three other ZSL algorithms.Comment: G. Sumbul, R. G. Cinbis, S. Aksoy, "Fine-Grained Object Recognition and Zero-Shot Learning in Remote Sensing Imagery", IEEE Transactions on Geoscience and Remote Sensing (TGRS), in press, 201

Remote Sensing for Monitoring the Mountaintop Mining Landscape: Applications for Land Cover Mapping at the Individual Mine Complex Scale

The aim of this dissertation was to investigate the potential for mapping land cover associated with mountaintop mining in Southern West Virginia using high spatial resolution aerial- and satellite-based multispectral imagery, as well as light detection and ranging (LiDAR) elevation data and terrain derivatives. The following research themes were explored: comparing aerial- and satellite-based imagery, combining data sets of multiple dates and types, incorporating measures of texture, using nonparametric, machine learning classification algorithms, and employing a geographical object-based image analysis (GEOBIA) framework. This research is presented as four interrelated manuscripts.;In a comparison of aerial National Agriculture Imagery Program (NAIP) orthophotography and satellite-based RapidEye data, the aerial imagery was found to provide statistically less accurate classifications of land cover. These lower accuracies are most likely due to inconsistent viewing geometry and radiometric normalization associated with the aerial imagery. Nevertheless, NAIP orthophotography has many characteristics that make it useful for surface mine mapping and monitoring, including its availability for multiple years, a general lack of cloud cover, contiguous coverage of large areas, ease of availability, and low cost. The lower accuracies of the NAIP classifications were somewhat remediated by decreasing the spatial resolution and reducing the number of classes mapped.;Combining LiDAR with multispectral imagery statistically improved the classification of mining and mine reclamation land cover in comparison to only using multispectral data for both pixel-based and GEOBIA classification. This suggests that the reduced spectral resolution of high spatial resolution data can be combated by incorporating data from another sensor.;Generally, the support vector machines (SVM) algorithm provided higher classification accuracies in comparison to random forests (RF) and boosted classification and regression trees (CART) for both pixel-based and GEOBIA classification. It also outperformed k-nearest neighbor, the algorithm commonly used for GEOBIA classification. However, optimizing user-defined parameters for the SVM algorithm tends to be more complex in comparison to the other algorithms. In particular, RF has fewer parameters, and the program seems robust regarding the parameter settings. RF also offers measures to assess model performance, such as estimates of variable importance and overall accuracy.;Textural measures were found to be of marginal value for pixel-based classification. For GEOBIA, neither measures of texture nor object-specific geometry improved the classification accuracy. Notably, the incorporation of additional information from LiDAR provided a greater improvement in classification accuracy then deriving complex textural and geometric measures.;Pre- and post-mining terrain data classified using GEOBIA and machine learning algorithms resulted in significantly more accurate differentiation of mine-reclaimed and non-mining grasslands than was possible with spectral data. The combination of pre- and post-mining terrain data or just pre-mining data generally outperformed post-mining data. Elevation change data were shown to be of particular value, as were terrain shape parameters. GEOBIA was a valuable tool for combining data collected using different sensors and gridded at variable cell sizes, and machine learning algorithms were particularly useful for incorporating the ancillary data derived from the digital elevation models (DEMs), since these most likely would not have met the basic assumptions of multivariate normality required for parametric classifiers.;Collectively, this research suggests that high spatial resolution remotely sensed data are valuable for mapping and monitoring surface mining and mine reclamation, especially when elevation and spectral data are combined. Machine learning algorithms and GEOBIA are useful for integrating such diverse data

Graph-based Data Modeling and Analysis for Data Fusion in Remote Sensing

Hyperspectral imaging provides the capability of increased sensitivity and discrimination over traditional imaging methods by combining standard digital imaging with spectroscopic methods. For each individual pixel in a hyperspectral image (HSI), a continuous spectrum is sampled as the spectral reflectance/radiance signature to facilitate identification of ground cover and surface material. The abundant spectrum knowledge allows all available information from the data to be mined. The superior qualities within hyperspectral imaging allow wide applications such as mineral exploration, agriculture monitoring, and ecological surveillance, etc. The processing of massive high-dimensional HSI datasets is a challenge since many data processing techniques have a computational complexity that grows exponentially with the dimension. Besides, a HSI dataset may contain a limited number of degrees of freedom due to the high correlations between data points and among the spectra. On the other hand, merely taking advantage of the sampled spectrum of individual HSI data point may produce inaccurate results due to the mixed nature of raw HSI data, such as mixed pixels, optical interferences and etc. Fusion strategies are widely adopted in data processing to achieve better performance, especially in the field of classification and clustering. There are mainly three types of fusion strategies, namely low-level data fusion, intermediate-level feature fusion, and high-level decision fusion. Low-level data fusion combines multi-source data that is expected to be complementary or cooperative. Intermediate-level feature fusion aims at selection and combination of features to remove redundant information. Decision level fusion exploits a set of classifiers to provide more accurate results. The fusion strategies have wide applications including HSI data processing. With the fast development of multiple remote sensing modalities, e.g. Very High Resolution (VHR) optical sensors, LiDAR, etc., fusion of multi-source data can in principal produce more detailed information than each single source. On the other hand, besides the abundant spectral information contained in HSI data, features such as texture and shape may be employed to represent data points from a spatial perspective. Furthermore, feature fusion also includes the strategy of removing redundant and noisy features in the dataset. One of the major problems in machine learning and pattern recognition is to develop appropriate representations for complex nonlinear data. In HSI processing, a particular data point is usually described as a vector with coordinates corresponding to the intensities measured in the spectral bands. This vector representation permits the application of linear and nonlinear transformations with linear algebra to find an alternative representation of the data. More generally, HSI is multi-dimensional in nature and the vector representation may lose the contextual correlations. Tensor representation provides a more sophisticated modeling technique and a higher-order generalization to linear subspace analysis. In graph theory, data points can be generalized as nodes with connectivities measured from the proximity of a local neighborhood. The graph-based framework efficiently characterizes the relationships among the data and allows for convenient mathematical manipulation in many applications, such as data clustering, feature extraction, feature selection and data alignment. In this thesis, graph-based approaches applied in the field of multi-source feature and data fusion in remote sensing area are explored. We will mainly investigate the fusion of spatial, spectral and LiDAR information with linear and multilinear algebra under graph-based framework for data clustering and classification problems

Data Preprocessing in Multi-Temporal Remote Sensing Data for Deforestation Analysis

In recent years, the contemporary data mining community has developed a plethora of algorithms and methods used for different tasks in knowledge discovery within large databases. Furthermore, algorithms become more complex and hybrid as algorithms combining several approaches are suggested, the task of implementing such algorithms from scratch becomes increasingly time consuming. Spatial data sets often contain large amounts of data arranged in multiple layers. These data may contain errors and may not be collected at a common set of coordinates. Therefore, various data pre-processing steps are often necessary to prepare data for further usage. It is important to understand the quality and characteristics of the chosen data. Careful selection, preprocessing, and transformation of the data are needed to ensure meaningful analysis and results

Optimum Feature Selection for Recognizing Objects from Satellite Imagery Using Genetic Algorithm

Object recognition is a research area that aims to associate objects to categories or classes. Usually recognition of object specific geospatial features, as building, tree, mountains, roads, and rivers from high-resolution satellite imagery is a time consuming and expensive problem in the maintenance cycle of a Geographic Information System (GIS). Feature selection is the task of selecting a small subset from original features that can achieve maximum classification accuracy and reduce data dimensionality. This subset of features has some very important benefits like, it reduces computational complexity of learning algorithms, saves time, improve accuracy and the selected features can be insightful for the people involved in problem domain. This makes feature selection as an indispensable task in classification task. In our work, we propose wrapper approach based on Genetic Algorithm (GA) as an optimization algorithm to search the space of all possible subsets related to object geospatial features set for the purpose of recognition. GA is wrapped with three different classifier algorithms namely neural network, k-nearest neighbor and decision tree J48 as subset evaluating mechanism. The GA-ANN, GA-KNN and GA-J48 methods are implemented using the WEKA software on dataset that contains 38 extracted features from satellite images using ENVI software. The proposed wrapper approach incorporated the Correlation Ranking Filter (CRF) for spatial features to remove unimportant features. Results suggest that GA based neural classifiers and using CRF for spatial features are robust and effective in finding optimal subsets of features from large data sets

DETERMINING WHERE INDIVIDUAL VEHICLES SHOULD NOT DRIVE IN SEMIARID TERRAIN IN VIRGINIA CITY, NV

This thesis explored elements involved in determining and mapping where a vehicle should not drive off-road in semiarid areas. Obstacles are anything which slows or obstructs progress (Meyer et al., 1977) or limits the space available for maneuvering (Spenko et al., 2006). This study identified the major factors relevant in determining which terrain features should be considered obstacles when off-road driving and thus should be avoided. These are elements relating to the vehicle itself and how it is driven as well as terrain factors of slope, vegetation, water, and soil. Identification of these in the terrain was done using inferential methods of Terrain Pattern Recognition (TPR), analyzing of remotely sensing data, and Digital Elevation Map (DEM) data analysis. Analysis was further refined using other reference information about the area. Other factors such as weather, driving angle, and environmental impact are discussed. This information was applied to a section of Virginia City, Nevada as a case-study. Analysis and mapping was done purposely without field work prior to mapping to determine what could be assessed using only remote means. Not all findings from the literature review could be implemented in this trafficability study. Some methods and trafficability knowledge could not be implemented and were omitted due to data being unavailable, un-acquirable, or being too coarsely mapped to be useful. Examples of these are Lidar mapping of the area, soil profiling of the terrain, and assessment of plant species present in the area for driven-over traction and tire punctures. The Virginia City section was analyzed and mapped utilizing hyperspectral remotely sensed image data, remote-sensor-derived DEM data was used in a Geographical Information Systems (GIS). Stereo-paired air photos of the study site were used in TPR. Other information on flora, historical weather, and a previous soil survey map were used in a Geographical Information System (GIS). Field validation was used to check findings.The case study's trafficability assessment demonstrated methodologies of terrain analysis which successfully classified many materials present and identified major areas where a vehicle should not drive. The methods used were: Manual TPR of the stereo-paired air photo using a stereo photo viewer to conduct drainage-tracing and slope analysis of the DEM was done using automated methods in ArcMap. The SpecTIR hyperspectral data was analyzed using the manual Environment for Visualizing Images (ENVI) software hourglass procedure. Visual analysis of the hyperspectral data and air photos along with known soil and vegetation characteristics were used to refine analyses. Processed data was georectified using SpecTIR Geographic Lookup Table (GLT) input geometry, and exported to and analyzed in ArcMap with the other data previously listed. Features were identified based on their spectral attributes, spatial properties, and through visual analysis. Inaccuracies in mapping were attributable largely to spatial resolution of Digital Elevation Maps (DEMs) which averaged out some non-drivable obstacles and parts of a drivable road, subjective human and computer decisions during the processing of the data, and grouping of spectral end-members during hyperspectral data analysis. Further refinements to the mapping process could have been made if fieldwork was done during the mapping process.Mapping and field validation found: several manmade and natural obstacles were visible from the ground, but these obstacles were too fine, thin, or small to be identified from the remote sensing data. . Examples are fences and some natural terrain surface roughness - where the terrain's surface deviated from being a smooth surface, exhibiting micro- variations in surface elevation and/or textures. Slope analysis using the 10-meter and 30-meter resolution DEMs did not accurately depict some manmade features [eg. some of the buildings, portions of roads, and fences], evident with a well-trafficked paved road showing in DEM analysis as having too steep a slope [beyond 15°] to be drivable. Some features had been spectrally grouped together during analysis, due to similar spectral properties. Spectral grouping is a process where the spectral class's pixel areas are reviewed and classes which have too few occurrences are averaged into similar classes or dropped entirely. This is done to reduce the number of spectrally unique material classes to those that are most relevant to the terrain mapped. These decisions are subjective and in one case two similar spectral material classes were combined. In later evaluation should have remained as two separate material classes. In field sample collection, some of the determined features; free-standing water and liquid tanks, were found to be inaccessible due to being on private land and/or fence secured. These had to be visually verified - photos were also taken. Further refinements to the mapping could have been made if fieldwork was done during the mapping process. Determining and mapping where a vehicle should not drive in semiarid areas is a complex task which involves many variables and reference data types. Processing, analyzing, and fusing these different references entails subjective manual and automated decisions which are subject to errors and/or inaccuracies at multiple levels that can individually or collectively skew results, causing terrain trafficability to be depicted incorrectly. That said, a usable reference map is creatable which can assist decision makers when determining their route(s)

Applied Remote Sensing Program (ARSP)

There are no author-identified significant results in this report

Benchmark of machine learning methods for classification of a Sentinel-2 image

Thanks to mainly ESA and USGS, a large bulk of free images of the Earth is readily available nowadays. One of the main goals of remote sensing is to label images according to a set of semantic categories, i.e. image classification. This is a very challenging issue since land cover of a specific class may present a large spatial and spectral variability and objects may appear at different scales and orientations. In this study, we report the results of benchmarking 9 machine learning algorithms tested for accuracy and speed in training and classification of land-cover classes in a Sentinel-2 dataset. The following machine learning methods (MLM) have been tested: linear discriminant analysis, k-nearest neighbour, random forests, support vector machines, multi layered perceptron, multi layered perceptron ensemble, ctree, boosting, logarithmic regression. The validation is carried out using a control dataset which consists of an independent classification in 11 land-cover classes of an area about 60 km2, obtained by manual visual interpretation of high resolution images (20 cm ground sampling distance) by experts. In this study five out of the eleven classes are used since the others have too few samples (pixels) for testing and validating subsets. The classes used are the following: (i) urban (ii) sowable areas (iii) water (iv) tree plantations (v) grasslands. Validation is carried out using three different approaches: (i) using pixels from the training dataset (train), (ii) using pixels from the training dataset and applying cross-validation with the k-fold method (kfold) and (iii) using all pixels from the control dataset. Five accuracy indices are calculated for the comparison between the values predicted with each model and control values over three sets of data: the training dataset (train), the whole control dataset (full) and with k-fold cross-validation (kfold) with ten folds. Results from validation of predictions of the whole dataset (full) show the random forests method with the highest values; kappa index ranging from 0.55 to 0.42 respectively with the most and least number pixels for training. The two neural networks (multi layered perceptron and its ensemble) and the support vector machines - with default radial basis function kernel - methods follow closely with comparable performanc