Investigating the Use of Generative Adversarial Networks (GANs) for Pansharpening Thermal Satellite Imagery

Abstract

Remotely sensed satellite imagery can be an important data source for observing and measuring environmental changes in urban areas. For example, thermal satellite imagery can be used to identify urban heat islands (UHIs) and provide motivation for urban cooling strategies. However, the spatial resolution of thermal satellite imagery is insufficient for many urban biodiversity applications (e.g., tree planting and shifting ranges of flora and fauna in response to climate change) because of the inherent heterogeneity and complexity of cities. In order to improve the usability of this imagery, artificial intelligence techniques can increase the spatial resolution of the thermal imagery without losing valuable spectral information. One such technique is pansharpening, which fuses high-spatial resolution panchromatic (single band gray-scale) images and lower-spatial resolution multispectral or thermal infrared images. This project uses a modified generative adversarial network (GAN) to pansharpen lower resolution, remotely sensed thermal satellite imagery using high spatial resolution red-green-blue (RGB) imagery. A focus on developing higher spatial resolution maps of heat across cities could enable such broader-scale investigations into the impacts of UHIs on biodiversity. This thesis develops a novel training dataset with patch-pairs of co-located thermal (70 m) and RGB imagery (3 m). Using this novel dataset, this thesis assesses whether a pansharpening model (PanColorGAN) trained on RGB and panchromatic imagery can be successfully applied to the thermal-optical patch-pairs to solve the thermal pansharpening problem. In addition, this thesis will determine if a pansharpening model (pix2pix) trained on the thermal-optical patch-pairs produces higher quality results than the model with pre-trained weights (PanColorGAN). This approach will be applied to five cities with variable climate-urban environments across the United States: Austin, TX, Boulder, CO, Chicago, IL, Los Angeles, CA, and Washington, D.C. The visual and quantitative results indicated that the PanColorGAN framework with pre-trained weights produced higher quality thermal images than the pix2pix framework trained on the thermal-optical patch-pair dataset. While thermal-optical pansharpening successfully recovered many of the spatial details of the high-resolution RGB imagery, it failed to fully retain the valuable spectral information from the thermal imagery