Super-resolution restoration of spaceborne HD videos using the UCL MAGiGAN system

We developed a novel SRR system, called Multi-Angle Gotcha image restoration with Generative Adversarial Network (MAGiGAN), to produce resolution enhancement of 3-5 times from multi-pass EO images. The MAGiGAN SRR system uses a combination of photogrammetric and machine vision approaches including image segmentation and shadow labelling, feature matching and densification, estimation of an image degradation model, and deep learning approaches, to retrieve image information from distorted features and training networks. We have tested the MAGiGAN SRR using the NVIDIA® Jetson TX-2 GPU card for onboard processing within a smart-satellite capturing high definition satellite videos, which will enable many innovative remote-sensing applications to be implemented in the future. In this paper, we show SRR processing results from a Planet® SkySat HD 70cm spaceborne video using a GPU version of the MAGiGAN system. Image quality and effective resolution enhancement are measured and discussed

Super-Resolution Restoration of MISR Images Using the UCL MAGiGAN System

High spatial resolution Earth observation imagery is considered desirable for many scientific and commercial applications. Given repeat multi-angle imagery, an imaging instrument with a specified spatial resolution, we can use image processing and deep learning techniques to enhance the spatial resolution. In this paper, we introduce the University College London (UCL) MAGiGAN super-resolution restoration (SRR) system based on multi-angle feature restoration and deep SRR networks. We explore the application of MAGiGAN SRR to a set of 9 MISR red band images (275 m) to produce up to a factor of 3.75 times resolution enhancement. We show SRR results over four different test sites containing different types of image content including urban and rural targets, sea ice and a cloud field. Different image metrics are introduced to assess the overall SRR performance, and these are employed to compare the SRR results with the original MISR input images and higher resolution Landsat images, where available. Significant resolution improvement over various types of image content is demonstrated and the potential of SRR for different scientific application is discussed

Single-Frame Super-Resolution of Solar Magnetograms: Investigating Physics-Based Metrics \& Losses

Breakthroughs in our understanding of physical phenomena have traditionally followed improvements in instrumentation. Studies of the magnetic field of the Sun, and its influence on the solar dynamo and space weather events, have benefited from improvements in resolution and measurement frequency of new instruments. However, in order to fully understand the solar cycle, high-quality data across time-scales longer than the typical lifespan of a solar instrument are required. At the moment, discrepancies between measurement surveys prevent the combined use of all available data. In this work, we show that machine learning can help bridge the gap between measurement surveys by learning to \textbf{super-resolve} low-resolution magnetic field images and \textbf{translate} between characteristics of contemporary instruments in orbit. We also introduce the notion of physics-based metrics and losses for super-resolution to preserve underlying physics and constrain the solution space of possible super-resolution outputs

Towards Streamlined Single-Image Super-Resolution: Demonstration with 10 m Sentinel-2 Colour and 10-60 m Multi-Spectral VNIR and SWIR Bands

Higher spatial resolution imaging data are considered desirable in many Earth observation applications. In this work, we propose and demonstrate the TARSGAN (learning Terrestrial image deblurring using Adaptive weighted dense Residual Super-resolution Generative Adversarial Network) system for Super-resolution Restoration (SRR) of 10 m/pixel Sentinel-2 “true” colour images as well as all the other multispectral bands. In parallel, the ELF (automated image Edge detection and measurements of edge spread function, Line spread function, and Full width at half maximum) system is proposed to achieve automated and precise assessments of the effective resolutions of the input and SRR images. Subsequent ELF measurements of the TARSGAN SRR results suggest an averaged effective resolution enhancement factor of about 2.91 times (equivalent to ~3.44 m/pixel for the 10 m/pixel bands) given a nominal SRR upscaling factor of 4 times. Several examples are provided for different types of scenes from urban landscapes to agricultural scenes and sea-ice floes

Subpixel-Scale Topography Retrieval of Mars Using Single-Image DTM Estimation and Super-Resolution Restoration

We propose using coupled deep learning based super-resolution restoration (SRR) and single-image digital terrain model (DTM) estimation (SDE) methods to produce subpixel-scale topography from single-view ESA Trace Gas Orbiter Colour and Stereo Surface Imaging System (CaSSIS) and NASA Mars Reconnaissance Orbiter High Resolution Imaging Science Experiment (HiRISE) images. We present qualitative and quantitative assessments of the resultant 2 m/pixel CaSSIS SRR DTM mosaic over the ESA and Roscosmos Rosalind Franklin ExoMars rover’s (RFEXM22) planned landing site at Oxia Planum. Quantitative evaluation shows SRR improves the effective resolution of the resultant CaSSIS DTM by a factor of 4 or more, while achieving a fairly good height accuracy measured by root mean squared error (1.876 m) and structural similarity (0.607), compared to the ultra-high-resolution HiRISE SRR DTMs at 12.5 cm/pixel. We make available, along with this paper, the resultant CaSSIS SRR image and SRR DTM mosaics, as well as HiRISE full-strip SRR images and SRR DTMs, to support landing site characterisation and future rover engineering for the RFEXM22

Subpixel-Scale Topography Retrieval of Mars Using Single-Image DTM Estimation and Super-Resolution Restoration

We propose using coupled deep learning based super-resolution restoration (SRR) and single-image digital terrain model (DTM) estimation (SDE) methods to produce subpixel-scale topography from single-view ESA Trace Gas Orbiter Colour and Stereo Surface Imaging System (CaSSIS) and NASA Mars Reconnaissance Orbiter High Resolution Imaging Science Experiment (HiRISE) images. We present qualitative and quantitative assessments of the resultant 2 m/pixel CaSSIS SRR DTM mosaic over the ESA and Roscosmos Rosalind Franklin ExoMars rover’s (RFEXM22) planned landing site at Oxia Planum. Quantitative evaluation shows SRR improves the effective resolution of the resultant CaSSIS DTM by a factor of 4 or more, while achieving a fairly good height accuracy measured by root mean squared error (1.876 m) and structural similarity (0.607), compared to the ultra-high-resolution HiRISE SRR DTMs at 12.5 cm/pixel. We make available, along with this paper, the resultant CaSSIS SRR image and SRR DTM mosaics, as well as HiRISE full-strip SRR images and SRR DTMs, to support landing site characterisation and future rover engineering for the RFEXM22

Contemporaneous Monitoring of the Whole Dynamic Earth System from Space, Part I: System Simulation Study Using GEO and Molniya Orbits

Despite the wealth of data produced by previous and current Earth Observation platforms feeding climate models, weather forecasts, disaster monitoring services and countless other applications, the public still lacks the ability to access a live, true colour, global view of our planet, and nudge them towards a realisation of its fragility. The ideas behind commercialization of Earth photography from space has long been dominated by the analytical value of the imagery. What specific knowledge and actionable intelligence can be garnered from these evermore frequent revisits of the planet’s surface? How can I find a market for this analysis? However, what is rarely considered is what is the educational value of the imagery? As students and children become more aware of our several decades of advance in viewing our current planetary state, we should find mechanisms which serve their curiosity, helping to satisfy our children’s simple quest to explore and learn more about what they are seeing. The following study describes the reasons why current GEO and LEO observation platforms are inadequate to provide truly global RGB coverage on an update time-scale of 5-min and proposes an alternative, low-cost, GEO + Molniya 3U CubeSat constellation to perform such an application

On Martian Surface Exploration: Development of Automated 3D Reconstruction and Super-Resolution Restoration Techniques for Mars Orbital Images

Very high spatial resolution imaging and topographic (3D) data play an important role in modern Mars science research and engineering applications. This work describes a set of image processing and machine learning methods to produce the “best possible” high-resolution and high-quality 3D and imaging products from existing Mars orbital imaging datasets. The research work is described in nine chapters of which seven are based on separate published journal papers. These include a) a hybrid photogrammetric processing chain that combines the advantages of different stereo matching algorithms to compute stereo disparity with optimal completeness, fine-scale details, and minimised matching artefacts; b) image and 3D co-registration methods that correct a target image and/or 3D data to a reference image and/or 3D data to achieve robust cross-instrument multi-resolution 3D and image co-alignment; c) a deep learning network and processing chain to estimate pixel-scale surface topography from single-view imagery that outperforms traditional photogrammetric methods in terms of product quality and processing speed; d) a deep learning-based single-image super-resolution restoration (SRR) method to enhance the quality and effective resolution of Mars orbital imagery; e) a subpixel-scale 3D processing system using a combination of photogrammetric 3D reconstruction, SRR, and photoclinometric 3D refinement; and f) an optimised subpixel-scale 3D processing system using coupled deep learning based single-view SRR and deep learning based 3D estimation to derive the best possible (in terms of visual quality, effective resolution, and accuracy) 3D products out of present epoch Mars orbital images. The resultant 3D imaging products from the above listed new developments are qualitatively and quantitatively evaluated either in comparison with products from the official NASA planetary data system (PDS) and/or ESA planetary science archive (PSA) releases, and/or in comparison with products generated with different open-source systems. Examples of the scientific application of these novel 3D imaging products are discussed