Atmospheric Correction Inter-comparison eXercise, ACIX-II Land: An assessment of atmospheric correction processors for Landsat 8 and Sentinel-2 over land

The correction of the atmospheric effects on optical satellite images is essential for quantitative and multitemporal remote sensing applications. In order to study the performance of the state-of-the-art methods in an integrated way, a voluntary and open-access benchmark Atmospheric Correction Inter-comparison eXercise (ACIX) was initiated in 2016 in the frame of Committee on Earth Observation Satellites (CEOS) Working Group on Calibration & Validation (WGCV). The first exercise was extended in a second edition wherein twelve atmospheric correction (AC) processors, a substantially larger testing dataset and additional validation metrics were involved. The sites for the inter-comparison analysis were defined by investigating the full catalogue of the Aerosol Robotic Network (AERONET) sites for coincident measurements with satellites' overpass. Although there were more than one hundred sites for Copernicus Sentinel-2 and Landsat 8 acquisitions, the analysis presented in this paper concerns only the common matchups amongst all processors, reducing the number to 79 and 62 sites respectively. Aerosol Optical Depth (AOD) and Water Vapour (WV) retrievals were consequently validated based on the available AERONET observations. The processors mostly succeeded in retrieving AOD for relatively light to medium aerosol loading (AOD 90% of the results falling within the suggested empirical specifications and with the Root Mean Square Error (RMSE) being mostly <0.25 g/cm2. Regarding Surface Reflectance (SR) validation two main approaches were followed. For the first one, a simulated SR reference dataset was computed over all of the test sites by using the 6SV (Second Simulation of the Satellite Signal in the Solar Spectrum vector code) full radiative transfer modelling (RTM) and AERONET measurements for the required aerosol variables and water vapour content. The performance assessment demonstrated that the retrievals were not biased for most of the bands. The uncertainties ranged from approximately 0.003 to 0.01 (excluding B01) for the best performing processors in both sensors' analyses. For the second one, measurements from the radiometric calibration network RadCalNet over La Crau (France) and Gobabeb (Namibia) were involved in the validation. The performance of the processors was in general consistent across all bands for both sensors and with low standard deviations (<0.04) between on-site and estimated surface reflectance. Overall, our study provides a good insight of AC algorithms' performance to developers and users, pointing out similarities and differences for AOD, WV and SR retrievals. Such validation though still lacks of ground-based measurements of known uncertainty to better assess and characterize the uncertainties in SR retrievals

ACIX-II Land: the second implementation of the Atmospheric Correction Inter-comparison eXercise over Land

The correction of the atmospheric effects on optical satellite images is essential for quantitative remote sensing applications. Open and free data access to Copernicus Sentinel-2 (EC/ESA) and Landsat 8 (NASA/USGS) missions increased significantly the scientific interest on atmospheric correction (AC) and several approaches have been introduced by involving different radiative transfer models, single or multitemporal images, various algorithms to estimate aerosol properties and water vapour content, constant or diverse aerosol models, various sources of ancillary data, etc. These methodologies are usually validated independently by developers and/or users based on a certain number of sites with available reference data and/or are compared with results of other AC processors. In order to investigate all the AC aspects and issues in an integrated way, a benchmark exercise (Atmosperic Correction Inter-compariosn eXercise, ACIX) was initiated in 2016 in the frame of CEOS Working Group on Calibration & Validation (WGCV) with the aim to compare the state-of-the-art AC processors. ACIX is a voluntary and open-access initiative to which every AC processor’s developer is invited to participate. ACIX-I was an initial attempt to study the variability of AC performances over diverse atmospheric and land cover conditions using Landsat 8 and Sentinel-2A input data. It was highly appreciated by the participants and considered as a useful tool to discover not only the assets and flaws of the approaches, but also ways to improve them. Thus, a second implementation of the experiment was requested to inter-compare the enhanced versions of the participating processors, but also to be expanded by including additional AC processors. In this second implementation, ACIX was split in two categories: Land and Aqua, with focus on the processors performing over land and water correspondingly. In this presentation, attention is given only to the Land part of the exercise. The sites for the inter-comparison analysis over land were defined by investigating the full catalogue of AERONET sites for available measurements within 30min (±15min) from the satellites’ overpass. Eventually a total of 123 and 110 AERONET sites, which were distributed globally and representing various land cover types, made the site list for Sentinel-2A, -2B and Landsat 8 acquisitions correspondingly. Based on these available AERONET measurements, Aerosol Optical Depth (AOD) and Water Vapour (WV) retrievals were validated with the help of various statistical metrics. Regarding Surface Reflectance (SR) validation, as there is not yet any global network of systematic ground-based measurements, alternative approaches had to be adopted. Therefore, simulated SR reference dataset was computed over all the test sites by using the 6SV full radiative transfer code, with the required aerosol and water vapour information to have been acquired from AERONET. Moreover, measurements from the calibration dedicated network RadCalNet over La Crau (France) and Gobabeb (Namimbia) were involved in the SR validation. The observations in this case were processed to the same sun and sensor geometry, as well as spectrally integrated to the corresponding sensor spectral bands of Sentinel-2 and Landsat 8. The analysis results varied depending on the AC product compared, the reference dataset and the metrics. In this presentation an overview of the analysis and results will be given and discussed

CMIX: Cloud Mask Intercomparison eXercise

Cloud cover is a major limiting factor in exploiting time-series data acquired by optical spaceborne remote sensing sensors. Multiple methods have been developed to address the problem of cloud detection in satellite imagery and a number of cloud masks have been developed for optical sensors but very few studies have carried out quantitative intercomparison of state-of-the-art methods in this domain. Here, we summarize results of the first Cloud Masking Intercomparison eXercise (CMIX) conducted within the Committee Earth Observation Satellites (CEOS) Working Group on Calibration & Validation (WGCV). CEOS is the forum for space agency coordination and cooperation on Earth observations, with activities organized under working groups. CMIX, as one such activity, is an international collaborative effort aimed at intercomparing cloud detection algorithms for moderate-spatial resolution (10-30 m) spaceborne optical sensors. The focus of CMIX is on open and free imagery acquired by the Landsat 8 (NASA/USGS) and Sentinel-2 (ESA) missions. Ten algorithms developed by nine teams from fourteen different organizations representing universities, research centers and industry, as well as space agencies (CNES, ESA, DLR, and NASA), were evaluated within the CMIX. Those algorithms varied in their approach and concepts utilized which were based on various spectral properties, spatial and temporal features, as well as machine learning methods. Algorithm outputs were evaluated against existing reference cloud mask datasets. Those datasets varied in sampling methods, geographical distribution, sample unit (points, polygons, or full image labels), and generation approach (experts annotations, machine learning, or sky images). Overall, the performance of algorithms varied depending on the reference dataset, which can be attributed to differences in cloud definitions used when producing the reference datasets. Average overall accuracy (across algorithms) varied 80.0±5.3% to 89.4±2.4% for Sentinel-2, and 79.8±7.1% to 97.6±0.8% for Landsat 8, depending on the reference dataset. An overall accuracy of 90% yields half the errors than an overall accuracy of 80%. The study identified algorithms that provided a balance between commission and omission errors, as well as algorithms, which are cloud conservative (high user’s accuracy) and non-cloud (clear) conservative (high producer’s accuracy). With repetitive observations like those of Sentinel-2, it seems reasonable to favor non-cloud conservative approaches, with maybe the exception of very cloudy regions where every cloud free observation is critical. When thin/semi-transparent clouds were not considered in the reference datasets algorithms’ performance generally improved: overall accuracy values increased by +1.5% to 7.4%. It should be noted though that these clouds are commonly occurring and are often present in optical imagery. Within CMIX, we also developed recommendations for further activities, which include provision of a quantitative definition for clouds (targeting moderate spatial resolution imagery by Landsat 8 and Sentinel-2), generation of new reference datasets, and expansion of the analysis framework (for example, multi-temporal analysis and application-driven validation). Such intercomparison studies will hopefully help the community to improve the algorithms and move towards standardization of cloud masking. Given the importance of cloud masking in optical satellite imagery we encourage CEOS to continue the CMIX activities