Grant, Mike

Groom, Steve

Taylor, Ben

Warren, Mark

Centre for Environmental Data Analysis Digital Repository

Data quality overviewThis report describes issues that should be considered when further processing any of the 2009 ARSF datasets.  The document will be updated over the course of the year, with the latest version available at:http://arsf-dan.nerc.ac.uk/trac/wiki/Reports Issues affecting all sensorsGeo-referencing accuracyARSF currently deliver data at level 1 (calibrated sensor data) rather than level 3 (geo-referenced). This allows users to generate level 2 products (e.g. atmospherically corrected radiances) if they wish, and/or map to any projection or datum that suits.Since ARSF does not currently process data to level 3 except for quality control purposes, instructions for applying precise geometric adjustments to align imagery to other data sets (such as vector overlays) are given to users with delivered datasets. Where a vector overlay or other ground truth information is available, ARSF provide an indication of the average error prior to any such adjustments.Timing errorsCorrection of the timing offsets between navigation and imagery has been necessary to correct errors in some flight lines. Timing errors cause scan lines to be positioned incorrectly and manifest visually as "wobbles" in the imagery correlated to but out of sync with movements of the aircraft.  An example is shown in Figure 1 below.This is under investigation at various levels and has been raised with Specim.We endeavour to correct all timing errors prior to delivery.  As this is a manual process and relies on finding suitable visible features in the imagery, some errors may still remain.  If any are found, please contact us at arsf-processing@pml.ac.uk.Figure 1a: timing error in an Eagle lineFigure 1b: corrected version of above (0.13 seconds difference)Specific sensor issuesSpecim Eagle and HawkRadiometric calibrationPrior to the start of the 2009 flying season, the Eagle and Hawk instruments were calibrated at Specim.  It is also anticipated that the NERC FSF will perform a calibration on the sensors later in the flying season.Quality of 2009 radiometric calibrationThe data quality appears to degrade at the low and high wavelength limits of both Eagle and Hawk. For example, a comparison of pixels near the high Eagle wavelengths and low Hawk wavelengths, over dark targets such as water reveals a mismatch (see Figure 2). Caution is advised when examining spectral responses at the edges of the usable range.Issue tracked at: http://arsf-dan.nerc.ac.uk/trac/ticket/232Figure 2a: Eagle vs. Hawk spectra over water (dark target), showing mismatch at the overlap pointFigure 2b: zoomed up version of Figure 2aOverflowed pixelsThe instruments have a limited dynamic range and must be set to capture data over the appropriate range of signal strength.  For example, if the area of interest is dark, then the instrument will be configured to capture as much low light detail as possible. This configuration is set based on operator experience, the principal investigator's indication of the areas of importance and the prevailing conditions.  Inevitably, some pixels are unexpectedly bright – e.g. sunglint over water or part of a cloud.  These pixels may exceed the maximum capture level and overflow. Overflowed pixels are marked with an invalid value (65535) in delivered level 1 data.  Typically they are not in areas of interest, but should be accounted for.In Hawk, overflows are marked for just the pixel/band in question.  However, Eagle uses a frame transfer CCD, where data are read out in rows.  Incoming light continues to accumulate in unread rows during the transfer and is removed by “smear correction” software, which relies on data from one row to correct the next.  If a pixel overflows, information is lost and all subsequent pixels in that column cannot be fully corrected.  In Eagle, the net effect is that an overflow at 600nm will cause all bluer bands (600nm -> ~400nm) to be undercorrected for that spatial pixel.When Eagle data with overflows are delivered, we mark as bad all bands following an overflow as they will incorporate some unknown additional light.  If your Eagle data contains overflows, the spectrum for  that pixel will look like the masked Eagle (solid line) in Figure 3.  The dashed line in Figure 3 shows what data were captured, although some are undercorrected – if you wish your data delivered unmasked, please contact us.Figure 3: Sample spectra for an overflowed pixel (overflow at ~650nm), showing the masked data (normally delivered) and the undercorrected (400-650nm) capture data.Bad pixelsThe Hawk instrument has a number of bad pixels that give inaccurate values. There are different types of error (e.g. constant pixel values, uncorrected offset, duplicating neighbouring pixels, etc), and ~1% of bad pixels are to be expected on the type of CCD used in the Hawk instrument.  Specim has provided a list of known bad pixels and these are filled with zero values in delivered data.While the majority of bad pixels are corrected or blanked out, some have not been detected and may appear in delivered data.  These appear in level 1 datasets as straight lines along the direction of flight and as undulating lines in level 3 following the motion of the aircraft (e.g. Figure 4).  Typically, they will only affect a single band and are difficult to detect.  A complete solution for detecting and removing these will be pursued in 2008.Issue tracked at: http://arsf-dan.nerc.ac.uk/trac/ticket/111Figure 4: A bad pixel on Hawk band 187, in a scene over water (images inverted to improve contrast on paper)Eagle/Hawk misalignmentCampaigns taking place between October 2008 and April 2009 suffer from a problem where the Eagle and Hawk sensors' data do not line up in the level 3 mapped images. Furthermore this does not appear to be a constant shift, but varies for different flightlines acquired on the same sortie.  The difficulty in identifying this error source has been compounded by the fact that it first occurred in an atmospheric campaign where the instruments were imaging cloud over the ocean.  This causes issues due to there being no suitable elevation model for the cloud surface, the possible dynamic influences and also there being no “hard targets” or vectors to use as references.  The current theory is that this has been caused by either the Eagle or Hawk instrument becoming loose within its casing.  For affected flights, data between the two sensors are being aligned by using per-line boresight corrections to try to minimise the error but may still result in differences of 2 to 8 metres at altitudes of 1000m, especially at swath edges.  The magnitude of the errors will likely increase with higher altitudes.  Where vectors are available the absolute positional error is minimised, otherwise only the relative error between the sensors is minimised.The sensor was been returned to the manufacturer for repair in mid-April and was operational again in mid-May 2009.Hawk dark stripingFlights during 2009 up until the end of July contain visible dark stripes in the Hawk data. This is worse from May onwards. This was due to dust getting into the sensor's internal optics. The values obtained from the sensor are lower in the affected areas of the image. The sensor was repaired at the end of July 2009.Figure 5: Hawk image showing dark stripesHawk frame skippingIn some flights from 2009, the Hawk suffered from a problem causing it to skip every other frame (but not to record this fact). This issue has been worked around in processing, and we believe that the values in the processed data delivered to the user are correct. However, because for processing the instrument was treated as running at half the framerate (rather than dropping every second line), this problem should be borne in mind if you find any issues with the magnitude of the data values.This problem affects all 2009 flights from Sept. 9th onwards. See http://arsf-dan.nerc.ac.uk/trac/ticket/281 for more information.

ARSF Hyperspectral Data Quality Report 02/12/2009

CEDA Repository

http://cedadocs.ceda.ac.uk/791/1/data_quality-20091202.pdf

ARSF Hyperspectral Data Quality Report 02/12/2009

Abstract

Similar works

Full text

Available Versions

Centre for Environmental Data Analysis Digital Repository

CEDA Repository