Assessing satellite-derived land product quality for earth system science applications: results from the ceos lpv sub-group

The value of satellite derived land products for science applications and research is dependent upon the known accuracy of the data. CEOS (Committee on Earth Observation Satellites), the space arm of the Group on Earth Observations (GEO), plays a key role in coordinating the land product validation process. The Land Product Validation (LPV) sub-group of the CEOS Working Group on Calibration and Validation (WGCV) aims to address the challenges associated with the validation of global land products. This paper provides an overview of LPV sub-group focus area activities, which cover seven terrestrial Essential Climate Variables (ECVs). The contribution will enhance coordination of the scientific needs of the Earth system communities with global LPV activities

Unmasking Clever Hans Predictors and Assessing What Machines Really Learn

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly "intelligent" behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.Comment: Accepted for publication in Nature Communication

Carving contiguous and fragmented files with fast object validation

http://dx.doi.org/10.1016/j.dlin.2007.06.017"File carving" reconstructs files based on their content, rather than using metadata that points to the content. Carving is widely used for forensics and data recovery, but no file carvers can automatically reassemble fragmented files. We survey files from more than 300 hard drives acquired on the secondary market and show that the ability to reassemble fragmented files is an important requirement for forensic work. Next we analyze the file carving problem, arguing that rapid, accurate carving is best performed by a multi-tier decision problem that seeks to quickly validate or discard candidate byte strings -- "object" -- from the media to be carved. Validators for the JPEG, Microsoft OLE (MSOLE) and ZIP file formats are discussed. Finally, we show how high speed validators can be used to reassemble fragmented files

The Validation of an Infrared Simulation System

A commonly-used term in the simulation domain is ‘validation, verification and accreditation’ (VVA). When analysing simulation predictions for the purpose of system solution development and decision-making, one key question persist: “What confidence can I have in the simulation and its results? ” Knowing the validation status of a simulation system is critical to express confidence in the simulation. A practical validation procedure must be simple and done in the regular course of work. A well-known and acknowledged validation model by Schlesinger depicts the interaction between three entities: Reality, Conceptual Model and Computer Model, and three processes: Analysis & Modelling, Programming and Verification, and Evaluation and Validation. We developed a systematic procedure where each of these six elements is evaluated, investigated and then quantified in terms of a set of criteria (or model properties). Many techniques exist to perform the validation procedure. They include: comparison with other models, face validity, extreme condition testing, historical data validation and predictive validation - to mention a few. The result is a two- dimensional matrix representing the confidence in validation of each of the criteria (model properties) along each of the verification and validation elements. Depending on the nature of the element, the quantification of each cell in this matrix is done numerically or heuristically. Most often literature on validation for simulation systems only provides guidance by means of a theoretical validation framework. This paper briefly describes the procedure used to validate software models in an infrared system simulation, and provides application examples of this process. The discussion includes practical validation techniques, quantification, visualisation, summary reports, and lessons learned during the course of a validation process. The framework presented in this paper is sufficiently general, so that the concepts could be applied to other simulation environments as well