Outlier removal for improved source estimation in atmospheric inverse problems

Estimation of the quantities of harmful substances emitted into the atmosphere is one of the main challenges in modern environmen- tal sciences. In most of the cases, this estimation requires solving a linear inverse problem. A key difficulty in evaluating the performance of any algorithm to solve this linear inverse problem is that the ground truth is typically unknown. In this paper we show that the noise encountered in this linear inverse problem is non-Gaussian. Next, we develop an algorithm to deal with the strong outliers present in the measurements. Finally, we test our approach on three different experiments: a simple synthetic experiment, a controlled real-world experiment, and real data from the Fukushima nuclear accident

Statistical framework for estimating GNSS bias

We present a statistical framework for estimating global navigation satellite system (GNSS) non-ionospheric differential time delay bias. The biases are estimated by examining differences of measured line integrated electron densities (TEC) that are scaled to equivalent vertical integrated densities. The spatio-temporal variability, instrumentation dependent errors, and errors due to inaccurate ionospheric altitude profile assumptions are modeled as structure functions. These structure functions determine how the TEC differences are weighted in the linear least-squares minimization procedure, which is used to produce the bias estimates. A method for automatic detection and removal of outlier measurements that do not fit into a model of receiver bias is also described. The same statistical framework can be used for a single receiver station, but it also scales to a large global network of receivers. In addition to the Global Positioning System (GPS), the method is also applicable to other dual frequency GNSS systems, such as GLONASS (Globalnaya Navigazionnaya Sputnikovaya Sistema). The use of the framework is demonstrated in practice through several examples. A specific implementation of the methods presented here are used to compute GPS receiver biases for measurements in the MIT Haystack Madrigal distributed database system. Results of the new algorithm are compared with the current MIT Haystack Observatory MAPGPS bias determination algorithm. The new method is found to produce estimates of receiver bias that have reduced day-to-day variability and more consistent coincident vertical TEC values.Comment: 18 pages, 5 figures, submitted to AM

Blind Deconvolution of Anisoplanatic Images Collected by a Partially Coherent Imaging System

Coherent imaging systems offer unique benefits to system operators in terms of resolving power, range gating, selective illumination and utility for applications where passively illuminated targets have limited emissivity or reflectivity. This research proposes a novel blind deconvolution algorithm that is based on a maximum a posteriori Bayesian estimator constructed upon a physically based statistical model for the intensity of the partially coherent light at the imaging detector. The estimator is initially constructed using a shift-invariant system model, and is later extended to the case of a shift-variant optical system by the addition of a transfer function term that quantifies optical blur for wide fields-of-view and atmospheric conditions. The estimators are evaluated using both synthetically generated imagery, as well as experimentally collected image data from an outdoor optical range. The research is extended to consider the effects of weighted frame averaging for the individual short-exposure frames collected by the imaging system. It was found that binary weighting of ensemble frames significantly increases spatial resolution

GREAT3 results I: systematic errors in shear estimation and the impact of real galaxy morphology

We present first results from the third GRavitational lEnsing Accuracy Testing (GREAT3) challenge, the third in a sequence of challenges for testing methods of inferring weak gravitational lensing shear distortions from simulated galaxy images. GREAT3 was divided into experiments to test three specific questions, and included simulated space- and ground-based data with constant or cosmologically-varying shear fields. The simplest (control) experiment included parametric galaxies with a realistic distribution of signal-to-noise, size, and ellipticity, and a complex point spread function (PSF). The other experiments tested the additional impact of realistic galaxy morphology, multiple exposure imaging, and the uncertainty about a spatially-varying PSF; the last two questions will be explored in Paper II. The 24 participating teams competed to estimate lensing shears to within systematic error tolerances for upcoming Stage-IV dark energy surveys, making 1525 submissions overall. GREAT3 saw considerable variety and innovation in the types of methods applied. Several teams now meet or exceed the targets in many of the tests conducted (to within the statistical errors). We conclude that the presence of realistic galaxy morphology in simulations changes shear calibration biases by $\sim 1$ per cent for a wide range of methods. Other effects such as truncation biases due to finite galaxy postage stamps, and the impact of galaxy type as measured by the S\'{e}rsic index, are quantified for the first time. Our results generalize previous studies regarding sensitivities to galaxy size and signal-to-noise, and to PSF properties such as seeing and defocus. Almost all methods' results support the simple model in which additive shear biases depend linearly on PSF ellipticity.Comment: 32 pages + 15 pages of technical appendices; 28 figures; submitted to MNRAS; latest version has minor updates in presentation of 4 figures, no changes in content or conclusion

Simulated vs. Actual Landsat Reflectance Spectra of Bare Soils

Simulated Landsat reflectance spectra of soil samples were compared to actual Landsat radiance values of soils in two fields (1 and 3) near Vance, Mississippi. The simulated reflectance spectra were calculated by combining Landsat spectral sensitivity with laboratory-based spectrophotometer reflectance values. The actual radiance data were obtained by extracting pixel values from Landsat images. Simple linear regression (SLR) yielded significant linear relationships for 1997 field-1 and 2001 field-3 data. Multiple linear regression (MLR) and weighted linear regression (WLR), which indirectly accounted for moisture content and spatial resolution, respectively, yielded improvement in R2 for most of the studied bands. The analyses generally satisfied the normality and constant variance assumptions, and removal of outliers improved the validity of the assumptions and R2. It was concluded that indirect measures of soil moisture content and spatial uncertainty can substantially improve the relationship between remotely sensed bare-soil spectra and laboratory spectra

The Bolocam Galactic Plane Survey: Survey Description and Data Reduction

We present the Bolocam Galactic Plane Survey (BGPS), a 1.1 mm continuum survey at 33" effective resolution of 170 square degrees of the Galactic Plane visible from the northern hemisphere. The survey is contiguous over the range -10.5 < l < 90.5, |b| < 0.5 and encompasses 133 square degrees, including some extended regions |b| < 1.5. In addition to the contiguous region, four targeted regions in the outer Galaxy were observed: IC1396, a region towards the Perseus Arm, W3/4/5, and Gem OB1. The BGPS has detected approximately 8400 clumps over the entire area to a limiting non-uniform 1-sigma noise level in the range 11 to 53 mJy/beam in the inner Galaxy. The BGPS source catalog is presented in a companion paper (Rosolowsky et al. 2010). This paper details the survey observations and data reduction methods for the images. We discuss in detail the determination of astrometric and flux density calibration uncertainties and compare our results to the literature. Data processing algorithms that separate astronomical signals from time-variable atmospheric fluctuations in the data time-stream are presented. These algorithms reproduce the structure of the astronomical sky over a limited range of angular scales and produce artifacts in the vicinity of bright sources. Based on simulations, we find that extended emission on scales larger than about 5.9' is nearly completely attenuated (> 90%) and the linear scale at which the attenuation reaches 50% is 3.8'. Comparison with other millimeter-wave data sets implies a possible systematic offset in flux calibration, for which no cause has been discovered. This presentation serves as a companion and guide to the public data release through NASA's Infrared Processing and Analysis Center (IPAC) Infrared Science Archive (IRSA). New data releases will be provided through IPAC IRSA with any future improvements in the reduction.Comment: Accepted for publication in Astrophysical Journal Supplemen

Blowing in the Wind:Regularizations and Outlier Removal

Every day tons of pollutants are emitted into the atmosphere all around the world. These pollutants are altering the equilibrium of our planet, causing profound changes in its climate, increasing global temperatures, and raising the sea level. The need to curb these emissions is clear and urgent. To do so, it is first necessary to estimate the quantity of pollutants that is being emitted. Hence, the central challenge of this thesis: how can we estimate the spatio-temporal emissions of a pollutant from many later observations of the concentration of that pollutant at different times and locations? Mathematically speaking, given such observations and an atmospheric dispersion model, this is a linear inverse problem. Using real datasets, we show that the main difficulties in solving this problem are ill-conditioning and outliers. Ill-conditioning amplifies the effect of additive noise, and each outlier strongly deflects our estimate from the ground truth. We proceed in two different ways to design new estimation methods that can handle these challenges. In the first approach, we enhance traditional estimators, which are already equipped to deal with ill-conditioning, with a preprocessing step to make them robust against outliers. This preprocessing step blindly localizes outliers in the dataset to remove them completely or to downgrade their influence. We propose two ways of localizing outliers: the first one uses several transport models, while the second one uses random sampling techniques. We show that our preprocessing step significantly improves the performance of traditional estimators, both in synthetic datasets as well as in real-world measurements. The second approach is based on enhancing existing robust estimators, which are already equipped to deal with outliers, with suitable regularizations, so that they are stable when the problem is ill-conditioned. We analyze the properties of our new estimators and compare them with the properties of existing estimators, showing the advantages of introducing the regularization. Our new estimators perform well both in the presence and in the absence of outliers, making them generally applicable. They have good performance with up to 50 % of outliers in the dataset. They are also stable when the problem is ill-conditioned. We demonstrate their performance using real-world measurements. Two different algorithms to compute the new estimators are given: one is based on an iterative re-weighted least squares algorithm and the other on a proximal gradient algorithm. Software implementations of all our proposed estimators, along with sample datasets, are provided as part of our commitment to reproducible results. In addition, we provide LinvPy, an open-source python package that contains tested, documented, and user-friendly implementations of our regularized robust algorithms

Use of Hyperspectral Remote Sensing to Estimate Water Quality

Approximating and forecasting water variables like phosphorus, nitrogen, chlorophyll, dissolved organic matter, and turbidity are of supreme importance due to their strong influence on water resource quality. This chapter is aimed at showing the practicability of merging water quality observations from remote sensing with water quality modeling for efficient and effective monitoring of water quality. We examine the spatial dynamics of water quality with hyperspectral remote sensing and present approaches that can be used to estimate water quality using hyperspectral images. The methods presented here have been embraced because the blue-green and green algae peak wavelengths reflectance are close together and make their distinction more challenging. It has also been established that hyperspectral imagers permit an improved recognition of chlorophyll and hereafter algae, due to acquired narrow spectral bands between 450 nm and 600 nm. We start by describing the practical application of hyperspectral remote sensing data in water quality modeling. The surface inherent optical properties of absorption and backscattering of chlorophyll a, colored dissolved organic matter (CDOM), and turbidity are estimated, and a detailed approach on analyzing ARCHER data for water quality estimation is presented