A universal rank-order transform to extract signals from noisy data

We introduce an ordinate method for noisy data analysis, based solely on rank information and thus insensitive to outliers. The method is nonparametric, objective, and the required data processing is parsimonious. Main ingredients are a rank-order data matrix and its transform to a stable form, which provide linear trends in excellent agreement with least squares regression, despite the loss of magnitude information. A group symmetry orthogonal decomposition of the 2D rank-order transform for iid (white) noise is further ordered by principal component analysis. This two-step procedure provides a noise "etalon" used to characterize arbitrary stationary stochastic processes. The method readily distinguishes both the Ornstein-Uhlenbeck process and chaos generated by the logistic map from white noise. Ranking within randomness differs fundamentally from that in deterministic chaos and signals, thus forming the basis for signal detection. To further illustrate the breadth of applications, we apply this ordinate method to the canonical nonlinear parameter estimation problem of two-species radioactive decay, outperforming special-purpose least square software. It is demonstrated that the method excels when extracting trends in heavy-tailed noise and, unlike the Thiele-Sen estimator, is not limited to linear regression. Lastly, a simple expression is given that yields a close approximation for signal extraction of an underlying generally nonlinear signal.Comment: 26 pages, 18 figure

Organization and oscillations in simulated shallow convective clouds

Physical insights into processes governing temporal organization and evolution of cloud fields are of great importance for climate research. Here using large eddy simulations with a bin microphysics scheme, we show that warm convective cloud fields exhibit oscillations with two distinct periods (~10 and ~90 min, for the case studied here). The shorter period dominates the nonprecipitating phase, and the longer period is related to the precipitating phase. We show that rain processes affect the domain\u27s thermodynamics, hence forcing the field into a low‐frequency recharge‐discharge cycle of developing cloudiness followed by precipitation‐driven depletion. The end result of precipitation is stabilization of the lower atmosphere by warming of the cloudy layer (due to latent heat release) and cooling of the subcloud layer (by rain evaporation, creating cold pools). As the thermodynamic instability weakens, so does the cloudiness, and the rain ceases. During the nonprecipitating phase of the cycle, surface fluxes destabilize the boundary layer until the next precipitation cycle. Under conditions that do not allow development of precipitation (e.g., high aerosol loading), high‐frequency oscillations dominate the cloud field. Clouds penetrating the stable inversion layer trigger gravity waves with a typical period of ~10 min. In return, the gravity waves modulate the clouds in the field by modifying the vertical velocity, temperature, and humidity fields. Subsequently, as the polluted nonprecipitating simulations evolve, the thermodynamic instability increases and the cloudy layer deepens until precipitation forms, shifting the oscillations from high to low frequency. The organization of cold pools and the spatial scale related to these oscillations are explored

Signals as departures from random walks

We study statistics of data ranking, focusing on the recently discovered distribution-invariant discrete eigenvalue spectrum for an independent and identically distributed (IID) process. We employ a variant of a cumulative distribution function in rank and time that maps the sampling variability for an IID process onto a set of random walks. This mapping admits confidence bounds on whether the residual (data with signal removed) arises solely from IID sampling variability. Any deviations judged significant are regarded as signals, whether deterministic, chaotic, or random. Unlike our recent work on extracting unknown signals in arbitrary noise, here we focus on aspects that are easily combined with any other methods of signal extraction. The ubiquitous case of a single trace receives particular attention. The approach is illustrated on dark current and gamma-ray arrival datasets where we examine the residual for consistency with the expected sampling variability of IID noise

Extraction of unknown signals in arbitrary noise

We devise a general method to extract weak signals of unknown form, buried in noise of arbitrary distribution. Central to it is signal-noise decomposition in rank and time: only stationary white noise generates data with a jointly uniform rank-time probability distribution, U(1,N)×U(1,N), for N points in a data sequence. We show that rank, averaged across jointly indexed series of noisy data, tracks the underlying weak signal via a simple relation, for all noise distributions. We derive an exact analytic, distribution-independent form for the discrete covariance matrix of cumulative distributions for independent and identically distributed noise and employ its eigenfunctions to extract unknown signals from single time series

A fast algorithm for computing a matrix transform used to detect trends in noisy data

A recently discovered universal rank-based matrix method to extract trends from noisy time series is described in Ierley and Kostinski (2019) but the formula for the output matrix elements, implemented there as an open-access supplement MATLAB computer code, is O(N4), with N the matrix dimension. This can become prohibitively large for time series with hundreds of sample points or more. Based on recurrence relations, here we derive a much faster O(N2) algorithm and provide code implementations in MATLAB and in open-source JULIA. In some cases one has the output matrix and needs to solve an inverse problem to obtain the input matrix. A fast algorithm and code for this companion problem, also based on the recurrence relations, are given. Finally, in the narrower, but common, domains of (i) trend detection and (ii) parameter estimation of a linear trend, users require, not the individual matrix elements, but simply their accumulated mean value. For this latter case we provide a yet faster O(N) heuristic approximation that relies on a series of rank one matrices. These algorithms are illustrated on a time series of high energy cosmic rays with N\u3e4×104. Program summary: Program Title: Pfromdata, QofP, mbasisandcoeffs, nonzerop, Qavgapprox, PofQ, mexact, CodeTesting CPC Library link to program files: http://dx.doi.org/10.17632/mkcxrky9jc.1 Licensing provisions: MIT Programming language: MATLAB and Julia Nature of problem: An order-rank data matrix and its transform to a stable form are used repeatedly to detect and/or extract trends from noisy data. An efficient yet accurate calculation of the matrix transform is therefore required. Solution method: We introduce and apply an analytic recursion relation, which speeds up the execution of the matrix transform from O(N4) arithmetic operations to O(N2). Since this matrix transform is called often during optimization, our improvement allows for far shorter optimization times, for a given sample size. For example, a transform whose time is extrapolated to an unrealistic 75 days on a Dell personal laptop computer with a 2.2 GHz quad-core AMD processor running 32 bit MATLAB version R2015b on 64 bit Windows 10 (N=5000), now takes a fraction of a second. References [1] Universal Rank-Order Transform to Extract Signals from Noisy Data, Glenn Ierley and Alex Kostinski, Phys. Rev. X 9 031039 (2019)

Characterizing IR dynamic response for foliage backgrounds

© 1990 SPIE. All rights reserved. There are several factors which can contribute to radiometric variability in IR imagery of foliage backgrounds, including variation in solar absorptivity, convective coupling, emissivity; as well as geometry considerations such as variation in leaf shading factors. One factor often neglected is scene variation resulting from differences in transient response for different foliage types under dynamic solar loading. This paper discusses the measurement and characterization of foliage dynamic response, and the effect foliage dynamic response has on scene variability under variable solar loading. Measurements of transient response were obtained under controlled laboratory conditions. Radiometric data was collected for two foliage types using step function solar loading input, using both imaging and nonimaging thermal sensors

Faint yet widespread glories reflect microphysics of marine clouds

Glory is a beautiful optical phenomenon observed in an atmosphere as concentric colored rings reflected by clouds or fog around an antisolar point. Here we report that true color glories, although faint, are discernible in raw unpolarized satellite images by a naked eye on a daily basis, thus constituting a large and untapped reservoir of cloud data for which a simple diffraction-like approximation links cloud droplet diameter and variance to the glory’s structure

Prediction of daily PM2.5 concentrations using aerosol optical depth retrievals from GOES satellite

Although ground-level PM2.5 (particulate matter with aerodynamic diameter <2.5 mu m) monitoring sites provide accurate measurements, their spatial coverage within a given region is limited and thus often insufficient for exposure and epidemiological studies. Satellite data expand spatial coverage, enhancing our ability to estimate location-and/or subject-specific exposures to PM2.5. In this study, the authors apply a mixed-effects model approach to aerosol optical depth (AOD) retrievals from the Geostationary Operational Environmental Satellite (GOES) to predict PM2.5 concentrations within the New England area of the United States. With this approach, it is possible to control for the inherent day-to-day variability in the AOD-PM2.5 relationship, which depends on time-varying parameters such as particle optical properties, vertical and diurnal concentration profiles, and ground surface reflectance. The model-predicted PM2.5 mass concentration are highly correlated with the actual observations, R-2 = 0.92. Therefore, adjustment for the daily variability in AOD-PM2.5 relationship allows obtaining spatially resolved PM2.5 concentration data that can be of great value to future exposure assessment and epidemiological studies.11Nsciescopu