AFQN: approximate Qn estimation in data streams

We present afqn (Approximate Fast Qn), a novel algorithm for approximate computation of the Qn scale estimator in a streaming setting, in the sliding window model. It is well-known that computing the Qn estimator exactly may be too costly for some applications, and the problem is a fortiori exacerbated in the streaming setting, in which the time available to process incoming data stream items is short. In this paper we show how to efficiently and accurately approximate the Qn estimator. As an application, we show the use of afqn for fast detection of outliers in data streams. In particular, the outliers are detected in the sliding window model, with a simple check based on the Qn scale estimator. Extensive experimental results on synthetic and real datasets confirm the validity of our approach by showing up to three times faster updates per second. Our contributions are the following ones: (i) to the best of our knowledge, we present the first approximation algorithm for online computation of the Qn scale estimator in a streaming setting and in the sliding window model; (ii) we show how to take advantage of our UDDSketch algorithm for quantile estimation in order to quickly compute the Qn scale estimator; (iii) as an example of a possible application of the Qn scale estimator, we discuss how to detect outliers in an input data stream

Fast online computation of the Qn estimator with applications to the detection of outliers in data streams

We present FQN (Fast Qn), a novel algorithm for online computation of the Qn scale estimator. The algorithm works in the sliding window model, cleverly computing the Qn scale estimator in the current window. We thoroughly compare our algorithm for online Qn with the state of the art competing algorithm by Nunkesser et al., and show that FQN (i) is faster, requiring only O(s) time in the worst case where s is the length of the window (ii) its computational complexity does not depend on the input distribution and (iii) it requires less space. To the best of our knowledge, our algorithm is the first that allows online computation of the Qn scale estimator in worst case time linear in the size of the window. As an example of a possible application, besides its use as a robust measure of statistical dispersion, we show how to use the Qn estimator for fast detection of outliers in data streams. Extensive experimental results on both synthetic and real datasets confirm the validity of our approach

Distribution System Disturbance Analysis and Outage Management Using Hybrid Data-Driven and Physics-Based Approaches

Securing cyber-power distribution systems (DS) against malicious events is critical with the integration of distributed energy resources (DERs), supporting automation and increasing vulnerabilities. Situational awareness utilizing power data (e.g., data from distribution phasor measurement units (D-PMUs)) and cyber data (e.g., network packets data) is the main focus of this dissertation by means of which, an opportunity for real-time monitoring and decision-making is provided. To further improve the DS’s reliable and resilient operation, in this Ph.D. work, the aim has been put towards the development of an automated tool consisting of multiple modules to precisely investigate any type of data anomalies, followed by root cause finding. For this purpose, a data aggregation scheme is developed to synchronize the resolution and time stamp of multiple metering sources throughout the DS, using an enhancement of the conventional Kalman Filter, named Ensemble Extended Kalman Filter (EEKF). EEKF is implemented as an automated module by exploiting the real-time measurements as well as deriving the system physics. Furthermore, this dissertation develops online cyber-physical event detection and classification as well as proposal of the novel Outage Root Cause Analysis (ORCA) system. Different sections of the work have been tested on IEEE and OPAL-RT test systems as well as real-filed measurements from installed actual hardwares

A new density estimation neural network to detect abnormal condition in streaming data

Along with the development of monitoring technologies, numerous measured data pour into monitoring system and form the high-volume and open-ended data stream. Usually, abnormal condition of monitored system can be characterized by the density variation of measured data stream. However, traditional density estimation methods can not dynamically track density variation of data stream due to the limitation of processing time and computation memory. In this paper, we propose a new density estimation neural network to continuously estimate the density of streaming data in a time-based sliding window. The network has a feedforward structure composed of discretization, input and summation layer. In the discretization layer, value range of data stream is discretized to network nodes with equal intervals. Measured data in the predefined time window are pushed into input layer and updated with the window sliding. In summation layer, the activation results between input neurons and discretization neurons are summed up and multiplied by a weight factor. The network outputs the kernel density estimators of sliding segment in data stream and achieves a one-pass estimation algorithm consuming constant computation memory. By subnet separation and local activation, computation load of the network is significantly reduced to catch up the pace of data stream. The nonlinear statistics, quantile and entropy, which can be consecutively figured out with the density estimators output by the density estimation neural network, are calculated as condition indictors to track the density variation of data stream. The proposed method is evaluated by a simulated data stream consisting of two mixing distribution data sets and a pressure data stream measured from a centrifugal compressor respectively. Results show that the underlying anomalies are successfully detected

Estimation and detection techniques for doubly-selective channels in wireless communications

A fundamental problem in communications is the estimation of the channel. The signal transmitted through a communications channel undergoes distortions so that it is often received in an unrecognizable form at the receiver. The receiver must expend significant signal processing effort in order to be able to decode the transmit signal from this received signal. This signal processing requires knowledge of how the channel distorts the transmit signal, i.e. channel knowledge. To maintain a reliable link, the channel must be estimated and tracked by the receiver. The estimation of the channel at the receiver often proceeds by transmission of a signal called the 'pilot' which is known a priori to the receiver. The receiver forms its estimate of the transmitted signal based on how this known signal is distorted by the channel, i.e. it estimates the channel from the received signal and the pilot. This design of the pilot is a function of the modulation, the type of training and the channel. [Continues.