Detecting changes in high frequency data streams, with applications

In recent years, problems relating to the analysis of data streams have become widespread. A data stream is a collection of time ordered observations x1, x2, ... generated from the random variables X1, X2, .... It is assumed that the observations are univariate and independent, and that they arrive in discrete time. Unlike traditional sequential analysis problems considered by statisticians, the size of a data stream is not assumed to be fixed, and new observations may be received over time. The rate at which these observations are received can be very high, perhaps several thousand every second. Therefore computational efficiency is very important, and methods used for analysis must be able to cope with potentially huge data sets. This paper is concerned with the task of detecting whether a data stream contains a change point, and extends traditional methods for sequential change detection to the streaming context. We focus on two different settings of the change point problem. The first is nonparametric change detection where, in contrast to most of the existing literature, we assume that nothing is known about either the pre- or post-change stream distribution. The task is then to detect a change from an unknown base distribution F0 to an unknown distribution F1. Further, we impose the constraint that change detection methods must have a bounded rate of false positives, which is important when it comes to assessing the significance of discovered change points. It is this constraint which makes the nonparametric problem difficult. We present several novel methods for this problem, and compare their performance via extensive experimental analysis. The second strand of our research is Bernoulli change detection, with application to streaming classification. In this setting, we assume a parametric form for the stream distribution, but one where both the pre- and post-change parameters are unknown. The task is again to detect changes, while having a control on the rate of false positives. After developing two different methods for tackling the pure Bernoulli change detection task, we then show how our approach can be deployed in streaming classification applications. Here, the goal is to classify objects into one of several categories. In the streaming case, the optimal classification rule can change over time, and classification techniques which are not able to adapt to these changes will suffer performance degradation. We show that by focusing only on the frequency of errors produced by the classifier, we can treat this as a Bernoulli change detection problem, and again perform extensive experimental analysis to show the value of our methods

Change-point Problem and Regression: An Annotated Bibliography

The problems of identifying changes at unknown times and of estimating the location of changes in stochastic processes are referred to as the change-point problem or, in the Eastern literature, as disorder . The change-point problem, first introduced in the quality control context, has since developed into a fundamental problem in the areas of statistical control theory, stationarity of a stochastic process, estimation of the current position of a time series, testing and estimation of change in the patterns of a regression model, and most recently in the comparison and matching of DNA sequences in microarray data analysis. Numerous methodological approaches have been implemented in examining change-point models. Maximum-likelihood estimation, Bayesian estimation, isotonic regression, piecewise regression, quasi-likelihood and non-parametric regression are among the methods which have been applied to resolving challenges in change-point problems. Grid-searching approaches have also been used to examine the change-point problem. Statistical analysis of change-point problems depends on the method of data collection. If the data collection is ongoing until some random time, then the appropriate statistical procedure is called sequential. If, however, a large finite set of data is collected with the purpose of determining if at least one change-point occurred, then this may be referred to as non-sequential. Not surprisingly, both the former and the latter have a rich literature with much of the earlier work focusing on sequential methods inspired by applications in quality control for industrial processes. In the regression literature, the change-point model is also referred to as two- or multiple-phase regression, switching regression, segmented regression, two-stage least squares (Shaban, 1980), or broken-line regression. The area of the change-point problem has been the subject of intensive research in the past half-century. The subject has evolved considerably and found applications in many different areas. It seems rather impossible to summarize all of the research carried out over the past 50 years on the change-point problem. We have therefore confined ourselves to those articles on change-point problems which pertain to regression. The important branch of sequential procedures in change-point problems has been left out entirely. We refer the readers to the seminal review papers by Lai (1995, 2001). The so called structural change models, which occupy a considerable portion of the research in the area of change-point, particularly among econometricians, have not been fully considered. We refer the reader to Perron (2005) for an updated review in this area. Articles on change-point in time series are considered only if the methodologies presented in the paper pertain to regression analysis

Inferential Modeling and Independent Component Analysis for Redundant Sensor Validation

The calibration of redundant safety critical sensors in nuclear power plants is a manual task that consumes valuable time and resources. Automated, data-driven techniques, to monitor the calibration of redundant sensors have been developed over the last two decades, but have not been fully implemented. Parity space methods such as the Instrumentation and Calibration Monitoring Program (ICMP) method developed by Electric Power Research Institute and other empirical based inferential modeling techniques have been developed but have not become viable options. Existing solutions to the redundant sensor validation problem have several major flaws that restrict their applications. Parity space method, such as ICMP, are not robust for low redundancy conditions and their operation becomes invalid when there are only two redundant sensors. Empirical based inferential modeling is only valid when intrinsic correlations between predictor variables and response variables remain static during the model training and testing phase. They also commonly produce high variance results and are not the optimal solution to the problem. This dissertation develops and implements independent component analysis (ICA) for redundant sensor validation. Performance of the ICA algorithm produces sufficiently low residual variance parameter estimates when compared to simple averaging, ICMP, and principal component regression (PCR) techniques. For stationary signals, it can detect and isolate sensor drifts for as few as two redundant sensors. It is fast and can be embedded into a real-time system. This is demonstrated on a water level control system. Additionally, ICA has been merged with inferential modeling technique such as PCR to reduce the prediction error and spillover effects from data anomalies. ICA is easy to use with, only the window size needing specification. The effectiveness and robustness of the ICA technique is shown through the use of actual nuclear power plant data. A bootstrap technique is used to estimate the prediction uncertainties and validate its usefulness. Bootstrap uncertainty estimates incorporate uncertainties from both data and the model. Thus, the uncertainty estimation is robust and varies from data set to data set. The ICA based system is proven to be accurate and robust; however, classical ICA algorithms commonly fail when distributions are multi-modal. This most likely occurs during highly non-stationary transients. This research also developed a unity check technique which indicates such failures and applies other, more robust techniques during transients. For linear trending signals, a rotation transform is found useful while standard averaging techniques are used during general transients

Changepoint detection for data intensive settings

Detecting a point in a data sequence where the behaviour alters abruptly, otherwise known as a changepoint, has been an active area of interest for decades. More recently, with the advent of the data intensive era, the need for automated and computationally efficient changepoint methods has grown. We here introduce several new techniques for doing this which address many of the issues inherent in detecting changes in a streaming setting. In short, these new methods, which may be viewed as non-trivial extensions of existing classical procedures, are intended to be as useful in as wide a set of situations as possible, while retaining important theoretical guarantees and ease of implementation. The first novel contribution concerns two methods for parallelising existing dynamic programming based approaches to changepoint detection in the single variate setting. We demonstrate that these methods can result in near quadratic computational gains, while retaining important theoretical guarantees. Our next area of focus is the multivariate setting. We introduce two new methods for data intensive scenarios with a fixed, but possibly large, number of dimensions. The first of these is an offline method which detects one change at a time using a new test statistic. We demonstrate that this test statistic has competitive power in a variety of possible settings for a given changepoint, while allowing the method to be versatile across a range of possible modelling assumptions. The other method we introduce for multivariate data is also suitable in the streaming setting. In addition, it is able to relax many standard modelling assumptions. We discuss the empirical properties of the procedure, especially insofar as they relate to a desired false alarm error rate

Wave function method to forecast foreign currencies exchange rates at ultra high frequency electronic trading in foreign currencies exchange markets

The accurate forecast of the foreign currencies exchange rates at the ultra high frequency electronic trading in the foreign currencies exchange markets is a main topic of our research: 1) the present state of the foreign currencies exchange markets in Asia, Europe and North America; 2) the research review on the classic forecast techniques of the foreign currencies exchange rates dynamics in the foreign currencies exchange markets in the classic finances theory; 3) the description on the quantum forecast techniques of the foreign currencies exchange rates dynamics in the foreign currencies exchange markets with the application of both the wave function and the time dependent / time independent wave equation in the quantum finances theory; 4) the derivation of the time dependent / time independent wave equation in the quantum finances theory; 5) the creation of the quantum system state prediction algorithm, based on both the wave function and the time dependent / time independent wave equation in the quantum finances theory; 6) the discussion on the developed software program with the embedded quantum system state prediction algorithm, using both the wave function and the time dependent / time independent wave equation in the quantum finances theory; 7) the final words on the perspectives of the quantum forecast techniques of the foreign currencies exchange rates dynamics in the foreign currencies exchange markets, applying both the wave function and the time dependent / time independent wave equation in the quantum finances theory

Wave function method to forecast foreign currencies exchange rates at ultra high frequency electronic trading in foreign currencies exchange markets

The accurate forecast of the foreign currencies exchange rates at the ultra high frequency electronic trading in the foreign currencies exchange markets is a main topic of our research: 1) the present state of the foreign currencies exchange markets in Asia, Europe and North America; 2) the research review on the classic forecast techniques of the foreign currencies exchange rates dynamics in the foreign currencies exchange markets in the classic finances theory; 3) the description on the quantum forecast techniques of the foreign currencies exchange rates dynamics in the foreign currencies exchange markets with the application of both the wave function and the time dependent / time independent wave equation in the quantum finances theory; 4) the derivation of the time dependent / time independent wave equation in the quantum finances theory; 5) the creation of the quantum system state prediction algorithm, based on both the wave function and the time dependent / time independent wave equation in the quantum finances theory; 6) the discussion on the developed software program with the embedded quantum system state prediction algorithm, using both the wave function and the time dependent / time independent wave equation in the quantum finances theory; 7) the final words on the perspectives of the quantum forecast techniques of the foreign currencies exchange rates dynamics in the foreign currencies exchange markets, applying both the wave function and the time dependent / time independent wave equation in the quantum finances theory

Spectral Time Series Analysis of Ocean Wave Buoy Measurements

Waves in the ocean can be as dangerous as they are impressive. In order to study the behaviour of such waves, buoys are commonly deployed to collect recordings of the ocean surface over time. This results in large quantities of high-frequency multivariate time series data. The statistical analysis of such data is of great importance in a variety of engineering and scientific contexts, from the design of coastal flood defences to offshore structures. We develop methodology for analysing such buoy data, investigating two key questions. Firstly, how should we perform parameter inference for models of the frequency domain behaviour of the surface, given recorded buoy data? Secondly, how can we detect statistically significant non-linearities present in these time series? For parameter inference, we find that pseudo-likelihood approaches greatly outperform state-of-the-art methodologies. As a result, not only can we obtain more reliable parameter estimates, but we can also perform inference for more complicated models, allowing for a more intricate description of the waves. Due to the improved performance of such estimates, we are able to see the evolution of these parameters throughout storm events, using recorded buoy data from both California and the North Sea. For detecting non-linearities, we develop a robust testing procedure by evaluating the bispectrum of the observed time series against the bispectrum of bootstrap simulated Gaussian processes with similar characteristics. We explore the performance of this technique in simulation studies, and apply the approach to buoy data from California

Forecast in Capital Markets

In the Schumpeterian technical and social innovations disruption age, we firmly believe that a big number of unlimited opportunities toward a new era of the ultra high frequency electronic trading in the foreign currencies exchange markets is being created due to an increasing application of the computations processing in the range of ultra high frequencies in the modern finances. In this book, we would like to focus on the capital markets in the finances, discussing a number of scientific methods for an accurate forecast of the foreign currencies exchange rates oscillations dynamics during the ultra high frequency electronic trading in the foreign currencies exchange markets in the short and long time periods. More specifically, we would like to precisely characterize the foreign currencies exchange rates at the ultra high frequencies electronic trading in the foreign currencies exchange markets in the short and long time periods, using the mathematical, financial, electronic and quantum analysis methods. In addition, we would like to propose the quantum winning virtuous strategies creation algorithm with the inductive, deductive, abductive and quantum logics to earn an increasing return premium during the ultra high frequencies electronic trading in the foreign currencies exchange markets in the short and long time periods

Forecast in Capital Markets

In the Schumpeterian technical and social innovations disruption age, we firmly believe that a big number of unlimited opportunities toward a new era of the ultra high frequency electronic trading in the foreign currencies exchange markets is being created due to an increasing application of the computations processing in the range of ultra high frequencies in the modern finances. In this book, we would like to focus on the capital markets in the finances, discussing a number of scientific methods for an accurate forecast of the foreign currencies exchange rates oscillations dynamics during the ultra high frequency electronic trading in the foreign currencies exchange markets in the short and long time periods. More specifically, we would like to precisely characterize the foreign currencies exchange rates at the ultra high frequencies electronic trading in the foreign currencies exchange markets in the short and long time periods, using the mathematical, financial, electronic and quantum analysis methods. In addition, we would like to propose the quantum winning virtuous strategies creation algorithm with the inductive, deductive, abductive and quantum logics to earn an increasing return premium during the ultra high frequencies electronic trading in the foreign currencies exchange markets in the short and long time periods

Investment in capital markets

Investment in Capital Markets creates a strategic vision on the financial capital investment in the capital markets with the aim to get an increased return premium in the short and long time periods. The book is written with a main goal to explain the pros and cons of the financial capital investment in the capital markets, discussing the sophisticated investment concepts and techniques in the simple understandable readable general format language. We would like to highlight the three interesting facts about the book: 1. It is centered on the consideration of the modern investment products, the investment vehicles and the investment mediums for the financial capital investment in the capital markets; 2. It is focused on the financial risk calculation and mitigation techniques for the financial capital investment in the financial capital markets. 3. It is aimed to describe the quantum winning virtuous investment strategies creation and execution techniques during the financial capital investment in the capital markets. The investors, financiers, economists, financial analysts, financial traders, financial advisers, lawmakers, policy analysts, subject experts, professors, and students will certainly enjoy a breathtaking splendid learning journey with the explained new ideas, established concepts and outlined future prospects toward the financial capital investment in the capital markets with the aim to get an increased return premium in the short and long time periods