This short paper describes our ongoing research on Greenhouse - a zero-positive machine learning system for time-series anomaly detection

Gottschlich, Justin E

Lee, Tae J

Metcalf, Eric

Tatbul, Nesime

Zdonik, Stan

English

arXiv

ScholarlyCommons@Penn

University of Pennsylvania ScholarlyCommons Machine Programming PRECISE (Penn Research in Embedded Computing and Integrated Engineering) 2018 Greenhouse: A Zero-Positive Machine Learning System for Time-Series Anomaly Detection Tae J. Lee Justin E. Gottschlich Nesime Tatbul Eric Metcalf Stan Zdonik Follow this and additional works at: https://repository.upenn.edu/cps_machine_programming This paper is posted at ScholarlyCommons. https://repository.upenn.edu/cps_machine_programming/10 For more information, please contact repository@pobox.upenn.edu. Greenhouse: A Zero-Positive Machine Learning System for Time-Series Anomaly Detection Abstract This short paper describes our ongoing research on Greenhouse - a zero-positive machine learning system for time-series anomaly detection. Greenhouse: A Zero-Positive Machine Learning Systemfor Time-Series Anomaly DetectionTae Jun Lee*MicrosoftJustin Gottschlich, Nesime TatbulIntel LabsEric Metcalf, Stan ZdonikBrown UniversityABSTRACTThis short paper describes our ongoing research on Greenhouse -a zero-positive machine learning system for time-series anomalydetection.1 INTRODUCTIONThe emerging “killer app” of the Internet of Things (IoT) envisionsa world made up of huge numbers of sensors and computing devicesthat interact in intelligent ways (e.g., self-driving cars, industrialautomation, mobile phone tracking). The sensors produce massiveamounts of data and the computing devices must figure out how touse it. The preponderance of this data is numerical time-series.Anomaly detection, i.e., the process of finding patterns that do notconform to expected behavior, over time-series is an important capa-bility in IoT with multiple potential applications. Through anomalydetection, we can identify unusual environmental situations that needhuman attention [4], distinguish outliers in sensor data cleaning [2],or pre-filter uninteresting portions of data to preserve computingresources, to name a few.Prior research on time-series anomaly detection largely relied ontraditional data mining and machine learning techniques [7]. Morerecently, new techniques using deep neural networks have gainedattention. Recurrent neural networks (RNNs), and in particular theirLong Short-Term Memory (LSTM) variant, excel at capturing short-term dependencies when making predictions over sequence data[8, 9]. Specifically, LSTM has the ability to support new data as wellas a way to gracefully forget old, and therefore less relevant, data.As such, it is a good fit for predicting time-varying patterns oversequential data, as in time-series anomaly detection [14].In this paper, we describe a novel time-series anomaly detectionsystem called Greenhouse. Our key goal in Greenhouse is to combinestate-of-the-art machine learning and data management techniquesfor efficient and accurate prediction of anomalous patterns overhigh volumes of time-series data. We have designed Greenhouseas a zero-positive machine learning system [3], in that, it does notrequire anomalous samples in its training datasets. This capabilityhas notable practical value, because it can be challenging to collectand label anomalous data when anomalies are rare and varying.In what follows, we first summarize our algorithmic machinelearning framework and preliminary results from its implementationon top of the TensorFlow machine learning library [1]. Then wediscuss our ongoing research agenda in extending this frameworkinto an end-to-end time-series anomaly detection system for IoT.*The work was done while a Brown student.SysML’18, February 2018, Stanford, CA, USA2018.2 ALGORITHMIC FRAMEWORKAt the heart of Greenhouse’s algorithmic framework is an LSTM-based neural network model, which is used to predict values atfuture time points based on values observed at past time points. Thismodel is built based on a training dataset which represents what isconsidered to be “normal”, and is then used for detecting anomalousvalues - those that sufficiently deviate from what the model predictswould be normally observed.Given a time-series that consists of an isochronous (i.e., evenlyspaced in time) sequence of (time, value) pairs as illustrated in Table1, there are two basic tasks that are repeatedly performed in ourframework:Making a prediction: For a given time point t , a window of mostrecently observed values [vt−B , ..,vt−1] of length B is used as “Look-Back” to predict a subsequent window of values [vt , ..,vt+F−1] oflength F as “Predict-Forward”. This is applied to all points in thegiven time-series in a sliding window fashion, resulting in F distinctvalue predictions for each time point. For example, in Table 1a, fort = 4, a Look-Back window with observed values [v1,v2,v3] is usedto predict a Predict-Forward window with values [p4.1,p5.1]. As aresult, we obtain two value predictions per time point (except theinitial point), e.g., p5.1 and p5.2 for t = 5.Computing an error vector: For a given time point t , an error vec-tor “Error Vector(t)” of length F is computed. This vector consists ofdifferences between predicted and observed values that correspondto time point t . For example, in Table 1b, the error vector for t = 5consists of two values: [p5.1 −v5,p5.2 −v5].Table 1: Predictions and Error Vectors for an example time-series (time, value) = [ (1,v1), (2,v2), (3,v3), (4,v4), (5,v5), ... ](a) Making a PredictionLook-Back Predict-Forward(B=3) (F=2)v1, v2, v3 p4.1, p5.1v2, v3, v4 p5.2, p6.1v3, v4, v5 p6.2, p7.1v4, v5, v6 p7.2, p8.1... ...(b) Computing an Error VectorTime ErrorPoint Vector (t)t = 5 [p5.1 − v5, p5.2 − v5]t = 6 [p6.1 − v6, p6.2 − v6]t = 7 [p7.1 − v7, p7.2 − v7]t = 8 [p8.1 − v8, p8.2 − v8]... ...As illustrated in Figure 1, our overall framework mainly consistsof two phases: (i) build an LSTM model (M) and a multi-variateprobability distribution for error vectors (N ), and determine an errorthreshold for anomalies (τ ) during a Training Phase; (ii) use M , N ,and τ to detect anomalies over previously unseen data sets during anInference Phase. We now provide a step-by-step summary of thesetwo phases.Training Phase: Training is performed in four key steps:arXiv:1801.03168v3  [cs.AI]  11 Feb 2018SysML’18, February 2018, Stanford, CA, USA T. J. Lee, J. Gottschlich, N. Tatbul, E. Metcalf, S. ZdonikFigure 1: Training and Inference Phases in Greenhouse(1) Split a non-anomalous dataset into three: TrainingDataset-1,TrainingDataset-2, TrainingDataset-3.(2) Train an LSTM prediction model M with TrainingDataset-1.(3) Apply M over TrainingDataset-2 to make predictions andcompute error vectors. Then fit the resulting error vectors intoa multi-variate normal distribution N .(4) Apply M over TrainingDataset-3 to make predictions andcompute error vectors. Then compute M-distances (Mahalano-bis-distances [13]), and fit the resulting M-distances into atruncated normal distribution T . Finally, evaluate the inversecumulative distribution function of T at a user-specified per-centile to be used as the anomaly detection threshold τ .Inference Phase: Inference involves the following steps:(1) Apply M over NewDataset to make predictions.(2) Compute error vectors.(3) Compute M-distances between these error vectors and thecenter of N .(4) Finally, label the time-series values whose M-distances ex-ceed τ as anomalies.Table 2: Greenhouse vs. LSTM-AD [14]Precision Recall F1 scoreGreenhouse (Twitter_AAPL) 0.49 0.06 0.11LSTM-AD (Twitter_AAPL) 0.22 0.14 0.17Greenhouse (nyc_taxi) 0.25 0.58 0.35LSTM-AD (nyc_taxi) 0.26 0.82 0.40We implemented this algorithm on top of TensorFlow [1], andcompared its anomaly detection accuracy with LSTM-AD [14],based on real-world datasets from the Numenta Anomaly Benchmark[11]. LSTM-AD fundamentally differs from Greenhouse in that, itis not zero-positive, i.e., it requires anomalous samples to be presentin its training datasets. We present a sample experimental result inTable 2. For two different datasets, Greenhouse performed favorablyover LSTM-AD in terms of its Precision and managed to maintain aclose F1 score, despite its lower Recall. This is a remarkable result,especially given that Greenhouse uses significantly smaller trainingdata (about 25% and 55% that of LSTM-AD, respectively) and doesnot rely on any anomalous samples. We believe these are powerfulqualities, making Greenhouse more generally applicable in practice.3 ONGOING RESEARCHGoing forward, we are working on a number of research issues toextend our algorithmic framework into a full-feature time-seriesanomaly detection system for IoT. We conclude the paper with anoverview of these issues.Training dataset management: In Greenhouse, we use multipledatasets during training. Furthermore, for sequential data like time-series, respecting order, regularity (isochronism), and continuity isimportant in correctly capturing patterns of interest. In order to avoidthe risk of over/under-fitting and to preserve continuity, we need topay attention to how we choose and arrange our training datasets.Range-based anomalies and their evaluation: Time-series anom-alies often manifest themselves over a period of time rather than atsingle time points (so-called range-based or collective anomalies[6, 7]). Furthermore, judging accuracy of results in this context ishighly intricate and application-dependent. We are extending Green-house with models and algorithms to handle range-based anomaliesin a principled manner [12].Real-time anomaly detection: Our algorithmic framework initiallyfocused on operating in an “offline mode”, where both training andinference are applied on finite datasets that have been collected inthe past and are being analyzed retrospectively after the fact. Inmany IoT applications, real-time anomaly detection on live data isalso important. Thus, we are extending Greenhouse to operate in an“online mode”, which requires continuous, low-latency inferencing(and possibly training) over streaming time-series. This includesthings like carefully analyzing the impact of algorithm and modelparameters on accuracy and performance, and properly tuning themas datasets change.Utilizing human feedback: When deployed in an online setting,Greenhouse will start making anomaly predictions on new datasetsas well as potentially saving these (self-labeled) datasets for furthertraining. Furthermore, feedback from human may be available tovalidate anomaly predictions or to course-correct. We plan to aug-ment Greenhouse with reinforcement learning techniques to utilizesuch feedback when available [5].Data management support: As in any deep learning system, datais an indispensable resource in Greenhouse. We will deploy Green-house and study its data management needs within the context of atime-series data management system environment (e.g., Metronome[15]). This includes things like providing methods of efficient andconsistent access to arbitrary windows of data needed by Greenhousealgorithms.Exploiting high-performance compute frameworks: While im-plemented on top of a state-of-the-art machine learning frame-work [1], Greenhouse does not yet take full advantage of high-performance compute facilities provided by such frameworks. Weplan to extend Greenhouse in this direction, including exploring itsuse in conjunction with other complementary tools and libraries,such as Intel® Nervana™ Graph [10].Support for distributed IoT deployments: IoT applications typi-cally execute on multi-tier deployments from wireless networks ofedge devices to more powerful cloud servers. We plan to investi-gate the use of Greenhouse in such distributed deployments withheterogeneous computing resources.Acknowledgments. This research has been funded in part by Intel.Greenhouse: A Zero-Positive Machine Learning Systemfor Time-Series Anomaly Detection SysML’18, February 2018, Stanford, CA, USAREFERENCES[1] Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, JeffreyDean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Man-junath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray,Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, YuanYu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-scale Ma-chine Learning. In 12th USENIX Symposium on Operating Systems Design andImplementation (OSDI). 265–283.[2] Charu C. Aggarwal. 2013. Outlier Analysis. Springer.[3] Mohammad Mejbah Ul Alam, Justin Gottschlich, and Abdullah Muzahid. 2017.AutoPerf: A Generalized Zero-Positive Learning System to Detect Software Perfor-mance Anomalies. Technical Report. http://arxiv.org/abs/1709.07536/[4] Peter Bailis, Edward Gan, Samuel Madden, Deepak Narayanan, Kexin Rong, andSahaana Suri. 2017. MacroBase: Prioritizing Attention in Fast Data. In ACMInternational Conference on Management of Data (SIGMOD). 541–556.[5] Bram Bakker. 2001. Reinforcement Learning with Long Short-term Memory. In14th International Conference on Neural Information Processing Systems (NIPS).1475–1482.[6] Loic Bontemps, Van Loi Cao, James McDermott, and Nhien-An Le-Khac. 2017.Collective Anomaly Detection based on Long Short Term Memory RecurrentNeural Network. Technical Report. https://arxiv.org/abs/1703.09752[7] Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly Detection:A Survey. ACM Computing Surveys 41, 3 (2009), 15:1–15:58.[8] Christopher Olah. 2015. Understanding LSTM Networks. http://colah.github.io/posts/2015-08-Understanding-LSTMs/. (2015).[9] Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory.Neural Computation 9, 8 (1997), 1735–1780.[10] Intel. 2017. Intel Nervana Graph. http://ngraph.nervanasys.com/. (2017).[11] Alexander Lavin and Subutai Ahmad. 2015. Evaluating Real-Time AnomalyDetection Algorithms - The Numenta Anomaly Benchmark. In 14th IEEE Inter-national Conference on Machine Learning and Applications (ICMLA). 38–44.[12] Tae Jun Lee, Justin Gottschlich, Nesime Tatbul, Eric Metcalf, and Stan Zdonik.2018. Precision and Recall for Range-Based Anomaly Detection. https://arxiv.org/abs/1801.03175/. In SysML Conference.[13] Prasanta Chandra Mahalanobis. 1936. On the Generalised Distance in Statistics.Proceedings of the National Institute of Sciences of India 2, 1 (1936), 49–55.[14] Pankaj Malhotra, Lovekesh Vig, Gautam Shroff, and Puneet Agarwal. 2015. LongShort Term Memory Networks for Anomaly Detection in Time Series. In Euro-pean Symposium on Artificial Neural Networks, Computational Intelligence andMachine Learning (ESANN). 89–94.[15] John Meehan, Cansu Aslantas, Stan Zdonik, Nesime Tatbul, and Jiang Du. 2017.Data Ingestion for the Connected World. In Conference on Innovative Data Sys-tems Research (CIDR).

Greenhouse: A Zero-Positive Machine Learning System for Time-Series Anomaly Detection

Abstract

Similar works

Full text

Available Versions

ScholarlyCommons@Penn