EWMA STATISTICS AND FUZZY LOGIC IN FUNCTION OF NETWORK ANOMALY DETECTION

Anomaly detection is used to monitor and capture traffic anomalies in network systems. Many anomalies manifest in changes in the intensity of network events. Because of the ability of EWMA control chart to monitor the rate of occurrences of events based on their intensity, this statistic is appropriate for implementation in control limits based algorithms. The performance of standard EWMA algorithm can be made more effective combining the logic of adaptive threshold algorithm and adequate application of fuzzy theory. This paper analyzes the theoretical possibility of applying EWMA statistics and fuzzy logic to detect network anomalies. Different aspects of fuzzy rules are discussed as well as different membership functions, trying to find the most adequate choice. It is shown that the introduction of fuzzy logic in standard EWMA algorithm for anomaly detection opens the possibility of previous warning from a network attack. Besides, fuzzy logic enables precise determination of degree of the risk

Applied Aspects of Modern Metrology

In the modern era of scientific and technological development, the role of measurements and metrology in scientific research is becoming more and more important due to the increase in the testing of various products. Moreover, requirements for the accuracy and reliability of measurement results are increasing significantly and their ranges are expanding. Improving measurement accuracy allows us to identify the shortcomings of certain technological processes and either eliminate them or reduce their influence. This leads to better-quality products and contributes to saving energy and other resources, as well as raw materials and materials. This book discusses relevant aspects of practical metrological activity to establish traceability of measurements while increasing their accuracy and reliability. It also presents procedures for the calibration and testing of measuring instruments

Technical analysis in the foreign exchange market

This article introduces the subject of technical analysis in the foreign exchange market, with emphasis on its importance for questions of market efficiency. Technicians view their craft, the study of price patterns, as exploiting traders’ psychological regularities. The literature on technical analysis has established that simple technical trading rules on dollar exchange rates provided 15 years of positive, risk-adjusted returns during the 1970s and 80s before those returns were extinguished. More recently, more complex and less studied rules have produced more modest returns for a similar length of time. Conventional explanations that rely on risk adjustment and/or central bank intervention are not plausible justifications for the observed excess returns from following simple technical trading rules. Psychological biases, however, could contribute to the profitability of these rules. We view the observed pattern of excess returns to technical trading rules as being consistent with an adaptive markets view of the world.Foreign exchange rates

Latent Structures based-Multivariate Statistical Process Control: a paradigm shift

The basic fundamentals of statistical process control (SPC) were proposed by Walter Shewhart for data-starved production environments typical in the 1920s and 1930s. In the 21st century, the traditional scarcity of data has given way to a data-rich environment typical of highly automated and computerized modern processes. These data often exhibit high correlation, rank deficiency, low signal-to-noise ratio, multistage and multiway structures, and missing values. Conventional univariate and multivariate SPC techniques are not suitable in these environments. This article discusses the paradigm shift to which those working in the quality improvement field should pay keen attention. We advocate the use of latent structure based multivariate statistical process control methods as efficient quality improvement tools in these massive data contexts. This is a strategic issue for industrial success in the tremendously competitive global market.This research work was partially supported by the Spanish Ministry of Economy and Competitiveness under the project DPI2011-28112-C04-02.Ferrer, A. (2014). Latent Structures based-Multivariate Statistical Process Control: a paradigm shift. Quality Engineering. 26(1):72-91. https://doi.org/10.1080/08982112.2013.846093S7291261Aparisi, F., Jabaioyes, J., & Carrion, A. (1999). Statistical properties of the lsi multivariate control chart. Communications in Statistics - Theory and Methods, 28(11), 2671-2686. doi:10.1080/03610929908832445Arteaga, F., & Ferrer, A. (2002). Dealing with missing data in MSPC: several methods, different interpretations, some examples. Journal of Chemometrics, 16(8-10), 408-418. doi:10.1002/cem.750Bersimis, S., Psarakis, S., & Panaretos, J. (2007). Multivariate statistical process control charts: an overview. Quality and Reliability Engineering International, 23(5), 517-543. doi:10.1002/qre.829Bharati, M. H., & MacGregor, J. F. (1998). Multivariate Image Analysis for Real-Time Process Monitoring and Control. Industrial & Engineering Chemistry Research, 37(12), 4715-4724. doi:10.1021/ie980334lBharati, M. H., MacGregor, J. F., & Tropper, W. (2003). Softwood Lumber Grading through On-line Multivariate Image Analysis Techniques. Industrial & Engineering Chemistry Research, 42(21), 5345-5353. doi:10.1021/ie0210560Bisgaard, S. (2012). The Future of Quality Technology: From a Manufacturing to a Knowledge Economy & From Defects to Innovations. Quality Engineering, 24(1), 30-36. doi:10.1080/08982112.2011.627010Box, G. E. P. (1954). Some Theorems on Quadratic Forms Applied in the Study of Analysis of Variance Problems, I. Effect of Inequality of Variance in the One-Way Classification. The Annals of Mathematical Statistics, 25(2), 290-302. doi:10.1214/aoms/1177728786Camacho, J., & Ferrer, A. (2012). Cross-validation in PCA models with the element-wise k-fold (ekf) algorithm: theoretical aspects. Journal of Chemometrics, 26(7), 361-373. doi:10.1002/cem.2440Duchesne, C., Liu, J. J., & MacGregor, J. F. (2012). Multivariate image analysis in the process industries: A review. Chemometrics and Intelligent Laboratory Systems, 117, 116-128. doi:10.1016/j.chemolab.2012.04.003Efron, B., & Gong, G. (1983). A Leisurely Look at the Bootstrap, the Jackknife, and Cross-Validation. The American Statistician, 37(1), 36-48. doi:10.1080/00031305.1983.10483087Ferrer, A. (2007). Multivariate Statistical Process Control Based on Principal Component Analysis (MSPC-PCA): Some Reflections and a Case Study in an Autobody Assembly Process. Quality Engineering, 19(4), 311-325. doi:10.1080/08982110701621304Fuchs, C. (1998). Multivariate Quality Control. doi:10.1201/9781482273731Geladi, P., & Kowalski, B. R. (1986). Partial least-squares regression: a tutorial. Analytica Chimica Acta, 185, 1-17. doi:10.1016/0003-2670(86)80028-9Helland, I. S. (1988). On the structure of partial least squares regression. Communications in Statistics - Simulation and Computation, 17(2), 581-607. doi:10.1080/03610918808812681Höskuldsson, A. (1988). PLS regression methods. Journal of Chemometrics, 2(3), 211-228. doi:10.1002/cem.1180020306Hunter, J. S. (1986). The Exponentially Weighted Moving Average. Journal of Quality Technology, 18(4), 203-210. doi:10.1080/00224065.1986.11979014Edward Jackson, J. (1985). Multivariate quality control. Communications in Statistics - Theory and Methods, 14(11), 2657-2688. doi:10.1080/03610928508829069Jackson, J. E., & Mudholkar, G. S. (1979). Control Procedures for Residuals Associated With Principal Component Analysis. Technometrics, 21(3), 341-349. doi:10.1080/00401706.1979.10489779Process analysis and abnormal situation detection: from theory to practice. (2002). IEEE Control Systems, 22(5), 10-25. doi:10.1109/mcs.2002.1035214Kourti, T. (2005). Application of latent variable methods to process control and multivariate statistical process control in industry. International Journal of Adaptive Control and Signal Processing, 19(4), 213-246. doi:10.1002/acs.859Kourti, T. (2006). Process Analytical Technology Beyond Real-Time Analyzers: The Role of Multivariate Analysis. Critical Reviews in Analytical Chemistry, 36(3-4), 257-278. doi:10.1080/10408340600969957Kourti, T., & MacGregor, J. F. (1996). Multivariate SPC Methods for Process and Product Monitoring. Journal of Quality Technology, 28(4), 409-428. doi:10.1080/00224065.1996.11979699Liu, R. Y. (1995). Control Charts for Multivariate Processes. Journal of the American Statistical Association, 90(432), 1380-1387. doi:10.1080/01621459.1995.10476643Liu, R. Y., Singh, K., & Teng*, J. H. (2004). DDMA-charts: Nonparametric multivariate moving average control charts based on data depth. Allgemeines Statistisches Archiv, 88(2), 235-258. doi:10.1007/s101820400170Liu, R. Y., & Tang, J. (1996). Control Charts for Dependent and Independent Measurements Based on Bootstrap Methods. Journal of the American Statistical Association, 91(436), 1694-1700. doi:10.1080/01621459.1996.10476740LOWRY, C. A., & MONTGOMERY, D. C. (1995). A review of multivariate control charts. IIE Transactions, 27(6), 800-810. doi:10.1080/07408179508936797MacGregor, J. F. (1997). Using On-Line Process Data to Improve Quality: Challenges for Statisticians. International Statistical Review, 65(3), 309-323. doi:10.1111/j.1751-5823.1997.tb00311.xMason, R. L., Champ, C. W., Tracy, N. D., Wierda, S. J., & Young, J. C. (1997). Assessment of Multivariate Process Control Techniques. Journal of Quality Technology, 29(2), 140-143. doi:10.1080/00224065.1997.11979743Montgomery, D. C., & Woodall, W. H. (1997). A Discussion on Statistically-Based Process Monitoring and Control. Journal of Quality Technology, 29(2), 121-121. doi:10.1080/00224065.1997.11979738Nelson, P. R. C., Taylor, P. A., & MacGregor, J. F. (1996). Missing data methods in PCA and PLS: Score calculations with incomplete observations. Chemometrics and Intelligent Laboratory Systems, 35(1), 45-65. doi:10.1016/s0169-7439(96)00007-xNomikos, P., & MacGregor, J. F. (1995). Multivariate SPC Charts for Monitoring Batch Processes. Technometrics, 37(1), 41-59. doi:10.1080/00401706.1995.10485888Prats-Montalbán, J. M., de Juan, A., & Ferrer, A. (2011). Multivariate image analysis: A review with applications. Chemometrics and Intelligent Laboratory Systems, 107(1), 1-23. doi:10.1016/j.chemolab.2011.03.002Prats-Montalbán, J. M., Ferrer, A., Malo, J. L., & Gorbeña, J. (2006). A comparison of different discriminant analysis techniques in a steel industry welding process. Chemometrics and Intelligent Laboratory Systems, 80(1), 109-119. doi:10.1016/j.chemolab.2005.08.005Prats-Montalbán, J. M., & Ferrer, A. (2007). Integration of colour and textural information in multivariate image analysis: defect detection and classification issues. Journal of Chemometrics, 21(1-2), 10-23. doi:10.1002/cem.1026Bisgaard, S., Doganaksoy, N., Fisher, N., Gunter, B., Hahn, G., Keller-McNulty, S., … Wu, C. F. J. (2008). The Future of Industrial Statistics: A Panel Discussion. Technometrics, 50(2), 103-127. doi:10.1198/004017008000000136Stoumbos, Z. G., Reynolds, M. R., Ryan, T. P., & Woodall, W. H. (2000). The State of Statistical Process Control as We Proceed into the 21st Century. Journal of the American Statistical Association, 95(451), 992-998. doi:10.1080/01621459.2000.10474292Tracy, N. D., Young, J. C., & Mason, R. L. (1992). Multivariate Control Charts for Individual Observations. Journal of Quality Technology, 24(2), 88-95. doi:10.1080/00224065.1992.12015232Wierda, S. J. (1994). Multivariate statistical process control—recent results and directions for future research. Statistica Neerlandica, 48(2), 147-168. doi:10.1111/j.1467-9574.1994.tb01439.xWold, S. (1978). Cross-Validatory Estimation of the Number of Components in Factor and Principal Components Models. Technometrics, 20(4), 397-405. doi:10.1080/00401706.1978.10489693Woodall, W. H. (2000). Controversies and Contradictions in Statistical Process Control. Journal of Quality Technology, 32(4), 341-350. doi:10.1080/00224065.2000.11980013Woodall, W. H., & Montgomery, D. C. (1999). Research Issues and Ideas in Statistical Process Control. Journal of Quality Technology, 31(4), 376-386. doi:10.1080/00224065.1999.11979944Yu, H., & MacGregor, J. F. (2003). Multivariate image analysis and regression for prediction of coating content and distribution in the production of snack foods. Chemometrics and Intelligent Laboratory Systems, 67(2), 125-144. doi:10.1016/s0169-7439(03)00065-0Yu, H., MacGregor, J. F., Haarsma, G., & Bourg, W. (2003). Digital Imaging for Online Monitoring and Control of Industrial Snack Food Processes. Industrial & Engineering Chemistry Research, 42(13), 3036-3044. doi:10.1021/ie020941

A systematic study on time between events control charts

Ph.DDOCTOR OF PHILOSOPH

Exploiting Robust Multivariate Statistics and Data Driven Techniques for Prognosis and Health Management

This thesis explores state of the art robust multivariate statistical methods and data driven techniques to holistically perform prognostics and health management (PHM). This provides a means to enable the early detection, diagnosis and prognosis of future asset failures. In this thesis, the developed PHM methodology is applied to wind turbine drive train components, specifically focussed on planetary gearbox bearings and gears. A novel methodology for the identification of relevant time-domain statistical features based upon robust statistical process control charts is presented for high frequency bearing accelerometer data. In total, 28 time-domain statistical features were evaluated for their capabilities as leading indicators of degradation. The results of this analysis describe the extensible multivariate “Moments’ model” for the encapsulation of bearing operational behaviour. This is presented, enabling the early degradation of detection, predictive diagnostics and estimation of remaining useful life (RUL). Following this, an extended physics of failure model based upon low frequency SCADA data for the quantification of wind turbine gearbox condition is described. This extends the state of the art, whilst defining robust performance charts for quantifying component condition. Normalisation against loading of the turbine and transient states based upon empirical data is performed in the bivariate domain, with extensibility into the multivariate domain if necessary. Prognosis of asset condition is found to be possible with the assistance of artificial neural networks in order to provide business intelligence to the planning and scheduling of effective maintenance actions. These multivariate condition models are explored with multivariate distance and similarity metrics for to exploit traditional data mining techniques for tacit knowledge extraction, ensemble diagnosis and prognosis. Estimation of bearing remaining useful life is found to be possible, with the derived technique correlating strongly to bearing life (r = .96

Development of a syndromic surveillance system to enhance early detection of emerging and re-emerging animal diseases

Animal health surveillance plays an important role in protecting animal health, production and welfare, public health and trade from the negative impacts of disease. To address the challenges posed by new, exotic or re-emerging diseases as well as the limitations of traditional surveillance, new approaches, including syndromic surveillance (SyS) and modern communication technologies have been developed to improve early disease detection. SyS is based on the continuous monitoring of unspecific pre-diagnostic health data in order to detect an unusual increase in counts which may indicate a health hazard in a timely manner. An increasing number of studies has been investigating different types of animal health data for a possible use in SyS. Although the potential of cattle mortality data routinely collected in national cattle registers for use in a SyS system was highlighted, the performance of aberration-detection algorithms applied to such data has not yet been investigated. Furthermore, knowledge about the impact of delayed reporting of these data on outbreak detection performance is limited. Clinical observations made by veterinary practitioners reported in real-time using web- and mobile-based communication tools may improve the timeliness of outbreak detection. The willingness of practitioners to report their observations is essential for the successful implementation of such systems. A lack of knowledge about factors that motivate or hinder practitioners to participate in surveillance was found. The aim of this work was to contribute to the development of a national surveillance system for the early detection of emerging and re-emerging animal diseases in Switzerland, focusing on two Swiss data sources: cattle mortality data routinely reported by farmers to the Swiss system for individual identification and registration of cattle (Tierverkehrsdatenbank TVD); clinical data voluntarily reported by veterinary practitioners to Equinella, an electronic reporting and information system for the early detection of infectious equine diseases in Switzerland. Time series of on-farm and perinatal cattle deaths, extracted from the TVD, were analysed with regard to data quality and explainable temporal patterns, e.g. day-of-week effect or seasonality. A set of three temporal aberration detection algorithms (Shewhart, CuSum, EWMA) was retrospectively applied to these data to assess their performance in detecting varying simulated disease outbreak scenarios. The effect of reporting delay on outbreak detection was investigated in a Bayesian framework. Participation of veterinary practitioners during the first 12 months of the new internet-based reporting platform of Equinella was assessed. Telephone interviews were conducted to gain insights into factors that motivate or hinder practitioners to participate in a voluntary surveillance system offering non-monetary incentives. Furthermore, the suitability of mobile devices such as smartphones for collecting health data was investigated. The TVD provided timely cattle mortality data with comprehensive geographical information, making it a valuable data source for Sys. Mortality time series exhibited temporal patterns, associated with non-health related factors, that had to be considered before applying aberration detection algorithms. The three evaluated control chart algorithms adequately performed under specific outbreak conditions, but none of them was superior in detecting outbreak signals across multiple evaluation metrics. Combining algorithms outputs according to different rules did not satisfactorily increase the system’s overall performance, further illustrating the difficulty in finding a balance between a high sensitivity and a manageable number of false alarms. The Bayesian approach performed similarly well in the scenario where delayed reporting was accounted for to the (ideal) scenario where it was absent. Non-monetary incentives were attractive to sentinel practitioners and overall participation was experienced positive. Insufficient understanding of the reporting system and of its relevance, as well as concerns over the electronic dissemination of health data were identified as potential challenges to sustainable reporting. Mobile devices were sporadically used during the first year and an awareness of the advantages of mobile-based surveillance was yet lacking among practitioners, indicating that they may require some time to become accustomed to novel reporting methods. This work highlighted the value of routinely collected cattle mortality data for use in SyS, but also the need to carefully optimise aberration detection algorithms for a particular data stream. Alternative methods to the binary alarm system may be chosen for a prospective use of cattle mortality data in a SyS system. The value of evidence framework may be suitable for surveillance systems with multiple syndromes and delayed reporting of data. Before integrating these data into a national surveillance system for the early detection of new, exotic or re-emerging diseases, health authorities need to define response protocols enabling investigation of the data that triggered a statistical alarm and to identify the underlying cause. Possibilities for improving sensitivity and specificity were identified that may be addressed when implementing a future SyS system. In addition, the potential of voluntary reporting surveillance system based on non-monetary incentives was shown. Many of the identified barriers to reporting can be addressed in the future, making the outcome of the pilot project favourable. Continued information feedback loops within voluntary sentinel networks will be important to ensure sustainable participation. Combining reporting of syndromic data and mobile devices in a One Health context has the potential to benefit animal and public health as well as to enhance interdisciplinary collaboration

Six Sigma

In the new millennium the increasing expectation of customers and products complexity has forced companies to find new solutions and better alternatives to improve the quality of their products. Lean and Six Sigma methodology provides the best solutions to many problems and can be used as an accelerator in industry, business and even health care sectors. Due to its flexible nature, the Lean and Six Sigma methodology was rapidly adopted by many top and even small companies. This book provides the necessary guidance for selecting, performing and evaluating various procedures of Lean and Six Sigma. In the book you will find personal experiences in the field of Lean and Six Sigma projects in business, industry and health sectors