Anomaly Detection in Aerial Videos with Transformers

Unmanned aerial vehicles (UAVs) are widely applied for purposes of inspection, search, and rescue operations by the virtue of low-cost, large-coverage, real-time, and high-resolution data acquisition capacities. Massive volumes of aerial videos are produced in these processes, in which normal events often account for an overwhelming proportion. It is extremely difficult to localize and extract abnormal events containing potentially valuable information from long video streams manually. Therefore, we are dedicated to developing anomaly detection methods to solve this issue. In this paper, we create a new dataset, named DroneAnomaly, for anomaly detection in aerial videos. This dataset provides 37 training video sequences and 22 testing video sequences from 7 different realistic scenes with various anomalous events. There are 87,488 color video frames (51,635 for training and 35,853 for testing) with the size of $640 \times 640$ at 30 frames per second. Based on this dataset, we evaluate existing methods and offer a benchmark for this task. Furthermore, we present a new baseline model, ANomaly Detection with Transformers (ANDT), which treats consecutive video frames as a sequence of tubelets, utilizes a Transformer encoder to learn feature representations from the sequence, and leverages a decoder to predict the next frame. Our network models normality in the training phase and identifies an event with unpredictable temporal dynamics as an anomaly in the test phase. Moreover, To comprehensively evaluate the performance of our proposed method, we use not only our Drone-Anomaly dataset but also another dataset. We will make our dataset and code publicly available. A demo video is available at https://youtu.be/ancczYryOBY. We make our dataset and code publicly available

Latent Structures based-Multivariate Statistical Process Control: a paradigm shift

The basic fundamentals of statistical process control (SPC) were proposed by Walter Shewhart for data-starved production environments typical in the 1920s and 1930s. In the 21st century, the traditional scarcity of data has given way to a data-rich environment typical of highly automated and computerized modern processes. These data often exhibit high correlation, rank deficiency, low signal-to-noise ratio, multistage and multiway structures, and missing values. Conventional univariate and multivariate SPC techniques are not suitable in these environments. This article discusses the paradigm shift to which those working in the quality improvement field should pay keen attention. We advocate the use of latent structure based multivariate statistical process control methods as efficient quality improvement tools in these massive data contexts. This is a strategic issue for industrial success in the tremendously competitive global market.This research work was partially supported by the Spanish Ministry of Economy and Competitiveness under the project DPI2011-28112-C04-02.Ferrer, A. (2014). Latent Structures based-Multivariate Statistical Process Control: a paradigm shift. Quality Engineering. 26(1):72-91. https://doi.org/10.1080/08982112.2013.846093S7291261Aparisi, F., Jabaioyes, J., & Carrion, A. (1999). Statistical properties of the lsi multivariate control chart. Communications in Statistics - Theory and Methods, 28(11), 2671-2686. doi:10.1080/03610929908832445Arteaga, F., & Ferrer, A. (2002). Dealing with missing data in MSPC: several methods, different interpretations, some examples. Journal of Chemometrics, 16(8-10), 408-418. doi:10.1002/cem.750Bersimis, S., Psarakis, S., & Panaretos, J. (2007). Multivariate statistical process control charts: an overview. Quality and Reliability Engineering International, 23(5), 517-543. doi:10.1002/qre.829Bharati, M. H., & MacGregor, J. F. (1998). Multivariate Image Analysis for Real-Time Process Monitoring and Control. Industrial & Engineering Chemistry Research, 37(12), 4715-4724. doi:10.1021/ie980334lBharati, M. H., MacGregor, J. F., & Tropper, W. (2003). Softwood Lumber Grading through On-line Multivariate Image Analysis Techniques. Industrial & Engineering Chemistry Research, 42(21), 5345-5353. doi:10.1021/ie0210560Bisgaard, S. (2012). The Future of Quality Technology: From a Manufacturing to a Knowledge Economy & From Defects to Innovations. Quality Engineering, 24(1), 30-36. doi:10.1080/08982112.2011.627010Box, G. E. P. (1954). Some Theorems on Quadratic Forms Applied in the Study of Analysis of Variance Problems, I. Effect of Inequality of Variance in the One-Way Classification. The Annals of Mathematical Statistics, 25(2), 290-302. doi:10.1214/aoms/1177728786Camacho, J., & Ferrer, A. (2012). Cross-validation in PCA models with the element-wise k-fold (ekf) algorithm: theoretical aspects. Journal of Chemometrics, 26(7), 361-373. doi:10.1002/cem.2440Duchesne, C., Liu, J. J., & MacGregor, J. F. (2012). Multivariate image analysis in the process industries: A review. Chemometrics and Intelligent Laboratory Systems, 117, 116-128. doi:10.1016/j.chemolab.2012.04.003Efron, B., & Gong, G. (1983). A Leisurely Look at the Bootstrap, the Jackknife, and Cross-Validation. The American Statistician, 37(1), 36-48. doi:10.1080/00031305.1983.10483087Ferrer, A. (2007). Multivariate Statistical Process Control Based on Principal Component Analysis (MSPC-PCA): Some Reflections and a Case Study in an Autobody Assembly Process. Quality Engineering, 19(4), 311-325. doi:10.1080/08982110701621304Fuchs, C. (1998). Multivariate Quality Control. doi:10.1201/9781482273731Geladi, P., & Kowalski, B. R. (1986). Partial least-squares regression: a tutorial. Analytica Chimica Acta, 185, 1-17. doi:10.1016/0003-2670(86)80028-9Helland, I. S. (1988). On the structure of partial least squares regression. Communications in Statistics - Simulation and Computation, 17(2), 581-607. doi:10.1080/03610918808812681Höskuldsson, A. (1988). PLS regression methods. Journal of Chemometrics, 2(3), 211-228. doi:10.1002/cem.1180020306Hunter, J. S. (1986). The Exponentially Weighted Moving Average. Journal of Quality Technology, 18(4), 203-210. doi:10.1080/00224065.1986.11979014Edward Jackson, J. (1985). Multivariate quality control. Communications in Statistics - Theory and Methods, 14(11), 2657-2688. doi:10.1080/03610928508829069Jackson, J. E., & Mudholkar, G. S. (1979). Control Procedures for Residuals Associated With Principal Component Analysis. Technometrics, 21(3), 341-349. doi:10.1080/00401706.1979.10489779Process analysis and abnormal situation detection: from theory to practice. (2002). IEEE Control Systems, 22(5), 10-25. doi:10.1109/mcs.2002.1035214Kourti, T. (2005). Application of latent variable methods to process control and multivariate statistical process control in industry. International Journal of Adaptive Control and Signal Processing, 19(4), 213-246. doi:10.1002/acs.859Kourti, T. (2006). Process Analytical Technology Beyond Real-Time Analyzers: The Role of Multivariate Analysis. Critical Reviews in Analytical Chemistry, 36(3-4), 257-278. doi:10.1080/10408340600969957Kourti, T., & MacGregor, J. F. (1996). Multivariate SPC Methods for Process and Product Monitoring. Journal of Quality Technology, 28(4), 409-428. doi:10.1080/00224065.1996.11979699Liu, R. Y. (1995). Control Charts for Multivariate Processes. Journal of the American Statistical Association, 90(432), 1380-1387. doi:10.1080/01621459.1995.10476643Liu, R. Y., Singh, K., & Teng*, J. H. (2004). DDMA-charts: Nonparametric multivariate moving average control charts based on data depth. Allgemeines Statistisches Archiv, 88(2), 235-258. doi:10.1007/s101820400170Liu, R. Y., & Tang, J. (1996). Control Charts for Dependent and Independent Measurements Based on Bootstrap Methods. Journal of the American Statistical Association, 91(436), 1694-1700. doi:10.1080/01621459.1996.10476740LOWRY, C. A., & MONTGOMERY, D. C. (1995). A review of multivariate control charts. IIE Transactions, 27(6), 800-810. doi:10.1080/07408179508936797MacGregor, J. F. (1997). Using On-Line Process Data to Improve Quality: Challenges for Statisticians. International Statistical Review, 65(3), 309-323. doi:10.1111/j.1751-5823.1997.tb00311.xMason, R. L., Champ, C. W., Tracy, N. D., Wierda, S. J., & Young, J. C. (1997). Assessment of Multivariate Process Control Techniques. Journal of Quality Technology, 29(2), 140-143. doi:10.1080/00224065.1997.11979743Montgomery, D. C., & Woodall, W. H. (1997). A Discussion on Statistically-Based Process Monitoring and Control. Journal of Quality Technology, 29(2), 121-121. doi:10.1080/00224065.1997.11979738Nelson, P. R. C., Taylor, P. A., & MacGregor, J. F. (1996). Missing data methods in PCA and PLS: Score calculations with incomplete observations. Chemometrics and Intelligent Laboratory Systems, 35(1), 45-65. doi:10.1016/s0169-7439(96)00007-xNomikos, P., & MacGregor, J. F. (1995). Multivariate SPC Charts for Monitoring Batch Processes. Technometrics, 37(1), 41-59. doi:10.1080/00401706.1995.10485888Prats-Montalbán, J. M., de Juan, A., & Ferrer, A. (2011). Multivariate image analysis: A review with applications. Chemometrics and Intelligent Laboratory Systems, 107(1), 1-23. doi:10.1016/j.chemolab.2011.03.002Prats-Montalbán, J. M., Ferrer, A., Malo, J. L., & Gorbeña, J. (2006). A comparison of different discriminant analysis techniques in a steel industry welding process. Chemometrics and Intelligent Laboratory Systems, 80(1), 109-119. doi:10.1016/j.chemolab.2005.08.005Prats-Montalbán, J. M., & Ferrer, A. (2007). Integration of colour and textural information in multivariate image analysis: defect detection and classification issues. Journal of Chemometrics, 21(1-2), 10-23. doi:10.1002/cem.1026Bisgaard, S., Doganaksoy, N., Fisher, N., Gunter, B., Hahn, G., Keller-McNulty, S., … Wu, C. F. J. (2008). The Future of Industrial Statistics: A Panel Discussion. Technometrics, 50(2), 103-127. doi:10.1198/004017008000000136Stoumbos, Z. G., Reynolds, M. R., Ryan, T. P., & Woodall, W. H. (2000). The State of Statistical Process Control as We Proceed into the 21st Century. Journal of the American Statistical Association, 95(451), 992-998. doi:10.1080/01621459.2000.10474292Tracy, N. D., Young, J. C., & Mason, R. L. (1992). Multivariate Control Charts for Individual Observations. Journal of Quality Technology, 24(2), 88-95. doi:10.1080/00224065.1992.12015232Wierda, S. J. (1994). Multivariate statistical process control—recent results and directions for future research. Statistica Neerlandica, 48(2), 147-168. doi:10.1111/j.1467-9574.1994.tb01439.xWold, S. (1978). Cross-Validatory Estimation of the Number of Components in Factor and Principal Components Models. Technometrics, 20(4), 397-405. doi:10.1080/00401706.1978.10489693Woodall, W. H. (2000). Controversies and Contradictions in Statistical Process Control. Journal of Quality Technology, 32(4), 341-350. doi:10.1080/00224065.2000.11980013Woodall, W. H., & Montgomery, D. C. (1999). Research Issues and Ideas in Statistical Process Control. Journal of Quality Technology, 31(4), 376-386. doi:10.1080/00224065.1999.11979944Yu, H., & MacGregor, J. F. (2003). Multivariate image analysis and regression for prediction of coating content and distribution in the production of snack foods. Chemometrics and Intelligent Laboratory Systems, 67(2), 125-144. doi:10.1016/s0169-7439(03)00065-0Yu, H., MacGregor, J. F., Haarsma, G., & Bourg, W. (2003). Digital Imaging for Online Monitoring and Control of Industrial Snack Food Processes. Industrial & Engineering Chemistry Research, 42(13), 3036-3044. doi:10.1021/ie020941

Dietary variations in three co-occurring rockfish species off the Pacific Northwest during anomalous oceanographic events in 1998 and 1999

Stomach samples from three rockfish species, yellowtail (Sebastes f lavidus), widow (S. entomelas), and canary (S. pinniger) rockfish, seasonally collected off the Pacific Northwest in 1998 and 1999, provided quantitative information on the food habits of these species during and after the 1997–98 El Niño event. Although euphausiids were the most common major prey of all three predators, gelatinous zooplankton and fishes were the most commonly consumed prey items during some seasonal quarters. The influence of the El Niño event was evident in the diets. Anomalous prey items, including the southern euphausiid species Nyctiphanes simplex and juveniles of Pacific whiting (Merluccius productus) frequently appeared in the diets in the spring and summer of 1998. The results of stomach contents analyses, based on 905 stomach samples from 49 trawl hauls during seven commercial fishing trips and from 56 stations during research surveys, were consistent with the timing of occurrence and the magnitude of change in biomass of some zooplankton species reported from zooplankton studies in the northern California Current during the 1997–98 El Niño. Our findings indicate that the observed variations of prey groups in some rockfish diets may be a function of prey variability related to climate and environment changes

Action Recognition in Videos: from Motion Capture Labs to the Web

This paper presents a survey of human action recognition approaches based on visual data recorded from a single video camera. We propose an organizing framework which puts in evidence the evolution of the area, with techniques moving from heavily constrained motion capture scenarios towards more challenging, realistic, "in the wild" videos. The proposed organization is based on the representation used as input for the recognition task, emphasizing the hypothesis assumed and thus, the constraints imposed on the type of video that each technique is able to address. Expliciting the hypothesis and constraints makes the framework particularly useful to select a method, given an application. Another advantage of the proposed organization is that it allows categorizing newest approaches seamlessly with traditional ones, while providing an insightful perspective of the evolution of the action recognition task up to now. That perspective is the basis for the discussion in the end of the paper, where we also present the main open issues in the area.Comment: Preprint submitted to CVIU, survey paper, 46 pages, 2 figures, 4 table

Bio-inspired relevant interaction modelling in cognitive crowd management

Cognitive algorithms, integrated in intelligent systems, represent an important innovation in designing interactive smart environments. More in details, Cognitive Systems have important applications in anomaly detection and management in advanced video surveillance. These algorithms mainly address the problem of modelling interactions and behaviours among the main entities in a scene. A bio-inspired structure is here proposed, which is able to encode and synthesize signals, not only for the description of single entities behaviours, but also for modelling cause–effect relationships between user actions and changes in environment configurations. Such models are stored within a memory (Autobiographical Memory) during a learning phase. Here the system operates an effective knowledge transfer from a human operator towards an automatic systems called Cognitive Surveillance Node (CSN), which is part of a complex cognitive JDL-based and bio-inspired architecture. After such a knowledge-transfer phase, learned representations can be used, at different levels, either to support human decisions, by detecting anomalous interaction models and thus compensating for human shortcomings, or, in an automatic decision scenario, to identify anomalous patterns and choose the best strategy to preserve stability of the entire system. Results are presented in a video surveillance scenario , where the CSN can observe two interacting entities consisting in a simulated crowd and a human operator. These can interact within a visual 3D simulator, where crowd behaviour is modelled by means of Social Forces. The way anomalies are detected and consequently handled is demonstrated, on synthetic and also on real video sequences, in both the user-support and automatic modes

Laboratory Development of an AI System for the Real-Time Monitoring of Water Quality and Detection of Anomalies Arising from Chemical Contamination

Monitoring water quality is critical for mitigating risks to human health and the environment. It is also essential for ensuring high quality water-based and water-dependent products and services. The monitoring and detection of chemical contamination are often based around a small set of parameters or substances. Conventional monitoring often involves the collection of water samples in the field and subsequent analyses in the laboratory. Such strategies are expensive, time consuming, and focused on a narrow set of potential risks. They also induce a significant time delay between a contamination event and a possible reactive measure. Here, we developed a real-time monitoring system based on Artificial Intelligence (AI) for field deployable sensors. We used data obtained from full-scan UV-spec and fluorescence sensors for validation in this study. This multi-sensor system consists of (a) anomaly detection that uses multivariate statistical methods to detect any anomalous state in an aqueous environment and (b) anomaly identification, using Machine Learning (ML) to classify the anomaly into one of the a priori known categories. For a proof of concept, we tested this methodology on a supply of municipal drinking water and a few representative organic chemical contaminants applied in a laboratory-controlled environment. The outcomes confirm the ability for the multi-sensor system to detect and identify changes in water quality due to incidences of chemical contamination. The method may be applied to numerous other areas where water quality should be measured online and in real time, such as in surface-water, urban runoff, or food and industrial process water.publishedVersio

Multivariate Statistical Process Control Charts and the Problem of Interpretation: A Short Overview and Some Applications in Industry

Woodall and Montgomery in a discussion paper, state that multivariate process control is one of the most rapidly developing sections of statistical process control. Nowadays, in industry, there are many situations in which the simultaneous monitoring or control, of two or more related quality - process characteristics is necessary. Process monitoring problems in which several related variables are of interest are collectively known as Multivariate Statistical Process Control (MSPC). This article has three parts. In the first part, we discuss in brief the basic procedures for the implementation of multivariate statistical process control via control charting. In the second part we present the most useful procedures for interpreting the out-of-control variable when a control charting procedure gives an out-of-control signal in a multivariate process. Finally, in the third, we present applications of multivariate statistical process control in the area of industrial process control, informatics, and businessQuality Control, Process Control, Multivariate Statistical Process Control, Hotelling's T², CUSUM, EWMA, PCA, PLS, Identification, Interpretation