ALOJA: A framework for benchmarking and predictive analytics in Hadoop deployments

This article presents the ALOJA project and its analytics tools, which leverages machine learning to interpret Big Data benchmark performance data and tuning. ALOJA is part of a long-term collaboration between BSC and Microsoft to automate the characterization of cost-effectiveness on Big Data deployments, currently focusing on Hadoop. Hadoop presents a complex run-time environment, where costs and performance depend on a large number of configuration choices. The ALOJA project has created an open, vendor-neutral repository, featuring over 40,000 Hadoop job executions and their performance details. The repository is accompanied by a test-bed and tools to deploy and evaluate the cost-effectiveness of different hardware configurations, parameters and Cloud services. Despite early success within ALOJA, a comprehensive study requires automation of modeling procedures to allow an analysis of large and resource-constrained search spaces. The predictive analytics extension, ALOJA-ML, provides an automated system allowing knowledge discovery by modeling environments from observed executions. The resulting models can forecast execution behaviors, predicting execution times for new configurations and hardware choices. That also enables model-based anomaly detection or efficient benchmark guidance by prioritizing executions. In addition, the community can benefit from ALOJA data-sets and framework to improve the design and deployment of Big Data applications.This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 639595). This work is partially supported by the Ministry of Economy of Spain under contracts TIN2012-34557 and 2014SGR1051.Peer ReviewedPostprint (published version

Self-tuning diagnosis of routine alarms in rotating plant items

Condition monitoring of rotating plant items in the energy generation industry is often achieved through examination of vibration signals. Engineers use this data to monitor the operation of turbine generators, gas circulators and other key plant assets. A common approach in such monitoring is to trigger an alarm when a vibration deviates from a predefined envelope of normal operation. This limit-based approach, however, generates a large volume of alarms not indicative of system damage or concern, such as operational transients that result in temporary increases in vibration. In the nuclear generation context, all alarms on rotating plant assets must be analysed and subjected to auditable review. The analysis of these alarms is often undertaken manually, on a case- by-case basis, but recent developments in monitoring research have brought forward the use of intelligent systems techniques to automate parts of this process. A knowledge- based system (KBS) has been developed to automatically analyse routine alarms, where the underlying cause can be attributed to observable operational changes. The initialisation and ongoing calibration of such systems, however, is a problem, as normal machine state is not uniform throughout asset life due to maintenance procedures and the wear of components. In addition, different machines will exhibit differing vibro- acoustic dynamics. This paper proposes a self-tuning knowledge-driven analysis system for routine alarm diagnosis across the key rotating plant items within the nuclear context common to the UK. Such a system has the ability to automatically infer the causes of routine alarms, and provide auditable reports to the engineering staff

Automatic Maritime Traffic Anomalous Behaviors Detection

Maritime traffic plays a very important role in the world economy, with over 90% of global trading done through naval transportation. The high amount of vessel traffic, mainly due to cargo transportation, leads to several new risks, threats, and concerns, such as increased criminal activity in the sea. The OVERSEE project is proprietary software developed by Crit ical Software and used by Marinha Portuguesa, Irish Coast Guard, and Papua New Guinea’s Coast Guard. The OVERSEE project displays vessel information in real-time through AIS messages, which are mandatory for most cargo vessels to report consistently. Anomaly de tection and behavior monitoring tools are computer-based systems that analyse real-time data to detect anomalous behaviors. This project aims to develop a solution capable of detecting anomalous behaviors committed by vessels using AIS messages, which will be re ported in real-time automatically via e-mail and the extant OVERSEE graphical interface. The solution is developed with the use of Long Short-Term Memory Recurrent Neural Net works, and a deeper analysis is provided to compare the obtained results with the ideal results. The network training and testing are done with real data, with cross-classification techniques to improve the trustworthiness of the algorithm, hence providing more accurate results.O tráfego marítimo desempenha um papel muito importante na economia mundial, com mais de 90% do comércio global feito por meio do transporte naval. O grande volume de tráfego de embarcações, principalmente devido ao transporte de cargas, leva a vários novos riscos, ameaças e preocupações, como o aumento da criminalidade no mar. O projeto OVERSEE é um software proprietário desenvolvido pela Critical Software e usado pela Marinha Portuguesa, Guarda Costeira Irlandesa e Guarda Costeira da Papua Nova Guiné. O projeto OVERSEE exibe informações da embarcação em tempo real por meio de mensagens AIS, cuja maioria das embarcações de carga são obrigadas a relatar num período de tempo regular. As ferramentas de detecção de anomalias e monitoramento de comportamento são sistemas baseados em computador que analisam dados em tempo real para detetar comportamentos anómalos. Este projeto visa desenvolver uma solução capaz de detetar comportamentos anómalos cometidos por embarcações por meio de mensagens AIS, que serão reportados em tempo real automaticamente via e-mail e interface gráfica existente do OVERSEE. A solução está desenvolvida com o uso de Redes Neurais Recorrentes1 de Memória-Curta de Longo Prazo2 . Uma análise mais profunda é fornecida para comparar os resultados obtidos com os resultados ideais. O treinamento e teste da rede são feitos com dados reais, com técnicas de classificação cruzada para melhorar a confiabilidade do algoritmo, fornecendo resultados mais precisos