Artificial intelligence driven anomaly detection for big data systems

The main goal of this thesis is to contribute to the research on automated performance anomaly detection and interference prediction by implementing Artificial Intelligence (AI) solutions for complex distributed systems, especially for Big Data platforms within cloud computing environments. The late detection and manual resolutions of performance anomalies and system interference in Big Data systems may lead to performance violations and financial penalties. Motivated by this issue, we propose AI-based methodologies for anomaly detection and interference prediction tailored to Big Data and containerized batch platforms to better analyze system performance and effectively utilize computing resources within cloud environments. Therefore, new precise and efficient performance management methods are the key to handling performance anomalies and interference impacts to improve the efficiency of data center resources. The first part of this thesis contributes to performance anomaly detection for in-memory Big Data platforms. We examine the performance of Big Data platforms and justify our choice of selecting the in-memory Apache Spark platform. An artificial neural network-driven methodology is proposed to detect and classify performance anomalies for batch workloads based on the RDD characteristics and operating system monitoring metrics. Our method is evaluated against other popular machine learning algorithms (ML), as well as against four different monitoring datasets. The results prove that our proposed method outperforms other ML methods, typically achieving 98–99% F-scores. Moreover, we prove that a random start instant, a random duration, and overlapped anomalies do not significantly impact the performance of our proposed methodology. The second contribution addresses the challenge of anomaly identification within an in-memory streaming Big Data platform by investigating agile hybrid learning techniques. We develop TRACK (neural neTwoRk Anomaly deteCtion in sparK) and TRACK-Plus, two methods to efficiently train a class of machine learning models for performance anomaly detection using a fixed number of experiments. Our model revolves around using artificial neural networks with Bayesian Optimization (BO) to find the optimal training dataset size and configuration parameters to efficiently train the anomaly detection model to achieve high accuracy. The objective is to accelerate the search process for finding the size of the training dataset, optimizing neural network configurations, and improving the performance of anomaly classification. A validation based on several datasets from a real Apache Spark Streaming system is performed, demonstrating that the proposed methodology can efficiently identify performance anomalies, near-optimal configuration parameters, and a near-optimal training dataset size while reducing the number of experiments up to 75% compared with naïve anomaly detection training. The last contribution overcomes the challenges of predicting completion time of containerized batch jobs and proactively avoiding performance interference by introducing an automated prediction solution to estimate interference among colocated batch jobs within the same computing environment. An AI-driven model is implemented to predict the interference among batch jobs before it occurs within system. Our interference detection model can alleviate and estimate the task slowdown affected by the interference. This model assists the system operators in making an accurate decision to optimize job placement. Our model is agnostic to the business logic internal to each job. Instead, it is learned from system performance data by applying artificial neural networks to establish the completion time prediction of batch jobs within the cloud environments. We compare our model with three other baseline models (queueing-theoretic model, operational analysis, and an empirical method) on historical measurements of job completion time and CPU run-queue size (i.e., the number of active threads in the system). The proposed model captures multithreading, operating system scheduling, sleeping time, and job priorities. A validation based on 4500 experiments based on the DaCapo benchmarking suite was carried out, confirming the predictive efficiency and capabilities of the proposed model by achieving up to 10% MAPE compared with the other models.Open Acces

An Autoethnographic Account of Innovation at the US Department of Veterans Affairs

The history of the U.S. Department of Veterans Affairs (VA) health information technology (HIT) has been characterized by both enormous successes and catastrophic failures. While the VA was once hailed as the way to the future of twenty-first-century health care, many programs have been mismanaged, delayed, or flawed, resulting in the waste of hundreds of millions of taxpayer dollars. Since 2015 the U.S. Government Accountability Office (GAO) has designated HIT at the VA as being susceptible to waste, fraud, and mismanagement. The timely central research question I ask in this study is, can healthcare IT at the VA be healed? To address this question, I investigate a HIT case study at the VA Center of Innovation (VACI), originally designed to be the flagship initiative of the open government transformation at the VA. The Open Source Electronic Health Record Alliance (OSEHRA) was designed to promote the open innovation ecosystem public-private-academic partnership. Based on my fifteen years of experience at the VA, I use an autoethnographic methodology to make a significant value-added contribution to understanding and modeling the VA’s approach to innovation. I use several theoretical information system framework models including People, Process, and Technology (PPT), Technology, Organization and Environment (TOE), and Technology Adaptive Model (TAM) and propose a new adaptive theory to understand the inability of VA HIT to innovate. From the perspective of people and culture, I study retaliation against whistleblowers, organization behavioral integrity, and lack of transparency in communications. I examine the VA processes, including the different software development methodologies used, the development and operations process (DevOps) of an open-source application developed at VACI, the Radiology Protocol Tool Recorder (RAPTOR), a Veterans Health Information Systems and Technology Architecture (VistA) radiology workflow module. I find that the VA has chosen to migrate away from inhouse application software and buy commercial software. The impact of these People, Process, and Technology findings are representative of larger systemic failings and are appropriate examples to illustrate systemic issues associated with IT innovation at the VA. This autoethnographic account builds on first-hand project experience and literature-based insights

The bridge of dreams::Towards a method for operational performance alignment in IT-enabled service supply chains

Concerns on performance alignment, especially on business-IT alignment, have been around for three decades. It is still considered to be one of the most important driving forces for business success, as well as one of the top concerns of many practitioners and organizational researchers. It is also found to be a major issue in two thirds of digital transformation projects. Many attempts from researchers in diverse disciplines have been made to tackle this issue. Unfortunately, they have been working separately and the research appears in various forms and names. This dissertation presents a piece of interdisciplinary research that focuses on identifying operational performance alignment issues, discovering and assessing their root causes with attention to the dynamics in operating IT-enabled service supply chain (SSC). It makes a modest contribution by providing a communication-centred instrument which can modularize complex SSC in terms of a hierarchically-structured set of services and analyze the performance causality between them. With a special focus on the impact of IT, it makes it possible to monitor and tune various performance issues in SSC. This research intends to provide a solution-oriented common ground where multiple service research streams can meet together. Following the framework proposed in this research, services, at different tiers of an SSC, are modelled with a balanced perspective on both business, technical service components and KPIs. It allows a holistic picture of service performances and interactions throughout the entire supply chain to be viewed through a different research lens and permits the causal impact of technology, business strategy, and service operations on supply chain performance to be unveiled

The Essence of Software Engineering

Software Engineering; Software Development; Software Processes; Software Architectures; Software Managemen

Standard Interfaces and Protocols at Sensor Network and Cloud Level Definition

In this paper we presented full design of the system for monitoring forest which consists of cloud platform, sensor networks and mobile (drone) technologies for data collection and cameras. We first present the advanced design and structural model of an advanced system for monitoring of forest area. This model integrate sensor networks and mobile (drone) technologies for data collection and acquisition of those data at existing Crisis Management Information Systems (CMIS). Then we demonstrate the possibility to map different technological solutions and the main result was the definition of the set of standard interfaces and protocols for network interoperability

End-userApplication for Early Forest Fire Detection and Prevention

n this paper, we describe a Web application that has been designed and implemented by Fulda University of Applied Sciences in the context of the ASPires project. The application extends the functionality available to Crisis Management Centers (CMC). Actual readings from sensors installed in the test areas, for example national parks, are made available to CMC personnel, as well as pictures from cameras that are either mounted on stationary observation towers or taken by Unmanned Aerial Vehicles (UAVs) in the area of an actual of supposed forest fire. Data are transmitted to the Aspires cloud and delivered swiftly to the Web application via an open interface. Furthermore, fire alarms raised by novel detection algorithms are forwarded automatically to the application. This clearly improves the potential for the early detection of forest fires in rural areas

Agile Processes in Software Engineering and Extreme Programming

This open access book constitutes the proceedings of the 21st International Conference on Agile Software Development, XP 2020, which was planned to be held during June 8-12, 2020, at the IT University of Copenhagen, Denmark. However, due to the COVID-19 pandemic the conference was postponed until an undetermined date. XP is the premier agile software development conference combining research and practice. It is a hybrid forum where agile researchers, academics, practitioners, thought leaders, coaches, and trainers get together to present and discuss their most recent innovations, research results, experiences, concerns, challenges, and trends. Following this history, for both researchers and seasoned practitioners XP 2020 provided an informal environment to network, share, and discover trends in Agile for the next 20 years. The 14 full and 2 short papers presented in this volume were carefully reviewed and selected from 37 submissions. They were organized in topical sections named: agile adoption; agile practices; large-scale agile; the business of agile; and agile and testing

Next Generation Internet of Things – Distributed Intelligence at the Edge and Human-Machine Interactions

This book provides an overview of the next generation Internet of Things (IoT), ranging from research, innovation, development priorities, to enabling technologies in a global context. It is intended as a standalone in a series covering the activities of the Internet of Things European Research Cluster (IERC), including research, technological innovation, validation, and deployment.The following chapters build on the ideas put forward by the European Research Cluster, the IoT European Platform Initiative (IoT–EPI), the IoT European Large-Scale Pilots Programme and the IoT European Security and Privacy Projects, presenting global views and state-of-the-art results regarding the next generation of IoT research, innovation, development, and deployment.The IoT and Industrial Internet of Things (IIoT) are evolving towards the next generation of Tactile IoT/IIoT, bringing together hyperconnectivity (5G and beyond), edge computing, Distributed Ledger Technologies (DLTs), virtual/ andaugmented reality (VR/AR), and artificial intelligence (AI) transformation.Following the wider adoption of consumer IoT, the next generation of IoT/IIoT innovation for business is driven by industries, addressing interoperability issues and providing new end-to-end security solutions to face continuous treats.The advances of AI technology in vision, speech recognition, natural language processing and dialog are enabling the development of end-to-end intelligent systems encapsulating multiple technologies, delivering services in real-time using limited resources. These developments are focusing on designing and delivering embedded and hierarchical AI solutions in IoT/IIoT, edge computing, using distributed architectures, DLTs platforms and distributed end-to-end security, which provide real-time decisions using less data and computational resources, while accessing each type of resource in a way that enhances the accuracy and performance of models in the various IoT/IIoT applications.The convergence and combination of IoT, AI and other related technologies to derive insights, decisions and revenue from sensor data provide new business models and sources of monetization. Meanwhile, scalable, IoT-enabled applications have become part of larger business objectives, enabling digital transformation with a focus on new services and applications.Serving the next generation of Tactile IoT/IIoT real-time use cases over 5G and Network Slicing technology is essential for consumer and industrial applications and support reducing operational costs, increasing efficiency and leveraging additional capabilities for real-time autonomous systems.New IoT distributed architectures, combined with system-level architectures for edge/fog computing, are evolving IoT platforms, including AI and DLTs, with embedded intelligence into the hyperconnectivity infrastructure.The next generation of IoT/IIoT technologies are highly transformational, enabling innovation at scale, and autonomous decision-making in various application domains such as healthcare, smart homes, smart buildings, smart cities, energy, agriculture, transportation and autonomous vehicles, the military, logistics and supply chain, retail and wholesale, manufacturing, mining and oil and gas

Next Generation Internet of Things – Distributed Intelligence at the Edge and Human-Machine Interactions

This book provides an overview of the next generation Internet of Things (IoT), ranging from research, innovation, development priorities, to enabling technologies in a global context. It is intended as a standalone in a series covering the activities of the Internet of Things European Research Cluster (IERC), including research, technological innovation, validation, and deployment.The following chapters build on the ideas put forward by the European Research Cluster, the IoT European Platform Initiative (IoT–EPI), the IoT European Large-Scale Pilots Programme and the IoT European Security and Privacy Projects, presenting global views and state-of-the-art results regarding the next generation of IoT research, innovation, development, and deployment.The IoT and Industrial Internet of Things (IIoT) are evolving towards the next generation of Tactile IoT/IIoT, bringing together hyperconnectivity (5G and beyond), edge computing, Distributed Ledger Technologies (DLTs), virtual/ andaugmented reality (VR/AR), and artificial intelligence (AI) transformation.Following the wider adoption of consumer IoT, the next generation of IoT/IIoT innovation for business is driven by industries, addressing interoperability issues and providing new end-to-end security solutions to face continuous treats.The advances of AI technology in vision, speech recognition, natural language processing and dialog are enabling the development of end-to-end intelligent systems encapsulating multiple technologies, delivering services in real-time using limited resources. These developments are focusing on designing and delivering embedded and hierarchical AI solutions in IoT/IIoT, edge computing, using distributed architectures, DLTs platforms and distributed end-to-end security, which provide real-time decisions using less data and computational resources, while accessing each type of resource in a way that enhances the accuracy and performance of models in the various IoT/IIoT applications.The convergence and combination of IoT, AI and other related technologies to derive insights, decisions and revenue from sensor data provide new business models and sources of monetization. Meanwhile, scalable, IoT-enabled applications have become part of larger business objectives, enabling digital transformation with a focus on new services and applications.Serving the next generation of Tactile IoT/IIoT real-time use cases over 5G and Network Slicing technology is essential for consumer and industrial applications and support reducing operational costs, increasing efficiency and leveraging additional capabilities for real-time autonomous systems.New IoT distributed architectures, combined with system-level architectures for edge/fog computing, are evolving IoT platforms, including AI and DLTs, with embedded intelligence into the hyperconnectivity infrastructure.The next generation of IoT/IIoT technologies are highly transformational, enabling innovation at scale, and autonomous decision-making in various application domains such as healthcare, smart homes, smart buildings, smart cities, energy, agriculture, transportation and autonomous vehicles, the military, logistics and supply chain, retail and wholesale, manufacturing, mining and oil and gas