Self-management and Optimization Framework. OpenIoT Deliverable D512

This deliverable describes the OpenIoT self-management and optimization framework, in terms of algorithms and mechanisms that it comprises as well as in terms of their implementation over the OpenIoT platform and associated cloud infrastructure. As a first step the main operations and functionalities of the OpenIoT self-management and optimization infrastructure are described and related to the structure of management operations defined in state-of-the-art frameworks for autonomic computing and self-management. Along with a brief description of the optimization techniques that are employed in OpenIoT, an initial mapping of the various techniques on the OpenIoT architecture is performed

Big Data and Artificial Intelligence in Digital Finance

This open access book presents how cutting-edge digital technologies like Big Data, Machine Learning, Artificial Intelligence (AI), and Blockchain are set to disrupt the financial sector. The book illustrates how recent advances in these technologies facilitate banks, FinTech, and financial institutions to collect, process, analyze, and fully leverage the very large amounts of data that are nowadays produced and exchanged in the sector. To this end, the book also describes some more the most popular Big Data, AI and Blockchain applications in the sector, including novel applications in the areas of Know Your Customer (KYC), Personalized Wealth Management and Asset Management, Portfolio Risk Assessment, as well as variety of novel Usage-based Insurance applications based on Internet-of-Things data. Most of the presented applications have been developed, deployed and validated in real-life digital finance settings in the context of the European Commission funded INFINITECH project, which is a flagship innovation initiative for Big Data and AI in digital finance. This book is ideal for researchers and practitioners in Big Data, AI, banking and digital finance

Non-intrusive slot layering in Hadoop

Hadoop, an open source implementation of MapReduce, uses slots to represent resource sharing. The number of slots in a Hadoop cluster node specifies the concurrency of task execution. Thus, the slot configuration has a significant impact on performance. The number of slots is by default hand-configured (static) and slots share resources "fairly". As resource capacity (e.g., #cores) continues to increase and application dynamics becomes increasingly diverse, the current practices of static slot configuration and fair resource sharing may not efficiently utilize resources. Besides, such fair sharing is against priority-based scheduling when high priority jobs are sharing resource with lower priority jobs. In this paper we study the optimization of resource utilization in Hadoop focusing on those two issues of current practices and present a non-intrusive slot layering solution. Our solution approach in essence uses two tiers of slot (Active and Passive) to increase the degree of concurrency with minimal performance interference between them. Tasks in the Passive slots proceed their execution when tasks in the Active slots are not fully using (CPU) resource, and tasks/slots in these tiers are dynamically and adaptively managed. To leverage the effectiveness of slot layering, we develop a layering-aware task scheduler. Our non-intrusive slot layering approach is unique in that (1) it is a generic way to manage resource sharing for parallel and distributed computing models (e.g., MPI and cloud computing) and (2) both overall throughput and high-priority job performance are improved. Our experimental results with 6 representative jobs show 3%-34% improvement in overall throughput and 13%-48% decrease in the executing time of high-priority jobs compared with static configurations.8 page(s