HEC: Collaborative Research: SAM^2 Toolkit: Scalable and Adaptive Metadata Management for High-End Computing

The increasing demand for Exa-byte-scale storage capacity by high end computing applications requires a higher level of scalability and dependability than that provided by current file and storage systems. The proposal deals with file systems research for metadata management of scalable cluster-based parallel and distributed file storage systems in the HEC environment. It aims to develop a scalable and adaptive metadata management (SAM2) toolkit to extend features of and fully leverage the peak performance promised by state-of-the-art cluster-based parallel and distributed file storage systems used by the high performance computing community. There is a large body of research on data movement and management scaling, however, the need to scale up the attributes of cluster-based file systems and I/O, that is, metadata, has been underestimated. An understanding of the characteristics of metadata traffic, and an application of proper load-balancing, caching, prefetching and grouping mechanisms to perform metadata management correspondingly, will lead to a high scalability. It is anticipated that by appropriately plugging the scalable and adaptive metadata management components into the state-of-the-art cluster-based parallel and distributed file storage systems one could potentially increase the performance of applications and file systems, and help translate the promise and potential of high peak performance of such systems to real application performance improvements. The project involves the following components: 1. Develop multi-variable forecasting models to analyze and predict file metadata access patterns. 2. Develop scalable and adaptive file name mapping schemes using the duplicative Bloom filter array technique to enforce load balance and increase scalability 3. Develop decentralized, locality-aware metadata grouping schemes to facilitate the bulk metadata operations such as prefetching. 4. Develop an adaptive cache coherence protocol using a distributed shared object model for client-side and server-side metadata caching. 5. Prototype the SAM2 components into the state-of-the-art parallel virtual file system PVFS2 and a distributed storage data caching system, set up an experimental framework for a DOE CMS Tier 2 site at University of Nebraska-Lincoln and conduct benchmark, evaluation and validation studies

Security in Computer and Information Sciences

This open access book constitutes the thoroughly refereed proceedings of the Second International Symposium on Computer and Information Sciences, EuroCybersec 2021, held in Nice, France, in October 2021. The 9 papers presented together with 1 invited paper were carefully reviewed and selected from 21 submissions. The papers focus on topics of security of distributed interconnected systems, software systems, Internet of Things, health informatics systems, energy systems, digital cities, digital economy, mobile networks, and the underlying physical and network infrastructures. This is an open access book

Timely processing of big data in collaborative large-scale distributed systems

Today’s Big Data phenomenon, characterized by huge volumes of data produced at very high rates by heterogeneous and geographically dispersed sources, is fostering the employment of large-scale distributed systems in order to leverage parallelism, fault tolerance and locality awareness with the aim of delivering suitable performances. Among the several areas where Big Data is gaining increasing significance, the protection of Critical Infrastructure is one of the most strategic since it impacts on the stability and safety of entire countries. Intrusion detection mechanisms can benefit a lot from novel Big Data technologies because these allow to exploit much more information in order to sharpen the accuracy of threats discovery. A key aspect for increasing even more the amount of data at disposal for detection purposes is the collaboration (meant as information sharing) among distinct actors that share the common goal of maximizing the chances to recognize malicious activities earlier. Indeed, if an agreement can be found to share their data, they all have the possibility to definitely improve their cyber defenses. The abstraction of Semantic Room (SR) allows interested parties to form trusted and contractually regulated federations, the Semantic Rooms, for the sake of secure information sharing and processing. Another crucial point for the effectiveness of cyber protection mechanisms is the timeliness of the detection, because the sooner a threat is identified, the faster proper countermeasures can be put in place so as to confine any damage. Within this context, the contributions reported in this thesis are threefold * As a case study to show how collaboration can enhance the efficacy of security tools, we developed a novel algorithm for the detection of stealthy port scans, named R-SYN (Ranked SYN port scan detection). We implemented it in three distinct technologies, all of them integrated within an SR-compliant architecture that allows for collaboration through information sharing: (i) in a centralized Complex Event Processing (CEP) engine (Esper), (ii) in a framework for distributed event processing (Storm) and (iii) in Agilis, a novel platform for batch-oriented processing which leverages the Hadoop framework and a RAM-based storage for fast data access. Regardless of the employed technology, all the evaluations have shown that increasing the number of participants (that is, increasing the amount of input data at disposal), allows to improve the detection accuracy. The experiments made clear that a distributed approach allows for lower detection latency and for keeping up with higher input throughput, compared with a centralized one. * Distributing the computation over a set of physical nodes introduces the issue of improving the way available resources are assigned to the elaboration tasks to execute, with the aim of minimizing the time the computation takes to complete. We investigated this aspect in Storm by developing two distinct scheduling algorithms, both aimed at decreasing the average elaboration time of the single input event by decreasing the inter-node traffic. Experimental evaluations showed that these two algorithms can improve the performance up to 30%. * Computations in online processing platforms (like Esper and Storm) are run continuously, and the need of refining running computations or adding new computations, together with the need to cope with the variability of the input, requires the possibility to adapt the resource allocation at runtime, which entails a set of additional problems. Among them, the most relevant concern how to cope with incoming data and processing state while the topology is being reconfigured, and the issue of temporary reduced performance. At this aim, we also explored the alternative approach of running the computation periodically on batches of input data: although it involves a performance penalty on the elaboration latency, it allows to eliminate the great complexity of dynamic reconfigurations. We chose Hadoop as batch-oriented processing framework and we developed some strategies specific for dealing with computations based on time windows, which are very likely to be used for pattern recognition purposes, like in the case of intrusion detection. Our evaluations provided a comparison of these strategies and made evident the kind of performance that this approach can provide

Adaptive intelligent personalised learning (AIPL) environment

As individuals the ideal learning scenario would be a learning environment tailored just for how we like to learn, personalised to our requirements. This has previously been almost inconceivable given the complexities of learning, the constraints within the environments in which we teach, and the need for global repositories of knowledge to facilitate this process. Whilst it is still not necessarily achievable in its full sense this research project represents a path towards this ideal.In this thesis, findings from research into the development of a model (the Adaptive Intelligent Personalised Learning (AIPL)), the creation of a prototype implementation of a system designed around this model (the AIPL environment) and the construction of a suite of intelligent algorithms (Personalised Adaptive Filtering System (PAFS)) for personalised learning are presented and evaluated. A mixed methods approach is used in the evaluation of the AIPL environment. The AIPL model is built on the premise of an ideal system being one which does not just consider the individual but also considers groupings of likeminded individuals and their power to influence learner choice. The results show that: (1) There is a positive correlation for using group-learning-paradigms. (2) Using personalisation as a learning aid can help to facilitate individual learning and encourage learning on-line. (3) Using learning styles as a way of identifying and categorising the individuals can improve their on-line learning experience. (4) Using Adaptive Information Retrieval techniques linked to group-learning-paradigms can reduce and improve the problem of mis-matching. A number of approaches for further work to extend and expand upon the work presented are highlighted at the end of the Thesis

Doctor of Philosophy

dissertationIn the past few years, we have seen a tremendous increase in digital data being generated. By 2011, storage vendors had shipped 905 PB of purpose-built backup appliances. By 2013, the number of objects stored in Amazon S3 had reached 2 trillion. Facebook had stored 20 PB of photos by 2010. All of these require an efficient storage solution. To improve space efficiency, compression and deduplication are being widely used. Compression works by identifying repeated strings and replacing them with more compact encodings while deduplication partitions data into fixed-size or variable-size chunks and removes duplicate blocks. While we have seen great improvements in space efficiency from these two approaches, there are still some limitations. First, traditional compressors are limited in their ability to detect redundancy across a large range since they search for redundant data in a fine-grain level (string level). For deduplication, metadata embedded in an input file changes more frequently, and this introduces more unnecessary unique chunks, leading to poor deduplication. Cloud storage systems suffer from unpredictable and inefficient performance because of interference among different types of workloads. This dissertation proposes techniques to improve the effectiveness of traditional compressors and deduplication in improving space efficiency, and a new IO scheduling algorithm to improve performance predictability and efficiency for cloud storage systems. The common idea is to utilize similarity. To improve the effectiveness of compression and deduplication, similarity in content is used to transform an input file into a compression- or deduplication-friendly format. We propose Migratory Compression, a generic data transformation that identifies similar data in a coarse-grain level (block level) and then groups similar blocks together. It can be used as a preprocessing stage for any traditional compressor. We find metadata have a huge impact in reducing the benefit of deduplication. To isolate the impact from metadata, we propose to separate metadata from data. Three approaches are presented for use cases with different constrains. For the commonly used tar format, we propose Migratory Tar: a data transformation and also a new tar format that deduplicates better. We also present a case study where we use deduplication to reduce storage consumption for storing disk images, while at the same time achieving high performance in image deployment. Finally, we apply the same principle of utilizing similarity in IO scheduling to prevent interference between random and sequential workloads, leading to efficient, consistent, and predictable performance for sequential workloads and a high disk utilization

Bridging the Gap between Application and Solid-State-Drives

Data storage is one of the important and often critical parts of the computing system in terms of performance, cost, reliability, and energy. Numerous new memory technologies, such as NAND flash, phase change memory (PCM), magnetic RAM (STT-RAM) and Memristor, have emerged recently. Many of them have already entered the production system. Traditional storage optimization and caching algorithms are far from optimal because storage I/Os do not show simple locality. To provide optimal storage we need accurate predictions of I/O behavior. However, the workloads are increasingly dynamic and diverse, making the long and short time I/O prediction challenge. Because of the evolution of the storage technologies and the increasing diversity of workloads, the storage software is becoming more and more complex. For example, Flash Translation Layer (FTL) is added for NAND-flash based Solid State Disks (NAND-SSDs). However, it introduces overhead such as address translation delay and garbage collection costs. There are many recent studies aim to address the overhead. Unfortunately, there is no one-size-fits-all solution due to the variety of workloads. Despite rapidly evolving in storage technologies, the increasing heterogeneity and diversity in machines and workloads coupled with the continued data explosion exacerbate the gap between computing and storage speeds. In this dissertation, we improve the data storage performance from both top-down and bottom-up approach. First, we will investigate exposing the storage level parallelism so that applications can avoid I/O contentions and workloads skew when scheduling the jobs. Second, we will study how architecture aware task scheduling can improve the performance of the application when PCM based NVRAM are equipped. Third, we will develop an I/O correlation aware flash translation layer for NAND-flash based Solid State Disks. Fourth, we will build a DRAM-based correlation aware FTL emulator and study the performance in various filesystems

Vulnerability to crop-raiding: an interdisciplinary investigation in Loango National Park, Gabon

Human-wildlife conflict is a major threat to long-term wildlife survival and to subsistence communities’ livelihoods in developing countries, particularly near protected areas. In this thesis, I use an interdisciplinary approach based on a threefold vulnerability framework to examine the specific issue of crop-raiding in Loango National Park, Gabon. First, I investigate the context of conservation at the study site, and how this limits, or intensifies, conflict over wildlife. People in Loango have an understanding of sustainability that shares common ground with modern conservation principles. However, local people resent and resist current conservation practices, which exclude local communities, threaten local environmental entitlement and thus exacerbate institutional vulnerability to crop-raiding. Next, I examine biophysical vulnerability to crop-raiding and find that elephants cause the most crop damage in Loango. Crop-raiding by elephants, when considered at the scale of the study site, follows a seasonal pattern which probably results from elephants’ use of water points. However, field isolation and surrounding forest types render some fields more vulnerable than others. Farmers use diverse deterrent methods to limit raids, but none seem effective. The lack of efficacy of deterrents stems from lack of access to labour, driven by rural exodus, which prevents their successful implementation. State mitigation strategies exist but are inadequate and ineffective. Demographic changes also make farmers increasingly vulnerable to poverty, which ultimately increases social vulnerability to crop-raiding. The consequences of crop-raiding, which span from increased food and economic insecurity to social marginalisation, create a negative spiral of vulnerability to poverty and to crop-raiding. Ultimately, spatial and social isolation are the main factors driving vulnerability to crop-raiding in Loango, and both need to be addressed. Vulnerability proves to be a very appropriate analytical framework for the holistic investigation of crop-raiding, and I recommend its use in future research on human-wildlife conflicts and in conservation