1,177 research outputs found
CRAID: Online RAID upgrades using dynamic hot data reorganization
Current algorithms used to upgrade RAID arrays typically require large amounts of data to be migrated, even those that move only the minimum amount of data required to keep a balanced data load. This paper presents CRAID, a self-optimizing RAID array that performs an online block reorganization of frequently used, long-term accessed data in order to reduce this migration even further. To achieve this objective, CRAID tracks frequently used, long-term data blocks and copies them to a dedicated partition spread across all the disks in the array. When new disks are added, CRAID only needs to extend this process to the new devices to redistribute this partition, thus greatly reducing the overhead of the upgrade process. In addition, the reorganized access patterns within this partition improve the array’s performance, amortizing the copy overhead and allowing CRAID to offer a performance competitive with traditional RAIDs.
We describe CRAID’s motivation and design and we evaluate it by replaying seven real-world workloads including a file server, a web server and a user share. Our experiments show that CRAID can successfully detect hot data variations and begin using new disks as soon as they are added to the array. Also, the usage of a dedicated
partition improves the sequentiality of relevant data access, which amortizes the cost of reorganizations. Finally, we prove that a full-HDD CRAID array with a small distributed partition (<1.28% per disk) can compete in performance with an ideally restriped RAID-5 and a hybrid RAID-5 with a small SSD cache.Peer ReviewedPostprint (published version
Storage Virtualization Promises Agility in the Data Center
Data storage and protection has moved to the forefront of Information Technology solutions because the business value of data has gained in rank and importance in the world of internet commerce. Modern business models are built around instant and continuous data availability and they would not be able to function without this quality. This level of data availability requires data storage technologies to be of increased flexibility and higher performance. However the more sophisticated technologies pose a greater challenge to the architects of data storage solutions who are required to evaluate products of much higher complexity and administrators who need to manage and monitor these installations. New tool sets are required to leverage the promise of the storage virtualization technologies and extract their full potential for an agile data center. New tool sets for storage virtualization will bring the IT organizations into the position of data service provider for the business groups
Achieving High Reliability and Efficiency in Maintaining Large-Scale Storage Systems through Optimal Resource Provisioning and Data Placement
With the explosive increase in the amount of data being generated by various applications, large-scale distributed and parallel storage systems have become common data storage solutions and been widely deployed and utilized in both industry and academia. While these high performance storage systems significantly accelerate the data storage and retrieval, they also bring some critical issues in system maintenance and management. In this dissertation, I propose three methodologies to address three of these critical issues.
First, I develop an optimal resource management and spare provisioning model to minimize the impact brought by component failures and ensure a highly operational experience in maintaining large-scale storage systems. Second, in order to cost-effectively integrate solid-state drives (SSD) into large-scale storage systems, I design a holistic algorithm which can adaptively predict the popularity of data objects by leveraging temporal locality in their access pattern and adjust their placement among solid-state drives and regular hard disk drives so that the data access throughput as well as the storage space efficiency of the large-scale heterogeneous storage systems can be improved. Finally, I propose a new checkpoint placement optimization model which can maximize the computation efficiency of large-scale scientific applications while guarantee the endurance requirements of the SSD-based burst buffer in high performance hierarchical storage systems. All these models and algorithms are validated through extensive evaluation using data collected from deployed large-scale storage systems and the evaluation results demonstrate our models and algorithms can significantly improve the reliability and efficiency of large-scale distributed and parallel storage systems
High availability using virtualization
High availability has always been one of the main problems for a data center.
Till now high availability was achieved by host per host redundancy, a highly
expensive method in terms of hardware and human costs. A new approach to the
problem can be offered by virtualization. Using virtualization, it is possible
to achieve a redundancy system for all the services running on a data center.
This new approach to high availability allows to share the running virtual
machines over the servers up and running, by exploiting the features of the
virtualization layer: start, stop and move virtual machines between physical
hosts. The system (3RC) is based on a finite state machine with hysteresis,
providing the possibility to restart each virtual machine over any physical
host, or reinstall it from scratch. A complete infrastructure has been
developed to install operating system and middleware in a few minutes. To
virtualize the main servers of a data center, a new procedure has been
developed to migrate physical to virtual hosts. The whole Grid data center
SNS-PISA is running at the moment in virtual environment under the high
availability system. As extension of the 3RC architecture, several storage
solutions have been tested to store and centralize all the virtual disks, from
NAS to SAN, to grant data safety and access from everywhere. Exploiting
virtualization and ability to automatically reinstall a host, we provide a sort
of host on-demand, where the action on a virtual machine is performed only when
a disaster occurs.Comment: PhD Thesis in Information Technology Engineering: Electronics,
Computer Science, Telecommunications, pp. 94, University of Pisa [Italy
Facilitating Data Driven Research Through a Hardware- and Software-Based Cyberinfrastructure Architecture
Cyberinfrastructure is the backbone of research and modern industry. As such, to have an environment conducive to research advancements, cyberinfrastructure must be well maintained and accessible by all researchers. Presented in this thesis is a method of centralizing aspects of cyberinfrastructure to allow for ease of collaboration and data management by researchers without requiring these researchers to manage the involved systems themselves. This centralized architecture includes dedicated machines for data transfers, a cluster designed to run microservices surrounding the method, a dashboard for performance and health monitoring, and network telemetry collection. As system administrators are responsible for maintaining the systems in place, a user study was conducted to assess the functionality of the dashboard they would utilize to receive alerts from and utilize to quickly gauge the status of involved hardware. This thesis aims to provide a template for deploying centralized data transfer cyberinfrastructure and a manual for utilizing these systems to support data driven research
- …