EVENODD: An Efficient Scheme for Tolerating Double Disk Failures in RAID Architectures

We present a novel method, that we call EVENODD, for tolerating up to two disk failures in RAID architectures. EVENODD employs the addition of only two redundant disks and consists of simple exclusive-OR computations. This redundant storage is optimal, in the sense that two failed disks cannot be retrieved with less than two redundant disks. A major advantage of EVENODD is that it only requires parity hardware, which is typically present in standard RAID-5 controllers. Hence, EVENODD can be implemented on standard RAID-5 controllers without any hardware changes. The most commonly used scheme that employes optimal redundant storage (i.e., two extra disks) is based on Reed-Solomon (RS) error-correcting codes. This scheme requires computation over finite fields and results in a more complex implementation. For example, we show that the complexity of implementing EVENODD in a disk array with 15 disks is about 50% of the one required when using the RS scheme. The new scheme is not limited to RAID architectures: it can be used in any system requiring large symbols and relatively short codes, for instance, in multitrack magnetic recording. To this end, we also present a decoding algorithm for one column (track) in error

Improved Cauchy Reed-Solomon Codes for Cloud Data Retrieval and Secured Data Storage using Role-Based Cryptographic Access and forensic investigation

Doling out client consent strategies to PC frameworks presents a huge test in guaranteeing legitimate approval, especially with the development of open frameworks and scattered stages like the cloud.  RBAC  has turned into a broadly involved strategy in cloud server applications because of its versatility. Granting access to cloud-stored data for investigating potential wrongdoings is crucial in computer forensic investigations. In cases where the cloud service provider's reliability is questionable, maintaining data confidentiality and establishing an efficient procedure for revoking access upon credential expiration is essential. As storage systems expand across vast networks, frequent component failures require stronger fault tolerance measures. Our work secure data-sharing system combines role (Authorized) based access control and AES encryption technology to provide safe key distribution and data sharing for dynamic groups. Data recovery entails protecting data dispersed over distributed systems by storing duplicate data and applying the erasure code technique. Erasure coding strategies, like Reed-Solomon codes, guarantee disc failure robustness while cutting down on data storage expenses dramatically. They do, however, also result in longer access times and more expensive repairs. Consequently, there has been a great deal of interest in academic and business circles for the investigation of novel coding strategies for cloud storage systems. The objective of this study is to present a novel coding method that utilizes the intricate Cauchy matrix in order to improve Reed-Solomon coding efficiency and strengthen fault tolerance

TOWARDS DIGITAL TWINS FOR OPTIMIZING METRICS IN DISTRIBUTED STORAGE SYSTEMS - A REVIEW

With the exponential data growth, there is a crucial need for highly available, scalable, reliable, and cost-effective Distributed Storage Systems (DSSs). To ensure such efficient and fault tolerant systems, replication and erasure coding techniques are typically used in traditional DSSs. However, these systems are prone to failure and require different failure prevention and recovery algorithms. Failure recovery of DSS and data reconstruction techniques take into consideration different performance metrics optimization in the recovery process. In this paper, DSS performance metrics are introduced. Several recent papers related to adopting erasure coding in DSSs are surveyed together with highlighting related performance metrics introduced in the context of these papers. Next, we present recent literature where Digital Twins (DTs) are involved in monitoring DSSs and assisting the data center managers in intelligent decision-making. Finally, important open issues are identified to inspire future studies for fully efficient DSSs

An erasure-resilient and compute-efficient coding scheme for storage applications

Driven by rapid technological advancements, the amount of data that is created, captured, communicated, and stored worldwide has grown exponentially over the past decades. Along with this development it has become critical for many disciplines of science and business to being able to gather and analyze large amounts of data. The sheer volume of the data often exceeds the capabilities of classical storage systems, with the result that current large-scale storage systems are highly distributed and are comprised of a high number of individual storage components. As with any other electronic device, the reliability of storage hardware is governed by certain probability distributions, which in turn are influenced by the physical processes utilized to store the information. The traditional way to deal with the inherent unreliability of combined storage systems is to replicate the data several times. Another popular approach to achieve failure tolerance is to calculate the block-wise parity in one or more dimensions. With better understanding of the different failure modes of storage components, it has become evident that sophisticated high-level error detection and correction techniques are indispensable for the ever-growing distributed systems. The utilization of powerful cyclic error-correcting codes, however, comes with a high computational penalty, since the required operations over finite fields do not map very well onto current commodity processors. This thesis introduces a versatile coding scheme with fully adjustable fault-tolerance that is tailored specifically to modern processor architectures. To reduce stress on the memory subsystem the conventional table-based algorithm for multiplication over finite fields has been replaced with a polynomial version. This arithmetically intense algorithm is better suited to the wide SIMD units of the currently available general purpose processors, but also displays significant benefits when used with modern many-core accelerator devices (for instance the popular general purpose graphics processing units). A CPU implementation using SSE and a GPU version using CUDA are presented. The performance of the multiplication depends on the distribution of the polynomial coefficients in the finite field elements. This property has been used to create suitable matrices that generate a linear systematic erasure-correcting code which shows a significantly increased multiplication performance for the relevant matrix elements. Several approaches to obtain the optimized generator matrices are elaborated and their implications are discussed. A Monte-Carlo-based construction method allows it to influence the specific shape of the generator matrices and thus to adapt them to special storage and archiving workloads. Extensive benchmarks on CPU and GPU demonstrate the superior performance and the future application scenarios of this novel erasure-resilient coding scheme

Erasure Coding Optimization for Data Storage: Acceleration Techniques and Delayed Parities Generation

Various techniques have been proposed in the literature to improve erasure code computation efficiency, including optimizing bitmatrix design and computation schedule, common XOR operation reduction, caching management techniques, and vectorization techniques. These techniques were largely proposed individually, and in this work, we seek to use them jointly. To accomplish this task, these techniques need to be thoroughly evaluated individually, and their relation better understood. Building on extensive testing, we develop methods to systematically optimize the computation chain together with the underlying bitmatrix. This led to a simple design approach of optimizing the bitmatrix by minimizing a weighted computation cost function, and also a straightforward coding procedure: follow a computation schedule produced from the optimized bitmatrix to apply XOR-level vectorization. This procedure provides better performances than most existing techniques (e.g., those used in ISA-L and Jerasure libraries), and sometimes can even compete against well-known but less general codes such as EVENODD, RDP, and STAR codes. One particularly important observation is that vectorizing the XOR operations is a better choice than directly vectorizing finite field operations, not only because of the flexibility in choosing finite field size and the better encoding throughput, but also its minimal migration efforts onto newer CPUs. A delayed parity generation technique for maximum distance separable (MDS) storage codes is proposed as well, for two possible applications: the first is to improve the write-speed during data intake where only a subset of the parities are initially produced and stored into the system, and the rest can be produced from the stored data during a later time of lower system load; the second is to provide better adaptivity, where a lower number of parities can be chosen initially in a storage system, and more parities can be produced when the existing ones are not sufficient to guarantee the needed reliability or performance. In both applications, it is important to reduce the data access as much as possible during the delayed parity generation procedure. For this purpose, we first identify the fundamental limit for delayed parity generation through a connection to the well-known multicast network coding problem, then provide an explicit and low-complexity code transformation that is applicable on any MDS codes to obtain optimal codes. The problem we consider is closely related to the regenerating code problem, however the proposed codes are much simpler and have a much smaller subpacketization factor than regenerating codes, and thus our result in fact shows that blindly adopting regenerating codes in these two settings is unnecessary and wasteful. Moreover, two aspects of this approach is addressed. The first is to optimize the underlying coding matrix, and the second is to understand its behavior in a system setting. For the former, we generalize the existing approach by allowing more flexibility in the code design, and then optimize the underlying coding matrix in the familiar bitmatrix-based coding framework. For the latter, we construct a prototype system, and conduct tests on a local storage network and on two virtual machine-based setups. In both cases, the results confirm the benefit of delayed parity generation when the system bottleneck is in the communication bandwidth instead of the computation

Techniques for ubiquitous reliable data storage

This portfolio thesis documents the work undertaken by the author under the auspices of the Engineering Doctorate (EngD) programme. The research work undertaken was completed at the sponsoring company, A2E Limited. There is a wide range of products and solutions to meet both commercial and personal electronic storage needs. This work documents research and development over a four year period into algorithms, implementations and product development for novel storage solutions for commercial and personal use. The design work and technical and business objectives were guided by the sponsoring company, A4E limited. This portfolio thesis considers the storage market at the start of the project, the commercial and technical aspects relevant to this market place and describes the development and testing of a RAID 6 algorithm in both hardware and software. The key contributions presented in this portfolio thesis include the implementation of the smallest and fastest FPGA based Reed-Solomon RAID 6 hardware accelerator. We also present the first commercial implementation of a Reed-Solomon RAID 6 intellectual property (IP) block. Both the hardware and software implementation are discussed in detail along with the supporting IP blocks and device drivers. Documentation of the product development stages and additional project work carried out to provide an understanding of product development stages and the requirements placed during such work are also examined. the results of testing and implementation are considered and the performance of the proposed solution is considered along with the commercial viability and success of the project

Hyfs: design and implementation of a reliable file system

Building reliable data storage systems is crucial to any commercial or scientific applications. Modern storage systems are complicated, and they are comprised of many components, from hardware to software. Problems may occur to any component of storage systems and cause data loss. When this kind of failures happens, storage systems cannot continue their data services, which may result in large revenue loss or even catastrophe to enterprises. Therefore, it is critically important to build reliable storage systems to ensure data reliability. In this dissertation, we propose to employ general erasure codes to build a reliable file system, called HyFS. HyFS is a cluster system, which can aggregate distributed storage servers to provide reliable data service. On client side, HyFS is implemented as a native file system so that applications can transparently run on top of HyFS. On server side, HyFS utilizes multiple distributed storage servers to provide highly reliable data service by employing erasure codes. HyFS is able to offer high throughput for either random or sequential file access, which makes HyFS an attractive choice for primary or backup storage systems. This dissertation studies five relevant topics of HyFS. Firstly, it presents several algorithms that can perform encoding operation efficiently for XOR-based erasure codes. Secondly, it discusses an efficient decoding algorithm for RAID-6 erasure codes. This algorithm can recover various types of disk failures. Thirdly, it describes an efficient algorithm to detect and correct errors for the STAR code, which further improves a storage system\u27s reliability. Fourthly, it describes efficient implementations for the arithmetic operations of large finite fields. This is to improve a storage system\u27s security. Lastly and most importantly, it presents the design and implementation of HyFS and evaluates the performance of HyFS