An Exploration of Optimization Algorithms and Heuristics for the Creation of Encoding and Decoding Schedules in Erasure Coding

Erasure codes are employed by disk systems to tolerate failures. They are typically characterized by bit-matrices that are used for encoding and decoding. The efficiency of an erasure code using a bit-matrix is directly related to the number of exclusive-or (XOR) operations required during the encoding process. Thus, a problem within the field of erasure coding is how to schedule the XOR operations for any given bit-matrix so that the fewest number of XOR operations are required. This paper develops an algorithm for finding the optimum solution and analyzes the performance of two known heuristics on a set of encoding matrices

An Exploration of Optimization Algorithms and Heuristics for the Creation of Encoding and Decoding Schedules in Erasure Coding

Erasure codes are employed by disk systems to tolerate failures. They are typically characterized by bit-matrices that are used for encoding and decoding. The efficiency of an erasure code using a bit-matrix is directly related to the number of exclusive-or (XOR) operations required during the encoding process. Thus, a problem within the field of erasure coding is how to schedule the XOR operations for any given bit-matrix so that the fewest number of XOR operations are required. This paper develops an algorithm for finding the optimum solution and analyzes the performance of two known heuristics on a set of encoding matrices

Content-access QoS in peer-to-peer networks using a fast MDS erasure code

This paper describes an enhancement of content access Quality of Service in peer to peer (P2P) networks. The main idea is to use an erasure code to distribute the information over the peers. This distribution increases the users’ choice on disseminated encoded data and therefore statistically enhances the overall throughput of the transfer. A performance evaluation based on an original model using the results of a measurement campaign of sequential and parallel downloads in a real P2P network over Internet is presented. Based on a bandwidth distribution, statistical content-access QoS are guaranteed in function of both the content replication level in the network and the file dissemination strategies. A simple application in the context of media streaming is proposed. Finally, the constraints on the erasure code related to the proposed system are analysed and a new fast MDS erasure code is proposed, implemented and evaluated

Improved Cauchy Reed-Solomon Codes for Cloud Data Retrieval and Secured Data Storage using Role-Based Cryptographic Access and forensic investigation

Doling out client consent strategies to PC frameworks presents a huge test in guaranteeing legitimate approval, especially with the development of open frameworks and scattered stages like the cloud.  RBAC  has turned into a broadly involved strategy in cloud server applications because of its versatility. Granting access to cloud-stored data for investigating potential wrongdoings is crucial in computer forensic investigations. In cases where the cloud service provider's reliability is questionable, maintaining data confidentiality and establishing an efficient procedure for revoking access upon credential expiration is essential. As storage systems expand across vast networks, frequent component failures require stronger fault tolerance measures. Our work secure data-sharing system combines role (Authorized) based access control and AES encryption technology to provide safe key distribution and data sharing for dynamic groups. Data recovery entails protecting data dispersed over distributed systems by storing duplicate data and applying the erasure code technique. Erasure coding strategies, like Reed-Solomon codes, guarantee disc failure robustness while cutting down on data storage expenses dramatically. They do, however, also result in longer access times and more expensive repairs. Consequently, there has been a great deal of interest in academic and business circles for the investigation of novel coding strategies for cloud storage systems. The objective of this study is to present a novel coding method that utilizes the intricate Cauchy matrix in order to improve Reed-Solomon coding efficiency and strengthen fault tolerance

Enhanced Cauchy Matrix Reed-Solomon Codes and Role-Based Cryptographic Data Access for Data Recovery and Security in Cloud Environment

In computer systems ensuring proper authorization is a significant challenge, particularly with the rise of open systems and dispersed platforms like the cloud. Role-Based Access Control (RBAC) has been widely adopted in cloud server applications due to its popularity and versatility. When granting authorization access to data stored in the cloud for collecting evidence against offenders, computer forensic investigations play a crucial role. As cloud service providers may not always be reliable, data confidentiality should be ensured within the system. Additionally, a proper revocation procedure is essential for managing users whose credentials have expired.  With the increasing scale and distribution of storage systems, component failures have become more common, making fault tolerance a critical concern. In response to this, a secure data-sharing system has been developed, enabling secure key distribution and data sharing for dynamic groups using role-based access control and AES encryption technology. Data recovery involves storing duplicate data to withstand a certain level of data loss. To secure data across distributed systems, the erasure code method is employed. Erasure coding techniques, such as Reed-Solomon codes, have the potential to significantly reduce data storage costs while maintaining resilience against disk failures. In light of this, there is a growing interest from academia and the corporate world in developing innovative coding techniques for cloud storage systems. The research goal is to create a new coding scheme that enhances the efficiency of Reed-Solomon coding using the sophisticated Cauchy matrix to achieve fault toleranc

Polynomial Ring Transforms for Efficient XOR-based Erasure Coding

The complexity of software implementations of MDS erasure codes mainly depends on the efficiency of the finite field operations implementation. In this paper, we propose a method to reduce the complexity of the finite field multiplication by using simple transforms between a field and a ring to perform the multiplication in a ring. We show that moving to a ring reduces the complexity of the operations. Then, we show that this construction allows the use of simple scheduling to reduce the number of operation

Efficient data reliability management of cloud storage systems for big data applications

Cloud service providers are consistently striving to provide efficient and reliable service, to their client's Big Data storage need. Replication is a simple and flexible method to ensure reliability and availability of data. However, it is not an efficient solution for Big Data since it always scales in terabytes and petabytes. Hence erasure coding is gaining traction despite its shortcomings. Deploying erasure coding in cloud storage confronts several challenges like encoding/decoding complexity, load balancing, exponential resource consumption due to data repair and read latency. This thesis has addressed many challenges among them. Even though data durability and availability should not be compromised for any reason, client's requirements on read performance (access latency) may vary with the nature of data and its access pattern behaviour. Access latency is one of the important metrics and latency acceptance range can be recorded in the client's SLA. Several proactive recovery methods, for erasure codes are proposed in this research, to reduce resource consumption due to recovery. Also, a novel cache based solution is proposed to mitigate the access latency issue of erasure coding

Erasure Coding Optimization for Data Storage: Acceleration Techniques and Delayed Parities Generation

Various techniques have been proposed in the literature to improve erasure code computation efficiency, including optimizing bitmatrix design and computation schedule, common XOR operation reduction, caching management techniques, and vectorization techniques. These techniques were largely proposed individually, and in this work, we seek to use them jointly. To accomplish this task, these techniques need to be thoroughly evaluated individually, and their relation better understood. Building on extensive testing, we develop methods to systematically optimize the computation chain together with the underlying bitmatrix. This led to a simple design approach of optimizing the bitmatrix by minimizing a weighted computation cost function, and also a straightforward coding procedure: follow a computation schedule produced from the optimized bitmatrix to apply XOR-level vectorization. This procedure provides better performances than most existing techniques (e.g., those used in ISA-L and Jerasure libraries), and sometimes can even compete against well-known but less general codes such as EVENODD, RDP, and STAR codes. One particularly important observation is that vectorizing the XOR operations is a better choice than directly vectorizing finite field operations, not only because of the flexibility in choosing finite field size and the better encoding throughput, but also its minimal migration efforts onto newer CPUs. A delayed parity generation technique for maximum distance separable (MDS) storage codes is proposed as well, for two possible applications: the first is to improve the write-speed during data intake where only a subset of the parities are initially produced and stored into the system, and the rest can be produced from the stored data during a later time of lower system load; the second is to provide better adaptivity, where a lower number of parities can be chosen initially in a storage system, and more parities can be produced when the existing ones are not sufficient to guarantee the needed reliability or performance. In both applications, it is important to reduce the data access as much as possible during the delayed parity generation procedure. For this purpose, we first identify the fundamental limit for delayed parity generation through a connection to the well-known multicast network coding problem, then provide an explicit and low-complexity code transformation that is applicable on any MDS codes to obtain optimal codes. The problem we consider is closely related to the regenerating code problem, however the proposed codes are much simpler and have a much smaller subpacketization factor than regenerating codes, and thus our result in fact shows that blindly adopting regenerating codes in these two settings is unnecessary and wasteful. Moreover, two aspects of this approach is addressed. The first is to optimize the underlying coding matrix, and the second is to understand its behavior in a system setting. For the former, we generalize the existing approach by allowing more flexibility in the code design, and then optimize the underlying coding matrix in the familiar bitmatrix-based coding framework. For the latter, we construct a prototype system, and conduct tests on a local storage network and on two virtual machine-based setups. In both cases, the results confirm the benefit of delayed parity generation when the system bottleneck is in the communication bandwidth instead of the computation