A differentiated proposal of three dimension i/o performance characterization model focusing on storage environments

The I/O bottleneck remains a central issue in high-performance environments. Cloud computing, high-performance computing (HPC) and big data environments share many underneath difficulties to deliver data at a desirable time rate requested by high-performance applications. This increases the possibility of creating bottlenecks throughout the application feeding process by bottom hardware devices located in the storage system layer. In the last years, many researchers have been proposed solutions to improve the I/O architecture considering different approaches. Some of them take advantage of hardware devices while others focus on a sophisticated software approach. However, due to the complexity of dealing with high-performance environments, creating solutions to improve I/O performance in both software and hardware is challenging and gives researchers many opportunities. Classifying these improvements in different dimensions allows researchers to understand how these improvements have been built over the years and how it progresses. In addition, it also allows future efforts to be directed to research topics that have developed at a lower rate, balancing the general development process. This research present a three-dimension characterization model for classifying research works on I/O performance improvements for large scale storage computing facilities. This classification model can also be used as a guideline framework to summarize researches providing an overview of the actual scenario. We also used the proposed model to perform a systematic literature mapping that covered ten years of research on I/O performance improvements in storage environments. This study classified hundreds of distinct researches identifying which were the hardware, software, and storage systems that received more attention over the years, which were the most researches proposals elements and where these elements were evaluated. In order to justify the importance of this model and the development of solutions that targets I/O performance improvements, we evaluated a subset of these improvements using a a real and complete experimentation environment, the Grid5000. Analysis over different scenarios using a synthetic I/O benchmark demonstrates how the throughput and latency parameters behaves when performing different I/O operations using distinct storage technologies and approaches.O gargalo de E/S continua sendo um problema central em ambientes de alto desempenho. Os ambientes de computação em nuvem, computação de alto desempenho (HPC) e big data compartilham muitas dificuldades para fornecer dados em uma taxa de tempo desejável solicitada por aplicações de alto desempenho. Isso aumenta a possibilidade de criar gargalos em todo o processo de alimentação de aplicativos pelos dispositivos de hardware inferiores localizados na camada do sistema de armazenamento. Nos últimos anos, muitos pesquisadores propuseram soluções para melhorar a arquitetura de E/S considerando diferentes abordagens. Alguns deles aproveitam os dispositivos de hardware, enquanto outros se concentram em uma abordagem sofisticada de software. No entanto, devido à complexidade de lidar com ambientes de alto desempenho, criar soluções para melhorar o desempenho de E/S em software e hardware é um desafio e oferece aos pesquisadores muitas oportunidades. A classificação dessas melhorias em diferentes dimensões permite que os pesquisadores entendam como essas melhorias foram construídas ao longo dos anos e como elas progridem. Além disso, também permite que futuros esforços sejam direcionados para tópicos de pesquisa que se desenvolveram em menor proporção, equilibrando o processo geral de desenvolvimento. Esta pesquisa apresenta um modelo de caracterização tridimensional para classificar trabalhos de pesquisa sobre melhorias de desempenho de E/S para instalações de computação de armazenamento em larga escala. Esse modelo de classificação também pode ser usado como uma estrutura de diretrizes para resumir as pesquisas, fornecendo uma visão geral do cenário real. Também usamos o modelo proposto para realizar um mapeamento sistemático da literatura que abrangeu dez anos de pesquisa sobre melhorias no desempenho de E/S em ambientes de armazenamento. Este estudo classificou centenas de pesquisas distintas, identificando quais eram os dispositivos de hardware, software e sistemas de armazenamento que receberam mais atenção ao longo dos anos, quais foram os elementos de proposta mais pesquisados e onde esses elementos foram avaliados. Para justificar a importância desse modelo e o desenvolvimento de soluções que visam melhorias no desempenho de E/S, avaliamos um subconjunto dessas melhorias usando um ambiente de experimentação real e completo, o Grid5000. Análises em cenários diferentes usando um benchmark de E/S sintética demonstra como os parâmetros de vazão e latência se comportam ao executar diferentes operações de E/S usando tecnologias e abordagens distintas de armazenamento

RAID Organizations for Improved Reliability and Performance: A Not Entirely Unbiased Tutorial (1st revision)

RAID proposal advocated replacing large disks with arrays of PC disks, but as the capacity of small disks increased 100-fold in 1990s the production of large disks was discontinued. Storage dependability is increased via replication or erasure coding. Cloud storage providers store multiple copies of data obviating for need for further redundancy. Varitaions of RAID based on local recovery codes, partial MDS reduce recovery cost. NAND flash Solid State Disks - SSDs have low latency and high bandwidth, are more reliable, consume less power and have a lower TCO than Hard Disk Drives, which are more viable for hyperscalers.Comment: Submitted to ACM Computing Surveys. arXiv admin note: substantial text overlap with arXiv:2306.0876

Letter from the Special Issue Editor

Editorial work for DEBULL on a special issue on data management on Storage Class Memory (SCM) technologies

Analysis of material efficiency aspects of personal computers product group

This report has been developed within the project ‘Technical support for environmental footprinting, material efficiency in product policy and the European Platform on Life Cycle Assessment’ (LCA) (2013-2017) funded by the Directorate-General for Environment. The report summarises the findings of the analysis of material-efficiency aspects of the personal-computer (PC) product group, namely durability, reusability, reparability and recyclability. It also aims to identify material-efficiency aspects which can be relevant for the current revision of the Ecodesign Regulation (EU) No 617/2013. Special focus was given to the content of EU critical raw materials (CRMs) ( ) in computers and computer components, and how to increase the efficient use of these materials, including material savings thanks to reuse and repair and recovery of the products at end of life. The analysis has been based mainly on the REAPro method ( ) developed by the Joint Research Centre for the material-efficiency assessment of products. This work has been carried out in the period June 2016-September 2017, in parallel with the development of The preparatory study on the review of Regulation 617/2013 (Lot 3) — computers and computer servers led by Viegand Maagøe and Vlaamse Instelling voor Technologisch Onderzoek NV (VITO) (2017) ( ). During this period, close communication was maintained with the authors of the preparatory study. This allowed ensuring consistency between input data and assumptions of the two studies. Moreover, outcomes of the present research were used as scientific basis for the preparatory study for the analysis of material-efficiency aspects for computers. The research has been differentiated as far as possible for different types of computers (i.e. tablet, notebooks and desktop computers). The report starts with the analysis of the technical and scientific background relevant for material-efficiency aspects of computers, such as market sales, expected lifetime, bill of materials, and a focus on the content of CRMs (especially cobalt in batteries, rare earths including neodymium in hard disk drives and palladium in printed circuit boards). Successively the report analyses the current practices for repair, reuse and recycling of computers. Based on results available from the literature, material efficiency of the product group has the potential to be improved, in particular the lifetime extension. The residence time ( ) of IT equipment put on the market in 2000 versus 2010 generally declined by approximately 10 % (Huisman et al., 2012), while consumers expressed their preference for durable goods, lasting considerably longer than they are typically used (Wieser and Tröger, 2016). Design barriers (such as difficulties for the disassembly of certain components or for their processing for data sanitisation) can hinder the repair and the reuse of products. Malfunction and accident rates are not negligible (IDC, 2016, 2010; SquareTrade, 2009) and difficulties in repair may bring damaged products to be discarded even if still functioning. Once a computer reaches the end of its useful life, it is addressed to ‘waste of electrical and electronic equipment’ (WEEE) recycling plants. Recycling of computers is usually based on a combination of manual dismantling of certain components (mainly components containing hazardous substances or valuable materials, e.g. batteries, printed circuit boards, display panels, data-storage components), followed by mechanical processing including shredding. The recycling of traditional desktop computers is perceived as non-problematic by recyclers, with the exception of some miniaturised new models (i.e. mini desktop computers), which still are not found in recycling plants and which could present some difficulties for the extraction of printed circuit boards and batteries (if present). The design of notebooks and tablets can originate some difficulties for the dismantling of batteries, especially for computers with compact design. Recycling of plastics from computers of all types is generally challenging due to the large use of different plastics with additives, such as flame retardants. According to all the interviewed recyclers, recycling of WEEE plastics with flame retardant is very poor or null with current technologies. Building on this analysis, the report then focuses on possible actions to improve material efficiency in computers, namely measures to improve (a) waste prevention, (b) repair and reuse and (c) design for recycling. The possible actions identified are listed hereinafter. (a) Waste prevention a.1 Implementation of dedicated functionality ( ) for the optimisation of the lifetime of batteries in notebooks: the lifetime of batteries could be extended by systematically implementing a preinstalled functionality on notebooks, which makes it possible to optimise the state of charge (SoC) of the battery when the device is used in grid operation (stationary). By preventing the battery remaining at full load when the notebook is in grid operation, the lifetime of batteries can be potentially extended by up to 50 %. Users could be informed about the existence and characteristics of such a functionality and the potential benefits related to its use. a.2 Decoupling external power supplies (EPS) from personal computers: the provision of information on the EPS specifications and the presence/absence of the EPS in the packaging of notebooks and tablets could facilitate the reuse by the consumer of already-available EPS with suitable characteristics. Such a measure could promote the use of common EPS across different devices, as well as the reuse of already-owned EPS. This would result in a reduction in material consumption for the production of unnecessary power supplies (and related packaging and transport) and overall a reduction of treatment of electronic waste. The International Electrotechnical Commission (IEC) technical specification (TS) 62700, the Standard Institute of Electrical and Electronics Engineers (IEEE) 1823 and Recommendation ITU-T L.1002 can be used to develop standards for the correct definition of connectors and power specifications. a.3 Provision of information about the durability of batteries: the analysis identified the existence of endurance tests suitable for the assessment of the durability of batteries in computers according to existing standards (e.g. EN 61960). The availability of information about these endurance tests could help users to get an indication on the residual capacity of the battery after a predefined number of charge/discharge cycles. Moreover, such information would allow for comparison between different products and potentially push the market towards longer-lasting batteries. a.4 Provision of information about the ‘liquid ingress protection (IP) class’ for personal computers: this can be assessed for a notebook or tablet by performing specific tests, developed according to existing standards (e.g. IEC 60529). Users can be informed about the level of protection of the computer against the ingress of liquids (e.g. dripping water or spraying water or water jets) and in this way prevent one of the most common causes of computer failure. The yearly rate of estimated material saving if dedicated functionality for the optimisation of the lifetime of batteries (a.1) were used ranges from around 2 360 to 5 400 tonnes (t) of different materials per year. About 450 t of cobalt, 100 t of lithium, 210 t of nickel and 730 t of copper could be saved every year. The estimated potential savings of materials when EPS are decoupled from notebooks and tablets (a.2) are in the range 2 300-4 600 t/year (80 % related to the notebook category, and 20 % to tablets). These values can be obtained when 10-20 % of notebooks and tablets are sold without an EPS, as users can reuse already-owned and compatible EPS. Under these conditions, for example, about 190-370 t of copper can be saved every year. This estimate may increase when the same EPS can be used for both notebooks and tablets (at the moment the assessment is based on the assumption that the two product types were kept separated). Further work is needed to assess the potential improvements thanks to the provision of information about the durability of batteries (a.3), and about the ‘liquid-IP class’ (a.4). The former option (a.3) has the potential to boost competition among battery manufacturers, resulting in more durable products. The latter option (a.4) has the potential to reduce computer damage due to liquid spillage, ranked among the most recurrent failure modes. (b) Repair/reuse b.1 and b.2 Provision of information to facilitate computer disassembly: the disassembly of relevant components (such as the display panel, keyboard, data storage, batteries, memory and internal power-supply units) plays a key role to enhance repair and reuse of personal computers. Some actions have therefore been discussed (b.1) to provide professional repair operators with documentation about the sequence of disassembly, extraction, replacement and reassembly operations needed for each relevant component of personal computers, and (b.2) to provide end-users with specific information about the disassembly and replacement of batteries in notebooks and tablets. b.3 Secure data deletion for personal computers: this is the process of deliberately, permanently and irreversibly erasing all traces of existing data from storage media, overwriting the data completely in such a way that access to the original data, or parts of them, becomes infeasible for a given level of effort. Secure data deletion is essential for the security of personal data and to allow the reuse of computers by a different user. Secure data deletion for personal computers can be ensured by means of built-in functionality. A number of existing national standards (HMG IS Standard No 5 (the United Kingdom), DIN 66399 (Germany), NIST 800-88r1 (the United States (US)) can be used as a basis to start standardisation activities on secure data deletion. The estimated potential savings of materials due to the provision of information and tools to facilitate computer disassembly were quantified in the range of 150-620 t/year for mobile computers (notebooks and tablets) within the first 2 years of use, and in the range of 610 2 460 t/year for mobile computers older than 2 years. Secure data deletion of personal computers, instead, is considered a necessary prerequisite to enhance reuse. The need to take action on this is related to policies on privacy and protection of personal data, as the General Data Protection Regulation (EU) 2016/679 and in particular its Article 25 on ‘data protection by design and by default’. Future work is needed to strengthen the analysis, however it was estimated that secure data deletion has the potential to double volume of desktop, notebook and tablet computers reused after the first useful lifetime. (c) Recyclability c.1 Provision of information to facilitate computer dismantling: computers could be designed so that crucial components for material aspects (e.g. content of hazardous substances and/or valuable materials) can be easily identified and extracted in order to be processed by means of specific recycling treatments. Design for dismantling can focus on components listed in Annex VII of the WEEE directive ( ). The ‘ease of dismantling’ can be supported by the provision of relevant information (such as a diagram of the product showing the location of the components, the content of hazardous substances, instructions on the sequence of operations needed to remove these components, including type and number of fastening techniques to be unlocked, and tool(s) required). c.2 Marking of plastic components: although all plastics are theoretically recyclable, in practice the recyclability of plastics in computers is generally low, mainly due to the large amount of different plastic components with flame retardants (FRs) and other additives. Marking of plastic components according to existing standards (e.g. ISO 11469 and ISO 1043 series) can facilitate identification and sorting of plastic components during the manual dismantling steps of the recycling. c.3 FR content: according to all the recyclers interviewed, FRs are a major barrier to plastics recycling. Current mechanical-sorting processes of shredded plastics are characterised by low efficiency, while innovative sorting systems are still at the pilot stage and have been shown to be effective only in certain cases. Therefore, the provision of information on the content of FRs in plastic components is a first step to contribute to the improvement of plastics recycling. Plastics marking (as discussed above) can contribute to the separation of plastics with FRs during the manual dismantling, allowing for their recycling at higher rates (in line with the prescription of IEC/TR 62635, 2015). However, detailed information about FRs content could be given in a more systematised way, for example through the development of specific indexes. These indexes could support recyclers in checking the use of FRs in computers and in developing future processes and technologies suitable for plastics recycling. Moreover, these indexes could support policymakers in monitoring the use of FRs in the products and, in the medium-long term, to promote products that use smaller quantities of FRs. An example of a FR content index is provided in this report. c.4 Battery marks: the identification of the chemistry type of batteries in computers is necessary in order to have efficient identification and sorting, and thus to improve the material efficiency during the recycling. It is proposed to start standardisation activities to establish standard marking symbols for batteries. The examples of the ‘battery-recycle mark’, developed by the Battery Association of Japan (BAJ), and the current standardisation activities for the IEC 62902 (standard marking symbols for batteries with a volume higher than 900 cm3) may be used as references to develop ad hoc standards. The benefits of actions for the design for recycling can be relevant. In particular, the proposed actions should contribute to increase the amounts of materials that will be recycled (6 350-8 900 t/year), in particular plastics (5 950-7 960 t/year of additional plastics), but also metals such as cobalt (55-110 t), copper (240-610 t), rare earths as neodymium and dysprosium (2 7 t) and various precious metals (gold (0.1-0.4 t), palladium (0.1-0.4 t) and silver (2 7 t)). Compared to the amount of materials recycled in the EU (2012 data), these values would represent a recycling increase of 1-2 % for cobalt, 2-5 % for palladium, and 13-50 % for rare earths.JRC.D.3-Land Resource

Improving Data Management and Data Movement Efficiency in Hybrid Storage Systems

University of Minnesota Ph.D. dissertation.July 2017. Major: Computer Science. Advisor: David Du. 1 computer file (PDF); ix, 116 pages.In the big data era, large volumes of data being continuously generated drive the emergence of high performance large capacity storage systems. To reduce the total cost of ownership, storage systems are built in a more composite way with many different types of emerging storage technologies/devices including Storage Class Memory (SCM), Solid State Drives (SSD), Shingle Magnetic Recording (SMR), Hard Disk Drives (HDD), and even across off-premise cloud storage. To make better utilization of each type of storage, industries have provided multi-tier storage through dynamically placing hot data in the faster tiers and cold data in the slower tiers. Data movement happens between devices on one single device and as well as between devices connected via various networks. Toward improving data management and data movement efficiency in such hybrid storage systems, this work makes the following contributions: To bridge the giant semantic gap between applications and modern storage systems, passing a piece of tiny and useful information (I/O access hints) from upper layers to the block storage layer may greatly improve application performance or ease data management in heterogeneous storage systems. We present and develop a generic and flexible framework, called HintStor, to execute and evaluate various I/O access hints on heterogeneous storage systems with minor modifications to the kernel and applications. The design of HintStor contains a new application/user level interface, a file system plugin and a block storage data manager. With HintStor, storage systems composed of various storage devices can perform pre-devised data placement, space reallocation and data migration polices assisted by the added access hints. Each storage device/technology has its own unique price-performance tradeoffs and idiosyncrasies with respect to workload characteristics they prefer to support. To explore the internal access patterns and thus efficiently place data on storage systems with fully connected (i.e., data can move from one device to any other device instead of moving tier by tier) differential pools (each pool consists of storage devices of a particular type), we propose a chunk-level storage-aware workload analyzer framework, simplified as ChewAnalyzer. With ChewAnalzyer, the storage manager can adequately distribute and move the data chunks across different storage pools. To reduce the duplicate content transferred between local storage devices and devices in remote data centers, an inline Network Redundancy Elimination (NRE) process with Content-Defined Chunking (CDC) policy can obtain a higher Redundancy Elimination (RE) ratio but may suffer from a considerably higher computational requirement than fixed-size chunking. We build an inline NRE appliance which incorporates an improved FPGA based scheme to speed up CDC processing. To efficiently utilize the hardware resources, the whole NRE process is handled by a Virtualized NRE (VNRE) controller. The uniqueness of this VNRE that we developed lies in its ability to exploit the redundancy patterns of different TCP flows and customize the chunking process to achieve a higher RE ratio

Feasibility study on the microeconomic impact of enforcement of competition policies on innovation: final report

Following seminal contributions from two of the giants of 20th century economics, Schumpeter and Arrow, the relationship between competition and innovation has long been hotly debated, but the general consensus is that competition, whether for the market or in the market, is an important stimulus to innovation. This provides an important additional justification for competition policy, beyond the static purely price-based perspective. Remarkably however, we know relatively little about how specific competition policy interventions have impacted on firms’ innovation activities. So whilst the impact evaluation literature has made important strides in recent decades in assessing the static gains which have been driven by anti-trust and merger control, there have only been very few studies evaluating the impacts of individual policy decisions in this area. The main objective of this study is to explore whether, and how far, such impact evaluation exercises are feasible for competition and innovation. For this reason DG COMP commissioned a team of academics led by Peter Ormosi at the Centre for Competition Policy, University of East Anglia, to review the existing literature, and to propose a rigorous analytical and methodological framework which can be used to evaluate cases. As an illustration of this framework in action, the study provides a pilot evaluation of the Seagate/Samsung and Western Digital/Hitachi mergers. The findings of this case study prove to be interesting in their own right – shedding some new light on these important mergers. But far more important for present purposes, it establishes that the methodology is viable, albeit with important lessons to be learnt. The objective of this study was to offer a detailed literature review, develop a methodological framework, collect data on three different areas (R&D spending, patents, and product characteristics), and analyse it. Our task was to identify what is feasible, what we can learn in terms of the applied methodology, and also to provide preliminary results on how innovation was affected by the 2012 consolidation of the HDD market

Disturbance attenuation with multi-sensing servo systems for high density storage devices

Ph.DDOCTOR OF PHILOSOPH

Leveraging Non-Volatile Memory in Modern Storage Management Architectures

Non-volatile memory technologies (NVM) introduce a novel class of devices that combine characteristics of both storage and main memory. Like storage, NVM is not only persistent, but also denser and cheaper than DRAM. Like DRAM, NVM is byte-addressable and has lower access latency. In recent years, NVM has gained a lot of attention both in academia and in the data management industry, with views ranging from skepticism to over excitement. Some critics claim that NVM is not cheap enough to replace flash-based SSDs nor is it fast enough to replace DRAM, while others see it simply as a storage device. Supporters of NVM have observed that its low latency and byte-addressability requires radical changes and a complete rewrite of storage management architectures. This thesis takes a moderate stance between these two views. We consider that, while NVM might not replace flash-based SSD or DRAM in the near future, it has the potential to reduce the gap between them. Furthermore, treating NVM as a regular storage media does not fully leverage its byte-addressability and low latency. On the other hand, completely redesigning systems to be NVM-centric is impractical. Proposals that attempt to leverage NVM to simplify storage management result in completely new architectures that face the same challenges that are already well-understood and addressed by the traditional architectures. Therefore, we take three common storage management architectures as a starting point, and propose incremental changes to enable them to better leverage NVM. First, in the context of log-structured merge-trees, we investigate the impact of storing data in NVM, and devise methods to enable small granularity accesses and NVM-aware caching policies. Second, in the context of B+Trees, we propose to extend the buffer pool and describe a technique based on the concept of optimistic consistency to handle corrupted pages in NVM. Third, we employ NVM to enable larger capacity and reduced costs in a index+log key-value store, and combine it with other techniques to build a system that achieves low tail latency. This thesis aims to describe and evaluate these techniques in order to enable storage management architectures to leverage NVM and achieve increased performance and lower costs, without major architectural changes.:1 Introduction 1.1 Non-Volatile Memory 1.2 Challenges 1.3 Non-Volatile Memory & Database Systems 1.4 Contributions and Outline 2 Background 2.1 Non-Volatile Memory 2.1.1 Types of NVM 2.1.2 Access Modes 2.1.3 Byte-addressability and Persistency 2.1.4 Performance 2.2 Related Work 2.3 Case Study: Persistent Tree Structures 2.3.1 Persistent Trees 2.3.2 Evaluation 3 Log-Structured Merge-Trees 3.1 LSM and NVM 3.2 LSM Architecture 3.2.1 LevelDB 3.3 Persistent Memory Environment 3.4 2Q Cache Policy for NVM 3.5 Evaluation 3.5.1 Write Performance 3.5.2 Read Performance 3.5.3 Mixed Workloads 3.6 Additional Case Study: RocksDB 3.6.1 Evaluation 4 B+Trees 4.1 B+Tree and NVM 4.1.1 Category #1: Buffer Extension 4.1.2 Category #2: DRAM Buffered Access 4.1.3 Category #3: Persistent Trees 4.2 Persistent Buffer Pool with Optimistic Consistency 4.2.1 Architecture and Assumptions 4.2.2 Embracing Corruption 4.3 Detecting Corruption 4.3.1 Embracing Corruption 4.4 Repairing Corruptions 4.5 Performance Evaluation and Expectations 4.5.1 Checksums Overhead 4.5.2 Runtime and Recovery 4.6 Discussion 5 Index+Log Key-Value Stores 5.1 The Case for Tail Latency 5.2 Goals and Overview 5.3 Execution Model 5.3.1 Reactive Systems and Actor Model 5.3.2 Message-Passing Communication 5.3.3 Cooperative Multitasking 5.4 Log-Structured Storage 5.5 Networking 5.6 Implementation Details 5.6.1 NVM Allocation on RStore 5.6.2 Log-Structured Storage and Indexing 5.6.3 Garbage Collection 5.6.4 Logging and Recovery 5.7 Systems Operations 5.8 Evaluation 5.8.1 Methodology 5.8.2 Environment 5.8.3 Other Systems 5.8.4 Throughput Scalability 5.8.5 Tail Latency 5.8.6 Scans 5.8.7 Memory Consumption 5.9 Related Work 6 Conclusion Bibliography A PiBenc