Achieving High Reliability and Efficiency in Maintaining Large-Scale Storage Systems through Optimal Resource Provisioning and Data Placement

With the explosive increase in the amount of data being generated by various applications, large-scale distributed and parallel storage systems have become common data storage solutions and been widely deployed and utilized in both industry and academia. While these high performance storage systems significantly accelerate the data storage and retrieval, they also bring some critical issues in system maintenance and management. In this dissertation, I propose three methodologies to address three of these critical issues. First, I develop an optimal resource management and spare provisioning model to minimize the impact brought by component failures and ensure a highly operational experience in maintaining large-scale storage systems. Second, in order to cost-effectively integrate solid-state drives (SSD) into large-scale storage systems, I design a holistic algorithm which can adaptively predict the popularity of data objects by leveraging temporal locality in their access pattern and adjust their placement among solid-state drives and regular hard disk drives so that the data access throughput as well as the storage space efficiency of the large-scale heterogeneous storage systems can be improved. Finally, I propose a new checkpoint placement optimization model which can maximize the computation efficiency of large-scale scientific applications while guarantee the endurance requirements of the SSD-based burst buffer in high performance hierarchical storage systems. All these models and algorithms are validated through extensive evaluation using data collected from deployed large-scale storage systems and the evaluation results demonstrate our models and algorithms can significantly improve the reliability and efficiency of large-scale distributed and parallel storage systems

A Toolkit for Resilience Evaluation of Land Use Alternatives in a Multifunctional Peri-Urban Landscape

Translating the concept of social-ecological resilience to practical applications in spatial planning remains challenging. The aim of this paper is to contribute to the scientific approach of spatial aspects of social-ecological resilience and adaptive capacity related to bioproductive space, with particular attention to food systems. We define ‘bioproductive space’ as all space providing ecosystem services through primary production processes and includes semi-natural as well as agricultural ecosystems. We argue that bioproductive space is resilient if it continues in delivering similar levels of ecosystem services under changing conditions. A toolkit was developed to explore spatial resilience of bioproductive space. The first stage in the toolkit is a spatially explicit evaluation of various ecosystem services for different land uses. In a second stage, bio-physical and socio-economic drivers or shocks are introduced that can influence the value society attributes to specific ecosystem services. Some of these variations are mostly society driven, e.g. changing bioenergy demand or more restrictive air quality targets. Others are rather driven by biophysical factors, like increasing need for buffering of extreme weather events under the impulse of global change. The third stage of the toolkit takes policy priorities into account. In a final stage, the output of the tool is synthesised by ranking the analysis results for different scnearios and policy priority settings. This toolkit allows spatial planners to explore and evaluate policy decisions against trade-offs between various land use alternatives, while taking ecosystem services into account. The toolkit is applied to a case study to demonstrate its use. Besides the potential for supporting policy makers, the toolkit provides useful feedback for adaptive farm and landscape management

Adaptation of WASH Services Delivery to Climate Change and Other Sources of Risk and Uncertainty

This report urges WASH sector practitioners to take more seriously the threat of climate change and the consequences it could have on their work. By considering climate change within a risk and uncertainty framework, the field can use the multitude of approaches laid out here to adequately protect itself against a range of direct and indirect impacts. Eleven methods and tools for this specific type of risk management are described, including practical advice on how to implement them successfully

Using Permuted States of Validated Simulation to Analyze Conflict Rates in Optimistic Replication

Optimistic replication provides high data availability in the presence of network outages. Although widely deployed, this relaxed consistency model introduces concurrent updates, whose behavior is poorly understood due to the vast state space. This paper introduces the notion of permuted states to eliminate system states that are redundant and unreachable, which can constitute the majority of states (4069 out of 4096 for four replicas). With the aid of permuted states, we are for the first time able to construct analytical models beyond the two-replica case. By examining the analysis for 2 to 4 replicas, we can demystify the process of forming identical conflicts—the most common conflict type at high replication factors. Additionally, we have automated and optimized the generation of permuted states, which allows us to explore higher replication factors (up to 10 replicas) using hybrid techniques. It also allows us to validate our results with existing simulations based on actual replication mechanisms, which previously were analytically validated with only one pair of replicas. Finally, we have discovered that update locality and bimodal access patterns are the primary factors contributing to the formation of identical conflicts

Data-Driven Intelligent Scheduling For Long Running Workloads In Large-Scale Datacenters

Cloud computing is becoming a fundamental facility of society today. Large-scale public or private cloud datacenters spreading millions of servers, as a warehouse-scale computer, are supporting most business of Fortune-500 companies and serving billions of users around the world. Unfortunately, modern industry-wide average datacenter utilization is as low as 6% to 12%. Low utilization not only negatively impacts operational and capital components of cost efficiency, but also becomes the scaling bottleneck due to the limits of electricity delivered by nearby utility. It is critical and challenge to improve multi-resource efficiency for global datacenters. Additionally, with the great commercial success of diverse big data analytics services, enterprise datacenters are evolving to host heterogeneous computation workloads including online web services, batch processing, machine learning, streaming computing, interactive query and graph computation on shared clusters. Most of them are long-running workloads that leverage long-lived containers to execute tasks. We concluded datacenter resource scheduling works over last 15 years. Most previous works are designed to maximize the cluster efficiency for short-lived tasks in batch processing system like Hadoop. They are not suitable for modern long-running workloads of Microservices, Spark, Flink, Pregel, Storm or Tensorflow like systems. It is urgent to develop new effective scheduling and resource allocation approaches to improve efficiency in large-scale enterprise datacenters. In the dissertation, we are the first of works to define and identify the problems, challenges and scenarios of scheduling and resource management for diverse long-running workloads in modern datacenter. They rely on predictive scheduling techniques to perform reservation, auto-scaling, migration or rescheduling. It forces us to pursue and explore more intelligent scheduling techniques by adequate predictive knowledges. We innovatively specify what is intelligent scheduling, what abilities are necessary towards intelligent scheduling, how to leverage intelligent scheduling to transfer NP-hard online scheduling problems to resolvable offline scheduling issues. We designed and implemented an intelligent cloud datacenter scheduler, which automatically performs resource-to-performance modeling, predictive optimal reservation estimation, QoS (interference)-aware predictive scheduling to maximize resource efficiency of multi-dimensions (CPU, Memory, Network, Disk I/O), and strictly guarantee service level agreements (SLA) for long-running workloads. Finally, we introduced a large-scale co-location techniques of executing long-running and other workloads on the shared global datacenter infrastructure of Alibaba Group. It effectively improves cluster utilization from 10% to averagely 50%. It is far more complicated beyond scheduling that involves technique evolutions of IDC, network, physical datacenter topology, storage, server hardwares, operating systems and containerization. We demonstrate its effectiveness by analysis of newest Alibaba public cluster trace in 2017. We are the first of works to reveal the global view of scenarios, challenges and status in Alibaba large-scale global datacenters by data demonstration, including big promotion events like Double 11 . Data-driven intelligent scheduling methodologies and effective infrastructure co-location techniques are critical and necessary to pursue maximized multi-resource efficiency in modern large-scale datacenter, especially for long-running workloads

Incorporating bioenergy into sustainable landscape designs

AbstractThe paper describes an approach to landscape design that focuses on integrating bioenergy production with other components of environmental, social and economic systems. Landscape design as used here refers to a spatially explicit, collaborative plan for management of landscapes and supply chains. Landscape design can involve multiple scales and build on existing practices to reduce costs or enhance services. Appropriately applied to a specific context, landscape design can help people assess trade-offs when making choices about locations, types of feedstock, transport, refining and distribution of bioenergy products and services. The approach includes performance monitoring and reporting along the bioenergy supply chain. Examples of landscape design applied to bioenergy production systems are presented. Barriers to implementation of landscape design include high costs, the need to consider diverse land-management objectives from a wide array of stakeholders, up-front planning requirements, and the complexity and level of effort needed for successful stakeholder involvement. A landscape design process may be stymied by insufficient data or participation. An impetus for coordination is critical, and incentives may be required to engage landowners and the private sector. Hence devising and implementing landscape designs for more sustainable outcomes require clear communication of environmental, social, and economic opportunities and concerns

The ecomics of ecosystems and biodiversity: scoping the scale

The G8 decided in March 2007 to initiate a “Review on the economics of biodiversity loss”, in the so called Potsdam Initiative: 'In a global study we will initiate the process of analysing the global economic benefit of biological diversity, the costs of the loss of biodiversity and the failure to take protective measures versus the costs of effective conservation. The study is being supported by the European Commission (together with the European Environmental Agency and in cooperation with the German Government. “The objective of the current study is to provide a coherent overview of existing scientific knowledge upon which to base the economics of the Review, and to propose a coherent global programme of scientific work, both for Phase 2 (consolidation) and to enable more robust future iterations of the Review beyond 2010.

Soil biodiversity: functions, threats and tools for policy makers

Human societies rely on the vast diversity of benefits provided by nature, such as food, fibres, construction materials, clean water, clean air and climate regulation. All the elements required for these ecosystem services depend on soil, and soil biodiversity is the driving force behind their regulation. With 2010 being the international year of biodiversity and with the growing attention in Europe on the importance of soils to remain healthy and capable of supporting human activities sustainably, now is the perfect time to raise awareness on preserving soil biodiversity. The objective of this report is to review the state of knowledge of soil biodiversity, its functions, its contribution to ecosystem services and its relevance for the sustainability of human society. In line with the definition of biodiversity given in the 1992 Rio de Janeiro Convention, soil biodiversity can be defined as the variation in soil life, from genes to communities, and the variation in soil habitats, from micro-aggregates to entire landscapes. Bio Intelligence Service, IRD, and NIOO, Report for European Commission (DG Environment