ElfStore: A Resilient Data Storage Service for Federated Edge and Fog Resources

Edge and fog computing have grown popular as IoT deployments become wide-spread. While application composition and scheduling on such resources are being explored, there exists a gap in a distributed data storage service on the edge and fog layer, instead depending solely on the cloud for data persistence. Such a service should reliably store and manage data on fog and edge devices, even in the presence of failures, and offer transparent discovery and access to data for use by edge computing applications. Here, we present Elfstore, a first-of-its-kind edge-local federated store for streams of data blocks. It uses reliable fog devices as a super-peer overlay to monitor the edge resources, offers federated metadata indexing using Bloom filters, locates data within 2-hops, and maintains approximate global statistics about the reliability and storage capacity of edges. Edges host the actual data blocks, and we use a unique differential replication scheme to select edges on which to replicate blocks, to guarantee a minimum reliability and to balance storage utilization. Our experiments on two IoT virtual deployments with 20 and 272 devices show that ElfStore has low overheads, is bound only by the network bandwidth, has scalable performance, and offers tunable resilience.Comment: 24 pages, 14 figures, To appear in IEEE International Conference on Web Services (ICWS), Milan, Italy, 201

Freecursive ORAM: [Nearly] Free Recursion and Integrity Verification for Position-based Oblivious RAM

Oblivious RAM (ORAM) is a cryptographic primitive that hides memory access patterns as seen by untrusted storage. Recently, ORAM has been architected into secure processors. A big challenge for hardware ORAM schemes is how to efficiently manage the Position Map (PosMap), a central component in modern ORAM algorithms. Implemented naively, the PosMap causes ORAM to be fundamentally unscalable in terms of on-chip area. On the other hand, a technique called Recursive ORAM fixes the area problem yet significantly increases ORAM's performance overhead. To address this challenge, we propose three new mechanisms. We propose a new ORAM structure called the PosMap Lookaside Buffer (PLB) and PosMap compression techniques to reduce the performance overhead from Recursive ORAM empirically (the latter also improves the construction asymptotically). Through simulation, we show that these techniques reduce the memory bandwidth overhead needed to support recursion by 95%, reduce overall ORAM bandwidth by 37% and improve overall SPEC benchmark performance by 1.27x. We then show how our PosMap compression techniques further facilitate an extremely efficient integrity verification scheme for ORAM which we call PosMap MAC (PMMAC). For a practical parameterization, PMMAC reduces the amount of hashing needed for integrity checking by >= 68x relative to prior schemes and introduces only 7% performance overhead. We prototype our mechanisms in hardware and report area and clock frequency for a complete ORAM design post-synthesis and post-layout using an ASIC flow in a 32~nm commercial process. With 2 DRAM channels, the design post-layout runs at 1~GHz and has a total area of .47~mm2. Depending on PLB-specific parameters, the PLB accounts for 10% to 26% area. PMMAC costs 12% of total design area. Our work is the first to prototype Recursive ORAM or ORAM with any integrity scheme in hardware.Qatar Computing Research Institute (QCRI-CSAIL Parternship)National Science Foundation (U.S.)American Society for Engineering Education. National Defense Science and Engineering Graduate Fellowshi

Comparison of scatter storage techniques using an analysis of variance model

An analysis of variance model is developed to determine if a significant difference exists between various scatter storage techniques. The model is a two-factor hierarchical mixed design with each combination of transformation and search method considered as a treatment. The data used in the analysis is obtained from a computer program which provides statistics on the number of probes needed to load the (k+l)st item into a table for the different treatments. An ANOVA table was then computed for various load factors. A significant difference among the treatments was detected for load factors above .4. Comparison of individual treatments using Tukey\u27s multiple range test shows that although some treatments are significantly inferior, most treatments are not significantly different in terms of the average number of probes needed to load an item --Abstract, page ii

Scalable Hash Tables

The term scalability with regards to this dissertation has two meanings: It means taking the best possible advantage of the provided resources (both computational and memory resources) and it also means scaling data structures in the literal sense, i.e., growing the capacity, by “rescaling” the table. Scaling well to computational resources implies constructing the fastest best per- forming algorithms and data structures. On today’s many-core machines the best performance is immediately associated with parallelism. Since CPU frequencies have stopped growing about 10-15 years ago, parallelism is the only way to take ad- vantage of growing computational resources. But for data structures in general and hash tables in particular performance is not only linked to faster computations. The most execution time is actually spent waiting for memory. Thus optimizing data structures to reduce the amount of memory accesses or to take better advantage of the memory hierarchy especially through predictable access patterns and prefetch- ing is just as important. In terms of scaling the size of hash tables we have identified three domains where scaling hash-based data structures have been lacking previously, i.e., space effi- cient growing, concurrent hash tables, and Approximate Membership Query data structures (AMQ-filter). Throughout this dissertation, we describe the problems in these areas and develop efficient solutions. We highlight three different libraries that we have developed over the course of this dissertation, each containing mul- tiple implementations that have shown throughout our testing to be among the best implementations in their respective domains. In this composition they offer a comprehensive toolbox that can be used to solve many kinds of hashing related problems or to develop individual solutions for further ones. DySECT is a library for space efficient hash tables specifically growing space effi- cient hash tables that scale with their input size. It contains the namesake DySECT data structure in addition to a number of different probing and cuckoo based im- plementations. Growt is a library for highly efficient concurrent hash tables. It contains a very fast base table and a number of extensions to adapt this table to match any purpose. All extension can be combined to create a variety of different interfaces. In our extensive experimental evaluation, each adaptation has shown to be among the best hash tables for their specific purpose. Lpqfilter is a library for concurrent approximate membership query (AMQ) data structures. It contains some original data structures, like the linear probing quotient filter, as well as some novel approaches to dynamically sized quotient filters

Robust optimisation of urban drought security for an uncertain climate

Abstract Recent experience with drought and a shifting climate has highlighted the vulnerability of urban water supplies to “running out of water” in Perth, south-east Queensland, Sydney, Melbourne and Adelaide and has triggered major investment in water source infrastructure which ultimately will run into tens of billions of dollars. With the prospect of continuing population growth in major cities, the provision of acceptable drought security will become more pressing particularly if the future climate becomes drier. Decision makers need to deal with significant uncertainty about future climate and population. In particular the science of climate change is such that the accuracy of model predictions of future climate is limited by fundamental irreducible uncertainties. It would be unwise to unduly rely on projections made by climate models and prudent to favour solutions that are robust across a range of possible climate futures. This study presents and demonstrates a methodology that addresses the problem of finding “good” solutions for urban bulk water systems in the presence of deep uncertainty about future climate. The methodology involves three key steps: 1) Build a simulation model of the bulk water system; 2) Construct replicates of future climate that reproduce natural variability seen in the instrumental record and that reflect a plausible range of future climates; and 3) Use multi-objective optimisation to efficiently search through potentially trillions of solutions to identify a set of “good” solutions that optimally trade-off expected performance against robustness or sensitivity of performance over the range of future climates. A case study based on the Lower Hunter in New South Wales demonstrates the methodology. It is important to note that the case study does not consider the full suite of options and objectives; preliminary information on plausible options has been generalised for demonstration purposes and therefore its results should only be used in the context of evaluating the methodology. “Dry” and “wet” climate scenarios that represent the likely span of climate in 2070 based on the A1F1 emissions scenario were constructed. Using the WATHNET5 model, a simulation model of the Lower Hunter was constructed and validated. The search for “good” solutions was conducted by minimizing two criteria, 1) the expected present worth cost of capital and operational costs and social costs due to restrictions and emergency rationing, and 2) the difference in present worth cost between the “dry” and “wet” 2070 climate scenarios. The constraint was imposed that solutions must be able to supply (reduced) demand in the worst drought. Two demand scenarios were considered, “1.28 x current demand” representing expected consumption in 2060 and “2 x current demand” representing a highly stressed system. The optimisation considered a representative range of options including desalination, new surface water sources, demand substitution using rainwater tanks, drought contingency measures and operating rules. It was found the sensitivity of solutions to uncertainty about future climate varied considerably. For the “1.28 x demand” scenario there was limited sensitivity to the climate scenarios resulting in a narrow range of trade-offs. In contrast, for the “2 x demand” scenario, the trade-off between expected present worth cost and robustness was considerable. The main policy implication is that (possibly large) uncertainty about future climate may not necessarily produce significantly different performance trajectories. The sensitivity is determined not only by differences between climate scenarios but also by other external stresses imposed on the system such as population growth and by constraints on the available options to secure the system against drought. Recent experience with drought and a shifting climate has highlighted the vulnerability of urban water supplies to “running out of water” in Perth, south-east Queensland, Sydney, Melbourne and Adelaide and has triggered major investment in water source infrastructure which ultimately will run into tens of billions of dollars. With the prospect of continuing population growth in major cities, the provision of acceptable drought security will become more pressing particularly if the future climate becomes drier. Decision makers need to deal with significant uncertainty about future climate and population. In particular the science of climate change is such that the accuracy of model predictions of future climate is limited by fundamental irreducible uncertainties. It would be unwise to unduly rely on projections made by climate models and prudent to favour solutions that are robust across a range of possible climate futures. This study presents and demonstrates a methodology that addresses the problem of finding “good” solutions for urban bulk water systems in the presence of deep uncertainty about future climate. The methodology involves three key steps: 1) Build a simulation model of the bulk water system; 2) Construct replicates of future climate that reproduce natural variability seen in the instrumental record and that reflect a plausible range of future climates; and 3) Use multi-objective optimisation to efficiently search through potentially trillions of solutions to identify a set of “good” solutions that optimally trade-off expected performance against robustness or sensitivity of performance over the range of future climates. A case study based on the Lower Hunter in New South Wales demonstrates the methodology. It is important to note that the case study does not consider the full suite of options and objectives; preliminary information on plausible options has been generalised for demonstration purposes and therefore its results should only be used in the context of evaluating the methodology. “Dry” and “wet” climate scenarios that represent the likely span of climate in 2070 based on the A1F1 emissions scenario were constructed. Using the WATHNET5 model, a simulation model of the Lower Hunter was constructed and validated. The search for “good” solutions was conducted by minimizing two criteria, 1) the expected present worth cost of capital and operational costs and social costs due to restrictions and emergency rationing, and 2) the difference in present worth cost between the “dry” and “wet” 2070 climate scenarios. The constraint was imposed that solutions must be able to supply (reduced) demand in the worst drought. Two demand scenarios were considered, “1.28 x current demand” representing expected consumption in 2060 and “2 x current demand” representing a highly stressed system. The optimisation considered a representative range of options including desalination, new surface water sources, demand substitution using rainwater tanks, drought contingency measures and operating rules. It was found the sensitivity of solutions to uncertainty about future climate varied considerably. For the “1.28 x demand” scenario there was limited sensitivity to the climate scenarios resulting in a narrow range of trade-offs. In contrast, for the “2 x demand” scenario, the trade-off between expected present worth cost and robustness was considerable. The main policy implication is that (possibly large) uncertainty about future climate may not necessarily produce significantly different performance trajectories. The sensitivity is determined not only by differences between climate scenarios but also by other external stresses imposed on the system such as population growth and by constraints on the available options to secure the system against drought. Please cite this report as: Mortazavi, M, Kuczera, G, Kiem, AS, Henley, B, Berghout, B,Turner, E, 2013 Robust optimisation of urban drought security for an uncertain climate. National Climate Change Adaptation Research Facility, Gold Coast, pp. 74

Datacenter Traffic Control: Understanding Techniques and Trade-offs

Datacenters provide cost-effective and flexible access to scalable compute and storage resources necessary for today's cloud computing needs. A typical datacenter is made up of thousands of servers connected with a large network and usually managed by one operator. To provide quality access to the variety of applications and services hosted on datacenters and maximize performance, it deems necessary to use datacenter networks effectively and efficiently. Datacenter traffic is often a mix of several classes with different priorities and requirements. This includes user-generated interactive traffic, traffic with deadlines, and long-running traffic. To this end, custom transport protocols and traffic management techniques have been developed to improve datacenter network performance. In this tutorial paper, we review the general architecture of datacenter networks, various topologies proposed for them, their traffic properties, general traffic control challenges in datacenters and general traffic control objectives. The purpose of this paper is to bring out the important characteristics of traffic control in datacenters and not to survey all existing solutions (as it is virtually impossible due to massive body of existing research). We hope to provide readers with a wide range of options and factors while considering a variety of traffic control mechanisms. We discuss various characteristics of datacenter traffic control including management schemes, transmission control, traffic shaping, prioritization, load balancing, multipathing, and traffic scheduling. Next, we point to several open challenges as well as new and interesting networking paradigms. At the end of this paper, we briefly review inter-datacenter networks that connect geographically dispersed datacenters which have been receiving increasing attention recently and pose interesting and novel research problems.Comment: Accepted for Publication in IEEE Communications Surveys and Tutorial