Modeling performance of Hadoop applications: A journey from queueing networks to stochastic well formed nets

Nowadays, many enterprises commit to the extraction of actionable knowledge from huge datasets as part of their core business activities. Applications belong to very different domains such as fraud detection or one-to-one marketing, and encompass business analytics and support to decision making in both private and public sectors. In these scenarios, a central place is held by the MapReduce framework and in particular its open source implementation, Apache Hadoop. In such environments, new challenges arise in the area of jobs performance prediction, with the needs to provide Service Level Agreement guarantees to the enduser and to avoid waste of computational resources. In this paper we provide performance analysis models to estimate MapReduce job execution times in Hadoop clusters governed by the YARN Capacity Scheduler. We propose models of increasing complexity and accuracy, ranging from queueing networks to stochastic well formed nets, able to estimate job performance under a number of scenarios of interest, including also unreliable resources. The accuracy of our models is evaluated by considering the TPC-DS industry benchmark running experiments on Amazon EC2 and the CINECA Italian supercomputing center. The results have shown that the average accuracy we can achieve is in the range 9–14%

Distributed Operating Systems

Distributed operating systems have many aspects in common with centralized ones, but they also differ in certain ways. This paper is intended as an introduction to distributed operating systems, and especially to current university research about them. After a discussion of what constitutes a distributed operating system and how it is distinguished from a computer network, various key design issues are discussed. Then several examples of current research projects are examined in some detail, namely, the Cambridge Distributed Computing System, Amoeba, V, and Eden. © 1985, ACM. All rights reserved

A Discrete Choice Experiment to Elicit the Willingness to Pay for Health Insurance by the Informal Sector Workers in Sierra Leone

The current health care financing system in Sierra Leone is unsustainable and poses challenges ranging from increased in out of pocket health care expenditure to accessibility problems, particularly in rural areas where living standards are low and health care facilities are scarce. This paper investigates whether privately financed health Insurance can improve the accessibility to formal health care in Sierra Leone and mitigate the effects of OOPs on poor households. To do so, we estimate the Willingness To Pay (WTP) for health insurance among informal sector workers in Sierra Leone using a Discrete Choice Experiment approach. Eight informal sector activities were selected namely – petty trading, subsistence farming, commercial bike riding, cattle rearing, fishing, tailoring, mining and quarrying. A random effect logit model is used to estimate households’ WTP for an improvement in coverage, choice of health care provider and a reduction in waiting time. Our study reveals that households were WTP more to have better attributes (better coverage, less waiting time) and to go to a faith - based provider. Our findings also suggest that location – rural versus urban – matters in determining the WTP since urban households were WTP more for health insurance than their rural counterparts, (SLL 54,348 or $7.34) and (SLL 37,250.5 or$5.03), respectively

Lightweight remote procedure call

Lightweight Remote Procedure Call (LRPC) is a communication facility designed and optimized for communication between protection domains on the same machine. In contemporary small-kernel operating systems, existing RPC systems incur an unnecessarily high cost when used for the type of communication that predominates-between protection domains on the same machine. This cost leads system designers to coalesce weakly related subsystems into the same protection domain, trading safety for performance. By reducing the overhead of same-machine communication, LRPC encourages both safety and performance. LRPC combines the control transfer and communication model of capability systems with the programming semantics and large-grained protection model of RPC. LRPC achieves a factor-of-three performance improvement over more traditional approaches based on independent threads exchanging messages, reducing the cost of same-machine communication to nearly the lower bound imposed by conventional hardware. LRPC has been integrated into the Taos operating system of the DEC SRC Firefly multiprocessor workstation