A Game-Theoretic Approach for Runtime Capacity Allocation in MapReduce

Nowadays many companies have available large amounts of raw, unstructured data. Among Big Data enabling technologies, a central place is held by the MapReduce framework and, in particular, by its open source implementation, Apache Hadoop. For cost effectiveness considerations, a common approach entails sharing server clusters among multiple users. The underlying infrastructure should provide every user with a fair share of computational resources, ensuring that Service Level Agreements (SLAs) are met and avoiding wastes. In this paper we consider two mathematical programming problems that model the optimal allocation of computational resources in a Hadoop 2.x cluster with the aim to develop new capacity allocation techniques that guarantee better performance in shared data centers. Our goal is to get a substantial reduction of power consumption while respecting the deadlines stated in the SLAs and avoiding penalties associated with job rejections. The core of this approach is a distributed algorithm for runtime capacity allocation, based on Game Theory models and techniques, that mimics the MapReduce dynamics by means of interacting players, namely the central Resource Manager and Class Managers

A cooperative approach for distributed task execution in autonomic clouds

Virtualization and distributed computing are two key pillars that guarantee scalability of applications deployed in the Cloud. In Autonomous Cooperative Cloud-based Platforms, autonomous computing nodes cooperate to offer a PaaS Cloud for the deployment of user applications. Each node must allocate the necessary resources for customer applications to be executed with certain QoS guarantees. If the QoS of an application cannot be guaranteed a node has mainly two options: to allocate more resources (if it is possible) or to rely on the collaboration of other nodes. Making a decision is not trivial since it involves many factors (e.g. the cost of setting up virtual machines, migrating applications, discovering collaborators). In this paper we present a model of such scenarios and experimental results validating the convenience of cooperative strategies over selfish ones, where nodes do not help each other. We describe the architecture of the platform of autonomous clouds and the main features of the model, which has been implemented and evaluated in the DEUS discrete-event simulator. From the experimental evaluation, based on workload data from the Google Cloud Backend, we can conclude that (modulo our assumptions and simplifications) the performance of a volunteer cloud can be compared to that of a Google Cluster

Airborne and Terrestrial Laser Scanning Data for the Assessment of Standing and Lying Deadwood: Current Situation and New Perspectives

LiDAR technology is finding uses in the forest sector, not only for surveys in producing forests but also as a tool to gain a deeper understanding of the importance of the three-dimensional component of forest environments. Developments of platforms and sensors in the last decades have highlighted the capacity of this technology to catch relevant details, even at finer scales. This drives its usage towards more ecological topics and applications for forest management. In recent years, nature protection policies have been focusing on deadwood as a key element for the health of forest ecosystems and wide-scale assessments are necessary for the planning process on a landscape scale. Initial studies showed promising results in the identification of bigger deadwood components (e.g., snags, logs, stumps), employing data not specifically collected for the purpose. Nevertheless, many efforts should still be made to transfer the available methodologies to an operational level. Newly available platforms (e.g., Mobile Laser Scanner) and sensors (e.g., Multispectral Laser Scanner) might provide new opportunities for this field of study in the near future

On the Economic Sustainability of Cloud Sharing Systems: Are Dynamic Single Resource Sharing Markets Stable?

The recent emergence of the small cloud (SC), both in concept and in practice, has been driven mainly by issues related to service cost and complexity of commercial cloud providers (e.g., Amazon) employing massive data centers. However, the resource inelasticity problem faced by the SCs due to their relatively scarce resources might lead to a potential degradation of customer QoS and loss of revenue. A proposed solution to this problem recommends the sharing of resources between competing SCs to alleviate the resource inelasticity issues that might arise. Based on this idea, a recent effort proposed SC-Share, a performance-driven static market model for competitive small cloud environments that results in an efficient market equilibrium jointly optimizing customer QoS satisfaction and SC revenue generation. However, an important question with a non-obvious answer still remains to be answered, without which SC sharing markets may not be guaranteed to sustain in the long-run - is it still possible to achieve a stable market efficient state when the supply of SC resources is dynamic in nature? In this short paper, we take a first step to addressing the problem of efficient market design for single SC resource sharing in dynamic environments. We answer our previous question in the affirmative through the use of Arrow and Hurwicz's disequilibrium process in economics, and the gradient play technique in game theory that allows us to iteratively converge upon efficient and stable market equilibria.Peer reviewe

Self-managed Cost-efficient Virtual Elastic Clusters on Hybrid Cloud Infrastructures

In this study, we describe the further development of Elastic Cloud Computing Cluster (EC3), a tool for creating self-managed cost-efficient virtual hybrid elastic clusters on top of Infrastructure as a Service (IaaS) clouds. By using spot instances and checkpointing techniques, EC3 can significantly reduce the total execution cost as well as facilitating automatic fault tolerance. Moreover, EC3 can deploy and manage hybrid clusters across on-premises and public cloud resources, thereby introducing cloud bursting capabilities. We present the results of a case study that we conducted to assess the effectiveness of the tool based on the structural dynamic analysis of buildings. In addition, we evaluated the checkpointing algorithms in a real cloud environment with existing workloads to study their effectiveness. The results demonstrate the feasibility and benefits of this type of cluster for computationally intensive applications. © 2016 Elsevier B.V. All rights reserved.This study was supported by the program "Ayudas para la contratacion de personal investigador en formacion de caracter pre doctoral, programa VALi+d" under grant number ACIF/2013/003 from the Conselleria d'Educacio of the Generalitat Valenciana. We are also grateful for financial support received from The Spanish Ministry of Economy and Competitiveness to develop the project "CLUVIEM" under grant reference TIN2013-44390-R. Finally, we express our gratitude to D. David Ruzafa for support with the arduous task of analyzing the executions data.Calatrava Arroyo, A.; Romero Alcalde, E.; Moltó Martínez, G.; Caballer Fernández, M.; Alonso Ábalos, JM. (2016). Self-managed Cost-efficient Virtual Elastic Clusters on Hybrid Cloud Infrastructures. Future Generation Computer Systems. 61:13-25. https://doi.org/10.1016/j.future.2016.01.018S13256