A Tale of Two Data-Intensive Paradigms: Applications, Abstractions, and Architectures

Scientific problems that depend on processing large amounts of data require overcoming challenges in multiple areas: managing large-scale data distribution, co-placement and scheduling of data with compute resources, and storing and transferring large volumes of data. We analyze the ecosystems of the two prominent paradigms for data-intensive applications, hereafter referred to as the high-performance computing and the Apache-Hadoop paradigm. We propose a basis, common terminology and functional factors upon which to analyze the two approaches of both paradigms. We discuss the concept of "Big Data Ogres" and their facets as means of understanding and characterizing the most common application workloads found across the two paradigms. We then discuss the salient features of the two paradigms, and compare and contrast the two approaches. Specifically, we examine common implementation/approaches of these paradigms, shed light upon the reasons for their current "architecture" and discuss some typical workloads that utilize them. In spite of the significant software distinctions, we believe there is architectural similarity. We discuss the potential integration of different implementations, across the different levels and components. Our comparison progresses from a fully qualitative examination of the two paradigms, to a semi-quantitative methodology. We use a simple and broadly used Ogre (K-means clustering), characterize its performance on a range of representative platforms, covering several implementations from both paradigms. Our experiments provide an insight into the relative strengths of the two paradigms. We propose that the set of Ogres will serve as a benchmark to evaluate the two paradigms along different dimensions.Comment: 8 pages, 2 figure

Towards a method for rigorous development of generic requirements patterns

We present work in progress on a method for the engineering, validation and verification of generic requirements using domain engineering and formal methods. The need to develop a generic requirement set for subsequent system instantiation is complicated by the addition of the high levels of verification demanded by safety-critical domains such as avionics. Our chosen application domain is the failure detection and management function for engine control systems: here generic requirements drive a software product line of target systems. A pilot formal specification and design exercise is undertaken on a small (twosensor) system element. This exercise has a number of aims: to support the domain analysis, to gain a view of appropriate design abstractions, for a B novice to gain experience in the B method and tools, and to evaluate the usability and utility of that method.We also present a prototype method for the production and verification of a generic requirement set in our UML-based formal notation, UML-B, and tooling developed in support. The formal verification both of the structural generic requirement set, and of a particular application, is achieved via translation to the formal specification language, B, using our U2B and ProB tools

Assessment of the development of aquifer management councils (COTAS) for sustainable groundwater management in Guanajuato, Mexico

Collective groundwater management by water users—self-regulation—is increasingly advocated as a complement to state regulation. This article analyzes the attempts by the Guanajuato State Water Commission (CEAG) in central Mexico to promote user self-regulation through the establishment and development of 14 Consejos Técnicos de Aguas (COTAS; Technical Water Councils). Based on a joint assessment by a former senior CEAG policy-maker and two researchers, Guanajuato’s groundwater-management policy is reviewed to understand why user self-regulation was less successful than expected. It concludes that increasing awareness and improving the knowledge base on groundwater is not enough to trigger self-regulation by groundwater users. A wider delegation of responsibilities to the COTAS is necessary, combined with: (1) functioning mechanisms for enforcing groundwater legislation, especially concerning well permits and pumped volumes, and (2) mechanisms that ensure the legitimacy and accountability of users’ representatives to both users and state agencie