32,317 research outputs found
Adaptive Resource Management for Distributed Data Analytics
Increasingly large datasets make scalable and distributed data analytics necessary. Frameworks such as Spark and Flink help users in efficiently utilizing cluster resources for their data analytics jobs. It is, however, usually difficult to anticipate the runtime behavior and resource demands of these distributed data analytics jobs. Yet, many resource management decisions would benefit from such information.
Addressing this general problem, this chapter presents our vision of adaptive resource management and reviews recent work in this area. The key idea is that workloads should be monitored for trends, patterns, and recurring jobs. These monitoring statistics should be analyzed and used for a cluster resource management calibrated to the actual workload. In this chapter, we motivate and present the idea of adaptive resource management. We also introduce a general system architecture and we review specific adaptive techniques for data placement, resource allocation, and job scheduling in the context of our architecture
A Game-Theoretic Based QoS-Aware Capacity Management for Real-Time EdgeIoT Applications
More and more real-time IoT applications such as smart cities or autonomous vehicles require big data analytics with reduced latencies. However, data streams produced from distributed sensing devices may not suffice to be processed traditionally in the remote cloud due to: (i) longer Wide Area Network (WAN) latencies and (ii) limited resources held by a single Cloud. To solve this problem, a novel Software-Defined Network (SDN) based InterCloud architecture is presented for mobile edge computing environments, known as EdgeIoT. An adaptive resource capacity management approach is proposed to employ a policy-based QoS control framework using principles in coalition games with externalities. To optimise resource capacity policy, the proposed QoS management technique solves, adaptively, a lexicographic ordering bi-criteria Coalition Structure Generation (CSG) problem. It is an onerous task to guarantee in a deterministic way that a real-time EdgeIoT application satisfies low latency requirement specified in Service Level Agreements (SLA). CloudSim 4.0 toolkit is used to simulate an SDN-based InterCloud scenario, and the empirical results suggest that the proposed approach can adapt, from an operational perspective, to ensure low latency QoS for real-time EdgeIoT application instances
Models of everywhere revisited: a technological perspective
The concept ‘models of everywhere’ was first introduced in the mid 2000s as a means of reasoning about the
environmental science of a place, changing the nature of the underlying modelling process, from one in which
general model structures are used to one in which modelling becomes a learning process about specific places, in
particular capturing the idiosyncrasies of that place. At one level, this is a straightforward concept, but at another
it is a rich multi-dimensional conceptual framework involving the following key dimensions: models of everywhere,
models of everything and models at all times, being constantly re-evaluated against the most current
evidence. This is a compelling approach with the potential to deal with epistemic uncertainties and nonlinearities.
However, the approach has, as yet, not been fully utilised or explored. This paper examines the
concept of models of everywhere in the light of recent advances in technology. The paper argues that, when first
proposed, technology was a limiting factor but now, with advances in areas such as Internet of Things, cloud
computing and data analytics, many of the barriers have been alleviated. Consequently, it is timely to look again
at the concept of models of everywhere in practical conditions as part of a trans-disciplinary effort to tackle the
remaining research questions. The paper concludes by identifying the key elements of a research agenda that
should underpin such experimentation and deployment
- …