5 research outputs found
Resource Management and Scheduling for Big Data Applications in Cloud Computing Environments
This chapter presents software architectures of the big data processing
platforms. It will provide an in-depth knowledge on resource management
techniques involved while deploying big data processing systems on cloud
environment. It starts from the very basics and gradually introduce the core
components of resource management which we have divided in multiple layers. It
covers the state-of-art practices and researches done in SLA-based resource
management with a specific focus on the job scheduling mechanisms.Comment: 27 pages, 9 figure
Adaptive Provisioning of Heterogeneous Cloud Resources for Big Data Processing
Efficient utilization of resources plays an important role in the performance of large scale task processing. In cases where heterogeneous types of resources are used within the same application, it is hard to achieve good utilization of all of the different types of resources. By taking advantage of recent developments in cloud infrastructure that enable the use of dynamic clusters of resources, and by dynamically altering the size of the available resources for all the different resource types, the overall utilization of resources, however, can be improved. Starting from this premise, this paper discusses a solution that aims to provide a generic algorithm to estimate the desired ratios of instance processing tasks as well as ratios of the resources that are used by these instances, without the necessity for trial runs or a priori knowledge of the execution steps. These ratios are then used as part of an adaptive system that is able to reconfigure itself to maximize utilization. To verify the solution, a reference framework which adaptively manages clusters of functionally different VMs to host a calculation scenario is implemented. Experiments are conducted based on a compute-heavy use case in which the probability of underground pipeline failures is determined based on the settlement of soils. These experiments show that the solution is capable of eliminating large amounts of under-utilization, resulting in increased throughput and lower lead times
A survey on the development status and application prospects of knowledge graph in smart grids
With the advent of the electric power big data era, semantic interoperability
and interconnection of power data have received extensive attention. Knowledge
graph technology is a new method describing the complex relationships between
concepts and entities in the objective world, which is widely concerned because
of its robust knowledge inference ability. Especially with the proliferation of
measurement devices and exponential growth of electric power data empowers,
electric power knowledge graph provides new opportunities to solve the
contradictions between the massive power resources and the continuously
increasing demands for intelligent applications. In an attempt to fulfil the
potential of knowledge graph and deal with the various challenges faced, as
well as to obtain insights to achieve business applications of smart grids,
this work first presents a holistic study of knowledge-driven intelligent
application integration. Specifically, a detailed overview of electric power
knowledge mining is provided. Then, the overview of the knowledge graph in
smart grids is introduced. Moreover, the architecture of the big knowledge
graph platform for smart grids and critical technologies are described.
Furthermore, this paper comprehensively elaborates on the application prospects
leveraged by knowledge graph oriented to smart grids, power consumer service,
decision-making in dispatching, and operation and maintenance of power
equipment. Finally, issues and challenges are summarised.Comment: IET Generation, Transmission & Distributio