979 research outputs found

    Application profiling and resource management for MapReduce

    Get PDF
    Scale of data generated and processed is exponential growth in the Big Data ear. It poses a challenge that is far beyond the goal of a single computing system. Processing such vast amount of data on a single machine is impracticable in term of time or cost. Hence, distributed systems, which can harness very large clusters of commodity computers and processing data within restrictive time deadlines, are imperative. In this thesis, we target two aspects of distributed systems: application profiling and resource management. We study a MapReduce system in detail, which is a programming paradigm for large scale distributed computing, and presents solutions to tackle three key problems. Firstly, this thesis analyzes the characteristics of jobs running on the MapReduce system to reveal the problem—the Application scope of MapReduce has been extended beyond the original design goal that was large-scale data processing. This problem enables us to present a Workload Characteristic Oriented Scheduler (WCO), which strives for co-locating tasks of possibly different MapReduce jobs with complementing resource usage characteristics. Secondly, this thesis studies the current job priority mechanism focusing on resource management. In the MapReduce system, job priority only exists at scheduling level. High priority jobs are placed at the front of the scheduling queue and dispatched first. Resource, however, is fairly shared among jobs running at the same worker node without any consideration for their priorities. In order to resolve this, this thesis presents a non-intrusive slot layering solution, which dynamically allocates resource between running jobs based on their priority and efficiently reduces the execution time of high priority jobs while improves overall throughput. Last, based on the fact of underutilization of resource at each individual worker node, this thesis propose a new way, Local Resource Shaper (LRS), to smooth resource consumption of each individual job by automatically tuning the execution of concurrent jobs to maximize resource utilization while minimizing resource contention

    Application profiling and resource management for MapReduce

    Get PDF
    Scale of data generated and processed is exponential growth in the Big Data ear. It poses a challenge that is far beyond the goal of a single computing system. Processing such vast amount of data on a single machine is impracticable in term of time or cost. Hence, distributed systems, which can harness very large clusters of commodity computers and processing data within restrictive time deadlines, are imperative. In this thesis, we target two aspects of distributed systems: application profiling and resource management. We study a MapReduce system in detail, which is a programming paradigm for large scale distributed computing, and presents solutions to tackle three key problems. Firstly, this thesis analyzes the characteristics of jobs running on the MapReduce system to reveal the problem—the Application scope of MapReduce has been extended beyond the original design goal that was large-scale data processing. This problem enables us to present a Workload Characteristic Oriented Scheduler (WCO), which strives for co-locating tasks of possibly different MapReduce jobs with complementing resource usage characteristics. Secondly, this thesis studies the current job priority mechanism focusing on resource management. In the MapReduce system, job priority only exists at scheduling level. High priority jobs are placed at the front of the scheduling queue and dispatched first. Resource, however, is fairly shared among jobs running at the same worker node without any consideration for their priorities. In order to resolve this, this thesis presents a non-intrusive slot layering solution, which dynamically allocates resource between running jobs based on their priority and efficiently reduces the execution time of high priority jobs while improves overall throughput. Last, based on the fact of underutilization of resource at each individual worker node, this thesis propose a new way, Local Resource Shaper (LRS), to smooth resource consumption of each individual job by automatically tuning the execution of concurrent jobs to maximize resource utilization while minimizing resource contention

    Resource management algorithms for real-time wireless sensor networks with applications in cyber-physical systems

    Get PDF
    Wireless Sensor Networks (WSN) are playing a key role in the efficient operation of Cyber Physical Systems (CPS). They provide cost efficient solutions to current and future CPS re- quirements such as real-time structural awareness, faster event localization, cost reduction due to condition based maintenance rather than periodic maintenance, increased opportunities for real-time preventive or corrective control action and fine grained diagnostic analysis. However, there are several critical challenges in the real world applicability of WSN. The low power, low data rate characteristics of WSNs coupled with constraints such as application specified latency and wireless interference present challenges to their efficient integration in CPSs. The existing state of the art solutions lack methods to address these challenges that impediment the easy integration of WSN in CPS. This dissertation develops efficient resource management algorithms enabling WSNs to perform reliable, real-time, cost efficient monitoring. This research addresses three important problems in resource management in the presence of different constraints such as latency, precedence and wireless interference constraints. Additionally, the dissertation proposes a solution to deploy WSNs based real-time monitoring of critical infrastructure such as electrical overhead transmission lines. Firstly, design and analysis of an energy-aware scheduling algorithm encompassing both computation and communication subsystems in the presence of deadline, precedence and in- terference constraints is presented. The energy-delay tradeoff presented by the energy saving technologies such as Dynamic Voltage Scaling (DVS) and Dynamic modulation Scaling (DMS) is studied and methods to leverage it by way of efficient schedule construction is proposed. Performance results show that the proposed polynomial-time heuristic scheduling algorithm offers comparable energy savings to that of the analytically derived optimal solution. Secondly, design, analysis and evaluation of adaptive online algorithms leveraging run- time variations is presented. Specifically, two widely used medium access control schemes are considered and online algorithms are proposed for each. For one, temporal correlation in sensor measurements is exploited and three heuristics with varying complexities are proposed to perform energy minimization using DMS. For another, an adaptive algorithm is proposed addressing channel and load conditions at a node by influencing the selection of either low energy or low delay transmission option. In both cases, the simulation results show that the proposed schemes provide much better energy savings as compared to the existing algorithms. The third component presents design and evaluation of a WSN based framework to mon- itor a CPS namely, electrical overhead transmission line infrastructure. The cost optimized hybrid hierarchical network architecture is composed of a combination of wired, wireless and cellular technologies. The proposed formulation is generic and addresses constraints such as bandwidth and latency; and real world scenarios such as asymmetric sensor data generation, unreliable wireless link behavior, non-uniform cellular coverage and is suitable for cost minimized incremental future deployment. In conclusion, this dissertation addresses several challenging research questions in the area of resource management in WSNs and their applicability in future CPSs through associated algorithms and analyses. The proposed research opens up new avenues for future research such as energy management through network coding and fault diagnosis for reliable monitoring

    BAG : Managing GPU as buffer cache in operating systems

    Get PDF
    This paper presents the design, implementation and evaluation of BAG, a system that manages GPU as the buffer cache in operating systems. Unlike previous uses of GPUs, which have focused on the computational capabilities of GPUs, BAG is designed to explore a new dimension in managing GPUs in heterogeneous systems where the GPU memory is an exploitable but always ignored resource. With the carefully designed data structures and algorithms, such as concurrent hashtable, log-structured data store for the management of GPU memory, and highly-parallel GPU kernels for garbage collection, BAG achieves good performance under various workloads. In addition, leveraging the existing abstraction of the operating system not only makes the implementation of BAG non-intrusive, but also facilitates the system deployment

    An Analytical Model for Wireless Mesh Networks with Collision-Free TDMA and Finite Queues

    Full text link
    Wireless mesh networks are a promising technology for connecting sensors and actuators with high flexibility and low investment costs. In industrial applications, however, reliability is essential. Therefore, two time-slotted medium access methods, DSME and TSCH, were added to the IEEE 802.15.4 standard. They allow collision-free communication in multi-hop networks and provide channel hopping for mitigating external interferences. The slot schedule used in these networks is of high importance for the network performance. This paper supports the development of efficient schedules by providing an analytical model for the assessment of such schedules, focused on TSCH. A Markov chain model for the finite queue on every node is introduced that takes the slot distribution into account. The models of all nodes are interconnected to calculate network metrics such as packet delivery ratio, end-to-end delay and throughput. An evaluation compares the model with a simulation of the Orchestra schedule. The model is applied to Orchestra as well as to two simple distributed scheduling algorithms to demonstrate the importance of traffic-awareness for achieving high throughput.Comment: 17 pages, 14 figure

    An Integrated Routing and Distributed Scheduling Approach for Hybrid IEEE 802.16E Mesh Networks For Vehicular Broadband Communications

    Get PDF
    An integrated routing and distributed scheduling approach for fast deployable IEEE 802.16e networks is presented where distributed base stations with dual radios form a mesh backhaul and subscriber stations communicate through these base stations. The mesh backhaul is formed via an IEEE 802.16e mesh mode radio on each base station, while the subscriber stations communicate with base stations via PMP mode radios. The proposed routing scheme divides the deployed network into several routing zones. Each routing zone contains several base stations that form the mesh backhaul with one base station equipped with either a fiber, satellite or any other point-to-point backhaul link to reach a gateway on the core network (for example, Internet or Enterprise Network). Traffic from the subscriber stations is routed by the serving base station through the mesh to the gateway-connected base station using min-hop routing metric. Mobile IP scheme is used to assign a care-of address to a subscriber station that moves from one routing zone to the other, thereby avoiding a change in IP address for network layer applications. The scheduling approach consists of two phases. In the first phase, a centralized mesh scheduling algorithm is applied with collected information on network topology, radio parameters, and initial QoS provisioning requirements. At the same time, each base station derives a PMP schedule for actual demands from associated subscriber stations constrained by the initial mesh schedule. In the second phase, each base station monitors its carried PMP traffic load statistics; to accommodate traffic load changes in a distributed fashion, each base station lends or borrows time slots from neighboring base stations to adjust its mesh and PMP radio schedules. The distributed schedule adaptation method not only allows individual base stations to accommodate short-term increases in bandwidth demands, it also provides the means for optimizing the mesh and PMP schedules with respect to actual bandwidth demands. Several deployment strategies are considered and an analytical model is developed to identify the achievable increase in overall network throughput using the proposed scheduling approach. Simulations are run in network simulator ns-2 to verify results obtained using the analytical model

    Scalable adaptive networking for the Internet of Underwater Things

    Get PDF
    Internet of Underwater Things (IoUT) systems comprising tens or hundreds of underwater acoustic communication nodes will become feasible in the near future. The development of scalable networking protocols is a key enabling technology for such IoUT systems, but this task is challenging due to the fundamental limitations of the underwater acoustic communication channel: extremely slow propagation and limited bandwidth. The aim of this paper is to propose the JOIN protocol to enable the integration of new nodes into an existing IoUT network without the control overhead of typical state-of-the-art solutions. The proposed solution is based on the capability of a joining node to incorporate local topology and schedule information into a probabilistic model that allows it to choose when to join the network to minimize the expected number of collisions. The proposed approach is tested in numerical simulations and validated in two sea trials. The simulations show that the JOIN protocol achieves fast convergence to a collision-free solution, fast network adaptation to new nodes, and negligible network disruption due to collisions caused by a joining node. The sea trials demonstrate the practical feasibility of this protocol in real UAN deployments and provide valuable insight for future work on the trade-off between control overhead and reliability of the JOIN protocol in a harsh acoustic communication environment

    Medium access control design for distributed opportunistic radio networks

    Get PDF
    Existing wireless networks are characterized by a fixed spectrum assignment policy. However, the scarcity of available spectrum and its inefficient usage demands for a new communication paradigm to exploit the existing spectrum opportunistically. Future Cognitive Radio (CR) devices should be able to sense unoccupied spectrum and will allow the deployment of real opportunistic networks. Still, traditional Physical (PHY) and Medium Access Control (MAC) protocols are not suitable for this new type of networks because they are optimized to operate over fixed assigned frequency bands. Therefore, novel PHY-MAC cross-layer protocols should be developed to cope with the specific features of opportunistic networks. This thesis is mainly focused on the design and evaluation of MAC protocols for Decentralized Cognitive Radio Networks (DCRNs). It starts with a characterization of the spectrum sensing framework based on the Energy-Based Sensing (EBS) technique considering multiple scenarios. Then, guided by the sensing results obtained by the aforementioned technique, we present two novel decentralized CR MAC schemes: the first one designed to operate in single-channel scenarios and the second one to be used in multichannel scenarios. Analytical models for the network goodput, packet service time and individual transmission probability are derived and used to compute the performance of both protocols. Simulation results assess the accuracy of the analytical models as well as the benefits of the proposed CR MAC schemes
    corecore