45,870 research outputs found

    DALiuGE: A Graph Execution Framework for Harnessing the Astronomical Data Deluge

    Full text link
    The Data Activated Liu Graph Engine - DALiuGE - is an execution framework for processing large astronomical datasets at a scale required by the Square Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex data reduction pipelines consisting of both data sets and algorithmic components and an implementation run-time to execute such pipelines on distributed resources. By mapping the logical view of a pipeline to its physical realisation, DALiuGE separates the concerns of multiple stakeholders, allowing them to collectively optimise large-scale data processing solutions in a coherent manner. The execution in DALiuGE is data-activated, where each individual data item autonomously triggers the processing on itself. Such decentralisation also makes the execution framework very scalable and flexible, supporting pipeline sizes ranging from less than ten tasks running on a laptop to tens of millions of concurrent tasks on the second fastest supercomputer in the world. DALiuGE has been used in production for reducing interferometry data sets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide Spectral Radioheliograph; and is being developed as the execution framework prototype for the Science Data Processor (SDP) consortium of the Square Kilometre Array (SKA) telescope. This paper presents a technical overview of DALiuGE and discusses case studies from the CHILES and MUSER projects that use DALiuGE to execute production pipelines. In a companion paper, we provide in-depth analysis of DALiuGE's scalability to very large numbers of tasks on two supercomputing facilities.Comment: 31 pages, 12 figures, currently under review by Astronomy and Computin

    Efficient energy management for the internet of things in smart cities

    Get PDF
    The drastic increase in urbanization over the past few years requires sustainable, efficient, and smart solutions for transportation, governance, environment, quality of life, and so on. The Internet of Things offers many sophisticated and ubiquitous applications for smart cities. The energy demand of IoT applications is increased, while IoT devices continue to grow in both numbers and requirements. Therefore, smart city solutions must have the ability to efficiently utilize energy and handle the associated challenges. Energy management is considered as a key paradigm for the realization of complex energy systems in smart cities. In this article, we present a brief overview of energy management and challenges in smart cities. We then provide a unifying framework for energy-efficient optimization and scheduling of IoT-based smart cities. We also discuss the energy harvesting in smart cities, which is a promising solution for extending the lifetime of low-power devices and its related challenges. We detail two case studies. The first one targets energy-efficient scheduling in smart homes, and the second covers wireless power transfer for IoT devices in smart cities. Simulation results for the case studies demonstrate the tremendous impact of energy-efficient scheduling optimization and wireless power transfer on the performance of IoT in smart cities

    An eco-friendly hybrid urban computing network combining community-based wireless LAN access and wireless sensor networking

    Get PDF
    Computer-enhanced smart environments, distributed environmental monitoring, wireless communication, energy conservation and sustainable technologies, ubiquitous access to Internet-located data and services, user mobility and innovation as a tool for service differentiation are all significant contemporary research subjects and societal developments. This position paper presents the design of a hybrid municipal network infrastructure that, to a lesser or greater degree, incorporates aspects from each of these topics by integrating a community-based Wi-Fi access network with Wireless Sensor Network (WSN) functionality. The former component provides free wireless Internet connectivity by harvesting the Internet subscriptions of city inhabitants. To minimize session interruptions for mobile clients, this subsystem incorporates technology that achieves (near-)seamless handover between Wi-Fi access points. The WSN component on the other hand renders it feasible to sense physical properties and to realize the Internet of Things (IoT) paradigm. This in turn scaffolds the development of value-added end-user applications that are consumable through the community-powered access network. The WSN subsystem invests substantially in ecological considerations by means of a green distributed reasoning framework and sensor middleware that collaboratively aim to minimize the network's global energy consumption. Via the discussion of two illustrative applications that are currently being developed as part of a concrete smart city deployment, we offer a taste of the myriad of innovative digital services in an extensive spectrum of application domains that is unlocked by the proposed platform
    corecore