160 research outputs found

    Deep generative models for network data synthesis and monitoring

    Get PDF
    Measurement and monitoring are fundamental tasks in all networks, enabling the down-stream management and optimization of the network. Although networks inherently have abundant amounts of monitoring data, its access and effective measurement is another story. The challenges exist in many aspects. First, the inaccessibility of network monitoring data for external users, and it is hard to provide a high-fidelity dataset without leaking commercial sensitive information. Second, it could be very expensive to carry out effective data collection to cover a large-scale network system, considering the size of network growing, i.e., cell number of radio network and the number of flows in the Internet Service Provider (ISP) network. Third, it is difficult to ensure fidelity and efficiency simultaneously in network monitoring, as the available resources in the network element that can be applied to support the measurement function are too limited to implement sophisticated mechanisms. Finally, understanding and explaining the behavior of the network becomes challenging due to its size and complex structure. Various emerging optimization-based solutions (e.g., compressive sensing) or data-driven solutions (e.g. deep learning) have been proposed for the aforementioned challenges. However, the fidelity and efficiency of existing methods cannot yet meet the current network requirements. The contributions made in this thesis significantly advance the state of the art in the domain of network measurement and monitoring techniques. Overall, we leverage cutting-edge machine learning technology, deep generative modeling, throughout the entire thesis. First, we design and realize APPSHOT , an efficient city-scale network traffic sharing with a conditional generative model, which only requires open-source contextual data during inference (e.g., land use information and population distribution). Second, we develop an efficient drive testing system — GENDT, based on generative model, which combines graph neural networks, conditional generation, and quantified model uncertainty to enhance the efficiency of mobile drive testing. Third, we design and implement DISTILGAN, a high-fidelity, efficient, versatile, and real-time network telemetry system with latent GANs and spectral-temporal networks. Finally, we propose SPOTLIGHT , an accurate, explainable, and efficient anomaly detection system of the Open RAN (Radio Access Network) system. The lessons learned through this research are summarized, and interesting topics are discussed for future work in this domain. All proposed solutions have been evaluated with real-world datasets and applied to support different applications in real systems

    Advances and Applications of DSmT for Information Fusion. Collected Works, Volume 5

    Get PDF
    This fifth volume on Advances and Applications of DSmT for Information Fusion collects theoretical and applied contributions of researchers working in different fields of applications and in mathematics, and is available in open-access. The collected contributions of this volume have either been published or presented after disseminating the fourth volume in 2015 in international conferences, seminars, workshops and journals, or they are new. The contributions of each part of this volume are chronologically ordered. First Part of this book presents some theoretical advances on DSmT, dealing mainly with modified Proportional Conflict Redistribution Rules (PCR) of combination with degree of intersection, coarsening techniques, interval calculus for PCR thanks to set inversion via interval analysis (SIVIA), rough set classifiers, canonical decomposition of dichotomous belief functions, fast PCR fusion, fast inter-criteria analysis with PCR, and improved PCR5 and PCR6 rules preserving the (quasi-)neutrality of (quasi-)vacuous belief assignment in the fusion of sources of evidence with their Matlab codes. Because more applications of DSmT have emerged in the past years since the apparition of the fourth book of DSmT in 2015, the second part of this volume is about selected applications of DSmT mainly in building change detection, object recognition, quality of data association in tracking, perception in robotics, risk assessment for torrent protection and multi-criteria decision-making, multi-modal image fusion, coarsening techniques, recommender system, levee characterization and assessment, human heading perception, trust assessment, robotics, biometrics, failure detection, GPS systems, inter-criteria analysis, group decision, human activity recognition, storm prediction, data association for autonomous vehicles, identification of maritime vessels, fusion of support vector machines (SVM), Silx-Furtif RUST code library for information fusion including PCR rules, and network for ship classification. Finally, the third part presents interesting contributions related to belief functions in general published or presented along the years since 2015. These contributions are related with decision-making under uncertainty, belief approximations, probability transformations, new distances between belief functions, non-classical multi-criteria decision-making problems with belief functions, generalization of Bayes theorem, image processing, data association, entropy and cross-entropy measures, fuzzy evidence numbers, negator of belief mass, human activity recognition, information fusion for breast cancer therapy, imbalanced data classification, and hybrid techniques mixing deep learning with belief functions as well

    Demand Response in Smart Grids

    Get PDF
    The Special Issue “Demand Response in Smart Grids” includes 11 papers on a variety of topics. The success of this Special Issue demonstrates the relevance of demand response programs and events in the operation of power and energy systems at both the distribution level and at the wide power system level. This reprint addresses the design, implementation, and operation of demand response programs, with focus on methods and techniques to achieve an optimized operation as well as on the electricity consumer

    Unsupervised energy disaggregation via convolutional sparse coding

    Full text link
    In this work, a method for unsupervised energy disaggregation in private households equipped with smart meters is proposed. This method aims to classify power consumption as active or passive, granting the ability to report on the residents' activity and presence without direct interaction. This lays the foundation for applications like non-intrusive health monitoring of private homes. The proposed method is based on minimizing a suitable energy functional, for which the iPALM (inertial proximal alternating linearized minimization) algorithm is employed, demonstrating that various conditions guaranteeing convergence are satisfied. In order to confirm feasibility of the proposed method, experiments on semi-synthetic test data sets and a comparison to existing, supervised methods are provided.Comment: 9 pages, 2 figures, 3 table

    A Distributed and Real-time Machine Learning Framework for Smart Meter Big Data

    Get PDF
    The advanced metering infrastructure allows smart meters to collect high-resolution consumption data, thereby enabling consumers and utilities to understand their energy usage at different levels, which has led to numerous smart grid applications. Smart meter data, however, poses different challenges to developing machine learning frameworks than classic theoretical frameworks due to their big data features and privacy limitations. Therefore, in this work, we aim to address the challenges of building machine learning frameworks for smart meter big data. Specifically, our work includes three parts: 1) We first analyze and compare different learning algorithms for multi-level smart meter big data. A daily activity pattern recognition model has been developed based on non-intrusive load monitoring for appliance-level smart meter data. Then, a consensus-based load profiling and forecasting system has been proposed for individual building level and higher aggregated level smart meter data analysis; 2) Following discussion of multi-level smart meter data analysis from an offline perspective, a universal online functional analysis model has been proposed for multi-level real-time smart meter big data analysis. The proposed model consists of a multi-scale load dynamic profiling unit based on functional clustering and a multi-scale online load forecasting unit based on functional deep neural networks. The two units enable online tracking of the dynamic cluster trajectories and online forecasting of daily multi-scale demand; 3) To enable smart meter data analysis in the distributed environment, FederatedNILM was proposed, which is then combined with differential privacy to provide privacy guarantees for the appliance-level distributed machine learning framework. Based on federated deep learning enhanced with two schemes, namely the utility optimization scheme and the privacy-preserving scheme, the proposed distributed and privacy-preserving machine learning framework enables electric utilities and service providers to offer smart meter services on a large scale

    A Framework for Offline Risk-aware Planning of Low-altitude Aerial Flights during Urban Disaster Response

    Get PDF
    Disaster response missions are dynamic and dangerous events for first responders. Active situational awareness is critical for effective decision-making, and unmanned aerial assets have successfully extended the range and output of sensors. Aerial assets have demonstrated their capability in disaster response missions via decentralized operations. However, literature and industry lack a systematic investigation of the algorithms, datasets, and tools for aerial system trajectory planning in urban disasters that optimizes mission performance and guarantee mission success. This work seeks to develop a framework and software environment to investigate the requirements for offline planning algorithms and flight risk models when applied to aerial assets exploring urban disaster zones. This is addressed through the creation of rapid urban maps, efficient flight planning algorithms, and formal risk metrics that are demonstrated in scenario-driven experiments using Monte Carlo simulation. First, rapid urban mapping strategies are independently compared for efficient processing and storage through obstacle and terrain layers. Open-source data is used when available and is supplemented with an urban feature prediction model trained on satellite imagery using deep learning. Second, sampling-based planners are evaluated for efficient and effective trajectory planning of nonlinear aerial dynamic systems. The algorithm can find collision-free, kinodynamic feasible trajectories using random open-loop control targets. Alternative open-loop control commands are formed to improve the planning algorithm’s speed and convergence. Third, a risk-aware implementation of the planning algorithm is developed that considers the uncertainty of energy, collisions, and onboard viewpoint data and maps them to a single measure of the likelihood of mission failure. The three modules are combined in a framework where the rapid urban maps and risk-aware planner are evaluated against benchmarks for mission success, performance, and speed while creating a unique set of benchmarks from open-source data and software. One, the rapid urban map module generates a 3D structure and terrain map within 20 meters of data and in less than 5 minutes. The Gaussian Process terrain model performs better than B-spline and NURBS models in small-scale, mountainous environments at 10-meter squared resolution. Supplementary data for structures and other urban landcover features is predicted using the Pix2Pix Generative Adversarial Network with a 3-channel encoding for nine labels. Structures, greenspaces, water, and roads are predicted with high accuracy according to the F1, OIU, and pixel accuracy metrics. Two, the sampling-based planning algorithm is selected for forming collision-free, 3D offline flight paths with a black-box dynamics model of a quadcopter. Sampling-based planners prove successful for efficient and optimal flight paths through randomly generated and rapid urban maps, even under wind and noise uncertainty. The Stable-Sparse-RRT, SST, algorithm is shown to improve trajectories for minimum Euclidean distance more consistently and efficiently than the RRT algorithm, with a 50% improvement in finite-time path convergence for large-scale urban maps. The forward propagation dynamics of the black-box model are replaced with 5-15 times more computationally efficient motion primitives that are generated using an inverse lower-order dynamics model and the Differential Dynamic Programming, DDP, algorithm. Third, the risk-aware planning algorithm is developed that generates optimal paths based on three risk metrics of energy, collision, and viewpoint risk and quantifies the likelihood of worst-case events using the Conditional-Value-at-Risk, CVaR, metric. The sampling-based planning algorithm is improved with informative paths, and three versions of the algorithm are compared for the best performance in different scenarios. Energy risk in the planning algorithm results in 5-35% energy reduction and 20-30% more consistency in finite-time convergence for flight paths in large-scale urban maps. All three risk metrics in the planning algorithm generally result in more energy use than the planner with only energy risk, but reduce the mean flight path risk by 10-50% depending on the environment, energy available, and viewpoint landmarks. A final experiment in an Atlanta flooding scenario demonstrates the framework’s full capability with the rapid urban map displaying essential features and the trajectory planner reporting flight time, energy consumption, and total risk. Furthermore, the simulation environment provides insight into offline planning limitations through Monte Carlo simulations with environment wind and system dynamics noise. The framework and software environment are made available to use as benchmarks in the field to serve as a foundation for increasing the effectiveness of first responders’ safety in the challenging task of urban disaster response.Ph.D

    Neural Natural Language Processing for Long Texts: A Survey of the State-of-the-Art

    Full text link
    The adoption of Deep Neural Networks (DNNs) has greatly benefited Natural Language Processing (NLP) during the past decade. However, the demands of long document analysis are quite different from those of shorter texts, while the ever increasing size of documents uploaded on-line renders automated understanding of long texts a critical area of research. This article has two goals: a) it overviews the relevant neural building blocks, thus serving as a short tutorial, and b) it surveys the state-of-the-art in long document NLP, mainly focusing on two central tasks: document classification and document summarization. Sentiment analysis for long texts is also covered, since it is typically treated as a particular case of document classification. Additionally, this article discusses the main challenges, issues and current solutions related to long document NLP. Finally, the relevant, publicly available, annotated datasets are presented, in order to facilitate further research.Comment: 53 pages, 2 figures, 171 citation

    Residential Energy Management for Renewable Energy Systems Incorporating Data-Driven Unravelling of User Behavior

    Get PDF
    The penetration of distributed energy resources (DERs) such as photovoltaic (PV) at the residential level has increased rapidly over the past year. It will inevitably induce a paradigm shift in end-user and operations of local energy markets. The energy community with high integration of DERs initiative allows its users to manage their generation (for prosumers) and consumption more efficiently, resulting in various economic, social, and environmental benefits. Specifically, the local energy communities and their members can legally engage in energy generation, distribution, supply, consumption, storage, and sharing to increase levels of autonomy from the power grid, advance energy efficiency, reduce energy costs, and decrease carbon emissions. Reducing energy consumption costs is difficult for residential energy management without understanding the users' preferences. The advanced measurement and communication technologies provide opportunities for individual consumers/prosumers and local energy communities to adopt a more active role in renewable-rich smart grids. Non-intrusive load monitoring (NILM) monitors the load activities from a single point source, such as a smart meter, based on the assumption that different appliances have different power consumption levels and features. NILM can extract the users' load consumption from the smart meter to support the development of the smart grid for better energy management and demand response (DR). Yet to date, how to design residential energy management, including home energy management systems (HEMS) and community energy management systems (CEMS), with an understanding of user preferences and willingness to participate in energy management, is still far from being fully investigated. This thesis aims to develop methodologies for a resident energy management system for renewable energy systems (RES) incorporating data-driven unravelling of the user's energy consumption behaviour
    corecore