A survey of big data and machine learning

This paper presents a detailed analysis of big data and machine learning (ML) in the electrical power and energy sector. Big data analytics for smart energy operations, applications, impact, measurement and control, and challenges are presented in this paper. Big data and machine learning approaches need to be applied after analyzing the power system problem carefully. Determining the match between the strengths of big data and machine learning for solving the power system problem is of utmost important. They can be of great help to plan and operate the traditional grid/smart grid (SG). The basics of big data and machine learning are described in detailed manner along with their applications in various fields such as electrical power and energy, health care and life sciences, government, telecommunications, web and digital media, retailers, finance, e-commerce and customer service, etc. Finally, the challenges and opportunities of big data and machine learning are presented in this paper

Storage Solutions for Big Data Systems: A Qualitative Study and Comparison

Big data systems development is full of challenges in view of the variety of application areas and domains that this technology promises to serve. Typically, fundamental design decisions involved in big data systems design include choosing appropriate storage and computing infrastructures. In this age of heterogeneous systems that integrate different technologies for optimized solution to a specific real world problem, big data system are not an exception to any such rule. As far as the storage aspect of any big data system is concerned, the primary facet in this regard is a storage infrastructure and NoSQL seems to be the right technology that fulfills its requirements. However, every big data application has variable data characteristics and thus, the corresponding data fits into a different data model. This paper presents feature and use case analysis and comparison of the four main data models namely document oriented, key value, graph and wide column. Moreover, a feature analysis of 80 NoSQL solutions has been provided, elaborating on the criteria and points that a developer must consider while making a possible choice. Typically, big data storage needs to communicate with the execution engine and other processing and visualization technologies to create a comprehensive solution. This brings forth second facet of big data storage, big data file formats, into picture. The second half of the research paper compares the advantages, shortcomings and possible use cases of available big data file formats for Hadoop, which is the foundation for most big data computing technologies. Decentralized storage and blockchain are seen as the next generation of big data storage and its challenges and future prospects have also been discussed

Who needs XAI in the Energy Sector? A Framework to Upgrade Black Box Explainability

Artificial Intelligence (AI)-based methods in the energy sector challenge companies, organizations, and societies. Organizational issues include traceability, certifiability, explainability, responsibility, and efficiency. Societal challenges include ethical norms, bias, discrimination, privacy, and information security. Explainable Artificial Intelligence (XAI) can address these issues in various application areas of the energy sector, e.g., power generation forecasting, load management, and network security operations. We derive Key Topics (KTs) and Design Requirements (DRs) and develop Design Principles (DPs) for efficient XAI applications through Design Science Research (DSR). We analyze 179 scientific articles to identify our 8 KTs for XAI implementation through text mining and topic modeling. Based on the KTs, we derive 15 DRs and develop 18 DPs. After that, we discuss and evaluate our results and findings through expert surveys. We develop a Three-Forces Model as a framework for implementing efficient XAI solutions. We provide recommendations and a further research agenda

Economic opportunities of AMI implementation : a review

Advanced Metering Infrastructure (AMI) is rapidly becoming a key element for the modernization and disruption of power grids, generating benefits and opportunities to all actors involved in its implementation. In order to guarantee a correct deployment of AMI, a wide knowledge of its advantages and challenges is needed, that takes into account previous experiences and latest advances that have been made in the field. In this paper, a review of literature is used as a mean to collect the relevant information concerning AMI, so as to conclude which are the opportunities that AMI provides to all parties involved. This is achieved by searching in the most important data bases and specialized sources such as IEEE and IEA. It was found that this infrastructure, does indeed help improve efficiency and leads to positive economic effects impacting variables like costs and prices

Big Data Analytics na energia: uma revisão sistemática da literatura

A transformação para uma sociedade mais digital tem vindo a produzir uma quantidade de dados incomparável com décadas anteriores. Big data analytics é agora um termo muito conhecido ao nível de dados financeiros, saúde, marketing, logística, entre outros. Sendo a energia uma das áreas afetadas pela transformação digital, como tem sido aplicado o conceito de big data analytics nesta área? Que aplicações surgiram devido ao volume de dados recolhidos? Que ferramentas são utilizadas? Como tem sido afetada por esta mudança? Esta dissertação pretende responder à pergunta-chave “Como tem sido aplicado big data analytics na área da energia?”. Para o efeito procede-se à realização de uma revisão sistemática da literatura, a partir da qual se elabora uma introdução das temáticas discutidas para leitores que possam estar a iniciar estudos nesta área de investigação, garantindo, ao mesmo tempo, que esta seja útil para leitores que já se encontram inseridos nestas áreas. Para a realização da revisão foi consultada a base de dados Science Direct, sendo construída uma frase booleana através das palavras-chave selecionadas durante a fase de pesquisa da base de dados sendo estas “Big Data”, “Analytics”, “Energy”, “Smart Grid”, “Smart Meters”, “Smart Sensor Network”, “Meter Data Analytics”, “Energy Management” e “Energy Consumption” tendo sido realizado um estudo bibliométrico aos resultados obtidos. Para a seleção dos estudos foram utilizados três métodos de filtragem: um primeiro centrado nos critérios de seleção; o segundo através da leitura dos títulos das publicações e dos resumos; e o terceiro através da realização de um teste de relevância. A exploração da informação permitiu concluir que big data analytics na energia tem sido aplicada em áreas como o estudo do consumidor, soluções para análise de dados recolhidos por smart meters, desenvolvimento de plataformas de análise das smart grids, segurança e gestão inteligente da energia que se refletem na forma como a energia é canalizada, aproveitada e utilizada através da associação de técnicas, processos e sistemas planeados para análise de grandes volumes de dados, resultando numa tomada de decisões mais consciente pelos fornecedores e consumidores de energi

Probabilistic short-term load forecasting at low voltage in distribution networks

Predmet istraživanja ove doktorske disertacije je kratkoročna probabili- stička prognoza opterećenja na niskom naponu u elektrodistributivnim mre- žama. Cilj istraživanja je da se razvije novo rešenje koje će uvažiti varija- bilnost opterećenja na niskom naponu i ponuditi konkurentnu tačnost prog- noze uz visoku efikasnost sa stanovišta zauzeća računarskih resursa. Predlo- ženo rešenje se zasniva na primeni statističkih metoda i metoda mašinskog (dubokog) učenja u reprezentaciji podataka (ekstrakciji i odabiru atributa), klasterovanju i regresiji. Efikasnost predloženog rešenja je verifikovana u studiji slučaja nad skupom realnih podataka sa pametnih brojila. Rezultat primene predloženog rešenja je visoka tačnost prognoze i kratko vreme izvr- šavanja u poređenju sa konkurentnim rešenjima iz aktuelnog stanja u oblasti.This Ph.D. thesis deals with the problem of probabilistic short-term load forecasting at the low voltage level in power distribution networks. The research goal is to develop a new solution that considers load variability and offers high forecasting accuracy without excessive hardware requirements. The proposed solution is based on the application of statistical methods and machine (deep) learning methods for data representation (feature extraction and selection), clustering, and regression. The efficiency of the proposed solution was verified in a case study on real smart meter data. The case study results confirm that the application of the proposed solution leads to high forecast accuracy and short execution time compared to related solutions

A Big Data Platform for Smart Meter Data Analytics

Smart grids have started generating an ever increasingly large volume of data. Extensive research has been done in meter data analytics for small data sets of electrical grid and electricity consumption. However limited research has investigated the methods, systems and tools to support data storage and data analytics for big data generated by smart grids. This work has proposed a new core-broker-client system architecture for big data analytics. Its implemented platform is named Smart Meter Analytics Scaled by Hadoop (SMASH). Our work has demonstrated that SMASH is able to perform data storage, query, analysis and visualization tasks on large data sets at 20 TB scale. The performance of SMASH in storing and querying large quantities of data are compared with the published results provided by Google Cloud, IBM, MongoDB, and AMPLab. The experimental results suggest that SMASH provides industry a competitive and easily operable platform to manage big energy data and visualize knowledge, with potential to support data-intensive decision making