10 research outputs found
Trends and challenges in smart metering analytics
With strong policy support globally, it is expected that the total amount of smart energy meters installed worldwide will reach 780 million by 2020, including 200 million in the EU and 30 Million in the UK alone. Smart metering can improve grid operation and maintenance of distribution networks through load forecasting, improve demand response measures, and enhance end-user experience through accurate billing and appliance-level energy feedback via Non-Intrusive Load Monitoring (NILM). In this paper, we review trends of smart metering applications and challenges in large-scale adoption, and provide case studies to demonstrate application of NILM for meaningful energy feedback
An Ensemble Detection Model using Multinomial Classification of Stochastic Gas Smart Meter Data to Improve Wellbeing Monitoring in Smart Cities
Fuel poverty has a negative impact on the wellbeing of individuals within a household; affecting not only comfort levels but also increased levels of seasonal mortality. Wellbeing solutions within this sector are moving towards identifying how the needs of people in vulnerable situations can be improved or monitored by means of existing supply networks and public institutions. Therefore, the focus of this research is towards wellbeing monitoring solution, through the analysis of gas smart meter data. Gas smart meters replace the traditional analogue electro-mechanical and diaphragm-based meters that required regular reading. They have received widespread popularity over the last 10 years. This is primarily due to the fact that by using this technology, customers are able to adapt their consumption behaviours based on real-time information provided by In-Home Devices. Yet, the granular nature of the datasets generated has also meant that this technology is ideal for further scalable wellbeing monitoring applications. For example, the autonomous detection of households at risk of energy poverty is possible and of growing importance in order to face up to the impacts of fuel poverty, quality of life and wellbeing of low-income housing. However, despite their popularity (smart meters), the analysis of gas smart meter data has been neglected. In this paper, an ensemble model is proposed to achieve autonomous detection, supported by four key measures from gas usage patterns, consisting of i) a tariff detection, ii) a temporally-aware tariff detection, iii) a routine consumption detection and iv) an age-group detection. Using a cloud-based machine learning platform, the proposed approach yielded promising classification results of up to 84.1% Area Under Curve (AUC), when the Synthetic Minority Over-sampling Technique (SMOTE) was utilised
A Hierarchical Framework for Smart Grid Anomaly Detection Using Large-Scale Smart Meter Data
Real-time monitoring and control of smart grids (SGs) is critical to the enhancement of reliability and operational efficiency of power utilities. We develop a real-time anomaly detection framework, which can be built based upon smart meter (SM) data collected at the consumers' premises. The model is designed to detect the occurrence of anomalous events and abnormal conditions at both lateral and customer levels. We propose a generative model for anomaly detection that takes into account the hierarchical structure of the network and the data collected from SMs. We also address three challenges existing in SG analytics: 1) large-scale multivariate count measurements; 2) missing points; and 3) variable selection. We present the effectiveness of our approach with numerical experiments
Real-Time Detection of Demand Manipulation Attacks on a Power Grid
An increased usage in IoT devices across the globe has posed a threat to the power grid. When an attacker has access to multiple IoT devices within the same geographical location, they can possibly disrupt the power grid by regulating a botnet of high-wattage IoT devices. Based on the time and situation of the attack, an adversary needs access to a fixed number of IoT devices to synchronously switch on/off all of them, resulting in an imbalance between the supply and demand. When the frequency of the power generators drops below a threshold value, it can lead to the generators tripping and potentially failing. Attacks such as these can cause an imbalance in the grid frequency, line failures and cascades, can disrupt a black start or increase the operating cost. The challenge lies in early detection of abnormal demand peaks in a large section of the power grid from the power operator’s side, as it only takes seconds to cause a generator failure before any action could be taken.
Anomaly detection comes handy to flag the power operator of an anomalous behavior while such an attack is taking place. However, it is difficult to detect anomalies especially when such attacks are taking place obscurely and for prolonged time periods. With this motive, we compare different anomaly detection systems in terms of detecting these anomalies collectively. We generate attack data using real-world power consumption data across multiple apartments to assess the performance of various prediction-based detection techniques as well as commercial detection applications and observe the cases when the attacks were not detected. Using static thresholds for the detection process does not reliably detect attacks when they are performed in different times of the year and also lets the attacker exploit the system to create the attack obscurely. To combat the effects of using static thresholds, we propose a novel dynamic thresholding mechanism, which improves the attack detection reaching up to 100% detection rate, when used with prediction-based anomaly score techniques
Big Data na gestão eficiente das Smart Grids. HDS: Uma Plataforma Híbrida, Dinâmica e Inteligente
[POR]Nos últimos anos tem-se verificado um acréscimo exponencial de informação gerada e disponibilizada
a cada dia. Devido ao rápido avanço tecnológico (dispositivos móveis; sensores; comunicação wireless;
etc.) biliões e biliões de bytes são criados todos os dias. Este fenómeno, denominado por Big Data, é
caracterizado por 5 Vs (i.e. Volume, Velocidade, Variedade, Veracidade, Valor) e cada um deles
representa verdadeiros desafios (e.g. como recolher e transportar um grande volume de informação;
como armazenar essa informação; como minerá-la, como analisá-la e extrair conhecimento, como
garantir a sua segurança e privacidade, como processá-la em tempo real, etc.). É unanime na comunidade
científica que o valor a extrair de toda esta informação constituirá um fator de extrema importância para
a tomada de decisão, determinante no sucesso das mais variadíssimas áreas económicas, bem como na
resolução de inúmeros problemas. Nestas áreas inclui-se o ecossistema energético que por razões
ecológicas, económicas e políticas conduziu ao repensar da forma como consumimos e produzimos
energia. Devido ao aumento das necessidades energéticas provocado pelo avanço tecnológico, ao
previsto esgotamento dos recursos energéticos não renováveis e devido às diretivas para a eficiência
energética impostas pela União Europeia, muitos têm sido os estudos feitos na área da gestão de recursos
energéticos. O termo Smart Grids surgiu nas últimas décadas com o objetivo de definir um ecossistema
energético inteligente, que visa não só a integração de inteligência, mas também de automação na
operabilidade extremamente complexa de todos os seus processos. As Smart Grids têm sido alvo de
grandes estudos e investimentos dos quais têm resultado avanços significativos. No entanto, alguns
desafios estão ainda por concretizar nomeadamente na gestão do seu complexo fluxo de dados. É neste
contexto que se enquadra a presente dissertação cujo principal objetivo se centra na obtenção de soluções
para alguns dos problemas identificados no domínio de Smart Grids com recurso às novas técnicas e
metodologias propostas na área de Big Data.
Este trabalho apresenta um estudo sobre os recentes e crescentes avanços tecnológicos realizados na
área de Big Data, onde são identificados os seus grandes desafios. Destes destacam-se a complexidade
na gestão de fluxos contínuos e desordenados, a necessidade de reduzir o tempo despendido na prépreparação
dos dados e o desafio de explorar soluções que proporcionem a automatização analítica. Por
outro lado, o estudo analisa o impacto da aplicação nas novas tecnologias no desenvolvimento das Smart
Grids, no qual se conclui que apesar de embrionária, a sua aplicação é imprescindível para a evolução
do ecossistema energético. Deste estudo resultou ainda a identificação dos principais desafios na área
das Smart Grids, dos quais se destacam a complexidade na gestão do seu fluxo de dados em tempo real
e a necessidade de melhorar a precisão das previsões de consumo e produção de energia.
Face aos desafios identificados foi proposto um modelo conceptual, baseado na arquitetura Docker
Container, para o desenvolvimento de uma plataforma. Este modelo objetiva a flexibilidade e agilidade
de forma a permitir a integração e validação das novas e crescentes abordagens tecnológicas propostas
na área de Big Data, necessárias ao desenvolvimento das Smart Grids. A fim de validar o modelo
proposto, foi desenvolvida uma stack onde foram implementados vários serviços que visaram contribuir
para os desafios identificados na área de Big Data e Smart Grids, nomeadamente: visualização e
monitorização dos dados recolhidos em tempo real; preparação dos dados recolhidos em tempo real;
previsão em tempo real de várias séries temporais simultaniamente; deteção de anomalias; avaliação da
precisão das previsões e geração de novos modelos para a previsão de consumo e produção de energia
segundo determinados critérios.
Finalmente foram desenvolvidos vários casos de estudo cujos resultados obtidos permitiram concluir
sobre a importância da pré-preparação dos dados na fase analítica, sobre a eficiência na automatização
analítica e sobre as vantagens da análise de ponta (Edge Analytics). Ao contrário de abordagens mais
tradicionais que visam a execução centralizada do processo analítico, o edge analytics explora a
possibilidade de executar a analise de dados de forma descentralizada a partir de um ponto não central
do sistema. Os resultados permitiram concluir que o edge analytics traz vantagens acrescidas para a
precisão das previsões. Permitiram ainda, inferir sobre como recolher os resultados a fim de se obter
uma melhor precisão nas previsões, i.e., quanto mais específica e ajustada ao contexto forem executadas
as previsões maior será a sua precisão.[ES]En los últimos años se ha verificado un aumento exponencial de información generada y disponible cada
día. Debido al rápido avance tecnológico (dispositivos móviles, sensores, comunicación inalámbrica,
etc.) billones y billones de bytes se crean todos los días. Este fenómeno, denominado Big Data, se
caracteriza por 5 Vs (es decir, Volumen, Velocidad, Variedad, Veracidad, Valor) y cada uno de ellos
representa verdaderos desafíos (por ejemplo, cómo recoger y transportar un gran volumen de
información, cómo almacenar esa información, minarla, cómo analizarla y extraer conocimiento, cómo
garantizar su seguridad y privacidad, cómo procesarla en tiempo real, etc.). Es unánime en la comunidad
científica que el valor a extraer de toda esta información constituirá un factor de extrema importancia
para la toma de decisión, determinante el éxito de las variadísimas áreas económicas, así como en la
resolución de innumerables problemas. En estas áreas se incluye el ecosistema energético que por
razones ecológicas, económicas y políticas condujo a repensar la forma en que consumimos y
producimos energía. Debido al aumento de las necesidades energéticas provocado por el avance
tecnológico, al previsto agotamiento de los recursos energéticos no renovables y debido a las directivas
para la eficiencia energética impuestas por la Unión Europea, muchos han sido los estudios realizados
en el ámbito de la gestión de recursos energéticos. El término Smart Grid surgió en las últimas décadas
con el objetivo de definir un ecosistema energético inteligente, que apunta no sólo a la integración de
inteligencia, sino también de automatización en la operatividad extremadamente compleja de todos sus
procesos. Las Smart Grids han sido objeto de grandes estudios e inversiones de los cuales han resultado
avances significativos. Sin embargo, algunos desafíos aún no se concretan en la gestión de su complejo
flujo de datos. Es en este contexto que se encuadra la presente disertación cuyo principal objetivo se
centra en la obtención de soluciones para algunos de los problemas identificados en el dominio de Smart
Grids utilizando las nuevas técnicas y metodologías propuestas en el área de Big Data.
Este trabajo presenta un estudio sobre los recientes y crecientes avances tecnológicos realizados en el
área de Big Data, donde se identifican sus grandes desafíos. De ellos se destacan la complejidad en la
gestión de flujos continuos y desordenados, la necesidad de reducir el tiempo empleado en la prepreparación
de los datos y el desafío de explorar soluciones que proporcionen la automatización
analítica. Por otro lado, el estudio analiza el impacto de la aplicación de nuevas tecnologías en el
desarrollo de las Smart Grids, en el que se concluye que, a pesar de embrionaria, su aplicación es
imprescindible para la evolución del ecosistema energético. De este estudio resultó también la
identificación de los principales desafíos en el área de las Smart Grids, de los cuales se destacan la
complejidad en la gestión de su flujo de datos en tiempo real y la necesidad de mejorar la precisión de
las previsiones de consumo y producción de energía.
En cuanto a los desafíos identificados, se propuso un modelo conceptual, basado en la arquitectura
Docker Container, para el desarrollo de una plataforma. Este modelo tiene como objetivo la flexibilidad y agilidad para permitir la integración y validación de los nuevos y crecientes enfoques tecnológicos
propuestos en el área de Big Data, necesarios para el desarrollo de las Smart Grids. Con el fin de validar
el modelo propuesto, se desarrolló una stack donde se implementaron varios servicios que pretendían
contribuir a los desafíos identificados en el área de Big Data y Smart Grids, en particular: visualización
y seguimiento de los datos recogidos en tiempo real; preparación de los datos recogidos en tiempo real;
previsión en tiempo real de multillas series temporales simultáneamente; detección de anomalías;
evaluación de la precisión del predicción y generación de nuevos modelos para la previsión de consumo
y producción de energía según ciertos criterios.
Finalmente, se desarrollaron una serie de casos de estudo cuyos resultados nos permitieron concluir
sobre la importancia de la preparación previa de los datos en la fase analítica, la eficiencia en la
automatización analítica y las ventajas del análisis de borde (Edge Analytics). A diferencia de los
enfoques más tradicionales para la ejecución centralizada del proceso analítico, el análisis de borde
explora la posibilidad de realizar análisis de datos de forma descentralizada desde un punto no central
del sistema. Los resultados permitieron concluir que el análisis de borde aporta ventajas añadidas a la
precisión de los pronósticos. También nos permitieron inferir cómo recopilar los resultados para obtener
una mejor precisión en las predicciones, por ejemplo, cuanto más precisos y ajustados al contexto se
ejecuten los pronósticos, mayor será su precisión.[EN]In recent years, there has been an exponential increase of information generated and made available
every day. Due to rapid technological advancement (e.g., mobile devices, sensors, wireless
communication, etc.) billions and billions of bytes are created every day. This phenomenon, called Big
Data, is characterized by 5 Vs (i.e., Volume, Velocity, Variety, Veracity, Value) and each represents
real challenges (e.g., how to collect and carry a large amount of information; how to store this
information; how mining it, analyzing it and extracting knowledge; how to ensure its security and
privacy; how to process it in real time, etc.). It is unanimous in the scientific community that the value
to be extracted from all this information will be a factor of extreme importance for the decision making,
determining the success of the most varied economic areas, as well as the resolution of numerous
problems. These areas include the energy ecosystem that, for ecological, economic and political reasons,
led us to rethink the way we consume and produce energy. Due to the increase in energy needs caused
by technological advances, the expected depletion of non-renewable energy resources and due to the
energy efficiency directives imposed by the European Union, many studies have been carried out in the
area of energy resources management. The term Smart Grid has emerged in the last decades with the
objective of defining an intelligent energy ecosystem, which aims not only to integrate intelligence but
also to automate the extremely complex operability of all its processes. Smart grids have been the subject
of major studies and investments which have resulted in significant advances. However, some
challenges have to be addressed in the management of its complex data flow. It is in this context that
the present dissertation falls, with the main objective on obtaining solutions to some of the problems
identified in the field of Smart Grids using new techniques and methodologies proposed in the area of
Big Data.
This paper presents a study on the recent and growing technological advances in the area of Big Data,
where its major challenges are identified. These include complexity in the management of continuous
and disordered flows, the need to reduce the time spent in pre-preparation of data and the challenge of
exploring solutions that provide analytical automation. On the other hand, the study analyzes the impact
of the application in the new technologies in the development of the Smart Grids, in which it is concluded
that, although embryonic, its application is essential for the evolution of the energy ecosystem. This
study also resulted in the identification of the main challenges in the area of Smart Grids, which highlight
the complexity in managing its data flow in real time and the need to improve the accuracy of energy
consumption and production forecasts.
Given the identified challenges, a conceptual model, based on the Docker Container architecture, was
proposed for the development of a platform. This model aims at flexibility and agility in order to allow
the integration and validation of the new and growing technological approaches proposed in the area of
Big Data, necessary for the development of Smart Grids. In order to validate the proposed model, a stack was developed where several services were implemented that aimed to contribute to the challenges
identified in the area of Big Data and Smart Grids, namely: visualization and monitoring of data
collected in real time; preparation of data collected in real time; real-time forecasting of multiple time
series simultaneously; detection of anomalies; evaluation of the accuracy of forecasting and generation
of new models for the forecast of consumption and production of energy according to certain criteria.
Finally, a number of case studies were developed whose results allowed us to conclude on the
importance of the pre-preparation of the data in the analytical phase, on the efficiency in the analytical
automation and on the advantages of the Edge Analytics. Unlike more traditional approaches to the
centralized execution of the analytic process, edge analytics explores the possibility of performing data
analysis in a decentralized way from a non-central point of the system. The results allowed to conclude
that edge analytics brings added advantages to the precision of the forecasts. Results allowed us to infer
how to collect the data in order to obtain a better precision in the predictions, i.e., the more precise and
context-adjusted the forecasts are executed the greater their accuracy
Recommended from our members
Deep learning driven data analytics for smart grids
This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonAs advanced metering infrastructure (AMI) and wide area monitoring systems (WAMSs) are being deployed rapidly and widely, the conventional power grid is transitioning towards the smart grid at an increasing speed. A number of smart metering devices and real-time monitoring systems are capable to generate a huge volume of data on a daily basis. However, a variety of generated data can be made full use of to advance the development of the smart grid through big data analytics, especially, deep learning. Thus, the thesis is focused on data analysis for smart grids from three different aspects.
Firstly, a real-time data driven event detection method is presented, which is quite robust when dealing with corrupted and significantly noisy data of phase measurement units (PMUs). To be specific, the presented event detection method is based on a novel combination of random matrix theory (RMT) and Kalman filtering. Furthermore, a dynamic Kalman filtering technique is proposed through the adjustment of the measurement noise covariance matrix as the data conditioner of the presented method in order to condition PMU data. The experimental results show that the presented method is indeed quite robust in such practical situations that include significant levels of noisy or missing PMU data.
Secondly, a short-term residential load forecasting method is proposed on the basis of deep learning and k-means clustering, which is capable to extract similarity of residential load effectively and perform prediction accurately at the individual residential level. Specifically, it makes full use of k-means clustering to extract similarity among residential load and deep learning to extract complex patterns of residential load. In addition, in order to improve the forecasting accuracy, a comprehensive feature expression strategy is utilised to describe load characteristics of each time step in detail. The experimental results suggest that the proposed method can achieve a high forecasting accuracy in terms of both root mean square error (RMSE) and mean absolute error (MAE).
Thirdly, an online individual residential load forecasting method is developed based on a combination of deep learning and dynamic mirror descent (DMD), which is able to predict residential load in real time and adjust the prediction error over time in order to improve the prediction performance. More specifically, it firstly employs a long short term memory (LSTM) network to build a prediction model offline, and then applies it online with DMD correcting the prediction error. In order to increase the prediction accuracy, a comprehensive feature expression strategy is used to describe load characteristics at each time step in detail. The experimental results indicate that the developed method can obtain a high prediction accuracy in terms of both RMSE and MAE.
To sum up, the proposed real-time event detection method contributes to the monitoring and operation of smart grids, while the proposed residential load forecasting methods contribute to the demand side response in smart grids.TDX-ASSIS