A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction

The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them can capture the long-term temporal dependencies appropriately and select the relevant driving series to make predictions. In this paper, we propose a dual-stage attention-based recurrent neural network (DA-RNN) to address these two issues. In the first stage, we introduce an input attention mechanism to adaptively extract relevant driving series (a.k.a., input features) at each time step by referring to the previous encoder hidden state. In the second stage, we use a temporal attention mechanism to select relevant encoder hidden states across all time steps. With this dual-stage attention scheme, our model can not only make predictions effectively, but can also be easily interpreted. Thorough empirical studies based upon the SML 2010 dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can outperform state-of-the-art methods for time series prediction.Comment: International Joint Conference on Artificial Intelligence (IJCAI), 201

Holistic Measures for Evaluating Prediction Models in Smart Grids

The performance of prediction models is often based on "abstract metrics" that estimate the model's ability to limit residual errors between the observed and predicted values. However, meaningful evaluation and selection of prediction models for end-user domains requires holistic and application-sensitive performance measures. Inspired by energy consumption prediction models used in the emerging "big data" domain of Smart Power Grids, we propose a suite of performance measures to rationally compare models along the dimensions of scale independence, reliability, volatility and cost. We include both application independent and dependent measures, the latter parameterized to allow customization by domain experts to fit their scenario. While our measures are generalizable to other domains, we offer an empirical analysis using real energy use data for three Smart Grid applications: planning, customer education and demand response, which are relevant for energy sustainability. Our results underscore the value of the proposed measures to offer a deeper insight into models' behavior and their impact on real applications, which benefit both data mining researchers and practitioners.Comment: 14 Pages, 8 figures, Accepted and to appear in IEEE Transactions on Knowledge and Data Engineering, 2014. Authors' final version. Copyright transferred to IEE

Machine Learning Modeling of Horizontal Photovoltaics Using Weather and Location Data

Solar energy is a key renewable energy source; however, its intermittent nature and potential for use in distributed systems make power prediction an important aspect of grid integration. This research analyzed a variety of machine learning techniques to predict power output for horizontal solar panels using 14 months of data collected from 12 northern-hemisphere locations. We performed our data collection and analysis in the absence of irradiation data—an approach not commonly found in prior literature. Using latitude, month, hour, ambient temperature, pressure, humidity, wind speed, and cloud ceiling as independent variables, a distributed random forest regression algorithm modeled the combined dataset with an R2 value of 0.94. As a comparative measure, other machine learning algorithms resulted in R2 values of 0.50–0.94. Additionally, the data from each location was modeled separately with R2 values ranging from 0.91 to 0.97, indicating a range of consistency across all sites. Using an input variable permutation approach with the random forest algorithm, we found that the three most important variables for power prediction were ambient temperature, humidity, and cloud ceiling. The analysis showed that machine learning potentially allowed for accurate power prediction while avoiding the challenges associated with modeled irradiation data

Predictive modeling of PV energy production: How to set up the learning task for a better prediction?

In this paper, we tackle the problem of power prediction of several photovoltaic (PV) plants spread over an extended geographic area and connected to a power grid. The paper is intended to be a comprehensive study of one-day ahead forecast of PV energy production along several dimensions of analysis: i) The consideration of the spatio-temporal autocorrelation, which characterizes geophysical phenomena, to obtain more accurate predictions.ii) The learning setting to be considered, i.e. using simple output prediction for each hour or structured output prediction for each day. iii) The learning algorithms: We compare artificial neural networks, most often used for PV prediction forecast, and regression trees for learning adaptive models. The results obtained on two PV power plant datasets show that: taking into account spatio/temporal autocorrelation is beneficial; the structured output prediction setting significantly outperforms the non-structured output prediction setting; and regression trees provide better models than artificial neural networks

Bidirectional Encoder-Decoder with Dual-Stage Attention for Multivariate Time-Series Prediction

학위논문(석사)--서울대학교 대학원 :공과대학 협동과정 기술경영·경제·정책전공,2019. 8. 조성준.자연어처리에 주로 사용되는 RNN 계열의 모델들은 순차적인 데이터를 다루는 데 적합하기 때문에 시계열 데이터 분석에도 다양하게 활용되고 있다. RNN 계열의 모델들이 가지고 있는 취약점은 기울기 값의 소실이라는 문제이다. 이를 해결하기 위해 배치 정규화 같은 기법들이 사용되고 있지만, 시계열 데이터의 경우는 과거에서부터 이어져 오는 추세를 잃어버릴 수 있다. 그리고 적절한 외생변수들을 선택하는 사항도 고려되어야 한다. 본 연구에서는 위에서 언급한 문제들을 해결하기 위해 듀얼 어텐션 메커니즘을 기반으로 하는 양방향 Encoder-Decoder LSTM 모델을 제안한다. Encoder에서 작동하는 어텐션 메커니즘은 이전 단계의 양방향 LSTM에서 전달받은 hidden state와 cell state를 참조하여 예측에 도움이 되는 외생변수들을 파악한다. Decoder는 시계열적인 특성을 반영한 모델 구조로 되어 있다. 먼저 타겟변수의 과거 값들과 추세를 나타내는 통계치를 입력으로 받아서 양방향 LSTM을 통해 학습하고, 지정한 시간 간격까지 매번 hidden state를 업데이트시킨다. Decoder에서의 어텐션은 업데이트된 hidden state와 Encoder에서 나온 출력값을 연결하여 입력으로 받는다. 따라서 예측에 적합한 외생변수의 시점을 파악하는 것뿐만 아니라 타겟변수의 긴 추세를 반영할 수 있다. 모델 평가에 사용된 데이터는 약 4년 치의 KODEX 200(ETF)과 KODEX 200에 포함된 회사들의 5분 단위 개별 주가 거래 데이터이다.Recurrent neural network has been widely applied for time-series prediction. However, the vanishing gradient is still a problem and only a few of them select the relevant dependent variables appropriately. In this paper, I propose a bidirectional encoder-decoder model using dual-stage attention to address the above problems. In the encoder, the input attention mechanism extracts relevant dependent variables by referring to the hidden state and cell state from the bidirectional LSTM of the previous time step. In the decoder, the attention mechanism is applied to the past values of the independent variable but it works differently with the first stage (encoder). A bidirectional LSTM runs through until the defined time step and the hidden state in the decoder is updated at each time step. The updated hidden state combines with the encoded input are used as input in the decoder. With the proposed method, the decoder can capture the information throughout the encoder. It learns a trend of independent variable efficiently and can make a better prediction in comparison with other encoder-decoder models. For the evaluation, the Korean stock market 5-minute trading data is used.제 1장 서론 1 제 2장 관련 연구 4 2.1 시계열 분석에 대한 선행 연구 4 2.2 Encoder-Decoder 모델에 대한 선행 연구 6 2.3 어텐션 메커니즘에 대한 선행 연구 9 제 3장 제안하는 방법 13 3.1 제안하는 Encoder-Decoder의 구조 13 3.2 Encoder의 내부구조 15 3.3 Decoder의 내부구조 17 제 4장 실험 결과 20 4.1 데이터 설명 20 4.2 데이터 전처리 및 학습 방법 22 4.3 성능 평가 및 비교 25 4.4 어텐션 가중치 분석 28 제 5장 결론 35 참고문헌 37 Abstract 42Maste

Sistema de predicción del Recurso Solar aplicado a centrales termosolares

Premio extraordinario de Trabajo Fin de Máster curso 2012-2013.Energías Renovables DistribuidasLa consecución global de este proyecto conlleva desarrollar un sistema de predicción a corto plazo del recurso solar, focalizando el resultado final a la extracción y transformación de la radiación directa. Más concretamente, este proyecto consiste en un desarrollo informático que permite calcular la superficie sombreada1 de un campo solar por acción de los obstáculos transitorios que constituyen las nubes. El origen de datos del sistema se compone de imágenes de frecuencia minutal provenientes de una cámara de nubes, datos de altura y composición de aerosoles del aire facilitados por un ceilómetro e información de radiación cedida por un pirheliómetro. La herramienta desarrollada analiza el contenido de una fotografía de la bóveda celeste realizando las siguientes sub-tareas: Filtrado paso alto para realzar los bordes, reducción del área a procesar en función del foco de visión deseado, cálculo de la posición de los elementos capturados que forman parte de la estructura de la cámara y aplicación de una máscara, extrapolación de la información de la imagen oculta tras los elementos de la estructura , aplicación de un algoritmo que determina qué información de la imagen se corresponde con cielo y qué parte se corresponde con una nube, cálculo de la posición relativa del Sol en la imagen y la zona de bóveda que infiere sobre el campo solar descartando el resto de información, superposición de las nubes sobre el campo solar y cálculo de la potencia solar teórica capturada.The global aim of this project is developing a short time solar forecasting system (nowcasting), focusing on the direct irradiation extraction. Within this global target, this project is a computer tool to calculate the shading surface of a solar field produced for transient obstacles, better said, the clouds. The system data source includes minutal frequency images from a total sky camera, information about height and air composition given for a ceilometer and irradiation data taken from a pyrheliometer. The developed tool analyses a whole sky picture and does several tasks: high pass filtering to encouraging the edges, reduction of processing area based on focus of vision parameter, detection of the camera arms position and application of a mask, calculation of hidden information by the arms, application of an algorithm in order to determinate the information of each pixel (if it is a cloud or a sky zone), calculation of relative solar position within the picture and the sky area involved in the solar field shading and ignoring the rest, superimposing the clouds over the solar field and finally, calculation of the theorical solar power captured