A Review of Solar Forecasting Techniques and the Role of Artificial Intelligence

Solar energy forecasting is essential for the effective integration of solar power into electricity grids and the optimal management of renewable energy resources. Distinguishing itself from the existing literature, this review study provides a nuanced contribution by centering on advancements in forecasting techniques. While preceding reviews have examined factors such as meteorological input parameters, time horizons, the preprocessing methodology, optimization, and sample size, our study uniquely delves into a diverse spectrum of time horizons, spanning ultrashort intervals (1 min to 1 h) to more extended durations (up to 24 h). This temporal diversity equips decision makers in the renewable energy sector with tools for enhanced resource allocation and refined operational planning. Our investigation highlights the prominence of Artificial Intelligence (AI) techniques, specifically focusing on Neural Networks in solar energy forecasting, and we review supervised learning, regression, ensembles, and physics-based methods. This showcases a multifaceted approach to address the intricate challenges associated with solar energy predictions. The integration of Satellite Imagery, weather predictions, and historical data further augments precision in forecasting. In assessing forecasting models, our study describes various error metrics. While the existing literature discusses the importance of metrics, our emphasis lies on the significance of standardized datasets and benchmark methods to ensure accurate evaluations and facilitate meaningful comparisons with naive forecasts. This study stands as a significant advancement in the field, fostering the development of accurate models crucial for effective renewable energy planning and emphasizing the imperative for standardization, thus addressing key gaps in the existing research landscape

Spatio-temporal solar forecasting

Current and future photovoltaic (PV) deployment levels require accurate forecasting to ensure grid stability. Spatio-temporal solar forecasting is a recent solar forecasting approach that explores spatially distributed solar data sets, either irradiance or photovoltaic power output, modeling cloud advection patterns to improve forecasting accuracy. This thesis contributes to further understanding of the potential and limitations of this approach, for different spatial and temporal scales, using different data sources; and its sensitivity to prevailing local weather patterns. Three irradiance data sets with different spatial coverages (from meters to hundreds of kilometers) and time resolutions (from seconds to days) were investigated using linear autoregressive models with external inputs (ARX). Adding neighboring data led to accuracy gains up to 20-40 % for all datasets. Spatial patterns matching the local prevailing winds could be identified in the model coefficients and the achieved forecast skill whenever the forecast horizon was of the order of scale of the distance between sensors divided by cloud speed. For one of the sets, it was shown that the ARX model underperformed for non-prevailing winds. Thus, a regime-based approach driven by wind information is proposed, where specialized models are trained for different ranges of wind speed and wind direction. Although forecast skill improves by up to 55.2 % for individual regimes, the overall improvement is only of 4.3 %, as those winds have a low representation in the data. By converting the highest resolution irradiance data set to PV power, it was also shown that forecast accuracy is sensitive to module tilt and orientation. Results are shown to be correlated with the difference in tilt and orientation between systems, indicating that clear-sky normalization is not totally effective in removing the geometry dependence of solar irradiance. Thus, non-linear approaches, such as machine learning algorithms, should be tested for modelling the non-linearity introduced by the mounting diversity from neighboring systems in spatio-temporal forecasting

Probabilistic Space Weather Modeling and Forecasting for the Challenge of Orbital Drag in Space Traffic Management

In the modern space age, private companies are crowding the already-congested low Earth orbit (LEO) regime with small satellite mega constellations. With over 25,000 objects larger than 10 cm already in LEO, this rapid expansion is forcing us towards the enterprise on Space Traffic Management (STM). STM is an operational effort that focuses on conjunction assessment and collision avoidance between objects. While the equations of motion for objects in orbit are well-known, there are many uncertain parameters that result in the uncertainty of an object\u27s future position. The force that the atmosphere exerts on satellite - known as drag - is the largest source of uncertainty in LEO. This is largely due to the difficulty in predicting mass density in the thermosphere - the neutral region in Earth\u27s upper atmosphere. Presently, most thermosphere models are deterministic and the treatment of uncertainty in density is highly simplified or nonexistent in operations. In this work, four probabilistic thermospheric mass density models are developed using machine learning (ML) to enable the investigation of the impact of model uncertainty on satellite position for the first time. Of these four models, two (HASDM-ML and TIE-GCM ROPE) are reduced order models based on outputs from existing thermosphere models while the other two (CHAMP-ML and MSIS-UQ) are based on in-situ thermosphere measurements. The data and model development are described, and the models\u27 capabilities, including the robustness of their uncertainty quantification (UQ) capabilities, are thoroughly assessed. Existing thermosphere models, and the ones developed here, use different space weather drivers to estimate density. In a forecasting environment, there are algorithms and models that forecast the drivers for a given period in order for a density model to make a forecast. The driver forecast models used by the United States Space Force for the HASDM system are assessed to benchmark our current capabilities. Using the error statistics for each driver, we can perturb the deterministic forecasts. This provides an avenue to use the ML thermosphere models to study the effect of driver uncertainty on satellite position, in addition to model uncertainty, for any period with available driver forecasts. Seven periods are considered with diverse space weather conditions to study the isolated effects of the two density uncertainty sources on a 72-hour satellite orbit. This provides insight into the relative importance of density uncertainty on satellite position for various space weather scenarios. This study also functions as a motivation to reconsider our current methods for STM in order to improve our capabilities and prevent future satellite collisions with increased confidence

고해상도 초분광영상을 활용한 하천 부유사농도 계측기법 개발

학위논문(박사) -- 서울대학교대학원 : 공과대학 건설환경공학부, 2022. 8. 서일원.기존의 하천 부유사 농도 계측은 샘플링 기반 직접계측 방식에 의존하여 시공간적 고해상도 자료 취득이 어려운 실정이다. 이러한 한계점을 극복하기 위해 최근 위성과 드론을 활용하여 촬영된 다분광 혹은 초분광 영상을 통해 고해상도의 부유사농도 시공간분포를 계측하는 기법에 대한 연구가 활발히 진행되고 있다. 하지만, 다른 하천 물리량 계측에 비해 부유사 계측 연구는 하천에 따라 부유사가 다양하게 분포하고 다른 부유물질 혹은 하상에 의한 바닥 반사의 영향 때문에 분광 자료를 통해 정확한 부유사농도 분포를 재현하기 어려운 실정이다. 특히, 부유사 분광 특성에 영향을 미치는 입도분포, 광물특성, 침강성 등이 하천에 따라 강한 지역성을 나타내기에 이러한 요인에서 야기되는 분광다양성으로 인해 특정 시기와 지역에만 적합한 원격탐사 기반 계측 모형들이 개발되어 왔다. 본 연구에서는 이러한 분광다양성을 반영하여 다양한 하천 및 유사 조건에서 적용 가능한 고해상도 초분광영상을 활용한 하천 부유사농도 계측방법을 개발하기 위해 초분광 군집화 기법과 다양한 파장대의 분광 밴드를 학습할 수 있는 기계학습 회귀 모형을 결합하여 CMR-OV라는 방법론을 제시하였다. CMR-OV 개발 및 검증은 1) 실험적 연구를 통한 하천 부유사 분광 특성의 주요 교란 요인 분석, 2) 최적 회귀모형 선정 및 초분광 클러스터링과의 결합, 3) 현장적용성 평가의 과정을 거쳐 수행되었다. 실험적 연구에서는 우선 실내 실험실에서 횡방향 혼합기를 활용하여 바닥 반사를 제거하고 완전 혼합된 상태에서 부유사의 고유 초분광 스펙트럼 자료를 수집하였다. 이를 바탕으로 실제 하천과 유사한 조건의 실규모 옥외 수로 실험에서 다양한 유사 특성(입도 및 광물)과 하상 특성(식생 및 모래)에 대한 초분광 자료를 수집하여 고유 초분광 스펙트럼과 비교하였다. 그 결과, 부유사의 분광 특성은 유사의 종류 및 입도에 따라 농도 증가에 따른 초분광 스펙트럼의 반사율 변화가 상이하게 나타났다. 또한, 1 m 이하의 얕은 수심 조건에서는 바닥 반사의 영향으로 하상 종류에 따라 초분광 스펙트럼의 개형이 크게 변화하였으며, 고농도의 부유사가 분포할 때도 바닥 반사가 크게 영향을 미치는 것을 확인하였다. 이러한 분광다양성이 반영된 부유사농도와 초분광 자료의 관계를 구축하기 위하여 기계학습 기반 랜덤포레스트 회귀 모형과 가우시안 혼합 모형 기반 초분광 군집 기법을 결합한 CMR-OV를 적용한 결과, 기존 연구들에서 주로 활용된 밴드비 기반의 모형과 단일 기계학습모형에 비해 정확도가 크게 향상하였다. 특히, 기존 최적 밴드비 분석 (OBRA) 방법은 비선형성을 고려해도 좁은 영역의 파장대만을 고려하는 한계점으로 인해 분광다양성을 반영하지 못하는 것으로 밝혀졌다. 하지만, CMR-OV는 폭 넓은 파장대 영역을 고려함과 동시에 높은 정확도를 산출하였다. 최종적으로 CMR-OV를 황강의 직선구간 및 사행구간과 낙동강과 황강의 합류부에 적용하여 현장검증을 수행한 결과, 기존 모형 대비 정확도와 부유사 농도 맵핑의 정밀성에서 큰 개선이 있었으며, 비학습지역에서도 높은 정확도를 산출하였다. 특히, 하천 합류부에서는 초분광 군집을 통해 두 하천 흐름의 경계층을 명확히 구별하였으며, 이를 바탕으로 지류와 본류에 대해 각각 분리된 회귀모형을 구축하여 복잡한 합류부 근역 경계층에서의 부유사 분포를 보다 정확하게 재현하였다. 또한, 나아가서 재현된 고해상도의 부유사 공간분포를 바탕으로 혼합도를 산정한 결과, 기존 점계측 대비 상세하게 부유사 혼합에 대한 정량적 평가가 가능한 것으로 나타났다. 따라서, 본 연구에서 개발한 초분광영상 기반 부유사 계측 기술을 통해 추후 하천 조사 및 관리 실무의 정확성 및 효율성을 크게 증진할 수 있을 것으로 기대된다.The conventional measurement method of suspended sediment concentration (SSC) in the riverine system is labor-intensive and time-consuming since it has been conducted using the sampling-based direct measurement method. For this reason, it is challenging to collect high-resolution datasets of SSC in rivers. In order to overcome this limitation, remote sensing-based techniques using multi- or hyper-spectral images from satellites or UAVs have been recently carried out to obtain high-resolution SSC distributions in water environments. However, suspended sediment in rivers is more dynamic and spatially heterogeneous than those in other fields. Moreover, the sediment and streambed properties have strong regional characteristics depending on the river type; thus, only models suitable for a specific period and region have been developed owing to the increased spectral variability of the water arising from various types of suspended matter in the water and the heterogeneous streambed properties. Therefore, to overcome the limitations of the existing monitoring system, this study proposed a robust hyperspectral imagery-based SSC measurement method, termed cluster-based machine learning regression with optical variability (CMR-OV). This method dealt with the spectral variability problem by combining hyperspectral clustering and machine learning regression with the Gaussian mixture model (GMM) and Random forest (RF) regression. The hyperspectral clustering separated the complex dataset into several homogeneous datasets according to spectral characteristics. Then, the machine learning regressors corresponding to clustered datasets were built to construct the relationship between the hyperspectral spectrum and SSC. The development and validation of the proposed method were carried out through the following processes: 1) analysis of confounding factors in the spectral variability through experimental studies, 2) selection of an optimal regression model and validation of hyperspectral clustering, and 3) evaluation of field applicability. In the experimental studies, the intrinsic hyperspectral spectra of suspended sediment were collected in a completely mixed state after removing the bottom reflection using a horizontal rotating cylinder. Then, hyperspectral data on various sediment properties (particle size and mineral contents) and river bed properties (sand and vegetation) were collected from sediment tracer experiments in field-scale open channels under different hydraulic conditions and compared with intrinsic hyperspectral spectra. Consequently, the change of the hyperspectral spectrum was different according to the sediment type and particle size distribution. In addition, under the shallow water depth condition of 1 m or less, the shape of the hyperspectral spectrum changed significantly depending on the bed type due to the bottom reflectance. The bottom reflectance substantially affected the hyperspectral spectrum even when the high SSC was distributed. As a result of combining the GMM and RF regression with building a relationship between the SSC and hyperspectral data reflecting the spectral variability, the accuracy was substantially improved compared to the other methods. In particular, even when nonlinearity is considered based on the existing optimal band ratio analysis (OBRA) method, spectral variability could not be reflected due to the limitation of considering only a narrow wavelength range. On the other hand, CMR-OV showed high accuracy while considering a wide range of wavelengths with clusters having distinct spectral characteristics. Finally, the CMR-OV was applied to the straight and meandering reaches of the Hwang River and the confluence of the Nakdong and Hwang Rivers in South Korea to assess field applicability. There was a remarkable improvement in the accuracy and precision of SSC mapping under various river conditions compared to the existing models, and CMR-OV showed robust performance even with non-calibrated datasets. At the river confluence, the mixing pattern between the main river and tributary was apparently retrieved from CMR-OV under optically complex conditions. Compared to the non-clustered model, hyperspectral clustering played a primary role in improving the performance by separating the water bodies originating from both rivers. It was also possible to quantitatively evaluate the complicated mixing pattern in detail compared to the existing point measurement. Therefore, it is expected that the accuracy and efficiency of river investigation will be significantly improved through the SSC measurement method presented in this study.Abstract of dissertation i List of figures ix List of tables xvii List of abbreviations xix List of symbols xxii 1. Introduction 1 2. Theoretical research 13 2.1.1 Pre-processing of hyperspectral image (HSI) 19 2.1.2 Optical characteristics of suspended sediment in rivers 28 2.1.2.1 Theory of solar radiation transfer in rivers 28 2.1.2.2 Heterogeneity of sediment properties 33 2.1.2.3 Effects of bottom reflectance 38 2.1.2.4 Vertical distribution of suspended sediment 41 2.1.3 Retrieval of suspended sediment from remote sensing data 46 2.1.3.1 Remote sensing-based regression approach 46 2.1.3.2 Clustering of remote sensing data 52 2.2 Mapping of suspended sediment concentration in rivers 56 2.2.1 Traditional method for spatial measurement 56 2.2.2 Spatial measurement at river confluences 57 2.2.2.1 Dynamics of flow and mixing at river confluences 57 2.2.2.2 Field experiments in river confluences 64 3. Experimental studies 68 3.1 Experimental cases 68 3.2 Laboratory experiment 74 3.2.1 Experimental setup 74 3.2.2 Experimental method 78 3.3 Field-scale experiments in River Experiment Center 83 3.3.1 Experiments in the straight channel 83 3.3.2 Experiments in the meandering channel 96 3.4 Field survey 116 3.4.1 Study area and field measurement 116 3.4.2 Hydraulic and sediment data in rivers with simple geometry 122 3.4.3 Hydraulic and sediment data in river confluences 126 3.5 Analysis of hyperspectral data of suspended sediment 141 3.5.1 Hyperspectral data of laboratory experiment 142 3.5.2 Hyperspectral data of field-scale experiments 146 3.5.2.1 Effect of bottom reflectance 146 3.5.2.2 Principal component analysis of the effect of suspended sediment properties 154 4. Development of suspended sediment concentration estimator using UAV-based hyperspectral imagery 164 4.1 Outline of Cluster-based Machine learning Regression with Optical Variability (CMR-OV) 164 4.2 Pre-processing of hyperspectral images 168 4.3 Regression models and clustering technique 173 4.3.1 Index-based regression models 173 4.3.2 Machine learning regression models 175 4.3.3 Relevant band selection 183 4.3.4 Gaussian mixture model for clustering 185 4.3.5 Performance criteria 188 4.4 Model development and evaluation 189 4.4.1 Comparison of regression models 189 4.4.1.1 OBRA-based explicit models 189 4.4.1.2 Machine learning-based implicit models 194 4.4.2 Assessment of hyperspectral clustering 200 4.4.3 Spatio-temporal SSCV mapping using CMR-OV 215 5. Evaluation of field applicability of CMR-OV 225 5.1 Outline of field applicability test 225 5.2 Cross-applicability validation of CMR-OV 227 5.3 Assessment of field applicability in rivers with simple geometry 234 5.4 Assessment of field applicability in river confluences 241 5.4.1 Classification of river regions using hyperspectral clustering 241 5.4.2 Retrievals of SSCV map 258 5.4.3 Mixing pattern extraction from SSCv map 271 6. Conclusions and future study 274 6.1 Conclusions 274 6.2 Future directions 278 Reference 280 Appendix 308 Appendix A. Breakthrough curve (BTC) analysis 308 Appendix B. Experimental data 310 Appendix B. 1. BTCs of in-situ measured SSC from field-scale experiments 310 Appendix B. 2. Dataset of spectra from hyperspectral images and corresponding SSC in rivers 330 Appendix C. CMR-OV code 331 국문초록 337박

The 8th International Conference on Time Series and Forecasting

The aim of ITISE 2022 is to create a friendly environment that could lead to the establishment or strengthening of scientific collaborations and exchanges among attendees. Therefore, ITISE 2022 is soliciting high-quality original research papers (including significant works-in-progress) on any aspect time series analysis and forecasting, in order to motivating the generation and use of new knowledge, computational techniques and methods on forecasting in a wide range of fields

Estimating the concentration of physico chemical parameters in hydroelectric power plant reservoir

The United Nations Educational, Scientific and Cultural Organization (UNESCO) defines the amazon region and adjacent areas, such as the Pantanal, as world heritage territories, since they possess unique flora and fauna and great biodiversity. Unfortunately, these regions have increasingly been suffering from anthropogenic impacts. One of the main anthropogenic impacts in the last decades has been the construction of hydroelectric power plants. As a result, dramatic altering of these ecosystems has been observed, including changes in water levels, decreased oxygenation and loss of downstream organic matter, with consequent intense land use and population influxes after the filling and operation of these reservoirs. This, in turn, leads to extreme loss of biodiversity in these areas, due to the large-scale deforestation. The fishing industry in place before construction of dams and reservoirs, for example, has become much more intense, attracting large populations in search of work, employment and income. Environmental monitoring is fundamental for reservoir management, and several studies around the world have been performed in order to evaluate the water quality of these ecosystems. The Brazilian Amazon, in particular, goes through well defined annual hydrological cycles, which are very importante since their study aids in monitoring anthropogenic environmental impacts and can lead to policy and decision making with regard to environmental management of this area. The water quality of amazon reservoirs is greatly influenced by this defined hydrological cycle, which, in turn, causes variations of microbiological, physical and chemical characteristics. Eutrophication, one of the main processes leading to water deterioration in lentic environments, is mostly caused by anthropogenic activities, such as the releases of industrial and domestic effluents into water bodies. Physico-chemical water parameters typically related to eutrophication are, among others, chlorophyll-a levels, transparency and total suspended solids, which can, thus, be used to assess the eutrophic state of water bodies. Usually, these parameters must be investigated by going out to the field and manually measuring water transparency with the use of a Secchi disk, and taking water samples to the laboratory in order to obtain chlorophyll-a and total suspended solid concentrations. These processes are time- consuming and require trained personnel. However, we have proposed other techniques to environmental monitoring studies which do not require fieldwork, such as remote sensing and computational intelligence. Simulations in different reservoirs were performed to determine a relationship between these physico-chemical parameters and the spectral response. Based on the in situ measurements, empirical models were established to relate the reflectance of the reservoir measured by the satellites. The images were calibrated and corrected atmospherically. Statistical analysis using error estimation was used to evaluate the most accurate methodology. The Neural Networks were trained by hydrological cycle, and were useful to estimate the physicalchemical parameters of the water from the reflectance of visible bands and NIR of satellite images, with better results for the period with few clouds in the regions analyzed. The present study shows the application of wavelet neural network to estimate water quality parameters using concentration of the water samples collected in the Amazon reservoir and Cefni reservoir, UK. Sattelite imagens from Landsats and Sentinel-2 were used to train the ANN by hydrological cycle. The trained ANNs demonstrated good results between observed and estimated after Atmospheric corrections in satellites images. The ANNs showed in the results are useful to estimate these concentrations using remote sensing and wavelet transform for image processing. Therefore, the techniques proposed and applied in the present study are noteworthy since they can aid in evaluating important physico-chemical parameters, which, in turn, allows for identification of possible anthropogenic impacts, being relevant in environmental management and policy decision-making processes. The tests results showed that the predicted values have good accurate. Improving efficiency to monitor water quality parameters and confirm the reliability and accuracy of the approaches proposed for monitoring water reservoirs. This thesis contributes to the evaluation of the accuracy of different methods in the estimation of physical-chemical parameters, from satellite images and artificial neural networks. For future work, the accuracy of the results can be improved by adding more satellite images and testing new neural networks with applications in new water reservoirs

Wind Power Prediction with Machine Learning Methods in Complex Terrain Areas

The increasing amount of intermittant wind energy sources connected to the power grid present several challenges in balancing the power network. Accurate prediction of wind power production is identified as one of the most important measures for balancing the power network while maintaining a sustainable integration of wind power in the power grid. However, the volatile nature of wind makes wind power forecasting a complicated task, and it is known that the performance of already established wind power prediction models decreases for wind farms in complex terrain sites. This thesis aims to forecast the future wind power output for five different wind farms in Northern Norway using methods from statistics and machine learning. The wind farm sites are generally characterized as complex terrain areas with good wind resources. Four different prediction models are developed for short to medium-term, multi- step prediction of wind power, ranging from traditional statistical models such as the arimax process to complex machine learning models. Additionally, two of the models are implemented both using the recursive and the direct multi- step forecasting technique. For each wind farm, the models are evaluated for an entire year and utilize multivariate input data with variables from a nwp model. The results of the experiments varied greatly across all locations. It was seen that the implemented models were outperformed by the persistence model for short forecasting horizons. However, when the forecasting horizon increased, several models showed a lower error than the persistence model

Radiation-Based Analytic Approaches to Investigate the Earth’s Atmosphere

Radiation, propagating through Earth’s atmosphere, plays an important role in the Earth system. Solar radiation is the major source of energy, followed by thermal infrared radiation emitted by the Earth. The total radiative energy budget affects dynamic, thermodynamics, photochemical and biological processes. In addition, by measuring the reflected and emitted radiation at a distance (e.g., satellite or aircraft), we can detect and monitor the physical characteristics of a region which can help researchers get a better understanding of Earth’s atmosphere. Therefore, radiation-based analytic approaches are powerful tools in Earth Science. This thesis focuses on using radiation-based analytic tools to study the Earth’s atmosphere and to understand human impacts on the Earth system. First, we develop novel machine learning methods for hyperspectral radiative transfer simulations. Hyperspectral technique is one of the most popular and powerful methods for atmospheric remote sensing and is widely used for temperature, gas, aerosol, and cloud retrievals. However, accurate forward radiative transfer simulations are computationally expensive since they require a larger number of monochromatic radiative transfer calculations. We, therefore explore the feasibility of machine learning techniques for fast hyperspectral radiative transfer simulations that perform calculations at a small fraction of hyperspectral wavelengths and extend them across the entire spectral range. The machine learning-based approach achieves better performance than the traditional principal component analysis (PCA) method. Second, we evaluate modeled hyperspectral infrared spectra against satellite all-sky observations. The national weather centers obtain data from hyperspectral infrared sounders on a global scale. The cloudless scenario of this data is used to initialize weather forecasts, including temperature, water vapor, water cloud, and ice cloud profiles on a global grid. Although the data from these satellites are sensitive to the vertical distribution of ice and liquid water in the clouds, this information is not fully utilized. In this study, we evaluate how well the modeled spectra compare to AIRS observations using different cloud overlap models. We hope that this information can be used to verify clouds in the National Meteorological Center model and to initialize forecasts in the future. In the last chapter, we use radiation-based analytic approaches to study human impacts on the Earth system. In the first study case, we show that the radiative forcing due to geospatially redistributed anthropogenic aerosols mainly determined the spatial variations of winter extreme weather in the Northern Hemisphere during 1970-2005, which is a unique transition period for global aerosol forcing. In the second case, we review satellite and ground-based observations and conduct state-of-art atmospheric model simulations during the COVID-19 lockdown period. The halted human activities during the COVID-19 pandemic in China provided a unique experiment to assess the efficiency of air-pollution mitigation.</p