Solar power harbors immense potential in mitigating climate change by
substantially reducing CO2​ emissions. Nonetheless, the inherent
variability of solar irradiance poses a significant challenge for seamlessly
integrating solar power into the electrical grid. While the majority of prior
research has centered on employing purely time series-based methodologies for
solar forecasting, only a limited number of studies have taken into account
factors such as cloud cover or the surrounding physical context. In this paper,
we put forth a deep learning architecture designed to harness spatio-temporal
context using satellite data, to attain highly accurate \textit{day-ahead}
time-series forecasting for any given station, with a particular emphasis on
forecasting Global Horizontal Irradiance (GHI). We also suggest a methodology
to extract a distribution for each time step prediction, which can serve as a
very valuable measure of uncertainty attached to the forecast. When evaluating
models, we propose a testing scheme in which we separate particularly difficult
examples from easy ones, in order to capture the model performances in crucial
situations, which in the case of this study are the days suffering from varying
cloudy conditions. Furthermore, we present a new multi-modal dataset gathering
satellite imagery over a large zone and time series for solar irradiance and
other related physical variables from multiple geographically diverse solar
stations. Our approach exhibits robust performance in solar irradiance
forecasting, including zero-shot generalization tests at unobserved solar
stations, and holds great promise in promoting the effective integration of
solar power into the grid