111 research outputs found

    Exploiting radar polarimetry for nowcasting thunderstorm hazards using deep learning

    Get PDF
    This work presents the importance of polarimetric variables as an additional data source for nowcasting thunderstorm hazards using an existing neural network architecture with recurrent-convolutional layers. The model can be trained to predict different target variables, which enables nowcasting of hail, lightning, and heavy rainfall for lead times up to 60 min with a 5 min resolution, in particular. The exceedance probabilities of Swiss thunderstorm warning thresholds are predicted. This study is based on observations from the Swiss operational radar network, which consists of five operational polarimetric C-band radars. The study area of the Alpine region is topographically complex and has a comparatively very high thunderstorm activity. Different model runs using combinations of single- and dual-polarimetric radar observations and radar quality indices are compared to the reference run using only single-polarimetric observations. Two case studies illustrate the performance difference when using all predictors compared to the reference model. The importance of the predictors is quantified by investigating the final training loss of the model, with skill scores such as critical success index (CSI), precision, recall, precision–recall area under the curve, and the Shapley value. Results indicate that single-polarization radar data are the most important data source. Adding polarimetric observations improves the model performance compared to reference model in term of the training loss for all three target variables. Adding quality indices does so, too. Including both polarimetric variables and quality indices at the same time improves the accuracy of nowcasting heavy precipitation and lightning, with the largest improvement found for heavy precipitation. No improvement could be achieved for nowcasting of the probability of hail in this way.</p

    Towards nowcasting in Europe in 2030

    Get PDF
    The increasing impact of severe weather over Europe on lives and weathersensitive economies can be mitigated by accurate 0–6 h forecasts (nowcasts), supporting a vital ‘last line of defence’ for civil protection and many other applications. Recognizing lack of skill in some complex situations, often at convective and local sub-kilometre scales and associated with rare events, we identify seven recommendations with the aim to improve nowcasting in Europe by the national meteorological and hydrological services (NMHSs) by 2030. These recommendations are based on a review of user needs, the state of the observing system, techniques based on observations and high-resolution numerical weather models, as well as tools, data and infrastructure supporting the nowcasting community in Europe. Denser and more accurate observations are necessary particularly in the boundary layer to better characterize the ingredients of severe storms. A key driver for improvement is next-generation European satellite data becoming available as of 2023. Seamless ensemble prediction methods to produce enhanced weather forecasts with 0–24 h lead times and probabilistic products require further development. Such products need to be understood and interpreted by skilled forecasters operating in an evolving forecasting context

    Short-term irradiance nowcasting based on camera and satellite images

    Get PDF

    Deep learning-based hybrid short-term solar forecast using sky images and meteorological data

    Get PDF
    The global growth of solar power generation is rapid, yet the complex nature of cloud movement introduces significant uncertainty to short-term solar irradiance, posing challenges for intelligent power systems. Accurate short-term solar irradiance and photovoltaic power generation predictions under cloudy skies are critical for sub-hourly electricity markets. Ground-based image (GSI) analysis using convolutional neural network (CNN) algorithms has emerged as a promising method due to advancements in machine vision models based on deep learning networks. In this work, a novel deep network, ”ViT-E,” based on an attention mechanism Transformer architecture for short-term solar irradiance forecasting has been proposed. This innovative model enables cross-modality data parsing by establishing mapping relationships within GSI and between GSI, meteorological data, historical irradiation, clear sky irradiation, and solar angles. The feasibility of the ViT-E network was assessed the Folsom dataset from California, USA. Quantitative analysis showed that the ViT-E network achieved RMSE values of 81.45 W/m2 , 98.68 W/m2 , and 104.91 W/m2 for 2, 6, and 10-minute forecasts, respectively, outperforming the persistence model by 4.87%, 16.06%, and 19.09% and displaying performance comparable to CNN-based models. Qualitative analysis revealed that the ViT-E network successfully predicted 20.21%, 33.26%, and 36.87% of solar slope events at 2, 6, and 10 minutes in advance, respectively, significantly surpassing the persistence model and currently prevalent CNN-based model by 9.43%, 3.91%, and -0.55% for 2, 6, and 10-minute forecasts, respectively. Transfer learning experiments were conducted to test the ViT-E model’s generalisation under different climatic conditions and its performance on smaller datasets. We discovered that the weights learned from the three-year Folsom dataset in the United States could be transferred to a half-year local dataset in Nottingham, UK. Training with a dataset one-fifth the size of the original dataset achieved baseline accuracy standards and reduced training time by 80.2%. Additionally, using a dataset equivalent to only 4.5% of the original size yielded a model with less than 2% accuracy below the baseline. These findings validated the generalisation and robustness of the model’s trained weights. Finally, the ViT-E model architecture and hyperparameters were optimised and searched. Our investigation revealed that directly applying migrated deep vision models leads to redundancy in solar forecasting. We identified the best hyperparameters for ViT-E through manual hyperparameter space exploration. As a result, the model’s computational efficiency improved by 60%, and prediction performance increased by 2.7%

    Inductive biases in deep learning models for weather prediction

    Full text link
    Deep learning has recently gained immense popularity in the Earth sciences as it enables us to formulate purely data-driven models of complex Earth system processes. Deep learning-based weather prediction (DLWP) models have made significant progress in the last few years, achieving forecast skills comparable to established numerical weather prediction (NWP) models with comparatively lesser computational costs. In order to train accurate, reliable, and tractable DLWP models with several millions of parameters, the model design needs to incorporate suitable inductive biases that encode structural assumptions about the data and modelled processes. When chosen appropriately, these biases enable faster learning and better generalisation to unseen data. Although inductive biases play a crucial role in successful DLWP models, they are often not stated explicitly and how they contribute to model performance remains unclear. Here, we review and analyse the inductive biases of six state-of-the-art DLWP models, involving a deeper look at five key design elements: input data, forecasting objective, loss components, layered design of the deep learning architectures, and optimisation methods. We show how the design choices made in each of the five design elements relate to structural assumptions. Given recent developments in the broader DL community, we anticipate that the future of DLWP will likely see a wider use of foundation models -- large models pre-trained on big databases with self-supervised learning -- combined with explicit physics-informed inductive biases that allow the models to provide competitive forecasts even at the more challenging subseasonal-to-seasonal scales

    Reproducible and relocatable regional ocean modelling: fundamentals and practices

    Get PDF
    In response to an increasing demand for bespoke or tailored regional ocean modelling configurations, we outline fundamental principles and practices that can expedite the process to generate new configurations. The paper develops the principle of reproducibility and advocates adherence by presenting benefits to the community and user. The elements of this principle are reproducible workflows and standardised assessment, with additional effort over existing working practices being balanced against the added value generated. The paper then decomposes the complex build process, for a new regional ocean configuration, into stages and presents guidance, advice and insight for each component. This advice is compiled from across the NEMO (Nucleus for European Modelling of the Ocean) user community and sets out principles and practises that encompass regional ocean modelling with any model. With detailed and region-specific worked examples in Sects. 3 and 4, the linked companion repositories and DOIs all target NEMOv4. The aim of this review and perspective paper is to broaden the user community skill base and to accelerate development of new configurations in order to increase the time available for exploiting the configurations

    Smart models to improve agrometeorological estimations and predictions

    Get PDF
    La población mundial, en continuo crecimiento, alcanzará de forma estimada los 9,7 mil millones de habitantes en el 2050. Este incremento, combinado con el aumento en los estándares de vida y la situación de emergencia climática (aumento de la temperatura, intensificación del ciclo del agua, etc.) nos enfrentan al enorme desafío de gestionar de forma sostenible los cada vez más escasos recursos disponibles. El sector agrícola tiene que afrontar retos tan importantes como la mejora en la gestión de los recursos naturales, la reducción de la degradación medioambiental o la seguridad alimentaria y nutricional. Todo ello condicionado por la escasez de agua y las condiciones de aridez: factores limitantes en la producción de cultivos. Para garantizar una producción agrícola sostenible bajo estas condiciones, es necesario que todas las decisiones que se tomen estén basadas en el conocimiento, la innovación y la digitalización de la agricultura de forma que se garantice la resiliencia de los agroecosistemas, especialmente en entornos áridos, semi-áridos y secos sub-húmedos en los que el déficit de agua es estructural. Por todo esto, el presente trabajo se centra en la mejora de la precisión de los actuales modelos agrometeorológicos, aplicando técnicas de inteligencia artificial. Estos modelos pueden proporcionar estimaciones y predicciones precisas de variables clave como la precipitación, la radiación solar y la evapotranspiración de referencia. A partir de ellas, es posible favorecer estrategias agrícolas más sostenibles, gracias a la posibilidad de reducir el consumo de agua y energía, por ejemplo. Además, se han reducido el número de mediciones requeridas como parámetros de entrada para estos modelos, haciéndolos más accesibles y aplicables en áreas rurales y países en desarrollo que no pueden permitirse el alto costo de la instalación, calibración y mantenimiento de estaciones meteorológicas automáticas completas. Este enfoque puede ayudar a proporcionar información valiosa a los técnicos, agricultores, gestores y responsables políticos de la planificación hídrica y agraria en zonas clave. Esta tesis doctoral ha desarrollado y validado nuevas metodologías basadas en inteligencia artificial que han ser vido para mejorar la precision de variables cruciales en al ámbito agrometeorológico: precipitación, radiación solar y evapotranspiración de referencia. En particular, se han modelado sistemas de predicción y rellenado de huecos de precipitación a diferentes escalas utilizando redes neuronales. También se han desarrollado modelos de estimación de radiación solar utilizando exclusivamente parámetros térmicos y validados en zonas con características climáticas similares a lugar de entrenamiento, sin necesidad de estar geográficamente en la misma región o país. Analógamente, se han desarrollado modelos de estimación y predicción de evapotranspiración de referencia a nivel local y regional utilizando también solamente datos de temperatura para todo el proceso: regionalización, entrenamiento y validación. Y finalmente, se ha creado una librería de Python de código abierto a nivel internacional (AgroML) que facilita el proceso de desarrollo y aplicación de modelos de inteligencia artificial, no solo enfocadas al sector agrometeorológico, sino también a cualquier modelo supervisado que mejore la toma de decisiones en otras áreas de interés.The world population, which is constantly growing, is estimated to reach 9.7 billion people in 2050. This increase, combined with the rise in living standards and the climate emergency situation (increase in temperature, intensification of the water cycle, etc.), presents us with the enormous challenge of managing increasingly scarce resources in a sustainable way. The agricultural sector must face important challenges such as improving natural resource management, reducing environmental degradation, and ensuring food and nutritional security. All of this is conditioned by water scarcity and aridity, limiting factors in crop production. To guarantee sustainable agricultural production under these conditions, it is necessary to based all the decision made on knowledge, innovation, and the digitization of agriculture to ensure the resilience of agroecosystems, especially in arid, semi-arid, and sub-humid dry environments where water deficit is structural. Therefore, this work focuses on improving the precision of current agrometeorological models by applying artificial intelligence techniques. These models can provide accurate estimates and predictions of key variables such as precipitation, solar radiation, and reference evapotranspiration. This way, it is possible to promote more sustainable agricultural strategies by reducing water and energy consumption, for example. In addition, the number of measurements required as input parameters for these models has been reduced, making them more accessible and applicable in rural areas and developing countries that cannot afford the high cost of installing, calibrating, and maintaining complete automatic weather stations. This approach can help provide valuable information to technicians, farmers, managers, and policy makers in key wáter and agricultural planning areas. This doctoral thesis has developed and validated new methodologies based on artificial intelligence that have been used to improve the precision of crucial variables in the agrometeorological field: precipitation, solar radiation, and reference evapotranspiration. Specifically, prediction systems and gap-filling models for precipitation at different scales have been modeled using neural networks. Models for estimating solar radiation using only thermal parameters have also been developed and validated in areas with similar climatic characteristics to the training location, without the need to be geographically in the same region or country. Similarly, models for estimating and predicting reference evapotranspiration at the local and regional level have been developed using only temperature data for the entire process: regionalization, training, and validation. Finally, an internationally open-source Python library (AgroML) has been created to facilitate the development and application of artificial intelligence models, not only focused on the agrometeorological sector but also on any supervised model that improves decision-making in other areas of interest

    Reproducible and relocatable regional ocean modelling: Fundamentals and practices

    Get PDF
    In response to an increasing demand for bespoke or tailored regional ocean modelling configurations, we outline fundamental principles and practices that can expedite the process to generate new configurations. The paper develops the principle of Reproducibility and advocates adherence by presenting benefits to the community and user. The elements to this principle are reproducible workflows and standardised assessment, with additional effort over existing working practices being balanced against the added value generated. The paper then decomposes the complex build process, for a new regional ocean configuration, into stages and presents guidance, advice and insight on each component. This advice is compiled from across the user community, is presented in the context of NEMOv4, though aims to transcend NEMO version. Detail and region specific worked examples are linked in companion repositories and DOIs. The aim is to broaden the user community skill base, and to accelerate development of new configurations in order to increase available time exploiting the configurations

    Methods in machine learning for probabilistic modelling of environment, with applications in meteorology and geology

    Get PDF
    Earth scientists increasingly deal with ‘big data’. Where once we may have struggled to obtain a handful of relevant measurements, we now often have data being collected from multiple sources, on the ground, in the air, and from space. These observations are accumulating at a rate that far outpaces our ability to make sense of them using traditional methods with limited scalability (e.g., mental modelling, or trial-and-error improvement of process based models). The revolution in machine learning offers a new paradigm for modelling the environment: rather than focusing on tweaking every aspect of models developed from the top down based largely on prior knowledge, we now have the capability to instead set up more abstract machine learning systems that can ‘do the tweaking for us’ in order to learn models from the bottom up that can be considered optimal in terms of how well they agree with our (rapidly increasing number of) observations of reality, while still being guided by our prior beliefs. In this thesis, with the help of spatial, temporal, and spatio-temporal examples in meteorology and geology, I present methods for probabilistic modelling of environmental variables using machine learning, and explore the considerations involved in developing and adopting these technologies, as well as the potential benefits they stand to bring, which include improved knowledge-acquisition and decision-making. In each application, the common theme is that we would like to learn predictive distributions for the variables of interest that are well-calibrated and as sharp as possible (i.e., to provide answers that are as precise as possible while remaining honest about their uncertainty). Achieving this requires the adoption of statistical approaches, but the volume and complexity of data available mean that scalability is an important factor — we can only realise the value of available data if it can be successfully incorporated into our models.Engineering and Physical Sciences Research Council (EPSRC

    Systematic Approaches for Telemedicine and Data Coordination for COVID-19 in Baja California, Mexico

    Get PDF
    Conference proceedings info: ICICT 2023: 2023 The 6th International Conference on Information and Computer Technologies Raleigh, HI, United States, March 24-26, 2023 Pages 529-542We provide a model for systematic implementation of telemedicine within a large evaluation center for COVID-19 in the area of Baja California, Mexico. Our model is based on human-centric design factors and cross disciplinary collaborations for scalable data-driven enablement of smartphone, cellular, and video Teleconsul-tation technologies to link hospitals, clinics, and emergency medical services for point-of-care assessments of COVID testing, and for subsequent treatment and quar-antine decisions. A multidisciplinary team was rapidly created, in cooperation with different institutions, including: the Autonomous University of Baja California, the Ministry of Health, the Command, Communication and Computer Control Center of the Ministry of the State of Baja California (C4), Colleges of Medicine, and the College of Psychologists. Our objective is to provide information to the public and to evaluate COVID-19 in real time and to track, regional, municipal, and state-wide data in real time that informs supply chains and resource allocation with the anticipation of a surge in COVID-19 cases. RESUMEN Proporcionamos un modelo para la implementación sistemática de la telemedicina dentro de un gran centro de evaluación de COVID-19 en el área de Baja California, México. Nuestro modelo se basa en factores de diseño centrados en el ser humano y colaboraciones interdisciplinarias para la habilitación escalable basada en datos de tecnologías de teleconsulta de teléfonos inteligentes, celulares y video para vincular hospitales, clínicas y servicios médicos de emergencia para evaluaciones de COVID en el punto de atención. pruebas, y para el tratamiento posterior y decisiones de cuarentena. Rápidamente se creó un equipo multidisciplinario, en cooperación con diferentes instituciones, entre ellas: la Universidad Autónoma de Baja California, la Secretaría de Salud, el Centro de Comando, Comunicaciones y Control Informático. de la Secretaría del Estado de Baja California (C4), Facultades de Medicina y Colegio de Psicólogos. Nuestro objetivo es proporcionar información al público y evaluar COVID-19 en tiempo real y rastrear datos regionales, municipales y estatales en tiempo real que informan las cadenas de suministro y la asignación de recursos con la anticipación de un aumento de COVID-19. 19 casos.ICICT 2023: 2023 The 6th International Conference on Information and Computer Technologieshttps://doi.org/10.1007/978-981-99-3236-
    corecore