Exploring Interpretable LSTM Neural Networks over Multi-Variable Data

For recurrent neural networks trained on time series with target and exogenous variables, in addition to accurate prediction, it is also desired to provide interpretable insights into the data. In this paper, we explore the structure of LSTM recurrent neural networks to learn variable-wise hidden states, with the aim to capture different dynamics in multi-variable time series and distinguish the contribution of variables to the prediction. With these variable-wise hidden states, a mixture attention mechanism is proposed to model the generative process of the target. Then we develop associated training methods to jointly learn network parameters, variable and temporal importance w.r.t the prediction of the target variable. Extensive experiments on real datasets demonstrate enhanced prediction performance by capturing the dynamics of different variables. Meanwhile, we evaluate the interpretation results both qualitatively and quantitatively. It exhibits the prospect as an end-to-end framework for both forecasting and knowledge extraction over multi-variable data.Comment: Accepted to International Conference on Machine Learning (ICML), 201

Air quality and urban sustainable development: the application of machine learning tools

[EN] Air quality has an efect on a population¿s quality of life. As a dimension of sustainable urban development, governments have been concerned about this indicator. This is refected in the references consulted that have demonstrated progress in forecasting pollution events to issue early warnings using conventional tools which, as a result of the new era of big data, are becoming obsolete. There are a limited number of studies with applications of machine learning tools to characterize and forecast behavior of the environmental, social and economic dimensions of sustainable development as they pertain to air quality. This article presents an analysis of studies that developed machine learning models to forecast sustainable development and air quality. Additionally, this paper sets out to present research that studied the relationship between air quality and urban sustainable development to identify the reliability and possible applications in diferent urban contexts of these machine learning tools. To that end, a systematic review was carried out, revealing that machine learning tools have been primarily used for clustering and classifying variables and indicators according to the problem analyzed, while tools such as artifcial neural networks and support vector machines are the most widely used to predict diferent types of events. The nonlinear nature and synergy of the dimensions of sustainable development are of great interest for the application of machine learning tools.Molina-Gómez, NI.; Díaz-Arévalo, JL.; López Jiménez, PA. (2021). Air quality and urban sustainable development: the application of machine learning tools. International Journal of Environmental Science and Technology. 18(4):1-18. https://doi.org/10.1007/s13762-020-02896-6S118184Al-Dabbous A, Kumar P, Khan A (2017) Prediction of airborne nanoparticles at roadside location using a feed–forward artificial neural network. Atmos Pollut Res 8:446–454. https://doi.org/10.1016/j.apr.2016.11.004Antanasijević D, Pocajt V, Povrenović D, Ristić M, Perić-Grujić A (2013) PM10 emission forecasting using artificial neural networks and genetic algorithm input variable optimization. Sci Total Environ 443:511–519. https://doi.org/10.1016/j.scitotenv.2012.10.110Brink H, Richards JW, Fetherolf M (2016) Real-world machine learning. Richards JW, Fetherolf M (eds) Manning Publications Co. Berkeley, CA. https://www.manning.com/books/real-world-machine-learning. Accessed 26 Apr 2020Cervone G, Franzese P, Ezber Y, Boybeyi Z (2008) Risk assessment of atmospheric emissions using machine learning. Nat Hazard Earth Syst 8:991–1000. https://doi.org/10.5194/nhess-8-991-2008Chen S, Kan G, Li J, Liang K, Hong Y (2018) Investigating China’s urban air quality using big data, information theory, and machine learning. Pol J Environ Stud 27:565–578. https://doi.org/10.15244/pjoes/75159Corani (2005) Air quality prediction in Milan: feed-forward neural networks, pruned neural networks and lazy learning. Ecol Model 185:513–529. https://doi.org/10.1016/j.ecolmodel.2005.01.008Cruz C, Gómez A, Ramírez L, Villalva A, Monge O, Varela J, Quiroz J, Duarte H (2017) Calidad del aire respecto de metales (Pb, Cd, Ni, Cu, Cr) y relación con salud respiratoria: caso Sonora, México. Rev Int Contam Ambient 33:23–34. https://doi.org/10.20937/RICA.2017.33.esp02.02de Hoogh K, Héritier H, Stafoggia M, Künzli N, Kloog I (2018) Modelling daily PM2.5 concentrations at high spatio-temporal resolution across Switzerland. Environ Pollut 233:1147–1154. https://doi.org/10.1016/j.envpol.2017.10.025Franceschi F, Cobo M, Figueredo M (2018) Discovering relationships and forecasting PM10 and PM2.5 concentrations in Bogotá, Colombia, using Artificial Neural Networks, Principal Component Analysis, and k-means clustering. Atmos Pollut Res 9:912–922. https://doi.org/10.1016/j.apr.2018.02.006García N, Combarro E, del Coz J, Montañes E (2013) A SVM-based regression model to study the air quality at local scale in Oviedo urban area (Northern Spain): a case study. Appl Math Comput 219:8923–8937. https://doi.org/10.1016/j.amc.2013.03.018Gibert K, Sànchez-Màrre M, Sevilla B (2012) Tools for environmental data mining and intelligent decision support. In iEMSs. Leipzig, Germany. http://www.iemss.org/society/index.php/iemss-2012-proceedings. Accessed 26 Nov 2018Gibert K, Sànchez-Marrè M, Izquierdo J (2016) A survey on pre-processing techniques: relevant issues in the context of environmental data mining. Ai Commun 29:627–663. https://doi.org/10.3233/AIC-160710Gounaridis D, Chorianopoulos I, Koukoulas S (2018) Exploring prospective urban growth trends under different economic outlooks and land-use planning scenarios: the case of Athens. Appl Geogr 90:134–144. https://doi.org/10.1016/j.apgeog.2017.12.001Holloway J, Mengersen K (2018) Statistical machine learning methods and remote sensing for sustainable development goals: a review. Remote Sens 10:1–21. https://doi.org/10.3390/rs10091365Ifaei P, Karbassi A, Lee S, Yoo Ch (2017) A renewable energies-assisted sustainable development plan for Iran using techno-econo-socio-environmental multivariate analysis and big data. Energy Convers Manag 153:257–277. https://doi.org/10.1016/j.enconman.2017.10.014Kadiyala A, Kumar A (2017a) Applications of R to evaluate environmental data science problems. Environ Prog Sustain 36:1358–1364. https://doi.org/10.1002/ep.12676Kadiyala A, Kumar A (2017b) Vector time series-based radial basis function neural network modeling of air quality inside a public transportation bus using available software. Environ Prog Sustain 36:4–10. https://doi.org/10.1002/ep.12523Karimian H, Li Q, Wu Ch, Qi Y, Mo Y, Chen G, Zhang X, Sachdeva S (2019) Evaluation of different machine learning approaches to forecasting PM2.5 mass concentrations. Aerosol Air Qual Res 19:1400–1410. https://doi.org/10.4209/aaqr.2018.12.0450Krzyzanowski M, Apte J, Bonjour S, Brauer M, Cohen A, Prüss-Ustun A (2014) Air pollution in the mega-cities. Curr Environ Health Rep 1:185–191. https://doi.org/10.1007/s40572-014-0019-7Lässig K, Morik (2016) Computat sustainability. Springer, Berlin. https://doi.org/10.1007/978-3-319-31858-5Li Y, Wu Y-X, Zeng Z-X, Guo L (2006) Research on forecast model for sustainable development of economy-environment system based on PCA and SVM. In: Proceedings of the 2006 international conference on machine learning and cybernetics, vol 2006. IEEE, Dalian, China, pp 3590–3593. https://doi.org/10.1109/ICMLC.2006.258576Liu B-Ch, Binaykia A, Chang P-Ch, Tiwari M, Tsao Ch-Ch (2017) Urban air quality forecasting based on multi- dimensional collaborative support vector regression (SVR): a case study of Beijing-Tianjin-Shijiazhuang. PLoS ONE 12:1–17. https://doi.org/10.1371/journal.pone.0179763Lubell M, Feiock R, Handy S (2009) City adoption of environmentally sustainable policies in California’s Central Valley. J Am Plan Assoc 75:293–308. https://doi.org/10.1080/01944360902952295Ma D, Zhang Z (2016) Contaminant dispersion prediction and source estimation with integrated Gaussian-machine learning network model for point source emission in atmosphere. J Hazard Mater 311:237–245. https://doi.org/10.1016/j.jhazmat.2016.03.022Madu C, Kuei N, Lee P (2017) Urban sustainability management: a deep learning perspective. Sustain Cities Soc 30:1–17. https://doi.org/10.1016/j.scs.2016.12.012Mellos K (1988) Theory of eco-development. In: Perspectives on ecology. Palgrave Macmillan, London. https://doi.org/10.1007/978-1-349-19598-5_4Ni XY, Huang H, Du WP (2017) Relevance analysis and short-term prediction of PM2.5 concentrations in Beijing based on multi-source data. Atmos Environ 150:146–161. https://doi.org/10.1016/j.atmosenv.2016.11.054Oprea M, Dragomir E, Popescu M, Mihalache S (2016) Particulate matter air pollutants forecasting using inductive learning approach. Rev Chim 67:2075–2081Paas B, Stienen J, Vorländer M, Schneider Ch (2017) Modelling of urban near-road atmospheric PM concentrations using an artificial neural network approach with acoustic data input. Environments 4:1–25. https://doi.org/10.3390/environments4020026Pandey G, Zhang B, Jian L (2013) Predicting submicron air pollution indicators: a machine learning approach. Environ Sci Proc Impacts 15:996–1005. https://doi.org/10.1039/c3em30890aPeng H, Lima A, Teakles A, Jin J, Cannon A, Hsieh W (2017) Evaluating hourly air quality forecasting in Canada with nonlinear updatable machine learning methods. Air Qual Atmos Health 10:195–211. https://doi.org/10.1007/s11869-016-0414-3Pérez-Ortíz M, de La Paz-Marín M, Gutiérrez PA, Hervás-Martínez C (2014) Classification of EU countries’ progress towards sustainable development based on ordinal regression techniques. Knowl Based Syst 66:178–189. https://doi.org/10.1016/j.knosys.2014.04.041Phillis Y, Kouikoglou V, Verdugo C (2017) Urban sustainability assessment and ranking of cities. Comput Environ Urban 64:254–265. https://doi.org/10.1016/j.compenvurbsys.2017.03.002Saeed S, Hussain L, Awan I, Idris A (2017) Comparative analysis of different statistical methods for prediction of PM2.5 and PM10 concentrations in advance for several hours. Int J Comput Sci Netw Secur 17:45–52Sayegh A, Munir S, Habeebullah T (2014) Comparing the performance of statistical models for predicting PM10 concentrations. Aerosol Air Qual Res 14:653–665. https://doi.org/10.4209/aaqr.2013.07.0259Shaban K, Kadri A, Rezk E (2016) Urban air pollution monitoring system with forecasting models. IEEE Sens J 16:2598–2606. https://doi.org/10.1109/JSEN.2016.2514378Sierra B (2006) Aprendizaje automático conceptos básicos y avanzados Aspectos prácticos utilizando el software Weka. Madrid Pearson Prentice Hall, MadridSingh K, Gupta S, Rai P (2013) Identifying pollution sources and predicting urban air quality using ensemble learning methods. Atmos Environ 80:426–437. https://doi.org/10.1016/j.atmosenv.2013.08.023Song L, Pang S, Longley I, Olivares G, Sarrafzadeh A (2014) Spatio-temporal PM2.5 prediction by spatial data aided incremental support vector regression. In: International joint conference on neural networks. IEEE, Beijing, pp 623–630. https://doi.org/10.1109/IJCNN.2014.6889521Souza R, Coelho G, da Silva A, Pozza S (2015) Using ensembles of artificial neural networks to improve PM10 forecasts. Chem Eng Trans 43:2161–2166. https://doi.org/10.3303/CET1543361Suárez A, García PJ, Riesgo P, del Coz JJ, Iglesias-Rodríguez FJ (2011) Application of an SVM-based regression model to the air quality study at local scale in the Avilés urban area (Spain). Math Comput Model 54:453–1466. https://doi.org/10.1016/j.mcm.2011.04.017Tamas W, Notton G, Paoli C, Nivet M, Voyant C (2016) Hybridization of air quality forecasting models using machine learning and clustering: an original approach to detect pollutant peaks. Aerosol Air Qual Res 16:405–416. https://doi.org/10.4209/aaqr.2015.03.0193Toumi O, Le Gallo J, Ben Rejeb J (2017) Assessment of Latin American sustainability. Renew Sustain Energy Rev 78:878–885. https://doi.org/10.1016/j.rser.2017.05.013Tzima F, Mitkas P, Voukantsis D, Karatzas K (2011) Sparse episode identification in environmental datasets: the case of air quality assessment. Expert Syst Appl 38:5019–5027. https://doi.org/10.1016/j.eswa.2010.09.148United Nations, Department of Economic and Social Affairs (2019) World urbanization prospects The 2018 Revision. New York. https://doi.org/10.18356/b9e995fe-enWang B (2019) Applying machine-learning methods based on causality analysis to determine air quality in China. Pol J Environ Stud 28:3877–3885. https://doi.org/10.15244/pjoes/99639Wang X, Xiao Z (2017) Regional eco-efficiency prediction with support vector spatial dynamic MIDAS. J Clean Prod 161:165–177. https://doi.org/10.1016/j.jclepro.2017.05.077Wang W, Men C, Lu W (2008) Online prediction model based on support vector machine. Neurocomputing 71:550–558. https://doi.org/10.1016/j.neucom.2007.07.020WCED (1987) Report of the world commission on environment and development: our common future: report of the world commission on environment and development. WCED, Oslo. https://doi.org/10.1080/07488008808408783Weizhen H, Zhengqiang L, Yuhuan Z, Hua X, Ying Z, Kaitao L, Donghui L, Peng W, Yan M (2014) Using support vector regression to predict PM10 and PM2.5. In: IOP conference series: earth and environmental science, vol 17. IOP. https://doi.org/10.1088/1755-1315/17/1/012268WHO (2016) OMS | La OMS publica estimaciones nacionales sobre la exposición a la contaminación del aire y sus repercusiones para la salud. WHO. http://www.who.int/mediacentre/news/releases/2016/air-pollution-estimates/es/. Accesed 26 Nov 2018Yeganeh N, Shafie MP, Rashidi Y, Kamalan H (2012) Prediction of CO concentrations based on a hybrid partial least square and support vector machine model. Atmos Environ 55:357–365. https://doi.org/10.1016/j.atmosenv.2012.02.092Zalakeviciute R, Bastidas M, Buenaño A, Rybarczyk Y (2020) A traffic-based method to predict and map urban air quality. Appl Sci. https://doi.org/10.3390/app10062035Zeng L, Guo J, Wang B, Lv J, Wang Q (2019) Analyzing sustainability of Chinese coal cities using a decision tree modeling approach. Resour Policy 64:101501. https://doi.org/10.1016/j.resourpol.2019.101501Zhan Y, Luo Y, Deng X, Grieneisen M, Zhang M, Di B (2018) Spatiotemporal prediction of daily ambient ozone levels across China using random forest for human exposure assessment. Environ Pollut 233:464–473. https://doi.org/10.1016/j.envpol.2017.10.029Zhang Y, Huan Q (2006) Research on the evaluation of sustainable development in Cangzhou city based on neural-network-AHP. In: Proceedings of the fifth international conference on machine learning and cybernetics, vol 2006. pp 3144–3147. https://doi.org/10.1109/ICMLC.2006.258407Zhang Y, Shang W, Wu Y (2009) Research on sustainable development based on neural network. In: 2009 Chinese control and decision conference. IEEE, pp 3273–3276. https://doi.org/10.1109/CCDC.2009.5192476Zhou Y, Chang F-J, Chang L-Ch, Kao I-F, Wang YS (2019) Explore a deep learning multi-output neural network for regional multi-step-ahead air quality forecasts. J Clean Prod 209:134–145. https://doi.org/10.1016/j.jclepro.2018.10.24

Spatial-temporal prediction of air quality based on recurrent neural networks

To predict air quality (PM2.5 concentrations, et al), many parametric regression models have been developed, while deep learning algorithms are used less often. And few of them takes the air pollution emission or spatial information into consideration or predict them in hour scale. In this paper, we proposed a spatial-temporal GRU-based prediction framework incorporating ground pollution monitoring (GPM), factory emissions (FE), surface meteorology monitoring (SMM) variables to predict hourly PM2.5 concentrations. The dataset for empirical experiments was built based on air quality monitoring in Shenyang, China. Experimental results indicate that our method enables more accurate predictions than all baseline models and by applying the convolutional processing to the GPM and FE variables notable improvement can be achieved in prediction accuracy

Spatiotemporal and temporal forecasting of ambient air pollution levels through data-intensive hybrid artificial neural network models

Outdoor air pollution (AP) is a serious public threat which has been linked to severe respiratory and cardiovascular illnesses, and premature deaths especially among those residing in highly urbanised cities. As such, there is a need to develop early-warning and risk management tools to alleviate its effects. The main objective of this research is to develop AP forecasting models based on Artificial Neural Networks (ANNs) according to an identified model-building protocol from existing related works. Plain, hybrid and ensemble ANN model architectures were developed to estimate the temporal and spatiotemporal variability of hourly NO2 levels in several locations in the Greater London area. Wavelet decomposition was integrated with Multilayer Perceptron (MLP) and Long Short-term Memory (LSTM) models to address the issue of high variability of AP data and improve the estimation of peak AP levels. Block-splitting and crossvalidation procedures have been adapted to validate the models based on Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Willmott’s index of agreement (IA). The results of the proposed models present better performance than those from the benchmark models. For instance, the proposed wavelet-based hybrid approach provided 39.15% and 28.58% reductions in RMSE and MAE indices, respectively, on the performance of the benchmark MLP model results for the temporal forecasting of NO2 levels. The same approach reduced the RMSE and MAE indices of the benchmark LSTM model results by 12.45% and 20.08%, respectively, for the spatiotemporal estimation of NO2 levels in one site at Central London. The proposed hybrid deep learning approach offers great potential to be operational in providing air pollution forecasts in areas without a reliable database. The model-building protocol adapted in this thesis can also be applied to studies using measurements from other sites.Outdoor air pollution (AP) is a serious public threat which has been linked to severe respiratory and cardiovascular illnesses, and premature deaths especially among those residing in highly urbanised cities. As such, there is a need to develop early-warning and risk management tools to alleviate its effects. The main objective of this research is to develop AP forecasting models based on Artificial Neural Networks (ANNs) according to an identified model-building protocol from existing related works. Plain, hybrid and ensemble ANN model architectures were developed to estimate the temporal and spatiotemporal variability of hourly NO2 levels in several locations in the Greater London area. Wavelet decomposition was integrated with Multilayer Perceptron (MLP) and Long Short-term Memory (LSTM) models to address the issue of high variability of AP data and improve the estimation of peak AP levels. Block-splitting and crossvalidation procedures have been adapted to validate the models based on Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Willmott’s index of agreement (IA). The results of the proposed models present better performance than those from the benchmark models. For instance, the proposed wavelet-based hybrid approach provided 39.15% and 28.58% reductions in RMSE and MAE indices, respectively, on the performance of the benchmark MLP model results for the temporal forecasting of NO2 levels. The same approach reduced the RMSE and MAE indices of the benchmark LSTM model results by 12.45% and 20.08%, respectively, for the spatiotemporal estimation of NO2 levels in one site at Central London. The proposed hybrid deep learning approach offers great potential to be operational in providing air pollution forecasts in areas without a reliable database. The model-building protocol adapted in this thesis can also be applied to studies using measurements from other sites

A Review of 21st-Century Studies

PM10 prediction has attracted special legislative and scientific attention due to its harmful effects on human health. Statistical techniques have the potential for high-accuracy PM10 prediction and accordingly, previous studies on statistical methods for temporal, spatial and spatio-temporal prediction of PM10 are reviewed and discussed in this paper. A review of previous studies demonstrates that Support Vector Machines, Artificial Neural Networks and hybrid techniques show promise for suitable temporal PM10 prediction. A review of the spatial predictions of PM10 shows that the LUR (Land Use Regression) approach has been successfully utilized for spatial prediction of PM10 in urban areas. Of the six introduced approaches for spatio-temporal prediction of PM10, only one approach is suitable for high-resolved prediction (Spatial resolution < 100 m; Temporal resolution ¤ 24 h). In this approach, based upon the LUR modeling method, short-term dynamic input variables are employed as explanatory variables alongside typical non-dynamic input variables in a non- linear modeling procedure

Graph Neural Network for spatiotemporal data: methods and applications

In the era of big data, there has been a surge in the availability of data containing rich spatial and temporal information, offering valuable insights into dynamic systems and processes for applications such as weather forecasting, natural disaster management, intelligent transport systems, and precision agriculture. Graph neural networks (GNNs) have emerged as a powerful tool for modeling and understanding data with dependencies to each other such as spatial and temporal dependencies. There is a large amount of existing work that focuses on addressing the complex spatial and temporal dependencies in spatiotemporal data using GNNs. However, the strong interdisciplinary nature of spatiotemporal data has created numerous GNNs variants specifically designed for distinct application domains. Although the techniques are generally applicable across various domains, cross-referencing these methods remains essential yet challenging due to the absence of a comprehensive literature review on GNNs for spatiotemporal data. This article aims to provide a systematic and comprehensive overview of the technologies and applications of GNNs in the spatiotemporal domain. First, the ways of constructing graphs from spatiotemporal data are summarized to help domain experts understand how to generate graphs from various types of spatiotemporal data. Then, a systematic categorization and summary of existing spatiotemporal GNNs are presented to enable domain experts to identify suitable techniques and to support model developers in advancing their research. Moreover, a comprehensive overview of significant applications in the spatiotemporal domain is offered to introduce a broader range of applications to model developers and domain experts, assisting them in exploring potential research topics and enhancing the impact of their work. Finally, open challenges and future directions are discussed

A Systematic Review for Transformer-based Long-term Series Forecasting

The emergence of deep learning has yielded noteworthy advancements in time series forecasting (TSF). Transformer architectures, in particular, have witnessed broad utilization and adoption in TSF tasks. Transformers have proven to be the most successful solution to extract the semantic correlations among the elements within a long sequence. Various variants have enabled transformer architecture to effectively handle long-term time series forecasting (LTSF) tasks. In this article, we first present a comprehensive overview of transformer architectures and their subsequent enhancements developed to address various LTSF tasks. Then, we summarize the publicly available LTSF datasets and relevant evaluation metrics. Furthermore, we provide valuable insights into the best practices and techniques for effectively training transformers in the context of time-series analysis. Lastly, we propose potential research directions in this rapidly evolving field