This work was supported by the Engineering and Physical Sciences Research Council [EP/W522089/1].Peer reviewedPublisher PD

Leontidis, Georgios

Potts, Rebecca Lauren

English

This work was supported by the Engineering and Physical Sciences Research Council [EP/W522089/1].Peer reviewe

Aberdeen University Research

Attention-Based Deep Learning Methods for Predicting GasTurbine EmissionsRebecca Potts and Georgios LeontidisSchool of Natural and Computing Sciences & Interdisciplinary Centre for Data and AI,University of Aberdeen, Aberdeen, UKAbstractPredictive emissions monitoring systems (PEMS)for gas turbines are critical for monitoring harm-ful pollutants being released into the atmosphere,while reducing the use of expensive continuousemissions monitoring systems (CEMS) which re-quire daily maintenance to achieve accurate read-ings. We consider two attention-based deep learn-ing models, FT-Transformer and SAINT, and com-pare with classical tree-based XGBoost to predictemissions from gas turbines. We find that theattention-based models outperform XGBoost forboth prediction tasks, i.e. carbon monoxide (CO)and nitrogen oxides (NOx).1 IntroductionA predictive emissions monitoring systems (PEMS)model is trained on past data, utilising processparameters such as temperatures and pressures,and uses real-time data to generate estimations foremissions. Gas turbine process data can be consid-ered tabular data. Classical methods such as XG-Boost (Chen and Guestrin [2016]) and CatBoost(Prokhorenkova et al. [2018]) have shown excellentresults in the tabular domain, and are often seen asthe standard solution for structured data problems.Previous comparisons between neural networks(NNs) and classical methods such as XGBoost fortabular regression generally conclude that classi-cal methods match or outperform the NN basedmodels. For example, Kossen et al. [2021] findthat taking the entire dataset as input and usingself-attention between datapoints is outperformedby CatBoost and only matches the performance ofXGBoost. Grinsztajn et al. [2022] find that classi-cal tree-based methods outperform the NN models,even on numerical only datasets, whereas a surveyby Borisov et al. [2021] concludes that deep learn-ing tabular methods outperform classical when adataset consists of mostly numerical features.Previous machine learning methods used for gasturbine PEMS include Vanderhaegen et al. [2010]and Si et al. [2019] who compared different config-urations of NN based models, Cuccu et al. [2017]compared 12 different machine learning algorithmsand Azzam et al. [2018] explored a genetic algo-rithm to tune the hyperparameters of a NN andsupport vector machines for their PEMS.Kaya et al. [2019] collected a novel PEMS datasetto develop a benchmark PEMS using an extremelearning machine (ELM) regressor using three fu-sion strategies: averaging, random forest, and ba-sic ELM. We use this dataset to utilise attention-based deep learning methods for PEMS, the firsttime these have been used for this problem.We compared two attention-based deep learningmodels, FT-Transformer (Gorishniy et al. [2021])and SAINT (Somepalli et al. [2021]), against thetree-based XGBoost and the three ELM fusionstrategies seen in Kaya et al. [2019] to predict gasturbine emissions.2 MethodologyFT-Transformer (Gorishniy et al. [2021]) is anadaptation of the Transformer architecture for tab-ular data. Transformers (Vaswani et al. [2017])utilise multi-head self-attention to jointly attend toinformation to determine which inputs are the mostimportant at a time. In FT-Transformer, input fea-1tures are embedded via the Feature Tokenizer mod-ule and a stack of Transformer layers are applied toall embedded input features, such that every Trans-former layer operates on the feature level of oneobject.SAINT (Somepalli et al. [2021]), the Self-Attention and Intersample Attention Transformer,is a hybrid deep learning approach for tabular dataproblems. Attention is performed over both rowsand columns, whereby the self-attention attends toindividual features within each data sample, and in-tersample attention relates each row to other rowsin the table.The gas turbine emissions dataset is publiclyavailable and consists of five years of data with36,733 instances. It has nine input parameters, in-cluding temperature and pressure, and two targetvariables, NOx and CO. It consists of only numer-ical features and no categorical.We split the dataset into train, valid and testsets as in the original work (Kaya et al. [2019]),with XGBoost utilising only train and test sets dueto the nature of XGBoost. A batch size of 32 wasused with a learning rate of 0.0001. The optimiserAdamW was used with MSE loss function. For themulti-head self-attention, 8 attention heads wereused for both FT-Transformer and SAINT.3 Results and DiscussionOverall mean absolute error (MAE) results areshown in Table 1 for test set data. Attention-basedmodels outperform XGBoost and the ELM fusionTable 1: MAE results for attention-based meth-ods FT-Transformer and SAINT compared to tree-based XGBoost and best results of three fusionschemes from Kaya et al. [2019]. Best results shownin bold.CO NOxMAE MAEFT-Transformer 0.50 2.57SAINT 0.84 2.51XGBoost 1.15 11.65Averaging 1.05 7.91Meta-ELM 1.26 24.05Random Forest 0.93 11.29Figure 1: Scatter plot of test set predictions forFT-Transformer, SAINT and XGBoost against realsensor readings for CO and NOx.schemes for both CO and NOx. FT-Transformerrecords the best results for CO, while SAINTmarginally records the best performance for NOx,with very similar results by FT-Transformer.Test set predictions for CO and NOx are plot-ted against corresponding real values in Figure 1,where perfect results would lie on the identity line.The attention-based methods have less spread com-pared to XGBoost for NOx, showing their superiorcorrelation.4 ConclusionWe applied and compared two attention-based deeplearning models, FT-Transformer and SAINT, andthe classical tree-based method XGBoost to pre-dict emissions for a gas turbine dataset and foundthat the attention-based deep learning models out-performed XGBoost for MAE values, as well asoutperforming the original baseline results for thisdataset.2AcknowledgementsThis work was supported by the Engineer-ing and Physical Sciences Research Council[EP/W522089/1].ReferencesM. Azzam, M. Awad, and J. Zeaiter. Application ofevolutionary neural networks and support vectormachines to model nox emissions from gas tur-bines. Journal of environmental chemical engi-neering, 6(1):1044–1052, 2018.V. Borisov, T. Leemann, K. Seßler, J. Haug,M. Pawelczyk, and G. Kasneci. Deep neural net-works and tabular data: A survey. arXiv preprintarXiv:2110.01889, 2021.T. Chen and C. Guestrin. Xgboost: A scalable treeboosting system. In Proceedings of the 22nd acmsigkdd international conference on knowledge dis-covery and data mining, pages 785–794, 2016.G. Cuccu, S. Danafar, P. Cudré-Mauroux,M. Gassner, S. Bernero, and K. Kryszczuk. Adata-driven approach to predict nox-emissions ofgas turbines. In 2017 IEEE International Con-ference on Big Data (Big Data), pages 1283–1288, 2017. doi: 10.1109/BigData.2017.8258056.Y. Gorishniy, I. Rubachev, V. Khrulkov, andA. Babenko. Revisiting deep learning models fortabular data. Advances in Neural InformationProcessing Systems, 34:18932–18943, 2021.L. Grinsztajn, E. Oyallon, and G. Varoquaux.Why do tree-based models still outperform deeplearning on tabular data? arXiv preprintarXiv:2207.08815, 2022.H. Kaya, P. Tüfekci, and E. Uzun. Predicting coand nox emissions from gas turbines: novel dataand a benchmark pems. Turkish Journal of Elec-trical Engineering and Computer Sciences, 27(6):4783–4796, 2019.J. Kossen, N. Band, C. Lyle, A. N. Gomez, T. Rain-forth, and Y. Gal. Self-attention between dat-apoints: Going beyond individual input-outputpairs in deep learning. Advances in Neural In-formation Processing Systems, 34:28742–28756,2021.L. Prokhorenkova, G. Gusev, A. Vorobev, A. V.Dorogush, and A. Gulin. Catboost: unbiasedboosting with categorical features. Advances inneural information processing systems, 31, 2018.M. Si, T. J. Tarnoczi, B. M. Wiens, and K. Du.Development of predictive emissions monitoringsystem using open source machine learning li-brary – keras: A case study on a cogenerationunit. IEEE Access, 7:113463–113475, 2019. doi:10.1109/ACCESS.2019.2930555.G. Somepalli, M. Goldblum, A. Schwarzschild,C. B. Bruss, and T. Goldstein. Saint: Improvedneural networks for tabular data via row atten-tion and contrastive pre-training. arXiv preprintarXiv:2106.01342, 2021.E. Vanderhaegen, M. l. Deneve, H. Laget, N. Faniel,and J. Mertens. Predictive emissions monitoringusing a continuously updating neural network.In Turbo Expo: Power for Land, Sea, and Air,volume 43970, pages 769–775, 2010.A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,L. Jones, A. N. Gomez,  L. Kaiser, and I. Polo-sukhin. Attention is all you need. Advances inneural information processing systems, 30, 2017.3

Attention-Based Deep Learning Methods for Predicting Gas Turbine Emissions

https://aura.abdn.ac.uk/bitstream/2164/19602/1/Potts_NLDL_extended_abstract.pdf

Attention-Based Deep Learning Methods for Predicting Gas Turbine Emissions

Abstract

Similar works

Full text

Available Versions

Aberdeen University Research