머신 러닝 기법과 정보 이론을 이용한 데이터 기반 이상 감지 및 진단

학위논문(박사) -- 서울대학교대학원 : 공과대학 화학생물공학부, 2021.8. 문경빈.공정 모니터링 시스템은 효과적이고 안전한 공정 운전을 위한 필수적인 요소이다. 공정 이상은 목표 생성물의 품질에 영향을 주거나 공정의 정상 가동을 방해하여 생산성을 저해할 수 있다. 폭발성 및 인화성 물질을 주로 다루는 화학공정의 경우 공정 이상은 가장 중요한 요소인 공정의 안전을 위협하는 요소로 작용할 수 있다. 한편, 현대의 공정의 범위가 확장되고 자동화와 고도화가 진행됨에 따라 점점 더 신뢰도 높은 모니터링 시스템이 요구되고 있다. 공정 모니터링은 크게 세 단계로 구분될 수 있다. 실시간으로 공정의 이상 여부를 판단하는 공정 이상 감지, 다음으로 감지된 이상의 원인을 파악하는 이상 진단, 마지막으로 공정 이상의 원인을 제거하고 정상 상태로 회복시키는 복원으로 나뉘어진다. 특히 공정 이상 감지와 진단 시스템을 위해 다양한 방법론들이 제안되어왔으며, 그 방법론들은 크게 세 가지로 구분할 수 있다. 물리 이론을 기반으로 한 모델 분석 방법과 특정 분야의 경험 지식을 바탕으로 한 지식 기반 방법론에 비해 범용적인 적용 가능성과 현대 공정의 풍부한 공정 데이터가 제공되는 조건의 충족으로 인해 데이터 기반 방법론이 널리 활용되어지고 있다. 또한, 데이터 기반 공정 모니터링 방법론들은 공정의 규모와 복잡도가 증가함에 따라 그 장점이 더욱 극대화되는 특징을 갖는다. 본 연구에서는 기존의 데이터 기반 공정 모니터링 방법론들의 성능을 개선하기 위한 공정 이상 감지 방법론과 이상 진단 방법론을 제안한다. 전통적인 공정 이상 감지 시스템은 차원 축소방법들을 기반으로 개발되었다. 차원 축소를 기반으로 한 공정 이상 감지 모델은 공정 데이터에 내재되어 있는 특징으로 정의되는 저차원의 잠재 공간을 정의하고, 이를 기준으로 모니터링을 수행한다. 대표적인 방법으로는 전통적인 다변량 공정 모니터링 방법인 주 성분 분석과 머신 러닝 기법인 오토인코더가 있다. 최근 풍부한 학습 데이터와 우수한 성능 덕분에 다양한 머신 러닝 기법을 사용한 이상 감지 시스템이 널리 활용되고 있지만, 앞서 소개한 현대 공정의 다양한 특징으로 인해 더욱 향상된 성능의 모니터링 기법의 개발이 요구되어지고 있다. 이러한 데이터 기반 모니터링 시스템의 성능 향상을 위해서 모델의 구조를 변경하거나 모델의 학습 절차를 변형하는 접근법들이 주로 제안되었다. 하지만, 데이터 기반 방법론들은 궁극적으로 학습 데이터의 품질에 의존적이라는 특성은 여전히 남아있다. 즉, 학습 데이터의 부족한 정보를 보완함으로써 모니터링 시스템의 완성도를 높일 수 있는 방법론이 요구된다. 따라서, 본 연구는 첫 번째 주제로 데이터 증강 기법을 결합한 공정 이상 감지 방법론을 제안한다. 데이터 증강 기법은 여러 집합을 구분하는 분류기 모델링시에 특정 집합의 학습 데이터가 부족한 경우에 주로 활용되었다. 이러한 경우 데이터 증강을 통해 학습 데이터의 균형을 맞춤으로써 모델의 학습 효율을 증진시킬 수 있다. 반면에, 본 연구에서의 데이터 증강은 한 집합 내에서의 불균형을 완화하기 위한 목적으로 사용되었다. 정상 조건의 공정 데이터는 정상과 이상의 경계에 분포하는 데이터가 희박하게 존재하는 특징을 갖는다. 이상 감지 시스템이 정상 상태의 저차원 특징 공간을 학습하고, 이를 통해 정상과 이상을 구분하는 모델이라는 점을 고려하면 경계 영역의 데이터의 증강이 특징 공간 학습에 긍정적으로 작용할 것을 기대해 볼 수 있다. 이와 같은 맥락에서 제안된 방법론은 다음과 같다. 먼저, 기존의 학습 데이터를 이용하여 인공 데이터를 생성하기위한 생성모델인 변분 오토인코더를 학습한다. 생성 모델로 학습한 정상 운전 데이터의 저차원 분포의 경계영역에 해당하는 데이터들을 인공 데이터로 생성하여 학습데이터에 증강시킨다. 이렇게 증강된 학습 데이터를 기반으로 이상 감지 모델을 위한 머신 러닝 기반 차원 축소 방법인 오토인코더를 학습하여 이상 감지 시스템을 구축한다. 증강된 학습 데이터를 사용함으로써 오토인코더의 잠재 공간 학습이 더 효과적으로 수행될 수 있고, 이는 곧 정상과 이상 상태를 구분하는 이상 감지 시스템의 성능 개선으로 이어질 수 있다. 차원 축소 기법은 전통적인 이상 진단 방법으로도 활용되었다. 하지만, 이는 차원 축소시의 정보의 손실로 인해 저조하고 일관성이 부족한 성능을 보였다. 전통적인 방법의 한계점을 개선하기 위해 공정 변수 간의 인과 관계를 직접적으로 분석하는 기법들이 개발되었다. 그 중 하나인 정보 이론 기반의 전달 엔트로피는 특정 모델이나 선형 가정을 기반으로 하지 않기 때문에 비선형 공정의 이상 진단에 대해 일반적으로 우수한 성능을 보인다고 알려져 있다. 하지만, 전달 엔트로피를 이용한 인과관계 분석 방법은 고비용의 밀도 추정을 필요로 한다는 단점으로 인해 소규모 공정에 대해서만 제한적으로 적용되어 왔다. 이러한 한계점을 개선하기 위한 방안으로 그래프 라쏘라는 조정 방법을 전달 엔트로피와 결합한 방법론을 제안하였다. 그래프 라쏘는 비 방향성 그래프 모델에서 성긴 구조를 학습하기 위한 방법론으로 전체 공정 그래프로부터 상관 관계가 높은 부분 그래프를 추출해낼 수 있다. 가장 높은 상관 관계를 갖는 부분 그래프와 독립된 나머지 변수들이 그래프 라쏘의 출력으로 제시되기 때문에, 나머지 변수들에 대한 반복적인 적용을 통해 전체 공정 변수들을 연관성이 높은 몇몇의 부분 그래프로 변환할 수 있다. 연관성이 낮은 관계를 사전에 배제함으로써 인과 관계 분석의 대상을 크게 축소할 수 있다. 즉, 이 단계를 통해 고비용의 전달 엔트로피의 한계점을 완화하고, 그 적용 가능성을 확장할 수 있도록 한다. 두 방법을 결합하여 다음과 같은 이상 진단 방법론을 제안하였다. 먼저, 공정 이상이 발생한 데이터를 대상으로 반복적 그래프 라쏘를 적용하여 전체 공정 변수들을 연관성이 높은 5개의 부분 집합으로 구분한다. 구분된 각각의 부분 집합을 대상으로 전달 엔트로피를 이용한 인과관계 척도를 계산하고, 가장 유력한 원인 변수를 판별해낸다. 즉, 그래프 라쏘를 통해 효과적으로 인과관계 분석의 대상을 축소함으로써 불필요한 전달 엔트로피 계산으로 발생하는 비용을 크게 절감할 수 있다. 따라서, 제안된 방법론은 대규모 산업 공정에 대해서도 전달 엔트로피를 이용한 이상 진단 기법의 적용을 가능하게 했다는 점에서 의의가 있다. 본 연구에서 제안된 방법론의 성능을 검증하기 위하여 산업 규모의 벤치마크 공정 모델인 테네시 이스트만 공정에 이를 적용하고 결과를 분석하였다. 벤치마크 공정 모델은 다수의 단위 공정을 포함하고, 재순환 흐름과 화학 반응을 포함하고 있어 실제 공정과 같은 복잡도를 갖는 공정 모델로서 제안한 방법론들의 성능을 시험해보기에 적합했다. 성능 테스트는 테네시 이스트만 공정 모델에 포함되어 있는 사전에 정의된 28개 종류의 공정 이상에 대하여 수행하였다. 제안한 데이터 증강을 접목한 공정 이상 감지 방법론은 기존 방법론 대비 높은 이상 감지율을 보였다. 일부의 경우 이상 감지 지연측면에서도 개선을 확인할 수 있었다. 또한, 이상 진단을 위해 전달 엔트로피와 그래프 라쏘를 결합한 제안한 방법론은 전체 공정에 전달 엔트로피를 직접 적용한 기존의 방법론 대비 약 20%의 계산 비용만으로도 효과적으로 이상의 원인을 파악해내는 것을 확인할 수 있었다. 또한, 성능 테스트 결과는 일부 공정 이상의 경우 제안한 방법론이 기존의 방법보다 더 정확한 이상 진단 결과를 제시할 수 있음을 보였다.Process monitoring system is an essential component for efficient and safe operation. Process faults can affect the quality of the product or interfere with the normal operation of the process, hindering productivity. In the case of chemical processes dealing with explosive and flammable materials, process fault can act as a threat to the process safety which should be the top priority. Meanwhile, modern processes demand a more advanced monitoring system as the scope of the process expands and the process automation and intensification progress. The framework of the process monitoring system can be classified into three stages. It is divided into process fault detection that determines the existence of process faults in a system in real-time, fault diagnosis that identifies the root cause of the faults, and finally, process recovery that removes the cause of the fault and normalizes the process. In particular, various methodologies for fault detection and diagnosis have been proposed, and they can be categorized into three approaches. Data-driven methodologies are widely utilized due to the general applicability and the conditions under which abundant process data are provided compared to analytical methods based on the detailed first-principle models and knowledge-based methods on the specific domain knowledge. Furthermore, the advantage of the data-driven methods can be prominent as the scale and complexity of the process increase. In this thesis, fault detection and diagnosis methodologies to improve the performance of existing data-driven methods are proposed. Conventional data-driven fault detection systems have been developed based on dimensionality reduction methods. The fault detection models using dimensionality reduction identify the low dimensional latent space defined by features inherent in process data, performing process monitoring based on it. As the representative methods, there are principal component analysis which is the conventional multivariate process monitoring approach, and autoencoder which is one of the machine learning techniques. Although the monitoring systems using various machine learning techniques have been widely utilized thanks to sufficient process data and good performance, a monitoring scheme that improves the performance of up-to-date methods is required due to the aforementioned factors. To improve the performance of such a data-driven monitoring system, approaches that change the structure of the model or learning procedure have been mainly discussed. Meanwhile, the nature that data-driven methods are ultimately dependent on the quality of the training dataset still remains. In other words, a methodology to enhance the completeness of the monitoring system by supplementing the insufficient information in the training dataset is required. Thus, a process fault detection method that combines data augmentation techniques is proposed in the first part of the thesis. Data augmentation has been mostly employed to manage the deficiency of certain classes, between-class imbalance, in a classification problem. In this case, data augmentation can be effectively applied to improve the training performance by balancing the amount of each class. Data augmentation in this study, on the other hand, is applied to alleviate the with-in-class imbalance. The process data in normal operation has characteristics that the data samples in the borderline of normal and abnormal state are relatively sparse. Given that the modeling of the fault detection system corresponds to defining the low-dimensional feature space and monitoring the system in it, it can be expected that the supplement of the samples on the boundary of the normal state would positively affect the training process. In this context, the proposed method is as follows. First, variational autoencoder which is a generative model is constructed to generate the synthetic data using the original training data. The sample vector corresponding to the boundary region of the low-dimensional distribution of the normal state learned by the generative model is generated as the synthetic data and augmented to the original training data. Based on the augmented training data the fault detection system is established using autoencoder, a machine learning algorithm for feature extraction. The feature learning of autoencoder can be performed more effectively by using the augmented training data, which can lead to the improvement of the fault detection system that distinguishes between normal and abnormal states. The dimensionality reduction methods have been also utilized as the fault isolation method known as the contribution charts. However, the approaches showed limited performance and inconsistent analysis results due to the information loss during the dimension reduction process. To resolve the limitations of the conventional method, the approaches that directly figure out the causal relationships between process variables have been developed. As one of them, transfer entropy, an information-theoretic causality measure, is generally known to have good fault isolation performance in the fault isolation of nonlinear processes because it is neither linearity assumption nor model-based method. However, it has been limitedly applied to the small-scale process because of the drawback that the causal analysis using transfer entropy requires costly density estimation. To resolve the limitation, the method that combines graphical lasso which is a regularization method with transfer entropy is proposed. Graphical lasso is a sparse structure learning algorithm of the undirected graph model, which can be used to sort out the most relevant sub-group in the entire graph model. As graphical lasso algorithm presents the output as a highly correlated subgroup with the rest of the variables, the iterative application of graphical lasso can substitute the entire process into several subgroups. This process can greatly reduce the subject of causal analysis by excluding relationships with little relevance in advance. Accordingly, the limitation of demanding cost of transfer entropy can be mitigated and thus the applicability of fault isolation using transfer entropy can be expanded through this process. Combining the two methods, the following fault isolation method is proposed. First of all, the entire process variables are divided into the five most relevant subgroups based on the data when the fault has occurred. The root cause variable can be isolated from the most significant relationship by calculating the causality measure using transfer entropy only within each subgroup. It is possible to significantly reduce the computational cost due to transfer entropy by efficiently decreasing the subject of causal analysis through graphical lasso. Therefore, the proposed method is noteworthy in that it enables the application of fault isolation using transfer entropy for industrial-scale processes. The proposed methodologies in each stage are verified by applying them to the industrial-scale benchmark process model, the Tennessee Eastman process (TEP). The benchmark process model is suitable to test the performance of the proposed methods because it is a process model with similar complexity as a real chemical process involving multiple unit operations, recycle stream, and chemical reactions in it. The performance test is performed with respect to the 28 predefined process faults scenarios in TEP model. Application results of the proposed fault detection method performed better than the case using the conventional approach in terms of the fault detection rate. In some fault cases, the fault detection delay, the time required to first detect a fault since it occurred, also showed improvement. Fault isolation results by the proposed method integrating transfer entropy with graphical lasso showed that it could effectively identify the cause of the process fault with only about 20% of the computational cost compared to the base case that directly applied the transfer entropy to the entire process for fault isolation. In addition, the demonstration results suggested that the proposed method could outperform the base case in terms of accuracy in some particular cases.Chapter 1 Introduction -2 1.1. Research Motivation -2 1.2. Research Objectives 5 1.3. Outline of the Thesis 7 Chapter 2 Backgrounds and Preliminaries 8 2.1. Autoencoder 8 2.2. Variational Autoencoder 3 2.3. Transfer Entropy 7 2.4. Graphical Lasso 11 Chapter 3 Process Fault Detection Using Autoencoder with Data Augmentation via Variational Autoencoder 23 3.1. Introduction 23 3.2. Process Fault Detection Model Integrated with Data Augmentation 28 3.2.1. Info-Variational Autoencoder for Data Augmentation 31 3.2.2. Autoencoder for Process Monitoring 33 3.3. Case study and Discussion 34 3.3.1. Tennessee Eastman Process 35 3.3.2. Implementation of the Proposed Methodology 39 3.3.3. Discussion of the Results 64 Chapter 4 Process Fault Isolation using Transfer Entropy and Graphical Lasso 80 4.1. Introduction 80 4.2. Fault Isolation using Transfer Entropy Integrated with Graphical Lasso 86 4.2.1. Graphical Lasso for Sub-group Modeling 89 4.2.2. Transfer Entropy for Fault Isolation 90 4.3. Case study and Discussion 1 92 4.3.1. Selective Catalytic Reduction Process 92 4.3.2. Implementation of the Proposed Methodology 97 4.3.3. Discussion of the Results 99 4.4. Case study and Discussion 2 102 4.4.1. Tennessee Eastman Process 102 4.4.2. Implementation of the Proposed Methodology 108 4.4.3. Discussion of the Results 109 Chapter 5 Concluding Remarks 130 5.1. Summary of the Contributions 130 5.2. Future Work 133 Bibliography 135박

Machine Learning-based Predictive Maintenance for Optical Networks

Optical networks provide the backbone of modern telecommunications by connecting the world faster than ever before. However, such networks are susceptible to several failures (e.g., optical fiber cuts, malfunctioning optical devices), which might result in degradation in the network operation, massive data loss, and network disruption. It is challenging to accurately and quickly detect and localize such failures due to the complexity of such networks, the time required to identify the fault and pinpoint it using conventional approaches, and the lack of proactive efficient fault management mechanisms. Therefore, it is highly beneficial to perform fault management in optical communication systems in order to reduce the mean time to repair, to meet service level agreements more easily, and to enhance the network reliability. In this thesis, the aforementioned challenges and needs are tackled by investigating the use of machine learning (ML) techniques for implementing efficient proactive fault detection, diagnosis, and localization schemes for optical communication systems. In particular, the adoption of ML methods for solving the following problems is explored: - Degradation prediction of semiconductor lasers, - Lifetime (mean time to failure) prediction of semiconductor lasers, - Remaining useful life (the length of time a machine is likely to operate before it requires repair or replacement) prediction of semiconductor lasers, - Optical fiber fault detection, localization, characterization, and identification for different optical network architectures, - Anomaly detection in optical fiber monitoring. Such ML approaches outperform the conventionally employed methods for all the investigated use cases by achieving better prediction accuracy and earlier prediction or detection capability

Deep Learning-Based Machinery Fault Diagnostics

This book offers a compilation for experts, scholars, and researchers to present the most recent advancements, from theoretical methods to the applications of sophisticated fault diagnosis techniques. The deep learning methods for analyzing and testing complex mechanical systems are of particular interest. Special attention is given to the representation and analysis of system information, operating condition monitoring, the establishment of technical standards, and scientific support of machinery fault diagnosis

Fault diagnosis in industrial process by using LSTM and an elastic net

[EN] Fault diagnosis is important for industrial processes because it permits to determine the necessity of emergency stops in a process and/or to propose a maintenance plan. Two strategies for fault diagnosis are compared in this work. On the one hand, the data are preprocessed using the independent components analysis for dimension reduction, then the wavelet transform is used in order to highlight the faulty signals, with this information an artificial neural network was fed. On the other hand, the second strategy, the main contribution of this work, is the implementation of a long short term memory. This memory is fed with the most representative variables selected by an elastic net to use both, the L1 and L2 norms. These strategies are applied in the Tennessee Eastman process, a benchmark widely used for fault diagnosis. The fault isolation had better results than those reported in the literature.[ES] El diagnóstico de fallas es importante en los procesos industriales, ya que permite determinar si es necesario detener el proceso en operación y/o proponer un plan de mantenimiento. En el presente trabajo se comparan dos estrategias para diagnosticar fallas. La primera realiza un preprocesamiento de datos usando el análisis de componentes independientes para reducir la dimensión de los datos, posteriormente, se emplea la transformada wavelet para resaltar las señales de falla, con esta información se alimenta una red neuronal artificial. Por su parte, la segunda estrategia, principal contribución de este trabajo, usa una memoria de corto y largo plazo. Esta memoria es alimentada por las variables más significativas seleccionadas mediante una red elástica para usar tanto la norma $L_1$ como la $L_2$. Como ejemplo de aplicación se utilizó el proceso químico Tennessee Eastman, un proceso ampliamente usado en el diagnóstico de fallas. El aislamiento de fallas mostró mejores resultados con respecto a los reportados en la literatura.Márquez-Vera, MA.; López-Ortega, O.; Ramos-Velasco, LE.; Ortega-Mendoza, RM.; Fernández-Neri, BJ.; Zúñiga-Peña, NS. (2021). Diagnóstico de fallas mediante una LSTM y una red elástica. Revista Iberoamericana de Automática e Informática industrial. 18(2):164-175. https://doi.org/10.4995/riai.2020.13611OJS164175182Adewole, A., Tzoneva, R., Behardien, S., 2016. Distribution network fault section identification and fault location using wavelet entropy and neural networks. Applied Soft Computing 46, 296-306. https://doi.org/10.1016/j.asoc.2016.05.013Alkaya, A., Eker, I., 2011. Variance sensitive adaptive threshold-based PCA method for fault detection with experimental application. ISA Transactions 50, 287-302. https://doi.org/10.1016/j.isatra.2010.12.004Barakat, S., Eteiba, M., Wahba, W., 2014. Fault location in underground cables using anfis nets and discrete wavelet transform. Journal of Electrical Systems and Information Technology 1, 198-211. https://doi.org/10.1016/j.jesit.2014.12.003Bathelt, A., Ricker, N., Jelali, M., 2015. Revision of the Tennessee Eastman process model. IFAC Papers-Online 48 (8), 309-314. https://doi.org/10.1016/j.ifacol.2015.08.199Boldt, F., Rauber, T., Varejao, F., October 2014. Evaluation of the extreme learning machine for automatic fault diagnosis of the Tennessee Eastman chemical process. In: IEEE (Ed.), Annual Conference of the IEEE Industrial Electronics Society. Vol. 40. Dallas, Texas, pp. 2551-2557. https://doi.org/10.1109/IECON.2014.7048865Chen, H., Tino, P., Yao, X., 2014. Cognitive fault diagnosis in Tennessee Eastman process using learning in the model space. Computers and Chemical Engineering 67, 33-42. https://doi.org/10.1016/j.compchemeng.2014.03.015Rodrigues, J., Filho, P., PeixotoJr., E., Kumar, A., deAlbuquerque, V., 2019. Classification of EEG signals to detect alcoholism using machine learning techniques. Pattern Recognition Letters 125, 140-149. https://doi.org/10.1016/j.patrec.2019.04.019Dixit, A., Majumdar, S., 2013. Comparative analysis of coiflet and daubechies wavelets using global threshold for image denoising. Intenational Journal of Advances in Engineering & Technology 6 (5), 2247-2252.Downs, J., Vogel, E., 1993. A plant-wide industrial process control problem. Computers and Chemical Engineering 17 (3), 245-255. https://doi.org/10.1016/0098-1354(93)80018-IFischer, T., Krauss, C., 2018. Deep learning with long short-term memory networks for financial market predictions. European Journal of Operational Research 270, 654-669. https://doi.org/10.1016/j.ejor.2017.11.054Gao, X., Hou, J., 2016. An improved SVM integrated GS-PCA fault diagnosis approach of Tennessee Eastman process. Neurocomputing 174, 906-911. https://doi.org/10.1016/j.neucom.2015.10.018Geng, Z., Li, Z., Han, Y., 2018. A new deep belief network based on RBM with glial chains. Information Sciences 463, 294-306. https://doi.org/10.1016/j.ins.2018.06.043Goodfellow, I., Bengio, Y., Courville, A., 2016. Deep Learning. MIT Press, United States of America, http://www.deeplearningbook.ogr.Han, L., Li, C., Guo, S., Su, X., 2015. Feature extraction method of bearing AE signal based on improved Fast-ICA and wavelet packet energy. Mechanical Systems and Signal Processing 62-63, 91-99. https://doi.org/10.1016/j.ymssp.2015.03.009Hastie, T., Tibshirani, R., Friedman, J., 2009. The elements of statistical learning: data mining, inference and prediction. Springer, New York. https://doi.org/10.1007/978-0-387-84858-7Hoang, D., Kang, H., 2019. A survey on deep learning based bearing fault diagnosis. Neurocomputing 335, 327-335. ttps://doi.org/10.1016/j.neucom.2018.06.078Hochreiter, S., Schmidhuber, J., 1997. Long short term memory. Neural Computation 9 (8), 1735-1780. ttps://doi.org/10.1162/neco.1997.9.8.1735Hyvärinen, A., Oja, E., 2000. Independent component analysis: Algorithms and applications. Neural Networks 13, 411-430. ttps://doi.org/10.1016/S0893-6080(00)00026-5Jing, C., Gao, X., Zhu, X., Lang, S., July 2014. Fault classificaction on Tennessee Eastman process: PCA and SVM. In: IEEE (Ed.), Intenational Conference on Mecatronics and Control. Jinzhou, China, pp. 2194-2197. https://doi.org/10.1109/ICMC.2014.7231958Jung, C., Kim, K., Lee, J., Klockl, B., 2007. Wavelet and neuro-fuzzy based fault location for combined transmission systems. Energy Systems 29, 445-454. https://doi.org/10.1016/j.ijepes.2006.11.003Kandula, V. K., 2011. Fault detection in process control plants using principal component analysis. Master's thesis, Louisiana State University, Department of Electrical Engineering.Karpenko, M., Sepehri, N., Octubre 2001. A neural network based fault detection and identification scheme for pneumatic process control valves. In: IEEE (Ed.), International Conference on Systems, Man and Cybernetics. Tucson, USA, pp. 93-98. https://doi.org/10.1109/ICSMC.2001.969794Khakipour, M., Safavi, A., Setoodeh, P., 2017. Bearing fault diagnosis with morphological gradient wavelet. Journal of the Franklin Institute 354, 2465-2476. https://doi.org/10.1016/j.jfranklin.2016.11.013Kuang, T., Yang, Z., Yao, Y., 2015. Multivariate fault isolation via variable selection in discriminant analysis. Journal of Process Control 35, 30-40. https://doi.org/10.1016/j.isatra.2017.06.014Kumar, R., Bansal, H., 2019. Hardware in the loop implementation of wavelet based strategy in shuntactive powerfilter to mitigate power quality issues. Electric Power Systems Research 169, 92-104. https://doi.org/10.1016/j.epsr.2019.01.001Lau, C., Ghosh, K., Hussain, M., Hassan, C. C., 2013. Fault disgnosis of Tennessee Eastman process with multi-scale PCA and ANFIS. Chemom. Intell. Lab. Syst. 120, 1-14. https://doi.org/10.1016/j.chemolab.2012.10.005Lee, J., Yoo, C., Lee, I., 2004. Statistical process monitoring with independent component analysis. Journal of Process Control 14 (5), 467-485. https://doi.org/10.1016/j.jprocont.2003.09.004Lei, J., Liu, C., Jiang, D., 2019. Fault diagnosis of wind turbine based on long short-term memory networks. Renewable Energy 133, 422-432. https://doi.org/10.1016/j.renene.2018.10.031Li, W., Monti, A., Ponci, F., 2014. Fault detection and classification in medium voltage DC shipboard power systems with wavelets and artificial neural networks. IEEE Transactions on Instrumentation and Measurement 63 (11), 2651-2665. https://doi.org/10.1109/TIM.2014.2313035Liang, P., Deng, C.,Wu, J., Yang, Z., Zhu, J., Zhang, Z., 2019. Compound fault diagnosis of gearboxes via multi-label convolutional neural network and wavelet transform. Computers in Industry 113, 103132. https://doi.org/10.1016/j.compind.2019.103132Lin, J., Zhang, A., 2005. Fault feature separation using wavelet-ICA filter. NDT&E International 38, 421-427. https://doi.org/10.1016/j.ndteint.2004.11.005Linker, R., Gutman, P., Seginer, I., 2002. Observer-based robust failure detection and isolation in greenhouses. Control Engineering Practice 10 (5), 519- 531. https://doi.org/10.1016/S0967-0661(02)00002-3Lou, W., Loparo, K., 2004. Bearing fault diagnosis based on wavelet transform and fuzzy inference. Mechanical Systems and Signal Processing 18, 1077-1095. https://doi.org/10.1016/S0888-3270(03)00077-3Lv, F.,Wen, C., Bao, Z., Liu, M., 2016. Fault diagnosis based on deep learning. In: AACC (Ed.), American Control Conference. Boston, USA, pp. 6851-6856. https://doi.org/10.1109/ACC.2016.7526751Lv, F., Wen, C., Liu, M., Bao, Z., 2017. Weighted time series fault diagnosis based on a staked sparce autoencoder. Journal of Chemometrics 31, 16 pages. https://doi.org/10.1002/cem.2912Lv, F., Fan, X., Wen, C., Bao, Z., 2018. Stacked sparse auto encoder network based multimode process monitoring. In: IEEE (Ed.), International Conference on Control Automation & Information Science. Hangzhou, China, pp. 227-232. https://doi.org/10.1109/ICCAIS.2018.8570618Maglaveras, N., Stamkopoulos, T., Diamantaras, K., Pappas, C., Strintzis, M., 1998. ECG pattern recognition and classification using non-linear transfor mations and neural networks: A review. International Journal of Medical Informatics 52, 191-208. https://doi.org/10.1016/S1386-5056(98)00138-5Methnani, S., Lafont, F., Gautier, J., Damak, T., Toumi, A., 2013. Actuator and sensor fault detection, isolation and identification in nonlinear dynamical systems, with applications to a waste water treatment plant. Journal of Computer Engineering and Informatics 1 (4), 112-125. https://doi.org/10.1080/21642583.2014.888525Muñoz-Cobo, J., Mendizábal, R., Miquel, A., Berna, C., Escrivá, A., 2017. Use of the principles of maximum entropy and maximum relative entropy for the determination of uncertain parameter distributions in engineering applications. Entropy 19, 486, 37 pages. https://doi.org/10.3390/e19090486Nguyen, B., Quyen, A., Nguyen, P., Ton, T., July 2017. Wavelet-based neural network for recognition of faults at nhabe power substation of the vietnam power system. In: IEEE (Ed.), International Conference on System Science and Engineering. Ho Chi Minh City, Vietnam, pp. 165-168. https://doi.org/10.1109/ICSSE.2017.8030858Ojeda-González, A., Mendes-Jr., O., Oliveira-Domingues, M., Menconi, V., 2014. Daubechies wavelet coeffcients: a tool to study interplanetary magnetic field fluctuations. Geof'ısica Internacional 53 (2), 101-115. https://doi.org/10.1016/S0016-7169(14)71494-1Oliveira, J., Pontes, K., Santori, I., Embirucu, M., 2017. Fault detection and diagnosis in dynamic systems using weightless neural networks. Expert Systems With Applications 84, 200-219. https://doi.org/10.1016/j.eswa.2017.05.020Patan, K., 2008. Artificial neural networks for the modelling and fault diagnosis of technical process. Lecture Notes in Control and Information Sciences. Springer, India.Rafiee, J., Rafiee, M., Tse, P., 2010. Application of mother wavelet functions for automatic gear and bearing fault diagnosis. Expert Systems with Applications 37, 4568-4579. https://doi.org/10.1016/j.eswa.2009.12.051Ramos-Velasco, L., Ramos-Fernández, J., Islar-Gómez, O., Espejel-Rivera, M., García-Lamont, J., Márquez-Vera, M., 2013. Identificación y control wavenet de un motor de ca. Revista Iberoamericana de Automática e Informática Industrial 10, 269-278. https://doi.org/10.1016/j.riai.2013.05.002Rato, T., Reis, M., 2013. Defining the structure of DPCA models and its impact on process monitoring and prediction ctivities. Chemometrics and Intelligent Laboratory Systems 125, 74-86. https://doi.org/10.1016/j.chemolab.2013.03.009Rockinger, M., Jondeau, E., 2002. Entropy densities with an application to autoregressive conditional skewness and kurtosis. Journal of Econometrics 106, 119-142. https://doi.org/10.1016/S0304-4076(01)00092-6Salahschoor, K., Kiasi, F., July 2008. On-line process monitoring based on wavelet-ICA methodology. In: IFAC (Ed.), Proceedings of the 17th World Congress. Seul- Korea, pp. 6-11. https://doi.org/10.3182/20080706-5-KR-1001.01253Salahshoor, K., Khoshro, M., Kordestani, M., 2011. Fault detection and diagnosis of an industrial steam turbine using a distributed configuration of adaptive neuro-fuzzy inference systems. Simulation Modelling Practice and Theory 19, 1280-1293. https://doi.org/10.1016/j.simpat.2011.01.005Sharif, I., Khare, S., 2014. Comparative analysis of Haar and Daubechies wavelet for hyper spectral image classification. In: Commission, I. T. (Ed.), VIII Symposium of The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Science. Hyderabad-India, pp. 937-941. https://doi.org/10.5194/isprsarchives-XL-8-937-2014Smirnov, E., Timoshenko, D., Adrianov, S., 2014. Comparison of regularization methods for imagenet classification with deep convolutional neural networks. AASRI Procedia 6, 89-94. https://doi.org/10.1016/j.aasri.2014.05.013Sobhani-Tehrani, E., Khorasani, K., 2009. Fault diagnosis of nonlinear systems using a hybrid approach. Fault detetion and diagnosis. Springer, Berlin, Ch. 2, pp. 22-49. https://doi.org/10.1007/978-0-387-92907-1_2Tayarani-Bathaie, S., Vanini, Z., Khorasani, K., 2014. Dynamic neural networkbased fault diagnosis of gas turbine engines. Neurocomputing 125, 153-165. https://doi.org/10.1016/j.neucom.2012.06.050Zvokelj, M., Zupan, S., Prebil, I., 2016. EEMD-based multiscale ICA method for slewing bearing fault detection and diagnosis. Journal of Sound and Vibration 26, 394-423. https://doi.org/10.1016/j.jsv.2016.01.046Wang, X., Qin, Y., Wang, Y., Xiang, S., Chen, H., 2019. ReLTanh: An activation function with vanishing gradient resistance for SAE-based DNNs and its application to rotating machinery fault diagnosis. Neurocomputing 363, 88-98. https://doi.org/10.1016/j.neucom.2019.07.017Wu, F., Tong, F., Yang, Z., 2016. EMGdi signal enhancement based on ICA decomposition and wavelet transform. Applied Soft Computing 43, 561-571. https://doi.org/10.1016/j.asoc.2016.03.002Wu, J., Hsu, C., Wu, G., 2009. Fault gear identification and classification using discrete wavelet transform and adaptive neuro-fuzzy inference. Expert Systems with Applications 36, 6244-6255. https://doi.org/10.1016/j.eswa.2008.07.023Wu, Q., Law, R., Wu, S., 2011. Fault diagnosis of car assembly line based on fuzzy wavelet kernel support vector classifier machine and modified genetic algorithm. Expert Systems with Applications 38, 9096-9104. https://doi.org/10.1016/j.eswa.2010.12.109Wu, H., Zhao., Jinsong, 2018. Deep convolutional neural network model based chemical process fault diagnosis. Computers and Chemical Engineering 115, 185-197. https://doi.org/10.1016/j.compchemeng.2018.04.009Xiao, C., Chen, N., Hu, C., Wang, K., Gong, J., Chen, Z., 2019. Short and midterm sea surface temperature prediction using time-series satellite data and LSTM-AdaBoost combination approach. Remote Sensing of Environment 233, 111358. https://doi.org/10.1016/j.rse.2019.111358Xie, D., Bai, L., December 2015. A hierarchical deep neural network for fault diagnosis on Tennessee-Eastman process. In: IEEE (Ed.), International Conference on Machine Learning and Applications. Vol. 14. Miami, USA, pp. 745-748. https://doi.org/10.1109/ICMLA.2015.208Yan, R., Gao, R., Chen, X., 2014. Wavelets for fault diagnosis of rotary machines: A review with applications. Signal Processing 351, 4555-4569. https://doi.org/10.1016/j.sigpro.2013.04.015Yan, Z., Yao, Y., 2015. Variable selection method for fault isolation using least absolute shrinkage and selection operator (LASSO). Chemometrics and Intelligent Laboratory Systems 146, 136-146. https://doi.org/10.1016/j.chemolab.2015.05.019Yao, G., Lei, T., Zhong, J., 2019. A review of convolutional-neural-networkbased action recognition. Pattern Recognition Letters 118, 14-22. https://doi.org/10.1016/j.patrec.2018.05.018Yin, S., Ding, S., Haghani, A., Hao, H., Zhang, P., 2012. A comparison study of basic data-driven fault diagnosis and process monitoring methods on the benchmark Tennessee Eastman process. Journal of Process Control 22, 1567-1581. https://doi.org/10.1016/j.jprocont.2012.06.009Zhang, Q., Yang, L., Chen, Z., Li, P., 2018. A survey on deep learning for big data. Information Fusion 42, 146-157. https://doi.org/10.1016/j.inffus.2017.10.006Zhang, X., Polycarpou, M., Parisini, T., 2002. A robust detection and isolation scheme for abrupt and incipient faults in nonlinear systems. IEEE Transactions on Automatic Control 47 (4), 576-593. https://doi.org/10.1109/9.995036Zhang, Y., Zhang, L., Zhang, H., 2012. Fault detection for industrial processes. Mathematical Problems in Engineering 2012, 18 pages. https://doi.org/10.1155/2012/757828Zhang, Z., Zhao, J., 2017. A deep belief network based fault diagnosis model for complex chemical process. Computers and Chemical Engineering 107, 395-407. https://doi.org/10.1016/j.compchemeng.2017.02.041Zhao, H., 2018. Neural component analysis for fault detection. Chemometrics and Intelligent Laboratory Systems 176, 11-21. https://doi.org/10.1016/j.chemolab.2018.02.001Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P., Gao, R., 2019. Deep learning and its applications to machine health monitoring. Mechanical Systems and Signal Processing 115, 213-237. https://doi.org/10.1016/j.ymssp.2018.05.050Zheng, J., Huang, W., Wang, Z., Liang, J., 2019. Mutual information-based sparse multiblock dissimilarity method for incipient fault detection and diagnosis in plant-wide process. Journal of Process Control 83, 63-76. https://doi.org/10.1016/j.jprocont.2019.09.00

Application of Deep Learning in Chemical Processes: Explainability, Monitoring and Observability

The last decade has seen remarkable advances in speech, image, and language recognition tools that have been made available to the public through computer and mobile devices’ applications. Most of these significant improvements were achieved by Artificial Intelligence (AI)/ deep learning (DL) algorithms (Hinton et al., 2006) that generally refers to a set of novel neural network architectures and algorithms such as long-short term memory (LSTM) units, convolutional networks (CNN), autoencoders (AE), t-distributed stochastic embedding (TSNE), etc. Although neural networks are not new, due to a combination of relatively novel improvements in methods for training the networks and the availability of increasingly powerful computers, one can now model much more complex nonlinear dynamic behaviour by using complex structures of neurons, i.e. more layers of neurons, than ever before (Goodfellow et al., 2016). However, it is recognized that the training of neural nets of such complex structures requires a vast amount of data. In this sense manufacturing processes are good candidates for deep learning applications since they utilize computers and information systems for monitoring and control thus generating a massive amount of data. This is especially true in pharmaceutical companies such as Sanofi Pasteur, the industrial collaborator for the current study, where large data sets are routinely stored for monitoring and regulatory purposes. Although novel DL algorithms have been applied with great success in image analysis, speech recognition, and language translation, their applications to chemical processes and pharmaceutical processes, in particular, are scarce. The current work deals with the investigation of deep learning in process systems engineering for three main areas of application: (i) Developing a deep learning classification model for profit-based operating regions. (ii) Developing both supervised and unsupervised process monitoring algorithms. (iii) Observability Analysis It is recognized that most empirical or black-box models, including DL models, have good generalization capabilities but are difficult to interpret. For example, using these methods it is difficult to understand how a particular decision is made, which input variable/feature is greatly influencing the decision made by the DL models etc. This understanding is expected to shed light on why biased results can be obtained or why a wrong class is predicted with a higher probability in classification problems. Hence, a key goal of the current work is on deriving process insights from DL models. To this end, the work proposes both supervised and unsupervised learning approaches to identify regions of process inputs that result in corresponding regions, i.e. ranges of values, of process profit. Furthermore, it will be shown that the ability to better interpret the model by identifying inputs that are most informative can be used to reduce over-fitting. To this end, a neural network (NN) pruning algorithm is developed that provides important physical insights on the system regarding the inputs that have positive and negative effect on profit function and to detect significant changes in process phenomenon. It is shown that pruning of input variables significantly reduces the number of parameters to be estimated and improves the classification test accuracy for both case studies: the Tennessee Eastman Process (TEP) and an industrial vaccine manufacturing process. The ability to store a large amount of data has permitted the use of deep learning (DL) and optimization algorithms for the process industries. In order to meet high levels of product quality, efficiency, and reliability, a process monitoring system is needed. The two aspects of Statistical Process Control (SPC) are fault detection and diagnosis (FDD). Many multivariate statistical methods like PCA and PLS and their dynamic variants have been extensively used for FD. However, the inherent non-linearities in the process pose challenges while using these linear models. Numerous deep learning FDD approaches have also been developed in the literature. However, the contribution plots for identifying the root cause of the fault have not been derived from Deep Neural Networks (DNNs). To this end, the supervised fault detection problem in the current work is formulated as a binary classification problem while the supervised fault diagnosis problem is formulated as a multi-class classification problem to identify the type of fault. Then, the application of the concept of explainability of DNNs is explored with its particular application in FDD problem. The developed methodology is demonstrated on TEP with non-incipient faults. Incipient faults are faulty conditions where signal to noise ratio is small and have not been widely studied in the literature. To address the same, a hierarchical dynamic deep learning algorithm is developed specifically to address the issue of fault detection and diagnosis of incipient faults. One of the major drawbacks of both the methods described above is the availability of labeled data i.e. normal operation and faulty operation data. From an industrial point of view, most data in an industrial setting, especially for biochemical processes, is obtained during normal operation and faulty data may not be available or may be insufficient. Hence, we also develop an unsupervised DL approach for process monitoring. It involves a novel objective function and a NN architecture that is tailored to detect the faults effectively. The idea is to learn the distribution of normal operation data to differentiate among the fault conditions. In order to demonstrate the advantages of the proposed methodology for fault detection, systematic comparisons are conducted with Multiway Principal Component Analysis (MPCA) and Multiway Partial Least Squares (MPLS) on an industrial scale Penicillin Simulator. Past investigations reported that the variability in productivity in the Sanofi's Pertussis Vaccine Manufacturing process may be highly correlated to biological phenomena, i.e. oxidative stresses, that are not routinely monitored by the company. While the company monitors and stores a large amount of fermentation data it may not be sufficiently informative about the underlying phenomena affecting the level of productivity. Furthermore, since the addition of new sensors in pharmaceutical processes requires extensive and expensive validation and certification procedures, it is very important to assess the potential ability of a sensor to observe relevant phenomena before its actual adoption in the manufacturing environment. This motivates the study of the observability of the phenomena from available data. An algorithm is proposed to check the observability for the classification task from the observed data (measurements). The proposed methodology makes use of a Supervised AE to reduce the dimensionality of the inputs. Thereafter, a criterion on the distance between the samples is used to calculate the percentage of overlap between the defined classes. The proposed algorithm is tested on the benchmark Tennessee Eastman process and then applied to the industrial vaccine manufacturing process

Sensors Fault Diagnosis Trends and Applications

Fault diagnosis has always been a concern for industry. In general, diagnosis in complex systems requires the acquisition of information from sensors and the processing and extracting of required features for the classification or identification of faults. Therefore, fault diagnosis of sensors is clearly important as faulty information from a sensor may lead to misleading conclusions about the whole system. As engineering systems grow in size and complexity, it becomes more and more important to diagnose faulty behavior before it can lead to total failure. In the light of above issues, this book is dedicated to trends and applications in modern-sensor fault diagnosis

Novel Deep Learning Techniques For Computer Vision and Structure Health Monitoring

This thesis proposes novel techniques in building a generic framework for both the regression and classification tasks in vastly different applications domains such as computer vision and civil engineering. Many frameworks have been proposed and combined into a complex deep network design to provide a complete solution to a wide variety of problems. The experiment results demonstrate significant improvements of all the proposed techniques towards accuracy and efficiency