The Multivariate State Estimation Technique (MSET) has been developed at Argonne National Laboratory (ANL) and applied for real time surveillance applications for the purposes of signal validation, sensor operability validation, equipment health monitoring, incipient component fault annunciation, and process anomaly identification. Although MSET was originally developed for applications in the commercial nuclear industry, it has recently been spun off for applications in fields such as aerospace, manufacturing, transportation, robotics, and ship propulsion. Notwithstanding these types of successful applications of MSET in industry, it is necessary for safety-critical and mission-critical applications of MSET to have reliability analysis methods, including a propagation-of-uncertainty tool, which is needed to support safety evaluations in a variety of industries, and technical-specification-change requests in the case of the nuclear industry. For these and related applications, a general purpose uncertainty analysis tool for MSET has been developed that uses Monte Carlo simulation with Latin Hypercube Sampling. For any new application of MSET, the uncertainty analysis tool developed here may be used to investigate quantitative propagation-of-uncertainty behavior for all sensors under surveillance. In addition to supporting safety analysis of surveillance systems that are based on MSET, the tool developed here can be employed in parametric studies to support system designers in evaluating the relative value of adding new sensors to an engineering system during early design stages or for equipment or facility upgrades

Gross, K. C.

Zavaljevski, N.

UNT Digital Library

A/!L/!2’%k%’-An% 74’*The paper submitted for presentation at the Maintenance and Reliability Conferencej“Maintenance and Reliability in the 21s’ Century”, May 7-11, 2000,Knoxville, Tennessee,USAUncertainty Analysis for Multivariate State Estimation in Safety-Criticaland Mission-Critical Maintenance ApplicationsbyNela Zavaljevski and Kenny C. Gross0Reactor Analysis DivisionArgonne National Laboratory9700 S. Cass AvenueArgonne, IL 60439The submitted manuscript has been created by theUniversity of Chicago as Operator of ArgonneNational Laboratory (“Argome”) under Conlract No.W-3 I-109-ENG-38 with the U.S. Department ofEnergy. The U.S. Government retains for itself, andothers acting on its behalf, a paid-up, non-exchsive,irrevocable worldwide license in said article toreproduce, prepare derivative works, dkibute copiesto the public, and perform publicly and dispIaypublicly, by or on behalf of the Government.I I*Work supported by the U. S. Department of Energy, Nuclear Energy Programs underContract W-31 -109-Eng-38DISCLAIMERThis report was prepared as an account of work sponsoredby an agency of the United States Government. Neitherthe United States Government nor any agency thereof, norany of their employees, make any warranty, express orimplied, or assumes any legal Iiabiiity or responsibility forthe accuracy, completeness, or usefulness of anyinformation, apparatus, product, or process disclosed, orrepresents that its use would not infringe privately ownedrights. Reference herein to any specific commercialproduct, process, or service by trade name, trademark,manufacturer, or otherwise does not necessarily constituteor imply its endorsement, recommendation, or favoring bythe United States Government or any agency thereof. Theviews and opinions of authors expressed herein do notnecessarily state or reflect those of the United StatesGovernment or any agency thereof.DISCLAIMERPortions of this document may be illegiblein electronic image products. Images areproduced from the best avaiiable originaldocument.Uncertainty Analysis for Multivariate State Estimation in Safety-Critical and Mission-Critical Maintenance ApplicationsNela Zavaljevski and Kenny C. GrossReactorAnalysisDivision,Argome NationalLaboratory9700 South Cliss Avenue,Argonne,IL 60439Phone:630-252-6617nelaz@ra.anl.~ov ggoss@ull.PovABSTRACTThe Multivariate State Estimation Technique (MSET) has been developed at Argonne NationalLaboratory (ANL) and applied for real time surveillance applications for the purposes of signalvalidation, sensor operability validation, equipment health monitoring, incipient component faultannunciation, and process anomaly identification. Although MSET was originally developed forapplications in the commercial nuclear industry, it has recently been spun off for applications in fieldssuch as aerospace, manufacturing, transportation, robotics, and ship propulsion. Notwithstandingthese types of successful applications of MSET in industry, it is necessary for safety-critical andmission-critical applications of MSET to have reliability analysis methods, including a propagation-of-uncertainty tool, which is needed to support safety evaluations in a variety of industries, andtechnical-specification-change requests in the case of the nuclear industry. For these and relatedapplications, a general purpose uncertainty analysis tool for MSET has been developed that usesmonte carlo simulation with Latin Hypercube Sampling. For any new application of MSET, theuncertainty analysis tool developed here may be used to investigate quantitative propagation-of-uncertainty behavior for all sensors under surveillance. In addition to supporting safety analysis ofsurveillance systems that are based on MSET, the tool developed here can be employed in parametricstudies to support system designers in evaluating the relative value of adding new sensors to anengineering system during early design stages or for equipment or facility upgrades.INTRODUCTIONA pattern recognition technique for sensor and equipment surveillance has been developed and patented byANL and implemented in a software system called MSET (Multivariate State Estimation Technique) (Grosset al, 1997). The MSET algorithm is a nonlinear deterministic model. It uses a set of nonlinear operators toobtain system state estimates that are based on data stored in a matrix called the memory matrix. Theuncertainty components associated with each element of the memory matrix, which arise due to the randomnature of the physical process itself or to measurement errors, contribute to the overall uncertainty of theMSET output estimates. In order to quantify the uncertainties and extend the domain of MSET applicationsto include safety-critical and missionaitical areas, we have developed uncertainty analysis tools that are basedon the Latin Hypercube Sampling (LHS) scheme (McKay, 1988). The LHS approach was adopted for ourinvestigation because it has an advantage over standard random sampling procedures in the manner in whichit systematically and uniformly covers the domain of uncertain parameters. This approach has been usedsuccessfully in a wide variety of safety-critical applications, including safety assessments for high levelradioactive waste repositories (Helton, 1993).METHODOLOGYThe total number of uncertain parameters M in MSET is equal to the product m x n, where m is the numberof training vectors (observations uniformly sampled in time) for each sensor, and n is the total number ofsensors under surveillance. The sampling for each of the M parameters in the LHS algorithm is performedin the following way. N distinct values of any given parameter are chosen to span the range of the parameterin a stratified manner. Accordingly, the range is divided into N intervals of equal-probability content (basedon the cumulative distribution function, CDF), and a value is randomly sampled from each of the stratifiedCDF intervals. The values are assigned at random to the N model runs. This is accomplished by generatinga random permutation of integers from 1 to N. For a LHS scheme with M parameters, M randompermutations of N integers are generated, and a permutation matrix with dimensions M x N is produced. Inmost practical MSET applications this matrix becomes very large. The number N should be at least of theorder of M if uncertainty measures in addition to the mean and variance are desired. This makes the problemcomputationally very intensive. For this reason, it is necessary to develop a program that will efficientlymakeuse of available computational resources. In particular, we have implemented the LHS algorithm fordistributed concurrent computing on a network of UNIX workstations using the Distributed Queuing System(Green, 1992).A general error model has been implemented that can generate a specified noise distribution for anycombination of sensors under surveillance. In addition to Gaussian noise, Laplacian noise has been added,characterized by larger tails. This will enable more conservative confidence interval estimation. Signalaut~orrekttion effects, which the authors have observed in many reactor and aerospace applications of MSET,can be added to either distribution. The option has also been provided to generate both deterministic andrandom drift for any combination of sensors in the model. In this case the start times for drift can be selectedfrom a uniform distribution. The drift slopes maybe selected from a triangular distribution, or constant driftslopes can be specified.The LHS-based algorithm developed for MSET reliability analysis is amenable to at least two levels ofdistributed concurrent computing. On the first, coarse level, noise distribution parameters are preassigned.For each set of parameters, a LHS analysis is performed by computing random realizations of the noise. Eachjob is assigned to a workstation in the network. This enables a faster throughput even though a large portionof the code remains sequential.A second and higher level of parallelism is achieved by partitioning the perturbation domain among availableprocessors. In this case the LHS algorithm requires a list of random permutations, computed in advance andsaved in a file. Each line of the file contains perturbed parameters for one sample case. The number ofsamples necessary for confidence interval estimation is proportional to the product of the number of sensorsand the number of data observations. This means that for realistic data sets the sample size will be on tkieorder of 104. The perturbation file is therefore divided into several files, and each new file is assigned to oneworkstation on the network. After completion of all processes, results are evaluated and confidence intervalsand other statistical measures are computed using a single processor.RESULTSTo illustrate the developed LHS uncertainty analysis methodology, we used a small subset of 5 signals fromthe Experimental Breeder Reactor-II (EBR-11). Both training and testing sets have 500 observations (for a totalof 1000 observations). This means that perturbations in 5000 data values shouId be introduced.Two classes of perturbations have been introduced to investigate the performance of the MSET estimationalgorithm: additive noise perturbations, and drift perturbations.a. Additive noise perturbationsGaussian additive noise in both the testing and the training data has been studied first. Sensor M of the fiveEBR-11sensors was selected and is defined as the “true” value. To this defined true signal we want to addsome noise to simulate measurement error, and we add noise to sensors #2 -#5 as well. Our objective is tothen compare the MSET estimate for signal #l to our original unperturbed signal #l. The difference betweenthe MSET estimate and the original unperturbed signal, called the residual fhnction, provides a measure of theerror in the estimation procedure embodied in MSET. Accordingly, sensor #1 of the five sensors wasperturbed with a relatively large noise level (signal to noise ratio, SNR = 3), while a smaller noise level wassuperimposed onto the other sensor signals (SNR = 100).To reduce the variance in the MSET estimates, Tikhonov reguhuization, recently implemented in MSET, hasbeen applied. Jn a previous paper we applied Generalized Cross Validation to select the regularizationparameters (Zavaljevski et al., 1999). Here we selected the regularization parameters empirically, based onavailable simulation results. Table 1 shows the bias and variance in MSET estimates for three values of theregularization parameter.Table 1. MSET Performance for Various Values of the Regularization Parameterx=O.000 k= 0.001 k==.oo5Mean 853.64 853.64 853.66Standard Deviation 0.371 0.333 0.264The standard deviation of the original EBR-11test signal, defined as the “true” signal for this exercise, was0.330. The standard deviation of the perturbed signal #1 was 0.385. Note from Table 1 that the standarddeviation of the noisy signal can be reduced from 0.385 to 0.371 using the regularization parameter k= 0.000,and to 0.333 using the regularization parameter l.= 0.001. Since the original signal standard deviation is0.330, further increases in the regularization parameter would not be desirable because when the variance ofthe MSET estimate is reduced below the variance of the “true” signal, there is a loss of signal informationcontent. Thus, for the example application reported here, the optimal value of the regularization parameteris very close to 1= 0.001.To check the robustness of MSET estimates with respect to the noise distribution, L.aplaciannoise has beenintroduced in the training data with the same variance as examined above for the Gaussian distribution. TheLaplacian distribution has larger tails, and the training data now have a larger range. However, the optimalregukwization parameter keeps the variance of the estimate at the same level as with the Gaussian training damthus effectively denoising the testing data set. The histograms of the testing data and the MSET estimates aregiven in Figure 1.Confidence intervals for the same test are given in Figure 2.b. Drift PerturbationsIn this test the training data set was the same as in the previous two cases. A drift perturbation was introducedin the testing data in such a way that at the end of the testing period the perturbation was equivalent to about2 standard deviations of the training data. The effects of regularization for this case are presented in Figures3-5.Figure 3 indicates that once MSET is trained with noisy data, it may not be able to recognize the driftperturbation within the training range unless regularization is applied. With the regularization parameterselected optimally, it is clear that there will be a significant increase in the residual, and hence the driftperturbation will be easily detected. The same effect is clearly seen in Figures 4 and 5. In each figure the“estimated” residual is the difference between the MSET estimate and the perturbed signal, while the trueresidual is the difference between the MSET estimate and the uncorrupted (or “true”) signal, which is knownin this simulation study.CONCLUSIONSA general-purpose uncertainty analysis tool for MSET has been developed using Latin Hypercube Samplingin a monte carlo paradigm. The uncertainly analysis results of MSET estimates show the stability of theestimates with respect to small perturbations introduced in the training and testing data. In particular, it hasbeen shown that a simple regularization procedure can reduce the training data noise and enable accuratesignal estimation. The same procedure enables very subtle perturbations in the testing data to be revealed evenwhen the training is performed with noise. Further study will be conducted to investigate the effects ofperturbations on the empirical false alarm and missed alarm probabilities obtained with the MSET surveillancesystem.1400012W01Oooc800C60QC400C200CcE. . . .. . . . . . . . . . ...... . . . . .. . . . . . . . . . . . .... . . . . . . . . . . . . . .... . . . . . . . . . . .:,A852. . . .. . . . . . .. . . . . . .. . . . . . .I...... ..- SignalM Estimate. . . . . . . .. . . . . . . . . . . ... . . . . . . . . . . . .. . . . . . . . . . . ... . . . . . . . . . . . . .. . . . . . . ... . . . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . . . .853 854 855 858 i7Figure 1. Histograms of the MSET Estimates and the Testing Data with Laplacian Noise95% Confidence Interval for MSET Estimate of signal 13556546536520 50 100 150 200 250 300 350 403 450 50095% Confidence Interval for MSET Estimate of Signal 2S91.5 1 1 t 1 I 1 I 1. . . . . . . . .. . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . .. . . ..4 .. . . ... . . . . . . . . ... . . . . . . . . .. .. . .:tik&G*h$%\@vi~~.:."..l..:...; . . . . . . . . . ... . . . . . . . . ... .. . . .. J........w.o 50 100 150 200 250 300 3s0 400 450 50095’7. Confidence Interval for MSET Estimate of Signal 3893892891890 I I I I I I 1 I 1 I Io 50 100 150 200 250 300 350 4oa 450 5c095% Confidence Interval for MSET Estimate of Sianal 4‘b:...:sa3 i “ :!%!d332 I [ I I I I I I I 1 Io 50 100 150 200 250 300 350 400 454 50095% Confidence Interval for MSET Estimate of Signal 5666 1 1 I I I I I 1 1— MSET Upper Sound. . . . .. . . . . ... .363 - . . . .. . . . . .. . . . . . . . . . .. . . . .. . . . . . .. . . . . . . . . ... . . . . ... . . . . . . . . .. . . . ,. ..:.. .1::::: :::: I“.-o 50 100 150 200 250 3C!JI 350 400 450 500Figure 2. Confidence Intervals for MSET Estimatesx 104 Histogram of Standardized Residuals (MSET)2[ I I I 1 1 1 1-h8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6Standardized Residual Samples (MSH)0.4 [!l !!!!: ‘!! I0.20-0.2-0.4-0.6 [ I t 1 I I I I t 1 I Io 50 100 150 200 250 300 350 400 450 500Histogram of Standardized Residuals (Regularized MSET)8000 r I 1 1 I 1. . . . . .. . . . . . . . . . . . . . . . . . ... . . . . . . . . . . . . ... . . . . . . . . . . . . . . .. . . . . ... . . . . . . . . . ... . . . . . . .. . . . .. . . . . . . . . . .. . . . .-’1.5 -1 -0.5 0Standardized Residual Samples (Regularized MSET)0.5 10.50-0.5-1-,..Jo 50 100 150 200 250 300 350 400 450 500Figure 3. Statistical Measures of the MSET Estimates for Drift PerturbationsRMS Residuals for Signal 10.40.30.20.100 50 100 150 200 250 300 350 40Q 450 500RMS Residuals for Signal 20.20.150.10.0500 50 100 150 200 250 3oa 350 400 450 5(20RMS Residuals for Signal 30.20.150.10.G5o0 50 100 150 200 250 300 350 400 450 5RMS Residuals for Signal 40.2 I I I 1 t & t 1 L-9- Eatimeted Residual0.15 . . . . . . . .. . . . . . . .. . . . . . . . . . .. . .. :..,. . . .. . . . . . . . . ... . . . . . . . ... . . . . + True Reakhl0.10.050 50 100 150 200 250 300 350 400 450 500RMS Residuals for Signal 5: -I:!’; ‘ ‘ ‘ ‘ ‘I I 1 t I I 1 t 1+3- EstimstedReskmal.y . . . . .. ..~ . . . . . ... . . . . . . . . ... . . . . . . . . ... . . . . . . . . ... . . . . . + TmeR=k@l0.10.0500 50 100 150 200 250 300 350 400 450 500Figure 4. Standard MSET Residuals for Drift Perturbations..RMS Residuals for Signal 10.5 i 1 I 1 1 I 1 1 1(3: :_ [ -9- Eetkneted Reeidual [0.40.30.20.100 50 100 150 200 250 300 350 400 450 500RMS Residuals for Signal 20.40.30.20.1n“o 50 100 150 200 250 300 350 400 450 500RMS Residuals for Signal 30.40.30.20.100 50 100 150 200 250 300 350 400 450 500RMS Residuals for Sianal 40.40.30.20.100 50 100 150 200 250- 300 35a 400 450 SWRMS Residuals for Signal 50.50.40.30.20.1n-o 50 100 150 200 250 300 350 400 450 500Figure 5. Regularized MSET Residuals for Drift Perturbations●REFERENCESK. C. Gross, R. M. Singer, S. W. Wegench, J. P. Herzog, R. Van Alstine, and F. Blockhorst (1997)“Application of a Model-Based Fault Detection System to Nuclear Plant Signals; Proc. 9’hML Conj OnIntelligent Systems Application to Power Systems, Seoul, Korea.M. D. McKay (1988) “Sensitivity and Uncertainty Analysis Using a Statistical Sample of Input Values: inUncertain~ Analysis, Y.Ronen, Ed, CRC Press, Boca Raton, Florida.J. C. Helton (1993) “Uncertainty and Sensitivity Analysis Techniques for Use in Performance Assessmentfor Radioactive Waste Disposal: Reliability Engineering and System Safety 42,327-367.T. P. Green (1992) “The Distributed Queuing System,” Supercomputer Computations Research Institute,Florida State University.N. Zavaljevski, K. C. Gross and S. W. Wegench (1999) “Regularization Methods for the MultivariateState Estimation Technique (MSET), “Proc. hztl. Conjl on Mathematics and Computations, ReactorPhysics and Environmental Analysis in Nuclear Applications, Madrid, Spain.

Uncertainty analysis for multivariate state estimation in safety-critical and mission-critical maintenance applications

Abstract

Similar works

Full text

Available Versions

UNT Digital Library