Back-propagation neural networks were used to classify PET scans as either normal or abnormal, with abnormal subjects defined as subjects who had previously been clinically diagnosed with memory disorders. Numerous neural network experiments were performed in order to achieve optimization with respect to number of hidden units and training duration. Optimizations and performance evaluations were based on ROC analysis, in which the area under the ROC curve was the figure of merit. The neural network's performance was better than that of dlscrlminant analysis, and comparable to the expert's performance, despite the low resolution image data, which consisted of one value per brain lobe, provided to the network

Barker, Warren W.

Duara, Ranjan

Kippenhan, Jonathan Shane

Nagel, Joachim H.

Pascal, Shlomo

OPUS Online Publikationen der Universität Stuttgart

Neural Networks 24.7-5 Optimization and Evaluation of a Neural-Network Classifier for PET Scans of Memory-Disorder Subjects J. Shane Kippenhan I .2, Warren W. Barkex2, Shlomo Pascal2, Ranjan Duara2, Joachim Nagell ' 1 Dept. of Biomedical Engineering, University of Miami, Coral Gables, FL 2 Wien Center for Memory Disorders, Mt. Sinai Medical etr., Miami Beach, FL ABSTRACT Back-propagation neural networks were used to classify PET scans as either normal or abnormal, with abnormal subjects defined as subjects who had previously been clinically diagnosed with memory disorders. Numerous neural network experiments were performed in order to achieve optimization with respect to number of hidden unJts and training duration. Optimizations and performance evaluations were based on ROC analysis, in which the area under the ROC curve was the figure of merit The neural network's performance was better than that of dlscrlminant analysis, and comparable to the expen's performance. despite the low resolution image data, which consisted of one value per brain lobe, provided to the network. INTRODUCI10N QuantitatJve approaches to the analysis and/or classification of Positron Emission Tomography (PE1) scans usually involve a reglon-of·interest (ROI) analysis, in which regional metabolic function in the brain is evaluated [I). Pattern recognition stUdies are then performed on these data. Various pattern recognltion technJques, including the back-propagation neural network (2), have been applied to the classification of normal and abnormal PET scans based on ROI data. Neural networks appear to perform better than standard statistical methods like dlscrlminant analysis (3). In the literature describing various recent applications of neural networks, there appears to be relatJvely little standardization In neural-network training. Networks are often trained to satisfy panicular "convergence criteria", which essentially specify how well the hypersurfaces defined by the network are able to separate the different classes comprising the trainJng set A more imponant consIderation in most drcumstances, however, is the abillty of a network to generalize and Identify previOUSly-unseen patterns, an issue which involves the number of traIning patterns, the dimensionality of these patterns, the architecture of the network (e.g., the number of hidden unJts) and number of trainJng Iterations. Complex networks trained on high-dlmenslonaJ patterns for an excessive number of iterations may tend to "memorize" their training sets, and learn criteria that are not generally applicable to populations of given pattern classes. Evaluation of a network's abillty to generalize is accomplished by cross-validation studies, that is, testing trained networks on new and independent data sets. The question then arises: what is the most appropriate figure of merit for performance evaluation? The ROC (Relative-Operating-Characteristic) method of analysis has recently come to be recognized as an Objective and comprehensive way to evaluate diagnostic systems, since It measures a diagnostic system's performance independent of dedsion biases and prior probabillties (4). The ROC curve represents a system's performance at several different settings of the particular dedsion criteria. The area under the curve is the "only performance measure available that is unin1Juenced by dedsion biases and prior probabillties. and it places the performances of diverse systems on a common, easily interpreted scale" (4). The method of optimization to be presented here is based on cross-validation studies, and employs the area under the ROC curve as the figure of merit Neural network performances were evaluated for subject groups with different levels of dementia, and performances were compared to the performances of an expen human reader (RO) and to those of discriminant analysis. OPTIMIZATION A back-propagation neural network with one hidden layer was applied to three separate groups of subjects, each group contaIning two classes: normal and abnormal (clinical diagnoses were used as reference standards). The "abnormal" class was represented by, in order of decreasing dementia severity, "Late Probable Alzheimer's Disease (AD)" subjects, "Probable AD" SUbjects and "Possible AD" subjects. Neural-network classification performances were evaluated for different combinations of number of hidden units and traIning duration. TrainIng and testing was performed twenty times for each combination. Each training session staned with a new set of randomized weights. All traIning was conducted with a learning rate of 0.7, and a momentum constant of 0.9 (2). Figure 1 depicts the results of experiments which tested performance on the "Possible AD" group. ~ a: « o o a: 1 4 LOG( NO. OF TRAINING ITERATIONS) FIgure 1: Cross-validation performance for "Possible AD" subject group over a range of traIning parameter values. Standard Error (SE) ranged from 0.003 to 0.01. Lowest SE's occurred around 10 iterations. Figure 1 indicates that the best performance was obtained by training for just ten iterations, and that a network with six hidden unJts performed no better at this point than one with four hidden unJts. Funher iterations resulted in lower ROC areas, indicating that the network was being "ovenrained". These trends were typical of experiments with other groups. It can be seen that the best cross-validation performance was achieved after relatively few iterations. The network attains its greatest generalization ability before it has converged to any significant degree. 1472 Annual International Conference of the IEEE EngIneering In MediCIne and BIology Society, Vol. 13. No.3. 1991 CH3068-4/9110000-1472 $01.00 " 1991 IEEE The claim that this combination is "optimum" Cor this group is best substantiated by cwnin.ing the network's performance with regard to its own training seL Figures 2 and 3 indicate that, after long training periods, the networks learn their training sets very well At this point, RMS output error is very low, and there is no impetus Cor further change in cross-valiciation performance. i5 II: ~ o o II: 4 LOG( NO. OF TRAINING ITERATIONS) FIgure 2: Results oC testing on network's training set over a range oC training-parameter values. II: o II: II: W ct> :E II: 1 2 :3 LOG( NO. OF TRAINING ITERATIONS) FIgure 3: Network RMS output error Cor training set over a range oC training-parameter values. EVALUATION For each subject group, the best performance over the given range oC hidden units and training duration was chosen, and an average performance (Cor the 20 experiments that were performed at this "optimum" combination) was calculated. For the "Late Probable AD" group, two hidden units proved sufficient, while Cour were required Cor the other two groups. Figure 4 depicts the ROC curve Cor the "Late Probable" group. o 1.0 ..... E-o D3 O.B r::I > 0.6 ..... E-o ..... [1J o p.. r::I ::> 0:= E-o 0.4 0.2 0.0 ---------- ---• NEURAL NET EXPERT DISC. ANAL. 0.0 0.2 0.4 0.6 O.B 1.0 FALSE POSITIVE RATIO FIgure 4: ROC curve Cor "Late Probable AD" group. Complete results arc summarized in the table below: Method Expert Network Dlscr. Anal. Group Late Probable AD ft. 093 0.92 0.87 Age-EqulY Normal Probable AD ft. 0.89 0.85 0.78 Age-EquJv Normal Possible AD ft. 0.81 0.81 0.74 Age-EqulY Normal Table: Classification performance oC various classification methods. Each value represents the area under the ROC curve Cor a given classification method. Direct comparisons between network performance and discriminant analysis performance are quite valid, since each method was given pattern information in the same Cormat: each pattern was represented by a group oC eight values, one Cor each lobe oC the brain. In comparing network performance to the expert's performance, the necessarily low-resolution "view" that the neural network had oC each PET study cenainly represented a significant handicap. Another factor which may have Influenced the results is the fact that, from a quantitatfve.classifier point oC view, the expert's "testing set" was not independent, since the expert was clinically Camlliar with all oC the tested subjects. Part oC the expert's training included subjects In the testing seL [1) (2) (3) (4) REFERENCES Haxby, JV. Resting state regional cerebral metabolism in dementia oC the Alzheimer type. In: Positron EmIssion Tomography In Dementia. R. Duara, ed. Wiley-Uss, pp. 93-116,1990. Rumelhart, DE, Hinton, GE and RJ Williams. Learning intemal representations by error propagation, In: Paralld Distributed Processing, VoLl, Rumelhart, D.E. and J.L. McClelland and the PDP Research Group, MIT Press, pp. 318-364, 1986. Kippenhan, JS and JH NageL Diagnosis and modelling oC Alzheimer's disease through neural network analyses oC PET studies,Proe 12th Ann Int Conr or IEEFJEMBS, 12, pp. 1449-SO,I990. Swets, JAo Measuring the accuracy oC diagnostic systems. Science, 240, pp. 1285-1293, 1988 Correspondence: Jonathan Shane Kippenhan Department oC Biomedical Engineering P.O. Box 248294 University oC MiamI Coral Gables, FL 33124 Annuallntematlonal Conference of the IEEE Engineering In Medicine and Biology Society. Vol. 13, No.3, 1991 CH3068-4/9110000·1473 $01.00 C 1991 IEEE 1473 

Optimization and evaluation of a neural network classifier for PET scans of memory disorder subjects

Abstract

Similar works

Full text

Available Versions

OPUS Online Publikationen der Universität Stuttgart