A Survey on Evaluation Metrics for Backchannel Prediction Models

In this paper we give an overview of the evaluation metrics used to measure the performance of backchannel prediction models. Both objective and subjective evaluation metrics are discussed. The survey shows that almost every backchannel prediction model is evaluated with a different evaluation metric. This makes comparison between developed models unreliable, even beside the other variables in play, such as different corpora, language, conversational setting, amount of data and/or definition of the term backchannel

Evaluation of Intelligent Intrusion Detection Models

This paper discusses an evaluation methodology that can be used to assess the performance of intelligent techniques at detecting, as well as predicting, unauthorised activities in networks. The effectiveness and the performance of any developed intrusion detection model will be determined by means of evaluation and validation. The evaluation and the learning prediction performance for this task will be discussed, together with a description of validation procedures. The performance of developed detection models that incorporate intelligent elements can be evaluated using well known standard methods, such as matrix confusion, ROC curves and Lift charts. In this paper these methods, as well as other useful evaluation approaches, are discussed.Peer reviewe

Performance and evaluation of the Viking lander camera performance prediction program

A computer program is described for predicting the performance of the Viking lander cameras. The predictions are primarily concerned with two objectives: (1) the picture quality of a reference test chart (of which there are three on each lander) to aid in diagnosing camera performance; and (2) the picture quality of cones with surface properties of a natural terrain to aid in predicting favorable illumination and viewing geometries and operational camera commands. Predictions made with this program are verified by experimental data obtained with a Viking-like laboratory facsimile camera

Sequential Complexity as a Descriptor for Musical Similarity

We propose string compressibility as a descriptor of temporal structure in audio, for the purpose of determining musical similarity. Our descriptors are based on computing track-wise compression rates of quantised audio features, using multiple temporal resolutions and quantisation granularities. To verify that our descriptors capture musically relevant information, we incorporate our descriptors into similarity rating prediction and song year prediction tasks. We base our evaluation on a dataset of 15500 track excerpts of Western popular music, for which we obtain 7800 web-sourced pairwise similarity ratings. To assess the agreement among similarity ratings, we perform an evaluation under controlled conditions, obtaining a rank correlation of 0.33 between intersected sets of ratings. Combined with bag-of-features descriptors, we obtain performance gains of 31.1% and 10.9% for similarity rating prediction and song year prediction. For both tasks, analysis of selected descriptors reveals that representing features at multiple time scales benefits prediction accuracy.Comment: 13 pages, 9 figures, 8 tables. Accepted versio

Evaluation of ceramics for stator application: Gas turbine engine report

Current ceramic materials, component fabrication processes, and reliability prediction capability for ceramic stators in an automotive gas turbine engine environment are assessed. Simulated engine duty cycle testing of stators conducted at temperatures up to 1093 C is discussed. Materials evaluated are SiC and Si3N4 fabricated from two near-net-shape processes: slip casting and injection molding. Stators for durability cycle evaluation and test specimens for material property characterization, and reliability prediction model prepared to predict stator performance in the simulated engine environment are considered. The status and description of the work performed for the reliability prediction modeling, stator fabrication, material property characterization, and ceramic stator evaluation efforts are reported