A direct measure of discriminant and characteristic capability for classifier building and assessment

AbstractPerformance measures are used in various stages of the process aimed at solving a classification problem. Unfortunately, most of these measures are in fact biased, meaning that they strictly depend on the class ratio – i.e. on the imbalance between negative and positive samples. After pointing to the source of bias for the best known measures, novel unbiased measures are defined which are able to capture the concepts of discriminant and characteristic capability. The combined use of these measures can give important information to researchers involved in machine learning or pattern recognition tasks, in particular for classifier performance assessment and feature selection

A two-tiered 2D visual tool for assessing classifier performance

In this article, a new kind of 2D tool is proposed, namely ⟨φ δ⟩ diagrams, able to highlight most of the information deemed relevant for classifier building and assessment. In particular, accuracy, bias and break-even points are immediately evident therein. These diagrams come in two different forms: the first is aimed at representing the phenomenon under investigation in a space where the imbalance between negative and positive samples is not taken into account, the second (which is a generalization of the first) is able to visualize relevant information in a space that accounts also for the imbalance. According to a specific design choice, all properties found in the first space hold also in the second. The combined use of φ and δ can give important information to researchers involved in the activity of building intelligent systems, in particular for classifier performance assessment and feature ranking/selection

What does touch tell us about emotions in touchscreen-based gameplay?

This is the post-print version of the Article. The official published version can be accessed from the link below - Copyright @ 2012 ACM. It is posted here by permission of ACM for your personal use. Not for redistribution.Nowadays, more and more people play games on touch-screen mobile phones. This phenomenon raises a very interesting question: does touch behaviour reflect the player’s emotional state? If possible, this would not only be a valuable evaluation indicator for game designers, but also for real-time personalization of the game experience. Psychology studies on acted touch behaviour show the existence of discriminative affective profiles. In this paper, finger-stroke features during gameplay on an iPod were extracted and their discriminative power analysed. Based on touch-behaviour, machine learning algorithms were used to build systems for automatically discriminating between four emotional states (Excited, Relaxed, Frustrated, Bored), two levels of arousal and two levels of valence. The results were very interesting reaching between 69% and 77% of correct discrimination between the four emotional states. Higher results (~89%) were obtained for discriminating between two levels of arousal and two levels of valence

Analysis of term roles along taxonomy nodes by adopting discriminant and characteristic capabilities

Taxonomies are becoming essential to a growing number of application, particularly for specific domains. Taxonomies, originally built by hand, have been recently focused on their automatic generation. In particular, a main issue on automatic taxonomy building regards the choice of the most suitable features. In this paper, we propose an analy- sis on how each feature changes its role along taxonomy nodes in a text categorization scenario, in which the features are the terms in textual documents. We deem that, in a hierarchical structure, each node should intuitively be represented with proper meaningful and discriminant terms (i.e., performing a feature selection task for each node), instead of con- sidering a fixed feature space. To assess the discriminant power of a term, we adopt two novel metrics able to measure it. Our conjecture is that a term could significantly change its discriminant power (hence, its role) along the taxonomy levels. We perform experiments aimed at proving that a significant number of terms play different roles in each taxonomy node, giving emphasis to the usefulness of a distinct feature selection for each node. We assert that this analysis should support automatic taxonomy building approaches

A Novel Malware Target Recognition Architecture for Enhanced Cyberspace Situation Awareness

The rapid transition of critical business processes to computer networks potentially exposes organizations to digital theft or corruption by advanced competitors. One tool used for these tasks is malware, because it circumvents legitimate authentication mechanisms. Malware is an epidemic problem for organizations of all types. This research proposes and evaluates a novel Malware Target Recognition (MaTR) architecture for malware detection and identification of propagation methods and payloads to enhance situation awareness in tactical scenarios using non-instruction-based, static heuristic features. MaTR achieves a 99.92% detection accuracy on known malware with false positive and false negative rates of 8.73e-4 and 8.03e-4 respectively. MaTR outperforms leading static heuristic methods with a statistically significant 1% improvement in detection accuracy and 85% and 94% reductions in false positive and false negative rates respectively. Against a set of publicly unknown malware, MaTR detection accuracy is 98.56%, a 65% performance improvement over the combined effectiveness of three commercial antivirus products

XSS-FP: Browser Fingerprinting using HTML Parser Quirks

There are many scenarios in which inferring the type of a client browser is desirable, for instance to fight against session stealing. This is known as browser fingerprinting. This paper presents and evaluates a novel fingerprinting technique to determine the exact nature (browser type and version, eg Firefox 15) of a web-browser, exploiting HTML parser quirks exercised through XSS. Our experiments show that the exact version of a web browser can be determined with 71% of accuracy, and that only 6 tests are sufficient to quickly determine the exact family a web browser belongs to

Credit scoring using the clustered support vector machine

This work investigates the practice of credit scoring and introduces the use of the clustered support vector machine (CSVM) for credit scorecard development. This recently designed algorithm addresses some of the limitations noted in the literature that is associated with traditional nonlinear support vector machine (SVM) based methods for classification. Specifically, it is well known that as historical credit scoring datasets get large, these nonlinear approaches while highly accurate become computationally expensive. Accordingly, this study compares the CSVM with other nonlinear SVM based techniques and shows that the CSVM can achieve comparable levels of classification performance while remaining relatively cheap computationally

Using RF-DNA Fingerprints to Discriminate ZigBee Devices in an Operational Environment

This research was performed to expand AFIT\u27s Radio Frequency Distinct Native Attribute (RF-DNA) fingerprinting process to support IEEE 802.15.4 ZigBee communication network applications. Current ZigBee bit-level security measures include use of network keys and MAC lists which can be subverted through interception and spoofing using open-source hacking tools. This work addresses device discrimination using Physical (PHY) waveform alternatives to augment existing bit-level security mechanisms. ZigBee network vulnerability to outsider threats was assessed using Receiver Operating Characteristic (ROC) curves to characterize both Authorized Device ID Verification performance (granting network access to authorized users presenting true bit-level credentials) and Rogue Device Rejection performance (denying network access to unauthorized rogue devices presenting false bit-level credentials). Radio Frequency Distinct Native Attribute (RF-DNA) features are extracted from time-domain waveform responses of 2.4 GHz CC2420 ZigBee transceivers to enable humanlike device discrimination. The fingerprints were constructed using a hybrid pool of emissions collected under a range of conditions, including anechoic chamber and an indoor office environment where dynamic multi-path and signal degradation factors were present. The RF-DNA fingerprints were input to a Multiple Discriminant Analysis, Maximum Likelihood (MDA/ML) discrimination process and a 1 vs. many Looks most like? classification assessment made. The hybrid MDA model was also used for 1 vs. 1 Looks how much like? verification assessment. ZigBee Device Classification performance was assessed using both full and reduced dimensional fingerprint sets. Reduced dimensional subsets were selected using Dimensional Reduction Analysis (DRA) by rank ordering 1) pre-classification KS-Test p-values and 2) post-classification GRLVQI feature relevance values. Assessment of Zigbee device ID verification capability