Estimating Confidences for Classifier Decisions using Extreme Value Theory

Classifiers generally lack a mechanism to compute decision confidences. As humans, when we sense that the confidence for a decision is low, we either conduct additional actions to improve our confidence or dismiss the decision. While this reasoning is natural to us, it is currently missing in most common decision algorithms (i.e., classifiers) used in computer vision or machine learning. This limits the capability for a machine to take further actions to either improve a result or dismiss the decision. In this thesis, we design algorithms for estimating the confidence for decisions made by classifiers such as nearest-neighbor or support vector machines. We developed these algorithms leveraging the theory of extreme values. We use the statistical models that this theory provides for modeling the classifier's decision scores for correct and incorrect outcomes. Our proposed algorithms exploit these statistical models in order to compute a correctness belief: the probability that the classifier's decision is correct. In this work, we show how these beliefs can be used to filter bad classifications and to speed up robust estimations via sample and consensus algorithms, which are used in computer vision for estimating camera motions and for reconstructing the scene's 3D structure. Moreover, we show how these beliefs improve the classification accuracy of one-class support vector machines. In conclusion, we show that extreme value theory leads to powerful mechanisms that can predict the correctness of a classifier's decision

A generic framework for context-dependent fusion with application to landmine detection.

For complex detection and classification problems, involving data with large intra-class variations and noisy inputs, no single source of information can provide a satisfactory solution. As a result, combination of multiple classifiers is playing an increasing role in solving these complex pattern recognition problems, and has proven to be a viable alternative to using a single classifier. Over the past few years, a variety of schemes have been proposed for combining multiple classifiers. Most of these were global as they assign a degree of worthiness to each classifier, that is averaged over the entire training data. This may not be the optimal way to combine the different experts since the behavior of each one may not be uniform over the different regions of the feature space. To overcome this issue, few local methods have been proposed in the last few years. Local fusion methods aim to adapt the classifiers\u27 worthiness to different regions of the feature space. First, they partition the input samples. Then, they identify the best classifier for each partition and designate it as the expert for that partition. Unfortunately, current local methods are either computationally expensive and/or perform these two tasks independently of each other. However, feature space partition and algorithm selection are not independent and their optimization should be simultaneous. In this dissertation, we introduce a new local fusion approach, called Context Extraction for Local Fusion (CELF). CELF was designed to adapt the fusion to different regions of the feature space. It takes advantage of the strength of the different experts and overcome their limitations. First, we describe the baseline CELF algorithm. We formulate a novel objective function that combines context identification and multi-algorithm fusion criteria into a joint objective function. The context identification component thrives to partition the input feature space into different clusters (called contexts), while the fusion component thrives to learn the optimal fusion parameters within each cluster. Second, we propose several variations of CELF to deal with different applications scenario. In particular, we propose an extension that includes a feature discrimination component (CELF-FD). This version is advantageous when dealing with high dimensional feature spaces and/or when the number of features extracted by the individual algorithms varies significantly. CELF-CA is another extension of CELF that adds a regularization term to the objective function to introduce competition among the clusters and to find the optimal number of clusters in an unsupervised way. CELF-CA starts by partitioning the data into a large number of small clusters. As the algorithm progresses, adjacent clusters compete for data points, and clusters that lose the competition gradually become depleted and vanish. Third, we propose CELF-M that generalizes CELF to support multiple classes data sets. The baseline CELF and its extensions were formulated to use linear aggregation to combine the output of the different algorithms within each context. For some applications, this can be too restrictive and non-linear fusion may be needed. To address this potential drawback, we propose two other variations of CELF that use non-linear aggregation. The first one is based on Neural Networks (CELF-NN) and the second one is based on Fuzzy Integrals (CELF-FI). The latter one has the desirable property of assigning weights to subsets of classifiers to take into account the interaction between them. To test a new signature using CELF (or its variants), each algorithm would extract its set of features and assigns a confidence value. Then, the features are used to identify the best context, and the fusion parameters of this context are used to fuse the individual confidence values. For each variation of CELF, we formulate an objective function, derive the necessary conditions to optimize it, and construct an iterative algorithm. Then we use examples to illustrate the behavior of the algorithm, compare it to global fusion, and highlight its advantages. We apply our proposed fusion methods to the problem of landmine detection. We use data collected using Ground Penetration Radar (GPR) and Wideband Electro -Magnetic Induction (WEMI) sensors. We show that CELF (and its variants) can identify meaningful and coherent contexts (e.g. mines of same type, mines buried at the same site, etc.) and that different expert algorithms can be identified for the different contexts. In addition to the land mine detection application, we apply our approaches to semantic video indexing, image database categorization, and phoneme recognition. In all applications, we compare the performance of CELF with standard fusion methods, and show that our approach outperforms all these methods

Information-Theoretic Perspectives on Unconscious Priming and Group Decisions: Retrieving Maximum Information From Human Responses

Seitdem die Informationstheorie in die psychologische Forschungsliteratur wäh-rend der Mitte des letzten Jahrhunderts eingeführt worden ist, hat das Thema Informationsverarbeitung im menschlichen Gehirn mehr und mehr an Aufmerksamkeit gewonnen. Von besonderem Interesse war die Frage, zu welchem Ausmaß Menschen Informationen unterbewusst verarbeiten können. Um unterbewusste Informationsverarbeitung nachzuweisen, war in den letzten zwei Jahrzehnten eines der prominentesten Paradigmen das unterbewusste Priming. Studien in diesem Paradigma folgen häufig einem \emph{Standardverfahren}: Diese Studien zeigen, dass Versuchsteilnehmer nahe am Rateniveau sind, wenn sie kaum sichtbare Stimuli identifizieren sollen. Gleichzeitig produzieren dieselben Stimuli eindeutige Effekte in indirekten Messungen wie zum Beispiel in Reaktionszeiten oder in Gehirnaktivierung. Von diesem Ergebnismuster wird im Standardverfahren geschlussfolgert, dass die Versuchsteilnehmer mehr Information über die Stimuli verarbeitet haben, als ihnen bewusst ist. Wir zeigen hier, dass das Standardverfahren fehlerhaft ist. Die Effekte auf die indirekten Messungen können meist vollständig durch residuale, bewusste Verarbeitung erklärt werden. Diese schwache, bewusste Verarbeitung zeigt sich in der Identifikationsleistung der Studienteilnehmer, welche zwar nahe am Zufallsniveau aber doch nicht exakt auf diesem liegt. Der irreführende Eindruck einer überlegenen, unterbewussten Verarbeitung entsteht durch einen methodisch unangemessenen Vergleich. Dabei erscheinen die direkt gegebenen Antworten, als basierten sie auf weniger Information über die Stimuli als die indirekten Messungen. Wir entwickeln hier eine Reanalysemethode für Ergebnisse aus früheren Studien und zeigen, dass große Teile der vielzitierten Forschungsliteratur zu unterbewusstem Priming wenig bis keine Beweise für die weitreichenden Interpretationen über unterbewusste Verarbeitung liefern. Im Forschungsfeld Gruppenentscheidungen gibt es einen analogen Fehler. In solchen Studien werden echte Gruppenentscheidungsprozesse simuliert, indem Aussagen individueller Gruppenmitglieder statistisch zusammengeführt werden. Diese statistischen Zusammenführungen dienen als \emph{simulierte} Gruppenentscheidungen, die dann mit den \emph{echten} Gruppenentscheidungen aus interaktiven Gruppendiskussionen verglichen werden. Ein Ergebnis solcher Studien ist, dass echte Gruppen häufiger korrekte Entscheidungen treffen als simulierte Gruppen. Aber die meisten Studien nutzen nicht die theoretisch optimale Methode, Mehrheitsbeschluss mit Stimmgewichtung (Confidence Weighted Majority Voting, CWMV). Ähnlich zum Problem beim unterbewussten Priming führen suboptimale Methoden bei der Simulation von Gruppenentscheidungen potenziell zu ungerechtfertigten Interpretationen beim Vergleich von echten mit simulierten Gruppen. Unterschiede können allein durch methodische Disparitäten auftreten und dürfen nicht ohne weiteres auf einen zugrundeliegenden, wahren Unterschied zwischen echten und simulierten Gruppen zurückgeführt werden. Wir stellen die theoretisch optimale Methode CWMV in den Blickpunkt und zeigen in einem Experiment, dass diese Methode gleiche Vorhersagegenauigkeit wie echte Gruppenentscheidungen erreicht. Das macht CWMV zu einem geeigneten Kandidaten, um echte Gruppenprozesse zu modellieren. Obwohl die Vorhersagegenauigkeit übereinstimmt, unterscheiden sich echte Gruppen systematisch von den Simulationen mit CWMV. Wir modellieren diese Abweichungen und zeigen, dass echte Gruppen die Aussagen ihrer Mitglieder gleichmäßiger gewichten und insgesamt weniger Sicherheitsbewertung in die Gruppenentscheidung legen als Simulationen mittels CWMV. Beide Forschungsbereiche -- unterbewusstes Priming und Gruppenentscheidungsprozesse -- vereint, dass die volle Information in den Antworten der Versuchsteilnehmer oft nicht vollständig berücksichtigt wird. Unsere Ergebnisse im Bereich der Gruppenentscheidungen werfen die zu-sätzliche Frage auf, ob die Vorhersagegenauigkeit der einzelnen Gruppenmitglieder die \emph{Vorhersagegenauigkeit der Gruppe} bestimmt. Diese Frage ist insbesondere im Bereich des maschinellen Lernens relevant, bei der nicht Menschen eine Gruppe bilden sondern einzelne Klassifikationsalgorithmen ein sogenanntes Ensemble. Wir erarbeiten hier ein Modell, in dem wir ein Negativergebnis nachweisen: Die Vorhersagegenauigkeit des Ensembles kann Werte in einer überraschend breiten Spanne annehmen, selbst wenn die Vorhersagegenauigkeit der einzelnen Klassifikationsalgorithmen konstant gehalten wird. Der Grund liegt in dem drastisch unterschiedlichen Informationsgehalt, den ein Klassifikationsalgorithmus trotz gleich gehaltener Genauigkeit übertragen kann. Wir beweisen, welche Vorhersagegenauigkeit eine Ensemble im besten und schlechtesten Fall bei gegebener Genauigkeit der einzelnen Algorithmen annehmen kann. Zusätzlich beweisen wir engere Schranken für den Fall, dass nicht nur die Klassifikationsgenauigkeit sondern auch der Informationsgehalt der einzelnen Algorithmen gegeben ist. Aus unseren konstruktiven Beweisen gehen Prinzipien für die Auswahl und Implementation von Klassifikationsalgorithmen für die Verwendung in Ensembles hervor. Diese Prinzipien gehen über die einfache Heuristik hinaus, dass Klassifikationsalgorithmen mit höherem Informationsgehalt gewählt werden sollten. Diese drei Forschungsgegenstände unterstreichen die Relevanz von Aspekten menschlicher und maschineller Vorhersagen, welche jenseits der Vorhersagegenauigkeit liegen. Diese Aspekte sind auf den ersten Blick leicht übersehen, da viele psychologische Forschungsbereiche sich auf das herkömmliche Maß der Klassifikationsgenaugkeit beschränken. Sie spielen nichtsdestotrotz eine wichtige thereotische und praktische Rolle, wie wir in den drei Bereichen zeigen.Since Information Theory was introduced to the field of psychology in the middle of the last century, Information processing in the human brain has gained attention. A question of particular interest has been: To what degree can humans process information unconsciously? For the past two decades, one of the most prominent paradigms in which this question has been investigated was \emph{unconscious priming}. Studies in this paradigm have frequently used a \emph{standard reasoning}: These studies show that participants perform close to random guessing when they have to identify barely visible stimuli but the same stimuli nevertheless produce clear effects in indirect measures such as reaction times or neuroimaging measures. From this pattern of results, the standard reasoning concludes that participants processed information about the stimuli beyond what they are consciously aware of. But we show here that the standard reasoning is flawed. The clear effects in indirect measures can often be fully explained by residual conscious processing that is reflected in participants' close to (but not exactly equal to) chance level guessing in consciously given responses. The erroneous appearance of more unconscious processing is due to an inappropriate comparison making conscious responses appear as if they were based on less information than the indirect measures. We develop a reanalysis method for results from these studies and demonstrate that a large body of heavily cited literature in the paradigm has little to no evidence for their strong claims on unconscious processing. In the field of group decision making, a similar methodological problem occurs. Here, researchers aim to model real group discussions via statistical aggregations of individual group members' responses. The statistically aggregated responses serve as \emph{simulated} group decisions that are then compared to the \emph{real} group decisions coming from an interactive group discussion. A common result is that real group decisions are more accurate than simulated group decisions. But most studies do not use the theoretically optimal method of Confidence Weighted Majority Voting (CWMV) to simulate group decisions. Similar to unconscious priming, suboptimal methods for simulations lead to inappropriate comparisons between simulated vs. real decisions. This in turn may lead to unwarranted interpretations due to methodological bias. We bring forward the theoretically optimal method and demonstrate in an experiment that simulated and real group decisions are equally accurate with this method. Despite matching in accuracy, real groups systematically deviate from CWMV simulations. We capture these deviations in a formal cognitive model showing that real groups treat each group member's vote more equally than CWMV predicts. Moreover, real groups exhibit an overall lower confidence than CWMV simulations. What ties group decision making and unconscious priming together is that the full information from human participants' responses was not fully taken into account when making comparisons. Our results raise the additional question of whether, based on the accuracies of the individuals, we can a priori determine the \emph{accuracy of the group}. This is particularly interesting in machine learning where not individual humans form a group but individual classifiers form an ensemble. We introduce a model in which we demonstrate a negative result: The ensemble accuracy can take values in a surprisingly large range even when the individual classifiers' accuracies are held constant. This is because individual classifiers with a fixed accuracy can still convey drastically varying amounts of information. We prove best- and worst-case ensemble accuracies for when the individual classifiers' accuracies are known. Additionally, we provide tighter bounds for cases in which not only accuracies but also individual classifiers' transmitted information is known. Our constructive proofs yield guiding principles for selecting and constructing classifiers for ensembles. These principles go beyond the simple notion of preferring classifiers with highest mutual information. These three strands of research highlight the relevance of certain aspects from responses given by humans or classifiers that go beyond classification accuracy. Such aspects are prima facie easily overlooked in many scenarios. But they still affect mutual information measures and can have theoretical and practical impact as we demonstrate in unconscious priming, decision making, and ensemble accuracy

Detection and classification of non-stationary signals using sparse representations in adaptive dictionaries

Automatic classification of non-stationary radio frequency (RF) signals is of particular interest in persistent surveillance and remote sensing applications. Such signals are often acquired in noisy, cluttered environments, and may be characterized by complex or unknown analytical models, making feature extraction and classification difficult. This thesis proposes an adaptive classification approach for poorly characterized targets and backgrounds based on sparse representations in non-analytical dictionaries learned from data. Conventional analytical orthogonal dictionaries, e.g., Short Time Fourier and Wavelet Transforms, can be suboptimal for classification of non-stationary signals, as they provide a rigid tiling of the time-frequency space, and are not specifically designed for a particular signal class. They generally do not lead to sparse decompositions (i.e., with very few non-zero coefficients), and use in classification requires separate feature selection algorithms. Pursuit-type decompositions in analytical overcomplete (non-orthogonal) dictionaries yield sparse representations, by design, and work well for signals that are similar to the dictionary elements. The pursuit search, however, has a high computational cost, and the method can perform poorly in the presence of realistic noise and clutter. One such overcomplete analytical dictionary method is also analyzed in this thesis for comparative purposes. The main thrust of the thesis is learning discriminative RF dictionaries directly from data, without relying on analytical constraints or additional knowledge about the signal characteristics. A pursuit search is used over the learned dictionaries to generate sparse classification features in order to identify time windows that contain a target pulse. Two state-of-the-art dictionary learning methods are compared, the K-SVD algorithm and Hebbian learning, in terms of their classification performance as a function of dictionary training parameters. Additionally, a novel hybrid dictionary algorithm is introduced, demonstrating better performance and higher robustness to noise. The issue of dictionary dimensionality is explored and this thesis demonstrates that undercomplete learned dictionaries are suitable for non-stationary RF classification. Results on simulated data sets with varying background clutter and noise levels are presented. Lastly, unsupervised classification with undercomplete learned dictionaries is also demonstrated in satellite imagery analysis

When in doubt ask the crowd : leveraging collective intelligence for improving event detection and machine learning

[no abstract

Interpretable methods in cancer diagnostics

Cancer is a hard problem. It is hard for the patients, for the doctors and nurses, and for the researchers working on understanding the disease and finding better treatments for it. The challenges faced by a pathologist diagnosing the disease for a patient is not necessarily the same as the ones faced by cell biologists working on experimental treatments and understanding the fundamentals of cancer. In this thesis we work on different challenges faced by both of the above teams. This thesis first presents methods to improve the analysis of the flow cy- tometry data used frequently in the diagnosis process, specifically for the two subtypes of non-Hodgkin Lymphoma which are our focus: Follicular Lymphoma and Diffuse Large B Cell Lymphoma. With a combination of concepts from graph theory, dynamic programming, and machine learning, we present methods to improve the diagnosis process and the analysis of the abovementioned data. The interpretability of the method helps a pathologist to better understand a patient’s disease, which itself improves their choices for a treatment. In the second part, we focus on the analysis of DNA-methylation and gene expression data, both of which presenting the challenge of being very high dimen- sional yet with a few number of samples comparatively. We present an ensemble model which adapts to different patterns seen in each given data, in order to adapt to noise and batch effects. At the same time, the interpretability of our model helps a pathologist to better find and tune the treatment for the patient: a step further towards personalized medicine.Krebs ist ein schweres Problem. Es ist schwer für die Patienten, für die Ärzte und Krankenschwestern und für die Forscher, die daran arbeiten, die Krankheit zu verstehen und eine bessere Behandlung dafür zu finden. Die Herausforderungen, mit denen ein Pathologe konfrontiert ist, um die Krankheit eines Patienten zu diagnostizieren, müssen nicht die gleichen sein, mit denen Zellbiologen konfrontiert sind, die an experimentellen Behandlungen arbeiten und die Grundlagen von Krebs verstehen. In dieser Arbeit beschäftigen wir uns mit verschiedenen Herausforderungen, denen sich beide oben genannten Teams stellen. In dieser Arbeit werden zunächst Methoden vorgestellt, um die Analyse der im Diagnoseverfahren häufig verwendeten Durchflusszytometriedaten zu verbessern, insbesondere für die beiden Subtypen des Non-Hodgkin-Lymphoms, auf die wir uns konzentrieren: das follikuläre Lymphom und das diffuse großzellige B-Zell-Lymphom. Mit einer Kombination von Konzepten aus Graphentheorie, dynamischer Programmierung und künstliche Intelligenz präsentieren wir Methoden zur Verbesserung des Diagnoseprozesses und der Analyse der oben genannten Daten. Die Interpretierbarkeit der Methode hilft einem Pathologen, die Apatientenkrankheit besser zu verstehen, was wiederum seine Wahlmöglichkeiten für eine Behandlung verbessert. Im zweiten Teil konzentrieren wir uns auf die Analyse von DNA-Methylierungsund Genexpressionsdaten, die beide die Herausforderung darstellen, sehr hochdimensional zu sein, jedoch mit nur wenigen Proben im Vergleich.Wir präsentieren ein Zusammenstellungsmodell, das sich an unterschiedliche Muster anpasst, die in den jeweiligen Daten zu sehen sind, um sich an Rauschen und Batch-Effekte anzupassen. Gleichzeitig hilft die Interpretierbarkeit unseres Modells einem Pathologen, die Behandlung für den Patienten besser zu finden und abzustimmen: ein Schritt weiter in Richtung personalisierter Medizin

Differentiating Pressure Ulcer Risk Levels through Interpretable Classification Models Based on Readily Measurable Indicators

Pressure ulcers carry a significant risk in clinical practice. This paper proposes a practical and interpretable approach to estimate the risk levels of pressure ulcers using decision tree models. In order to address the common problem of imbalanced learning in nursing classification datasets, various oversampling configurations are analyzed to improve the data quality prior to modeling. The decision trees built are based on three easily identifiable and clinically relevant pressure ulcer risk indicators: mobility, activity, and skin moisture. Additionally, this research introduces a novel tabular visualization method to enhance the usability of the decision trees in clinical practice. Thus, the primary aim of this approach is to provide nursing professionals with valuable insights for assessing the potential risk levels of pressure ulcers, which could support their decision-making and allow, for example, the application of suitable preventive measures tailored to each patient’s requirements. The interpretability of the models proposed and their performance, evaluated through stratified cross-validation, make them a helpful tool for nursing care in estimating the pressure ulcer risk level

Colour Communication Within Different Languages

For computational methods aiming to reproduce colour names that are meaningful to speakers of different languages, the mapping between perceptual and linguistic aspects of colour is a problem of central information processing. This thesis advances the field of computational colour communication within different languages in five main directions. First, we show that web-based experimental methodologies offer considerable advantages in obtaining a large number of colour naming responses in British and American English, Greek, Russian, Thai and Turkish. We continue with the application of machine learning methods to discover criteria in linguistic, behavioural and geometric features of colour names that distinguish classes of colours. We show that primary colour terms do not form a coherent class, whilst achromatic and basic classes do. We then propose and evaluate a computational model trained by human responses in the online experiment to automate the assignment of colour names in different languages across the full three-dimensional colour gamut. Fourth, we determine for the first time the location of colour names within a physiologically-based cone excitation space through an unconstrained colour naming experiment using a calibrated monitor under controlled viewing conditions. We show a good correspondence between online and offline datasets; and confirm the validity of both experimental methodologies for estimating colour naming functions in laboratory and real-world monitor settings. Finally, we present a novel information theoretic measure, called dispensability, for colour categories that predicts a gradual scale of basicness across languages from both web- and laboratory- based unconstrained colour naming datasets. As a result, this thesis contributes experimental and computational methodologies towards the development of multilingual colour communication schemes

Adaptive visual sampling

PhDVarious visual tasks may be analysed in the context of sampling from the visual field. In visual psychophysics, human visual sampling strategies have often been shown at a high-level to be driven by various information and resource related factors such as the limited capacity of the human cognitive system, the quality of information gathered, its relevance in context and the associated efficiency of recovering it. At a lower-level, we interpret many computer vision tasks to be rooted in similar notions of contextually-relevant, dynamic sampling strategies which are geared towards the filtering of pixel samples to perform reliable object association. In the context of object tracking, the reliability of such endeavours is fundamentally rooted in the continuing relevance of object models used for such filtering, a requirement complicated by realworld conditions such as dynamic lighting that inconveniently and frequently cause their rapid obsolescence. In the context of recognition, performance can be hindered by the lack of learned context-dependent strategies that satisfactorily filter out samples that are irrelevant or blunt the potency of models used for discrimination. In this thesis we interpret the problems of visual tracking and recognition in terms of dynamic spatial and featural sampling strategies and, in this vein, present three frameworks that build on previous methods to provide a more flexible and effective approach. Firstly, we propose an adaptive spatial sampling strategy framework to maintain statistical object models for real-time robust tracking under changing lighting conditions. We employ colour features in experiments to demonstrate its effectiveness. The framework consists of five parts: (a) Gaussian mixture models for semi-parametric modelling of the colour distributions of multicolour objects; (b) a constructive algorithm that uses cross-validation for automatically determining the number of components for a Gaussian mixture given a sample set of object colours; (c) a sampling strategy for performing fast tracking using colour models; (d) a Bayesian formulation enabling models of object and the environment to be employed together in filtering samples by discrimination; and (e) a selectively-adaptive mechanism to enable colour models to cope with changing conditions and permit more robust tracking. Secondly, we extend the concept to an adaptive spatial and featural sampling strategy to deal with very difficult conditions such as small target objects in cluttered environments undergoing severe lighting fluctuations and extreme occlusions. This builds on previous work on dynamic feature selection during tracking by reducing redundancy in features selected at each stage as well as more naturally balancing short-term and long-term evidence, the latter to facilitate model rigidity under sharp, temporary changes such as occlusion whilst permitting model flexibility under slower, long-term changes such as varying lighting conditions. This framework consists of two parts: (a) Attribute-based Feature Ranking (AFR) which combines two attribute measures; discriminability and independence to other features; and (b) Multiple Selectively-adaptive Feature Models (MSFM) which involves maintaining a dynamic feature reference of target object appearance. We call this framework Adaptive Multi-feature Association (AMA). Finally, we present an adaptive spatial and featural sampling strategy that extends established Local Binary Pattern (LBP) methods and overcomes many severe limitations of the traditional approach such as limited spatial support, restricted sample sets and ad hoc joint and disjoint statistical distributions that may fail to capture important structure. Our framework enables more compact, descriptive LBP type models to be constructed which may be employed in conjunction with many existing LBP techniques to improve their performance without modification. The framework consists of two parts: (a) a new LBP-type model known as Multiscale Selected Local Binary Features (MSLBF); and (b) a novel binary feature selection algorithm called Binary Histogram Intersection Minimisation (BHIM) which is shown to be more powerful than established methods used for binary feature selection such as Conditional Mutual Information Maximisation (CMIM) and AdaBoost