Detecting change via competence model

In real world applications, interested concepts are more likely to change rather than remain stable, which is known as concept drift. This situation causes problems on predictions for many learning algorithms including case-base reasoning (CBR). When learning under concept drift, a critical issue is to identify and determine "when" and "how" the concept changes. In this paper, we developed a competence-based empirical distance between case chunks and then proposed a change detection method based on it. As a main contribution of our work, the change detection method provides an approach to measure the distribution change of cases of an infinite domain through finite samples and requires no prior knowledge about the case distribution, which makes it more practical in real world applications. Also, different from many other change detection methods, we not only detect the change of concepts but also quantify and describe this change. © 2010 Springer-Verlag

Generating estimates of classification confidence for a case-based spam filter

Producing estimates of classification confidence is surprisingly difficult. One might expect that classifiers that can produce numeric classification scores (e.g. k-Nearest Neighbour, Na¨ıve Bayes or Support Vector Machines) could readily produce confidence estimates based on thresholds. In fact, this proves not to be the case, probably because these are not probabilistic classifiers in the strict sense. The numeric scores coming from k-Nearest Neighbour, Na¨ıve Bayes and Support Vector Machine classifiers are not well correlated with classification confidence. In this paper we describe a case-based spam filtering application that would benefit significantly from an ability to attach confidence predictions to positive classifications (i.e. messages classified as spam). We show that ‘obvious’ confidence metrics for a case-based classifier are not effective. We propose an ensemble-like solution that aggregates a collection of confidence metrics and show that this offers an effective solution in this spam filtering domain

Tidal rates of settlement of the intertidal barnacles and in western Europe: the influence of the night/day cycle

Chthamalus montagui and Chthamalus stellatus are abundant barnacles in western Europe. Tidal settlement of Chthamalus in SW Ireland and SW Portugal was studied in relation to a night and day factor and at different temporal (dates) and spatial (shores and sites) scales. Based on the identifiable cyprids and metamorphs, Chthamalus settlement in SW Ireland was comprised mainly of C. stellatus but was composed of C. montagui only in SW Portugal.In SW Ireland and SW Portugal, settlement rates of Chthamalus (mean number of settlers per 25 cm2±S.E.) were higher after one day tidal cycle (597±158.7 in SW Ireland, 144±23.6 in SW Portugal) than one night tidal cycle (55±12.1 in SW Ireland, 13±2.2 in SW Portugal), but significant differences were only detected in SW Portugal. Different models were proposed for explaining this pattern related to night and day variability of the physical processes responsible for transporting cyprids to shore (1), and/or of pre-settlement behaviour (2) and/or of settlement behaviour of cyprids (3).Spatial patterns of tidal settlement of both species or at both locations seem similar with small scale variability (between sites, 5 to 30 m apart) in settlement of Chthamalus being the only spatial scale at which variability was detected in both locations.The composition of Chthamalus cohorts settling during one tidal cycle differed considerably between locations/species: settlers of C. stellatus were mainly cyprids; settlers of C. montagui during the day (when most settlement occurred) were essentially metamorphs

CBTV: Visualising Case Bases for Similarity Measure Design and Selection

In CBR the design and selection of similarity measures is paramount. Selection can benefit from the use of exploratory visualisation- based techniques in parallel with techniques such as cross-validation ac- curacy comparison. In this paper we present the Case Base Topology Viewer (CBTV) which allows the application of different similarity mea- sures to a case base to be visualised so that system designers can explore the case base and the associated decision boundary space. We show, using a range of datasets and similarity measure types, how the idiosyncrasies of particular similarity measures can be illustrated and compared in CBTV allowing CBR system designers to make more informed choices

Towards Transparent Systems: Semantic Characterization of Failure Modes

Abstract. Today’s computer vision systems are not perfect. They fail frequently. Even worse, they fail abruptly and seemingly inexplicably. We argue that making our systems more transparent via an explicit human understandable characterization of their failure modes is desirable. We propose characterizing the failure modes of a vision system using semantic attributes. For example, a face recognition system may say “If the test image is blurry, or the face is not frontal, or the person to be recognized is a young white woman with heavy make up, I am likely to fail. ” This information can be used at training time by researchers to design better features, models or collect more focused training data. It can also be used by a downstream machine or human user at test time to know when to ignore the output of the system, in turn making it more reliable. To generate such a “specification sheet”, we discriminatively cluster incorrectly classified images in the semantic attribute space using L1-regularized weighted logistic regression. We show that our specification sheets can predict oncoming failures for face and animal species recognition better than several strong baselines. We also show that lay people can easily follow our specification sheets.