A review onquantification learning

The task of quantification consists in providing an aggregate estimation (e.g. the class distribution in a classification problem) for unseen test sets, applying a model that is trained using a training set with a different data distribution. Several real-world applications demand this kind of methods that do not require predictions for individual examples and just focus on obtaining accurate estimates at an aggregate level. During the past few years, several quantification methods have been proposed from different perspectives and with different goals. This paper presents a unified review of the main approaches with the aim of serving as an introductory tutorial for newcomers in the fiel

Continuous Sweep: an improved, binary quantifier

Quantification is a supervised machine learning task, focused on estimating the class prevalence of a dataset rather than labeling its individual observations. We introduce Continuous Sweep, a new parametric binary quantifier inspired by the well-performing Median Sweep. Median Sweep is currently one of the best binary quantifiers, but we have changed this quantifier on three points, namely 1) using parametric class distributions instead of empirical distributions, 2) optimizing decision boundaries instead of applying discrete decision rules, and 3) calculating the mean instead of the median. We derive analytic expressions for the bias and variance of Continuous Sweep under general model assumptions. This is one of the first theoretical contributions in the field of quantification learning. Moreover, these derivations enable us to find the optimal decision boundaries. Finally, our simulation study shows that Continuous Sweep outperforms Median Sweep in a wide range of situations

Dataset Shift and the Adjustment of Probabilistic Classifiers

Klassifitseerimine on masinõppe ülesanne, kus igale andmepunktiletuleb tema tunnuste põhjal määrata klass. Tõenäosuslik klassifitseerimine onkitsam ülesanne, kus kõikidele võimalikele klassidele tuleb määrata iga andmepunkti puhul tõenäosus, mis näitaks klassifitseerija enesekindlust andmepunktile antud klassi määramisel.Klassikalises masinõppes eeldatakse, et kõik andmepunktid, mida kasutatakseklassifitseerija treenimiseks või testimiseks on valitud sõltumatult ja samasttunnuste ja märgendite ühisjaotusest. See on aga päriselulistes rakendustes väga ebatõenäoline, kuna sageli andmete jaotus muutub aja jooksul. Muutust andmete jaotuses klassifitseerija treenimise ja hilisema rakendamise vahel tuntakse kui andmenihet.Antud töös pakutakse välja uus meetod mistahes selliste tõenäosuslike klassifitseerijate töö parandamiseks, mille puhul on andmetes klassijaotust muutev nihe - omadus, mis on enamikel andmenihetel. Välja pakutud meetod baseerub kohandamise protsessil, mille käigus sobitatakse tõenäosusliku klassifitseerija oodatav väljund andmete klassijaotusega. Varasemas töös on näidatud, et kohandamine vähendab oodatavat kahju keskmise ruutvea ja KL-divergentsi puhul.Need kaks kaofunktsiooni on osa laiemast funktsioonide perest, mida kutsutakse puhasteks skoorireegliteks.Välja pakutud protseduuri kutsume edaspidi üldiseks kohandamiseks, kuna seevähendab oodatavat kahju kõikide puhaste skoorireeglite korral. Üldisel kohandamisel on kaks variatsiooni: piiramata ja piiratud. Piiramata üldine kohandamine annab keskmise ruutvea ja KL-divergentsi korral sama tulemuse nagu juba eksisteerivad kohandamise protseduurid. Piiratud üldine kohandamine on täiendus, mis vähendab oodatavat kahju vähemalt sama palju või rohkem kui piiramata versioon. Mõlemad meetodid lahenduvad kui kumerad minimiseerimisülesanded ning on seega arvutuslikult efektiivsed.Eksperimentide tulemused näitavad, et piiratud üldine kohandamine vähendabkahju praktilistes olukordades, kus uue andmejaotuse klassijaotus ei pruugi ollatäpselt teada. Isegi mõõduka veaga hinnatud klassijaotuse korral suudab piiratud üldine kohandamine enamikel juhtudel kahju vähendada.Classification is the machine learning problem of assigning a class toa given instance of data defined by a set of features. Probabilistic classificationis the stricter problem of assigning probabilities to each possible class given aninstance, indicating the classifiers confidence in that class being correct for thegiven instance.The underlying assumption of classical machine learning is that any instance used to train or test the classifier is sampled independently and identically distributed from the same joint probability distribution of features and labels. This, however, is a very unlikely situation in real world applications, as the distribution of data frequently changes over time. The change in the distribution of data between the time of training the classifier and a future point in the classifier’s life cycle (testing, deployment, etc.) is known as dataset shift.In this thesis, a novel procedure is presented which improves the performanceof a probabilistic classifier experiencing any pattern of shift that causes the class distribution to change, a property most patterns of shift share. This new technique is based off of adjustment, the process of matching the probabilistic classifier’s expected output to the class distribution of the data. In previous works it has been shown that adjustment can be used to reduce expected loss for mean squared error and KL divergence. These two loss functions are a part of a wider family of loss functions called proper scoring rules.The proposed novel procedure is termed general adjustment, since it reducesexpected loss for all proper scoring rules. It comes in two varieties, unboundedand bounded. Unbounded general adjustment gives results equivalent to the previously described adjustment procedures for mean squared error and KL divergence.Bounded general adjustment is a further refinement, reducing expected loss asmuch or more than its unbounded form. Both are convex minimization tasks, andtherefore computationally efficient to compute.The results of a series of experiments show that bounded general adjustmentreduces loss in a practical setting, where the exact value of the new class distribution may not be known. Even with moderate error in the estimation of the new class distribution, bounded general adjustment still reduces loss in most cases