1,752 research outputs found

    On the Significance of Distance in Machine Learning

    Get PDF
    Avstandsbegrepet er grunnleggende i maskinlæring. Hvordan vi velger å måle avstand har betydning, men det er ofte utfordrende å finne et passende avstandsmål. Metrisk læring kan brukes til å lære funksjoner som implementerer avstand eller avstandslignende mål. Vanlige dyplæringsmodeller er sårbare for modifikasjoner av input som har til hensikt å lure modellen (adversarial examples, motstridende eksempler). Konstruksjon av modeller som er robuste mot denne typen angrep er av stor betydning for å kunne utnytte maskinlæringsmodeller i større skala, og et passende avstandsmål kan brukes til å studere slik motstandsdyktighet. Ofte eksisterer det hierarkiske relasjoner blant klasser, og disse relasjonene kan da representeres av den hierarkiske avstanden til klasser. I klassifiseringsproblemer som må ta i betraktning disse klasserelasjonene, kan hierarkiinformert klassifisering brukes. Jeg har utviklet en metode kalt /distance-ratio/-basert (DR) metrisk læring. I motsetning til den formuleringen som normalt anvendes har DR-formuleringen to gunstige egenskaper. For det første er det skala-invariant med hensyn til rommet det projiseres til. For det andre har optimale klassekonfidensverdier på klasserepresentantene. Dersom rommet for å konstruere modifikasjoner er tilstrekklig stort, vil man med standard adversarial accuracy (SAA, standard motstridende nøyaktighet) risikere at naturlige datapunkter blir betraktet som motstridende eksempler. Dette kan være en årsak til SAA ofte går på bekostning av nøyaktighet. For å løse dette problemet har jeg utviklet en ny definisjon på motstridende nøyaktighet kalt Voronoi-epsilon adversarial accuracy (VAA, Voronoi-epsilon motstridende nøyaktighet). VAA utvider studiet av lokal robusthet til global robusthet. Klassehierarkisk informasjon er ikke tilgjengelig for alle datasett. For å håndtere denne utfordringen har jeg undersøkt om klassifikasjonsbaserte metriske læringsmodeller kan brukes til å utlede klassehierarkiet. Videre har jeg undersøkt de mulige effektene av robusthet på feature space (egenskapsrom). Jeg fant da at avstandsstrukturen til et egenskapsrom trent for robusthet har større likhet med avstandsstrukturen i rådata enn et egenskapsrom trent uten robusthet.The notion of distance is fundamental in machine learning. The choice of distance matters, but it is often challenging to find an appropriate distance. Metric learning can be used for learning distance(-like) functions. Common deep learning models are vulnerable to the adversarial modification of inputs. Devising adversarially robust models is of immense importance for the wide deployment of machine learning models, and distance can be used for the study of adversarial robustness. Often, hierarchical relationships exist among classes, and these relationships can be represented by the hierarchical distance of classes. For classification problems that must take these class relationships into account, hierarchy-informed classification can be used. I propose distance-ratio-based (DR) formulation for metric learning. In contrast to the commonly used formulation, DR formulation has two favorable properties. First, it is invariant of the scale of an embedding. Secondly, it has optimal class confidence values on class representatives. For a large perturbation budget, standard adversarial accuracy (SAA) allows natural data points to be considered as adversarial examples. This could be a reason for the tradeoff between accuracy and SAA. To resolve the issue, I proposed a new definition of adversarial accuracy named Voronoi-epsilon adversarial accuracy (VAA). VAA extends the study of local robustness to global robustness. Class hierarchical information is not available for all datasets. To handle this challenge, I investigated whether classification-based metric learning models can be used to infer class hierarchy. Furthermore, I explored the possible effects of adversarial robustness on feature space. I found that the distance structure of robustly trained feature space resembles that of input space to a greater extent than does standard trained feature space.Doktorgradsavhandlin

    Inspecting class hierarchies in classification-based metric learning models

    Full text link
    Most classification models treat all misclassifications equally. However, different classes may be related, and these hierarchical relationships must be considered in some classification problems. These problems can be addressed by using hierarchical information during training. Unfortunately, this information is not available for all datasets. Many classification-based metric learning methods use class representatives in embedding space to represent different classes. The relationships among the learned class representatives can then be used to estimate class hierarchical structures. If we have a predefined class hierarchy, the learned class representatives can be assessed to determine whether the metric learning model learned semantic distances that match our prior knowledge. In this work, we train a softmax classifier and three metric learning models with several training options on benchmark and real-world datasets. In addition to the standard classification accuracy, we evaluate the hierarchical inference performance by inspecting learned class representatives and the hierarchy-informed performance, i.e., the classification performance, and the metric learning performance by considering predefined hierarchical structures. Furthermore, we investigate how the considered measures are affected by various models and training options. When our proposed ProxyDR model is trained without using predefined hierarchical structures, the hierarchical inference performance is significantly better than that of the popular NormFace model. Additionally, our model enhances some hierarchy-informed performance measures under the same training options. We also found that convolutional neural networks (CNNs) with random weights correspond to the predefined hierarchies better than random chance.Comment: The main manuscript is 22 pages. The whole paper is 49 pages. The codes for our experiments will be available in https://github.com/hjk92g/Inspecting_Hierarchies_ML . The plankton datasets are available from the Norwegian Marine Data Center (MicroS: https://doi.org/10.21335/NMDC-2102309336 , MicroL: https://doi.org/10.21335/NMDC-573815973 , MesoZ: https://doi.org/10.21335/NMDC-1805578916
    corecore