Comprehensive Evaluation of Machine Learning Experiments: Algorithm Comparison, Algorithm Performance and Inferential Reproducibility

This doctoral thesis addresses critical methodological aspects within machine learning experimentation, focusing on enhancing the evaluation and analysis of algorithm performance. The established "train-dev-test paradigm" commonly guides machine learning practitioners, involving nested optimization processes to optimize model parameters and meta-parameters and benchmarking against test data. However, this paradigm overlooks crucial aspects, such as algorithm variability and the intricate relationship between algorithm performance and meta-parameters. This work introduces a comprehensive framework that employs statistical techniques to bridge these gaps, advancing the methodological standards in empirical machine learning research. The foundational premise of this thesis lies in differentiating between algorithms and classifiers, recognizing that an algorithm may yield multiple classifiers due to inherent stochasticity or design choices. Consequently, algorithm performance becomes inherently probabilistic and cannot be captured by a single metric. The contributions of this work are structured around three core themes: Algorithm Comparison: A fundamental aim of empirical machine learning research is algorithm comparison. To this end, the thesis proposes utilizing Linear Mixed Effects Models (LMEMs) for analyzing evaluation data. LMEMs offer distinct advantages by accommodating complex data structures beyond the typical independent and identically distributed (iid) assumption. Thus LMEMs enable a holistic analysis of algorithm instances and facilitate the construction of nuanced conditional models of expected risk, supporting algorithm comparisons based on diverse data properties. Algorithm Performance Analysis: Contemporary evaluation practices often treat algorithms and classifiers as black boxes, hindering insights into their performance and parameter dependencies. Leveraging LMEMs, specifically implementing Variance Component Analysis, the thesis introduces methods from psychometrics to quantify algorithm performance homogeneity (reliability) and assess the influence of meta-parameters on performance. The flexibility of LMEMs allows a granular analysis of this relationship and extends these techniques to analyze data annotation processes linked to algorithm performance. Inferential Reproducibility: Building upon the preceding chapters, this section showcases a unified approach to analyze machine learning experiments comprehensively. By leveraging the full range of generated model instances, the analysis provides a nuanced understanding of competing algorithms. The outcomes offer implementation guidelines for algorithmic modifications and consolidate incongruent findings across diverse datasets, contributing to a coherent empirical perspective on algorithmic effects. This work underscores the significance of addressing algorithmic variability, meta-parameter impact, and the probabilistic nature of algorithm performance. This thesis aims to enhance machine learning experiments' transparency, reproducibility, and interpretability by introducing robust statistical methodologies facilitating extensive empirical analysis. It extends beyond conventional guidelines, offering a principled approach to advance the understanding and evaluation of algorithms in the evolving landscape of machine learning and data science

Towards Inferential Reproducibility of Machine Learning Research

Reliability of machine learning evaluation -- the consistency of observed evaluation scores across replicated model training runs -- is affected by several sources of nondeterminism which can be regarded as measurement noise. Current tendencies to remove noise in order to enforce reproducibility of research results neglect inherent nondeterminism at the implementation level and disregard crucial interaction effects between algorithmic noise factors and data properties. This limits the scope of conclusions that can be drawn from such experiments. Instead of removing noise, we propose to incorporate several sources of variance, including their interaction with data properties, into an analysis of significance and reliability of machine learning evaluation, with the aim to draw inferences beyond particular instances of trained models. We show how to use linear mixed effects models (LMEMs) to analyze performance evaluation scores, and to conduct statistical inference with a generalized likelihood ratio test (GLRT). This allows us to incorporate arbitrary sources of noise like meta-parameter variations into statistical significance testing, and to assess performance differences conditional on data properties. Furthermore, a variance component analysis (VCA) enables the analysis of the contribution of noise sources to overall variance and the computation of a reliability coefficient by the ratio of substantial to total variance.Comment: Published at ICLR 2023 (see https://openreview.net/pdf?id=li4GQCQWkv

Ensembling Neural Networks for Improved Prediction and Privacy in Early Diagnosis of Sepsis

Ensembling neural networks is a long-standing technique for improving the generalization error of neural networks by combining networks with orthogonal properties via a committee decision. We show that this technique is an ideal fit for machine learning on medical data: First, ensembles are amenable to parallel and asynchronous learning, thus enabling efficient training of patient-specific component neural networks. Second, building on the idea of minimizing generalization error by selecting uncorrelated patient-specific networks, we show that one can build an ensemble of a few selected patient-specific models that outperforms a single model trained on much larger pooled datasets. Third, the non-iterative ensemble combination step is an optimal low-dimensional entry point to apply output perturbation to guarantee the privacy of the patient-specific networks. We exemplify our framework of differentially private ensembles on the task of early prediction of sepsis, using real-life intensive care unit data labeled by clinical experts.Comment: Accepted at MLHC 202

A first broad-scale molecular phylogeny of Prionoceridae (Coleoptera: Cleroidea) provides insight into taxonomy, biogeography and life history evolution

© Senckenberg Gesellschaft fur Naturforschung, 2016. This is an open access article. Authors are permitted to post a PDF of their own articles, as provided by the publisher, on their personal web pages or the web page of their institution. Any commercial use is excluded. The attached file is the published version of the article

Realization and training of an inverter-based printed neuromorphic computing system

Emerging applications in soft robotics, wearables, smart consumer products or IoT-devices benefit from soft materials, flexible substrates in conjunction with electronic functionality. Due to high production costs and conformity restrictions, rigid silicon technologies do not meet application requirements in these new domains. However, whenever signal processing becomes too comprehensive, silicon technology must be used for the high-performance computing unit. At the same time, designing everything in flexible or printed electronics using conventional digital logic is not feasible yet due to the limitations of printed technologies in terms of performance, power and integration density. We propose to rather use the strengths of neuromorphic computing architectures consisting in their homogeneous topologies, few building blocks and analog signal processing to be mapped to an inkjet-printed hardware architecture. It has remained a challenge to demonstrate non-linear elements besides weighted aggregation. We demonstrate in this work printed hardware building blocks such as inverter-based comprehensive weight representation and resistive crossbars as well as printed transistor-based activation functions. In addition, we present a learning algorithm developed to train the proposed printed NCS architecture based on specific requirements and constraints of the technology

Characterization of Coal Particles in the Soil of a Former Rail Yard and Urban Brownfield: Liberty State Park, Jersey City (NJ), USA

From the 1850\u27s until the 1960\u27s, the Central Railroad of New Jersey was among several major railways shipping anthracite and bituminous coal to the New York City area, transferring coal from railcar to barge at its extensive rail yard and port facility in Jersey City. The 490 ha Liberty State Park was developed on the site after the rail yard closed, but a ca. 100 ha brownfield zone within the park remains off limits to visitors pending future remediation. As part of an environmental forensic and industrial archeological investigation of this zone, the present study characterizes anthracite and bituminous coal particles present in abundance in the soil by scanning electron microscopy (SEM) and pyrolysis-gas chromatography-mass spectrometry (Py-GC-MS). A simple pretreatment procedure employing density separation improved the analytical results. This detailed information about the nature of contaminants at the site will help to inform the remediation effort in the public interest

X-Ray Emission from Slow Highly Charged Ar Ions Interacting with a Ge Surface

We have measured K x-ray spectra and yields from Ar17+ ions slowly approaching a single-crystal Ge surface. The yields were measured as a function of the projectile velocity component perpendicular to the surface. From the data a characteristic time of approximately 1 psec was extracted for the in-flight filling above the surface of the Ar K vacancy

Counterfeit Detection and Prevention in Additive Manufacturing Based on Unique Identification of Optical Fingerprints of Printed Structures

Printed Electronics (PE) based on additive manufacturing has a rapidly growing market. Due to large feature sizes and reduced complexity of PE applications compared to silicon counterparts, they are more prone to counterfeiting. Common solutions to detect counterfeiting insert watermarks or extract unique fingerprints based on (irreproducible) process variations of valid components. Commonly, such fingerprints have been extracted through electrical methods, similar to those of physically unclonable functions (PUFs). Hence, they introduce overhead to the production resulting in additional costs. While such costs may be negligible for application domains targeted by silicon-based technologies, they are detrimental to the ultra-low-cost PE applications. In this paper, we propose an optical unique identification, by extracting fingerprints from the optically visible variations of printed inks in the PE components. The images can be obtained from optical cameras, such as cell phones, thanks to large feature sizes of PE, by trusted parties, such as an end user wanting to verify the authenticity of a particular product. Since this approach does not require any additional circuitry, the fingerprint production cost consists of merely acquisition, processing and saving an image of the circuit components, matching the requirements of ultra-low-cost applications of PE. To further decrease the storage costs for the unique fingerprints, we utilize image downscaling resulting in a compression rate between 83– 188× , while preserving the reliability and uniqueness of the fingerprints. The proposed fingerprint extraction methodology is applied to four datasets and the results show that the optical variation printed inks is suitable to prevent counterfeiting in PE

Evaluation of the vaginal flora in pregnant women receiving opioid maintenance therapy: a matched case-control study

Dataset supporting the conclusions of this article. (XLS 2634 kb