DMLR: Data-centric Machine Learning Research -- Past, Present and Future

Drawing from discussions at the inaugural DMLR workshop at ICML 2023 and meetings prior, in this report we outline the relevance of community engagement and infrastructure development for the creation of next-generation public datasets that will advance machine learning science. We chart a path forward as a collective effort to sustain the creation and maintenance of these datasets and methods towards positive scientific, societal and business impact.Comment: This editorial report accompanies the inaugural Data-centric Machine Learning Research (DMLR) Workshop that took place at ICML 2023 https://dmlr.ai

Security of Cyber-Physical Systems: A Generalized Algorithm for Intrusion Detection and Determining Security Robustness of Cyber Physical Systems using Logical Truth Tables

The recent proliferation of embedded cyber components in modern physical systems [1] has generated a variety of new security risks which threaten not only cyberspace, but our physical environment as well. Whereas earlier security threats resided primarily in cyberspace, the increasing marriage of digital technology with mechanical systems in cyber-physical systems (CPS), suggests the need for more advanced generalized CPS security measures. To address this problem, in this paper we consider the first step toward an improved security model: detecting the security attack. Using logical truth tables, we have developed a generalized algorithm for intrusion detection in CPS for systems which can be defined over discrete set of valued states. Additionally, a robustness algorithm is given which determines the level of security of a discrete-valued CPS against varying combinations of multiple signal alterations. These algorithms, when coupled with encryption keys which disallow multiple signal alteration, provide for a generalized security methodology for both cyber-security and cyber-physical systems

Learning with confident examples: Rank pruning for robust classification with noisy labels

P N learning is the problem of binary classification when training examples may be mislabeled (flipped) uniformly with noise rate ρ1 for positive examples and ρ0 for negative examples. We propose Rank Pruning (RP) to solve PN learning and the open problem of estimating the noise rates. Unlike prior solutions, RP is efficient and general, requiring O(T) for any unrestricted choice of probabilistic classifier with T fitting time. We prove RP achieves consistent noise estimation and equivalent expected risk as learning with uncorrupted labels in ideal conditions, and derive closed-form solutions when conditions are non-ideal. RP achieves state-of-the-art noise estimation and F1, error, and AUC-PR for both MNIST and CIFAR datasets, regardless of the amount of noise. To highlight, RP with a CNN classifier can predict if an MNIST digit is a one or not with only 0:25% error, and 0:46% error across all digits, even when 50% of positive examples are mislabeled and 50% of observed positive labels are mislabeled negative examples

Detecting and preventing “multiple-account” cheating in massive open online courses

© 2016 Elsevier Ltd. All rights reserved. We describe a cheating strategy enabled by the features of massive open online courses (MOOCs) and detectable by virtue of the sophisticated data systems that MOOCs provide. The strategy, Copying Answers using Multiple Existences Online (CAMEO), involves a user who gathers solutions to assessment questions using a "harvester" account and then submits correct answers using a separate "master" account. We use a small-scale experiment to verify CAMEO and estimate a "lower bound" for its prevalence among 1.9 million course participants in 115 MOOCs from two universities. Using conservative thresholds, we estimate CAMEO prevalence at 1237 certificates, accounting for 1.3% of the certificates in the 69 MOOCs with CAMEO users. Among earners of 20 or more certificates, 25% have used the CAMEO strategy. CAMEO users are more likely to be young, male, and international than other MOOC certificate earners. We identify preventive strategies that can decrease CAMEO rates and show evidence of their effectiveness in science courses