16 research outputs found
Classification with Costly Features using Deep Reinforcement Learning
We study a classification problem where each feature can be acquired for a
cost and the goal is to optimize a trade-off between the expected
classification error and the feature cost. We revisit a former approach that
has framed the problem as a sequential decision-making problem and solved it by
Q-learning with a linear approximation, where individual actions are either
requests for feature values or terminate the episode by providing a
classification decision. On a set of eight problems, we demonstrate that by
replacing the linear approximation with neural networks the approach becomes
comparable to the state-of-the-art algorithms developed specifically for this
problem. The approach is flexible, as it can be improved with any new
reinforcement learning enhancement, it allows inclusion of pre-trained
high-performance classifier, and unlike prior art, its performance is robust
across all evaluated datasets.Comment: AAAI 201
NIPS - Not Even Wrong? A Systematic Review of Empirically Complete Demonstrations of Algorithmic Effectiveness in the Machine Learning and Artificial Intelligence Literature
Objective: To determine the completeness of argumentative steps necessary to
conclude effectiveness of an algorithm in a sample of current ML/AI supervised
learning literature.
Data Sources: Papers published in the Neural Information Processing Systems
(NeurIPS, n\'ee NIPS) journal where the official record showed a 2017 year of
publication.
Eligibility Criteria: Studies reporting a (semi-)supervised model, or
pre-processing fused with (semi-)supervised models for tabular data.
Study Appraisal: Three reviewers applied the assessment criteria to determine
argumentative completeness. The criteria were split into three groups,
including: experiments (e.g real and/or synthetic data), baselines (e.g
uninformed and/or state-of-art) and quantitative comparison (e.g. performance
quantifiers with confidence intervals and formal comparison of the algorithm
against baselines).
Results: Of the 121 eligible manuscripts (from the sample of 679 abstracts),
99\% used real-world data and 29\% used synthetic data. 91\% of manuscripts did
not report an uninformed baseline and 55\% reported a state-of-art baseline.
32\% reported confidence intervals for performance but none provided references
or exposition for how these were calculated. 3\% reported formal comparisons.
Limitations: The use of one journal as the primary information source may not
be representative of all ML/AI literature. However, the NeurIPS conference is
recognised to be amongst the top tier concerning ML/AI studies, so it is
reasonable to consider its corpus to be representative of high-quality
research.
Conclusion: Using the 2017 sample of the NeurIPS supervised learning corpus
as an indicator for the quality and trustworthiness of current ML/AI research,
it appears that complete argumentative chains in demonstrations of algorithmic
effectiveness are rare