41 research outputs found
C2AE: Class Conditioned Auto-Encoder for Open-set Recognition
Models trained for classification often assume that all testing classes are
known while training. As a result, when presented with an unknown class during
testing, such closed-set assumption forces the model to classify it as one of
the known classes. However, in a real world scenario, classification models are
likely to encounter such examples. Hence, identifying those examples as unknown
becomes critical to model performance. A potential solution to overcome this
problem lies in a class of learning problems known as open-set recognition. It
refers to the problem of identifying the unknown classes during testing, while
maintaining performance on the known classes. In this paper, we propose an
open-set recognition algorithm using class conditioned auto-encoders with novel
training and testing methodology. In contrast to previous methods, training
procedure is divided in two sub-tasks, 1. closed-set classification and, 2.
open-set identification (i.e. identifying a class as known or unknown). Encoder
learns the first task following the closed-set classification training
pipeline, whereas decoder learns the second task by reconstructing conditioned
on class identity. Furthermore, we model reconstruction errors using the
Extreme Value Theory of statistical modeling to find the threshold for
identifying known/unknown class samples. Experiments performed on multiple
image classification datasets show proposed method performs significantly
better than state of the art.Comment: CVPR2019 (Oral
An Improved Baseline for Sentence-level Relation Extraction
Sentence-level relation extraction (RE) aims at identifying the relationship
between two entities in a sentence. Many efforts have been devoted to this
problem, while the best performing methods are still far from perfect. In this
paper, we revisit two problems that affect the performance of existing RE
models, namely entity representation and noisy or ill-defined labels. Our
improved baseline model, incorporated with entity representations with typed
markers, achieves an F1 of 74.6% on TACRED, significantly outperforms previous
SOTA methods. Furthermore, the presented new baseline achieves an F1 of 91.1%
on the refined Re-TACRED dataset, demonstrating that the pre-trained language
models achieve unexpectedly high performance on this task. We release our code
to the community for future research.Comment: Code available at https://github.com/wzhouad/RE_improved_baselin
Learning Large Margin Sparse Embeddings for Open Set Medical Diagnosis
Fueled by deep learning, computer-aided diagnosis achieves huge advances.
However, out of controlled lab environments, algorithms could face multiple
challenges. Open set recognition (OSR), as an important one, states that
categories unseen in training could appear in testing. In medical fields, it
could derive from incompletely collected training datasets and the constantly
emerging new or rare diseases. OSR requires an algorithm to not only correctly
classify known classes, but also recognize unknown classes and forward them to
experts for further diagnosis. To tackle OSR, we assume that known classes
could densely occupy small parts of the embedding space and the remaining
sparse regions could be recognized as unknowns. Following it, we propose Open
Margin Cosine Loss (OMCL) unifying two mechanisms. The former, called Margin
Loss with Adaptive Scale (MLAS), introduces angular margin for reinforcing
intra-class compactness and inter-class separability, together with an adaptive
scaling factor to strengthen the generalization capacity. The latter, called
Open-Space Suppression (OSS), opens the classifier by recognizing sparse
embedding space as unknowns using proposed feature space descriptors. Besides,
since medical OSR is still a nascent field, two publicly available benchmark
datasets are proposed for comparison. Extensive ablation studies and feature
visualization demonstrate the effectiveness of each design. Compared with
state-of-the-art methods, MLAS achieves superior performances, measured by ACC,
AUROC, and OSCR
Controlling Risk of Web Question Answering
Web question answering (QA) has become an indispensable component in modern
search systems, which can significantly improve users' search experience by
providing a direct answer to users' information need. This could be achieved by
applying machine reading comprehension (MRC) models over the retrieved passages
to extract answers with respect to the search query. With the development of
deep learning techniques, state-of-the-art MRC performances have been achieved
by recent deep methods. However, existing studies on MRC seldom address the
predictive uncertainty issue, i.e., how likely the prediction of an MRC model
is wrong, leading to uncontrollable risks in real-world Web QA applications. In
this work, we first conduct an in-depth investigation over the risk of Web QA.
We then introduce a novel risk control framework, which consists of a qualify
model for uncertainty estimation using the probe idea, and a decision model for
selectively output. For evaluation, we introduce risk-related metrics, rather
than the traditional EM and F1 in MRC, for the evaluation of risk-aware Web QA.
The empirical results over both the real-world Web QA dataset and the academic
MRC benchmark collection demonstrate the effectiveness of our approach.Comment: 42nd International ACM SIGIR Conference on Research and Development
in Information Retrieva