Recent works seek to endow recognition systems with the ability to handle the
open world. Few shot learning aims for fast learning of new classes from
limited examples, while open-set recognition considers unknown negative class
from the open world. In this paper, we study the problem of few-shot open-set
recognition (FSOR), which learns a recognition system robust to queries from
new sources with few examples and from unknown open sources. To achieve that,
we mimic human capability of envisioning new concepts from prior knowledge, and
propose a novel task-adaptive negative class envision method (TANE) to model
the open world. Essentially we use an external memory to estimate a negative
class representation. Moreover, we introduce a novel conjugate episode training
strategy that strengthens the learning process. Extensive experiments on four
public benchmarks show that our approach significantly improves the
state-of-the-art performance on few-shot open-set recognition. Besides, we
extend our method to generalized few-shot open-set recognition (GFSOR), where
we also achieve performance gains on MiniImageNet