We introduce a method for flexible continual learning in open-vocabulary
image classification, drawing inspiration from the complementary learning
systems observed in human cognition. We propose a "tree probe" method, an
adaption of lazy learning principles, which enables fast learning from new
examples with competitive accuracy to batch-trained linear models. Further, we
propose a method to combine predictions from a CLIP zero-shot model and the
exemplar-based model, using the zero-shot estimated probability that a sample's
class is within any of the exemplar classes. We test in data incremental, class
incremental, and task incremental settings, as well as ability to perform
flexible inference on varying subsets of zero-shot and learned categories. Our
proposed method achieves a good balance of learning speed, target task
effectiveness, and zero-shot effectiveness.Comment: In revie