Domain Transfer in Histopathology using Multi-ProtoNets with Interactive Prototype Adaptation

Few-shot learning addresses the problem of classification when little data or few labels are available. This is especially relevant in histopathology, where labeling must be carried out by highly trained medical experts. Prototypical Networks promise transferability to new domains by using a pre-trained encoder and classifying by way of a prototypical representation of each class learned with few samples. We examine the applicability of this approach by attempting domain transfer from colon tissue (for training the encoder) to urothelial tissue. Furthermore, we address the problems arising from representing a class via a small amount of representatives (prototypes) by testing two different prototype calculation strategies. We compare the original “Prototype per Class” (PPC) approach to our “Prototype per Annotation” (PPA) method, which calculates one prototype for each example annotation made by the pathologist. We test the domain transfer capability of our approach on a dataset of 55 whole slide images (WSIs) containing six subtypes of urothelial carcinoma in two granularities: “Superclasses”, which combines the tumorous subtypes into a single “tumor” class on top of a aggregated “healthy” and additional “necrosis” class, and “subtypes”, which considers all eleven classes separately. We evaluate the classic PPC approach as well as our PPA approach on this data set. Our results show that the adaptation of the Prototypical Network from colon tissue to urothelial tissue was successful, yielding an F1 score of 0.91 for the “superclasses”. Furthermore, the PPA approach performs very comparably to the PPC strategy. This makes it a viable alternative that places more value on the intent of the pathologist during annotation

Towards interactive AI-authoring with prototypical few-shot classifiers in histopathology

A vast multitude of tasks in histopathology could potentially benefit from the support of artificial intelligence (AI). Many examples have been shown in the literature and first commercial products with FDA or CE-IVDR clearance are available. However, two key challenges remain: (1) a scarcity of thoroughly annotated images, respectively the laboriousness of this task, and (2) the creation of robust models that can cope with the data heterogeneity in the field (domain generalization). In this work, we investigate how the combination of prototypical few-shot classification models and data augmentation can address both of these challenges. Based on annotated data sets that include multiple centers, multiple scanners, and two tumor entities, we examine the robustness and the adaptability of few-shot classifiers in multiple scenarios. We demonstrate that data from one scanner and one site are sufficient to train robust few-shot classification models by applying domain-specific data augmentation. The models achieved classification performance of around 90% on a multiscanner and multicenter database, which is on par with the accuracy achieved on the primary single-center single-scanner data. Various convolutional neural network (CNN) architectures can be used for feature extraction in the few-shot model. A comparison of nine state-of-the-art architectures yielded that EfficientNet B0 provides the best trade-off between accuracy and inference time. The classification of prototypical few-shot models directly relies on class prototypes derived from example images of each class. Therefore, we investigated the influence of prototypes originating from images from different scanners and evaluated their performance also on the multiscanner database. Again, our few-shot model showed a stable performance with an average absolute deviation in accuracy compared to the primary prototypes of 1.8% points. Finally, we examined the adaptability to a new tumor entity: classification of tissue sections containing urothelial carcinoma into normal, tumor, and necrotic regions. Only three annotations per subclass (e.g., muscle and adipose tissue are subclasses of normal tissue) were provided to adapt the few-shot model, which obtained an overall accuracy of 93.6%. These results demonstrate that prototypical few-shot classification is an ideal technology for realizing an interactive AI authoring system as it only requires few annotations and can be adapted to new tasks without involving retraining of the underlying feature extraction CNN, which would in turn require a selection of hyper-parameters based on data science expert knowledge. Similarly, it can be regarded as a guided annotation system. To this end, we realized a workflow and user interface that targets non-technical users