Segment Anything Model for Medical Images?

The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It designed a novel promotable segmentation task, ensuring zero-shot image segmentation using the pre-trained model via two main modes including automatic everything and manual prompt. SAM has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging due to the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. SAM has achieved impressive results on various natural image segmentation tasks. Meanwhile, zero-shot and efficient MIS can well reduce the annotation time and boost the development of medical image analysis. Hence, SAM seems to be a potential tool and its performance on large medical datasets should be further validated. We collected and sorted 52 open-source datasets, and build a large medical segmentation dataset with 16 modalities, 68 objects, and 553K slices. We conducted a comprehensive analysis of different SAM testing strategies on the so-called COSMOS 553K dataset. Extensive experiments validate that SAM performs better with manual hints like points and boxes for object perception in medical images, leading to better performance in prompt mode compared to everything mode. Additionally, SAM shows remarkable performance in some specific objects and modalities, but is imperfect or even totally fails in other situations. Finally, we analyze the influence of different factors (e.g., the Fourier-based boundary complexity and size of the segmented objects) on SAM's segmentation performance. Extensive experiments validate that SAM's zero-shot segmentation capability is not sufficient to ensure its direct application to the MIS.Comment: 23 pages, 14 figures, 12 table

When employer brand image aids employee satisfaction and engagement

Purpose – The purpose of this paper is to test whether employee characteristics (age, gender, role and experience) influence the effects of employer brand image, for warmth and competence, on employee satisfaction and engagement. Design/methodology/approach – Members of the public were surveyed as to their satisfaction and engagement with their employer and their view of their employer brand image. Half were asked to evaluate their employer’s “warmth” and half its “competence”. The influence of employee characteristics was tested on a “base model” linking employer image to satisfaction and engagement using a mediated moderation model. Findings – The base model proved valid; satisfaction partially mediates the influence of employer brand image on engagement. Age, experience gender, and whether the role involved customer contact moderate both the influence of the employer brand image and of satisfaction on engagement. Practical implications – Engagement varies with employee characteristics, and both segmenting employees and promoting the employer brand image differentially to specific groups are ways to counter this effect. Originality/value – The contexts in which employer brand image can influence employees in general and specific groups of employees in particular are not well understood. This is the first empirical study of the influence of employer brand image on employee engagement and one of few that considers the application of employee segmentation

Improving Fine-Grain Segmentation via Interpretable Modifications: A Case Study in Fossil Segmentation

Most interpretability research focuses on datasets containing thousands of images of commonplace objects. However, many high-impact datasets, such as those in medicine and the geosciences, contain fine-grain objects that require domain-expert knowledge to recognize and are time-consuming to collect and annotate. As a result, these datasets contain few annotated images, and current machine vision models cannot train intensively on them. Thus, adapting interpretability techniques to maximize the amount of information that models can learn from small, fine-grain datasets is an important endeavor. Using a Mask R-CNN to segment ancient reef fossils in rock sample images, we present a general paradigm for identifying and mitigating model weaknesses. Specifically, we apply image perturbations to expose the Mask R-CNN's inability to distinguish between different classes of fossils and its inconsistency in segmenting fossils with different textures. To address these shortcomings, we extend an existing model-editing method for correcting systematic mistakes in image classification to image segmentation and introduce a novel application of the technique: encouraging a greater separation between positive and negative pixels for a given class. Through extensive experiments, we find that editing the model by perturbing all pixels for a given class in one image is most effective (compared to using multiple images and/or fewer pixels). Our paradigm may also generalize to other segmentation models trained on small, fine-grain datasets

Segmentation-aware Image Denoising Without Knowing True Segmentation

Recent works have discussed application-driven image restoration neural networks capable of not only removing noise in images but also preserving their semantic-aware details, making them suitable for various high-level computer vision tasks as the pre-processing step. However, such approaches require extra annotations for their high-level vision tasks in order to train the joint pipeline using hybrid losses, yet the availability of those annotations is often limited to a few image sets, thereby restricting the general applicability of these methods to simply denoise more unseen and unannotated images. Motivated by this, we propose a segmentation-aware image denoising model dubbed U-SAID, based on a novel unsupervised approach with a pixel-wise uncertainty loss. U-SAID does not require any ground-truth segmentation map, and thus can be applied to any image dataset. It is capable of generating denoised images with comparable or even better quality than that of its supervised counterpart and even more general “application-agnostic” denoisers, and its denoised results show stronger robustness for subsequent semantic segmentation tasks. Moreover, plugging its “universal” denoiser without fine-tuning, we demonstrate the superior generalizability of U-SAID in three-folds: (1) denoising unseen types of images; (2) denoising as preprocessing for segmenting unseen noisy images; and (3) denoising for unseen high-level tasks. Extensive experiments were conducted to assess the effectiveness and robustness of the proposed U-SAID model against various popular image sets

imageseg: An R package for deep learning-based image segmentation

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. © 2022 The Authors. Methods in Ecology and Evolution published by John Wiley & Sons Ltd on behalf of British Ecological SocietyConvolutional neural networks (CNNs) and deep learning are powerful and robust tools for ecological applications and are particularly suited for image data. Image segmentation (the classification of all pixels in images) is one such application and can, for example, be used to assess forest structural metrics. While CNN-based image segmentation methods for such applications have been suggested, widespread adoption in ecological research has been slow, likely due to technical difficulties in implementation of CNNs and lack of toolboxes for ecologists. Here, we present R package imageseg which implements a CNN-based workflow for general purpose image segmentation using the U-Net and U-Net++ architectures in R. The workflow covers data (pre)processing, model training and predictions. We illustrate the utility of the package with image recognition models for two forest structural metrics: tree canopy density and understorey vegetation density. We trained the models using large and diverse training datasets from a variety of forest types and biomes, consisting of 2877 canopy images (both canopy cover and hemispherical canopy closure photographs) and 1285 understorey vegetation images. Overall segmentation accuracy of the models was high with a Dice score of 0.91 for the canopy model and 0.89 for the understorey vegetation model (assessed with 821 and 367 images respectively). The image segmentation models performed significantly better than commonly used thresholding methods and generalized well to data from study areas not included in training. This indicates robustness to variation in input images and good generalization strength across forest types and biomes. The package and its workflow allow simple yet powerful assessments of forest structural metrics using pretrained models. Furthermore, the package facilitates custom image segmentation with single or multiple classes and based on colour or grayscale images, for example, for applications in cell biology or for medical images. Our package is free, open source and available from CRAN. It will enable easier and faster implementation of deep learning-based image segmentation within R for ecological applications and beyond.publishedVersio

The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot

Segmentation is an essential step for remote sensing image processing. This study aims to advance the application of the Segment Anything Model (SAM), an innovative image segmentation model by Meta AI, in the field of remote sensing image analysis. SAM is known for its exceptional generalization capabilities and zero-shot learning, making it a promising approach to processing aerial and orbital images from diverse geographical contexts. Our exploration involved testing SAM across multi-scale datasets using various input prompts, such as bounding boxes, individual points, and text descriptors. To enhance the model's performance, we implemented a novel automated technique that combines a text-prompt-derived general example with one-shot training. This adjustment resulted in an improvement in accuracy, underscoring SAM's potential for deployment in remote sensing imagery and reducing the need for manual annotation. Despite the limitations encountered with lower spatial resolution images, SAM exhibits promising adaptability to remote sensing data analysis. We recommend future research to enhance the model's proficiency through integration with supplementary fine-tuning techniques and other networks. Furthermore, we provide the open-source code of our modifications on online repositories, encouraging further and broader adaptations of SAM to the remote sensing domain.Comment: 20 pages, 9 figure

Meta-learning with implicit gradients in a few-shot setting for medical image segmentation

Widely used traditional supervised deep learning methods require a large number of training samples but often fail to generalize on unseen datasets. Therefore, a more general application of any trained model is quite limited for medical imaging for clinical practice. Using separately trained models for each unique lesion category or a unique patient population will require sufficiently large curated datasets, which is not practical to use in a real-world clinical set-up. Few-shot learning approaches can not only minimize the need for an enormous number of reliable ground truth labels that are labour-intensive and expensive, but can also be used to model on a dataset coming from a new population. To this end, we propose to exploit an optimization-based implicit model agnostic meta-learning (iMAML) algorithm under few-shot settings for medical image segmentation. Our approach can leverage the learned weights from diverse but small training samples to perform analysis on unseen datasets with high accuracy. We show that, unlike classical few-shot learning approaches, our method improves generalization capability. To our knowledge, this is the first work that exploits iMAML for medical image segmentation and explores the strength of the model on scenarios such as meta-training on unique and mixed instances of lesion datasets. Our quantitative results on publicly available skin and polyp datasets show that the proposed method outperforms the naive supervised baseline model and two recent few-shot segmentation approaches by large margins. In addition, our iMAML approach shows an improvement of 2%–4% in dice score compared to its counterpart MAML for most experiments