SAM Meets Robotic Surgery: An Empirical Study on Generalization, Robustness and Adaptation

The Segment Anything Model (SAM) serves as a fundamental model for semantic segmentation and demonstrates remarkable generalization capabilities across a wide range of downstream scenarios. In this empirical study, we examine SAM's robustness and zero-shot generalizability in the field of robotic surgery. We comprehensively explore different scenarios, including prompted and unprompted situations, bounding box and points-based prompt approaches, as well as the ability to generalize under corruptions and perturbations at five severity levels. Additionally, we compare the performance of SAM with state-of-the-art supervised models. We conduct all the experiments with two well-known robotic instrument segmentation datasets from MICCAI EndoVis 2017 and 2018 challenges. Our extensive evaluation results reveal that although SAM shows remarkable zero-shot generalization ability with bounding box prompts, it struggles to segment the whole instrument with point-based prompts and unprompted settings. Furthermore, our qualitative figures demonstrate that the model either failed to predict certain parts of the instrument mask (e.g., jaws, wrist) or predicted parts of the instrument as wrong classes in the scenario of overlapping instruments within the same bounding box or with the point-based prompt. In fact, SAM struggles to identify instruments in complex surgical scenarios characterized by the presence of blood, reflection, blur, and shade. Additionally, SAM is insufficiently robust to maintain high performance when subjected to various forms of data corruption. We also attempt to fine-tune SAM using Low-rank Adaptation (LoRA) and propose SurgicalSAM, which shows the capability in class-wise mask prediction without prompt. Therefore, we can argue that, without further domain-specific fine-tuning, SAM is not ready for downstream surgical tasks.Comment: Accepted as Oral Presentation at MedAGI Workshop - MICCAI 2023 1st International Workshop on Foundation Models for General Medical AI. arXiv admin note: substantial text overlap with arXiv:2304.1467

S2^2ME: Spatial-Spectral Mutual Teaching and Ensemble Learning for Scribble-supervised Polyp Segmentation

Fully-supervised polyp segmentation has accomplished significant triumphs over the years in advancing the early diagnosis of colorectal cancer. However, label-efficient solutions from weak supervision like scribbles are rarely explored yet primarily meaningful and demanding in medical practice due to the expensiveness and scarcity of densely-annotated polyp data. Besides, various deployment issues, including data shifts and corruption, put forward further requests for model generalization and robustness. To address these concerns, we design a framework of Spatial-Spectral Dual-branch Mutual Teaching and Entropy-guided Pseudo Label Ensemble Learning (S$^2$ME). Concretely, for the first time in weakly-supervised medical image segmentation, we promote the dual-branch co-teaching framework by leveraging the intrinsic complementarity of features extracted from the spatial and spectral domains and encouraging cross-space consistency through collaborative optimization. Furthermore, to produce reliable mixed pseudo labels, which enhance the effectiveness of ensemble learning, we introduce a novel adaptive pixel-wise fusion technique based on the entropy guidance from the spatial and spectral branches. Our strategy efficiently mitigates the deleterious effects of uncertainty and noise present in pseudo labels and surpasses previous alternatives in terms of efficacy. Ultimately, we formulate a holistic optimization objective to learn from the hybrid supervision of scribbles and pseudo labels. Extensive experiments and evaluation on four public datasets demonstrate the superiority of our method regarding in-distribution accuracy, out-of-distribution generalization, and robustness, highlighting its promising clinical significance. Our code is available at https://github.com/lofrienger/S2ME.Comment: MICCAI 2023 Early Acceptanc

An Efficient MLP-based Point-guided Segmentation Network for Ore Images with Ambiguous Boundary

The precise segmentation of ore images is critical to the successful execution of the beneficiation process. Due to the homogeneous appearance of the ores, which leads to low contrast and unclear boundaries, accurate segmentation becomes challenging, and recognition becomes problematic. This paper proposes a lightweight framework based on Multi-Layer Perceptron (MLP), which focuses on solving the problem of edge burring. Specifically, we introduce a lightweight backbone better suited for efficiently extracting low-level features. Besides, we design a feature pyramid network consisting of two MLP structures that balance local and global information thus enhancing detection accuracy. Furthermore, we propose a novel loss function that guides the prediction points to match the instance edge points to achieve clear object boundaries. We have conducted extensive experiments to validate the efficacy of our proposed method. Our approach achieves a remarkable processing speed of over 27 frames per second (FPS) with a model size of only 73 MB. Moreover, our method delivers a consistently high level of accuracy, with impressive performance scores of 60.4 and 48.9 in~$AP_{50}^{box}$ and~$AP_{50}^{mask}$ respectively, as compared to the currently available state-of-the-art techniques, when tested on the ore image dataset. The source code will be released at \url{https://github.com/MVME-HBUT/ORENEXT}.Comment: 10 pages, 8 figure

Monolithic quantum-dot distributed feedback laser array on silicon

Electrically-pumped lasers directly grown on silicon are key devices interfacing silicon microelectronics and photonics. We report here, for the first time, an electrically-pumped, room-temperature, continuous-wave (CW) and single-mode distributed feedback (DFB) laser array fabricated in InAs/GaAs quantum-dot (QD) gain material epitaxially grown on silicon. CW threshold currents as low as 12 mA and single-mode side mode suppression ratios (SMSRs) as high as 50 dB have been achieved from individual devices in the array. The laser array, compatible with state-of-the-art coarse wavelength division multiplexing (CWDM) systems, has a well-aligned channel spacing of 20 0.2 nm and exhibits a record wavelength coverage range of 100 nm, the full span of the O-band. These results indicate that, for the first time, the performance of lasers epitaxially grown on silicon is elevated to a point approaching real-world CWDM applications, demonstrating the great potential of this technology

Effects of bisphenol A exposure at different circadian time on hepatic lipid metabolism in mice

BackgroundLipid metabolism in liver shows circadian-dependent profiles. The hepatotoxicity of environmental chemicals is dependent on circadian time. ObjectiveTo observe the effects of bisphenol A (BPA) exposure at different zeitgeber time (ZT) on hepatic and blood lipid metabolism and decipher the underlying mechanisms related to circadian rhythm in mice. MethodsThirty-five female C57BL/6J mice were sacrificed every 4 h in a light-dark cycle (12 h/12 h). The liver tissues were collected to describe the circadian profiles of hepatic Rev-erba, Bmal1, Clock, Srebp1c, and Chrebp mRNA expression levels within 24 h. Thirty female mice were divided into 6 groups by the timing (ZT3 represents the 3 h after light on, ZT15 represents the 3 h after light off) and dose (50 or 500 μg·kg−1·d−1) of BPA exposure to observe hepatotoxicity. Mice were gavaged with designed doses of BPA once per day for 4 weeks. Mice were maintained with ad libitum access to food and water and measured body weight weekly. After the experiment, mice were euthanatized and liver tissues were separated to determine the biochemical indicators of lipid metabolism and lipid metabolism- and circadian-related gene mRNA expressions. ResultsHepatic Rev-erba, Bmal1, Clock, Srebp1c, and Chrebp mRNA expression levels were rhythmic during a 24 h period in mice. At ZT3 and ZT15, BPA did not alter body weight, plasma glucose, plasma total cholesterol, plasma low density lipoprotein cholesterol, and plasma triglycerides (P>0.05). The plasma high density lipoprotein cholesterol decreased in the 50 μg·kg−1·d−1 BPA group at ZT3 by 14.56% compared with the control group (P<0.05). The liver triglycerides increased in the 50 μg·kg−1·d−1 BPA group at ZT15 by 115.20% compared with the control group (P<0.05). BPA decreased Srebp1c mRNA expression level when dosing at ZT3 and increased Chrebp, Srebp1c, and Acc1 mRNA expression levels when dosing at ZT15 compared with the control group (P<0.05). BPA increased Bmal1 mRNA expression level and decreased Rev-erbα mRNA expression level at ZT3 exposure and decreased Bmal1 and increased Rev-erbα mRNA expression level at ZT15 exposure (P<0.05). ConclusionBPA exposure at light or dark period has different effects on hepatic lipid metabolism in mice. Hepatic lipid deposit appears when BPA is dosed at dark period. Rev-erbα-Bmal1 regulation circuits and the subsequent upregulation of Srebp1c and Chrebp and the target gene Acc1 may be involved

Imaging-guided synergistic targeting-promoted photo-chemotherapy against cancers by methotrexate-conjugated hyaluronic acid nanoparticles

Abstract(#br)A combination of chemotherapy and photothermal therapy (PTT) against cancer, overcoming the intrinsic limitations of single-modal chemotherapy or PTT, has emerged as a promising strategy to achieve synergistic therapeutic effect. However, the lack of precise drug delivery and intelligent drug release based on photo-chemotherapy at specific tumor sites remained a challenge. Hence, the both tumor-specific targeting molecule (methotrexate) and ligand (hyaluronic acid)-introduced, glutathione-responsive amphiphiles (deoxycholic acid-hyaluronic acid-methotrexate, DA-SS-HA-MTX) were developed for synchronous delivery of indocyanine green (ICG) and doxorubicin (DOX). The as-synthesized DOX/ICG@DSHM remarkably improved the intracellular drug uptake and accumulation owing to both the CD44/folate receptors-mediated synergistic targeting and the glutathione-triggered rapid drug release. Moreover, DOX/ICG@DSHM efficiently accumulated at the tumor sites, realizing the notable tumor ablation under the guidance of dual-modal optical imaging. Taken together, this study provided a promising nanotheranostic agent for imaging-guided chemo-photothermal combination therapy

SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge

Surgical tool segmentation and action recognition are fundamental building blocks in many computer-assisted intervention applications, ranging from surgical skills assessment to decision support systems. Nowadays, learning-based action recognition and segmentation approaches outperform classical methods, relying, however, on large, annotated datasets. Furthermore, action recognition and tool segmentation algorithms are often trained and make predictions in isolation from each other, without exploiting potential cross-task relationships. With the EndoVis 2022 SAR-RARP50 challenge, we release the first multimodal, publicly available, in-vivo, dataset for surgical action recognition and semantic instrumentation segmentation, containing 50 suturing video segments of Robotic Assisted Radical Prostatectomy (RARP). The aim of the challenge is twofold. First, to enable researchers to leverage the scale of the provided dataset and develop robust and highly accurate single-task action recognition and tool segmentation approaches in the surgical domain. Second, to further explore the potential of multitask-based learning approaches and determine their comparative advantage against their single-task counterparts. A total of 12 teams participated in the challenge, contributing 7 action recognition methods, 9 instrument segmentation techniques, and 4 multitask approaches that integrated both action recognition and instrument segmentation. The complete SAR-RARP50 dataset is available at: https://rdr.ucl.ac.uk/projects/SARRARP50_Segmentation_of_surgical_instrumentation_and_Action_Recognition_on_Robot-Assisted_Radical_Prostatectomy_Challenge/19109

A multimechanistic antibody targeting receptor-binding sites potently cross-protects against influenza B viruses

流感病毒HA是研制流感药物和流感疫苗的重要靶标，但HA具有高度变异性，如何在高变异HA中找到不变之处，即高度保守表位，是研制流感特效药物和广谱疫苗的关键。近年来国外报道的流感HA广谱中和单抗的识别位点均在较为保守的HA茎部区，而针对流感病毒与细胞受体结合部位的HA头部区尤其是RBS区，一直未能发现广谱中和抗体。夏宁邵教授团队通过探索多种免疫策略和筛选策略，成功筛选出一株广谱中和单抗12G6，识别一个位于HA头部RBS上的全新保守性表位。体外实验显示12G6人源化改造的C12G6抗体能高效中和1940-2016年间世界各地历年流行的代表三个遗传变异亚系的18个乙型流感病毒代表株对细胞的感染，并能保护小鼠致死性感染，治疗效果显著优于已报道的代表性抗体以及抗流感药物；C12G6与“达菲”联合用药具有明显的协同效果。此外，雪貂感染模型的预防和治疗效果进一步证实了C12G6作为抗体药物的治疗潜能。研究还显示该表位是病毒感染复制的关键表位，该位点的突变会造成病毒毒力显著下降。最后，研究揭示了C12G6通过五种不同的抗病毒作用机制发挥作用，提示其高效的抗病毒活性得益于多机制协同效应，这也是目前国内外第一次发现一个流感抗体能通过如此全面的抗病毒机制发挥作用。 该发现为研制能抵抗各种变异株的乙型流感特效治疗药物和通用疫苗带来新希望。 该研究工作依托分子疫苗学和分子诊断学国家重点实验室（厦门大学）、国家传染病诊断试剂与疫苗工程技术研究中心、厦门大学养生堂生物药物联合实验室完成。陈毅歆副教授、夏宁邵教授为该研究论文的共同通讯作者。在读博士研究生沈晨光、陈俊煜、李睿、王国松和硕士研究生张梦娅等为共同第一作者。【Abstract】Influenza B virus causes considerable disease burden worldwide annually, highlighting the limitations of current influenza vaccines and antiviral drugs. In recent years, broadly neutralizing antibodies (bnAbs) against hemagglutinin (HA) have emerged as a new approach for combating influenza. We describe the generation and characterization of a chimeric monoclonal antibody, C12G6, that cross-neutralizes representative viruses spanning the 76 years of influenza B antigenic evolution since 1940, including viruses belonging to the Yamagata, Victoria, and earlier lineages. Notably, C12G6 exhibits broad cross-lineage hemagglutination inhibition activity against influenza B viruses and has higher potency and breadth of neutralization when compared to four previously reported influenza B bnAbs. In vivo, C12G6 confers stronger cross-protection against Yamagata and Victoria lineages of influenza B viruses in mice and ferrets than other bnAbs or the anti-influenza drug oseltamivir and has an additive antiviral effect when administered in combination with oseltamivir. Epitope mapping indicated that C12G6 targets a conserved epitope that overlaps with the receptor binding site in the HA region of influenza B virus, indicating why it neutralizes virus so potently. Mechanistic analyses revealed that C12G6 inhibits influenza B viruses via multiple mechanisms, including preventing viral entry, egress, and HA-mediated membrane fusion and triggering antibody-dependent cell-mediated cytotoxicity and complement-dependent cytotoxicity responses. C12G6 is therefore a promising candidate for the development of prophylactics or therapeutics against influenza B infection and may inform the design of a truly universal influenza vaccine.This research was supported by grants from the National Natural Science Foundation of China (31670934 and 81371817), the Ministry of Science and Technology of the People’s Republic of China (2011ZX09102-009-12 and 2012DFH30020), the Research Grants Council of the Hong Kong Special Administrative Region (7629/13M, 17103214, and 17154516), and a sponsored research agreement from Sanofi Pasteur. 研究工作得到了香港大学新发传染病国家重点实验室和赛诺菲巴斯德公司的技术支持和帮助，获得国家自然科学基金、新药创制国家科技重大专项、科技部对港科技合作项目等课题资助

Morphological diversity of single neurons in molecularly defined cell types.

Dendritic and axonal morphology reflects the input and output of neurons and is a defining feature of neuronal types1,2, yet our knowledge of its diversity remains limited. Here, to systematically examine complete single-neuron morphologies on a brain-wide scale, we established a pipeline encompassing sparse labelling, whole-brain imaging, reconstruction, registration and analysis. We fully reconstructed 1,741 neurons from cortex, claustrum, thalamus, striatum and other brain regions in mice. We identified 11 major projection neuron types with distinct morphological features and corresponding transcriptomic identities. Extensive projectional diversity was found within each of these major types, on the basis of which some types were clustered into more refined subtypes. This diversity follows a set of generalizable principles that govern long-range axonal projections at different levels, including molecular correspondence, divergent or convergent projection, axon termination pattern, regional specificity, topography, and individual cell variability. Although clear concordance with transcriptomic profiles is evident at the level of major projection type, fine-grained morphological diversity often does not readily correlate with transcriptomic subtypes derived from unsupervised clustering, highlighting the need for single-cell cross-modality studies. Overall, our study demonstrates the crucial need for quantitative description of complete single-cell anatomy in cell-type classification, as single-cell morphological diversity reveals a plethora of ways in which different cell types and their individual members may contribute to the configuration and function of their respective circuits

La montaña artificial: el zoo como lugar de construcción de la naturaleza

El zoo es un mediador complejo entre la Naturaleza y la sociedad humana. Hoy en día, los estudios sobre los zoológicos se han desarrollado habitualmente como una actividad interdisciplinaria en el contexto de las humanidades y de las ciencias sociales. El presente trabajo intenta aportar un discurso específico desde la perspectiva arquitectónica. Es evidente que las casas de fieras afrontan algunas de las problemáticas básicas de la arquitectura y del paisaje, tales como el habitar, el mirar, el limitar y la reconstrucción de la Naturaleza. Para no perderse en un tema tan extenso, la investigación se concentra en torno a la construcción de montaña falsa del parque zoológico a principios del Siglo XX. Antes de entrar propiamente a los casos de estudios, se argumenta temas fundamentales para contextualizar, tales como la ¨artelización¨ de la Naturaleza artificial, el zoo como lugar de heterotopía, la ideología de la montaña. Una vez argumentadas estas hipótesis de partida, se concentra en los objetos principales del tema estudiado. El análisis aborda los tres casos más representativos de montañas artificiales en zoológicos, que son el Panorama del Tierpark Hagenbeck (1907), The Mappin Terrace del London Zoo (1913) y Le Grand Rocher del Zoo de Vincennes (1934), investigando las construcciones desde varias ópticas. En paralelo, en relación con el propio tema, se organiza una serie de atlas como un discurso dialectico hacia la propia tesis. Esta parte gráfica se presenta intercalando en los textos en un formato menor y no afecta a la integridad de la lectura principal