Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis

Histopathology image analysis is the golden standard of clinical diagnosis for Cancers. In doctors daily routine and computer-aided diagnosis, the Whole Slide Image (WSI) of histopathology tissue is used for analysis. Because of the extremely large scale of resolution, previous methods generally divide the WSI into a large number of patches, then aggregate all patches within a WSI by Multi-Instance Learning (MIL) to make the slide-level prediction when developing computer-aided diagnosis tools. However, most previous WSI-MIL models using global-attention without pairwise interaction and any positional information, or self-attention with absolute position embedding can not well handle shape varying large WSIs, e.g. testing WSIs after model deployment may be larger than training WSIs, since the model development set is always limited due to the difficulty of histopathology WSIs collection. To deal with the problem, in this paper, we propose to amend position embedding for shape varying long-contextual WSI by introducing Linear Bias into Attention, and adapt it from 1-d long sequence into 2-d long-contextual WSI which helps model extrapolate position embedding to unseen or under-fitted positions. We further utilize Flash-Attention module to tackle the computational complexity of Transformer, which also keep full self-attention performance compared to previous attention approximation work. Our method, Long-contextual MIL (Long-MIL) are evaluated on extensive experiments including 4 dataset including WSI classification and survival prediction tasks to validate the superiority on shape varying WSIs. The source code will be open-accessed soon

Test-Time Training for Semantic Segmentation with Output Contrastive Loss

Although deep learning-based segmentation models have achieved impressive performance on public benchmarks, generalizing well to unseen environments remains a major challenge. To improve the model's generalization ability to the new domain during evaluation, the test-time training (TTT) is a challenging paradigm that adapts the source-pretrained model in an online fashion. Early efforts on TTT mainly focus on the image classification task. Directly extending these methods to semantic segmentation easily experiences unstable adaption due to segmentation's inherent characteristics, such as extreme class imbalance and complex decision spaces. To stabilize the adaptation process, we introduce contrastive loss (CL), known for its capability to learn robust and generalized representations. Nevertheless, the traditional CL operates in the representation space and cannot directly enhance predictions. In this paper, we resolve this limitation by adapting the CL to the output space, employing a high temperature, and simplifying the formulation, resulting in a straightforward yet effective loss function called Output Contrastive Loss (OCL). Our comprehensive experiments validate the efficacy of our approach across diverse evaluation scenarios. Notably, our method excels even when applied to models initially pre-trained using domain adaptation methods on test domain data, showcasing its resilience and adaptability.\footnote{Code and more information could be found at~ \url{https://github.com/dazhangyu123/OCL}

Exploring Unsupervised Cell Recognition with Prior Self-activation Maps

The success of supervised deep learning models on cell recognition tasks relies on detailed annotations. Many previous works have managed to reduce the dependency on labels. However, considering the large number of cells contained in a patch, costly and inefficient labeling is still inevitable. To this end, we explored label-free methods for cell recognition. Prior self-activation maps (PSM) are proposed to generate pseudo masks as training targets. To be specific, an activation network is trained with self-supervised learning. The gradient information in the shallow layers of the network is aggregated to generate prior self-activation maps. Afterward, a semantic clustering module is then introduced as a pipeline to transform PSMs to pixel-level semantic pseudo masks for downstream tasks. We evaluated our method on two histological datasets: MoNuSeg (cell segmentation) and BCData (multi-class cell detection). Compared with other fully-supervised and weakly-supervised methods, our method can achieve competitive performance without any manual annotations. Our simple but effective framework can also achieve multi-class cell detection which can not be done by existing unsupervised methods. The results show the potential of PSMs that might inspire other research to deal with the hunger for labels in medical area.Comment: MICCAI 2023. arXiv admin note: substantial text overlap with arXiv:2210.0786

Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference

For middle-school math students, interactive question-answering (QA) with tutors is an effective way to learn. The flexibility and emergent capabilities of generative large language models (LLMs) has led to a surge of interest in automating portions of the tutoring process - including interactive QA to support conceptual discussion of mathematical concepts. However, LLM responses to math questions can be incorrect or mismatched to the educational context - such as being misaligned with a school's curriculum. One potential solution is retrieval-augmented generation (RAG), which involves incorporating a vetted external knowledge source in the LLM prompt to increase response quality. In this paper, we designed prompts that retrieve and use content from a high-quality open-source math textbook to generate responses to real student questions. We evaluate the efficacy of this RAG system for middle-school algebra and geometry QA by administering a multi-condition survey, finding that humans prefer responses generated using RAG, but not when responses are too grounded in the textbook content. We argue that while RAG is able to improve response quality, designers of math QA systems must consider trade-offs between generating responses preferred by students and responses closely matched to specific educational resources.Comment: 6 pages, presented at NeurIPS'23 Workshop on Generative AI for Education (GAIED

Unleashing the Power of Prompt-driven Nucleus Instance Segmentation

Nucleus instance segmentation in histology images is crucial for a broad spectrum of clinical applications. Current dominant algorithms rely on regression of nuclear proxy maps. Distinguishing nucleus instances from the estimated maps requires carefully curated post-processing, which is error-prone and parameter-sensitive. Recently, the Segment Anything Model (SAM) has earned huge attention in medical image segmentation, owing to its impressive generalization ability and promptable property. Nevertheless, its potential on nucleus instance segmentation remains largely underexplored. In this paper, we present a novel prompt-driven framework that consists of a nucleus prompter and SAM for automatic nucleus instance segmentation. Specifically, the prompter learns to generate a unique point prompt for each nucleus while the SAM is fine-tuned to output the corresponding mask for the prompted nucleus. Furthermore, we propose the inclusion of adjacent nuclei as negative prompts to enhance the model's capability to identify overlapping nuclei. Without complicated post-processing, our proposed method sets a new state-of-the-art performance on three challenging benchmarks. Code is available at \url{github.com/windygoo/PromptNucSeg}Comment: under revie

PathAsst: Redefining Pathology through Generative Foundation AI Assistant for Pathology

As advances in large language models (LLMs) and multimodal techniques continue to mature, the development of general-purpose multimodal large language models (MLLMs) has surged, with significant applications in natural image interpretation. However, the field of pathology has largely remained untapped in this regard, despite the growing need for accurate, timely, and personalized diagnostics. To bridge the gap in pathology MLLMs, we present the PathAsst in this study, which is a generative foundation AI assistant to revolutionize diagnostic and predictive analytics in pathology. To develop PathAsst, we collect over 142K high-quality pathology image-text pairs from a variety of reliable sources, including PubMed, comprehensive pathology textbooks, reputable pathology websites, and private data annotated by pathologists. Leveraging the advanced capabilities of ChatGPT/GPT-4, we generate over 180K instruction-following samples. Furthermore, we devise additional instruction-following data, specifically tailored for the invocation of the pathology-specific models, allowing the PathAsst to effectively interact with these models based on the input image and user intent, consequently enhancing the model's diagnostic capabilities. Subsequently, our PathAsst is trained based on Vicuna-13B language model in coordination with the CLIP vision encoder. The results of PathAsst show the potential of harnessing the AI-powered generative foundation model to improve pathology diagnosis and treatment processes. We are committed to open-sourcing our meticulously curated dataset, as well as a comprehensive toolkit designed to aid researchers in the extensive collection and preprocessing of their own datasets. Resources can be obtained at https://github.com/superjamessyx/Generative-Foundation-AI-Assistant-for-Pathology.Comment: 13 pages, 5 figures, conferenc

Opposite effects of single-dose and multidose administration of the ethanol extract of danshen on

The aim of this study was to investigate the effect of single-and multidose administration of the ethanol extract of danshen on in vivo CYP3A activity in healthy volunteers. A sequential, open-label, and three-period pharmacokinetic interaction study design was used based on 12 healthy male individuals. The plasma concentrations of midazolam and its metabolite 1-hydroxymidazolam were measured. Treatment with single dose of the extract caused the mean max of midazolam to increase by 87% compared with control. After 10 days of the danshen extract intake, the mean AUC 0-12 , max , and 1/2 of midazolam were decreased by 79.9%, 66.6%, and 43.8%, respectively. The mean clearance of midazolam was increased by 501.6% compared with control. The in vitro study showed that dihydrotanshinone I in the extract could inhibit CYP3A, while tanshinone IIA and cryptotanshinone could induce CYP3A. In conclusion, a single-dose administration of the danshen extract can inhibit intestinal CYP3A, but multidose administration can induce intestinal and hepatic CYP3A

Epigenetic Regulation of Organ Regeneration in Zebrafish

The zebrafish is broadly used for investigating de novo organ regeneration, because of its strong regenerative potential. Over the past two decades of intense study, significant advances have been made in identifying both the regenerative cell sources and molecular signaling pathways in a variety of organs in adult zebrafish. Epigenetic regulation has gradually moved into the center-stage of this research area, aided by comprehensive work demonstrating that DNA methylation, histone modifications, chromatin remodeling complexes, and microRNAs are essential for organ regeneration. Here, we present a brief review of how these epigenetic components are induced upon injury, and how they are involved in sophisticated organ regeneration. In addition, we highlight several prospective research directions and their potential implications for regenerative medicine

Construction of Pt-M (M = Co, Ni, Fe)/g-C₃N₄ composites for highly efficient photocatalytic H₂ generation

Platinum (Pt) is recognized as an excellent cocatalyst which not only suppresses the charge carrier recombination of the photocatalyst but also reduces the overpotential for photocatalytic H2 generation. Albeit of its good performance, the high cost and low abundance restricted the utilization of Pt in large-scale photocatalytic H2 generation. Pt based transition metal alloys are demonstrated to reveal enhanced activities towards various catalytic reactions, suggesting the possibility to substitute Pt as the cocatalyst. In the present work, Pt was partially substituted with Co, Ni, and Fe and Pt-M (M = Co, Ni, and Fe)/g-C3N4 composites were constructed through co-reduction of H2PtCl6 and transition metal salts by the reductant of ethylene glycol. The crystal structure and valence states were measured by X-ray diffractometer (XRD) and X-ray photoelectron spectrometer (XPS), respectively. The higher degree of XRD peaks and larger binding energies for Pt 4f5/2 and Pt 4f7/2 after incorporating Co2+ ions indicated that Co was successfully introduced into the lattice of Pt and Pt-Co bimetallic alloys was attained through the solvothermal treatment. The morphology was subsequently observed by transmission electron microscope (TEM), which showed a good dispersion of Pt-Co nanoparticles on the surface of g-C3N4. Meanwhile, the shrinkage of lattice fringe after introducing cobalt salt further confirmed the presence of Pt-Co bimetallic alloys. The UV-Vis absorption spectra of g-C3N4 and Pt, Pt-Co deposited g-C3N4 were subsequently performed. It was found that the absorption edges were all consistent for all three samples as anticipated, implying that the band gap energy was maintained after hybridizing with Pt or Pt-Co alloys. Furthermore, the photocatalytic H2 generation was carried out over the as-prepared composites with triethanolamine (TEOA) as sacrificial reagent. Under visible-light illumination, the1% (w) Pt2.5M/g-C3N4 (M = Co, Fe, Ni) composites all exhibited higher or comparable activity towards photocatalytic H2 generation when compared to 1% (w) Pt loaded counterpart. In addition, the atomic ratios of Pt/Co and the loading amount of Pt-Co cocatalyst were modified to optimize the photocatalytic performance, among which, 1% (w) Pt2.5Co/g-C3N4 composite revealed the highest activity with a 1.6-time enhancement. Electrochemical impedance spectra (EIS) and photoluminescence (PL) spectra indicated that the enhancement might be attributed to improved charge transfer from g-C3N4 to Pt2.5Co cocatalyst and inhibited charge carrier recombination in the presence of Pt2.5Co cocatalyst. Therefore, the present study demonstrates the great potential to partially replace Pt with low-cost and abundant transition metals and to fabricate Pt based bimetallic alloys as promising cocatalysts for highly efficient photocatalytic H2 generation. 铂(Pt)是公认的优秀助催化剂：一方面，Pt能抑制光催化过程中光生载流子的复合；另一方面，Pt能降低光解水制 氢反应过电势。尽管如此，高昂的价格和极低的丰度限制了Pt在光解水制氢中的应用。Pt基过渡金属合金在多种催化反 应中呈现出卓越的催化性能，是潜在的助催化材料。在本工作中，我们利用Co、Ni、Fe等过渡金属部分取代贵金属Pt，并 以乙二醇为还原剂，通过原位还原H2PtCl6和过渡金属盐制备了Pt-M/g-C3N4 (M = Co, Ni, Fe)复合材料。在可见光照射 下，1% (w) Pt2.5M/g-C3N4 (M = Co, Ni, Fe)均表现出比同样条件下Pt负载的复合材料更高或者相当的光解水制氢性能。其 次，我们通过调节Pt/Co的原子比例以及Pt-Co合金的负载量进一步优化了光催化性能。结果显示：1% (w) Pt2.5Co/g-C3N4 复合材料表现出最高的光解水制氢性能，是同样条件下Pt负载的1.6倍。电化学阻抗谱(EIS)和光致发光光谱(PL)表明光 催化性能的提升得益于电子从g-C3N4向Pt2.5Co的有效传输以及光生载流子复合被有效抑制。这一工作显示Pt基过渡金属 合金在高效光解水制氢中具有潜在的应用前景，对于开发低成本、高效助催化剂具有一定的指导意义。Published versionThe project was supported by the Jiangsu Provincial Natural Science Foundation, China (BK20160987)