Unmasked Teacher: Towards Training-Efficient Video Foundation Models

Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our method enables faster convergence and multimodal friendliness. With a progressive pre-training framework, our model can handle various tasks including scene-related, temporal-related, and complex video-language understanding. Using only public sources for pre-training in 6 days on 32 A100 GPUs, our scratch-built ViT-L/16 achieves state-of-the-art performances on various video tasks. The code and models will be released at https://github.com/OpenGVLab/unmasked_teacher.Comment: 16 pages, 5 figures, 28 table

Harvest Video Foundation Models via Efficient Post-Pretraining

Building video-language foundation models is costly and difficult due to the redundant nature of video data and the lack of high-quality video-language datasets. In this paper, we propose an efficient framework to harvest video foundation models from image ones. Our method is intuitively simple by randomly dropping input video patches and masking out input text during the post-pretraining procedure. The patch dropping boosts the training efficiency significantly and text masking enforces the learning of cross-modal fusion. We conduct extensive experiments to validate the effectiveness of our method on a wide range of video-language downstream tasks including various zero-shot tasks, video question answering, and video-text retrieval. Despite its simplicity, our method achieves state-of-the-art performances, which are comparable to some heavily pretrained video foundation models. Our method is extremely efficient and can be trained in less than one day on 8 GPUs, requiring only WebVid-10M as pretraining data. We hope our method can serve as a simple yet strong counterpart for prevalent video foundation models, provide useful insights when building them, and make large pretrained models more accessible and sustainable. This is part of the InternVideo project \url{https://github.com/OpenGVLab/InternVideo}

Mendelian randomization supports genetic liability to hospitalization for COVID-19 as a risk factor of pre-eclampsia

BackgroundPre-eclampsia and eclampsia are among the major threats to pregnant women and fetuses, but they can be mitigated by prevention and early screening. Existing observational research presents conflicting evidence regarding the causal effects of coronavirus disease 2019 (COVID-19) on pre-eclampsia risk. Through Mendelian randomization (MR), this study aims to investigate the causal effect of three COVID-19 severity phenotypes on the risk of pre-eclampsia and eclampsia to provide more rigorous evidence.MethodsTwo-sample MR was utilized to examine causal effects. Summary-level data from genome-wide association studies (GWAS) of individuals of European ancestry were acquired from the GWAS catalog and FinnGen databases. Single-nucleotide polymorphisms associated with COVID-19 traits at p < 5 × −8 were obtained and pruned for linkage disequilibrium to generate instrumental variables for COVID-19. Inverse variance weighted estimates were used as the primary MR results, with weighted median and MR-Egger as auxiliary analyses. The robustness of the MR findings was also evaluated through sensitivity analyses. Bonferroni correction was applied to primary results, with a p < 0.0083 considered significant evidence and a p within 0.083–0.05 considered suggestive evidence.ResultsCritical ill COVID-19 [defined as hospitalization for COVID-19 with either a death outcome or respiratory support, OR (95% CI): 1.17 (1.03–1.33), p = 0.020] and hospitalized COVID-19 [defined as hospitalization for COVID-19, OR (95% CI): 1.10 (1.01–1.19), p = 0.026] demonstrated suggestive causal effects on pre-eclampsia, while general severe acute respiratory syndrome coronavirus 2 infection did not exhibit a significant causal effect on pre-eclampsia. None of the three COVID-19 severity phenotypes exhibited a significant causal effect on eclampsia.ConclusionsOur investigation demonstrates a suggestive causal effect of genetic susceptibility to critical ill COVID-19 and hospitalized COVID-19 on pre-eclampsia. The COVID-19 severity exhibited a suggestive positive dose–response relationship with the risk of pre-eclampsia. Augmented attention should be paid to pregnant women hospitalized for COVID-19, especially those needing respiratory support

InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation

This paper introduces InternVid, a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for multimodal understanding and generation. The InternVid dataset contains over 7 million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of total 4.1B words. Our core contribution is to develop a scalable approach to autonomously build a high-quality video-text dataset with large language models (LLM), thereby showcasing its efficacy in learning video-language representation at scale. Specifically, we utilize a multi-scale approach to generate video-related descriptions. Furthermore, we introduce ViCLIP, a video-text representation learning model based on ViT-L. Learned on InternVid via contrastive learning, this model demonstrates leading zero-shot action recognition and competitive video retrieval performance. Beyond basic video understanding tasks like recognition and retrieval, our dataset and model have broad applications. They are particularly beneficial for generating interleaved video-text data for learning a video-centric dialogue system, advancing video-to-text and text-to-video generation research. These proposed resources provide a tool for researchers and practitioners interested in multimodal video understanding and generation.Comment: Data and Code: https://github.com/OpenGVLab/InternVideo/tree/main/Data/InternVi

Expert Consensus on Microtransplant for Acute Myeloid Leukemia in Elderly Patients -Report From the International Microtransplant Interest Group

Recent studies have shown that microtransplant (MST) could improve outcome of patients with elderly acute myeloid leukemia (EAML). To further standardize the MST therapy and improve outcomes in EAML patients, based on analysis of the literature on MST, especially MST with EAML from January 1st, 2011 to November 30th, 2022, the International Microtransplant Interest Group provides recommendations and considerations for MST in the treatment of EAML. Four major issues related to MST for treating EAML were addressed: therapeutic principle of MST (1), candidates for MST (2), induction chemotherapy regimens (3), and post-remission therapy based on MST (4). Others included donor screening, infusion of donor cells, laboratory examinations, and complications of treatment

A Novel Address-Matching Framework Based on Region Proposal

Geocoding is a fundamental component of geographic information science that plays a crucial role in various geographical studies and applications involving text data. Current mainstream geocoding methods fall into two categories: geodesic-grid prediction and address matching. However, the geodesic-grid-prediction method’s localization accuracy is hindered by the density of grid partitioning, struggling to strike a balance between prediction accuracy and grid density. Address-matching methods mainly focus on the semantics of query text. However, they tend to ignore keyword information that can be used to distinguish candidates and introduce potential interference, which reduces matching accuracy. Inspired by the human map-usage process, we propose a two-stage address-matching approach that integrates geodesic-grid prediction and text-matching models. Initially, a multi-level text-classification model is used to generate a retrieval region proposal for an input query text. Subsequently, we search for the most relevant point of interest (POI) within the region-proposal area using a semantics-based text-retrieval model. We evaluated the proposed method using POI data from the Beijing Chaoyang District. The experimental results indicate that the proposed method provides high address-matching accuracy, increasing Recall@1 by 0.55 to 1.56 percentage points and MRR@5 by 0.54 to 1.68 percentage points

Detection of Genetic Mutations by Next-Generation Sequencing for Predicting Prognosis of Extensive-Stage Small-Cell Lung Cancer

Some studies have revealed that specific genetic mutations could be associated with chemotherapy response or even survival in small-cell lung cancer (SCLC). Our retrospective study aimed to identify the correlation between genetic mutations and progression-free survival (PFS) in extensive-stage SCLC after first-line chemotherapy. A total of 75 patients with extensive-stage SCLC confirmed by histopathology from February 2018 to February 2019 were retrospectively analyzed. The biopsy specimens of all patients were analyzed by Next-Generation Sequencing (NGS). All patients received first-line chemotherapy and follow-up at Shanghai Chest Hospital. Eleven genes were mutated in, at least, 10% of the 75 patients, including TP53 (96%), RB1 (77%), SMAD4 (32%), NOTCH1 (21%), PTEN (16%), FGFR1 (16%), KDR (15%), PIK3CA (15%), ROS1 (15%), BRCA2 (13%), and ERBB4 (10%). The median number of mutated genes among all patients was 5. Patients with more than 5 mutated genes (PFS = 6.7 months, P=0.004), mutant TP53 (PFS = 5.0 months, P=0.011), and mutant BRCA2 (PFS = 6.7 months, P=0.046) had better PFS after first-line chemotherapy than other patients. Multivariate Cox regression analysis showed that patients who achieved a PR (HR 3.729, 95% CI 2.038–6.822), had more than 5 mutated genes (HR 1.929, 95% CI 1.096–3.396), had BRCA2 mutations (HR 4.581, 95% CI 1.721–12.195), and had no liver metastasis (HR 0.415, 95% CI 0.181–0.951) showed improvements in PFS after first-line chemotherapy. In conclusion, the number of mutated genes and BRCA2 mutation status in extensive-stage SCLC were significantly related to PFS after first-line chemotherapy

VideoChat: Chat-Centric Video Understanding

In this study, we initiate an exploration into video understanding by introducing VideoChat, an end-to-end chat-centric video understanding system. It integrates video foundation models and large language models via a learnable neural interface, excelling in spatiotemporal reasoning, event localization, and causal relationship inference. To instructively tune this system, we propose a video-centric instruction dataset, composed of thousands of videos matched with detailed descriptions and conversations. This dataset emphasizes spatiotemporal reasoning and causal relationships, providing a valuable asset for training chat-centric video understanding systems. Preliminary qualitative experiments reveal our system's potential across a broad spectrum of video applications and set the standard for future research. Access our code and data at https://github.com/OpenGVLab/Ask-AnythingComment: Technical repor

Table3_Mendelian randomization supports genetic liability to hospitalization for COVID-19 as a risk factor of pre-eclampsia.xlsx

BackgroundPre-eclampsia and eclampsia are among the major threats to pregnant women and fetuses, but they can be mitigated by prevention and early screening. Existing observational research presents conflicting evidence regarding the causal effects of coronavirus disease 2019 (COVID-19) on pre-eclampsia risk. Through Mendelian randomization (MR), this study aims to investigate the causal effect of three COVID-19 severity phenotypes on the risk of pre-eclampsia and eclampsia to provide more rigorous evidence.MethodsTwo-sample MR was utilized to examine causal effects. Summary-level data from genome-wide association studies (GWAS) of individuals of European ancestry were acquired from the GWAS catalog and FinnGen databases. Single-nucleotide polymorphisms associated with COVID-19 traits at p −8 were obtained and pruned for linkage disequilibrium to generate instrumental variables for COVID-19. Inverse variance weighted estimates were used as the primary MR results, with weighted median and MR-Egger as auxiliary analyses. The robustness of the MR findings was also evaluated through sensitivity analyses. Bonferroni correction was applied to primary results, with a p ResultsCritical ill COVID-19 [defined as hospitalization for COVID-19 with either a death outcome or respiratory support, OR (95% CI): 1.17 (1.03–1.33), p = 0.020] and hospitalized COVID-19 [defined as hospitalization for COVID-19, OR (95% CI): 1.10 (1.01–1.19), p = 0.026] demonstrated suggestive causal effects on pre-eclampsia, while general severe acute respiratory syndrome coronavirus 2 infection did not exhibit a significant causal effect on pre-eclampsia. None of the three COVID-19 severity phenotypes exhibited a significant causal effect on eclampsia.ConclusionsOur investigation demonstrates a suggestive causal effect of genetic susceptibility to critical ill COVID-19 and hospitalized COVID-19 on pre-eclampsia. The COVID-19 severity exhibited a suggestive positive dose–response relationship with the risk of pre-eclampsia. Augmented attention should be paid to pregnant women hospitalized for COVID-19, especially those needing respiratory support.</p

Table2_Mendelian randomization supports genetic liability to hospitalization for COVID-19 as a risk factor of pre-eclampsia.xlsx

BackgroundPre-eclampsia and eclampsia are among the major threats to pregnant women and fetuses, but they can be mitigated by prevention and early screening. Existing observational research presents conflicting evidence regarding the causal effects of coronavirus disease 2019 (COVID-19) on pre-eclampsia risk. Through Mendelian randomization (MR), this study aims to investigate the causal effect of three COVID-19 severity phenotypes on the risk of pre-eclampsia and eclampsia to provide more rigorous evidence.MethodsTwo-sample MR was utilized to examine causal effects. Summary-level data from genome-wide association studies (GWAS) of individuals of European ancestry were acquired from the GWAS catalog and FinnGen databases. Single-nucleotide polymorphisms associated with COVID-19 traits at p −8 were obtained and pruned for linkage disequilibrium to generate instrumental variables for COVID-19. Inverse variance weighted estimates were used as the primary MR results, with weighted median and MR-Egger as auxiliary analyses. The robustness of the MR findings was also evaluated through sensitivity analyses. Bonferroni correction was applied to primary results, with a p ResultsCritical ill COVID-19 [defined as hospitalization for COVID-19 with either a death outcome or respiratory support, OR (95% CI): 1.17 (1.03–1.33), p = 0.020] and hospitalized COVID-19 [defined as hospitalization for COVID-19, OR (95% CI): 1.10 (1.01–1.19), p = 0.026] demonstrated suggestive causal effects on pre-eclampsia, while general severe acute respiratory syndrome coronavirus 2 infection did not exhibit a significant causal effect on pre-eclampsia. None of the three COVID-19 severity phenotypes exhibited a significant causal effect on eclampsia.ConclusionsOur investigation demonstrates a suggestive causal effect of genetic susceptibility to critical ill COVID-19 and hospitalized COVID-19 on pre-eclampsia. The COVID-19 severity exhibited a suggestive positive dose–response relationship with the risk of pre-eclampsia. Augmented attention should be paid to pregnant women hospitalized for COVID-19, especially those needing respiratory support.</p