Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training

Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for both 2D and 3D computer vision. However, existing MAE-style methods can only learn from the data of a single modality, i.e., either images or point clouds, which neglect the implicit semantic and geometric correlation between 2D and 3D. In this paper, we explore how the 2D modality can benefit 3D masked autoencoding, and propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training. Joint-MAE randomly masks an input 3D point cloud and its projected 2D images, and then reconstructs the masked information of the two modalities. For better cross-modal interaction, we construct our JointMAE by two hierarchical 2D-3D embedding modules, a joint encoder, and a joint decoder with modal-shared and model-specific decoders. On top of this, we further introduce two cross-modal strategies to boost the 3D representation learning, which are local-aligned attention mechanisms for 2D-3D semantic cues, and a cross-reconstruction loss for 2D-3D geometric constraints. By our pre-training paradigm, Joint-MAE achieves superior performance on multiple downstream tasks, e.g., 92.4% accuracy for linear SVM on ModelNet40 and 86.07% accuracy on the hardest split of ScanObjectNN.Comment: Accepted by IJCAI 202

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts

Contrastive Language-Image Pre-training (CLIP) has drawn increasing attention recently for its transferable visual representation learning. However, due to the semantic gap within datasets, CLIP's pre-trained image-text alignment becomes sub-optimal on downstream tasks, which severely harms its transferring performance. To better adapt the cross-modality embedding space, we propose to enhance CLIP via Visual-guided Texts, named VT-CLIP. Specifically, we guide textual features of different categories to adaptively explore informative regions on the image and aggregate visual features by attention mechanisms. In this way, the texts become visual-guided, namely, more semantically correlated with downstream images, which greatly benefits the category-wise matching process. In few-shot settings, we evaluate our VT-CLIP on 11 well-known classification datasets to demonstrate its effectiveness

Posterior Sampling for Competitive RL: Function Approximation and Partial Observation

This paper investigates posterior sampling algorithms for competitive reinforcement learning (RL) in the context of general function approximations. Focusing on zero-sum Markov games (MGs) under two critical settings, namely self-play and adversarial learning, we first propose the self-play and adversarial generalized eluder coefficient (GEC) as complexity measures for function approximation, capturing the exploration-exploitation trade-off in MGs. Based on self-play GEC, we propose a model-based self-play posterior sampling method to control both players to learn Nash equilibrium, which can successfully handle the partial observability of states. Furthermore, we identify a set of partially observable MG models fitting MG learning with the adversarial policies of the opponent. Incorporating the adversarial GEC, we propose a model-based posterior sampling method for learning adversarial MG with potential partial observability. We further provide low regret bounds for proposed algorithms that can scale sublinearly with the proposed GEC and the number of episodes $T$. To the best of our knowledge, we for the first time develop generic model-based posterior sampling algorithms for competitive RL that can be applied to a majority of tractable zero-sum MG classes in both fully observable and partially observable MGs with self-play and adversarial learning.Comment: NeurIPS 202

An improved contrastive learning network for semi-supervised multi-structure segmentation in echocardiography

Cardiac diseases have high mortality rates and are a significant threat to human health. Echocardiography is a commonly used imaging technique to diagnose cardiac diseases because of its portability, non-invasiveness and low cost. Precise segmentation of basic cardiac structures is crucial for cardiologists to efficiently diagnose cardiac diseases, but this task is challenging due to several reasons, such as: (1) low image contrast, (2) incomplete structures of cardiac, and (3) unclear border between the ventricle and the atrium in some echocardiographic images. In this paper, we applied contrastive learning strategy and proposed a semi-supervised method for echocardiographic images segmentation. This proposed method solved the above challenges effectively and made use of unlabeled data to achieve a great performance, which could help doctors improve the accuracy of CVD diagnosis and screening. We evaluated this method on a public dataset (CAMUS), achieving mean Dice Similarity Coefficient (DSC) of 0.898, 0.911, 0.916 with 1/4, 1/2 and full labeled data on two-chamber (2CH) echocardiography images, and of 0.903, 0.921, 0.928 with 1/4, 1/2 and full labeled data on four-chamber (4CH) echocardiography images. Compared with other existing methods, the proposed method had fewer parameters and better performance. The code and models are available at https://github.com/gpgzy/CL-Cardiac-segmentation

Natural Coevolution of Tumor and Immunoenvironment in Glioblastoma.

Isocitrate dehydrogenase (IDH) wild-type glioblastoma (GBM) has a dismal prognosis. A better understanding of tumor evolution holds the key to developing more effective treatment. Here we study GBM\u27s natural evolutionary trajectory by using rare multifocal samples. We sequenced 61,062 single cells from eight multifocal IDH wild-type primary GBMs and defined a natural evolution signature (NES) of the tumor. We show that the NES significantly associates with the activation of transcription factors that regulate brain development, including MYBL2 and FOSL2. Hypoxia is involved in inducing NES transition potentially via activation of the HIF1A-FOSL2 axis. High-NES tumor cells could recruit and polarize bone marrow-derived macrophages through activation of the FOSL2-ANXA1-FPR1/3 axis. These polarized macrophages can efficiently suppress T-cell activity and accelerate NES transition in tumor cells. Moreover, the polarized macrophages could upregulate CCL2 to induce tumor cell migration. SIGNIFICANCE: GBM progression could be induced by hypoxia via the HIF1A-FOSL2 axis. Tumor-derived ANXA1 is associated with recruitment and polarization of bone marrow-derived macrophages to suppress the immunoenvironment. The polarized macrophages promote tumor cell NES transition and migration. This article is highlighted in the In This Issue feature, p. 2711

Boosting with an aerosolized Ad5-nCoV elicited robust immune responses in inactivated COVID-19 vaccines recipients

IntroductionThe SARS-CoV-2 Omicron variant has become the dominant SARS-CoV-2 variant and exhibits immune escape to current COVID-19 vaccines, the further boosting strategies are required.MethodsWe have conducted a non-randomized, open-label and parallel-controlled phase 4 trial to evaluate the magnitude and longevity of immune responses to booster vaccination with intramuscular adenovirus vectored vaccine (Ad5-nCoV), aerosolized Ad5-nCoV, a recombinant protein subunit vaccine (ZF2001) or homologous inactivated vaccine (CoronaVac) in those who received two doses of inactivated COVID-19 vaccines. ResultsThe aerosolized Ad5-nCoV induced the most robust and long-lasting neutralizing activity against Omicron variant and IFNg T-cell response among all the boosters, with a distinct mucosal immune response. SARS-CoV-2-specific mucosal IgA response was substantially generated in subjects boosted with the aerosolized Ad5-nCoV at day 14 post-vaccination. At month 6, participants boosted with the aerosolized Ad5-nCoV had remarkably higher median titer and seroconversion of the Omicron BA.4/5-specific neutralizing antibody than those who received other boosters. DiscussionOur findings suggest that aerosolized Ad5-nCoV may provide an efficient alternative in response to the spread of the Omicron BA.4/5 variant.Clinical trial registrationhttps://www.chictr.org.cn/showproj.html?proj=152729, identifier ChiCTR2200057278

Robust estimation of bacterial cell count from optical density

Optical density (OD) is widely used to estimate the density of cells in liquid culture, but cannot be compared between instruments without a standardized calibration protocol and is challenging to relate to actual cell count. We address this with an interlaboratory study comparing three simple, low-cost, and highly accessible OD calibration protocols across 244 laboratories, applied to eight strains of constitutive GFP-expressing E. coli. Based on our results, we recommend calibrating OD to estimated cell count using serial dilution of silica microspheres, which produces highly precise calibration (95.5% of residuals <1.2-fold), is easily assessed for quality control, also assesses instrument effective linear range, and can be combined with fluorescence calibration to obtain units of Molecules of Equivalent Fluorescein (MEFL) per cell, allowing direct comparison and data fusion with flow cytometry measurements: in our study, fluorescence per cell measurements showed only a 1.07-fold mean difference between plate reader and flow cytometry data