FusionRCNN: LiDAR-Camera Fusion for Two-stage 3D Object Detection

3D object detection with multi-sensors is essential for an accurate and reliable perception system of autonomous driving and robotics. Existing 3D detectors significantly improve the accuracy by adopting a two-stage paradigm which merely relies on LiDAR point clouds for 3D proposal refinement. Though impressive, the sparsity of point clouds, especially for the points far away, making it difficult for the LiDAR-only refinement module to accurately recognize and locate objects.To address this problem, we propose a novel multi-modality two-stage approach named FusionRCNN, which effectively and efficiently fuses point clouds and camera images in the Regions of Interest(RoI). FusionRCNN adaptively integrates both sparse geometry information from LiDAR and dense texture information from camera in a unified attention mechanism. Specifically, it first utilizes RoIPooling to obtain an image set with a unified size and gets the point set by sampling raw points within proposals in the RoI extraction step; then leverages an intra-modality self-attention to enhance the domain-specific features, following by a well-designed cross-attention to fuse the information from two modalities.FusionRCNN is fundamentally plug-and-play and supports different one-stage methods with almost no architectural changes. Extensive experiments on KITTI and Waymo benchmarks demonstrate that our method significantly boosts the performances of popular detectors.Remarkably, FusionRCNN significantly improves the strong SECOND baseline by 6.14% mAP on Waymo, and outperforms competing two-stage approaches. Code will be released soon at https://github.com/xxlbigbrother/Fusion-RCNN.Comment: 7 pages, 3 figure

Deep Semantic Graph Matching for Large-scale Outdoor Point Clouds Registration

Current point cloud registration methods are mainly based on local geometric information and usually ignore the semantic information contained in the scenes. In this paper, we treat the point cloud registration problem as a semantic instance matching and registration task, and propose a deep semantic graph matching method (DeepSGM) for large-scale outdoor point cloud registration. Firstly, the semantic categorical labels of 3D points are obtained using a semantic segmentation network. The adjacent points with the same category labels are then clustered together using the Euclidean clustering algorithm to obtain the semantic instances, which are represented by three kinds of attributes including spatial location information, semantic categorical information, and global geometric shape information. Secondly, the semantic adjacency graph is constructed based on the spatial adjacency relations of semantic instances. To fully explore the topological structures between semantic instances in the same scene and across different scenes, the spatial distribution features and the semantic categorical features are learned with graph convolutional networks, and the global geometric shape features are learned with a PointNet-like network. These three kinds of features are further enhanced with the self-attention and cross-attention mechanisms. Thirdly, the semantic instance matching is formulated as an optimal transport problem, and solved through an optimal matching layer. Finally, the geometric transformation matrix between two point clouds is first estimated by the SVD algorithm and then refined by the ICP algorithm. Experimental results conducted on the KITTI Odometry dataset demonstrate that the proposed method improves the registration performance and outperforms various state-of-the-art methods.Comment: 12 pages, 6 figure

CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds

We present a novel two-stage fully sparse convolutional 3D object detection framework, named CAGroup3D. Our proposed method first generates some high-quality 3D proposals by leveraging the class-aware local group strategy on the object surface voxels with the same semantic predictions, which considers semantic consistency and diverse locality abandoned in previous bottom-up approaches. Then, to recover the features of missed voxels due to incorrect voxel-wise segmentation, we build a fully sparse convolutional RoI pooling module to directly aggregate fine-grained spatial information from backbone for further proposal refinement. It is memory-and-computation efficient and can better encode the geometry-specific features of each 3D proposal. Our model achieves state-of-the-art 3D detection performance with remarkable gains of +\textit{3.6\%} on ScanNet V2 and +\textit{2.6}\% on SUN RGB-D in term of [email protected]. Code will be available at https://github.com/Haiyang-W/CAGroup3D.Comment: Accept by NeurIPS202

Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions

Robust 3D perception under corruption has become an essential task for the realm of 3D vision. While current data augmentation techniques usually perform random transformations on all point cloud objects in an offline way and ignore the structure of the samples, resulting in over-or-under enhancement. In this work, we propose an alternative to make sample-adaptive transformations based on the structure of the sample to cope with potential corruption via an auto-augmentation framework, named as AdaptPoint. Specially, we leverage a imitator, consisting of a Deformation Controller and a Mask Controller, respectively in charge of predicting deformation parameters and producing a per-point mask, based on the intrinsic structural information of the input point cloud, and then conduct corruption simulations on top. Then a discriminator is utilized to prevent the generation of excessive corruption that deviates from the original data distribution. In addition, a perception-guidance feedback mechanism is incorporated to guide the generation of samples with appropriate difficulty level. Furthermore, to address the paucity of real-world corrupted point cloud, we also introduce a new dataset ScanObjectNN-C, that exhibits greater similarity to actual data in real-world environments, especially when contrasted with preceding CAD datasets. Experiments show that our method achieves state-of-the-art results on multiple corruption benchmarks, including ModelNet-C, our ScanObjectNN-C, and ShapeNet-C.Comment: Accepted by ICCV2023; code: https://github.com/Roywangj/AdaptPoin

Case report: Sex-specific characteristics of epilepsy phenotypes associated with Xp22.31 deletion: A case report and review

Deletion in the Xp22.31 region is increasingly suggested to be involved in the etiology of epilepsy. Little is known regarding the genomic and clinical delineations of X-linked epilepsy in the Chinese population or the sex-stratified difference in epilepsy characteristics associated with deletions in the Xp22.31 region. In this study, we reported two siblings with a 1.69 Mb maternally inherited microdeletion at Xp22.31 involving the genes VCX3A, HDHD1, STS, VCX, VCX2, and PNPLA4 presenting with easily controlled focal epilepsy and language delay with mild ichthyosis in a Chinese family with a traceable 4-generation history of skin ichthyosis. Both brain magnetic resonance imaging results were normal, while EEG revealed epileptic abnormalities. We further performed an exhaustive literature search, documenting 25 patients with epilepsy with gene defects in Xp22.31, and summarized the epilepsy heterogeneities between sexes. Males harboring the Xp22.31 deletion mainly manifested with child-onset, easily controlled focal epilepsy accompanied by X-linked ichthyosis; the deletions were mostly X-linked recessive, with copy number variants (CNVs) in the classic region of deletion (863.38 kb–2 Mb). In contrast, epilepsy in females tended to be earlier-onset, and relatively refractory, with pathogenic CNV sizes varying over a larger range (859 kb–56.36 Mb); the alterations were infrequently inherited and almost combined with additional CNVs. A candidate region encompassing STS, HDHD1, and MIR4767 was the likely pathogenic epilepsy-associated region. This study filled in the knowledge gap regarding the genomic and clinical delineations of X-linked recessive epilepsy in the Chinese population and extends the understanding of the sex-specific characteristics of Xp22.31 deletion in regard to epilepsy

Chromosomal aberrations in pediatric patients with moderate/severe developmental delay/intellectual disability with abundant phenotypic heterogeneities: A single-center study

Background: This study aimed to examine the clinical usefulness of chromosome microarray (CMA) for selective implementation in patients with unexplained moderate or severe developmental delay/intellectual disability (DD/ID) and/or combined with different dysphonic features in the Han Chinese population. Methods: We retrospectively analyzed data on 122 pediatric patients with unexplained isolated moderate/severe DD/ID with or without autism spectrum disorders, epilepsy, dystonia, and congenital abnormalities from a single-center neurorehabilitation clinic in southern China. Results: A total of 46 probands (37.7%) had abnormal CMA results among the 122 study patients. With the exclusion of aneuploidies, uniparental disomies, and multiple homozygotes, 37 patients harbored 39 pathogenic copy number variations (pCNVs) (median [interquartile range] size: 3.57 [1.6 to 7.1] Mb; 33 deletions and 6 duplications), enriched in chromosomes 5, 7, 15, 17, and 22, with a markedly high prevalence of Angelman/Prader-Willi syndrome (24.3% [nine of 37]). Three rare deletions in the regions 5q33.2q34, 17p13.2, and 13q33.2 were reported, with specific delineation of clinical phenotypes. The frequencies of pCNVs were 18%, 33.3%, 38.89%, 41.67%, and 100% for patients with 1, 2, 3, 4, and 5 study phenotypes, respectively; patients with more concomitant abnormalities in the heart, brain, craniofacial region, and/or other organs had a higher CMA diagnostic yield and pCNV prevalence (P \u3c 0.05). Conclusions: Clinical application of CMA as a first-tier test among patients with moderate/severe DD/ID combined with congenital structural anomalies improved diagnostic yields and the quality of clinical management in this series of patients

Cre-loxP-mediated genetic lineage tracing: Unraveling cell fate and origin in the developing heart

The Cre-loxP-mediated genetic lineage tracing system is essential for constructing the fate mapping of single-cell progeny or cell populations. Understanding the structural hierarchy of cardiac progenitor cells facilitates unraveling cell fate and origin issues in cardiac development. Several prospective Cre-loxP-based lineage-tracing systems have been used to analyze precisely the fate determination and developmental characteristics of endocardial cells (ECs), epicardial cells, and cardiomyocytes. Therefore, emerging lineage-tracing techniques advance the study of cardiovascular-related cellular plasticity. In this review, we illustrate the principles and methods of the emerging Cre-loxP-based genetic lineage tracing technology for trajectory monitoring of distinct cell lineages in the heart. The comprehensive demonstration of the differentiation process of single-cell progeny using genetic lineage tracing technology has made outstanding contributions to cardiac development and homeostasis, providing new therapeutic strategies for tissue regeneration in congenital and cardiovascular diseases (CVDs)

Learning Point-wise Abstaining Penalty for Point Cloud Anomaly Detection

LiDAR-based semantic scene understanding is an important module in the modern autonomous driving perception stack. However, identifying Out-Of-Distribution (OOD) points in a LiDAR point cloud is challenging as point clouds lack semantically rich features when compared with RGB images. We revisit this problem from the perspective of selective classification, which introduces a selective function into the standard closed-set classification setup. Our solution is built upon the basic idea of abstaining from choosing any known categories but learns a point-wise abstaining penalty with a marginbased loss. Synthesizing outliers to approximate unlimited OOD samples is also critical to this idea, so we propose a strong synthesis pipeline that generates outliers originated from various factors: unrealistic object categories, sampling patterns and sizes. We demonstrate that learning different abstaining penalties, apart from point-wise penalty, for different types of (synthesized) outliers can further improve the performance. We benchmark our method on SemanticKITTI and nuScenes and achieve state-of-the-art results. Risk-coverage analysis further reveals intrinsic properties of different methods. Codes and models will be publicly available.Comment: codes is available at https://github.com/Daniellli/PAD.gi