Robo3D: Towards Robust and Reliable 3D Perception against Corruptions

The robustness of 3D perception systems under natural corruptions from environments and sensors is pivotal for safety-critical applications. Existing large-scale 3D perception datasets often contain data that are meticulously cleaned. Such configurations, however, cannot reflect the reliability of perception models during the deployment stage. In this work, we present Robo3D, the first comprehensive benchmark heading toward probing the robustness of 3D detectors and segmentors under out-of-distribution scenarios against natural corruptions that occur in real-world environments. Specifically, we consider eight corruption types stemming from adversarial weather conditions, external disturbances, and internal sensor failure. We uncover that, although promising results have been progressively achieved on standard benchmarks, state-of-the-art 3D perception models are at risk of being vulnerable to corruptions. We draw key observations on the use of data representations, augmentation schemes, and training strategies, that could severely affect the model's performance. To pursue better robustness, we propose a density-insensitive training framework along with a simple flexible voxelization strategy to enhance the model resiliency. We hope our benchmark and approach could inspire future research in designing more robust and reliable 3D perception models. Our robustness benchmark suite is publicly available.Comment: 33 pages, 26 figures, 26 tables; code at https://github.com/ldkong1205/Robo3D project page at https://ldkong.com/Robo3

LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusion

LiDAR-camera fusion methods have shown impressive performance in 3D object detection. Recent advanced multi-modal methods mainly perform global fusion, where image features and point cloud features are fused across the whole scene. Such practice lacks fine-grained region-level information, yielding suboptimal fusion performance. In this paper, we present the novel Local-to-Global fusion network (LoGoNet), which performs LiDAR-camera fusion at both local and global levels. Concretely, the Global Fusion (GoF) of LoGoNet is built upon previous literature, while we exclusively use point centroids to more precisely represent the position of voxel features, thus achieving better cross-modal alignment. As to the Local Fusion (LoF), we first divide each proposal into uniform grids and then project these grid centers to the images. The image features around the projected grid points are sampled to be fused with position-decorated point cloud features, maximally utilizing the rich contextual information around the proposals. The Feature Dynamic Aggregation (FDA) module is further proposed to achieve information interaction between these locally and globally fused features, thus producing more informative multi-modal features. Extensive experiments on both Waymo Open Dataset (WOD) and KITTI datasets show that LoGoNet outperforms all state-of-the-art 3D detection methods. Notably, LoGoNet ranks 1st on Waymo 3D object detection leaderboard and obtains 81.02 mAPH (L2) detection performance. It is noteworthy that, for the first time, the detection performance on three classes surpasses 80 APH (L2) simultaneously. Code will be available at \url{https://github.com/sankin97/LoGoNet}.Comment: Accepted by CVPR202

UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebase

Point-, voxel-, and range-views are three representative forms of point clouds. All of them have accurate 3D measurements but lack color and texture information. RGB images are a natural complement to these point cloud views and fully utilizing the comprehensive information of them benefits more robust perceptions. In this paper, we present a unified multi-modal LiDAR segmentation network, termed UniSeg, which leverages the information of RGB images and three views of the point cloud, and accomplishes semantic segmentation and panoptic segmentation simultaneously. Specifically, we first design the Learnable cross-Modal Association (LMA) module to automatically fuse voxel-view and range-view features with image features, which fully utilize the rich semantic information of images and are robust to calibration errors. Then, the enhanced voxel-view and range-view features are transformed to the point space,where three views of point cloud features are further fused adaptively by the Learnable cross-View Association module (LVA). Notably, UniSeg achieves promising results in three public benchmarks, i.e., SemanticKITTI, nuScenes, and Waymo Open Dataset (WOD); it ranks 1st on two challenges of two benchmarks, including the LiDAR semantic segmentation challenge of nuScenes and panoptic segmentation challenges of SemanticKITTI. Besides, we construct the OpenPCSeg codebase, which is the largest and most comprehensive outdoor LiDAR segmentation codebase. It contains most of the popular outdoor LiDAR segmentation algorithms and provides reproducible implementations. The OpenPCSeg codebase will be made publicly available at https://github.com/PJLab-ADG/PCSeg.Comment: ICCV 2023; 21 pages; 9 figures; 18 tables; Code at https://github.com/PJLab-ADG/PCSe

Wafer-scale heterogeneous integration InP on trenched Si with a bubble-free interface

Heterogeneous integration of compound semiconductors on a Si platform leads to advanced device applications in the field of Si photonics and high frequency electronics. However, the unavoidable bubbles formed at the bonding interface are detrimental for achieving a high yield of dissimilar semiconductor integration by the direct wafer bonding technology. In this work, lateral outgassing surface trenches (LOTs) are introduced to efficiently inhibit the bubbles. It is found that the chemical reactions in InP-Si bonding are similar to those in Si-Si bonding, and the generated gas can escape via the LOTs. The outgassing efficiency is dominated by LOTs\u27 spacing, and moreover, the relationship between bubble formation and the LOT\u27s structure is well described by a thermodynamic model. With the method explored in this work, a 2-in. bubble-free crystalline InP thin film integrated on the Si substrate with LOTs is obtained by the ion-slicing and wafer bonding technology. The quantum well active region grown on this Si-based InP film shows a superior photoemission efficiency, and it is found to be 65% as compared to its bulk counterpart

Susceptibilities of Yersinia pestis to Twelve Antimicrobial Agents in China

Streptomycin is the preferred choice for therapy of plague in China and other countries. However, Yersinia pestis exhibiting plasmid-mediated antimicrobial agent–resistant traits had been reported in Madagascar. In this study, we evaluated the susceptibility of traditional or newer antimicrobial agents used for treatment and/or prophylaxis of plague. Following Clinical and Laboratory Standards Institute (CLSI) recommendations, the susceptibility of 12 antimicrobial agents was evaluated by the agar microdilution method in 1,012 strains of Y. pestis isolated from 1943 to 2017 in 12 natural plague foci in China. One clinical Y. pestis isolate (S19960127) was found to be highly resistant to streptomycin, while the strain was still sensitive to other 11 antibiotics, that is, ciprofloxacin, ofloxacin, kanamycin, chloramphenicol, ampicillin, ceftriaxone, cefuroxime, trimethoprim-sulfamethoxazole, tetracycline, spectinomycin and moxifloxacin. The remaining 1,011 Y. pestis strains in this study demonstrated susceptibility to the above-mentioned 12 antimicrobial agents. Antimicrobial sensitivity surveillance of Y. pestis isolates, including dynamic monitoring of streptomycin resistance during various clinical plague treatments, should be carried out routinely

Histopathological Observation of Immunized Rhesus Macaques with Plague Vaccines after Subcutaneous Infection of Yersinia pestis

In our previous study, complete protection was observed in Chinese-origin rhesus macaques immunized with SV1 (20 µg F1 and 10 µg rV270) and SV2 (200 µg F1 and 100 µg rV270) subunit vaccines and with EV76 live attenuated vaccine against subcutaneous challenge with 6×106 CFU of Y. pestis. In the present study, we investigated whether the vaccines can effectively protect immunized animals from any pathologic changes using histological and immunohistochemical techniques. In addition, the glomerular basement membranes (GBMs) of the immunized animals and control animals were checked by electron microscopy. The results show no signs of histopathological lesions in the lungs, livers, kidneys, lymph nodes, spleens and hearts of the immunized animals at Day 14 after the challenge, whereas pathological alterations were seen in the corresponding tissues of the control animals. Giemsa staining, ultrastructural examination, and immunohistochemical staining revealed bacteria in some of the organs of the control animals, whereas no bacterium was observed among the immunized animals. Ultrastructural observation revealed that no glomerular immune deposits on the GBM. These observations suggest that the vaccines can effectively protect animals from any pathologic changes and eliminate Y. pestis from the immunized animals. The control animals died from multi-organ lesions specifically caused by the Y. pestis infection. We also found that subcutaneous infection of animals with Y. pestis results in bubonic plague, followed by pneumonic and septicemic plagues. The histopathologic features of plague in rhesus macaques closely resemble those of rodent and human plagues. Thus, Chinese-origin rhesus macaques serve as useful models in studying Y. pestis pathogenesis, host response and the efficacy of new medical countermeasures against plague

Reliability of three-dimensional pseudo-continuous arterial spin labeling MR imaging for measuring visual cortex perfusion on two 3T scanners.

Cerebral blood flow (CBF) in the human primary visual cortex is correlated with the loss of visual function in neuro-ophthalmological diseases. Advanced three-dimensional pseudo-continuous arterial spin labeling (3D pCASL), as a non-invasive method to access the CBF, can be a novel measurement to detect the visual cortex. The objective of the study was to assess the intra- and inter-scanner reliability of 3D pCASL of the visual cortex in healthy adults and suggest the selection of different post-labeling delay times (PLDs). For this reason, 3D pCASL was conducted in two 3.0T MR three times with twelve healthy volunteers at an interval of 10-15 days. The 1st and 3rd tests were performed on scanner-1, and the 2nd test was performed on scanner-2. The value of the CBF was abstracted from the visual cortex with two PLDs. The intra- and inter-scanner reliability and reproducibility were evaluated with the intraclass correlation coefficient (ICC) and Bland-Altman plots. By estimating the mean value of the CBF in the visual cortex, the intra-scanner results demonstrated the higher reliability (ICC for PLD = 1.5 second presented at 0.743 compared with 0.829 for PLD = 2.5 seconds), and the Bland-Altman plots showed the reproducibility at a longer PLD. We conclude that the calibrated 3D pCASL approach provides a highly reproducible measurement of the CBF of the visual cortex that can serve as a useful quantitative probe for research conducted at multiple centers and for the long-term observation of the clinical effects of neuro-opthalmological diseases

The separated ICC values of the 6 ROIs for the intra- and inter-scanner using 3DpCASL.

<p>The separated ICC values of the 6 ROIs for the intra- and inter-scanner using 3DpCASL.</p