Pre-training across 3D vision and language remains under development because
of limited training data. Recent works attempt to transfer vision-language
pre-training models to 3D vision. PointCLIP converts point cloud data to
multi-view depth maps, adopting CLIP for shape classification. However, its
performance is restricted by the domain gap between rendered depth maps and
images, as well as the diversity of depth distributions. To address this issue,
we propose CLIP2Point, an image-depth pre-training method by contrastive
learning to transfer CLIP to the 3D domain, and adapt it to point cloud
classification. We introduce a new depth rendering setting that forms a better
visual effect, and then render 52,460 pairs of images and depth maps from
ShapeNet for pre-training. The pre-training scheme of CLIP2Point combines
cross-modality learning to enforce the depth features for capturing expressive
visual and textual features and intra-modality learning to enhance the
invariance of depth aggregation. Additionally, we propose a novel Dual-Path
Adapter (DPA) module, i.e., a dual-path structure with simplified adapters for
few-shot learning. The dual-path structure allows the joint use of CLIP and
CLIP2Point, and the simplified adapter can well fit few-shot tasks without
post-search. Experimental results show that CLIP2Point is effective in
transferring CLIP knowledge to 3D vision. Our CLIP2Point outperforms PointCLIP
and other self-supervised 3D networks, achieving state-of-the-art results on
zero-shot and few-shot classification