2D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervision

We present a Multimodal Interlaced Transformer (MIT) that jointly considers 2D and 3D data for weakly supervised point cloud segmentation. Research studies have shown that 2D and 3D features are complementary for point cloud segmentation. However, existing methods require extra 2D annotations to achieve 2D-3D information fusion. Considering the high annotation cost of point clouds, effective 2D and 3D feature fusion based on weakly supervised learning is in great demand. To this end, we propose a transformer model with two encoders and one decoder for weakly supervised point cloud segmentation using only scene-level class tags. Specifically, the two encoders compute the self-attended features for 3D point clouds and 2D multi-view images, respectively. The decoder implements interlaced 2D-3D cross-attention and carries out implicit 2D and 3D feature fusion. We alternately switch the roles of queries and key-value pairs in the decoder layers. It turns out that the 2D and 3D features are iteratively enriched by each other. Experiments show that it performs favorably against existing weakly supervised point cloud segmentation methods by a large margin on the S3DIS and ScanNet benchmarks. The project page will be available at https://jimmy15923.github.io/mit_web/.Comment: ICCV 2023 (main + supp). Website: https://jimmy15923.github.io/mit_web

Physically Plausible Animation of Human Upper Body from a Single Image

We present a new method for generating controllable, dynamically responsive, and photorealistic human animations. Given an image of a person, our system allows the user to generate Physically plausible Upper Body Animation (PUBA) using interaction in the image space, such as dragging their hand to various locations. We formulate a reinforcement learning problem to train a dynamic model that predicts the person's next 2D state (i.e., keypoints on the image) conditioned on a 3D action (i.e., joint torque), and a policy that outputs optimal actions to control the person to achieve desired goals. The dynamic model leverages the expressiveness of 3D simulation and the visual realism of 2D videos. PUBA generates 2D keypoint sequences that achieve task goals while being responsive to forceful perturbation. The sequences of keypoints are then translated by a pose-to-image generator to produce the final photorealistic video.Comment: WACV 202

Motor neuron-derived Thsd7a is essential for zebrafish vascular development via the Notch-dll4 signaling pathway.

BackgroundDevelopment of neural and vascular systems displays astonishing similarities among vertebrates. This parallelism is under a precise control of complex guidance signals and neurovascular interactions. Previously, our group identified a highly conserved neural protein called thrombospondin type I domain containing 7A (THSD7A). Soluble THSD7A promoted and guided endothelial cell migration, tube formation and sprouting. In addition, we showed that thsd7a could be detected in the nervous system and was required for intersegmental vessels (ISV) patterning during zebrafish development. However, the exact origin of THSD7A and its effect on neurovascular interaction remains unclear.ResultsIn this study, we discovered that zebrafish thsd7a was expressed in the primary motor neurons. Knockdown of Thsd7a disrupted normal primary motor neuron formation and ISV sprouting in the Tg(kdr:EGFP/mnx1:TagRFP) double transgenic zebrafish. Interestingly, we found that Thsd7a morphants displayed distinct phenotypes that are very similar to the loss of Notch-delta like 4 (dll4) signaling. Transcript profiling further revealed that expression levels of notch1b and its downstream targets, vegfr2/3 and nrarpb, were down-regulated in the Thsd7a morphants. These data supported that zebrafish Thsd7a could regulate angiogenic sprouting via Notch-dll4 signaling during development.ConclusionsOur results suggested that motor neuron-derived Thsd7a plays a significant role in neurovascular interactions. Thsd7a could regulate ISV angiogenesis via Notch-dll4 signaling. Thus, Thsd7a is a potent angioneurin involved in the development of both neural and vascular systems