70 research outputs found
360MonoDepth: High-Resolution 360° Monocular Depth Estimation
360{\deg} cameras can capture complete environments in a single shot, which
makes 360{\deg} imagery alluring in many computer vision tasks. However,
monocular depth estimation remains a challenge for 360{\deg} data, particularly
for high resolutions like 2K (2048x1024) and beyond that are important for
novel-view synthesis and virtual reality applications. Current CNN-based
methods do not support such high resolutions due to limited GPU memory. In this
work, we propose a flexible framework for monocular depth estimation from
high-resolution 360{\deg} images using tangent images. We project the 360{\deg}
input image onto a set of tangent planes that produce perspective views, which
are suitable for the latest, most accurate state-of-the-art perspective
monocular depth estimators. To achieve globally consistent disparity estimates,
we recombine the individual depth estimates using deformable multi-scale
alignment followed by gradient-domain blending. The result is a dense,
high-resolution 360{\deg} depth map with a high level of detail, also for
outdoor scenes which are not supported by existing methods. Our source code and
data are available at https://manurare.github.io/360monodepth/.Comment: CVPR 2022. Project page: https://manurare.github.io/360monodepth
OmniPhotos: Casual 360° VR Photography
Virtual reality headsets are becoming increasingly popular, yet it remains difficult for casual users to capture immersive 360° VR panoramas. State-of-the-art approaches require capture times of usually far more than a minute and are often limited in their supported range of head motion. We introduce OmniPhotos, a novel approach for quickly and casually capturing high-quality 360° panoramas with motion parallax. Our approach requires a single sweep with a consumer 360° video camera as input, which takes less than 3 seconds to capture with a rotating selfie stick or 10 seconds handheld. This is the fastest capture time for any VR photography approach supporting motion parallax by an order of magnitude. We improve the visual rendering quality of our OmniPhotos by alleviating vertical distortion using a novel deformable proxy geometry, which we fit to a sparse 3D reconstruction of captured scenes. In addition, the 360° input views significantly expand the available viewing area, and thus the range of motion, compared to previous approaches. We have captured more than 50 OmniPhotos and show video results for a large variety of scenes.This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 66599
A survey on human performance capture and animation
With the rapid development of computing technology, three-dimensional (3D) human body
models and their dynamic motions are widely used in the digital entertainment industry. Human perfor-
mance mainly involves human body shapes and motions. Key research problems include how to capture
and analyze static geometric appearance and dynamic movement of human bodies, and how to simulate
human body motions with physical e�ects. In this survey, according to main research directions of human body performance capture and animation, we summarize recent advances in key research topics, namely
human body surface reconstruction, motion capture and synthesis, as well as physics-based motion sim-
ulation, and further discuss future research problems and directions. We hope this will be helpful for
readers to have a comprehensive understanding of human performance capture and animatio
AdaTT: Adaptive Task-to-Task Fusion Network for Multitask Learning in Recommendations
Multi-task learning (MTL) aims at enhancing the performance and efficiency of
machine learning models by training them on multiple tasks simultaneously.
However, MTL research faces two challenges: 1) modeling the relationships
between tasks to effectively share knowledge between them, and 2) jointly
learning task-specific and shared knowledge. In this paper, we present a novel
model Adaptive Task-to-Task Fusion Network (AdaTT) to address both challenges.
AdaTT is a deep fusion network built with task specific and optional shared
fusion units at multiple levels. By leveraging a residual mechanism and gating
mechanism for task-to-task fusion, these units adaptively learn shared
knowledge and task specific knowledge. To evaluate the performance of AdaTT, we
conduct experiments on a public benchmark and an industrial recommendation
dataset using various task groups. Results demonstrate AdaTT can significantly
outperform existing state-of-the-art baselines
Cluster-Induced Mask Transformers for Effective Opportunistic Gastric Cancer Screening on Non-contrast CT Scans
Gastric cancer is the third leading cause of cancer-related mortality
worldwide, but no guideline-recommended screening test exists. Existing methods
can be invasive, expensive, and lack sensitivity to identify early-stage
gastric cancer. In this study, we explore the feasibility of using a deep
learning approach on non-contrast CT scans for gastric cancer detection. We
propose a novel cluster-induced Mask Transformer that jointly segments the
tumor and classifies abnormality in a multi-task manner. Our model incorporates
learnable clusters that encode the texture and shape prototypes of gastric
cancer, utilizing self- and cross-attention to interact with convolutional
features. In our experiments, the proposed method achieves a sensitivity of
85.0% and specificity of 92.6% for detecting gastric tumors on a hold-out test
set consisting of 100 patients with cancer and 148 normal. In comparison, two
radiologists have an average sensitivity of 73.5% and specificity of 84.3%. We
also obtain a specificity of 97.7% on an external test set with 903 normal
cases. Our approach performs comparably to established state-of-the-art gastric
cancer screening tools like blood testing and endoscopy, while also being more
sensitive in detecting early-stage cancer. This demonstrates the potential of
our approach as a novel, non-invasive, low-cost, and accurate method for
opportunistic gastric cancer screening.Comment: MICCAI 202
Improved Prognostic Prediction of Pancreatic Cancer Using Multi-Phase CT by Integrating Neural Distance and Texture-Aware Transformer
Pancreatic ductal adenocarcinoma (PDAC) is a highly lethal cancer in which
the tumor-vascular involvement greatly affects the resectability and, thus,
overall survival of patients. However, current prognostic prediction methods
fail to explicitly and accurately investigate relationships between the tumor
and nearby important vessels. This paper proposes a novel learnable neural
distance that describes the precise relationship between the tumor and vessels
in CT images of different patients, adopting it as a major feature for
prognosis prediction. Besides, different from existing models that used CNNs or
LSTMs to exploit tumor enhancement patterns on dynamic contrast-enhanced CT
imaging, we improved the extraction of dynamic tumor-related texture features
in multi-phase contrast-enhanced CT by fusing local and global features using
CNN and transformer modules, further enhancing the features extracted across
multi-phase CT images. We extensively evaluated and compared the proposed
method with existing methods in the multi-center (n=4) dataset with 1,070
patients with PDAC, and statistical analysis confirmed its clinical
effectiveness in the external test set consisting of three centers. The
developed risk marker was the strongest predictor of overall survival among
preoperative factors and it has the potential to be combined with established
clinical factors to select patients at higher risk who might benefit from
neoadjuvant therapy.Comment: MICCAI 202
CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans
Human readers or radiologists routinely perform full-body multi-organ
multi-disease detection and diagnosis in clinical practice, while most medical
AI systems are built to focus on single organs with a narrow list of a few
diseases. This might severely limit AI's clinical adoption. A certain number of
AI models need to be assembled non-trivially to match the diagnostic process of
a human reading a CT scan. In this paper, we construct a Unified Tumor
Transformer (CancerUniT) model to jointly detect tumor existence & location and
diagnose tumor characteristics for eight major cancers in CT scans. CancerUniT
is a query-based Mask Transformer model with the output of multi-tumor
prediction. We decouple the object queries into organ queries, tumor detection
queries and tumor diagnosis queries, and further establish hierarchical
relationships among the three groups. This clinically-inspired architecture
effectively assists inter- and intra-organ representation learning of tumors
and facilitates the resolution of these complex, anatomically related
multi-organ cancer image reading tasks. CancerUniT is trained end-to-end using
a curated large-scale CT images of 10,042 patients including eight major types
of cancers and occurring non-cancer tumors (all are pathology-confirmed with 3D
tumor masks annotated by radiologists). On the test set of 631 patients,
CancerUniT has demonstrated strong performance under a set of clinically
relevant evaluation metrics, substantially outperforming both multi-disease
methods and an assembly of eight single-organ expert models in tumor detection,
segmentation, and diagnosis. This moves one step closer towards a universal
high performance cancer screening tool.Comment: ICCV 2023 Camera Ready Versio
- …