85 research outputs found
Rethinking Implicit Neural Representations for Vision Learners
Implicit Neural Representations (INRs) are powerful to parameterize
continuous signals in computer vision. However, almost all INRs methods are
limited to low-level tasks, e.g., image/video compression, super-resolution,
and image generation. The questions on how to explore INRs to high-level tasks
and deep networks are still under-explored. Existing INRs methods suffer from
two problems: 1) narrow theoretical definitions of INRs are inapplicable to
high-level tasks; 2) lack of representation capabilities to deep networks.
Motivated by the above facts, we reformulate the definitions of INRs from a
novel perspective and propose an innovative Implicit Neural Representation
Network (INRN), which is the first study of INRs to tackle both low-level and
high-level tasks. Specifically, we present three key designs for basic blocks
in INRN along with two different stacking ways and corresponding loss
functions. Extensive experiments with analysis on both low-level tasks (image
fitting) and high-level vision tasks (image classification, object detection,
instance segmentation) demonstrate the effectiveness of the proposed method
Implicit Feature Networks for Texture Completion from Partial 3D Data
Prior work to infer 3D texture use either texture atlases, which require
uv-mappings and hence have discontinuities, or colored voxels, which are memory
inefficient and limited in resolution. Recent work, predicts RGB color at every
XYZ coordinate forming a texture field, but focus on completing texture given a
single 2D image. Instead, we focus on 3D texture and geometry completion from
partial and incomplete 3D scans. IF-Nets have recently achieved
state-of-the-art results on 3D geometry completion using a multi-scale deep
feature encoding, but the outputs lack texture. In this work, we generalize
IF-Nets to texture completion from partial textured scans of humans and
arbitrary objects. Our key insight is that 3D texture completion benefits from
incorporating local and global deep features extracted from both the 3D partial
texture and completed geometry. Specifically, given the partial 3D texture and
the 3D geometry completed with IF-Nets, our model successfully in-paints the
missing texture parts in consistence with the completed geometry. Our model won
the SHARP ECCV'20 challenge, achieving highest performance on all challenges.Comment: SHARP Workshop, European Conference on Computer Vision (ECCV), 202
Dual Arbitrary Scale Super-Resolution for Multi-Contrast MRI
Limited by imaging systems, the reconstruction of Magnetic Resonance Imaging
(MRI) images from partial measurement is essential to medical imaging research.
Benefiting from the diverse and complementary information of multi-contrast MR
images in different imaging modalities, multi-contrast Super-Resolution (SR)
reconstruction is promising to yield SR images with higher quality. In the
medical scenario, to fully visualize the lesion, radiologists are accustomed to
zooming the MR images at arbitrary scales rather than using a fixed scale, as
used by most MRI SR methods. In addition, existing multi-contrast MRI SR
methods often require a fixed resolution for the reference image, which makes
acquiring reference images difficult and imposes limitations on arbitrary scale
SR tasks. To address these issues, we proposed an implicit neural
representations based dual-arbitrary multi-contrast MRI super-resolution
method, called Dual-ArbNet. First, we decouple the resolution of the target and
reference images by a feature encoder, enabling the network to input target and
reference images at arbitrary scales. Then, an implicit fusion decoder fuses
the multi-contrast features and uses an Implicit Decoding Function~(IDF) to
obtain the final MRI SR results. Furthermore, we introduce a curriculum
learning strategy to train our network, which improves the generalization and
performance of our Dual-ArbNet. Extensive experiments in two public MRI
datasets demonstrate that our method outperforms state-of-the-art approaches
under different scale factors and has great potential in clinical practice.Comment: Accepted by MICCAI202
MeshfreeFlowNet: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework
We propose MeshfreeFlowNet, a novel deep learning-based super-resolution
framework to generate continuous (grid-free) spatio-temporal solutions from the
low-resolution inputs. While being computationally efficient, MeshfreeFlowNet
accurately recovers the fine-scale quantities of interest. MeshfreeFlowNet
allows for: (i) the output to be sampled at all spatio-temporal resolutions,
(ii) a set of Partial Differential Equation (PDE) constraints to be imposed,
and (iii) training on fixed-size inputs on arbitrarily sized spatio-temporal
domains owing to its fully convolutional encoder. We empirically study the
performance of MeshfreeFlowNet on the task of super-resolution of turbulent
flows in the Rayleigh-Benard convection problem. Across a diverse set of
evaluation metrics, we show that MeshfreeFlowNet significantly outperforms
existing baselines. Furthermore, we provide a large scale implementation of
MeshfreeFlowNet and show that it efficiently scales across large clusters,
achieving 96.80% scaling efficiency on up to 128 GPUs and a training time of
less than 4 minutes.Comment: Supplementary Video: https://youtu.be/mjqwPch9gDo. Accepted to SC2
SwIPE: Efficient and Robust Medical Image Segmentation with Implicit Patch Embeddings
Modern medical image segmentation methods primarily use discrete
representations in the form of rasterized masks to learn features and generate
predictions. Although effective, this paradigm is spatially inflexible, scales
poorly to higher-resolution images, and lacks direct understanding of object
shapes. To address these limitations, some recent works utilized implicit
neural representations (INRs) to learn continuous representations for
segmentation. However, these methods often directly adopted components designed
for 3D shape reconstruction. More importantly, these formulations were also
constrained to either point-based or global contexts, lacking contextual
understanding or local fine-grained details, respectively--both critical for
accurate segmentation. To remedy this, we propose a novel approach, SwIPE
(Segmentation with Implicit Patch Embeddings), that leverages the advantages of
INRs and predicts shapes at the patch level--rather than at the point level or
image level--to enable both accurate local boundary delineation and global
shape coherence. Extensive evaluations on two tasks (2D polyp segmentation and
3D abdominal organ segmentation) show that SwIPE significantly improves over
recent implicit approaches and outperforms state-of-the-art discrete methods
with over 10x fewer parameters. Our method also demonstrates superior data
efficiency and improved robustness to data shifts across image resolutions and
datasets. Code is available on Github.Comment: Accepted to 2023 International Conference on Medical Image Computing
and Computer Assisted Intervention (MICCAI'23
- …