530 research outputs found
On Quantizing Implicit Neural Representations
The role of quantization within implicit/coordinate neural networks is still
not fully understood. We note that using a canonical fixed quantization scheme
during training produces poor performance at low-rates due to the network
weight distributions changing over the course of training. In this work, we
show that a non-uniform quantization of neural weights can lead to significant
improvements. Specifically, we demonstrate that a clustered quantization
enables improved reconstruction. Finally, by characterising a trade-off between
quantization and network capacity, we demonstrate that it is possible (while
memory inefficient) to reconstruct signals using binary neural networks. We
demonstrate our findings experimentally on 2D image reconstruction and 3D
radiance fields; and show that simple quantization methods and architecture
search can achieve compression of NeRF to less than 16kb with minimal loss in
performance (323x smaller than the original NeRF).Comment: 10 pages, 10 figure
Compression with Bayesian Implicit Neural Representations
Many common types of data can be represented as functions that map
coordinates to signal values, such as pixel locations to RGB values in the case
of an image. Based on this view, data can be compressed by overfitting a
compact neural network to its functional representation and then encoding the
network weights. However, most current solutions for this are inefficient, as
quantization to low-bit precision substantially degrades the reconstruction
quality. To address this issue, we propose overfitting variational Bayesian
neural networks to the data and compressing an approximate posterior weight
sample using relative entropy coding instead of quantizing and entropy coding
it. This strategy enables direct optimization of the rate-distortion
performance by minimizing the -ELBO, and target different
rate-distortion trade-offs for a given network architecture by adjusting
. Moreover, we introduce an iterative algorithm for learning prior
weight distributions and employ a progressive refinement process for the
variational posterior that significantly enhances performance. Experiments show
that our method achieves strong performance on image and audio compression
while retaining simplicity.Comment: Preprin
Capturing dynamical correlations using implicit neural representations
The observation and description of collective excitations in solids is a
fundamental issue when seeking to understand the physics of a many-body system.
Analysis of these excitations is usually carried out by measuring the dynamical
structure factor, S(Q, ), with inelastic neutron or x-ray scattering
techniques and comparing this against a calculated dynamical model. Here, we
develop an artificial intelligence framework which combines a neural network
trained to mimic simulated data from a model Hamiltonian with automatic
differentiation to recover unknown parameters from experimental data. We
benchmark this approach on a Linear Spin Wave Theory (LSWT) simulator and
advanced inelastic neutron scattering data from the square-lattice spin-1
antiferromagnet LaNiO. We find that the model predicts the unknown
parameters with excellent agreement relative to analytical fitting. In doing
so, we illustrate the ability to build and train a differentiable model only
once, which then can be applied in real-time to multi-dimensional scattering
data, without the need for human-guided peak finding and fitting algorithms.
This prototypical approach promises a new technology for this field to
automatically detect and refine more advanced models for ordered quantum
systems.Comment: 12 pages, 7 figure
Implicit Neural Representations for Deformable Image Registration
Deformable medical image registration has in past years been revolutionized by the use of convolutional neural networks. These methods surpass conventional image registration techniques in speed but not in accuracy. Here, we present an alternative approach to leveraging neural networks for image registration. Instead of using a convolutional neural network to predict the transformation between images, we optimize a multi-layer perceptron to represent this transformation function. Using recent insights from differentiable rendering, we show how such an implicit deformable image registration (idir) model can be naturally combined with regularization terms based on standard automatic differentiation techniques. We demonstrate the effectiveness of this model on 4D chest CT registration in the DIR-LAB data set and find that a three-layer multi-layer perceptron with periodic activation functions outperforms all published deep learning-based results on this problem, without any folding and without the need for training data. The model is implemented using standard deep learning libraries and flexible enough to be extended to include different losses, regularizers, and optimization schemes.</p
Rethinking Implicit Neural Representations for Vision Learners
Implicit Neural Representations (INRs) are powerful to parameterize
continuous signals in computer vision. However, almost all INRs methods are
limited to low-level tasks, e.g., image/video compression, super-resolution,
and image generation. The questions on how to explore INRs to high-level tasks
and deep networks are still under-explored. Existing INRs methods suffer from
two problems: 1) narrow theoretical definitions of INRs are inapplicable to
high-level tasks; 2) lack of representation capabilities to deep networks.
Motivated by the above facts, we reformulate the definitions of INRs from a
novel perspective and propose an innovative Implicit Neural Representation
Network (INRN), which is the first study of INRs to tackle both low-level and
high-level tasks. Specifically, we present three key designs for basic blocks
in INRN along with two different stacking ways and corresponding loss
functions. Extensive experiments with analysis on both low-level tasks (image
fitting) and high-level vision tasks (image classification, object detection,
instance segmentation) demonstrate the effectiveness of the proposed method
VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations
Recent advancements in implicit neural representations have contributed to
high-fidelity surface reconstruction and photorealistic novel view synthesis.
However, the computational complexity inherent in these methodologies presents
a substantial impediment, constraining the attainable frame rates and
resolutions in practical applications. In response to this predicament, we
propose VQ-NeRF, an effective and efficient pipeline for enhancing implicit
neural representations via vector quantization. The essence of our method
involves reducing the sampling space of NeRF to a lower resolution and
subsequently reinstating it to the original size utilizing a pre-trained VAE
decoder, thereby effectively mitigating the sampling time bottleneck
encountered during rendering. Although the codebook furnishes representative
features, reconstructing fine texture details of the scene remains challenging
due to high compression rates. To overcome this constraint, we design an
innovative multi-scale NeRF sampling scheme that concurrently optimizes the
NeRF model at both compressed and original scales to enhance the network's
ability to preserve fine details. Furthermore, we incorporate a semantic loss
function to improve the geometric fidelity and semantic coherence of our 3D
reconstructions. Extensive experiments demonstrate the effectiveness of our
model in achieving the optimal trade-off between rendering quality and
efficiency. Evaluation on the DTU, BlendMVS, and H3DS datasets confirms the
superior performance of our approach.Comment: Submitted to the 38th Annual AAAI Conference on Artificial
Intelligenc
Continuous PDE Dynamics Forecasting with Implicit Neural Representations
Effective data-driven PDE forecasting methods often rely on fixed spatial and
/ or temporal discretizations. This raises limitations in real-world
applications like weather prediction where flexible extrapolation at arbitrary
spatiotemporal locations is required. We address this problem by introducing a
new data-driven approach, DINo, that models a PDE's flow with continuous-time
dynamics of spatially continuous functions. This is achieved by embedding
spatial observations independently of their discretization via Implicit Neural
Representations in a small latent space temporally driven by a learned ODE.
This separate and flexible treatment of time and space makes DINo the first
data-driven model to combine the following advantages. It extrapolates at
arbitrary spatial and temporal locations; it can learn from sparse irregular
grids or manifolds; at test time, it generalizes to new grids or resolutions.
DINo outperforms alternative neural PDE forecasters in a variety of challenging
generalization scenarios on representative PDE systems
- …