7 research outputs found
Towards Visual Foundational Models of Physical Scenes
We describe a first step towards learning general-purpose visual
representations of physical scenes using only image prediction as a training
criterion. To do so, we first define "physical scene" and show that, even
though different agents may maintain different representations of the same
scene, the underlying physical scene that can be inferred is unique. Then, we
show that NeRFs cannot represent the physical scene, as they lack extrapolation
mechanisms. Those, however, could be provided by Diffusion Models, at least in
theory. To test this hypothesis empirically, NeRFs can be combined with
Diffusion Models, a process we refer to as NeRF Diffusion, used as unsupervised
representations of the physical scene. Our analysis is limited to visual data,
without external grounding mechanisms that can be provided by independent
sensory modalities.Comment: TLDR: Physical scenes are equivalence classes of sufficient
statistics, and can be inferred uniquely by any agent measuring the same
finite data; We formalize and implement an approach to representation
learning that overturns "naive realism" in favor of an analytical approach of
Russell and Koenderink. NeRFs cannot capture the physical scenes, but
combined with Diffusion Models they ca