Approaching human 3D shape perception with neurally mappable models

Humans effortlessly infer the 3D shape of objects. What computations underlie this ability? Although various computational models have been proposed, none of them capture the human ability to match object shape across viewpoints. Here, we ask whether and how this gap might be closed. We begin with a relatively novel class of computational models, 3D neural fields, which encapsulate the basic principles of classic analysis-by-synthesis in a deep neural network (DNN). First, we find that a 3D Light Field Network (3D-LFN) supports 3D matching judgments well aligned to humans for within-category comparisons, adversarially-defined comparisons that accentuate the 3D failure cases of standard DNN models, and adversarially-defined comparisons for algorithmically generated shapes with no category structure. We then investigate the source of the 3D-LFN's ability to achieve human-aligned performance through a series of computational experiments. Exposure to multiple viewpoints of objects during training and a multi-view learning objective are the primary factors behind model-human alignment; even conventional DNN architectures come much closer to human behavior when trained with multi-view objectives. Finally, we find that while the models trained with multi-view learning objectives are able to partially generalize to new object categories, they fall short of human alignment. This work provides a foundation for understanding human shape inferences within neurally mappable computational architectures and highlights important questions for future work

Computational ethics

Technological advances are enabling roles for machines that present novel ethical challenges. The study of 'AI ethics' has emerged to confront these challenges, and connects perspectives from philosophy, computer science, law, and economics. Less represented in these interdisciplinary efforts is the perspective of cognitive science. We propose a framework – computational ethics – that specifies how the ethical challenges of AI can be partially addressed by incorporating the study of human moral decision-making. The driver of this framework is a computational version of reflective equilibrium (RE), an approach that seeks coherence between considered judgments and governing principles. The framework has two goals: (i) to inform the engineering of ethical AI systems, and (ii) to characterize human moral judgment and decision-making in computational terms. Working jointly towards these two goals will create the opportunity to integrate diverse research questions, bring together multiple academic communities, uncover new interdisciplinary research topics, and shed light on centuries-old philosophical questions.publishedVersio

Computational ethics

Technological advances are enabling roles for machines that present novel ethical challenges. The study of 'AI ethics' has emerged to confront these challenges, and connects perspectives from philosophy, computer science, law, and economics. Less represented in these interdisciplinary efforts is the perspective of cognitive science. We propose a framework – computational ethics – that specifies how the ethical challenges of AI can be partially addressed by incorporating the study of human moral decision-making. The driver of this framework is a computational version of reflective equilibrium (RE), an approach that seeks coherence between considered judgments and governing principles. The framework has two goals: (i) to inform the engineering of ethical AI systems, and (ii) to characterize human moral judgment and decision-making in computational terms. Working jointly towards these two goals will create the opportunity to integrate diverse research questions, bring together multiple academic communities, uncover new interdisciplinary research topics, and shed light on centuries-old philosophical questions

Masses, radii, and orbits of small Kepler planets : The transition from gaseous to rocky planets

We report on the masses, sizes, and orbits of the planets orbiting 22 Kepler stars. There are 49 planet candidates around these stars, including 42 detected through transits and 7 revealed by precise Doppler measurements of the host stars. Based on an analysis of the Kepler brightness measurements, along with high-resolution imaging and spectroscopy, Doppler spectroscopy, and (for 11 stars) asteroseismology, we establish low false-positive probabilities (FPPs) for all of the transiting planets (41 of 42 have an FPP under 1%), and we constrain their sizes and masses. Most of the transiting planets are smaller than three times the size of Earth. For 16 planets, the Doppler signal was securely detected, providing a direct measurement of the planet's mass. For the other 26 planets we provide either marginal mass measurements or upper limits to their masses and densities; in many cases we can rule out a rocky composition. We identify six planets with densities above 5 g cm-3, suggesting a mostly rocky interior for them. Indeed, the only planets that are compatible with a purely rocky composition are smaller than 2 R ⊕. Larger planets evidently contain a larger fraction of low-density material (H, He, and H2O).Peer reviewedFinal Accepted Versio

Masses, radii, and orbits of small Kepler planets: the transition from gaseous to rocky planets

We report on the masses, sizes, and orbits of the planets orbiting 22 Kepler stars. There are 49 planet candidates around these stars, including 42 detected through transits and 7 revealed by precise Doppler measurements of the host stars. Based on an analysis of the Kepler brightness measurements, along with high-resolution imaging and spectroscopy, Doppler spectroscopy, and (for 11 stars) asteroseismology, we establish low false-positive probabilities (FPPs) for all of the transiting planets (41 of 42 have an FPP under 1%), and we constrain their sizes and masses. Most of the transiting planets are smaller than three times the size of Earth. For 16 planets, the Doppler signal was securely detected, providing a direct measurement of the planet's mass. For the other 26 planets we provide either marginal mass measurements or upper limits to their masses and densities; in many cases we can rule out a rocky composition. We identify six planets with densities above 5 g cm-3, suggesting a mostly rocky interior for them. Indeed, the only planets that are compatible with a purely rocky composition are smaller than 2 R ⊕. Larger planets evidently contain a larger fraction of low-density material (H, He, and H2O)

Generative Modeling of Audible Shapes for Object Perception

© 2017 IEEE. Humans infer rich knowledge of objects from both auditory and visual cues. Building a machine of such competency, however, is very challenging, due to the great difficulty in capturing large-scale, clean data of objects with both their appearance and the sound they make. In this paper, we present a novel, open-source pipeline that generates audiovisual data, purely from 3D object shapes and their physical properties. Through comparison with audio recordings and human behavioral studies, we validate the accuracy of the sounds it generates. Using this generative model, we are able to construct a synthetic audio-visual dataset, namely Sound-20K, for object perception tasks. We demonstrate that auditory and visual information play complementary roles in object perception, and further, that the representation learned on synthetic audio-visual data can transfer to real-world scenarios

Finding Fallen Objects Via Asynchronous Audio-Visual Integration

The way an object looks and sounds provide complementary reflections of its physical properties. In many settings cues from vision and audition arrive asynchronously but must be integrated, as when we hear an object dropped on the floor and then must find it. In this paper, we introduce a setting in which to study multi-modal object localization in 3D virtual environments. An object is dropped somewhere in a room. An embodied robot agent, equipped with a camera and microphone, must determine what object has been dropped -- and where -- by combining audio and visual signals with knowledge of the underlying physics. To study this problem, we have generated a large-scale dataset -- the Fallen Objects dataset -- that includes 8000 instances of 30 physical object categories in 64 rooms. The dataset uses the ThreeDWorld platform which can simulate physics-based impact sounds and complex physical interactions between objects in a photorealistic setting. As a first step toward addressing this challenge, we develop a set of embodied agent baselines, based on imitation learning, reinforcement learning, and modular planning, and perform an in-depth analysis of the challenge of this new task.Comment: CVPR 2022. Project page: http://fallen-object.csail.mit.ed