Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models

Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. For example, in a search and rescue mission, a legged robot could climb over debris, crawl through gaps, and navigate out of dead ends. However, the robot\u27s controller needs to respond intelligently to such varied obstacles, and this requires handling unexpected and unusual scenarios successfully. This presents an open challenge to current learning methods, which often struggle with generalization to the long tail of unexpected situations without heavy human supervision. To address this issue, we investigate how to leverage the broad knowledge about the structure of the world and commonsense reasoning capabilities of vision-language models (VLMs) to aid legged robots in handling difficult, ambiguous situations. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection with VLMs: (1) in-context adaptation over previous robot interactions and (2) planning multiple skills into the future and replanning. We evaluate VLM-PC on several challenging real-world obstacle courses, involving dead ends and climbing and crawling, on a Go1 quadruped robot. Our experiments show that by reasoning over the history of interactions and future plans, VLMs enable the robot to autonomously perceive, navigate, and act in a wide range of complex scenarios that would otherwise require environment-specific engineering or human guidance.27 page

A 9-Month Hubble Space Telescope Near-UV Survey of M87. I. Light and Color Curves of 94 Novae, and a Re-determination of the Nova Rate

M87 has been monitored with a cadence of 5 days over a 9 month-long span through the near-ultraviolet (NUV:F275W) and optical (F606W) filters of the Wide Field Camera 3 (WFC3) of the $\textit{Hubble Space Telescope}$. This unprecedented dataset yields the NUV and optical light and color curves of 94 M87 novae, characterizing the outburst and decline properties of the largest extragalactic nova dataset in the literature (after M31 and M81). We test and confirm nova modelers' prediction that recurrent novae cannot erupt more frequently that once every 45 days; show that there are zero rapidly recurring novae in the central $\sim$ 1/3 of M87 with recurrence times $<$ 130 days; demonstrate that novae closely follow the K-band light of M87 to within a few arcsec of the galaxy nucleus; show that nova NUV light curves are as heterogeneous as their optical counterparts, and usually peak 5 to 30 days after visible light maximum; determine our observations' annual detection completeness to be 71 - 77\%; and measure the rate Rnova of nova eruptions in M87 as $352_{-37}^{+37}$/yr. The corresponding luminosity-specific classical nova rate for this galaxy is $7.91_{-1.20}^{+1.20}/yr/10^{10}L_\odot,_{K}$. These rates confirm that ground-based observations of extragalactic novae miss most faint, fast novae and those near the centers of galaxies. An annual M87 nova rate of 300 or more seems inescapable. A luminosity-specific nova rate of $\sim$ $7 - 10/yr/10^{10}L_\odot,_{K}$ in ${\it all}$ types of galaxies is indicated by the data available in 2023.Comment: Accepted/In Press in ApJS; 3 Tables, 108 Figures, 180 page