The dynamic response of the legged robot locomotion is non-Lipschitz and can
be stochastic due to environmental uncertainties. To test, validate, and
characterize the safety performance of legged robots, existing solutions on
observed and inferred risk can be incomplete and sampling inefficient. Some
formal verification methods suffer from the model precision and other surrogate
assumptions. In this paper, we propose a scenario sampling based testing
framework that characterizes the overall safety performance of a legged robot
by specifying (i) where (in terms of a set of states) the robot is potentially
safe, and (ii) how safe the robot is within the specified set. The framework
can also help certify the commercial deployment of the legged robot in
real-world environment along with human and compare safety performance among
legged robots with different mechanical structures and dynamic properties. The
proposed framework is further deployed to evaluate a group of state-of-the-art
legged robot locomotion controllers from various model-based, deep neural
network involved, and reinforcement learning based methods in the literature.
Among a series of intended work domains of the studied legged robots (e.g.
tracking speed on sloped surface, with abrupt changes on demanded velocity, and
against adversarial push-over disturbances), we show that the method can
adequately capture the overall safety characterization and the subtle
performance insights. Many of the observed safety outcomes, to the best of our
knowledge, have never been reported by the existing work in the legged robot
literature