Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

Model-free deep reinforcement learning algorithms have been shown to be capable of learning a wide range of robotic skills, but typically require a very large number of samples to achieve good performance. Model-based algorithms, in principle, can provide for much more efficient learning, but have proven difficult to extend to expressive, high-capacity models such as deep neural networks. In this work, we demonstrate that medium-sized neural network models can in fact be combined with model predictive control (MPC) to achieve excellent sample complexity in a model-based reinforcement learning algorithm, producing stable and plausible gaits to accomplish various complex locomotion tasks. We also propose using deep neural network dynamics models to initialize a model-free learner, in order to combine the sample efficiency of model-based approaches with the high task-specific performance of model-free methods. We empirically demonstrate on MuJoCo locomotion tasks that our pure model-based approach trained on just random action data can follow arbitrary trajectories with excellent sample efficiency, and that our hybrid algorithm can accelerate model-free learning on high-speed benchmark tasks, achieving sample efficiency gains of 3-5x on swimmer, cheetah, hopper, and ant agents. Videos can be found at https://sites.google.com/view/mbm

Learning Image-Conditioned Dynamics Models for Control of Under-actuated Legged Millirobots

Millirobots are a promising robotic platform for many applications due to their small size and low manufacturing costs. Legged millirobots, in particular, can provide increased mobility in complex environments and improved scaling of obstacles. However, controlling these small, highly dynamic, and underactuated legged systems is difficult. Hand-engineered controllers can sometimes control these legged millirobots, but they have difficulties with dynamic maneuvers and complex terrains. We present an approach for controlling a real-world legged millirobot that is based on learned neural network models. Using less than 17 minutes of data, our method can learn a predictive model of the robot's dynamics that can enable effective gaits to be synthesized on the fly for following user-specified waypoints on a given terrain. Furthermore, by leveraging expressive, high-capacity neural network models, our approach allows for these predictions to be directly conditioned on camera images, endowing the robot with the ability to predict how different terrains might affect its dynamics. This enables sample-efficient and effective learning for locomotion of a dynamic legged millirobot on various terrains, including gravel, turf, carpet, and styrofoam. Experiment videos can be found at https://sites.google.com/view/imageconddy

Detection of Slippery Terrain with a Heterogeneous Team of Legged Robots

Legged robots come in a range of sizes and capabilities. By combining these robots into heterogeneous teams, joint locomotion and perception tasks can be achieved by utilizing the diversified features of each robot. In this work we present a framework for using a heterogeneous team of legged robots to detect slippery terrain. StarlETH, a large and highly capable quadruped uses the VelociRoACH as a novel remote probe to detect regions of slippery terrain. StarlETH localizes the team using internal state estimation. To classify slippage of the VelociRoACH, we develop several Support Vector Machines (SVM) based on data from both StarlETH and VelociRoACH. By combining the team’s information about the motion of VelociRoACH, a classifier was built which could detect slippery spots with 92% (125/135) accuracy using only four features

NYMPH: A multiprocessor for manipulation applications

The robotics group of the Stanford Artificial Intelligence Laboratory is currently developing a new computational system for robotics applications. Stanford's NYMPH system uses multiple NSC 32016 processors and one MC68010 based processor, sharing a common Intel Multibus. The 32K processors provide the raw computational power needed for advanced robotics applications, and the 68K provides a pleasant interface with the rest of the world. Software has been developed to provide useful communications and synchronization primitives, without consuming excessive processor resources or bus bandwidth. NYMPH provides both large amounts of computing power and a good programming environment, making it an effective research tool

Wearable Microfluidic Diaphragm Pressure Sensor for Health and Tactile Touch Monitoring

Flexible pressure sensors have many potential applications in wearable electronics, robotics, health monitoring, and more. In particular, liquid-metal-based sensors are especially promising as they can undergo strains of over 200% without failure. However, current liquid-metal-based strain sensors are incapable of resolving small pressure changes in the few kPa range, making them unsuitable for applications such as heart-rate monitoring, which require a much lower pressure detection resolution. In this paper, a microfluidic tactile diaphragm pressure sensor based on embedded Galinstan microchannels (70 µm width × 70 µm height) capable of resolving sub-50 Pa changes in pressure with sub-100 Pa detection limits and a response time of 90 ms is demonstrated. An embedded equivalent Wheatstone bridge circuit makes the most of tangential and radial strain fields, leading to high sensitivities of a 0.0835 kPa^(−1) change in output voltage. The Wheatstone bridge also provides temperature self-compensation, allowing for operation in the range of 20–50 °C. As examples of potential applications, a polydimethylsiloxane (PDMS) wristband with an embedded microfluidic diaphragm pressure sensor capable of real-time pulse monitoring and a PDMS glove with multiple embedded sensors to provide comprehensive tactile feedback of a human hand when touching or holding objects are demonstrated

Wearable Microfluidic Diaphragm Pressure Sensor for Health and Tactile Touch Monitoring

Flexible pressure sensors have many potential applications in wearable electronics, robotics, health monitoring, and more. In particular, liquid-metal-based sensors are especially promising as they can undergo strains of over 200% without failure. However, current liquid-metal-based strain sensors are incapable of resolving small pressure changes in the few kPa range, making them unsuitable for applications such as heart-rate monitoring, which require a much lower pressure detection resolution. In this paper, a microfluidic tactile diaphragm pressure sensor based on embedded Galinstan microchannels (70 µm width × 70 µm height) capable of resolving sub-50 Pa changes in pressure with sub-100 Pa detection limits and a response time of 90 ms is demonstrated. An embedded equivalent Wheatstone bridge circuit makes the most of tangential and radial strain fields, leading to high sensitivities of a 0.0835 kPa^(−1) change in output voltage. The Wheatstone bridge also provides temperature self-compensation, allowing for operation in the range of 20–50 °C. As examples of potential applications, a polydimethylsiloxane (PDMS) wristband with an embedded microfluidic diaphragm pressure sensor capable of real-time pulse monitoring and a PDMS glove with multiple embedded sensors to provide comprehensive tactile feedback of a human hand when touching or holding objects are demonstrated