Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear

Developing embodied agents in simulation has been a key research topic in recent years. Exciting new tasks, algorithms, and benchmarks have been developed in various simulators. However, most of them assume deaf agents in silent environments, while we humans perceive the world with multiple senses. We introduce Sonicverse, a multisensory simulation platform with integrated audio-visual simulation for training household agents that can both see and hear. Sonicverse models realistic continuous audio rendering in 3D environments in real-time. Together with a new audio-visual VR interface that allows humans to interact with agents with audio, Sonicverse enables a series of embodied AI tasks that need audio-visual perception. For semantic audio-visual navigation in particular, we also propose a new multi-task learning model that achieves state-of-the-art performance. In addition, we demonstrate Sonicverse's realism via sim-to-real transfer, which has not been achieved by other simulators: an agent trained in Sonicverse can successfully perform audio-visual navigation in real-world environments. Sonicverse is available at: https://github.com/StanfordVL/Sonicverse.Comment: In ICRA 2023. Project page: https://ai.stanford.edu/~rhgao/sonicverse/. Code: https://github.com/StanfordVL/sonicverse. Gao and Li contributed equally to this work and are in alphabetical orde

Towards Jacamo-rest: A Resource-Oriented Abstraction for Managing Multi-Agent Systems

The Multi-Agent Oriented Programming (MAOP) paradigm provides abstractions to model and implements entities of agents, as well as of their organisations and environments. In recent years, researchers have started to explore the integration of MAOP and the resource-oriented web architecture (REST). This paper further advances this line of research by presenting an ongoing work on jacamo-rest, a resource-oriented web-based abstraction for the multi-agent programming platform JaCaMo. Jacamo-rest takes Multi-Agent System (MAS) interoperability to a new level, enabling MAS to not only interact with services or applications of the World Wide Web but also to be managed and updated in their specifications by other applications. To add a developer interface to JaCaMo that is suitable for the Web, we provide a novel conceptual perspective on the management of MAOP specification entities as web resources. We tested jacamo-rest using it as a middleware of a programming interface application that provides modern software engineering facilities such as continuous deployments and iterative software development for MAS.Comment: 11 pages, 5 figures, Accepted to present on 14th Workshop-School on Agents, Environments, and Applications (WESAAC 2020

A software toolkit for web-based virtual environments based on a shared database

We propose a software toolkit for developing complex web-based user interfaces, incorporating such things as multi-user facilities, virtual environments (VEs), and interface agents. The toolkit is based on a novel software architecture that combines ideas from multi-agent platforms and user interface (UI) architectures. It provides a distributed shared database with publish-subscribe facilities. This enables UI components to observe the state and activities of any other components in the system easily. The system runs in a web-based environment. The toolkit is comprised of several programming and other specification languages, providing a complete suite of systems design languages. We illustrate the toolkit by means of a couple of examples

Agents for educational games and simulations

This book consists mainly of revised papers that were presented at the Agents for Educational Games and Simulation (AEGS) workshop held on May 2, 2011, as part of the Autonomous Agents and MultiAgent Systems (AAMAS) conference in Taipei, Taiwan. The 12 full papers presented were carefully reviewed and selected from various submissions. The papers are organized topical sections on middleware applications, dialogues and learning, adaption and convergence, and agent applications

Towards Multi-Modal Interactions in Virtual Environments: A Case Study

We present research on visualization and interaction in a realistic model of an existing theatre. This existing ‘Muziek¬centrum’ offers its visitors information about performances by means of a yearly brochure. In addition, it is possible to get information at an information desk in the theatre (during office hours), to get information by phone (by talking to a human or by using IVR). The database of the theater holds the information that is available at the beginning of the ‘theatre season’. Our aim is to make this information more accessible by using multi-modal accessible multi-media web pages. A more general aim is to do research in the area of web-based services, in particu¬lar interactions in virtual environments

Reinforcement Learning for UAV Attitude Control

Autopilot systems are typically composed of an "inner loop" providing stability and control, while an "outer loop" is responsible for mission-level objectives, e.g. way-point navigation. Autopilot systems for UAVs are predominately implemented using Proportional, Integral Derivative (PID) control systems, which have demonstrated exceptional performance in stable environments. However more sophisticated control is required to operate in unpredictable, and harsh environments. Intelligent flight control systems is an active area of research addressing limitations of PID control most recently through the use of reinforcement learning (RL) which has had success in other applications such as robotics. However previous work has focused primarily on using RL at the mission-level controller. In this work, we investigate the performance and accuracy of the inner control loop providing attitude control when using intelligent flight control systems trained with the state-of-the-art RL algorithms, Deep Deterministic Gradient Policy (DDGP), Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO). To investigate these unknowns we first developed an open-source high-fidelity simulation environment to train a flight controller attitude control of a quadrotor through RL. We then use our environment to compare their performance to that of a PID controller to identify if using RL is appropriate in high-precision, time-critical flight control.Comment: 13 pages, 9 figure