FlightGoggles: A Modular Framework for Photorealistic Camera, Exteroceptive Sensor, and Dynamics Simulation

FlightGoggles is a photorealistic sensor simulator for perception-driven robotic vehicles. The key contributions of FlightGoggles are twofold. First, FlightGoggles provides photorealistic exteroceptive sensor simulation using graphics assets generated with photogrammetry. Second, it provides the ability to combine (i) synthetic exteroceptive measurements generated in silico in real time and (ii) vehicle dynamics and proprioceptive measurements generated in motio by vehicle(s) in a motion-capture facility. FlightGoggles is capable of simulating a virtual-reality environment around autonomous vehicle(s). While a vehicle is in flight in the FlightGoggles virtual reality environment, exteroceptive sensors are rendered synthetically in real time while all complex extrinsic dynamics are generated organically through the natural interactions of the vehicle. The FlightGoggles framework allows for researchers to accelerate development by circumventing the need to estimate complex and hard-to-model interactions such as aerodynamics, motor mechanics, battery electrochemistry, and behavior of other agents. The ability to perform vehicle-in-the-loop experiments with photorealistic exteroceptive sensor simulation facilitates novel research directions involving, e.g., fast and agile autonomous flight in obstacle-rich environments, safe human interaction, and flexible sensor selection. FlightGoggles has been utilized as the main test for selecting nine teams that will advance in the AlphaPilot autonomous drone racing challenge. We survey approaches and results from the top AlphaPilot teams, which may be of independent interest.Comment: Initial version appeared at IROS 2019. Supplementary material can be found at https://flightgoggles.mit.edu. Revision includes description of new FlightGoggles features, such as a photogrammetric model of the MIT Stata Center, new rendering settings, and a Python AP

An Extendable Multiagent Model for Behavioural Animation

This paper presents a framework for visually simulating the behaviour of actors in virtual environments. In principle, the environmental interaction follows a cyclic processing of perception, decision, and action. As natural life-forms perceive their environment by active sensing, our approach also tends to let the artificial actor actively sense the virtual world. This allows us to place the characters in non-preprocessed virtual dynamic environments, what we call generic environments. A main aspect within our framework is the strict distinction between a behaviour pattern, that we term model, and its instances, named characters, which use the pattern. This allows them sharing one or more behaviour models. Low-level tasks like sensing or acting are took over by so called subagents, which are subordinated modules extendedly plugged in the character. In a demonstration we exemplarily show the application of our framework. We place the same character in different environments and let it climb and descend stairs, ramps and hills autonomously. Additionally the reactiveness for moving objects is tested. In future, this approach shall go into action for a simulation of an urban environment

Gulliver project: performers and visitors

This paper discusses two projects in our research environment. The Gulliver project, an ambitious project conceived by some artists connected to our research efforts, and the Aveiro-project, as well ambitious, but with goals that can be achieved beause of technological developments, rather than be dependent on artistic and 'political' (read: financial) sources. Both projects are on virtual and augmented reality. The main goal is to design inhabited environments, where 'inhabited' refers to autonomous agents and agents that represent humans, realtime or off-line, visiting the virtual environment and interacting with other agents. The Gulliver environment has been designed by two artists: Matjaz Stuk and Alena Hudcovicova. The Aveiro project is a research effort of a group of researchers trying to design models of intelligence and interaction underlying the behavior of (groups of) agents inhabiting virtual worlds. In this paper we survey the current state of both projects and we discuss current and future attempts to have music performances by virtual and real performers in these environments

Business Case and Technology Analysis for 5G Low Latency Applications

A large number of new consumer and industrial applications are likely to change the classic operator's business models and provide a wide range of new markets to enter. This article analyses the most relevant 5G use cases that require ultra-low latency, from both technical and business perspectives. Low latency services pose challenging requirements to the network, and to fulfill them operators need to invest in costly changes in their network. In this sense, it is not clear whether such investments are going to be amortized with these new business models. In light of this, specific applications and requirements are described and the potential market benefits for operators are analysed. Conclusions show that operators have clear opportunities to add value and position themselves strongly with the increasing number of services to be provided by 5G.Comment: 18 pages, 5 figure

Agents for educational games and simulations

This book consists mainly of revised papers that were presented at the Agents for Educational Games and Simulation (AEGS) workshop held on May 2, 2011, as part of the Autonomous Agents and MultiAgent Systems (AAMAS) conference in Taipei, Taiwan. The 12 full papers presented were carefully reviewed and selected from various submissions. The papers are organized topical sections on middleware applications, dialogues and learning, adaption and convergence, and agent applications

Internet of robotic things : converging sensing/actuating, hypoconnectivity, artificial intelligence and IoT Platforms

The Internet of Things (IoT) concept is evolving rapidly and influencing newdevelopments in various application domains, such as the Internet of MobileThings (IoMT), Autonomous Internet of Things (A-IoT), Autonomous Systemof Things (ASoT), Internet of Autonomous Things (IoAT), Internetof Things Clouds (IoT-C) and the Internet of Robotic Things (IoRT) etc.that are progressing/advancing by using IoT technology. The IoT influencerepresents new development and deployment challenges in different areassuch as seamless platform integration, context based cognitive network integration,new mobile sensor/actuator network paradigms, things identification(addressing, naming in IoT) and dynamic things discoverability and manyothers. The IoRT represents new convergence challenges and their need to be addressed, in one side the programmability and the communication ofmultiple heterogeneous mobile/autonomous/robotic things for cooperating,their coordination, configuration, exchange of information, security, safetyand protection. Developments in IoT heterogeneous parallel processing/communication and dynamic systems based on parallelism and concurrencyrequire new ideas for integrating the intelligent “devices”, collaborativerobots (COBOTS), into IoT applications. Dynamic maintainability, selfhealing,self-repair of resources, changing resource state, (re-) configurationand context based IoT systems for service implementation and integrationwith IoT network service composition are of paramount importance whennew “cognitive devices” are becoming active participants in IoT applications.This chapter aims to be an overview of the IoRT concept, technologies,architectures and applications and to provide a comprehensive coverage offuture challenges, developments and applications

Towards virtual communities on the Web: Actors and audience

We report about ongoing research in a virtual reality environment where visitors can interact with agents that help them to obtain information, to perform certain transactions and to collaborate with them in order to get some tasks done. Our environment models a theatre in our hometown. We discuss attempts to let this environment evolve into a theatre community where we do not only have goal-directed visitors, but also visitors that that are not sure whether they want to buy or just want information or visitors who just want to look around. It is shown that we need a multi-user and multiagent environment to realize our goals. Since our environment models a theatre it is also interesting to investigate the roles of performers and audience in this environment. For that reason we discuss capabilities and personalities of agents. Some notes on the historical development of networked communities are included

In the Truman show: generating dynamic scenarios in a driving simulator

All the devices, animals, and people make their decisions based on what you're doing, but you don't know it or even notice it. Your world is that of Truman Burbank, from the 1998 movie The Truman Show. With this idea in mind, we've taken the movie metaphor to implement a prototype simulation system where the user steps into Truman's shoes. The set of our "movie" is a driving simulator, and the user is learning to drive a car. During the driving lessons, users drive in a virtual world that lets them experience all kinds of traffic scenarios. The system generates the scenarios with the student as the focal point, and the other traffic entities respond to the student's behavior, without the student noticing. To control the traffic scenarios and make them more effective, our prototype employs an agent-based framework. In this framework, each entity in the simulator is an actor agent playing a role. The prototype also includes a hierarchy of directors that directs the main action and the behind-the-scenes activity. The advantage of the movie metaphor is that it helps separate scenario description from scenario playing. The agents can read their required information from a script and perform their actions based on that information. Using this framework lets us build software that's extensible, maintainable, and easy to understan