General intelligence requires rethinking exploration

We are at the cusp of a transition from 'learning from data' to 'learning what data to learn from' as a central focus of artificial intelligence (AI) research. While the first-order learning problem is not completely solved, large models under unified architectures, such as transformers, have shifted the learning bottleneck from how to effectively train models to how to effectively acquire and use task-relevant data. This problem, which we frame as exploration, is a universal aspect of learning in open-ended domains like the real world. Although the study of exploration in AI is largely limited to the field of reinforcement learning, we argue that exploration is essential to all learning systems, including supervised learning. We propose the problem of generalized exploration to conceptually unify exploration-driven learning between supervised learning and reinforcement learning, allowing us to highlight key similarities across learning settings and open research challenges. Importantly, generalized exploration is a necessary objective for maintaining open-ended learning processes, which in continually learning to discover and solve new problems, provides a promising path to more general intelligence

Agents for educational games and simulations

This book consists mainly of revised papers that were presented at the Agents for Educational Games and Simulation (AEGS) workshop held on May 2, 2011, as part of the Autonomous Agents and MultiAgent Systems (AAMAS) conference in Taipei, Taiwan. The 12 full papers presented were carefully reviewed and selected from various submissions. The papers are organized topical sections on middleware applications, dialogues and learning, adaption and convergence, and agent applications

Urban traffic modeling with microscopic approach using cellular automata

Zastoj prometa je jedan od najvećih problema prenapućenih gradova i potrebno ga je riješiti. U ovom se radu, primjenom celularnih automata (CA), uz znakove maksimalne brzine ispitivao učinak znakova minimalnog ograničenja brzine i njihova lokacija u prometu. Gradski je promet modeliran dvodimenzionalnim CA. Model uključuje prometne znakove, prometna svjetla i neke vrste vozila (automobili, furgoni, autobusi, metro autobusi) koja su česta u prometu.Traffic jam is one of the hardest problems of the crowded cities, and it needs to be solved. In this study, the effect of the minimum speed limit signs in addition to the maximum speed signs and their locations in traffic flow has been examined by using cellular automata (CA). Urban traffic is modeled by two dimensional CA. The model includes traffic signs, traffic lights and some kinds of vehicles (such as automobiles, vans, buses, metro buses) that are often encountered in traffic

Visual Attention in Dynamic Environments and its Application to Playing Online Games

Abstract In this thesis we present a prototype of Cognitive Programs (CPs) - an executive controller built on top of Selective Tuning (ST) model of attention. CPs enable top-down control of visual system and interaction between the low-level vision and higher-level task demands. Abstract We implement a subset of CPs for playing online video games in real time using only visual input. Two commercial closed-source games - Canabalt and Robot Unicorn Attack - are used for evaluation. Their simple gameplay and minimal controls put the emphasis on reaction speed and attention over planning. Abstract Our implementation of Cognitive Programs plays both games at human expert level, which experimentally proves the validity of the concept. Additionally we resolved multiple theoretical and engineering issues, e.g. extending the CPs to dynamic environments, finding suitable data structures for describing the task and information flow within the network and determining the correct timing for each process

Surgical Subtask Automation for Intraluminal Procedures using Deep Reinforcement Learning

Intraluminal procedures have opened up a new sub-field of minimally invasive surgery that use flexible instruments to navigate through complex luminal structures of the body, resulting in reduced invasiveness and improved patient benefits. One of the major challenges in this field is the accurate and precise control of the instrument inside the human body. Robotics has emerged as a promising solution to this problem. However, to achieve successful robotic intraluminal interventions, the control of the instrument needs to be automated to a large extent. The thesis first examines the state-of-the-art in intraluminal surgical robotics and identifies the key challenges in this field, which include the need for safe and effective tool manipulation, and the ability to adapt to unexpected changes in the luminal environment. To address these challenges, the thesis proposes several levels of autonomy that enable the robotic system to perform individual subtasks autonomously, while still allowing the surgeon to retain overall control of the procedure. The approach facilitates the development of specialized algorithms such as Deep Reinforcement Learning (DRL) for subtasks like navigation and tissue manipulation to produce robust surgical gestures. Additionally, the thesis proposes a safety framework that provides formal guarantees to prevent risky actions. The presented approaches are evaluated through a series of experiments using simulation and robotic platforms. The experiments demonstrate that subtask automation can improve the accuracy and efficiency of tool positioning and tissue manipulation, while also reducing the cognitive load on the surgeon. The results of this research have the potential to improve the reliability and safety of intraluminal surgical interventions, ultimately leading to better outcomes for patients and surgeons

Simulation and Learning for Urban Mobility: City-scale Traffic Reconstruction and Autonomous Driving

Traffic congestion has become one of the most critical issues worldwide. The costs due to traffic gridlock and jams are approximately $160 billion in the United States, more than £13 billion in the United Kingdom, and over one trillion dollars across the globe annually. As more metropolitan areas will experience increasingly severe traffic conditions, the ability to analyze, understand, and improve traffic dynamics becomes critical. This dissertation is an effort towards achieving such an ability. I propose various techniques combining simulation and machine learning to tackle the problem of traffic from two perspectives: city-scale traffic reconstruction and autonomous driving. Traffic, by its definition, appears in an aggregate form. In order to study it, we have to take a holistic approach. I address the problem of efficient and accurate estimation and reconstruction of city-scale traffic. The reconstructed traffic can be used to analyze congestion causes, identify network bottlenecks, and experiment with novel transport policies. City-scale traffic estimation and reconstruction have proven to be challenging for two particular reasons: first, traffic conditions that depend on individual drivers are intrinsically stochastic; second, the availability and quality of traffic data are limited. Traditional traffic monitoring systems that exist on highways and major roads can not produce sufficient data to recover traffic at scale. GPS data, in contrast, provide much broader coverage of a city thus are more promising sources for traffic estimation and reconstruction. However, GPS data are limited by their spatial-temporal sparsity in practice. I develop a framework to statically estimate and dynamically reconstruct traffic over a city-scale road network by addressing the limitations of GPS data. Traffic is also formed of individual vehicles propagating through space and time. If we can improve the efficiency of them, collectively, we can improve traffic dynamics as a whole. Recent advancements in automation and its implication for improving the safety and efficiency of the traffic system have prompted widespread research of autonomous driving. While exciting, autonomous driving is a complex task, consider the dynamics of an environment and the lack of accurate descriptions of a desired driving behavior. Learning a robust control policy for driving remains challenging as it requires an effective policy architecture, an efficient learning mechanism, and substantial training data covering a variety of scenarios, including rare cases such as accidents. I develop a framework, named ADAPS (Autonomous Driving via Principled Simulations), for producing robust control policies for autonomous driving. ADAPS consists of two simulation platforms which are used to generate and analyze simulated accidents while automatically generating labeled training data, and a hierarchical control policy which takes into account the features of driving behaviors and road conditions. ADAPS also represents a more efficient online learning mechanism compared to previous techniques, in which the number of iterations required to learn a robust control policy is reduced.Doctor of Philosoph

Search-based Test Generation for Automated Driving Systems: From Perception to Control Logic

abstract: Automated driving systems are in an intensive research and development stage, and the companies developing these systems are targeting to deploy them on public roads in a very near future. Guaranteeing safe operation of these systems is crucial as they are planned to carry passengers and share the road with other vehicles and pedestrians. Yet, there is no agreed-upon approach on how and in what detail those systems should be tested. Different organizations have different testing approaches, and one common approach is to combine simulation-based testing with real-world driving. One of the expectations from fully-automated vehicles is never to cause an accident. However, an automated vehicle may not be able to avoid all collisions, e.g., the collisions caused by other road occupants. Hence, it is important for the system designers to understand the boundary case scenarios where an autonomous vehicle can no longer avoid a collision. Besides safety, there are other expectations from automated vehicles such as comfortable driving and minimal fuel consumption. All safety and functional expectations from an automated driving system should be captured with a set of system requirements. It is challenging to create requirements that are unambiguous and usable for the design, testing, and evaluation of automated driving systems. Another challenge is to define useful metrics for assessing the testing quality because in general, it is impossible to test every possible scenario. The goal of this dissertation is to formalize the theory for testing automated vehicles. Various methods for automatic test generation for automated-driving systems in simulation environments are presented and compared. The contributions presented in this dissertation include (i) new metrics that can be used to discover the boundary cases between safe and unsafe driving conditions, (ii) a new approach that combines combinatorial testing and optimization-guided test generation methods, (iii) approaches that utilize global optimization methods and random exploration to generate critical vehicle and pedestrian trajectories for testing purposes, (iv) a publicly-available simulation-based automated vehicle testing framework that enables application of the existing testing approaches in the literature, including the new approaches presented in this dissertation.Dissertation/ThesisDoctoral Dissertation Computer Engineering 201