Learning to Select State Machines Using Expert Advice on an Autonomous Robot

Hierarchical state machines have proven to be a powerful tool for controlling autonomous robots due to their flexibility and modularity. For most real robot implementations, however, it is often the case that the control hierarchy is hand-coded. As a result, the development process is often time intensive and error prone. In this paper, we explore the use of an experts learning approach, based on Auer and colleagues’ Exp3 [1], to help overcome some of these limitations. In particular, we develop a modified learning algorithm, which we call rExp3, that exploits the structure provided by a control hierarchary by treating each state machine as an ’expert’. Our experiments validate the performance of rExp3 on a real robot performing a task, and demonstrate that rExp3 is able to quickly learn to select the best state machine expert to execute. Through our investigations in these environments, we identify a need for faster learning recovery when the relative performances of experts reorder, such as in response to a discrete environment change. We introduce a modified learning rule to improve the recovery rate in these situations and demonstrate through simulation experiments that rExp3 performs as well or better than Exp3 under such conditions

Learning to Select State Machines using Expert Advice on an Autonomous Robot

Abstract — Hierarchical state machines have proven to be a powerful tool for controlling autonomous robots due to their flexibility and modularity. For most real robot implementations, however, it is often the case that the control hierarchy is hand-coded. As a result, the development process is often time intensive and error prone. In this paper, we explore the use of an experts learning approach, based on Auer and colleagues ’ Exp3 [1], to help overcome some of these limitations. In particular, we develop a modified learning algorithm, which we call rExp3, that exploits the structure provided by a control hierarchary by treating each state machine as an ’expert’. Our experiments validate the performance of rExp3 on a real robot performing a task, and demonstrate that rExp3 is able to quickly learn to select the best state machine expert to execute. Through our investigations in these environments, we identify a need for faster learning recovery when the relative performances of experts reorder, such as in response to a discrete environment change. We introduce a modified learning rule to improve the recovery rate in these situations and demonstrate through simulation experiments that rExp3 performs as well or better than Exp3 under such conditions. I

Dynamic Behavior Sequencing in a Hybrid Robot Architecture

Hybrid robot control architectures separate plans, coordination, and actions into separate processing layers to provide deliberative and reactive functionality. This approach promotes more complex systems that perform well in goal-oriented and dynamic environments. In various architectures, the connections and contents of the functional layers are tightly coupled so system updates and changes require major changes throughout the system. This work proposes an abstract behavior representation, a dynamic behavior hierarchy generation algorithm, and an architecture design to reduce this major change incorporation process. The behavior representation provides an abstract interface for loose coupling of behavior planning and execution components. The hierarchy generation algorithm utilizes the interface allowing dynamic sequencing of behaviors based on behavior descriptions and system objectives without knowledge of the low-level implementation or the high-level goals the behaviors achieve. This is accomplished within the proposed architecture design, which is based on the Three Layer Architecture (TLA) paradigm. The design provides functional decomposition of system components with respect to levels of abstraction and temporal complexity. The layers and components within this architecture are independent of surrounding components and are coupled only by the linking mechanisms that the individual components and layers allow. The experiments in this thesis demonstrate that the: 1) behavior representation provides an interface for describing a behavior’s functionality without restricting or dictating its actual implementation; 2) hierarchy generation algorithm utilizes the representation interface for accomplishing high-level tasks through dynamic behavior sequencing; 3) representation, control logic, and architecture design create a loose coupling, but defined link, between the planning and behavior execution layer of the hybrid architecture, which creates a system-of-systems implementation that requires minimal reprogramming for system modifications