Baur, C.

Fong, T.

Parker, Lynne E.

Schultz, Alan C.

Thorpe, C.

Infoscience - École polytechnique fédérale de Lausanne

ROBOT AS PARTNER:VEHICLE TELEOPERATION WITHCOLLABORATIVE CONTROLTerrence Fong and Charles ThorpeThe Robotics Institute, Carnegie Mellon University, Pittsburgh, PennsylvaniaCharles BaurSwiss Federal Institute of Technology, Lausanne, SwitzerlandAbstract We have developed a new teleoperation system model called collabora-tive control. With this model, the robot asks the human questions, toobtain assistance with cognition and perception during task execution.This enables the human to support the robot and to compensate for in-adequacies in autonomy. In the following, we review the system modelsconventionally used in teleoperation, describe collaborative control, anddiscuss its use.Keywords: Collaborative control, human-robot interaction, vehicle teleoperation.1. IntroductionIn teleoperation, a robot is commonly viewed as a tool: a device ca-pable of performing tasks on command. As such, a robot has limitedfreedom to act and will always perform poorly whenever its capabili-ties are ill-suited for the task at hand. This is particularly true whenhigh-level perceptual functions (e.g., object recognition) are involved.Moreover, even if a robot realizes that it is performing poorly, it usuallyhas no way to ask for (or to gain) assistance.The problem is that the “robot as tool” paradigm is extremely limit-ing, i.e., it restricts the human-robot relationship to that of master-slave.As a result, the system’s capability is strictly bound to the operator’sskill and the quality of the user interface. In order to make teleoperationbetter performing, therefore, we must ﬁnd a new approach. What weneed is a paradigm that is more ﬂexible, that encourages human-robotsynergy, and that allows robots to work as partners (if not as peers).122. Collaborative Control2.1 A Robot-Centric System ModelTo address this need, we have developed collaborative control, a sys-tem model in which human and robot work together[4, 5]. Instead of asupervisor dictating to a subordinate, the human and the robot engagein dialogue to exchange information, to ask questions, and to resolve dif-ferences. With this approach, human-robot interaction is more natural,more balanced, and more direct than conventional approaches.With collaborative control, the human functions as a resource forthe robot, providing information and processing just like other systemmodules. In particular, the robot is allowed to ask the human questionsas it works, to obtain assistance with perception and cognition. Thisallows the human to compensate for limitations of autonomy. Moreover,since the robot is aware that the human may not respond, collaborativecontrol enables dynamic, ﬁne-grained sharing of control. With otherforms of teleoperation, the division of labor is pre-deﬁned or is coarselyswitched on a per-task basis.To understand how this works in practice, consider the following sit-uation: a mobile robot is driving forward when it has diﬃculty decidingif there is an obstacle in its way (e.g., range sensors return conﬂictinginformation). At this point, the robot must make a decision. Withconventional design, there are three choices: wait for the path to be-come clear, look for a way around, or ignore the conﬂict and continueforward. All of these strategies have signiﬁcant problems: “wait untilclear” may cause indeﬁnite delay; “make a detour” may consume exces-sive resources; and “drive through” may result in excessive damage.With collaborative control, the robot can ask for human assistance.In this case, the robot can present the sensor data (e.g., a camera image)to the human and ask his opinion. Once the human sees the data, hemay decide (based on experience or interpretation) that “drive through”is acceptable. In other words, through collaboration, the robot can avoidneedless delay and having to make an unnecessary detour.Collaborative control provides an eﬀective framework for coordina-tion. Since the human can only attend to one module (or robot) at atime, we arbitrate among the requests to select which is presented. Thisallows human attention to be directed where it is most needed, in termsof safety, priority, etc. Additionally, because collaborative control in-corporates a user model for dialogue management, it can accommodateusers with varied backgrounds and capabilities. Thus, the robot canask questions and can interpret responses, based on the user’s expertise,preferences, and other characteristics.Robot as Partner: Vehicle Teleoperation with Collaborative Control 33. Related ResearchSome robot control architectures have addressed the problem of mix-ing humans with robots. One method is to incorporate humans as asystem module[8]. Another method is prioritized control, the classic ex-ample of which is NASREM[1]. In both these approaches, the humanonly provides command input. With collaborative control, however, thehuman contributes his expertise wherever it can be used.Teleassistance tries to improve teleoperation by supplying aid to theoperator in the same manner that an expert would[7]. Collaborativecontrol takes the opposite approach: it provides the robot with humanassistance. Although adjustable autonomy (e.g., [3]) shares some as-pects of collaborative control, human-robot dialogue is not used as amechanism for adaptation and coordination.In many ways, collaborative control is most similar to remote expertsystems. For example, Bauer et al. describe a wearable computer witha camera that enables oﬃce-based experts to see what a ﬁeld techniciansees[2]. The parallel with collaborative control is clear: the robot islike the ﬁeld technician (i.e., it is skilled, but may need help) and theoperator is like the expert (i.e., he can provide assistance when needed).4. Conventional System Models(a) (b) (c) (d)Figure 1. Teleoperation system models: a, direct control; b, supervisory control; c,fully autonomous control; d, collaborative control.The most common method for vehicle teleoperation is direct control:the operator operates the vehicle using hand-controllers while monitoringvideo displays (Figure 1a). Because all control decisions depend on thehuman, system performance is directly linked to human capabilities.Many factors including skill, training, etc., all play a role in how thesystem functions. Other factors, such as communication bandwidth,may also inﬂuence operational eﬃcacy.4The supervisory control system model is shown in Figure 1b. Withsupervisory control, the human divides a problem into a sequence oftasks which the robot performs on its own [9]. Once he gives control tothe robot, the human typically assumes a monitoring role. However, thehuman may also intermittently (i.e., trade or share) control the robot byclosing a command loop or he may control some variables while leavingthe others to the robot.Fully autonomous control is somewhat of a misnomer because it rarelyis fully automatic. With this system model, the human gives high-levelgoals, which the robot independently achieves (Figure 1c). The diﬀer-ence between supervisory and fully autonomous control is the nature ofthe goal. In the former, goals are limited and task planning is performedprimarily by the human. With the latter, goals are more abstract andthe robot is responsible for planning.With conventional system models, poor performance (or failure) willoccur if the human fails to recognize that the robot is ill-suited for thetask or situation. Additionally, none of the models can eﬀectively accom-modate a wide range of users. Direct control, for example, is generallylimited to trained, expert users because diﬃculty and risk are high.5. Collaborative Control System ModelCollaborative control addresses the limitations of conventional systemmodels through collaboration and dialogue. In supervisory or fully au-tonomous control, if the robot has diﬃculties, the only choices it hasare to continue performing poorly or to stop. With collaborative con-trol, however, the robot has the option of asking the human to assist:providing information, helping perform perception or cognition, etc.Another way in which collaborative control diﬀers from conventionalsystem models is that it provides ﬁne-grained sharing/trading of con-trol and autonomy adjustments. Because work is dynamically allocatedthrough dialogue, the human is automatically included in the controlloop as needed. This is a signiﬁcant diﬀerence from other models, whichrequire the user to decide how, when, and where control should be allo-cated.With collaborative control, therefore, the human may be involvedin multiple control loops (Figure 1d). As with supervisory control, hemay close a command loop or monitor task execution through interfacedisplays. As a resource for the robot, however, the human may alsoclose a perception loop, a cognition loop, or some combination of thetwo. Furthermore, the human may interact with the robot at diﬀerentlevels of abstraction.Robot as Partner: Vehicle Teleoperation with Collaborative Control 56. System DesignWe have implemented collaborative control as a set of modules (di-alogue management, robot control, etc.), each of which operates withvariable autonomy in a message-based architecture[4]. Whether a mod-ule operates at a low level (strongly dependent on the human) or highlevel (little or no human interaction) of autonomy is determined by fac-tors including situational demands, module competency, and user model.In our system, dialogue is the exchange of messages between hu-man and robot. Our current system has approximately thirty mes-sages: human-to-robot (command, queries, responses) and robot-to-human messages (information statements, queries). At present, therobot is able to ask two types of queries. Safeguard queries concernsafety issues (e.g., “Stopped due to rollover danger. Can you come overand help?”. Task queries describe task-speciﬁc functions, such as “Mo-tion detected. Is this an intruder (image)? If you answer yes, I willfollow him.”Each robot query is described by a number of attributes, some ofwhich are operator-dependent (required response accuracy, required ex-pertise) and others which are operator-independent (expiration, priority,etc). These attributes are used to select which queries will be asked. Afailure to respond, whether intentional or not, can trigger behavioralchanges (e.g., the robot’s level of autonomy may be increased).7. Results7.1 Remote Driving TestsDuring the past year, we have conducted a variety of remote driv-ing tests to evaluate the use of collaborative control[4, 5]. In one test,we examined the use of multiple mobile robots for reconnaissance andsurveillance. One of the attractive features of collaborative control isthat it directs the human’s limited resources (attention, cognition, etc.)where they are needed. This relieves the human of the burden of simul-taneously monitoring each robot.Figure 2 shows an experiment in which an operator used two robotsto perform outdoor reconnaissance. During the test, each robot askedsafety-related questions while traversing unknown terrain. For example,both robots had questions about setting safety levels. Although it iscommon practice to deﬁne “normal” safety levels, there are some occa-sions when a system must be used beyond its design speciﬁcations. Thisis particularly true for situations in which system loss is acceptable aslong as the goal is achieved (e.g., military combat missions).6Figure 2. Outdoor reconnaissance with two robots7.2 User StudyTo examine how collaborative control inﬂuences human-robot inter-action, we recently performed a Contextual Inquiry (CI) user study[4].CI is a structured interviewing method for grounding interactive systemdesign in the context of the work[6]. In the study, users were required toexplore a cluttered environment while assisting safeguarding autonomy.The study revealed several interesting ﬁndings. We observed that: (1)diﬀerent users may respond quite diﬀerently to the same question; (2)users may grow weary of answering questions; (3) a question withoutadequate detail is hard to answer; (4) dialogue can make users personifythe robot; and (5) indicating the urgency of questions is important.Overall, we found dialogue to be valuable for teleoperation. In particu-lar, novices reported that dialogue signiﬁcantly helped them understandthe problems the robot encountered during task execution. Although ex-perts were generally less satisﬁed than novices, primarily because theygrew tired of answering questions, they also stated that dialogue was auseful in keeping them involved and engaged in system operation.Robot as Partner: Vehicle Teleoperation with Collaborative Control 78. Discussion8.1 Beneﬁts of Collaborative ControlUnlike other forms of teleoperation, in which the division of labor isdeﬁned a priori, collaborative control allows human-robot interactionand autonomy to vary as needed. If the robot is capable of handlinga task autonomously, it can do so. But, if it cannot, the human canprovide assistance.The use of dialogue makes human-robot interaction adaptable. Sincethe robot is aware of the user, it can always decide if asking a questionwill be useful. Because it has knowledge of the human’s expertise, accu-racy, etc., the robot can consider whether to accept a response at facevalue or to weigh it against other factors. In this way, system operationcan adapt to diﬀerent operators.Dialogue helps the human to be eﬀective. By focusing attention whereit is needed, dialogue helps coordinate and direct problem solving. Inparticular, in situations in which the robot does not know what to do, orwhen it is working poorly, a simple human answer (even a single bit ofinformation from a novice) is often all that is required to get the robotout of trouble.8.2 Limitations of Collaborative ControlAlthough collaborative control is beneﬁcial to teleoperation, there arelimits to what it can provide. If human-robot interaction is adaptive,then control and information ﬂow will vary with time and situation. Thiscan make validation and veriﬁcation diﬃcult because it becomes harderto duplicate an error condition or a failure situation.Another consideration is that when humans and robots interact toachieve common goals, they are subject to team related issues. In par-ticular, teamwork requires team members to coordinate and synchro-nize their activities, to exchange information and communicate eﬀec-tively, and to minimize the potential for interference between themselves.Moreover, there are numerous factors which can impact and limit groupperformance including resource distribution, timing, progress monitor-ing, and procedure maintenance.Finally, working in collaboration requires that each partner trust andunderstand the other. To do this, each collaborator needs to have anaccurate model of (1) what the other is capable of doing and (2) how hewill carry out a given assignment. If the model is inaccurate, or if thepartner cannot be expected to perform correctly, then the collaborationwill not work well.89. Open IssuesAs we have discussed, collaborative control provides a framework forcoordinating and adapting robot operation. The issue of scaleability,however, remains to be addressed. For example, if the human mustinteract with a large number of robots, it might not be possible forhim to assist each one individually. Instead, it would be more practicalto focus on group interaction and have the robots work in formation.Similarly, if a robot has many varied modules, it may be diﬃcult for thehuman to help diﬀerent types of autonomy. In this case, it might bemore eﬃcient for the human to only assist certain modules.In more general terms, human assistance is clearly a limited resource.Hence, we need to ﬁnd ways of motivating the user to respond. Thisis particularly important when the robot must operate for long periods.One approach would be to develop an algorithm for choosing questionsthat are “signiﬁcant” to the user, i.e., having some level of informationtheoretical content (bits), matching user interest, etc.AcknowledgmentsThis work was partially supported by grants from the DARPA ITOMARS program, the National Science Foundation, and SAIC.References[1] J. Albus et al. NASA/NBS Standard reference model for telerobot control systemarchitecture (NASREM). Technical Note 1235, NIST, 1987.[2] M. Bauer et al. A Collaborative Wearable System with Remote Sensing. In Proc.of 2nd International Symposium on Wearable Computers, 1998.[3] G. Dorais et al. Adjustable autonomy for human-centered autonomous systems.In Proc. of Workshop on Adjustable Autonomy Systems (IJCAI), 1999.[4] T. Fong. Collaborative control: a robot-centric model for vehicle teleoperation.PhD thesis, Carnegie Mellon University, 2001.[5] T. Fong et al. Multi-robot remote driving with collaborative control. IEEETransactions on Industrial Electronics (in press), 2002.[6] K. Holtzblatt and S. Jones. Contextual inquiry: a participatory technique for sys-tem design. In D. Schuler and A. Namioka (eds.), Participatory design: principlesand practice. Lawrence Erlbaum, 1993.[7] R. Murphy and E. Rogers. Cooperative assistance for remote robot supervision.Presence, 5(2), 1996.[8] J. Rosenblatt DAMN: A distributed architecture for mobile navigation. In Proc.of AAAI Spring Symposium on Lessons Learned from Implemented Software Ar-chitecture for Physical Agents, 1995.[9] T. Sheridan. Telerobotics, automation, and human sup. control. MIT Press, 1992.

Robot as Partner: Vehicle Teleoperation with Collaborative Control

Abstract

Similar works

Full text

Available Versions

Infoscience - École polytechnique fédérale de Lausanne