Inferring team task plans from human meetings: A generative modeling approach with logic-based prior

We aim to reduce the burden of programming and deploying autonomous systems to work in concert with people in time-critical domains such as military field operations and disaster response. Deployment plans for these operations are frequently negotiated on-the-fly by teams of human planners. A human operator then translates the agreed-upon plan into machine instructions for the robots. We present an algorithm that reduces this translation burden by inferring the final plan from a processed form of the human team's planning conversation. Our hybrid approach combines probabilistic generative modeling with logical plan validation used to compute a highly structured prior over possible plans, enabling us to overcome the challenge of performing inference over a large solution space with only a small amount of noisy data from the team planning session. We validate the algorithm through human subject experimentations and show that it is able to infer a human team's final plan with 86% accuracy on average. We also describe a robot demonstration in which two people plan and execute a first-response collaborative task with a PR2 robot. To the best of our knowledge, this is the first work to integrate a logical planning technique within a generative model to perform plan inference.United States. Dept. of Defense. Assistant Secretary of Defense for Research & Engineering (United States. Air Force Contract FA8721-05-C-0002

Plan-based integration of natural language and graphics generation

Multimodal interfaces combining natural language and graphics take advantage of both the individual strength of each communication mode and the fact that several modes can be employed in parallel. The central claim of this paper is that the generation of a multimodal presentation system WIP which allows the generation of alternate presentations of the same content taking into account various contextual factors. We discuss how the plan-based approach to presentation design can be exploited so that graphics generation influences the production of text and vice versa. We show that well-known concepts from the area of natural language processing like speech acts, anaphora, and rhetorical relations take on an extended meaning in the context of multimodal communication. Finally, we discuss two detailed examples illustrating and reinforcing our theoretical claims

Visual recognition of multi-agent action

Thesis (Ph.D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 1999.Includes bibliographical references (p. 167-184).Developing computer vision sensing systems that work robustly in everyday environments will require that the systems can recognize structured interaction between people and objects in the world. This document presents a new theory for the representation and recognition of coordinated multi-agent action from noisy perceptual data. The thesis of this work is as follows: highly structured, multi-agent action can be recognized from noisy perceptual data using visually grounded goal-based primitives and low-order temporal relationships that are integrated in a probabilistic framework. The theory is developed and evaluated by examining general characteristics of multi-agent action, analyzing tradeoffs involved when selecting a representation for multi-agent action recognition, and constructing a system to recognize multi-agent action for a real task from noisy data. The representation, which is motivated by work in model-based object recognition and probabilistic plan recognition, makes four principal assumptions: (1) the goals of individual agents are natural atomic representational units for specifying the temporal relationships between agents engaged in group activities, (2) a high-level description of temporal structure of the action using a small set of low-order temporal and logical constraints is adequate for representing the relationships between the agent goals for highly structured, multi-agent action recognition, (3) Bayesian networks provide a suitable mechanism for integrating multiple sources of uncertain visual perceptual feature evidence, and (4) an automatically generated Bayesian network can be used to combine uncertain temporal information and compute the likelihood that a set of object trajectory data is a particular multi-agent action. The recognition algorithm is tested using a database of American football play descriptions. A system is described that can recognize single-agent and multi-agent actions in this domain given noisy trajectories of object movements. The strengths and limitations of the recognition system are discussed and compared with other multi-agent recognition algorithms.by Stephen Sean Intille.Ph.D

International Workshop on Description Logics : Bonn, May 28/29, 1994

This collection of papers forms the permanent record of the 1994 Description Logic Workshop, that was held at the Gustav Stresemann Institut in Bonn, Germany on 28 and 29 May 1994, immediately after the Fourth International Conference on Principles of Knowledge Representation and Reasoning. The workshop was set up to be as informal as possible, so this collection cannot hope to capture the discussions associated with the workshop. However, we hope that it will serve to remind participants of their discussion at the workshop, and provide non-participants with indications of the topics that were discussed at the workshop. The workshop consisted of seven regular sessions and one panel session. Each regular session had about four short presentations on a single theme, but also had considerable time reserved for discussion. The themes of the sessions were Foundations of Description Logics, Architecture of Description Logics and Description Logic Systems, Language Extensions, Expanding Description Logics, General Applications of Description Logics, Natural Language Applications of Description Logics, Connections between Description Logics and Databases, and the Future of Description Logics and Description Logic Systems. The session on Foundations of Description Logics concentrated on computational properties of description logics, correspondences between description logics and other formalisms, and on semantics of description logics, Similarly, there is discussion on how to develop tractable desription logics, for some notion of tractable, and whether it is useful to worry about achieving tractability at all. Several of the participants argued in favour of a very expressive description logic. This obviously precludes tractability or even decidability of complete reasoning. Klaus Schild proposed that for some purposes one could employ "model checking" (i .e., a closed world assumption) instead of "theorem proving," and has shown that this is still tractable for very large languages. Maurizio Lenzerini's opinion was that it is important to have decidable languages. Tractability cannot be achieved in several application areas because there one needs very expressive constructs: e.g., axioms, complex role constructors, and cycles with fixed-point semantics. For Bob MacGregor, not even decidability is an issue since he claims that Loom's incomplete reasoner is sufficient for his applications. The discussion addressed the question of whether there is still need for foundations, and whether the work on foundation done until now really solved the problems that the designers of early DL systems had. Both questions were mostly answered in the affirmative, with the caveat that new research on foundations should make sure that it is concerned with "real" problems, and not just generates new problems. In the session on Architecture of Description Logics and Description Logic Systems the participants considered different ways of putting together description logics and description logic systems. One way of doing this is to have a different kind of inference strategy for description logics, such as one based on intuitionistic logics or one based directly on rules of inference-thus allowing variant systems. Another way of modifying description logic systems is to divide them up in different ways, such as making a terminology consist of a schema portion and a view portion. Some discussion in this session concerned whether architectures should be influenced by application areas, or even by particular applications. There was considerable discussion at the workshop on how Description Logics should be extended or expanded to make them more useful. There are several methods to do this. The first is to extend the language of descriptions, e.g ., to represent n-ary relations, temporal information, or whole-part relationships, all of which were discussed at the workshop. The second is to add in another kind of reasoning, such as default reasoning, while still keeping the general framework of description logic reasoning. The third is to incorporate descriptions or description-like constructs in a larger reasoner, such as a first order reasoner. This was the approach taken in OMEGA and is the approach being taken in the Loom project. There have been many extensions of the first two kinds proposed for description logics, including several presented at the workshop. One quest ion discussed at the workshop was whether these extensions fit in well with the philosophy of description logic. Another question was whether the presence of many proposals for extensions means that description logics are easy to expand, or that description logics are inadequate representation formalisms? The general consensus was that description logics adequately capture a certain kind of core reasoning and that they lend themselves to incorporation with other kinds of reasoning. Care must be taken, however, to keep the extended versions true to the goals of description logics. The sessions on Applications of Description Logics had presentations on applications of description logics in various areas, including configuration, tutoring, natural language processing, and domain modeling. Most of these applications are research applications, funded by government research programs. There was discussion of what is needed to have more fielded applications of description logics. The session on Connections between Description Logics and Databases considered three kinds of connections between Description Logics and Databases: 1. using Description Logics for expressing database schemas, including local schemas, integrated schemas, and views, integrity constraints, and queries; 2. using Description Logic reasoning for various database-related reasoning, including schema integration and validation, and query optimization, and query validation and organization; and 3. making Description Logic reasoners more like Database Mangagement Systems via optimization. All three of these connections are being actively investigated by the description logic community. The panel session on the Future of Description Logics and Description Logic Systems discussed where the future of description logics will lie. There seems to be a consensus that description logics must forge tighter connections with other formalisms, such as databases or object-oriented systems. In this way, perhaps, description logics will find more real applications

International Workshop on Description Logics : Bonn, May 28/29, 1994

This collection of papers forms the permanent record of the 1994 Description Logic Workshop, that was held at the Gustav Stresemann Institut in Bonn, Germany on 28 and 29 May 1994, immediately after the Fourth International Conference on Principles of Knowledge Representation and Reasoning. The workshop was set up to be as informal as possible, so this collection cannot hope to capture the discussions associated with the workshop. However, we hope that it will serve to remind participants of their discussion at the workshop, and provide non-participants with indications of the topics that were discussed at the workshop. The workshop consisted of seven regular sessions and one panel session. Each regular session had about four short presentations on a single theme, but also had considerable time reserved for discussion. The themes of the sessions were Foundations of Description Logics, Architecture of Description Logics and Description Logic Systems, Language Extensions, Expanding Description Logics, General Applications of Description Logics, Natural Language Applications of Description Logics, Connections between Description Logics and Databases, and the Future of Description Logics and Description Logic Systems. The session on Foundations of Description Logics concentrated on computational properties of description logics, correspondences between description logics and other formalisms, and on semantics of description logics, Similarly, there is discussion on how to develop tractable desription logics, for some notion of tractable, and whether it is useful to worry about achieving tractability at all. Several of the participants argued in favour of a very expressive description logic. This obviously precludes tractability or even decidability of complete reasoning. Klaus Schild proposed that for some purposes one could employ "model checking" (i .e., a closed world assumption) instead of "theorem proving," and has shown that this is still tractable for very large languages. Maurizio Lenzerini\u27s opinion was that it is important to have decidable languages. Tractability cannot be achieved in several application areas because there one needs very expressive constructs: e.g., axioms, complex role constructors, and cycles with fixed-point semantics. For Bob MacGregor, not even decidability is an issue since he claims that Loom\u27s incomplete reasoner is sufficient for his applications. The discussion addressed the question of whether there is still need for foundations, and whether the work on foundation done until now really solved the problems that the designers of early DL systems had. Both questions were mostly answered in the affirmative, with the caveat that new research on foundations should make sure that it is concerned with "real" problems, and not just generates new problems. In the session on Architecture of Description Logics and Description Logic Systems the participants considered different ways of putting together description logics and description logic systems. One way of doing this is to have a different kind of inference strategy for description logics, such as one based on intuitionistic logics or one based directly on rules of inference-thus allowing variant systems. Another way of modifying description logic systems is to divide them up in different ways, such as making a terminology consist of a schema portion and a view portion. Some discussion in this session concerned whether architectures should be influenced by application areas, or even by particular applications. There was considerable discussion at the workshop on how Description Logics should be extended or expanded to make them more useful. There are several methods to do this. The first is to extend the language of descriptions, e.g ., to represent n-ary relations, temporal information, or whole-part relationships, all of which were discussed at the workshop. The second is to add in another kind of reasoning, such as default reasoning, while still keeping the general framework of description logic reasoning. The third is to incorporate descriptions or description-like constructs in a larger reasoner, such as a first order reasoner. This was the approach taken in OMEGA and is the approach being taken in the Loom project. There have been many extensions of the first two kinds proposed for description logics, including several presented at the workshop. One quest ion discussed at the workshop was whether these extensions fit in well with the philosophy of description logic. Another question was whether the presence of many proposals for extensions means that description logics are easy to expand, or that description logics are inadequate representation formalisms? The general consensus was that description logics adequately capture a certain kind of core reasoning and that they lend themselves to incorporation with other kinds of reasoning. Care must be taken, however, to keep the extended versions true to the goals of description logics. The sessions on Applications of Description Logics had presentations on applications of description logics in various areas, including configuration, tutoring, natural language processing, and domain modeling. Most of these applications are research applications, funded by government research programs. There was discussion of what is needed to have more fielded applications of description logics. The session on Connections between Description Logics and Databases considered three kinds of connections between Description Logics and Databases: 1. using Description Logics for expressing database schemas, including local schemas, integrated schemas, and views, integrity constraints, and queries; 2. using Description Logic reasoning for various database-related reasoning, including schema integration and validation, and query optimization, and query validation and organization; and 3. making Description Logic reasoners more like Database Mangagement Systems via optimization. All three of these connections are being actively investigated by the description logic community. The panel session on the Future of Description Logics and Description Logic Systems discussed where the future of description logics will lie. There seems to be a consensus that description logics must forge tighter connections with other formalisms, such as databases or object-oriented systems. In this way, perhaps, description logics will find more real applications

International Workshop on Description Logics : Bonn, May 28/29, 1994

This collection of papers forms the permanent record of the 1994 Description Logic Workshop, that was held at the Gustav Stresemann Institut in Bonn, Germany on 28 and 29 May 1994, immediately after the Fourth International Conference on Principles of Knowledge Representation and Reasoning. The workshop was set up to be as informal as possible, so this collection cannot hope to capture the discussions associated with the workshop. However, we hope that it will serve to remind participants of their discussion at the workshop, and provide non-participants with indications of the topics that were discussed at the workshop. The workshop consisted of seven regular sessions and one panel session. Each regular session had about four short presentations on a single theme, but also had considerable time reserved for discussion. The themes of the sessions were Foundations of Description Logics, Architecture of Description Logics and Description Logic Systems, Language Extensions, Expanding Description Logics, General Applications of Description Logics, Natural Language Applications of Description Logics, Connections between Description Logics and Databases, and the Future of Description Logics and Description Logic Systems. The session on Foundations of Description Logics concentrated on computational properties of description logics, correspondences between description logics and other formalisms, and on semantics of description logics, Similarly, there is discussion on how to develop tractable desription logics, for some notion of tractable, and whether it is useful to worry about achieving tractability at all. Several of the participants argued in favour of a very expressive description logic. This obviously precludes tractability or even decidability of complete reasoning. Klaus Schild proposed that for some purposes one could employ "model checking" (i .e., a closed world assumption) instead of "theorem proving," and has shown that this is still tractable for very large languages. Maurizio Lenzerini's opinion was that it is important to have decidable languages. Tractability cannot be achieved in several application areas because there one needs very expressive constructs: e.g., axioms, complex role constructors, and cycles with fixed-point semantics. For Bob MacGregor, not even decidability is an issue since he claims that Loom's incomplete reasoner is sufficient for his applications. The discussion addressed the question of whether there is still need for foundations, and whether the work on foundation done until now really solved the problems that the designers of early DL systems had. Both questions were mostly answered in the affirmative, with the caveat that new research on foundations should make sure that it is concerned with "real" problems, and not just generates new problems. In the session on Architecture of Description Logics and Description Logic Systems the participants considered different ways of putting together description logics and description logic systems. One way of doing this is to have a different kind of inference strategy for description logics, such as one based on intuitionistic logics or one based directly on rules of inference-thus allowing variant systems. Another way of modifying description logic systems is to divide them up in different ways, such as making a terminology consist of a schema portion and a view portion. Some discussion in this session concerned whether architectures should be influenced by application areas, or even by particular applications. There was considerable discussion at the workshop on how Description Logics should be extended or expanded to make them more useful. There are several methods to do this. The first is to extend the language of descriptions, e.g ., to represent n-ary relations, temporal information, or whole-part relationships, all of which were discussed at the workshop. The second is to add in another kind of reasoning, such as default reasoning, while still keeping the general framework of description logic reasoning. The third is to incorporate descriptions or description-like constructs in a larger reasoner, such as a first order reasoner. This was the approach taken in OMEGA and is the approach being taken in the Loom project. There have been many extensions of the first two kinds proposed for description logics, including several presented at the workshop. One quest ion discussed at the workshop was whether these extensions fit in well with the philosophy of description logic. Another question was whether the presence of many proposals for extensions means that description logics are easy to expand, or that description logics are inadequate representation formalisms? The general consensus was that description logics adequately capture a certain kind of core reasoning and that they lend themselves to incorporation with other kinds of reasoning. Care must be taken, however, to keep the extended versions true to the goals of description logics. The sessions on Applications of Description Logics had presentations on applications of description logics in various areas, including configuration, tutoring, natural language processing, and domain modeling. Most of these applications are research applications, funded by government research programs. There was discussion of what is needed to have more fielded applications of description logics. The session on Connections between Description Logics and Databases considered three kinds of connections between Description Logics and Databases: 1. using Description Logics for expressing database schemas, including local schemas, integrated schemas, and views, integrity constraints, and queries; 2. using Description Logic reasoning for various database-related reasoning, including schema integration and validation, and query optimization, and query validation and organization; and 3. making Description Logic reasoners more like Database Mangagement Systems via optimization. All three of these connections are being actively investigated by the description logic community. The panel session on the Future of Description Logics and Description Logic Systems discussed where the future of description logics will lie. There seems to be a consensus that description logics must forge tighter connections with other formalisms, such as databases or object-oriented systems. In this way, perhaps, description logics will find more real applications