An integrated theory of language production and comprehension

Currently, production and comprehension are regarded as quite distinct in accounts of language processing. In rejecting this dichotomy, we instead assert that producing and understanding are interwoven, and that this interweaving is what enables people to predict themselves and each other. We start by noting that production and comprehension are forms of action and action perception. We then consider the evidence for interweaving in action, action perception, and joint action, and explain such evidence in terms of prediction. Specifically, we assume that actors construct forward models of their actions before they execute those actions, and that perceivers of others' actions covertly imitate those actions, then construct forward models of those actions. We use these accounts of action, action perception, and joint action to develop accounts of production, comprehension, and interactive language. Importantly, they incorporate well-defined levels of linguistic representation (such as semantics, syntax, and phonology). We show (a) how speakers and comprehenders use covert imitation and forward modeling to make predictions at these levels of representation, (b) how they interweave production and comprehension processes, and (c) how they use these predictions to monitor the upcoming utterances. We show how these accounts explain a range of behavioral and neuroscientific data on language processing and discuss some of the implications of our proposal

On the Action Semantics of Concurrent Programming Languages

Action semantics is a framework for semantic description of prograrnming languages. In this framework, actions are semantic entities, used to represent the potential behaviour of programs --- also the contributions that parts of programs make to such behaviour. The notation for expressing actions, called action notation, is combinator-based. It is used in much the same way that lambda-notation is used in denotational semantics. However, the essence of action notation is operational, rather than mathematical, and its meaning is formally defined by a structural operational semantics together with a bisimulation equivalence.This paper briefly motivates action semantics, and explains the basic concepts. It then illustrates the use of the framework by giving an action semantic description of a small example language. This language includes a simple form of concurrency: tasks that may synchronize by means of rendezvous. The paper also discusses the operational semantics of action notation, focusing on the primitive actions that represent asynchronous message transmission and process initiation

Learning the Semantics of Manipulation Action

In this paper we present a formal computational framework for modeling manipulation actions. The introduced formalism leads to semantics of manipulation action and has applications to both observing and understanding human manipulation actions as well as executing them with a robotic mechanism (e.g. a humanoid robot). It is based on a Combinatory Categorial Grammar. The goal of the introduced framework is to: (1) represent manipulation actions with both syntax and semantic parts, where the semantic part employs $\lambda$-calculus; (2) enable a probabilistic semantic parsing schema to learn the $\lambda$-calculus representation of manipulation action from an annotated action corpus of videos; (3) use (1) and (2) to develop a system that visually observes manipulation actions and understands their meaning while it can reason beyond observations using propositional logic and axiom schemata. The experiments conducted on a public available large manipulation action dataset validate the theoretical framework and our implementation

CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos

Temporal action localization is an important yet challenging problem. Given a long, untrimmed video consisting of multiple action instances and complex background contents, we need not only to recognize their action categories, but also to localize the start time and end time of each instance. Many state-of-the-art systems use segment-level classifiers to select and rank proposal segments of pre-determined boundaries. However, a desirable model should move beyond segment-level and make dense predictions at a fine granularity in time to determine precise temporal boundaries. To this end, we design a novel Convolutional-De-Convolutional (CDC) network that places CDC filters on top of 3D ConvNets, which have been shown to be effective for abstracting action semantics but reduce the temporal length of the input data. The proposed CDC filter performs the required temporal upsampling and spatial downsampling operations simultaneously to predict actions at the frame-level granularity. It is unique in jointly modeling action semantics in space-time and fine-grained temporal dynamics. We train the CDC network in an end-to-end manner efficiently. Our model not only achieves superior performance in detecting actions in every frame, but also significantly boosts the precision of localizing temporal boundaries. Finally, the CDC network demonstrates a very high efficiency with the ability to process 500 frames per second on a single GPU server. We will update the camera-ready version and publish the source codes online soon.Comment: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 201

Semantic Component Composition

Building complex software systems necessitates the use of component-based architectures. In theory, of the set of components needed for a design, only some small portion of them are "custom"; the rest are reused or refactored existing pieces of software. Unfortunately, this is an idealized situation. Just because two components should work together does not mean that they will work together. The "glue" that holds components together is not just technology. The contracts that bind complex systems together implicitly define more than their explicit type. These "conceptual contracts" describe essential aspects of extra-system semantics: e.g., object models, type systems, data representation, interface action semantics, legal and contractual obligations, and more. Designers and developers spend inordinate amounts of time technologically duct-taping systems to fulfill these conceptual contracts because system-wide semantics have not been rigorously characterized or codified. This paper describes a formal characterization of the problem and discusses an initial implementation of the resulting theoretical system.Comment: 9 pages, submitted to GCSE/SAIG '0

Theory and Practice of Action Semantics

Action Semantics is a framework for the formal descriptionof programming languages. Its main advantage over other frameworksis pragmatic: action-semantic descriptions (ASDs) scale up smoothly torealistic programming languages. This is due to the inherent extensibilityand modifiability of ASDs, ensuring that extensions and changes tothe described language require only proportionate changes in its description.(In denotational or operational semantics, adding an unforeseenconstruct to a language may require a reformulation of the entire description.)After sketching the background for the development of action semantics,we summarize the main ideas of the framework, and provide a simpleillustrative example of an ASD. We identify which features of ASDsare crucial for good pragmatics. Then we explain the foundations ofaction semantics, and survey recent advances in its theory and practicalapplications. Finally, we assess the prospects for further developmentand use of action semantics.The action semantics framework was initially developed at the Universityof Aarhus by the present author, in collaboration with David Watt(University of Glasgow). Groups and individuals scattered around fivecontinents have since contributed to its theory and practice

Representing First-Order Causal Theories by Logic Programs

Nonmonotonic causal logic, introduced by Norman McCain and Hudson Turner, became a basis for the semantics of several expressive action languages. McCain's embedding of definite propositional causal theories into logic programming paved the way to the use of answer set solvers for answering queries about actions described in such languages. In this paper we extend this embedding to nondefinite theories and to first-order causal logic.Comment: 29 pages. To appear in Theory and Practice of Logic Programming (TPLP); Theory and Practice of Logic Programming, May, 201

Maude Object-Oriented Action Tool

MAIDL, André Murbach; CARVILHE, Claudio; MUSICANTE, Martin A. Maude Object-Oriented Action Tool. Electronic Notes in Theoretical Computer Science. [S.l:s.n], 2008.Object-Oriented Action Semantics (OOAS) incorporates object-oriented concepts to the Action Semantics formalism. Its main goal is to obtain more readable and reusable semantics specifications. Moreover, it supports syntax-independent specifications, due to the way classes are written. Maude Object-Oriented Action Tool (MOOAT) is an executable environment for Object-Oriented Action Semantics implemented as a conservative extension of Full Maude and Maude MSOS Tool (MMT). The Modular SOS of Action Notation has been implemented using MMT transitions and Full Maude has been used to implement the Classes Notation. The syntax created by MOOAT is fairly similar to the original Object-Oriented Action Semantics syntax. In addition to it, the tool combines the modularity aspects observed in the object-oriented approach with the efficient execution and analysis of the Maude system. We use MOOAT to describe syntaxindependent specifications of programming languages. In this way, we show how Constructive Object-Oriented Action Semantics (COOAS) may be achieved as a combination between Object-Oriented Action Semantics and Constructive Action Semantics (CAS) using MOOAT, in order to increase the modularity aspects observed in the object-oriented formalism. This paper reports on the development of Maude Object-Oriented Action Tool and its application to the formal specification of programming languages