Seeing What You're Told: Sentence-Guided Activity Recognition In Video

We present a system that demonstrates how the compositional structure of events, in concert with the compositional structure of language, can interplay with the underlying focusing mechanisms in video action recognition, thereby providing a medium, not only for top-down and bottom-up integration, but also for multi-modal integration between vision and language. We show how the roles played by participants (nouns), their characteristics (adjectives), the actions performed (verbs), the manner of such actions (adverbs), and changing spatial relations between participants (prepositions) in the form of whole sentential descriptions mediated by a grammar, guides the activity-recognition process. Further, the utility and expressiveness of our framework is demonstrated by performing three separate tasks in the domain of multi-activity videos: sentence-guided focus of attention, generation of sentential descriptions of video, and query-based video search, simply by leveraging the framework in different manners.Comment: To appear in CVPR 201

Intensional and Extensional Semantics of Bounded and Unbounded Nondeterminism

We give extensional and intensional characterizations of nondeterministic functional programs: as structure preserving functions between biorders, and as nondeterministic sequential algorithms on ordered concrete data structures which compute them. A fundamental result establishes that the extensional and intensional representations of non-deterministic programs are equivalent, by showing how to construct a unique sequential algorithm which computes a given monotone and stable function, and describing the conditions on sequential algorithms which correspond to continuity with respect to each order. We illustrate by defining may and must-testing denotational semantics for a sequential functional language with bounded and unbounded choice operators. We prove that these are computationally adequate, despite the non-continuity of the must-testing semantics of unbounded nondeterminism. In the bounded case, we prove that our continuous models are fully abstract with respect to may and must-testing by identifying a simple universal type, which may also form the basis for models of the untyped lambda-calculus. In the unbounded case we observe that our model contains computable functions which are not denoted by terms, by identifying a further "weak continuity" property of the definable elements, and use this to establish that it is not fully abstract

Non-null Infinitesimal Micro-steps: a Metric Temporal Logic Approach

Many systems include components interacting with each other that evolve with possibly very different speeds. To deal with this situation many formal models adopt the abstraction of "zero-time transitions", which do not consume time. These however have several drawbacks in terms of naturalness and logic consistency, as a system is modeled to be in different states at the same time. We propose a novel approach that exploits concepts from non-standard analysis to introduce a notion of micro- and macro-steps in an extension of the TRIO metric temporal logic, called X-TRIO. We use X-TRIO to provide a formal semantics and an automated verification technique to Stateflow-like notations used in the design of flexible manufacturing systems.Comment: 20 pages, 2 figures, submitted to the conference "FORMATS: Formal Modelling and Analysis of Timed Systems" 201

Real-time and Probabilistic Temporal Logics: An Overview

Over the last two decades, there has been an extensive study on logical formalisms for specifying and verifying real-time systems. Temporal logics have been an important research subject within this direction. Although numerous logics have been introduced for the formal specification of real-time and complex systems, an up to date comprehensive analysis of these logics does not exist in the literature. In this paper we analyse real-time and probabilistic temporal logics which have been widely used in this field. We extrapolate the notions of decidability, axiomatizability, expressiveness, model checking, etc. for each logic analysed. We also provide a comparison of features of the temporal logics discussed

Grounding the Lexical Semantics of Verbs in Visual Perception using Force Dynamics and Event Logic

This paper presents an implemented system for recognizing the occurrence of events described by simple spatial-motion verbs in short image sequences. The semantics of these verbs is specified with event-logic expressions that describe changes in the state of force-dynamic relations between the participants of the event. An efficient finite representation is introduced for the infinite sets of intervals that occur when describing liquid and semi-liquid events. Additionally, an efficient procedure using this representation is presented for inferring occurrences of compound events, described with event-logic expressions, from occurrences of primitive events. Using force dynamics and event logic to specify the lexical semantics of events allows the system to be more robust than prior systems based on motion profile