ROTUNDE - A Smart Meeting Cinematography Initiative: Tools, Datasets, and Benchmarks for Cognitive Interpretation and Control

We construe smart meeting cinematography with a focus on professional situations such as meetings and seminars, possibly conducted in a distributed manner across socio-spatially separated groups. The basic objective in smart meeting cinematography is to interpret professional interactions involving people, and automatically produce dynamic recordings of discussions, debates, presentations etc in the presence of multiple communication modalities. Typical modalities include gestures (e.g., raising one's hand for a question, applause), voice and interruption, electronic apparatus (e.g., pressing a button), movement (e.g., standing-up, moving around) etc. ROTUNDE, an instance of smart meeting cinematography concept, aims to: (a) develop functionality-driven benchmarks with respect to the interpretation and control capabilities of human-cinematographers, real-time video editors, surveillance personnel, and typical human performance in everyday situations; (b) Develop general tools for the commonsense cognitive interpretation of dynamic scenes from the viewpoint of visuo-spatial cognition centred perceptual narrativisation. Particular emphasis is placed on declarative representations and interfacing mechanisms that seamlessly integrate within large-scale cognitive (interaction) systems and companion technologies consisting of diverse AI sub-components. For instance, the envisaged tools would provide general capabilities for high-level commonsense reasoning about space, events, actions, change, and interaction.Comment: Appears in AAAI-2013 Workshop on: Space, Time, and Ambient Intelligence (STAMI 2013

Between Sense and Sensibility: Declarative narrativisation of mental models as a basis and benchmark for visuo-spatial cognition and computation focussed collaborative cognitive systems

What lies between `\emph{sensing}' and `\emph{sensibility}'? In other words, what kind of cognitive processes mediate sensing capability, and the formation of sensible impressions ---e.g., abstractions, analogies, hypotheses and theory formation, beliefs and their revision, argument formation--- in domain-specific problem solving, or in regular activities of everyday living, working and simply going around in the environment? How can knowledge and reasoning about such capabilities, as exhibited by humans in particular problem contexts, be used as a model and benchmark for the development of collaborative cognitive (interaction) systems concerned with human assistance, assurance, and empowerment? We pose these questions in the context of a range of assistive technologies concerned with \emph{visuo-spatial perception and cognition} tasks encompassing aspects such as commonsense, creativity, and the application of specialist domain knowledge and problem-solving thought processes. Assistive technologies being considered include: (a) human activity interpretation; (b) high-level cognitive rovotics; (c) people-centred creative design in domains such as architecture & digital media creation, and (d) qualitative analyses geographic information systems. Computational narratives not only provide a rich cognitive basis, but they also serve as a benchmark of functional performance in our development of computational cognitive assistance systems. We posit that computational narrativisation pertaining to space, actions, and change provides a useful model of \emph{visual} and \emph{spatio-temporal thinking} within a wide-range of problem-solving tasks and application areas where collaborative cognitive systems could serve an assistive and empowering function.Comment: 5 pages, research statement summarising recent publication

Cognitive Interpretation of Everyday Activities - Toward Perceptual Narrative Based Visuo-Spatial Scene Interpretation

We position a narrative-centred computational model for high-level knowledge representation and reasoning in the context of a range of assistive technologies concerned with visuo-spatial perception and cognition tasks. Our proposed narrative model encompasses aspects such as space, events, actions, change, and interaction from the viewpoint of commonsense reasoning and learning in large-scale cognitive systems. The broad focus of this paper is on the domain of human-activity interpretation in smart environments, ambient intelligence etc. In the backdrop of a smart meeting cinematography domain, we position the proposed narrative model, preliminary work on perceptual narrativisation, and the immediate outlook on constructing general-purpose open-source tools for perceptual narrativisation

Accurately Measuring the Satisfaction of Visual Properties in Virtual Camera Control

International audienceAbstract. Declarative approaches to camera control model inputs as properties on the camera and then rely on constraint-based and/or optimization techniques to compute the camera parameters or paths that best satisfy those properties. To reach acceptable performances, such approaches often (if not always) compute properties satisfaction in an approximate way. Therefore, it is difficult to measure results in terms of accuracy, and also compare approaches that use different approxima- tions. In this paper, we propose a simple language which can be used to express most of the properties proposed in the literature and whose semantics provide a way to accurately measure their satisfaction. The language can be used for several purposes, for example to measure how accurate a specific approach is and to compare two distinct approaches in terms of accuracy

Techniques de mise en scène pour le jeu vidéo et l'animation

Eurographics State of the Art Report (STAR).International audienceOver the last forty years, researchers in computer graphics have proposed a large variety of theoretical models and computer implementations of a virtual film director, capable of creating movies from minimal input such as a screenplay or storyboard. The underlying film directing techniques are also in high demand to assist and automate the generation of movies in computer games and animation. The goal of this survey is to characterize the spectrum of applications that require film directing, to present a historical and up-to-date summary of research in algorithmic film directing, and to identify promising avenues and hot topics for future research.Depuis quarante ans, les chercheurs en informatique graphique ont proposé une grande variété de modèles théoriques et d'implémentations de réalisateurs virtuels, capables de créer des films automatiquement à partir de scénarios ou de storyboards. Les techniques de mise en scène sous-jacentes peuvent également être très utiles pour assister et automatiser la création de films dans le jeu vidéo et l'animation. Le but de cet état de l'art est de caractériser le spectre des applications qui peuvent bénéficier des techniques de mise en scène, de donner un compte rendu historique de la recherche en mise en scène algorithmique, et d'identifier les tendances et perspectives du domaine

Applications of CSP solving in computer games (camera control)

While camera control systems of commercial 3D games have improved greatly in recent years, they are not as fully developed as are other game components such as graphics and physics engines. Bourne and Sattar (2006) have proposed a reactive constraint based third person perspective camera control system. We have extended the capability of their system to handle occlusion while following the main character, and have used camera cuts to find appropriate camera positions for a few difficult situations. We have developed a reactive constraint based third person perspective chase camera control system to follow a character in a 3D environment. The camera follows the character from (near) optimal positions defined by a camera profile. The desired values of the height and distance constraints of the camera profile are changed appropriately whenever the character enters a semi-enclosed or an enclosed area, and the desired value of the orientation constraint of the camera profile is changed incrementally whenever theoptimal camera view is obstructed. Camera cuts are used whenever the main character backs up to a wall or any other obstructions, or comes out of a semi-enclosed or an enclosed area. Two auxiliary cameras to observe the main camera positions from top and side views have been added. The chase camera control system achieved real-time performance while following the main character in a typical 3D environment, and maintained an optimal view based on a user specified/selected camera profile

CameraBots: Cinematography for Games with Non-Player Characters as Camera Operators

Cinematography can be defined as the art of film making [1]. Among other things, it describes principles and techniques pertaining to the effective use of cameras to film live action. The correct application of these principles and techniques produces filmed content that is more engaging, compelling and absorbing for the viewer. 3D computer games employ virtual cameras in order to provide the player with an appropriate view of the game world. These virtual cameras can simulate all of the functionality of their real-world counterparts yet little effort is usually made to incorporate cinematographic techniques and principles into their operation. Typically, severe constraints are placed on the positioning of these cameras: for example a third-person camera is positioned at a fixed distance behind the player’s avatar (the character that the player controls), and a first person camera directly simulates the avatar’s viewpoint. The exception to this is the case of non-interactive cut-scenes where more sophisticated camera work is common. In this paper we describe our work on enabling the virtual camera in a 3D computer game to employ principles from cinematography throughout the game play. The successful employment of this approach can result in a more dramatic and compelling experience as the full arsenal of cinematic camera operations, such as close-ups, pans, tilts, zooms and so on, are potentially available. Cinematography provides guidelines as to how these can be used in order to make the viewer more engrossed in the action [5], and also advises how to employ consistent camera work to prevent the viewer from becoming disoriented, a common occurrence with current configurations in games. Certain camera angles or movements can be used to inform the viewer about imminent events (e.g. the camera may focus on a door when a person is about to walk through it) or to help them interpret the events on the screen. Conversely, for dramatic effect, certain events or parts of a scene can be hidden from view until the appropriate time. Cinematography achieves much of its effect by making appropriate cuts between different camera positions at the correct instances [1]. This presents an immediate problem as games typically rely on a single virtual camera and therefore it is not possible to make cuts. We solve this problem by introducing multiple cameras controlled by CameraBots, autonomous agents within a game whose role it is to film the action in much the same way that real camera operators do on a film set. These CameraBots are closely modelled on the existing Non-Player Characters (NPCs) [2, 3, 4, 6] found in most game engines. They can navigate around the game world but do not participate in the action, and hence are not rendered onscreen. Multiple CameraBots will typically be active at any instant during the gameplay, and the system can thus cut between the views that they provide. We describe five classes of CameraBot, each of which employs guidelines from cinematography in order to orient and position itself to accomplish a particular type of shot. The EstablishingCBot is designed to provide establishing shots for a particular scene. This involves filming from a sufficient distance and appropriate angle such that a good proportion of it and the characters in it are visible [5]. It is often used when the action moves to a new setting. The CharacterCBot shoots character shots which frame one or more characters. The CloseUpCBot shoots closer and more dramatic shots of a single character. The FirstPersonCBot films through the avatar’s eyes, as employed in first-person shooter games, and is used when the player requires close control and accuracy. The OTSCBot provides over-the-shoulder shots that follow the avatar when moving. In practice this means the bot is positioned directly behind the avatar and gives a good general view of what’s ahead and where the avatar is positioned in the setting. The CameraBots have various parameters which can be used to specify which events to film (or for the CharacterCBot and CloseUpCBot which characters to film) and what style, e.g. steady or hand-held, to use. In our implementation we are using the existing code that drives NPCs in the Quake II game engine to create our CameraBots. This provides us with an established method of adding artificially intelligent characters to a game and so we can harness functionality already present. In order to coordinate the CameraBots such that guidelines for shooting different types of scenes may be employed, we introduce two additional entities, the Director module and the Cinematographer module. The Director continually examines the game and uses criteria informed by cinematography to decide what action is to be filmed. These include whether or not the action being examined relates to the avatar (the protagonist from a cinematic point-of-view) and how much character interaction is occurring relative to that in other parts of the game. The Cinematographer examines the selected action and chooses a suitable method to use to film it. The Director may provide input into the choice of method. The role of the Cinematographer then involves introducing and removing CameraBots, telling them what to film, and cutting between the resultant views at the appropriate time. Of great importance is that the camera work produced does not prevent the game player from carrying out required tasks. We incorporate task specific information into our camera system to ensure this does not occur. We also consider providing views to game spectators in addition to players. In this instance it is possible to employ more concepts from cinematography since task-relevant views are not required. References 1. Brown, B. (2002). Cinematography: Image Making for Cinematographers, Directors and Videographers. Oxford: Focal. 2. Fairclough, C., Fagan, M., Mac Namee, B. and Cunningham, P. (2001). Research Directions for AI in Computer Games. Proceedings of the Twelfth Irish Conference on Artificial Intelligence and Cognitive Science pp. 333 – 344, 2001. 3. Laird, J.E. and Duchi, J.C. (2000). Creating Human-Like Synthetic Characters with Multiple Skill Levels: A Case Study Using the Soar Quakebot. AAAI tech. report, SS-00-03, AAAI Press, Menlo Park, Calif., 2000. 4. Laird, J. E. (2000). It Knows What You’re Going To Do: Adding anticipation to a QuakeBot. AAAI 2000 Spring Symposium on Artificial Intelligence and Interactive Entertainment. AAAI Technical Report SS00–02. Menlo Park, CA: AAAI Press. 5. Mascelli, J. V. (1965). The Five C’s of Cinematography. Los Angeles: Silman-James Press. 6. Reynolds, C. (1999). Steering Behaviors For Autonomous Characters. Game Developers Conference 1999

A Prolog application for reasoning on maths puzzles with diagrams

open5noDespite the indisputable progresses of artificial intelligence, some tasks that are rather easy for a human being are still challenging for a machine. An emblematic example is the resolution of mathematical puzzles with diagrams. Sub-symbolical approaches have proven successful in fields like image recognition and natural language processing, but the combination of these techniques into a multimodal approach towards the identification of the puzzle’s answer appears to be a matter of reasoning, more suitable for the application of a symbolic technique. In this work, we employ logic programming to perform spatial reasoning on the puzzle’s diagram and integrate the deriving knowledge into the solving process. Analysing the resolution strategies required by the puzzles of an international competition for humans, we draw the design principles of a Prolog reasoning library, which interacts with image processing software to formulate the puzzle’s constraints. The library integrates the knowledge from different sources, and relies on the Prolog inference engine to provide the answer. This work can be considered as a first step towards the ambitious goal of a machine autonomously solving a problem in a generic context starting from its textual-graphical presentation. An ability that can help potentially every human–machine interaction.openBuscaroli, Riccardo; Chesani, Federico; Giuliani, Giulia; Loreti, Daniela; Mello, PaolaBuscaroli, Riccardo; Chesani, Federico; Giuliani, Giulia; Loreti, Daniela; Mello, Paol