A taxonomy of situated language in natural contexts

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 161-163).This thesis develops a multi-modal dataset consisting of transcribed speech along with the locations in which that speech took place. Speech with location attached is called situated language, and is represented here as spatial distributions, or two-dimensional histograms over locations in a home. These histograms are organized in the form of a taxonomy, where one can explore, compare, and contrast various slices along several axes of interest. This dataset is derived from raw data collected as part of the Human Speechome Project, and consists of semi-automatically transcribed spoken language and time-aligned overhead video collected over 15 months in a typical home environment. As part of this thesis, the vocabulary of the child before the age of two is derived from transcription, as well as the age at which the child first produced each of the 658 words in his vocabulary. Locations are derived using an efficient tracking algorithm, developed as part of this thesis, called 2C. This system maintains high accuracy when compared to similar systems, while dramatically reducing processing time, an essential feature when processing a corpus of this size. Spatial distributions are produced for many different cuts through the data, including temporal segments (i.e. morning, day, and night), speaker identities (i.e. mother, father, child), and linguistic content (i.e. per-word, aggregate by word type). Several visualization types and statistics are developed, which prove useful for organizing and exploring the dataset. It will then be shown that spatial distributions contain a wealth of information, and that this information can be exploited in various ways to derive meaningful insights and numerical results from the data.by George Macaulay Shaw.S.M

An immersive system for browsing and visualizing surveillance video

HouseFly is an interactive data browsing and visualization system that synthesizes audio-visual recordings from multiple sensors, as well as the meta-data derived from those recordings, into a unified viewing experience. The system is being applied to study human behavior in both domestic and retail situations grounded in longitudinal video recordings. HouseFly uses an immersive video technique to display multiple streams of high resolution video using a realtime warping procedure that projects the video onto a 3D model of the recorded space. The system interface provides the user with simultaneous control over both playback rate and vantage point, enabling the user to navigate the data spatially and temporally. Beyond applications in video browsing, this system serves as an intuitive platform for visualizing patterns over time in a variety of multi-modal data, including person tracks and speech transcripts.United States. Office of Naval Research (Award no. N000140910187