Dorsal stream : from algorithm to neuroscience

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 173-195).The dorsal stream in the primate visual cortex is involved in the perception of motion and the recognition of actions. The two topics, motion processing in the brain, and action recognition in videos, have been developed independently in the field of neuroscience and computer vision. We present a dorsal stream model that can be used for the recognition of actions as well as explaining neurophysiology in the dorsal stream. The model consists of a spatio-temporal feature detectors of increasing complexity: an input image sequence is first analyzed by an array of motion sensitive units which, through a hierarchy of processing stages, lead to position and scale invariant representation of motion in a video sequence. The model outperforms or on par with the state-of-the-art computer vision algorithms on a range of human action datasets. We then describe the extension of the model into a high-throughput system for the recognition of mouse behaviors in their homecage. We provide software and a very large manually annotated video database used for training and testing the system. Our system outperforms a commercial software and performs on par with human scoring, as measured from the ground-truth manual annotations of more than 10 hours of videos of freely behaving mice. We complete the neurobiological side of the model by showing it could explain the motion processing as well as action selectivity in the dorsal stream, based on comparisons between model outputs and the neuronal responses in the dorsal stream. Specifically, the model could explain pattern and component sensitivity and distribution [161], local motion integration [97], and speed-tuning [144] of MT cells. The model, when combining with the ventral stream model [173], could also explain the action and actor selectivity in the STP area. There exists only a few models for the motion processing in the dorsal stream, and these models were not be applied to the real-world computer vision tasks. Our model is one that agrees with (or processes) data at different levels: from computer vision algorithm, practical software, to neuroscience.by Hueihan Jhuang.Ph.D

A biologically inspired system for action recognition

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2007.Includes bibliographical references (p. 51-58).We present a biologically-motivated system for the recognition of actions from video sequences. The approach builds on recent work on object recognition based on hierarchical feedforward architectures and extends a neurobiological model of motion processing in the visual cortex. The system consists of a hierarchy of spatio-temporal feature detectors of increasing complexity: an input sequence is first analyzed by an array of motion-direction sensitive units which, through a hierarchy of processing stages, lead to position-invariant spatio-temporal feature detectors. We experiment with different types of motion-direction sensitive units as well as different system architectures. Besides, we find that sparse features in intermediate stages outperform dense ones and that using a simple feature selection approach leads to an efficient system that performs better with far fewer features. We test the approach on different publicly available action datasets, in all cases achieving the best results reported to date.by Hueihan Jhuang.S.M

Territoriality, Resistance and Indigenous Development in Protected Areas: A Political Ecology Analysis of Truku People in Eastern Taiwan

Indigenous areas in Taiwan were a ‘special administrative region’ during the Japanese colonial period (1895-1945). The Japanese police controlled the primary aspects of everyday life of indigenous people. Some policies concerning indigenous people have been continued in the post-colonial regimes of Han Chinese until now. Protected areas (PAs) have been established since the 1980s by central government when Taiwan was still under the martial law. National parks are typical protected area with rigorous conservation restrictions. Some protected areas actually overlapped with the traditional domains of indigenous people. Community conservation is a participatory protected area and has emerged around the 1990s. It is seen as a reform of fortress protected areas such as parks because it integrates both objectives of conservation and development. The rolling back of the state and empowerment of the local community are assumed to be the features of such a reformed policy. Community conservation has become popular among indigenous communities of Taiwan since 2000. This study aims to look at the interactions between state authorities and local indigenous people in PAs. Two Truku villages in east Taiwan were selected as case studies, as one is in Taroko National Park while the other conducted a community conservation project in the 2000s. Qualitative methods were employed for data collection. Drawing from the theory of political ecology, a framework is constructed drawing together human territoriality, resistance, and social impacts. This analysis framework was employed to examine the acts of state agencies and local Truku people, and social repercussions in the Truku examples in the context of PAs. Research results showed that the establishment of PAs and conservation policy implementations in PAs by state agencies were acts of internal territorialisation. Such a spatial classification restricted the locals’ exploitation of natural resources according to the imposed regulations. Through the control enforcement by state agencies and judicial authorities, conflicts between the local indigenous people and state agencies have happened. Even the co-management arrangement of the Park and the planning of scenic areas for local development revealed the domination of power by the government. These restrictions resulted in unpleasant social impacts such as difficulties of cultural practices and livelihood selections as well as the undermining social capital in the local indigenous communities. Accordingly, the local Truku people mobilised resistance to the conservation interventions via individual everyday practices and collective protests. Their resistance aimed to express their sustenance demands and ethnic claims. Differences between covert and overt resistance depended on the degree of empowerment. Through the process of empowerment, local protesters gained more information and political dynamics for their collective action, open resistance. I primarily contend that the establishment of PAs and conservation policy implementations by governmental agencies, whether through parks or community conservation, are acts of internal territoriality. Territorialisation of the state tends to result in resistance by the local indigenous residents due to the negative social impacts as a result of conservation interventions. This argument also interprets the unexpected consequence, resistance of the local indigenous people, of PA policies in Taiwan. To avoid the undesired outcome of policy implementation and social cost, it is necessary to build trust between them. A participatory project which confers genuine power and accords with local norms may be feasible. Decentralised power could be the first step of a breakthrough

Expanding the Family of Grassmannian Kernels: An Embedding Perspective

Modeling videos and image-sets as linear subspaces has proven beneficial for many visual recognition tasks. However, it also incurs challenges arising from the fact that linear subspaces do not obey Euclidean geometry, but lie on a special type of Riemannian manifolds known as Grassmannian. To leverage the techniques developed for Euclidean spaces (e.g, support vector machines) with subspaces, several recent studies have proposed to embed the Grassmannian into a Hilbert space by making use of a positive definite kernel. Unfortunately, only two Grassmannian kernels are known, none of which -as we will show- is universal, which limits their ability to approximate a target function arbitrarily well. Here, we introduce several positive definite Grassmannian kernels, including universal ones, and demonstrate their superiority over previously-known kernels in various tasks, such as classification, clustering, sparse coding and hashing

HMDB: A Large Video Database for Human Motion Recognition

With nearly one billion online videos viewed everyday, an emerging new frontier in computer vision research is recognition and search in video. While much effort has been devoted to the collection and annotation of large scalable static image datasets containing thousands of image categories, human action datasets lag far behind. Current action recognition databases contain on the order of ten different action categories collected under fairly controlled conditions. State-of-the-art performance on these datasets is now near ceiling and thus there is a need for the design and creation of new benchmarks. To address this issue we collected the largest action video database to-date with 51 action categories, which in total contain around 7,000 manually annotated clips extracted from a variety of sources ranging from digitized movies to YouTube. We use this database to evaluate the performance of two representative computer vision systems for action recognition and explore the robustness of these methods under various conditions such as camera motion, viewpoint, video quality and occlusion.United States. Defense Advanced Research Projects Agency. Information Processing Techniques OfficeUnited States. Defense Advanced Research Projects Agency. System Science Division. Defense Sciences OfficeNational Science Foundation (U.S.) (NSF-0640097)National Science Foundation (U.S.) (NSF-0827427)United States. Air Force Office of Scientific Research (FA8650-05- C-7262)Adobe SystemsKing Abdullah University of Science and TechnologyNEC ElectronicsSony CorporationEugene McDermott FoundationBrown University. Center for Computing and VisualizationRobert J. and Nancy D. Carney Fund for Scientific InnovationUnited States. Defense Advanced Research Projects Agency (DARPA-BAA-09-31)United States. Office of Naval Research (ONR-BAA-11-001)Ministry of Science, Research and the Arts of Baden Württemberg, German

Trainable, vision-based automated home cage behavioral phenotyping

We describe a fully trainable computer vision system enabling the automated analysis of complex mouse behaviors. Our system computes a sequence of feature descriptors for each video sequence and a classifier is used to learn a mapping from these features to behaviors of interest. We collected a very large manually annotated video database of mouse behaviors for training and testing the system. Our system performs on par with human scoring, as measured from the ground-truth manual annotations of thousands of clips of freely behaving mice. As a validation of the system, we characterized the home cage behaviors of two standard inbred and two nonstandard mouse strains. From this data, we were able to predict the strain identity of individual mice with high accuracy.California Institute of Technology. Broad Fellows Program in Brain CircuitryNational Science Council of Taiwan (TMS-094-1-A032

Automated home-cage behavioral phenotyping of mice

We describe a trainable computer vision system enabling the automated analysis of complex mouse behaviors. We provide software and a very large manually annotated video database used for training and testing the system. Our system outperforms leading commercial software and performs on par with human scoring, as measured from the ground-truth manual annotations of thousands of clips of freely behaving animals. We show that the home-cage behavior profiles provided by the system is sufficient to accurately predict the strain identity of individual animals in the case of two standard inbred and two non-standard mouse strains. Our software should complement existing sensor-based automated approaches and help develop an adaptable, comprehensive, high-throughput, fine-grained, automated analysis of rodent behavior