Concurrence-Aware Long Short-Term Sub-Memories for Person-Person Action Recognition

Recently, Long Short-Term Memory (LSTM) has become a popular choice to model individual dynamics for single-person action recognition due to its ability of modeling the temporal information in various ranges of dynamic contexts. However, existing RNN models only focus on capturing the temporal dynamics of the person-person interactions by naively combining the activity dynamics of individuals or modeling them as a whole. This neglects the inter-related dynamics of how person-person interactions change over time. To this end, we propose a novel Concurrence-Aware Long Short-Term Sub-Memories (Co-LSTSM) to model the long-term inter-related dynamics between two interacting people on the bounding boxes covering people. Specifically, for each frame, two sub-memory units store individual motion information, while a concurrent LSTM unit selectively integrates and stores inter-related motion information between interacting people from these two sub-memory units via a new co-memory cell. Experimental results on the BIT and UT datasets show the superiority of Co-LSTSM compared with the state-of-the-art methods

Some shortcomings of long-term working memory

Within the framework of their long-term working memory theory, Ericsson and Kintsch (1995) propose that experts rapidly store information in long-term memory through two mechanisms: elaboration of long-term memory patterns and schemas and use of retrieval structures. They use chess players’ memory as one of their most compelling sources of empirical evidence. In this paper, I show that evidence from chess memory, far from supporting their theory, limits its generality. Evidence from other domains reviewed by Ericsson and Kintsch, such as medical expertise, is not as strong as claimed, and sometimes contradicts the theory outright. I argue that Ericsson and Kintsch’s concept of retrieval structure conflates three different types of memory structures that possess quite different properties. One of these types of structures—generic, general-purpose retrieval structures—has a narrower use than proposed by Ericsson and Kintsch: it applies only in domains where there is a conscious, deliberate intent by individuals to improve their memory. Other mechanisms, including specific retrieval structures, exist that permit a rapid encoding into long-term memory under other circumstances

Learning and Production of Movement Sequences: Behavioral, Neurophysiological, and Modeling Perspectives

A growing wave of behavioral studies, using a wide variety of paradigms that were introduced or greatly refined in recent years, has generated a new wealth of parametric observations about serial order behavior. What was a mere trickle of neurophysiological studies has grown to a more steady stream of probes of neural sites and mechanisms underlying sequential behavior. Moreover, simulation models of serial behavior generation have begun to open a channel to link cellular dynamics with cognitive and behavioral dynamics. Here we summarize the major results from prominent sequence learning and performance tasks, namely immediate serial recall, typing, 2XN, discrete sequence production, and serial reaction time. These populate a continuum from higher to lower degrees of internal control of sequential organization. The main movement classes covered are speech and keypressing, both involving small amplitude movements that are very amenable to parametric study. A brief synopsis of classes of serial order models, vis-à-vis the detailing of major effects found in the behavioral data, leads to a focus on competitive queuing (CQ) models. Recently, the many behavioral predictive successes of CQ models have been joined by successful prediction of distinctively patterend electrophysiological recordings in prefrontal cortex, wherein parallel activation dynamics of multiple neural ensembles strikingly matches the parallel dynamics predicted by CQ theory. An extended CQ simulation model-the N-STREAMS neural network model-is then examined to highlight issues in ongoing attemptes to accomodate a broader range of behavioral and neurophysiological data within a CQ-consistent theory. Important contemporary issues such as the nature of working memory representations for sequential behavior, and the development and role of chunks in hierarchial control are prominent throughout.Defense Advanced Research Projects Agency/Office of Naval Research (N00014-95-1-0409); National Institute of Mental Health (R01 DC02852

Pushing Typists Back on the Learning Curve: Memory Chunking Improves Retrieval of Prior Typing Episodes

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.Hierarchical control of skilled performance depends on chunking of several lower-level units into a single higher-level unit. The present study examined the relationship between chunking and recognition of trained materials in the context of typewriting. In 3 experiments, participants were trained with typing nonwords and were later tested on their recognition of the trained materials. In Experiment 1, participants typed the same words or nonwords in 5 consecutive trials while performing a concurrent memory task. In Experiment 2, participants typed the materials with lags between repetitions without a concurrent memory task. In both experiments, recognition of typing materials was associated with better chunking of the materials. Experiment 3 used the remember-know procedure to test the recollection and familiarity components of recognition. Remember judgments were associated with better chunking than know judgments or nonrecognition. These results indicate that chunking is associated with explicit recollection of prior typing episodes. The relevance of the existing memory models to chunking in typewriting was considered, and it is proposed that memory chunking improves retrieval of trained typing materials by integrating contextual cues into the memory traces

Pushing Typists Back on the Learning Curve: Memory Chunking in the Hierarchical Control of Skilled Typewriting

Hierarchical control of skilled performance depends on the ability of higher level control to process several lower level units as a single chunk. The present study investigated the development of hierarchical control of skilled typewriting, focusing on the process of memory chunking. In the first 3 experiments, skilled typists typed words or nonwords under concurrent memory load. Memory chunks developed and consolidated into long-term memory when the same typing materials were repeated in 6 consecutive trials, but chunks did not develop when repetitions were spaced. However, when concurrent memory load was removed during training, memory chunks developed more efficiently with longer lags between repetitions than shorter lags. From these results, it is proposed that memory chunking requires 2 representations of the same letter string to be maintained simultaneously in short-term memory: 1 representation from the current trial, and the other from an earlier trial that is either retained from the immediately preceding trial or retrieved from long-term memory (i.e., study state retrieval)

The Complementary Brain: From Brain Dynamics To Conscious Experiences

How do our brains so effectively achieve adaptive behavior in a changing world? Evidence is reviewed that brains are organized into parallel processing streams with complementary properties. Hierarchical interactions within each stream and parallel interactions between streams create coherent behavioral representations that overcome the complementary deficiencies of each stream and support unitary conscious experiences. This perspective suggests how brain design reflects the organization of the physical world with which brains interact, and suggests an alternative to the computer metaphor suggesting that brains are organized into independent modules. Examples from perception, learning, cognition, and action are described, and theoretical concepts and mechanisms by which complementarity is accomplished are summarized.Defense Advanced Research Projects and the Office of Naval Research (N00014-95-1-0409); National Science Foundation (ITI-97-20333); Office of Naval Research (N00014-95-1-0657

The Complementary Brain: A Unifying View of Brain Specialization and Modularity

Defense Advanced Research Projects Agency and Office of Naval Research (N00014-95-I-0409); National Science Foundation (ITI-97-20333); Office of Naval Research (N00014-95-I-0657

Two-Stream RNN/CNN for Action Recognition in 3D Videos

The recognition of actions from video sequences has many applications in health monitoring, assisted living, surveillance, and smart homes. Despite advances in sensing, in particular related to 3D video, the methodologies to process the data are still subject to research. We demonstrate superior results by a system which combines recurrent neural networks with convolutional neural networks in a voting approach. The gated-recurrent-unit-based neural networks are particularly well-suited to distinguish actions based on long-term information from optical tracking data; the 3D-CNNs focus more on detailed, recent information from video data. The resulting features are merged in an SVM which then classifies the movement. In this architecture, our method improves recognition rates of state-of-the-art methods by 14% on standard data sets.Comment: Published in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS

Serial Position Effects in Short-term Visual Memory: A SIMPLE Explanation?

A version of Sternberg’s (1966) short-term, visual memory recognition paradigm with pictures of unfamiliar faces as stimuli was used in three experiments to assess the applicability of the distinctiveness based SIMPLE model proposed by Brown, Neath & Chater (2002). Initial simulations indicated that the amount of recency predicted increased as the parameter measuring the psychological distinctiveness of the stimulus material (c) increased, and that the amount of primacy was dependent on the extent of proactive interference from previously presented stimuli. The data from experiment 1, which used memory lists of four and five faces varying in visual similarity confirmed the predicted, extended recency effect. However, changes in visual similarity were not found to produce changes in c. In Experiments 2 and 3, the conditions that influence the magnitude of c were explored. These revealed that both the familiarity of the stimulus class before testing, and changes in familiarity due to perceptual learning, influenced distinctiveness as indexed by the parameter c. Overall the empirical data from all three experiments were well-fit by SIMPLE