The role of real-world size in object representation

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Brain and Cognitive Sciences, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 117-128).Every object in the world has a physical size which is intrinsic to how we interact with it: we pick up small objects like coins with our fingers, we throw footballs and swing tennis rackets, we orient our body to bigger objects like chairs and tables and we navigate with respect to landmarks like fountains and buildings. Here I argue that the size of objects in the world is a basic property of object representation with both behavioral and neural consequences. Specifically, I suggest that objects have a canonical visual size based on their real-world size (Chapter 2), and that we automatically access real-world size information when we recognize an object (Chapter 3). Further, I present evidence that there are neural consequences of realworld size for the large-scale organization of object knowledge in ventral visual cortex (Chapter 4). Specifically, there are regions with differential selectivity for big and small objects, that span from along the dorsal and lateral surfaces of occipito-temporal cortex in a mirrored organization. Finally, I suggest that the empirical findings can be coherently explained by thinking about the experience of an observer situated in a three-dimensional world. This work provides testable predictions about retinal size biases in visual experience, and an approach in which to understand the neural representation of any object in the world.by Talia Konkle.Ph.D

Tripartite Organization of the Ventral Stream by Animacy and Object Size

Occipito-temporal cortex is known to house visual object representations, but the organization of the neural activation patterns along this cortex is still being discovered. Here we found a systematic, large-scale structure in the neural responses related to the interaction between two major cognitive dimensions of object representation: animacy and real-world size. Neural responses were measured with functional magnetic resonance imaging while human observers viewed images of big and small animals and big and small objects. We found that real-world size drives differential responses only in the object domain, not the animate domain, yielding a tripartite distinction in the space of object representation. Specifically, cortical zones with distinct response preferences for big objects, all animals, and small objects, are arranged in a spoked organization around the occipital pole, along a single ventromedial, to lateral, to dorsomedial axis. The preference zones are duplicated on the ventral and lateral surface of the brain. Such a duplication indicates that a yet unknown higher-order division of labor separates object processing into two substreams of the ventral visual pathway. Broadly, we suggest that these large-scale neural divisions reflect the major joints in the representational structure of objects and thus place informative constraints on the nature of the underlying cognitive architecture.Psycholog

Visual Awareness Is Limited by the Representational Architecture of the Visual System

Visual perception and awareness have strict limitations. We suggest that one source of these limitations is the representational architecture of the visual system. Under this view, the extent to which items activate the same neural channels constrains the amount of information that can be processed by the visual system and ultimately reach awareness. Here, we measured how well stimuli from different categories (e.g., faces and cars) blocked one another from reaching awareness using two distinct paradigms that render stimuli invisible: visual masking and continuous flash suppression. Next, we used fMRI to measure the similarity of the neural responses elicited by these categories across the entire visual hierarchy. Overall, we found strong brain–behavior correlations within the ventral pathway, weaker correlations in the dorsal pathway, and no correlations in early visual cortex (V1–V3). These results suggest that the organization of higher level visual cortex constrains visual awareness and the overall processing capacity of visual cognition.National Science Foundation (U.S.). Graduate Research FellowshipNational Institutes of Health (U.S.). Ruth L. Kirschstein National Research Service Award (F32EY024483

Real-World Objects Are Not Represented as Bound Units: Independent Forgetting of Different Object Details from Visual Memory

Are real-world objects represented as bound units? Although a great deal of research has examined binding between the feature dimensions of simple shapes, little work has examined whether the featural properties of real-world objects are stored in a single unitary object representation. In a first experiment, we found that information about an object's color is forgotten more rapidly than the information about an object's state (e.g., open, closed), suggesting that observers do not forget objects as entirely bound units. In a second and third experiment, we examined whether state and exemplar information are forgotten separately or together. If these properties are forgotten separately, the probability of getting one feature correct should be independent of whether the other feature was correct. We found that after a short delay, observers frequently remember both state and exemplar information about the same objects, but after a longer delay, memory for the two properties becomes independent. This indicates that information about object state and exemplar are forgotten separately over time. We thus conclude that real-world objects are not represented in a single unitary representation in visual memory.Psycholog

Visual Long-Term Memory Has the Same Limit on Fidelity as Visual Working Memory

Visual long-term memory can store thousands of objects with surprising visual detail, but just how detailed are these representations, and how can one quantify this fidelity? Using the property of color as a case study, we estimated the precision of visual information in long-term memory, and compared this with the precision of the same information in working memory. Observers were shown real-world objects in random colors and were asked to recall the colors after a delay. We quantified two parameters of performance: the variability of internal representations of color (fidelity) and the probability of forgetting an object’s color altogether. Surprisingly, the fidelity of color information in long-term memory was comparable to the asymptotic precision of working memory. These results suggest that long-term memory and working memory may be constrained by a common limit, such as a bound on the fidelity required to retrieve a memory representation.Psycholog

A Real-World Size Organization of Object Responses in Occipitotemporal Cortex

While there are selective regions of occipitotemporal cortex that respond to faces, letters, and bodies, the large-scale neural organization of most object categories remains unknown. Here, we find that object representations can be differentiated along the ventral temporal cortex by their real-world size. In a functional neuroimaging experiment, observers were shown pictures of big and small real-world objects (e.g., table, bathtub; paperclip, cup), presented at the same retinal size. We observed a consistent medial-to-lateral organization of big and small object preferences in the ventral temporal cortex, mirrored along the lateral surface. Regions in the lateral-occipital, inferotemporal, and parahippocampal cortices showed strong peaks of differential real-world size selectivity and maintained these preferences over changes in retinal size and in mental imagery. These data demonstrate that the real-world size of objects can provide insight into the spatial topography of object representation.National Science Foundation (U.S.). Graduate Research Fellowship ProgramNational Eye Institute (Grant EY020484

A review of visual memory capacity: Beyond individual items and toward structured representations.

Traditional memory research has focused on identifying separate memory systems and exploring different stages of memory processing. This approach has been valuable for establishing a taxonomy of memory systems and characterizing their function but has been less informative about the nature of stored memory representations. Recent research on visual memory has shifted toward a representation-based emphasis, focusing on the contents of memory and attempting to determine the format and structure of remembered information. The main thesis of this review will be that one cannot fully understand memory systems or memory processes without also determining the nature of memory representations. Nowhere is this connection more obvious than in research that attempts to measure the capacity of visual memory. We will review research on the capacity of visual working memory and visual long-term memory, highlighting recent work that emphasizes the contents of memory. This focus impacts not only how we estimate the capacity of the systemVgoing beyond quantifying how many items can be remembered and moving toward structured representationsVbut how we model memory systems and memory processes. Keywords: memory, working memory, long-term memory, visual cognition, memory capacity, memory fidelity Citation: Brady, T. F., Konkle, T., & Alvarez, G. A. (2011). A review of visual memory capacity: Beyond individual items and toward structured representations. Journal of Vision, 11(5):4, 1-34, http://www.journalofvision.org/content/11/5/4, doi:10.1167/11.5.4. Introduction Tulving Early on, William James (1890) proposed the distinction between primary memoryVthe information held in the "conscious present"Vand secondary memory, which consists of information that is acquired, stored outside of conscious awareness, and then later remembered. This distinction maps directly onto the modern distinction between short-term memory (henceforth working memory) and long-term memory The emphasis on memory systems and memory processes has been quite valuable in shaping cognitive and neural models of memory. In general, this approach aims to characterize memory systems in a way that generalizes over representational content Research on visual perception takes the opposite approach, attempting to determine what is being represented and to generalize across processes. For example, early stages of visual representation consist of orientation and spatial frequency features. Vision research has measured the properties of these features, such as their tuning curves and sensitivity (e.g., Here, we review recent research in the domains of visual working memory and visual long-term memory, focusing on how models of these memory systems are altered and refined by taking the contents of memory into account. Visual working memory The working memory system is used to hold information actively in mind and to manipulate that information to perform a cognitive task The study of visual working memory has largely focused on the capacity of the system, both because limited capacity is one of the main hallmarks of working memory and because individual differences in measures of working memory capacity are correlated with differences in fluid intelligence, reading comprehension, and academic achievement In the broader working memory literature, a significant amount of research has focused on characterizing memory limits based on how quickly information can be refreshed (e.g., Here, we review research that focuses on working memory representations, including their fidelity, structure, and effects of stored knowledge. While not an exhaustive review of the literature, these examples highlight the fact that working memory representations have a great deal of structure beyond the level of individual items. This structure can be characterized as a hierarchy of properties, from individual features to individual objects to acrossobject ensemble features (spatial context and featural context). Together, the work reviewed here illustrates how a representation-based approach has led to important advances, not just in understanding the nature of stored representations themselves but also in characterizing working memory capacity and shaping models of visual working memory. The fidelity of visual working memory Recent progress in modeling visual working memory has resulted from an emphasis on estimating the fidelity of visual working memory representations. In general, the capacity of any memory system should be characterized both in terms of the number of items that can be stored and in terms of the fidelity with which each individual item can be stored. Consider the case of a USB drive that can store exactly 1000 images: the number of images alone is not a complete estimate of this USB drive's storage capacity. It is also important to consider the resolution with which those images can be stored: if each image can be stored with a very low resolution, say 16  16 pixels, then the drive has a lower capacity than if it can store the same number of images with a high resolution, say 1024  768 pixels. In general, the true capacity of a memory system can be estimated by multiplying the maximum number of items that can be stored by the fidelity with which each individual item can be stored (capacity = quantity  fidelity). For a memory system such as your USB drive, there is only an information limit on memory storage, so the number of files that can be stored is limited only by the size of those files. Whether visual working memory is best characterized as an information- or whether it has a predetermined and fixed item limit Importantly, this standard change detection paradigm provides little information about how well each individual object was remembered. The change detection paradigm indicates only that items were remembered with sufficient fidelity to distinguish an object's color from a categorically different color. How much information do observers actually remember about each object? Several new methods have been used to address this question (see Journal of Vision Fidelity of storage for complex objects While early experiments using large changes in a change detection paradigm found evidence for a slot model, in which memory is limited to storing a fixed number of items, subsequent experiments with newer paradigms that focused on the precision of memory representations have suggested an information-limited model. Specifically, This result was not due to an inability to discriminate the more complex shapes, such as 3D cubes: observers could easily detect a change between cubes when only a single cube was remembered, but they could not detect the same change when they tried to remember 4 cubes. This result suggests that encoding additional items reduced the resolution with which each individual item could be remembered, consistent with the idea that there is an information limit on memory. Using the same paradigm but varying the difficulty of the memory test, Awh, Fidelity of simple feature dimensions While the work of Alvarez and Cavanagh Wilken and Ma's However, Zhang and Luck Conclusion To summarize, by focusing on the contents of visual working memory, and on the fidelity of representations in particular, there has been significant progress in models of visual working memory and its capacity. At present, there is widespread agreement in the visual working memory literature that visual working memory has an extremely limited capacity and that it can represent 1 item with greater fidelity than 3-4 items. This finding requires the conclusion that working memory is limited by a resource that is shared among the representations of different items (i.e., information-limited). Some models claim that resource allocation is discrete and quantized into slots Research on the fidelity of working memory places important constraints on both continuous and discrete models. If working memory is slot-limited, then those slots must be recast as a flexible resource, all of which can be allocated to a single item to gain precision in its representation or which can be divided separately among multiple items yielding relatively low-resolution representations of each item. If memory capacity is informationlimited, then it is necessary to explain why under some conditions it appears that there is an upper bound on memory storage of 3-4 objects (e.g., The representation of features vs. objects in visual working memory Any estimate of memory capacity must be expressed with some unit, and what counts as the appropriate unit depends upon how information is represented. Since George Miller's (1956) seminal paper claiming a limit of 7 T 2 chunks as the capacity of working memory, a significant amount of work has attempted to determine the units of storage in working memory. In the domain of verbal memory, for example, debate has flourished about the extent to which working memory capacity is limited by storing a fixed number of chunks vs. time-based decay Objects are not always encoded in their entirety A significant body of work has demonstrated that observers do not always encode objects in their entirety. When multiple features of an object appear on distinct object parts, observers are significantly impaired at representing the entire object Costs for encoding multiple features within an object Furthermore, another body of work has demonstrated that encoding more than one feature of the same object does not always come without cost. In addition to limits on the number of values that may be stored within a particular feature dimension, data on the fidelity of representations suggest that even separate visual features from the same object are not stored completely independently. In an elegant design combining elements of the original work of Benefits of object-based storage beyond separate buffers While observers cannot completely represent 3-4 objects independently of their information load, there is a benefit to encoding multiple features from the same object compared to the same number of features on different objects Journal of Vision (2011) 11(5):4, 1-34 Brady, Konkle, & Alvarez 6 Jiang showed that it is easier to remember the color and orientation of 2 objects (4 features in total) than the color of 2 objects and the orientation of 2 separate objects (still 4 features in total). In addition, while Conclusion So what is the basic unit of representation in visual working memory? While there are significant benefits to encoding multiple features of the same object compared to multiple features across different objects (e.g., One possibility is that the initial encoding process is object-based (or location-based), but that the "unit" of visual working memory is a hierarchically structured feature bundle This proposal for the structure of memory representations is consistent with the full pattern of evidence described above, including the benefit for remembering multiple features from the same objects relative to different objects and the cost for remembering multiple features from the same object. Moreover, this hierarchical working memory theory is consistent with evidence showing a specific impairment in object-based working memory when attention is withdrawn from items (e.g. Furthermore, there is some direct evidence for separate capacities for feature-based and object-based working memory representations, with studies showing separable priming effects and memory capacities It is important to note that our proposed hierarchical feature bundle model is not compatible with a straightforward item-based or chunk-based model of working memory capacity. A key part of such proposals (e.g., Thus far, we have considered only the structure of individual items in working memory. Next, we review research demonstrating that working memory representations include another level of organization that represents properties that are computed across sets of items. Interactions between items in visual working memory In the previous two sections, we discussed the representation of individual items in visual working memory. However, research focusing on contextual effects in memory demonstrates that items are not stored in memory completely independent of one another. In particular, several studies have shown that items are encoded along with spatial context information (the spatial layout of items in the display) and with featural context information (the ensemble statistics of items in the display). These results suggest that visual working memory representations have a great deal of structure beyond the individual Journal of Vision Influences of spatial context Visual working memory paradigms often require observers to remember not only the featural properties of items (size, color, shape, identity) but also where those items appeared in the display. In these cases, memory for the features of individual items may be dependent on spatial working memory as well (for a review of spatial working memory, see Influence of feature context or "ensemble statistics" In addition to spatial context effects on item memory, it is likely that there are feature context effects as well. For instance, even in a display of squares with random colors, some displays will tend to have more "warm colors" on average, whereas others will have more "cool colors" on average, and others still will have no clear across-item structure. This featural context, or "ensemble statistics" (Alvarez, 2011), could influence memory for individual items (e.g., Given that ensemble information would be useful for remembering individual items, it is important to consider the possibility that these ensemble statistics will influence Journal of Vision Perceptual grouping and dependence between items Other research has shown that items tend to be influenced by the other items in visual working memory, although such work has not explicitly attempted to distinguish influences due to the storage of individual items and influences from ensemble statistics. For example, Lin and Luck (2008; using colored squares) and Viswanathan, Perl, Bisscher, Kahana, and Sekuler (2010; using Gabor stimuli) showed improved memory performance when items appear more similar to one another (see also Cases of explicit perceptual grouping make the nonindependence between objects even more clear. For example, Woodman, Vecera, and Luck Perceptual grouping vs. chunking vs. hierarchically structured memory What is the relationship between perceptual grouping, chunking, and the hierarchically structured memory model we have described? Perceptual grouping and chunking are both processes by which multiple elements are combined into a single higher order description. For example, a series of 10 evenly spaced dots could be grouped into a single line, and the letters F, B, and I can be chunked into the familiar acronym FBI (e.g., :4, 1-34 Brady, Konkle, & Alvarez 9 assume that the only limits on memory capacity come from the number of chunks or groups that can be encoded Conclusion Taken together, these results provide significant evidence that individual items are not represented independent of other items on the same display and that visual working memory stores information beyond the level of individual items. Put another way, every display has multiple levels of structure, from the level of feature representations to individual items to the level of groups or ensembles, and these levels of structure interact. It is important to note that these levels of structure exist and vary across trials, even if the display consists of randomly positioned objects that have randomly selected feature values. The visual system efficiently extracts and encodes structure from the spatial and featural information across the visual scene, even when, in the long run over displays, there may not be any consistent regularities. This suggests that any theory of visual working memory that specifies only the representation of individual items or groups cannot be a complete model of visual working memory. The effects of stored knowledge on visual working memory Most visual working memory research requires observers to remember meaningless, unrelated items, such as randomly selected colors or shapes. This is done to minimize the role of stored knowledge and to isolate working memory limitations from long-term memory. However, in the real world, working memory does not operate over meaningless, unrelated items. Observers have stored knowledge about most items in the real world, and this stored knowledge constrains what features and objects we expect to see and where we expect to see them. The role of such stored knowledge in modulating visual working memory representations has been controversial. In the broader working memory literature, there is clear evidence of the use of stored knowledge to increase the number of items remembered in working memory Biases from stored knowledge One uncontroversial effect of long-term memory on working memory is that there are biases in working memory resulting from prototypes or previous experience. For example, Stored knowledge effects on memory capacity While these biases in visual working memory representations are systematic and important, they do not address the question of whether long-term knowledge can be used to store more items in visual working memory. This Journal of Vision In contrast to this earlier work, Brady, Konkle, and Alvarez (2009) have recently shown clear effects of learned knowledge on working memory. In their paradigm, observers were shown standard working memory stimuli in which they had to remember the color of multiple objects It is possible that Brady, Konkle, and Alvarez (2009) found evidence for the use of stored knowledge in working memory coding because their paradigm teaches associations between items rather than attempting to make the items themselves more familiar. For instance, seeing the same set of colors for hundreds of trials might not improve the encoding of colors or shapes, because the One group of observers saw certain color pairs more often than others (e.g., yellow and green might occur next to each other 80% of the time), whereas the other group saw completely random color pairs. For the group that saw repeated color pairs, the number of color remembered increased across blocks, nearly doubling the number remembered by the random group by the end of the session. Journal of Vision Conclusion Observers have stored knowledge about most items in the real world, and this stored knowledge constrains what features and objects we expect to see and where we expect to see them. There is significant evidence that the representation of items in working memory is dependent on this stored knowledge. Thus, items for which we have expertise, like faces, are represented with more fidelity Visual working memory conclusion A great deal of research on visual working memory has focused on how to characterize the capacity of the system. We have argued that in order to characterize working memory capacity, it is important to take into account both the number of individual items remembered and the fidelity with which each individual item is remembered. Moreover, it is necessary to specify what the units of working memory storage are, how multiple units in memory interact, and how stored knowledge affects the representation of information in memory. In general, we believe that theories and models of working memory must be expanded to include memory representations that go beyond the representation of individual items and include hierarchically structured representations, both at the individual item level (hierarchical feature bundles) and across individual items. There is considerable evidence that working memory representations are not based on independent items, that working memory also stores ensembles that summarize the spatial and featural information across the display, and further, that there are interactions between working memory and stored knowledge even in simple displays. Moving beyond individual items toward structured representations certainly complicates any attempt to estimate working memory capacity. The answer to how many items can you hold in visual working memory depends on what kind of items you are trying to remember, how precisely they must be remembered, how they are presented on the display, and your history with those items. Even representations of simple items have structure at multiple levels. Thus, models that wish to accurately account for the full breadth of data and memory phenomena must make use of structured representations, especially as we move beyond colored dot objects probed by their locations toward items with more featural dimensions or toward real-world objects in scenes. Visual long-term memory Before discussing the capacity of long-term memory, it is important to make the distinction between visual longterm memory and stored knowledge. By "visual long-term memory," we refer to the ability to explicitly remember an image that was seen previously but that has not been continuously held actively in mind. Thus, visual long-term memory is the passive storage and subsequent retrieval of visual episodic information. By "stored knowledge," we refer to the preexisting visual representations that underlie our ability to perceive and recognize visual input. For example, when we first see an image, say of a red apple, stored knowledge about the visual form and features of apples in general enables us to recognize the object as such. If we are shown another picture of an apple hours later, vis

Compression in visual working memory: Using statistical regularities to form more efficient memory representations

The information that individuals can hold in working memory is quite limited, but researchers have typically studied this capacity using simple objects or letter strings with no associations between them. However, in the real world there are strong associations and regularities in the input. In an information theoretic sense, regularities introduce redundancies that make the input more compressible. The current study shows that observers can take advantage of these redundancies, enabling them to remember more items in working memory. In 2 experiments, covariance was introduced between colors in a display so that over trials some color pairs were more likely to appear than other color pairs. Observers remembered more items from these displays than from displays where the colors were paired randomly. The improved memory performance cannot be explained by simply guessing the high-probability color pair, suggesting that observers formed more efficient representations to remember more items. Further, as observers learned the regularities, their working memory performance improved in a way that is quantitatively predicted by a Bayesian learning model and optimal encoding scheme. These results suggest that the underlying capacity of the individuals ’ working memory is unchanged, but the information they have to remember can be encoded in a more compressed fashion