A Neural Model of Motion Processing and Visual Navigation by Cortical Area MST

Cells in the dorsal medial superior temporal cortex (MSTd) process optic flow generated by self-motion during visually-guided navigation. A neural model shows how interactions between well-known neural mechanisms (log polar cortical magnification, Gaussian motion-sensitive receptive fields, spatial pooling of motion-sensitive signals, and subtractive extraretinal eye movement signals) lead to emergent properties that quantitatively simulate neurophysiological data about MSTd cell properties and psychophysical data about human navigation. Model cells match MSTd neuron responses to optic flow stimuli placed in different parts of the visual field, including position invariance, tuning curves, preferred spiral directions, direction reversals, average response curves, and preferred locations for stimulus motion centers. The model shows how the preferred motion direction of the most active MSTd cells can explain human judgments of self-motion direction (heading), without using complex heading templates. The model explains when extraretinal eye movement signals are needed for accurate heading perception, and when retinal input is sufficient, and how heading judgments depend on scene layouts and rotation rates.Defense Research Projects Agency (N00014-92-J-4015); Office of Naval Research (N00014-92-J-1309, N00014-95-1-0409, N00014-95-1-0657, N00014-91-J-4100, N0014-94-I-0597); Air Force Office of Scientific Research (F49620-92-J-0334)

A Neural Model of How the Brain Computes Heading from Optic Flow in Realistic Scenes

Animals avoid obstacles and approach goals in novel cluttered environments using visual information, notably optic flow, to compute heading, or direction of travel, with respect to objects in the environment. We present a neural model of how heading is computed that describes interactions among neurons in several visual areas of the primate magnocellular pathway, from retina through V1, MT+, and MSTd. The model produces outputs which are qualitatively and quantitatively similar to human heading estimation data in response to complex natural scenes. The model estimates heading to within 1.5° in random dot or photo-realistically rendered scenes and within 3° in video streams from driving in real-world environments. Simulated rotations of less than 1 degree per second do not affect model performance, but faster simulated rotation rates deteriorate performance, as in humans. The model is part of a larger navigational system that identifies and tracks objects while navigating in cluttered environments.National Science Foundation (SBE-0354378, BCS-0235398); Office of Naval Research (N00014-01-1-0624); National-Geospatial Intelligence Agency (NMA201-01-1-2016

Multisensory Integration in Self Motion Perception

Self motion perception involves the integration of visual, vestibular, somatosensory and motor signals. This article reviews the findings from single unit electrophysiology, functional and structural magnetic resonance imaging and psychophysics to present an update on how the human and non-human primate brain integrates multisensory information to estimate one's position and motion in space. The results indicate that there is a network of regions in the non-human primate and human brain that processes self motion cues from the different sense modalities

Perceived heading during simulated torsional eye movements

AbstractObserver translation through the environment can be accompanied by rotation of the eye about any axis. For rotation about the vertical axis (horizontal rotation) during translation in the horizontal plane, it is known that the absence of depth in the scene and an extra retinal signal leads to a systematic error in the observer’s perceived direction of heading. This heading error is related in magnitude and direction to the shift of the centre of retinal flow (CF) that occurs because of the rotation. Rotation about any axis that deviates from the heading direction results in a CF shift. So far, however, the effect of rotation about the line of sight (torsion) on perceived heading has not been investigated. We simulated observer translation towards a wall or cloud, while simultaneously simulating eye rotation about the vertical axis, the torsional axis or combinations thereof. We find only small systematic effects of torsion on the set of 2D perceived headings, regardless of the simulated horizontal rotation. In proportion to the CF shift, the systematic errors are significantly smaller for pure torsion than for pure horizontal rotation. In contrast to errors caused by horizontal rotation, the torsional errors are hardly reduced by addition of depth to the scene. We suggest the difference in behaviour reflects the difference in symmetry of the field of view relative to the axis of rotation: the higher symmetry in the case of torsion may allow for a more accurate estimation of the rotational flow. Moreover, we report a new phenomenon. Simulated horizontal rotation during simulated wall approach increases the heading-dependency of errors, causing a larger compression of perceived heading in the horizontal direction than in the vertical direction

Analysis of Complex Motion Patterns by Form/Cue Invariant MSTd Neurons

Several groups have proposed that area MSTd of the macaque monkey has a role in processing optical flow information used in the analysis of self motion, based on its neurons’ selectivity for large-field motion patterns such as expansion, contraction, and rotation. It has also been suggested that this cortical region may be important in analyzing the complex motions of objects. More generally, MSTd could be involved in the generic function of complex motion pattern representation, with its cells responsible for integrating local motion signals sent forward from area MT into a more unified representation. If MSTd is extracting generic motion pattern signals, it would be important that the preferred tuning of MSTd neurons not depend on the particular features and cues that allow these motions to be represented. To test this idea, we examined the diversity of stimulus features and cues over which MSTd cells can extract information about motion patterns such as expansion, contraction, rotation, and spirals. The different classes of stimuli included: coherently moving random dot patterns, solid squares, outlines of squares, a square aperture moving in front of an underlying stationary pattern of random dots, a square composed entirely of flicker, and a square of nonFourier motion. When a unit was tuned with respect to motion patterns across these stimulus classes, the motion pattern producing the most vigorous response in a neuron was nearly the same for each class. Although preferred tuning was invariant, the magnitude and width of the tuning curves often varied between classes. Thus, MSTd is form/cue invariant for complex motions, making it an appropriate candidate for analysis of object motion as well as motion introduced by observer translation

Analysis of Complex Motion Patterns by Form/Cue Invariant MSTd Neurons

Several groups have proposed that area MSTd of the macaque monkey has a role in processing optical flow information used in the analysis of self motion, based on its neurons’ selectivity for large-field motion patterns such as expansion, contraction, and rotation. It has also been suggested that this cortical region may be important in analyzing the complex motions of objects. More generally, MSTd could be involved in the generic function of complex motion pattern representation, with its cells responsible for integrating local motion signals sent forward from area MT into a more unified representation. If MSTd is extracting generic motion pattern signals, it would be important that the preferred tuning of MSTd neurons not depend on the particular features and cues that allow these motions to be represented. To test this idea, we examined the diversity of stimulus features and cues over which MSTd cells can extract information about motion patterns such as expansion, contraction, rotation, and spirals. The different classes of stimuli included: coherently moving random dot patterns, solid squares, outlines of squares, a square aperture moving in front of an underlying stationary pattern of random dots, a square composed entirely of flicker, and a square of nonFourier motion. When a unit was tuned with respect to motion patterns across these stimulus classes, the motion pattern producing the most vigorous response in a neuron was nearly the same for each class. Although preferred tuning was invariant, the magnitude and width of the tuning curves often varied between classes. Thus, MSTd is form/cue invariant for complex motions, making it an appropriate candidate for analysis of object motion as well as motion introduced by observer translation

Modeling the development of cortical responses in primate dorsal (“where”) pathway to optic flow using hierarchical neural field models

Although there is a plethora of modeling literature dedicated to the object recognition processes of the ventral (“what”) pathway of primate visual systems, modeling studies on the motion-sensitive regions like the Medial superior temporal area (MST) of the dorsal (“where”) pathway are relatively scarce. Neurons in the MST area of the macaque monkey respond selectively to different types of optic flow sequences such as radial and rotational flows. We present three models that are designed to simulate the computation of optic flow performed by the MST neurons. Model-1 and model-2 each composed of three stages: Direction Selective Mosaic Network (DSMN), Cell Plane Network (CPNW) or the Hebbian Network (HBNW), and the Optic flow network (OF). The three stages roughly correspond to V1-MT-MST areas, respectively, in the primate motion pathway. Both these models are trained stage by stage using a biologically plausible variation of Hebbian rule. The simulation results show that, neurons in model-1 and model-2 (that are trained on translational, radial, and rotational sequences) develop responses that could account for MSTd cell properties found neurobiologically. On the other hand, model-3 consists of the Velocity Selective Mosaic Network (VSMN) followed by a convolutional neural network (CNN) which is trained on radial and rotational sequences using a supervised backpropagation algorithm. The quantitative comparison of response similarity matrices (RSMs), made out of convolution layer and last hidden layer responses, show that model-3 neuron responses are consistent with the idea of functional hierarchy in the macaque motion pathway. These results also suggest that the deep learning models could offer a computationally elegant and biologically plausible solution to simulate the development of cortical responses of the primate motion pathway

Visual perceptual stability and the processing of self-motion information: neurophysiology, psychophysics and neuropsychology

While we move through our environment, we constantly have to deal with new sensory input. Especially the visual system has to deal with an ever-changing input signal, since we continuously move our eyes. For example, we change our direction of gaze about three times every second to a new area within our visual field with a fast, ballistic eye movement called a saccade. As a consequence, the entire projection of the surrounding world on our retina moves. Yet, we do not perceive this shift consciously. Instead, we have the impression of a stable world around us, in which objects have a well-defined location. In my thesis I aimed to investigate the underlying neural mechanisms of the visual perceptual stability of our environment. One hypothesis is that there is a coordinate transformation of the retinocentric input signal to a craniocentric (egocentric) and eventually even to a world centered (allocentric) frame of reference. Such a transformation into a craniocentric reference frame requires information about both the location of a stimulus on the retina and the current eye position within the head. The physicist Hermann von Helmholtz was one of the first who suggested that such an eye-position signal is available in the brain as an internal copy of the motor plan, which is sent to the eye muscles. This so-called efference copy allows the brain to classify actions as self-generated and differentiate them from being externally triggered. If we are the creator of an action, we are able to predict its outcome and can take this prediction into consideration for the further processing. For example, if the projection of the environment moves across the retina due to an eye movement, the shift is registered as self-induced and the brain maintains a stable percept of the world. However, if one gently pushes the eye from the side with a finger, we perceive a moving environment. Along the same lines, it is necessary to correctly attribute the movement of the visual field to our own self-motion, e.g. to perform eye movements accounting for the additional influences of our movements. The first study of my thesis shows that the perceived location of a stimulus might indeed be a combination of two independent neuronal signals, i.e. the position of the stimulus on the retina and information about the current eye-position or eye-movement, respectively. In this experiment, the mislocalization of briefly presented stimuli, which is characteristic for each type of eye-movement, leads to a perceptual localization of stimuli within the area of the blind spot on the retina. Yet, this is the region where the optic nerve leaves the eye, meaning that there are no photoreceptors available to convert light into neuronal signals. Physically, subjects should be blind for stimuli presented in this part of the visual field. In fact, a combination of the actual stimulus position with the specific, error-inducing eye-movement information is able to explain the experimentally measured behavior. The second study in my thesis investigates the underlying neural mechanism of the mislocalization of briefly presented stimuli during eye-movements. Many previous studies using animal models (the rhesus monkey) revealed internal representations of eye-position signals in various brain regions and therefore confirmed the hypothesis of an efference copy signal within the brain. Although these eye-position signals basically reflect the actual eye-position with good accuracy, there are also some spatial and temporal inaccuracies. These erroneous representations have been previously suggested as the source of perceptual mislocalization during saccades. The second study of my thesis extends this hypothesis to the mislocalization during smooth pursuit eye-movements. We usually perform such an eye movement when we want to continuously track a moving object with our eyes. I showed that the activity of neurons in the ventral intraparietal area of the rhesus monkey adequately represents the actual eye-position during smooth pursuit. However, there was a constant lead of the internal eye-position signal as compared to the real eye-position in direction of the ongoing eye-movement. In combination with a distortion of the visual map due to an uneven allocation of attention in direction of the future stimulus position, this results in a mislocalization pattern during smooth pursuit, which almost exactly resembles those typically measured in psychophysical experiments. Hence, on the one hand the efference copy of the eye-position signal provides the required signal to perform a coordinate transformation in order to preserve a stable perception of our environment. On the other hand small inaccuracies within this signal seem to cause perceptual errors when the visual system is experimentally pushed to its limits. The efference copy also plays a role in dysfunctions of the brain in neurological or psychiatric diseases. For example, many symptoms of schizophrenia patients could be explained by an impaired efference copy mechanism and a resulting misattribution of agency to self- and externally-produced actions. Following this hypothesis, the typically observed auditory hallucinations in these patients might be the result of an erroneously assigned agency of their own thoughts. To make a detailed analysis of this potentially impaired efference copy mechanism possible, the third study of my thesis investigated eye movements of schizophrenia patients and tried to step outside the limited capabilities of laboratory setups into the real world. This study showed that results of previous laboratory studies only partly resemble those obtained in the real world. For example, schizophrenia patients, when compared to healthy controls, usually show a more inaccurate smooth pursuit eye-movement in the laboratory. Yet, in the real world when they track a stationary object with their eyes while they are moving towards it, there are no differences between patients and healthy controls, although both types of eye-movements are closely related. This might be due to the fact that patients were able to use additional sources of information in the real world, e.g. self-motion information, to compensate for some of their deficits under certain conditions. Similarly, the fourth study of my thesis showed that typical impairments of eye-movements during healthy aging can be equalized by other sources of information available under natural conditions. At the same time, this work underlined the need of eye-movement measurements in the real world as a complement to laboratory studies to accurately describe the visual system, all mechanisms of perception and their interactions under natural circumstances. For example, experiments in the laboratory usually analyze particularly selected eye-movement parameters within a specific range, such as saccades of a certain amplitude. However, this does not reflect everyday life in which parameters like that are typically continuous and not normally distributed. Furthermore, motion-selective areas in the brain might play a much bigger role in natural environments, since we generally move our head and/or ourselves. To correctly analyze the contribution to and influences on eye-movements, one has to perform eye-movement studies under conditions as realistic as possible. The fifth study of my thesis aimed to investigate a possible application of eye-movement studies in the diagnosis of neuronal diseases. We showed that basic eye-movement parameters like saccade peak-velocity can be used to differentiate patients with Parkinson’s disease from patients with an atypical form of Parkinsonism, progressive supranuclear palsy. This differentiation is of particular importance since both diseases share a similar onset but have a considerably different progression and outcome, requiring different types of therapies. An early differential diagnosis, preferably in a subclinical stage, is needed to ensure the optimal treatment of the patients in order to ease the symptoms and eventually even improve the prognosis. The study showed that mobile eye-trackers are particularly well-suited to investigate eye movements in the daily clinical routine, due to their promising results in differential diagnosis and their easy, fast and reliable handling. In conclusion, my thesis underlines the importance of an interaction of all the different neuroscientific methods such as psychophysics, eye-movement measurements in the real world, electrophysiology and the investigation of neuropsychiatric patients to get a complete picture of how the brain works. The results of my thesis contribute to extent the current knowledge about the processing of information and the perception of our environment in the brain, point towards fields of application of eye-movement measurements and can be used as groundwork for future research