Synchronized Audio-Visual Transients Drive Efficient Visual Search for Motion-in-Depth

In natural audio-visual environments, a change in depth is usually correlated with a change in loudness. In the present study, we investigated whether correlating changes in disparity and loudness would provide a functional advantage in binding disparity and sound amplitude in a visual search paradigm. To test this hypothesis, we used a method similar to that used by van der Burg et al. to show that non-spatial transient (square-wave) modulations of loudness can drastically improve spatial visual search for a correlated luminance modulation. We used dynamic random-dot stereogram displays to produce pure disparity modulations. Target and distractors were small disparity-defined squares (either 6 or 10 in total). Each square moved back and forth in depth in front of the background plane at different phases. The target’s depth modulation was synchronized with an amplitude-modulated auditory tone. Visual and auditory modulations were always congruent (both sine-wave or square-wave). In a speeded search task, five observers were asked to identify the target as quickly as possible. Results show a significant improvement in visual search times in the square-wave condition compared to the sine condition, suggesting that transient auditory information can efficiently drive visual search in the disparity domain. In a second experiment, participants performed the same task in the absence of sound and showed a clear set-size effect in both modulation conditions. In a third experiment, we correlated the sound with a distractor instead of the target. This produced longer search times, indicating that the correlation is not easily ignored

Dynamic and static cues for binocular vision – a systematic comparison

Background Patients who are diagnosed as stereo blind, during clinical assessment have reported a compelling, volumetric perception of depth during stereoscopic viewing at the cinema. This effect cannot entirely be explained by the monocular cues present in the cinematic presentation. This lead to the theory that depth from binocular cues may be more apparent when motion is included in the scene. As an object approaches in space is detected through the use of two binocular cues, changing disparity over time, and intraocular velocity difference. These cues have been previously investigated in terms of detecting the presence of motion and discriminating the direction of motion. In this thesis I am to investigate the contribution of stereomotion to the detection of depth. Methods A four alternate forced choice adaptive staircase presentation paradigm was used to assess the ability of participants to detect which of four random dot patterned stimuli patches appeared closest to them in space. The outcome measure for every experiment was depth detection threshold. The experiments were presented using either linear polarised or dichoptic stereoscopic display methods. The stimulus patches were designed to only define depth through binocular disparity, with care taken to avoid any monocular cues. The target patch was identical to all other stimuli patches other than variations to test the following dynamic characteristics: z-location change, X-location change, changing disparity only, interocular velocity difference change only and changes in pattern. These were all comparable to a static condition, where depth was defined by disparity only. All z-axis (or depth) changes were defined by ‘on-screen’ separations of half images (the images separated to the left and right eyes in turn). A number of control experiments were also included to assess the effect of fusional demand, of spurious temporal correlations, of variations in speed of changes in depth and of cue construction on depth detection thresholds. Results 410 subjects were assessed, (aged mean (SD) age 21(5) years) across all experiments. In comparison to the static disparity conditions (415”), depth detection thresholds were statistically significantly lower for the stereomotion conditions, with (CDOT 360”) and without (Z-LOCATION CHANGE 310”) pattern change (p0.05). Conclusion The threshold for detecting depth in stimuli that contain z-motion, is better (lower) than for static stimuli, providing an explanation for the experience of compelling depth at the cinema. As z-motion depth detection thresholds were significantly lower than static thresholds, this suggests motion provides an advantage to extracting depth, above serial static disparity detection alone. The assessment of stereoacuity should include the measurement of depth detection thresholds using changing depth stimuli, in order to fully investigate binocular potential

Organisation of audio-visual three-dimensional space

Le terme stéréopsie renvoie à la sensation de profondeur qui est perçue lorsqu une scène est vue de manière binoculaire. Le système visuel s appuie sur les disparités horizontales entre les images projetées sur les yeux gauche et droit pour calculer une carte des différentes profondeurs présentes dans la scène visuelle. Il est communément admis que le système stéréoscopique est encapsulé et fortement contraint par les connexions neuronales qui s étendent des aires visuelles primaires (V1/V2) aux aires intégratives des voies dorsales et ventrales (V3, cortex temporal inférieur, MT). A travers quatre projets expérimentaux, nous avons étudié comment le système visuel utilise la disparité binoculaire pour calculer la profondeur des objets. Nous avons montré que le traitement de la disparité binoculaire peut être fortement influencé par d autres sources d information telles que l occlusion binoculaire ou le son. Plus précisément, nos résultats expérimentaux suggèrent que : (1) La stéréo de da Vinci est résolue par un mécanisme qui intègre des processus de stéréo classiques (double fusion), des contraintes géométriques (les objets monoculaires sont nécessairement cachés à un œil, par conséquent ils sont situés derrière le plan de l objet caché) et des connaissances à priori (une préférence pour les faibles disparités). (2) Le traitement du mouvement en profondeur peut être influencé par une information auditive : un son temporellement corrélé avec une cible définie par le mouvement stéréo peut améliorer significativement la recherche visuelle. Les détecteurs de mouvement stéréo sont optimalement adaptés pour détecter le mouvement 3D mais peu adaptés pour traiter le mouvement 2D. (3) Grouper la disparité binoculaire avec un signal auditif dans une dimension orthogonale (hauteur tonale) peut améliorer l acuité stéréo d approximativement 30%Stereopsis refers the perception of depth that arises when a scene is viewed binocularly. The visual system relies on the horizontal disparities between the images from the left and right eyes to compute a map of the different depth values present in the scene. It is usually thought that the stereoscopic system is encapsulated and highly constrained by the wiring of neurons from the primary visual areas (V1/V2) to higher integrative areas in the ventral and dorsal streams (V3, inferior temporal cortex, MT). Throughout four distinct experimental projects, we investigated how the visual system makes use of binocular disparity to compute the depth of objects. In summary, we show that the processing of binocular disparity can be substantially influenced by other types of information such as binocular occlusion or sound. In more details, our experimental results suggest that: (1) da Vinci stereopsis is solved by a mechanism that integrates classic stereoscopic processes (double fusion), geometrical constraints (monocular objects are necessarily hidden to one eye, therefore they are located behind the plane of the occluder) and prior information (a preference for small disparities). (2) The processing of motion-in-depth can be influenced by auditory information: a sound that is temporally correlated with a stereomotion defined target can substantially improve visual search. Stereomotion detectors are optimally suited to track 3D motion but poorly suited to process 2D motion. (3) Grouping binocular disparity with an orthogonal auditory signal (pitch) can increase stereoacuity by approximately 30%PARIS5-Bibliotheque electronique (751069902) / SudocSudocFranceF

Vergence eye movements in patients with schizophrenia

AbstractPrevious studies have shown that smooth pursuit eye movements are impaired in patients with schizophrenia. However, under normal viewing conditions, targets move not only in the frontoparallel plane but also in depth, and tracking them requires both smooth pursuit and vergence eye movements. Although previous studies in humans and non-human primates suggest that these two eye movement subsystems are relatively independent of one another, to our knowledge, there have been no prior studies of vergence tracking behavior in patients with schizophrenia. Therefore, we have investigated these eye movements in patients with schizophrenia and in healthy controls. We found that patients with schizophrenia exhibited substantially lower gains compared to healthy controls during vergence tracking at all tested speeds (e.g. 0.25Hz vergence tracking mean gain of 0.59 vs. 0.86). Further, consistent with previous reports, patients with schizophrenia exhibited significantly lower gains than healthy controls during smooth pursuit at higher target speeds (e.g. 0.5Hz smooth pursuit mean gain of 0.64 vs. 0.73). In addition, there was a modest (r≈0.5), but significant, correlation between smooth pursuit and vergence tracking performance in patients with schizophrenia. Our observations clearly demonstrate substantial vergence tracking deficits in patients with schizophrenia. In these patients, deficits for smooth pursuit and vergence tracking are partially correlated suggesting overlap in the central control of smooth pursuit and vergence eye movements

Spatial and temporal integration of binocular disparity in the primate brain

Le système visuel du primate s'appuie sur les légères différences entre les deux projections rétiniennes pour percevoir la profondeur. Cependant, on ne sait pas exactement comment ces disparités binoculaires sont traitées et intégrées par le système nerveux. D'un côté, des enregistrements unitaires chez le macaque permettent d'avoir accès au codage neuronal de la disparité à un niveau local. De l'autre côté, la neuroimagerie fonctionnelle (IRMf) chez l'humain met en lumière les réseaux corticaux impliqués dans le traitement de la disparité à un niveau macroscopique mais chez une espèce différente. Dans le cadre de cette thèse, nous proposons d'utiliser la technique de l'IRMf chez le macaque pour permettre de faire le lien entre les enregistrements unitaires chez le macaque et les enregistrements IRMf chez l'humain. Cela, afin de pouvoir faire des comparaisons directes entre les deux espèces. Plus spécifiquement, nous nous sommes intéressés au traitement spatial et temporal des disparités binoculaires au niveau cortical mais aussi au niveau perceptif. En étudiant l'activité corticale en réponse au mouvement tridimensionnel (3D), nous avons pu montrer pour la première fois 1) qu'il existe un réseau dédié chez le macaque qui contient des aires allant au-delà du cluster MT et des aires environnantes et 2) qu'il y a des homologies avec le réseau trouvé chez l'humain en réponse à des stimuli similaires. Dans une deuxième étude, nous avons tenté d'établir un lien entre les biais perceptifs qui reflètent les régularités statistiques 3D ans l'environnement visuel et l'activité corticale. Nous nous sommes demandés si de tels biais existent et peuvent être reliés à des réponses spécifiques au niveau macroscopique. Nous avons trouvé de plus fortes activations pour le stimulus reflétant les statistiques naturelles chez un sujet, démontrant ainsi une possible influence des régularités spatiales sur l'activité corticale. Des analyses supplémentaires sont cependant nécessaires pour conclure de façon définitive. Néanmoins, nous avons pu confirmer de façon robuste l'existence d'un vaste réseau cortical répondant aux disparités corrélées chez le macaque. Pour finir, nous avons pu mesurer pour la première fois les points rétiniens correspondants au niveau du méridien vertical chez un sujet macaque qui réalisait une tâche comportementale (procédure à choix forcé). Nous avons pu comparer les résultats obtenus avec des données également collectées chez des participants humains avec le même protocole. Dans les différentes sections de discussion, nous montrons comment nos différents résultats ouvrent la voie à de nouvelles perspectives.The primate visual system strongly relies on the small differences between the two retinal projections to perceive depth. However, it is not fully understood how those binocular disparities are computed and integrated by the nervous system. On the one hand, single-unit recordings in macaque give access to neuronal encoding of disparity at a very local level. On the other hand, functional neuroimaging (fMRI) studies in human shed light on the cortical networks involved in disparity processing at a macroscopic level but with a different species. In this thesis, we propose to use an fMRI approach in macaque to bridge the gap between single-unit and fMRI recordings conducted in the non-human and human primate brain, respectively, by allowing direct comparisons between the two species. More specifically, we focused on the temporal and spatial processing of binocular disparities at the cortical but also at the perceptual level. Investigating cortical activity in response to motion-in-depth, we could show for the first time that 1) there is a dedicated network in macaque that comprises areas beyond the MT cluster and its surroundings and that 2) there are homologies with the human network involved in processing very similar stimuli. In a second study, we tried to establish a link between perceptual biases that reflect statistical regularities in the three-dimensional visual environment and cortical activity, by investigating whether such biases exist and can be related to specific responses at a macroscopic level. We found stronger activity for the stimulus reflecting natural statistics in one subject, demonstrating a potential influence of spatial regularities on the cortical activity. Further work is needed to firmly conclude about such a link. Nonetheless, we robustly confirmed the existence of a vast cortical network responding to correlated disparities in the macaque brain. Finally, we could measure for the first time retinal corresponding points on the vertical meridian of a macaque subject performing a behavioural task (forced-choice procedure) and compare it to the data we also collected in several human observers with the very same protocol. In the discussion sections, we showed how these findings open the door to varied perspectives

3D motion : encoding and perception

The visual system supports perception and inferences about events in a dynamic, three-dimensional (3D) world. While remarkable progress has been made in the study of visual information processing, the existing paradigms for examining visual perception and its relation to neural activity often fail to generalize to perception in the real world which has complex dynamics and 3D spatial structure. This thesis focuses on the case of 3D motion, developing dynamic tasks for studying visual perception and constructing a neural coding framework to relate neural activity to perception in a 3D environment. First, I introduce target-tracking as a psychophysical method and develop an analysis framework based on state space models and the Kalman filter. I demonstrate that target-tracking in conjunction with a Kalman filter analysis framework produce estimates of visual sensitivity that are comparable to those obtained with a traditional forced-choice task and a signal detection theory analysis. Next, I use the target-tracking paradigm in a series of experiments examining 3D motion perception, specifically comparing the perception of frontoparallel motion with the perception of motion-through-depth. I find that continuous tracking of motion-through-depth is selectively impaired due to the relatively small retinal projections resulting from motion-through-depth and the slower processing of binocular disparities. The thesis then turns the neural representation of 3D motion and how that underlies perception. First I introduce a theoretical framework that extends the standard neural coding approach, incorporating the environment-to-retina transformation. Neural coding typically treats the visuals stimulus as a direct proxy for the pattern of stimulation that falls on the retina. Incorporating the environment-to-retina transformation results in a neural representation fundamentally shaped by the projective geometry of the world onto the retina. This model explains substantial anomalies in existing neurophysiological recordings in primate visual cortical neurons during presentations of 3D motion and in psychophysical studies of human perception. In a series of psychophysical experiments, I systematically examine the predictions of the model for human perception by observing how perceptual performance changes as a function of viewing distance and eccentricity. Performance in these experiments suggests a reliance on a neural representation similar to the one described by the model. Taken together, the experimental and theoretical findings reported here advance the understanding of the neural representation and perception of the dynamic 3D world, and adds to the behavioral tools available to vision scientists.Neuroscienc

Neural mechanisms of binocular motion in depth perception

Motion in depth (MID) can be cued by two binocular sources of information. These are changes in retinal disparity over time (changing disparity, CD), and binocular opponent velocity vectors (inter-ocular velocity difference, IOVD). This thesis presents a series of psychophysical and fMRI experiments investigating the neural pathways supporting the perception of CD and IOVD. The first two experiments investigated how CD and IOVD mechanisms draw on information encoded in the magnocellular, parvocellular and koniocellular pathways. The chromaticity of CD and IOVD-isolating stimuli was manipulated to bias activity in these three pathways. Although all stimulus types and chromaticities supported a MID percept, fMRI revealed an especially dominant koniocellular contribution to the IOVD mechanism. Because IOVD depends on eye-specific velocity signals, experiment three sought to identify an area in the brain that encodes motion direction and eye of origin information. Classification and multivariate pattern analysis techniques were applied to fMRI data, but no area where both types of information were present simultaneously was identified. Results suggested that IOVD mechanisms inherit eye-specific information from V1. Finally, experiment four asked whether activity elicited by CD and IOVD stimuli could also be modulated by an attentional task where participants were asked to detect changes in MID or local contrast. fMRI activity was strongly modulated by attentional state, and activity in motion-selective areas was predictive of whether participants correctly identified the change in CD or IOVD MID. This suggests that these areas contain populations of neurons that are crucial for detecting, and behaviourally responding to, both types of MID. The work presented in this thesis detail a thorough investigation of the neural pathways that underlie the computation of CD and IOVD cues to MID

Role of feedback in the accuracy of perceived direction of motion-in-depth and control of interceptive action

AbstractWe quantified the accuracy of the perception of the absolute direction of motion-in-depth (MID) of a simulated approaching object using a perceptual task and compared those data with the accuracy of estimating the passing distance measured by means of a simulated catching task. For the simulated catching task, movements of the index finger and thumb of the observer’s hand were tracked as participants tried to “catch” the simulated approaching object. A sensation of MID was created by providing monocular and/or binocular retinal image information. Visual stimuli were identical for perceptual and simulated catching tasks. We confirm previous reports that in the perceptual task, observers judged the object to pass wider of the head than indicated by the visual information provided. Although accuracy improved when auditory feedback was added to the perceptual (button pressing) task, consistent overestimates were still recorded. For the no-feedback simulated catching task, observers consistently overreached, i.e., the hand was further away from the midline than the simulated object at the time of hand closure. When auditory feedback was added to the simulated catching task successful catching was achieved. The relative accuracy in binocular and monocular conditions for individual observers could be partially explained by individual differences in sensitivity to unidirectional changes in angular size and changes in relative disparity. We conclude that catching an approaching ball requires that errors in the perceived direction of MID are corrected by feedback-driven learning in the motor system, and that this learning is more easily achieved for the catching action than for button pressing

Perceived Acceleration in Stereoscopic Animation

In stereoscopic media, a sensation of depth is produced through the differences of images presented to the left and the right eyes. These differences are a result of binocular parallax caused by the separation of the cameras used to capture the scene. Creators of stereoscopic media face the challenge of producing compelling depth while restricting the amount of parallax to a comfortable range. Control of camera separation is a key manipulation to control parallax. Sometimes, stereoscopic warping is used in post-production process to selectively increase or decrease depth in certain regions of the image. However, mismatches between camera geometry and natural stereoscopic geometry can theoretically produce nonlinear distortions of perceived space. The relative expansion or compression of the stereoscopic space, in theory, should affect the perceived acceleration of objects moving through that space. This thesis suggests that viewers are tolerant of effects of distortions when perceiving acceleration in a stereoscopic scene