Effects of cross-modal asynchrony on informational masking in human cortex

In many everyday listening situations, an otherwise audible sound may go unnoticed amid multiple other sounds. This auditory phenomenon, called informational masking (IM), is sensitive to visual input and involves early (50-250 msec) activity in the auditory cortex (the so-called awareness-related negativity). It is still unclear whether and how the timing of visual input influences the neural correlates of IM in auditory cortex. To address this question, we obtained simultaneous behavioral and neural measures of IM from human listeners in the presence of a visual input stream and varied the asynchrony between the visual stream and the rhythmic auditory target stream (in-phase, antiphase, or random). Results show effects of cross-modal asynchrony on both target detectability (RT and sensitivity) and the awareness-related negativity measured with EEG, which were driven primarily by antiphasic audiovisual stimuli. The neural effect was limited to the interval shortly before listeners' behavioral report of the target. Our results indicate that the relative timing of visual input can influence the IM of a target sound in the human auditory cortex. They further show that this audiovisual influence occurs early during the perceptual buildup of the target sound. In summary, these findings provide novel insights into the interaction of IM and multisensory interaction in the human brain.</p

Auditory affective processing requires awareness

Recent work has challenged the previously widely accepted belief that affective processing does not require awareness and can be carried out with more limited resources than semantic processing. This debate has focused exclusively on visual perception, even though evidence from both human and animal studies suggests that existence for nonconscious affective processing would be physiologically more feasible in the auditory system. Here we contrast affective and semantic processing of nonverbal emotional vocalizations under different levels of awareness in three experiments, using explicit (two-alternative forced choice masked affective and semantic categorization tasks, Experiments 1 and 2) and implicit (masked affective and semantic priming, Experiment 3) measures. Identical stimuli and design were used in the semantic and affective tasks. Awareness was manipulated by altering stimulus-mask signal-to-noise ratio during continuous auditory masking. Stimulus awareness was measured on each trial using a four-point perceptual awareness scale. In explicit tasks, neither affective nor semantic categorization could be performed in the complete absence of awareness, while both tasks could be performed above chance level when stimuli were consciously perceived. Semantic categorization was faster than affective evaluation. When the stimuli were partially perceived, semantic categorization accuracy exceeded affective evaluation accuracy. In implicit tasks neither affective nor semantic priming occurred in the complete absence of awareness, whereas both affective and semantic priming emerged when participants were aware of the primes. We conclude that auditory semantic processing is faster than affective processing, and that both affective and semantic auditory processing are dependent on awareness

Neural correlates of auditory perceptual organization measured with direct cortical recordings in humans

Thesis (Ph. D.)--Harvard-MIT Division of Health Sciences and Technology, September 2011."August, 2011." Vita. Cataloged from PDF version of thesis.Includes bibliographical references.One of the primary functions of the human auditory system is to separate the complex mixture of sound arriving at the ears into neural representations of individual sound sources. This function is thought to be crucial for survival and communication in noisy settings, and allows listeners to selectively and dynamically attend to a sound source of interest while suppressing irrelevant information. How the brain works to perceptually organize the acoustic environment remains unclear despite the multitude of recent studies utilizing microelectrode recordings in experimental animals or non-invasive human neuroimaging. In particular, the role that brain areas outside the auditory cortex might play is, comparatively, vastly understudied. The experiments described in this thesis combined classic behavioral paradigms with electrical recordings made directly from the cortical surface of neurosurgical patients undergoing clinically-indicated invasive monitoring for localization of epileptogenic foci. By sampling from widespread brain areas with high temporal resolution while participants simultaneously engaged in streaming and jittered multi-tone masking paradigms, the present experiments sought to overcome limitations inherent in previous work, namely sampling extent, resolution in time and space, and direct knowledge of the perceptual experience of the listener. In experiment 1, participants listened to sequences of tones alternating in frequency (i.e., ABA-) and indicated whether they perceived the tones as grouped ("1 stream") or segregated ("2 streams"). As has been reported in neurologically-normal listeners since the 1950s, patients heard the sequences as grouped when the frequency separation between the A and B tones was small and segregated when it was large. Evoked potentials from widespread brain areas showed amplitude correlations with frequency separation but surprisingly did not differ based solely on perceptual organization in the absence of changes in the stimuli. In experiment 2, participants listened to sequences of jittered multi-tone masking stimuli on which a regularly-repeating target stream of tones was sometimes superimposed and indicated when they heard the target stream. Target detectability, as indexed behaviorally, increased throughout the course of each sequence. Evoked potentials and high-gamma activity differed strongly based on the listener's subjective perception of the target tones. These results extend and constrain theories of how the brain subserves auditory perceptual organization and suggests several new avenues of research for understanding the neural mechanisms underlying this critical function.by Andrew R. Dykstra.Ph.D

Attention, Awareness, and the Perception of Auditory Scenes

Auditory perception and cognition entails both low-level and high-level processes, which are likely to interact with each other to create our rich conscious experience of soundscapes. Recent research that we review has revealed numerous influences of high-level factors, such as attention, intention, and prior experience, on conscious auditory perception. And recently, studies have shown that auditory scene analysis tasks can exhibit multistability in a manner very similar to ambiguous visual stimuli, presenting a unique opportunity to study neural correlates of auditory awareness and the extent to which mechanisms of perception are shared across sensory modalities. Research has also led to a growing number of techniques through which auditory perception can be manipulated and even completely suppressed. Such findings have important consequences for our understanding of the mechanisms of perception and also should allow scientists to precisely distinguish the influences of different higher-level influences

Psychophysical and electrophysiological investigations into the mechanisms supporting everyday communication

Thesis (Ph.D.)--Boston UniversityHumans solve the so-called "cocktail party problem" with relative ease, and are generally able to selectively direct their attention to process and recall acoustic information from one sound source in the presence of other irrelevant stimuli that are competing for cognitive resources. This ability depends on a variety of factors, including volitional control of selective attention, the ability to store information in memory for recall at a later time, and the ability to integrate information across multiple sensory modalities. Here, psychophysical and electroencephalography (EEG) experiments were conducted to study these three factors. The effects of selective attention on cortical and subcortical structures were examined using EEG recorded during a dichotic listening task. Cortical potentials showed robust effects of attention (demonstrated by the ability to classify responses to attended and ignored speech based on short segments of EEG responses); however, potentials originating in the brainstem did not, even though stimuli were engineered to maximize the separability of the neural representation of the competing sources in the auditory periphery and thus the possibility of seeing attention-specific modulation of subcortical responses. In another study, the relationship between object formation and memory processing was explored in a psychophysical experiment examining how sequences of nonverbal auditory stimuli are stored and recalled from short-term memory. The results of this study support the notion that auditory short-term memory, like visual short-term memory, can be explained in terms of object formation. In particular, short-term memory performance is affected by stream formation and the perceptual costs involved in switching attention between multiple streams. Finally, effects of audiovisual integration were studied in a psychophysical experiment using complex speech-like stimuli (zebra finch songs). Results show visual cues improve performance differently depending on whether target identification is limited by energetic masking or whether it is limited by object formation difficulties and uncertainty about when a target occurs. Together, these studies support the idea that everyday communication depends on an interplay of many mechanisms including attention, memory, and multisensory integration, each of which is influenced by perceptual organization

The Effect of Visual Perceptual Load on Auditory Processing

Many fundamental aspects of auditory processing occur even when we are not attending to the auditory environment. This has led to a popular belief that auditory signals are analysed in a largely pre-attentive manner, allowing hearing to serve as an early warning system. However, models of attention highlight that even processes that occur by default may rely on access to perceptual resources, and so can fail in situations when demand on sensory systems is particularly high. If this is the case for auditory processing, the classic paradigms employed in auditory attention research are not sufficient to distinguish between a process that is truly automatic (i.e., will occur regardless of any competing demands on sensory processing) and one that occurs passively (i.e., without explicit intent) but is dependent on resource-availability. An approach that addresses explicitly whether an aspect of auditory analysis is contingent on access to capacity-limited resources is to control the resources available to the process; this can be achieved by actively engaging attention in a different task that depletes perceptual capacity to a greater or lesser extent. If the critical auditory process is affected by manipulating the perceptual demands of the attended task this suggests that it is subject to the availability of processing resources; in contrast a process that is automatic should not be affected by the level of load in the attended task. This approach has been firmly established within vision, but has been used relatively little to explore auditory processing. In the experiments presented in this thesis, I use MEG, pupillometry and behavioural dual-task designs to explore how auditory processing is impacted by visual perceptual load. The MEG data presented illustrate that both the overall amplitude of auditory responses, and the computational capacity of the auditory system are affected by the degree of perceptual load in a concurrent visual task. These effects are mirrored by the pupillometry data in which pupil dilation is found to reflect both the degree of load in the attended visual task (with larger pupil dilation to the high compared to the low load visual load task), and the sensory processing of irrelevant auditory signals (with reduced dilation to sounds under high versus low visual load). The data highlight that previous assumptions that auditory processing can occur automatically may be too simplistic; in fact, though many aspects of auditory processing occur passively and benefit from the allocation of spare capacity, they are not strictly automatic. Moreover, the data indicate that the impact of visual load can be seen even on the early sensory cortical responses to sound, suggesting not only that cortical processing of auditory signals is dependent on the availability of resources, but also that these resources are part of a global pool shared between vision and audition

Is the perceived present a predictive model of the objective present?

Processing latencies for coherent, high level percepts in vision are at least 100 ms and possibly as much as 500 ms. Processing latencies are less in other modalities, but still significant. This seems to imply that perception lags behind reality by an amount equal to the processing latency. It has been proposed that the brain can compensate for perceptual processing latencies by using the most recent available information to extrapolate forward, thereby constructing a model of what the world beyond the senses is like now. The present paper reviews several lines of evidence relating to this hypothesis, including the flash-lag effect, motion-induced position shifts, representational momentum, static visual illusions, and motion extrapolation at the retina. There are alternative explanations for most of the results but there are some findings for which no competing explanation has yet been proposed. Collectively, the evidence for extrapolation to the present is suggestive but not yet conclusive. An alternative account of compensation for processing latencies, based on the hypothesis of rapid emergence of percepts, is proposed

Sounds in noise: Behavioral and neural studies of illusory continuity and discontinuity

ability to parse an auditory scene into meaningful components varies greatly between individuals; some are able to parse out and write down competing musical pieces while others struggle to understand each word whenever they have to converse in a noisy environment. Using a simple discrimination task, healthy, normally-heari ng adult participants were asked to judge whether a pure tone (with or without amplitude modulation) was continuous or contained a gap. One quarter of the participants consistently heard a gap when none was present, if the tone was accompanied by a higher-frequency noise burst with a lower edge beginning one octave away from the tone (that did not have any energy overlapping the tone). This novel form of informational masking (perceptual interference between components with non-overlapping sound energy) was named 'illusory auditory discontinuity\u2019. The phenomenon appears to reflect natural differences in auditory processing rather than differences in decision-making strategies because: (1) susceptibility to illusory discontinuity correlates with individual differences in auditory streaming (measured using a classical ABA sequential paradigm); and (2) electroencephalographic responses elicited by tones overlaid by short noise bursts (when these sounds are not the focus of attention) are significantly correlated with the occurrence of illusory auditory discontinuity in both an early event-related potential (ERP) component (40-66 ms), and a later ERP component (270-350 ms) after noise onset. Participants prone to illusory discontinuity also tended not to perceive the \u2018auditory continuity illusion\u2019 (in which a tone is heard continuing under a burst of noise centered on the tone frequency that completely masks it) at short noise durations, but reliably perceived the auditory continuity illusion at longer noise durations. These results suggest that a number of attributes describing how individuals differentially parse complex auditory scenes are related to individual differences in two potentially independent attributes of neural processing, reflected here by EEG waveform differences at ~50 msec and ~300 msec after noise onset. Neural correlates of the auditory continuity illusion were also investigated by adjusting masker loudness, so that when listeners were given physically identical stimuli, they correctly detected the gap in a target tone on some trials, while on other trials they reported the tone as continuous (experiencing illusory continuity). High er power of low-frequency EEG activity (in the delta-theta range, <6 Hz) was observed prior to the onset of tones that were subsequently judged as discontinuous, with no other consistent EEG differences found after the onset of tones. These data suggest that the occurrence of the continuity illusion may depend on the brain state that exists immediately before a trial begins

Time marking in perception

Several authors have proposed that perceptual information carries labels that identify temporal features, including time of occurrence, ordinal temporal relations, and brief durations. These labels serve to locate and organise perceptual objects, features, and events in time. In some proposals time marking has local, specific functions such as synchronisation of different features in perceptual processing. In other proposals time marking has general significance and is responsible for rendering perceptual experience temporally coherent, just as various forms of spatial information render the visual environment spatially coherent. These proposals, which all concern time marking on the millisecond time scale, are reviewed. It is concluded that time marking is vital to the construction of a multisensory perceptual world in which things are orderly with respect to both space and time, but that much more research is needed to ascertain its functions in perception and its neurophysiological foundations

Concurrency in auditory displays for connected television

Many television experiences depend on users being both willing and able to visually attend to screen-based information. Auditory displays offer an alternative method for presenting this information and could benefit all users. This thesis explores how this may be achieved through the design and evaluation of auditory displays involving varying degrees of concurrency for two television use cases: menu navigation and presenting related content alongside a television show. The first study, on the navigation of auditory menus, looked at onset asynchrony and word length in the presentation of spoken menus. The effects of these on task duration, accuracy and workload were considered. Onset asynchrony and word length both caused significant effects on task duration and accuracy, while workload was only affected by onset asynchrony. An optimum asynchrony was identified, which was the same for both long and short words, but better performance was obtained with the shorter words that no longer overlapped. The second experiment investigated how disruption, workload, and preference are affected when presenting additional content accompanying a television programme. The content took the form of sound from different spatial locations or as text on a smartphone and the programme's soundtrack was either modified or left unaltered. Leaving the soundtrack unaltered or muting it negatively impacted user experience. Removing the speech from the television programme and presenting the secondary content as sound from a smartphone was the best auditory approach. This was found to compare well with the textual presentation, resulting in less visual disruption and imposing a similar workload. Additionally, the thesis reviews the state-of-the-art in television experiences and auditory displays. The human auditory system is introduced and important factors in the concurrent presentation of speech are highlighted. Conclusions about the utility of concurrency within auditory displays for television are made and areas for further work are identified