THE ACOUSTIC QUALITIES THAT INFLUENCE AUDITORY OBJECT AND EVENT RECOGNITION

Throughout the course of a given day, human listeners encounter an immense variety of sounds in their environment. These are quickly transformed into mental representations of objects and events in the world, which guide more complex cognitive processes and behaviors. Through five experiments in this dissertation, I investigated the rapid formation of auditory object and event representations (i.e., shortly after sound onset) with a particular focus on understanding what acoustic information the auditory system uses to support this recognition process. The first three experiments analyzed behavioral (dissimilarity ratings in Experiment 1; duration-gated identification in Experiment 2) and neural (MEG decoding in Experiment 3) responses to a diverse array of natural sound recordings as a function of the acoustic qualities of the stimuli and their temporal development alongside participants’ concurrently developing responses. The findings from these studies highlight the importance of acoustic qualities related to noisiness, spectral envelope, spectrotemporal change over time, and change in fundamental frequency over time for sound recognition. Two additional studies further tested these results via syntheszied stimuli that explicitly manipulated these acoustic cues, interspersed among a new set of natural sounds. Findings from these acoustic manipulations as well as replications of my previous findings (with new stimuli and tasks) again revealed the importance of aperiodicity, spectral envelope, spectral variability and fundamental frequency in sound-category representations. Moreover, analyses of the synthesized stimuli suggested that aperiodicity is a particularly robust cue for some categories and that speech is difficult to characterize acoustically, at least based on this set of acoustic dimensions and synthesis approach. While the study of the perception of these acoustic cues has a long history, a fuller understanding of how these qualities contribute to natural auditory object recognition in humans has been difficult to glean. This is in part because behaviorally important categories of sound (studied together in this work) have previously been studied in isolation. By bringing these literatures together over these five experiments, this dissertation begins to outline a feature space that encapsulates many different behaviorally relevant sounds with dimensions related to aperiodicity, spectral envelope, spectral variability and fundamental frequency

The rapid emergence of auditory object representations in cortex reflect central acoustic attributes

Human listeners are bombarded by acoustic information that the brain rapidly organizes into coherent percepts of objects and events in the environment, which aids speech and music perception. The efficiency of auditory object recognition belies the critical constraint that acoustic stimuli necessarily require time to unfold. Using magentoencephalography (MEG), we studied the time course of the neural processes that transform dynamic acoustic information into auditory object representations. Participants listened to a diverse set of 36 tokens comprising everyday sounds from a typical human environment. Multivariate pattern analysis was used to decode the sound tokens from the MEG recordings. We show that sound tokens can be decoded from brain activity beginning 90 milliseconds after stimulus onset with peak decoding performance occurring at 155 milliseconds post stimulus onset. Decoding performance was primarily driven by differences between category representations (e.g., environmental vs. instrument sounds), although within-category decoding was better than chance. Representational similarity analysis revealed that these emerging neural representations were related to harmonic and spectrotemporal differences among the stimuli, which correspond to canonical acoustic features processed by the auditory pathway. Our findings begin to link the processing of physical sound properties with the perception of auditory objects and events in cortex

Separable neural representations of sound sources: Speaker identity and musical timbre

Human listeners can quickly and easily recognize different sound sources (objects and events) in their environment. Understanding how this impressive ability is accomplished can improve signal processing and machine intelligence applications along with assistive listening technologies. However, it is not clear how the brain represents the many sounds that humans can recognize (such as speech and music) at the level of individual sources, categories and acoustic features. To examine the cortical organization of these representations, we used patterns of fMRI responses to decode 1) four individual speakers and instruments from one another (separately, within each category), 2) the superordinate category labels associated with each stimulus (speech or instrument), and 3) a set of simple synthesized sounds that could be differentiated entirely on their acoustic features. Data were collected using an interleaved silent steady state sequence to increase the temporal signal-to-noise ratio, and mitigate issues with auditory stimulus presentation in fMRI. Largely separable clusters of voxels in the temporal lobes supported the decoding of individual speakers and instruments from other stimuli in the same category. Decoding the superordinate category of each sound was more accurate and involved a larger portion of the temporal lobes. However, these clusters all overlapped with areas that could decode simple, acoustically separable stimuli. Thus, individual sound sources from different sound categories are represented in separate regions of the temporal lobes that are situated within regions implicated in more general acoustic processes. These results bridge an important gap in our understanding of cortical representations of sounds and their acoustics

Psychophysiological Indices of Music-Evoked Emotions in Musicians

A number of psychophysiological measures indexing autonomic and somatovisceral activation to music have been proposed in line with the wider emotion literature. However, attempts to replicate experimental findings and provide converging evidence for music-evoked emotions through physiological changes, overt expression, and subjective measures have had mixed success. This may be due to issues in stimulus and participant selection. Therefore, the aim of Experiment 1 was to select musical stimuli that were controlled for instrumentation, musical form, style, and familiarity. We collected a wide range of subjective responses from 30 highly trained musicians to music varying along the affective dimensions of arousal and valence. Experiment 2 examined a set of psychophysiological correlates of emotion in 20 different musicians by measuring heart rate, skin conductance, and facial electromyography during listening without requiring behavioral reports. [...

Comparison of methods for collecting and modeling dissimilarity data: applications to complex sound stimuli

Sorting procedures are frequently adopted as an alternative to dissimilarity ratings to measure the dissimilarity of large sets of stimuli in a comparatively short time. However, systematic empirical research on the consequences of this experiment-design choice is lacking. We carried out a behavioral experiment to assess the extent to which sorting procedures compare to dissimilarity ratings in terms of efficiency, reliability, and accuracy, and the extent to which data from different data-collection methods are redundant and are better fit by different distance models. Participants estimated the dissimilarity of either semantically charged environmental sounds or semantically neutral synthetic sounds. We considered free and hierarchical sorting and derived indications concerning the properties of constrained and truncated hierarchical sorting methods from hierarchical sorting data. Results show that the higher efficiency of sorting methods comes at a considerable cost in terms of data reliability and accuracy. This loss appears to be minimized with truncated hierarchical sorting methods that start from a relatively low number of groups of stimuli. Finally, variations in data-collection method differentially affect the fit of various distance models at the group-average and individual levels. On the basis of these results, we suggest adopting sorting as an alternative to dissimilarity-rating methods only when strictly necessary. We also suggest analyzing the raw behavioral dissimilarities, and avoiding modeling them with one single distance model