Computational Auditory Scene Analysis

An overview of the work of the Laboratory for Recognition and Organization of Speech and Audio, Department of Electrical Engineering, Columbia University, on signal separation

Computational Models of Auditory Scene Analysis: A Review

Auditory scene analysis (ASA) refers to the process(es) of parsing the complex acoustic input into auditory perceptual objects representing either physical sources or temporal sound patterns, such as melodies, which contributed to the sound waves reaching the ears. A number of new computational models accounting for some of the perceptual phenomena of ASA have been published recently. Here we provide a theoretically motivated review of these computational models, aiming to relate their guiding principles to the central issues of the theoretical framework of ASA. Specifically, we ask how they achieve the grouping and separation of sound elements and whether they implement some form of competition between alternative interpretations of the sound input. We consider the extent to which they include predictive processes, as important current theories suggest that perception is inherently predictive, and also how they have been evaluated. We conclude that current computational models of ASA are fragmentary in the sense that rather than providing general competing interpretations of ASA, they focus on assessing the utility of specific processes (or algorithms) for finding the causes of the complex acoustic signal. This leaves open the possibility for integrating complementary aspects of the models into a more comprehensive theory of ASA

Computational auditory scene analysis exploiting speech-recognition knowledge

The field of computational auditory scene analysis (CASA) strives to build computer models of the human ability to interpret sound mixtures as the combination of distinct sources. A major obstacle to this enterprise is defining and incorporating the kind of high level knowledge of real-world signal structure exploited by listeners. Speech recognition, while typically ignoring the problem of nonspeech inclusions, has been very successful at deriving powerful statistical models of speech structure from training data. In this paper, we describe a scene analysis system that includes both speech and nonspeech components, addressing the problem of working backwards from speech recognizer output to estimate the speech component of a mixture. Ultimately, such hybrid approaches will require more radical adaptation of current speech recognition approaches

Generation and Analysis of a Social Network: Hamlet

This paper examines the generation and analysis of a social network produced from Shakespeare’s play Hamlet. An XML file of Hamlet was parsed to extract the characters within the play and also identify when the characters appeared within the same scene. After parsing the speakers and the connections between characters, a network graph was generated that displayed all the characters in Hamlet, represented by nodes, and edges that represented the connections between characters as measured by their scene co-appearance. The results of the network graph were then compared to a published social network for Hamlet created by hand. The two social networks showed strong similarities in character centrality but also showed differences in the number of character nodes and edges. In addition to the case study, we present a suite of tools that provide a framework for computational analysis of future plays

Computational Audiovisual Scene Analysis

Yan R. Computational Audiovisual Scene Analysis. Bielefeld: Universitätsbibliothek Bielefeld; 2014.In most real-world situations, a robot is interacting with multiple people. In this case, understanding of the dialogs is essential. However, dialog scene analysis is missing in most existing systems of human-robot interaction. In such systems, only one speaker can talk with the robot or each speaker wears an attached microphone or a headset. The target of Computational AudioVisual Scene Analysis (CAVSA) is therefore making dialogs between humans and robots more natural and flexible. The CAVSA system is able to learn how many speakers are in the scenario, where the speakers are and who is currently speaking. CAVSA is a challenging task due to the complexity of dialogue scenarios. First, speakers are unknown in advance, thus a database for training high-level features beforehand to recognize faces or voices is not available. Second, people can dynamically come into and leave the scene, may move all the time and even change their locations outside the camera field of view. Third, the robot can not see all the people at the same time due to limited camera field of view and head movements. Moreover, a sound could be related to a person who stands outside the camera field of view and has never been seen. I will show that the CAVSA system is able to assign words to corresponding speakers. A speaker is recognized again when he leaves and enters the scene, or changes his position even with a newly appearing person

Auditory Scene Analysis: phenomena, theories and computational models

Lecture on auditory scene analysis

Integrating CASA with other approaches

Discusses how Computational Auditory Scene Analysis may be integrated with other separation mechanisms