462 research outputs found
LAVSS: Location-Guided Audio-Visual Spatial Audio Separation
Existing machine learning research has achieved promising results in monaural
audio-visual separation (MAVS). However, most MAVS methods purely consider what
the sound source is, not where it is located. This can be a problem in VR/AR
scenarios, where listeners need to be able to distinguish between similar audio
sources located in different directions. To address this limitation, we have
generalized MAVS to spatial audio separation and proposed LAVSS: a
location-guided audio-visual spatial audio separator. LAVSS is inspired by the
correlation between spatial audio and visual location. We introduce the phase
difference carried by binaural audio as spatial cues, and we utilize positional
representations of sounding objects as additional modality guidance. We also
leverage multi-level cross-modal attention to perform visual-positional
collaboration with audio features. In addition, we adopt a pre-trained monaural
separator to transfer knowledge from rich mono sounds to boost spatial audio
separation. This exploits the correlation between monaural and binaural
channels. Experiments on the FAIR-Play dataset demonstrate the superiority of
the proposed LAVSS over existing benchmarks of audio-visual separation. Our
project page: https://yyx666660.github.io/LAVSS/.Comment: Accepted by WACV202
Wave Field Synthesis in a listening room
This thesis investigates the influence of the listening room on sound fields synthesised by Wave Field Synthesis. Methods are developed that allow for investigation of the spatial and timbral perception of Wave Field Synthesis in a reverberant environment using listening experiments based on simulation by binaural synthesis and room acoustical simulation. The results can serve as guidelines for the design of listening rooms for Wave Field Synthesis.Diese Dissertation untersucht den Einfluss des Wiedergaberaums auf Schallfelder, die mit Wellenfeldsynthese synthetisiert werden. Es werden Methoden zur Untersuchung von räumlicher und klangfarblicher Wahrnehmung von Wellenfeldsynthese in einer reflektierenden Umgebung mittels Hörversuchen entwickelt, die auf Simulation mit Binauralsynthese und raumakustischer Simulation beruhen. Die Ergebnisse können als Richtlinien zur Gestaltung von Wiedergaberäumen für Wellenfeldsynthese dienen
Facial Action Recognition Combining Heterogeneous Features via Multi-Kernel Learning
International audienceThis paper presents our response to the first interna- tional challenge on Facial Emotion Recognition and Analysis. We propose to combine different types of features to automatically detect Action Units in facial images. We use one multi-kernel SVM for each Action Unit we want to detect. The first kernel matrix is computed using Local Gabor Binary Pattern histograms and a histogram intersection kernel. The second kernel matrix is computed from AAM coefficients and an RBF kernel. During the training step, we combine these two types of features using the recently proposed SimpleMKL algorithm. SVM outputs are then averaged to exploit temporal information in the sequence. To eval- uate our system, we perform deep experimentations on several key issues: influence of features and kernel function in histogram- based SVM approaches, influence of spatially-independent in- formation versus geometric local appearance information and benefits of combining both, sensitivity to training data and interest of temporal context adaptation. We also compare our results to those of the other participants and try to explain why our method had the best performance during the FERA challenge
A Survey on Deep Learning in Medical Image Analysis
Deep learning algorithms, in particular convolutional networks, have rapidly
become a methodology of choice for analyzing medical images. This paper reviews
the major deep learning concepts pertinent to medical image analysis and
summarizes over 300 contributions to the field, most of which appeared in the
last year. We survey the use of deep learning for image classification, object
detection, segmentation, registration, and other tasks and provide concise
overviews of studies per application area. Open challenges and directions for
future research are discussed.Comment: Revised survey includes expanded discussion section and reworked
introductory section on common deep architectures. Added missed papers from
before Feb 1st 201
Multi-Sensory Interaction for Blind and Visually Impaired People
This book conveyed the visual elements of artwork to the visually impaired through various sensory elements to open a new perspective for appreciating visual artwork. In addition, the technique of expressing a color code by integrating patterns, temperatures, scents, music, and vibrations was explored, and future research topics were presented. A holistic experience using multi-sensory interaction acquired by people with visual impairment was provided to convey the meaning and contents of the work through rich multi-sensory appreciation. A method that allows people with visual impairments to engage in artwork using a variety of senses, including touch, temperature, tactile pattern, and sound, helps them to appreciate artwork at a deeper level than can be achieved with hearing or touch alone. The development of such art appreciation aids for the visually impaired will ultimately improve their cultural enjoyment and strengthen their access to culture and the arts. The development of this new concept aids ultimately expands opportunities for the non-visually impaired as well as the visually impaired to enjoy works of art and breaks down the boundaries between the disabled and the non-disabled in the field of culture and arts through continuous efforts to enhance accessibility. In addition, the developed multi-sensory expression and delivery tool can be used as an educational tool to increase product and artwork accessibility and usability through multi-modal interaction. Training the multi-sensory experiences introduced in this book may lead to more vivid visual imageries or seeing with the mind’s eye
- …