The temporal pattern of impulses in primary afferents analogously encodes touch and hearing information

An open question in neuroscience is the contribution of temporal relations between individual impulses in primary afferents in conveying sensory information. We investigated this question in touch and hearing, while looking for any shared coding scheme. In both systems, we artificially induced temporally diverse afferent impulse trains and probed the evoked perceptions in human subjects using psychophysical techniques. First, we investigated whether the temporal structure of a fixed number of impulses conveys information about the magnitude of tactile intensity. We found that clustering the impulses into periodic bursts elicited graded increases of intensity as a function of burst impulse count, even though fewer afferents were recruited throughout the longer bursts. The interval between successive bursts of peripheral neural activity (the burst-gap) has been demonstrated in our lab to be the most prominent temporal feature for coding skin vibration frequency, as opposed to either spike rate or periodicity. Given the similarities between tactile and auditory systems, second, we explored the auditory system for an equivalent neural coding strategy. By using brief acoustic pulses, we showed that the burst-gap is a shared temporal code for pitch perception between the modalities. Following this evidence of parallels in temporal frequency processing, we next assessed the perceptual frequency equivalence between the two modalities using auditory and tactile pulse stimuli of simple and complex temporal features in cross-sensory frequency discrimination experiments. Identical temporal stimulation patterns in tactile and auditory afferents produced equivalent perceived frequencies, suggesting an analogous temporal frequency computation mechanism. The new insights into encoding tactile intensity through clustering of fixed charge electric pulses into bursts suggest a novel approach to convey varying contact forces to neural interface users, requiring no modulation of either stimulation current or base pulse frequency. Increasing control of the temporal patterning of pulses in cochlear implant users might improve pitch perception and speech comprehension. The perceptual correspondence between touch and hearing not only suggests the possibility of establishing cross-modal comparison standards for robust psychophysical investigations, but also supports the plausibility of cross-sensory substitution devices

The AffectToolbox: Affect Analysis for Everyone

In the field of affective computing, where research continually advances at a rapid pace, the demand for user-friendly tools has become increasingly apparent. In this paper, we present the AffectToolbox, a novel software system that aims to support researchers in developing affect-sensitive studies and prototypes. The proposed system addresses the challenges posed by existing frameworks, which often require profound programming knowledge and cater primarily to power-users or skilled developers. Aiming to facilitate ease of use, the AffectToolbox requires no programming knowledge and offers its functionality to reliably analyze the affective state of users through an accessible graphical user interface. The architecture encompasses a variety of models for emotion recognition on multiple affective channels and modalities, as well as an elaborate fusion system to merge multi-modal assessments into a unified result. The entire system is open-sourced and will be publicly available to ensure easy integration into more complex applications through a well-structured, Python-based code base - therefore marking a substantial contribution toward advancing affective computing research and fostering a more collaborative and inclusive environment within this interdisciplinary field

The knowledge domain of affective computing: a scientometric review

Purpose – The aim of this study is to investigate the bibliographical information about Affective Computing identifying advances, trends, major papers, connections, and areas of research. Design/methodology/approach – A scientometric analysis was applied using CiteSpace, of 5,078 references about Affective Computing imported from the Web-of-Science Core Collection, covering the period of 1991-2016. Findings – The most cited, creative, burts and central references are displayed by areas of research, using metrics and througout-time visualization. Research limitations/implications – Interpretation is limited to references retrieved from theWeb-of-Science Core Collection in the fields of management, psychology and marketing. Nevertheless, the richness of bibliographical data obtained, largely compensates this limitation. Practical implications – The study provides managers with a sound body of knowledge on Affective Computing, with which they can capture general public emotion in respect of their products and services, and on which they can base their marketing intelligence gathering, and strategic planning. Originality/value – The paper provides new opportunities for companies to enhance their capabilities in terms of customer relationships.info:eu-repo/semantics/acceptedVersio

A Hierarchical Regression Chain Framework for Affective Vocal Burst Recognition

As a common way of emotion signaling via non-linguistic vocalizations, vocal burst (VB) plays an important role in daily social interaction. Understanding and modeling human vocal bursts are indispensable for developing robust and general artificial intelligence. Exploring computational approaches for understanding vocal bursts is attracting increasing research attention. In this work, we propose a hierarchical framework, based on chain regression models, for affective recognition from VBs, that explicitly considers multiple relationships: (i) between emotional states and diverse cultures; (ii) between low-dimensional (arousal & valence) and high-dimensional (10 emotion classes) emotion spaces; and (iii) between various emotion classes within the high-dimensional space. To address the challenge of data sparsity, we also use self-supervised learning (SSL) representations with layer-wise and temporal aggregation modules. The proposed systems participated in the ACII Affective Vocal Burst (A-VB) Challenge 2022 and ranked first in the "TWO'' and "CULTURE'' tasks. Experimental results based on the ACII Challenge 2022 dataset demonstrate the superior performance of the proposed system and the effectiveness of considering multiple relationships using hierarchical regression chain models.Comment: 5 pages, 3 figures, 5 table

Changes in the McGurk Effect Across Phonetic Contexts

To investigate the process underlying audiovisual speech perception, the McGurk illusion was examined across a range of phonetic contexts. Two major changes were found. First, the frequency of illusory /g/ fusion percepts increased relative to the frequency of illusory /d/ fusion percepts as vowel context was shifted from /i/ to /a/ to /u/. This trend could not be explained by biases present in perception of the unimodal visual stimuli. However, the change found in the McGurk fusion effect across vowel environments did correspond systematically with changes in second format frequency patterns across contexts. Second, the order of consonants in illusory combination percepts was found to depend on syllable type. This may be due to differences occuring across syllable contexts in the timecourses of inputs from the two modalities as delaying the auditory track of a vowel-consonant stimulus resulted in a change in the order of consonants perceived. Taken together, these results suggest that the speech perception system either fuses audiovisual inputs into a visually compatible percept with a similar second formant pattern to that of the acoustic stimulus or interleaves the information from different modalities, at a phonemic or subphonemic level, based on their relative arrival times.National Institutes of Health (R01 DC02852

Global timing: a conceptual framework to investigate the neural basis of rhythm perception in humans and non-human species

Timing cues are an essential feature of music. To understand how the brain gives rise to our experience of music we must appreciate how acoustical temporal patterns are integrated over the range of several seconds in order to extract global timing. In music perception, global timing comprises three distinct but often interacting percepts: temporal grouping, beat, and tempo. What directions may we take to further elucidate where and how the global timing of music is processed in the brain? The present perspective addresses this question and describes our current understanding of the neural basis of global timing perception

The effect of emotions on brand recall by gender using voice emotion response with optimal data analysis

Purpose—To analyses the effect of emotions obtained by oral reproduction of advertising slogans established via Voice Emotion Response software on brand recall by gender; and to show the relevance for marketing communication of combining “human–computer Interaction (HCI)” with “affective computing (AC)” as part of their mission. Design/methodology/approach—A qualitative data analysis did the review of the scientific literature retrieved from Web-of-Science Core Collection (WoSCC), using CiteSpace’ scientometric technique; the quantitative data analysis did the analysis of brand recall over a sample of Taiwan’ participants by “optimal data analysis”. Findings—Advertising effectiveness has a positive association with emotions; brand recall varies with gender; and “HCI” connected with “AC” is an emerging area of research. Research limitations/implications—The selection of articles obtained depend on the terms used in WoSCC, and this study used only five emotions. Still the richness of the data gives some compensation. Practical implications—Marketers involved with brands need a body of knowledge on which to base their marketing communication intelligence gathering and strategic planning. Originality/value—It provides exploratory research findings related to the use of automatic tools capable of mining emotions by gender in real time, which could enhance the feedback of customers toward their brands.info:eu-repo/semantics/acceptedVersio

Perceptual processing of partially and fully assimilated words in French

International audienceModels of speech perception attribute a different role to contextual information in the processing of assimilated speech. The present study examined perceptual processing of regressive voice assimilation in French. This phonological variation is asymmetric in that assimilation is partial for voiced stops and near-complete for voiceless stops. Two auditory- visual cross-modal form priming experiments were used to examine perceptual compensation for assimilation in French words with voiceless versus voiced stop offsets. The results show that, for the former segments, assimilating context enhances underlying form recovery, whereas it does not for the latter. These results suggest that two sources of information -- contextual information, and bottom-up information from the assimilated forms themselves -- are complementary and both come into play during the processing of fully or partially assimilated word forms

Optimality and limitations of audio-visual integration for cognitive systems

Multimodal integration is an important process in perceptual decision-making. In humans, this process has often been shown to be statistically optimal, or near optimal: sensory information is combined in a fashion that minimizes the average error in perceptual representation of stimuli. However, sometimes there are costs that come with the optimization, manifesting as illusory percepts. We review audio-visual facilitations and illusions that are products of multisensory integration, and the computational models that account for these phenomena. In particular, the same optimal computational model can lead to illusory percepts, and we suggest that more studies should be needed to detect and mitigate these illusions, as artifacts in artificial cognitive systems. We provide cautionary considerations when designing artificial cognitive systems with the view of avoiding such artifacts. Finally, we suggest avenues of research toward solutions to potential pitfalls in system design. We conclude that detailed understanding of multisensory integration and the mechanisms behind audio-visual illusions can benefit the design of artificial cognitive systems.Human-Robot Interactio