Synesthetic Sensor Fusion via a Cross-Wired Artificial Neural Network.

The purpose of this interdisciplinary study was to examine the behavior of two artificial neural networks cross-wired based on the synesthesia cross-wiring hypothesis. Motivation for the study was derived from the study of psychology, robotics, and artificial neural networks, with perceivable application in the domain of mobile autonomous robotics where sensor fusion is a current research topic. This model of synesthetic sensor fusion does not exhibit synesthetic responses. However, it was observed that cross-wiring two independent networks does not change the functionality of the individual networks, but allows the inputs to one network to partially determine the outputs of the other network in some cases. Specifically, there are measurable influences of network A on network B, and yet network B retains its ability to respond independently

Music Notation-to-Color Synesthesia and Early Stages of Music Education: A Grounded Theory Study

Problem Synesthesia is a neurological condition characterized by over-abundant neural connectivity between commonly highly specialized areas of the brain. The developmental form of the condition often results in automatic and consistent cross-sensory associations between perceived stimuli and commonly unrelated brain regions. This research contemplates the specific form of music notation-to-color synesthesia and its impact on early stages of music education. Synesthetes with this mode of the condition tend to involuntarily yet consistently associate music-notational concepts with colors, thus rendering their assimilation of these concepts unique and individualized. The purpose of this study is to determine the extent of these individualized experiences from original narratives. Method This study entails a grounded theory qualitative approach, through which 12 from participants were interviewed cross-culturally (7 featured nationalities). All participants were adults with music notation-to-color synesthesia who experienced music instruction in a Western cultural context. Data collection methodology involved a written survey, inperson (or live Zoom) interviews, and shared document analysis. Qualitative analytical methodology was used via coding strategies to discover surfacing themes, emerging issues, and commonalities among the narratives. Results Five overarching categories of commonalities were identified in this study. Firstly, participants shared generalities of synesthetic perceptions of music notation involving color, such as their awareness about their condition, the qualities of their experiences, the conceptual basis of their associations, among other characteristics. Interviewees also alluded to the mechanisms involved in the perception of music notation, such as the positive impact of their music notation-to-color synesthesia on memory as well as the negative implications of synesthetic incongruence. The spatial location of synesthetic perceptions varied among participants. Interviewees reported projecting on the page of music and associating in their mind\u27s eye —two common themes in the literature. Some participants, however, have also mentioned a middleground location that does not fit only one of these categories. Finally, this study analyzed themes relating to the implications of this form of synesthesia for music education, with attention to awareness on the part of educators, instructional intentionality, validation, reinforcement of student individuality, and conscious use of the condition. Moreover, other themes and future research possibilities were analyzed. -- Conclusion This study arrived at two grounded theory models. The first comprises a grounded theory of the experiences shared by participants. This theoretical model articulates the salient themes, such as positive and negative traits of notation-to-color perceptions and spatial location of perceptions. In special, this theory argues for a tendency for conceptually-based notation-to-color synesthesia among participants. The second grounded theory model advanced in this research entails an educational approach that would benefit awareness and intentionality in addressing students with music notation-to-color synesthesia. It discusses philosophical foundations, a theoretical framework, and methodological considerations that may transform how music notation-to-color students are accounted for in curricula. The study concludes by offering pedagogical suggestions derived from the methodological considerations. Firstly, it advances a linear process for identification, verification, and addressing of synesthesia. Secondly, it proposes the elimination of excessive notational information and gradual learning as initial strategies that could benefit music notation-to-color synesthetes in learning new notational elements

Investigation of audio feature extraction and audiovisual correspondences for bio-inspired auditory to visual substitution systems

Résumé: Les systèmes de substitution sensorielle convertissent des stimuli d’une modalité sensorielle en des stimuli d’une autre modalité. Ils peuvent fournir les moyens pour les personnes handicapées de percevoir des stimuli d’une modalité défectueuse par une autre modalité. Le but de ce projet de recherche était d’étudier des systèmes de substitution de l’audition vers la vision. Ce type de substitution n’est pas bien étudié probablement en raison de la complexité du système auditif et des difficultés résultant de la désadaptation entre les sons audibles qui peuvent changer avec des fréquences allant jusqu’à 20000 Hz et des stimuli visuels qui changent très lentement avec le temps afin d’être perçus. Deux problèmes spécifiques des systèmes de substitution de l’audition vers la vision ont été ciblés par cette étude: la recherche de correspondances audiovisuelles et l’extraction de caractéristiques auditives. Une expérience audiovisuelle a été réalisée en ligne pour trouver les associations entre les caractéristiques auditives (la fréquence fondamentale et le timbre) et visuelles (la forme, la couleur, et la position verticale). Une forte corrélation entre le timbre des sons utilisés et des formes visuelles a été observée. Les sujets ont fortement associé des timbres “doux” avec des formes arrondies bleues, vertes ou gris clair, des timbres “durs” avec des formes angulaires pointues rouges, jaunes ou gris foncé et des timbres comportant simultanément des éléments de douceur et de dureté avec un mélange des deux formes visuelles arrondies et angulaires. La fréquence fondamentale n’a pas été associée à la position verticale, ni le niveau de gris ou la couleur. Étant donné la correspondance entre le timbre et une forme visuelle, dans l’étape sui- vante, un modèle hiérarchique flexible et polyvalent bio-inspiré pour analyser le timbre et extraire des caractéristiques importantes du timbre a été développé. Inspiré par les découvertes dans les domaines des neurosciences, neurosciences computationnelles et de la psychoacoustique, non seulement le modèle extrait-il des caractéristiques spectrales et temporelles d’un signal, mais il analyse également les modulations d’amplitude sur différentes échelles de temps. Il utilise un banc de filtres cochléaires pour résoudre les composantes spectrales d’un son, l’inhibition latérale pour améliorer la résolution spectrale, et un autre banc de filtres de modulation pour extraire l’enveloppe temporelle et la rugosité du son à partir des modulations d’amplitude. Afin de démontrer son potentiel pour la représentation du timbre, le modèle a été évalué avec succès pour trois applications : 1) la comparaison avec les valeurs subjectives de la rugosité 2) la classification d’instruments de musique 3) la sélection de caractéristiques pour les sons qui ont été regroupés en fonction de la forme visuelle qui leur avait été attribuée dans l’expérience audiovisuelle. La correspondance entre le timbre et la forme visuelle qui a été révélée par cette étude et le modèle proposé pour l’analyse de timbre peuvent être utilisés pour développer des systèmes de substitution de l’audition vers la vision intuitifs codant le timbre en formes visuelles.Abstract: Sensory substitution systems encode a stimulus modality into another stimulus modality. They can provide the means for handicapped people to perceive stimuli of an impaired modality through another modality. The purpose of this study was to investigate auditory to visual substitution systems. This type of sensory substitution is not well-studied probably because of the complexities of the auditory system and the difficulties arising from the mismatch between audible sounds that can change with frequencies up to 20000 Hz and visual stimuli that should change very slowly with time to be perceived. Two specific problems of auditory to visual substitution systems were targeted in this research: the investigation of audiovisual correspondences and the extraction of auditory features. An audiovisual experiment was conducted online to find the associations between the auditory (pitch and timbre) and visual (shape, color, height) features. One hundred and nineteen subjects took part in the experiments. A strong association between timbre of envelope normalized sounds and visual shapes was observed. Subjects strongly associated soft timbres with blue, green or light gray rounded shapes, harsh timbres with red, yellow or dark gray sharp angular shapes and timbres having elements of softness and harshness together with a mixture of the previous two shapes. Fundamental frequency was not associated with height, grayscale or color. Given the correspondence between timbre and shapes, in the next step, a flexible and multipurpose bio-inspired hierarchical model for analyzing timbre and extracting the important timbral features was developed. Inspired by findings in the fields of neuroscience, computational neuroscience, and psychoacoustics, not only does the model extract spectral and temporal characteristics of a signal, but it also analyzes amplitude modulations on different timescales. It uses a cochlear filter bank to resolve the spectral components of a sound, lateral inhibition to enhance spectral resolution, and a modulation filter bank to extract the global temporal envelope and roughness of the sound from amplitude modulations. To demonstrate its potential for timbre representation, the model was successfully evaluated in three applications: 1) comparison with subjective values of roughness, 2) musical instrument classification, and 3) feature selection for labeled timbres. The correspondence between timbre and shapes revealed by this study and the proposed model for timbre analysis can be used to develop intuitive auditory to visual substitution systems that encode timbre into visual shapes

Seeing Beyond the Visual: Sensory Perception and Synesthesia in Contemporary Installation Art

In this paper, I utilize scientific theories surrounding the neurological condition synesthesia to identify how multisensory installation art can challenge our understanding of the singularity of the senses. This exploration will form the basis of my theory of “synthetic synesthesia,” a term derived from my observation of synesthetic components in installation artworks that possess elements of multisensoriality, embodiment and immersion. This theory is applied to my analysis of select pieces by Marla Hlady and Peter De Cupere whose works involve sensory combinations, such as vision/hearing/touch (Hlady), and vision/olfactory (De Cupere). The works of both of my case studies reconsider how the participant interacts with an artwork by creating installations which augment the traditional visual art experience

Multi-Sensory Interaction for Blind and Visually Impaired People

This book conveyed the visual elements of artwork to the visually impaired through various sensory elements to open a new perspective for appreciating visual artwork. In addition, the technique of expressing a color code by integrating patterns, temperatures, scents, music, and vibrations was explored, and future research topics were presented. A holistic experience using multi-sensory interaction acquired by people with visual impairment was provided to convey the meaning and contents of the work through rich multi-sensory appreciation. A method that allows people with visual impairments to engage in artwork using a variety of senses, including touch, temperature, tactile pattern, and sound, helps them to appreciate artwork at a deeper level than can be achieved with hearing or touch alone. The development of such art appreciation aids for the visually impaired will ultimately improve their cultural enjoyment and strengthen their access to culture and the arts. The development of this new concept aids ultimately expands opportunities for the non-visually impaired as well as the visually impaired to enjoy works of art and breaks down the boundaries between the disabled and the non-disabled in the field of culture and arts through continuous efforts to enhance accessibility. In addition, the developed multi-sensory expression and delivery tool can be used as an educational tool to increase product and artwork accessibility and usability through multi-modal interaction. Training the multi-sensory experiences introduced in this book may lead to more vivid visual imageries or seeing with the mind’s eye

Metaphor, theories of concepts and biological reductionism

This thesis is an attempt to present a multidisciplinary approach to adjectival polysemy, particularly adjectival polysemy of a metaphorical type, and its underlying conceptual structure. The last ten years have clearly shown a tendency towards reducing the number of meanings and the idea of metaphor as a mechanism of concept formation has been gaining much force, influencing research in linguistics, psychology and cognitive science. Despite that fact, the long-standing tradition of analytic philosophy did not succumb to the attack. However few contentions are shared in these different fields, one is held unquestioningly by almost everyone. It is the literal-metaphorical distinction which is at the heart of both traditional philosophy and the theory of embodied realism. Drawing extensively on evidence from research on cross-modal perception, synesthesia, double-function terms in cross-cultural studies, child development, psycholinguistic experiments and experiments with brain-damaged subjects, reinterpreting the available data and analyzing in detail theories of concepts contained in cognitive linguistics, lexical semantics and informational semantics, the thesis casts doubt on the validity of the literalmetaphorical distinction, for this class of examples. It stipulates the existence of psychologically primitive concepts, which are likely to be atomic and innate, and offers a no-polysemy view of conceptual structure with implications for linguistic polysemy. It also shows the limits of biological reductionism and emphasizes the need for functional approaches to cognition. The proposed alternative is both unexpected and exciting, and may serve as a basis for bringing together empirical evidence and philosophical coherence in a non-contradictory wa

How touch and hearing influence visual processing in sensory substitution, synaesthesia and cross-modal correspondences

Sensory substitution devices (SSDs) systematically turn visual dimensions into patterns of tactile or auditory stimulation. After training, a user of these devices learns to translate these audio or tactile sensations back into a mental visual picture. Most previous SSDs translate greyscale images using intuitive cross-sensory mappings to help users learn the devices. However more recent SSDs have started to incorporate additional colour dimensions such as saturation and hue. Chapter two examines how previous SSDs have translated the complexities of colour into hearing or touch. The chapter explores if colour is useful for SSD users, how SSD and veridical colour perception differ and how optimal cross-sensory mappings might be considered. After long-term training, some blind users of SSDs report visual sensations from tactile or auditory stimulation. A related phenomena is that of synaesthesia, a condition where stimulation of one modality (i.e. touch) produces an automatic, consistent and vivid sensation in another modality (i.e. vision). Tactile-visual synaesthesia is an extremely rare variant that can shed light on how the tactile-visual system is altered when touch can elicit visual sensations. Chapter three reports a series of investigations on the tactile discrimination abilities and phenomenology of tactile-vision synaesthetes, alongside questionnaire data from synaesthetes unavailable for testing. Chapter four introduces a new SSD to test if the presentation of colour information in sensory substitution affects object and colour discrimination. Chapter five presents experiments on intuitive auditory-colour mappings across a wide variety of sounds. These findings are used to predict the reported colour hallucinations resulting from LSD use while listening to these sounds. Chapter six uses a new sensory substitution device designed to test the utility of these intuitive sound-colour links for visual processing. These findings are discussed with reference to how cross-sensory links, LSD and synaesthesia can inform optimal SSD design for visual processing

Self-Thinking and Self-Learning:Designing a Personal Tool for Making Visual Music

This Major Research Project illustrates the Inclusive Design principle of “one-size-fits-one” through an exploration of the process of designing a prototype of Visual Music making tool to match the needs and capabilities of one user—myself. Through an auto-ethnographic self-exploration and self-analysis of my dual identities as an international student from China and a junior graphic designer, I discovered my needs and capabilities as user of visual music tools. Using this information, I reviewed three popular visual music tools, to arrive at features for a prototype tool for myself, supplemented by basic features gathered through literature review. With these features, I designed a prototype music visualization tool named MyTool with interface mock-ups. MyTool is an experiment for creating customized tools through self-study, which could urge designers of popular tools to create more flexible tools, thereby expanding the user community of visual music makers like myself and kicking in a virtuous cycle of inclusion in the context of visual music production and consumption