Research Proposal for Study: Can Artifical Emulation of Synesthesia Aid Visual Word Recognition?

The field of synesthesia has largely been unexplored beyond mere analysis of the condition and only a few papers on the topic of new applications have been published within the last decade. The condition can potentially provide great benefits to the study of the psychology of language along with its role in language use itself. As understanding of this condition continues to grow, one can draw links between it and its effect on language use, thereby enabling a greater understanding of the language process itself. Synesthesia involves the stimulation of one sense along with involuntary activation of another sensory pathway. For those with synesthesia, ninety-two percent of cases are triggered via linguistic inducers, showing deep ties between the pathways used by the condition and those involved with language faculties. This paper will first cover synesthesia along with some of the many permutations the condition takes, then review previous research into the role of synesthesia in psycholinguistics including the subject of improved recall. After providing background, this paper will provide a framework for carrying out artificial simulation of the condition to improve participant’s word recognition. Finally, it will attempt to answer the question of what could be done with the results gleaned. The linking of sensory activation, or “cross talk” as it is commonly known, points to new understandings of how humans process information, especially language. Research can also be applied to further investigation of how those without the condition could apply similar techniques to facilitate visual word recognition. The role of synesthesia has not been adequately researched in light of its relation to lexical decision tasks. It has been previously shown how cognitive research can be effectively 3 informed by synesthesia research, yet there are virtually no extant studies on the application of synesthesia along with the possibility of artificial simulation. This study attempts to address this issue by simulating synesthesia through a period of training; subsequently, the results of the study will provide a baseline for the work involved in future application of synesthesia to other areas

Artificial Odour-Vision Syneasthesia via Olfactory Sensory Argumentation

The phenomenology of synaesthesia provides numerous cognitive benefits, which could be used towards augmenting interactive experiences with more refined multisensorial capabilities leading to more engaging and enriched experiences, better designs, and more transparent human-machine interfaces. In this study, we report a novel framework for the transformation of odours into the visual domain by applying the ideology from synaesthesia, to a low cost, portable, augmented reality/virtual reality system. The benefits of generating an artificial form of synesthesia are outlined and implemented using a custom made electronic nose to gather information about odour sources which is then sent to a mobile computing engine for characterisation, classification, and visualisation. The odours are visualised in the form of coloured 2D abstract shapes in real-time. Our results show that our affordable system has the potential to increase human odour discrimination comparable to that of natural syneasthesia highlighting the prospects for augmenting human-machine interfaces with an artificial form of this phenomenon

Virtual Synaesthesia: Crossmodal Correspondences and Synesthetic Experiences

As technology develops to allow for the integration of additional senses into interactive experiences, there is a need to bridge the divide between the real and the virtual in a manner that stimulates the five senses consistently and in harmony with the sensory expectations of the user. Applying the philosophy of a neurological condition known as synaesthesia and crossmodal correspondences, defined as the coupling of the senses, can provide numerous cognitive benefits and offers an insight into which senses are most likely to be ‘bound’ together. This thesis aims to present a design paradigm called ‘virtual synaesthesia’ the goal of the paradigm is to make multisensory experiences more human-orientated by considering how the brain combines senses in both the general population (crossmodal correspondences) and within a select few individuals (natural synaesthesia). Towards this aim, a literature review is conducted covering the related areas of research umbrellaed by the concept of ‘virtual synaesthesia’. Its research areas are natural synaesthesia, crossmodal correspondences, multisensory experiences, and sensory substitution/augmentation. This thesis examines augmenting interactive and multisensory experiences with strong (natural synaesthesia) and weak (crossmodal correspondences) synaesthesia. This thesis answers the following research questions: Is it possible to replicate the underlying cognitive benefits of odour-vision synaesthesia? Do people have consistent correspondences between olfaction and an aggregate of different sensory modalities? What is the nature and origin of these correspondences? And Is it possible to predict the crossmodal correspondences attributed to odours? The benefits of augmenting a human-machine interface using an artificial form of odour-vision synaesthesia are explored to answer these questions. This concept is exemplified by transforming odours transduced using a custom-made electronic nose and transforming an odour's ‘chemical footprint’ into a 2D abstract shape representing the current odour. Electronic noses can transform odours in the vapour phase generating a series of electrical signals that represent the current odour source. Weak synaesthesia (crossmodal correspondences) is then investigated to determine if people have consistent correspondences between odours and the angularity of shapes, the smoothness of texture, perceived pleasantness, pitch, musical, and emotional dimensions. Following on from this research, the nature and origin of these correspondences were explored using the underlying hedonic (values relating to pleasantness), semantic (knowledge of the identity of the odour) and physicochemical (the physical and chemical characteristics of the odour) dependencies. The final research chapter investigates the possibility of removing the bottleneck of conducting extensive human trials by determining what the crossmodal correspondences towards specific odours are by developing machine learning models to predict the crossmodal perception of odours using their underlying physicochemical features. The work presented in this thesis provides some insight and evidence of the benefits of incorporating the concept ‘virtual synaesthesia’ into human-machine interfaces and research into the methodology embodied by ‘virtual synaesthesia’, namely crossmodal correspondences. Overall, the work presented in this thesis shows potential for augmenting multisensory experiences with more refined capabilities leading to more enriched experiences, better designs, and a more intuitive way to convey information crossmodally

How touch and hearing influence visual processing in sensory substitution, synaesthesia and cross-modal correspondences

Sensory substitution devices (SSDs) systematically turn visual dimensions into patterns of tactile or auditory stimulation. After training, a user of these devices learns to translate these audio or tactile sensations back into a mental visual picture. Most previous SSDs translate greyscale images using intuitive cross-sensory mappings to help users learn the devices. However more recent SSDs have started to incorporate additional colour dimensions such as saturation and hue. Chapter two examines how previous SSDs have translated the complexities of colour into hearing or touch. The chapter explores if colour is useful for SSD users, how SSD and veridical colour perception differ and how optimal cross-sensory mappings might be considered. After long-term training, some blind users of SSDs report visual sensations from tactile or auditory stimulation. A related phenomena is that of synaesthesia, a condition where stimulation of one modality (i.e. touch) produces an automatic, consistent and vivid sensation in another modality (i.e. vision). Tactile-visual synaesthesia is an extremely rare variant that can shed light on how the tactile-visual system is altered when touch can elicit visual sensations. Chapter three reports a series of investigations on the tactile discrimination abilities and phenomenology of tactile-vision synaesthetes, alongside questionnaire data from synaesthetes unavailable for testing. Chapter four introduces a new SSD to test if the presentation of colour information in sensory substitution affects object and colour discrimination. Chapter five presents experiments on intuitive auditory-colour mappings across a wide variety of sounds. These findings are used to predict the reported colour hallucinations resulting from LSD use while listening to these sounds. Chapter six uses a new sensory substitution device designed to test the utility of these intuitive sound-colour links for visual processing. These findings are discussed with reference to how cross-sensory links, LSD and synaesthesia can inform optimal SSD design for visual processing

Developing an artificial colour-sound association for musical composition

Some great composers - Messiaen, Scriabin, Liszt in exemplum - have been found to have an internal colour world that responds to music and characterises the way they experience and express music. Many of these artists, it could be strongly argued, had the neurological trait synaesthesia. The author, a non-synaesthete, creates a logical correspondence between colour and sound and uses it to explore the tonality of aesthetic colour combinations in nature and modern life. He argues that if the colour-sound practitioner is consistent in their colour-sound association, they can benefit in harmonic discoveries as the synaesthete does. It was found that the harmonies produced were strange, new tonalities that do not repeat in each octave but form something akin to macro-chords, shifting density in different registers. The author produced a series of short scores for small ensembles to explore the possible merits of drawing harmony in music from harmony in colour

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

Synaesthesia Materialisation: Approaches to Applying Synaesthesia as a Provocation for Generating Creative Ideas Within the Context of Design

For the past three decades, research on the topic of synaesthesia has been largely dominated by the field of psychology and neuroscience, and has focused on scientifically investigating its experience and causes to define the phenomenon of synaesthesia. However, the scientific research on this subject is now enquiring into potential future implementations and asking how this subject may be useful to wider audiences, and is attempting to expand its research spectrum beyond the mere scientific analysis. This PhD research in design by practice attempts to contribute and expand this scope: it shares a creative interpretation of synaesthesia research and questions its existing boundary. Synaesthesia is one of those rare subjects where both science and creative context intersect and nurture each other. By looking into this PhD research, readers may gain insights of how a designer tries to discover a new value within this interdisciplinary context. This research contributes three types of new knowledge and new perspectives. Firstly, it provides a new interpretation and awareness in and of synaesthesia research, and expands its research boundaries, moving from analysis based research to application based research. Secondly, it outlines three approaches, a range of themes and toolkits for using synaesthesia as a provocation in generating creative ideas in the design process. Thirdly, it identifies the differences between previous synaesthesia application research and current application research within the context of design