Enabling technologies for audio augmented reality systems

Audio augmented reality (AAR) refers to technology that embeds computer-generated auditory content into a user's real acoustic environment. An AAR system has specific requirements that set it apart from regular human--computer interfaces: an audio playback system to allow the simultaneous perception of real and virtual sounds; motion tracking to enable interactivity and location-awareness; the design and implementation of auditory display to deliver AAR content; and spatial rendering to display spatialised AAR content. This thesis presents a series of studies on enabling technologies to meet these requirements. A binaural headset with integrated microphones is assumed as the audio playback system, as it allows mobility and precise control over the ear input signals. Here, user position and orientation tracking methods are proposed that rely on speech signals recorded at the binaural headset microphones. To evaluate the proposed methods, the head orientations and positions of three conferees engaged in a discussion were tracked. The binaural microphones improved tracking performance substantially. The proposed methods are applicable to acoustic tracking with other forms of user-worn microphones. Results from a listening test investigating the effect of auditory display parameters on user performance are reported. The parameters studied were derived from the design choices to be made when implementing auditory display. The results indicate that users are able to detect a sound sample among distractors and estimate sample numerosity accurately with both speech and non-speech audio, if the samples are presented with adequate temporal separation. Whether or not samples were separated spatially had no effect on user performance. However, with spatially separated samples, users were able to detect a sample among distractors and simultaneously localise it. The results of this study are applicable to a variety of AAR applications that require conveying sample presence or numerosity. Spatial rendering is commonly implemented by convolving virtual sounds with head-related transfer functions (HRTFs). Here, a framework is proposed that interpolates HRTFs measured at arbitrary directions and distances. The framework employs Delaunay triangulation to group HRTFs into subsets suitable for interpolation and barycentric coordinates as interpolation weights. The proposed interpolation framework allows the realtime rendering of virtual sources in the near-field via HRTFs measured at various distances

Numerical Cognition and Autism Spectrum Traits in Adults

Evidence suggests that individuals with high-functioning autism spectrum disorder (ASD) may be particularly inclined toward math proficiency, especially in adulthood. There is also evidence, however, that many of those with an ASD struggle in math as children compared to their typically-developing peers. These ostensibly inconsistent findings may indicate that individuals with an ASD struggle with number sense, a precursor to formal math, rather than with formal math per se. This account is compatible with evidence of a specific form of neural dysregulation, excitatory/inhibitory imbalance, in ASD that results in reduced signal-to-noise ratios (SNR) for processes that occur in downstream neural regions (such as association cortex). Based on this view, formal math, a task with enhanced SNR due to standardization, would likely be intact for individuals with an ASD, while number sense, a domain localized to association cortex that lacks SNR enhancement via standardization, would take longer to sufficiently refine and would delay formal math acquisition for this population. The current studies examined whether a neural dysregulation account of ASD effectively predicts and explains numerical cognition performance across ASD traits. Experiment 1 examined whether scores on the Autism-Spectrum Quotient and the Systemizing Quotient predict performance on measures of numerical cognition consistent with a neural dysregulation account and in contrast to a traditional hyper-systemizing account of ASD. Experiment 2 examined whether strengthening the stimulus signal by presenting stimuli multimodally improves number sense performance across the range of ASD traits, as well as whether manipulation of high-level stimulus features affects multisensory integration in a manner consistent with a neural dysregulation account

The role of short-term memory and visuo-spatial skills in numerical magnitude processing: evidence from Turner syndrome

Peer reviewe

Ubiquitous Log Odds: A Common Representation of Probability and Frequency Distortion in Perception, Action, and Cognition

In decision from experience, the source of probability information affects how probability is distorted in the decision task. Understanding how and why probability is distorted is a key issue in understanding the peculiar character of experience-based decision. We consider how probability information is used not just in decision-making but also in a wide variety of cognitive, perceptual, and motor tasks. Very similar patterns of distortion of probability/frequency information have been found in visual frequency estimation, frequency estimation based on memory, signal detection theory, and in the use of probability information in decision-making under risk and uncertainty. We show that distortion of probability in all cases is well captured as linear transformations of the log odds of frequency and/or probability, a model with a slope parameter, and an intercept parameter. We then consider how task and experience influence these two parameters and the resulting distortion of probability. We review how the probability distortions change in systematic ways with task and report three experiments on frequency distortion where the distortions change systematically in the same task. We found that the slope of frequency distortions decreases with the sample size, which is echoed by findings in decision from experience. We review previous models of the representation of uncertainty and find that none can account for the empirical findings

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

An Eye on Numbers: The Processing of Numerical Information in the Context of Visual Perception

The capability of understanding and processing numerical information is a critical skill that allows humans to compare, calculate, judge and remember numbers and numerosities. Without this capability, countless processes in everyday life would be very hard to accomplish. This ranges from simple actions like playing dice to the invention of modern techniques, such as personal computers and satellite-based navigation. Hence, it is important to understand the neural processes underlying the (human) perception of numbers and numerosities. As a contribution to this very complex research field I performed three studies using psychophysical methods and electroencephalography (EEG) with the aim to draw general conclusions on human number perception and the processing of numerical information. In the first two studies, I investigated the effect of spatial numerical association of response codes (SNARC). This effect is commonly seen as evidence for the concept of a mental number line (MNL), which is a metaphor for the fact, that the human brain organizes numbers on a mentally conceived line with small numbers on the left and large numbers on the right. In my first study I showed the effector dependence of the SNARC effect, by measuring the SNARC effect for three different effectors: bimanual finger responses, arm pointing responses and saccadic responses. In my second study, I showed that the concept of the mental number line can be extended to a frontoparallel mental number plane, where small numbers are represented left and down and large numbers are represented right and up. I achieved this result by investigating the SNARC effect for cardinal axes (horizontal and vertical) and for diagonal axes in one and the same subject. This approach allowed me to conclude that the strength of the SNARC effect on the diagonal axes can be expressed as a linear combination of the strength of the SNARC effect along the two cardinal axes. In this second study I measured the SNARC effect also regarding two sensory modalities (visual presented Arabic digits and spoken number words). The comparison of the SNARC effect elicited by these two modalities revealed that the strength of the SNARC effect depended on the modality of number presentation. Together with the results of the effector dependency of the SNARC effect from my first study this led me to propose the existence of a distributed “SNARC network” in the human brain. Within the framework of this proposal the SNARC effect is elicited in a central number stage (CNS) as a consequence of the interaction between numbers and space in the human brain (e.g. as explicated by the MNL). But in addition, the SNARC effect is further modulated by early, modality dependent processing stages and late, effector dependent processing stages. I hypothesize that these stages modulate the SNARC effect, but not the relationship between numbers and space per se. My first two studies, explored the SNARC effect, based on abstract numbers represented in the, so-called, approximate number system (ANS). In addition to the number processing in the ANS, it is known that the human brain is capable of perceiving very small magnitudes (up to four) immediately, a phenomenon called subitizing. Previous studies showed that this perception, although very fast, might be influenced by attentional load (Railo et al., 2008; Olivers & Watson, 2008; Anobile et al., 2012). In my third study, I measured the neural basis of the processing of numerical information non-invasively by means of EEG and used the effect of visual mismatch negativity to demonstrate the pre-attentive processing of quantities in the subitizing range. In this experiment, I rapidly pre-sented stimuli, consisting of one, two or three circular patches. To ensure that numerosity was the relevant factor, patches were varied for low-level visual features (luminance vs. individual patch size). While participants were engaged in a difficult visual detection task, changes of the number of patches (standard vs. deviant) were processed pre-attentively. The results of my study provide evidence for the idea that numerosity in this small (subitizing) range is processed pre-attentively. Taken together, I showed that the mental number line could be extended to a frontoparallel mental number plane and eventually even to a three-dimensional mental number space. I found evidence for the dependence of the SNARC effect on sensory modalities as well as on response effectors, suggesting the existence of a distributed SNARC-brain-network. Finally, I revealed some evidence that number processing of small magnitudes in the subitizing range might be pre-attentive

Visual Enumeration and Estimation: Brain mechanisms, Attentional demands and Number representations.

The work presented in this thesis explored the roles of attention and number awareness in visual enumeration and estimation through a variety of methods. First, a distinction was made between different attentional modes underlying estimation and enumeration in an in-depth single case study of a patient with simultagnosia. Subsequently I demonstrated that, in visual enumeration, subitizing and counting are dissociable processes and they rely on different brain structures. This was done through a neuropsychological single case study as well as through the first large sample neuropsychological group study using a voxel-based correlation method. Following this, behavioural methods were used to examine the relations between subitizing and estimation. I found that, under conditions encouraging estimation, subitizing is an automatic process and may lead to the exact representation of small numbers, which contrasts with approximate representations for larger numerosities. Finally, a functional MRI study was conducted to highlight the brain regions that are activated for subitizable numerosities, but not for larger numerosities under distributed attention conditions. The imaging study provided converging evidence for automatic subitizing leading to an exact number representation. The last chapter discusses the implications of the contrast between subitization and counting for understanding numerical processing

Numerical Cognition in Rhesus Monkeys (Macaca mulatta)

Over the past few decades, researchers have firmly established that a wide range of nonhuman animals exhibit some form of numerical competence. The focus of this research was to define further the extent of numerical ability in rhesus monkeys, and specifically to determine whether the animals possess a symbolic understanding of Arabic numerals. This required examining the stimulus attributes (e.g., number vs. hedonic value) represented by the numerals, as well as the precision (e.g., absolute vs. relative) and generality of those representations. In chapters 2 and 3, monkeys were required to compare and order numerals and were rewarded with either proportional or probabilistic rewards. The results indicated that monkeys were relying on the ordinal or absolute numerical values associated with each numeral and not hedonic value or learned 2-choice discriminations. The studies in chapters 4 and 5 indicated that monkeys can use numerals to symbolize an approximate number of sequential motor responses. The study in Chapter 6 tested the generality of the monkeys’ symbolic number concept using transfer tests. The results indicated that some monkeys are able to abstract number across presentation mode, but this ability is only exhibited under limited conditions. Collectively, these studies provide evidence that rhesus monkeys view Arabic numerals as more than sign-stimuli associated with specific response-reward histories, but that numerals do not have the same precise symbolic meaning as they do for humans