Tracking interacting targets in multi-modal sensors

PhDObject tracking is one of the fundamental tasks in various applications such as surveillance, sports, video conferencing and activity recognition. Factors such as occlusions, illumination changes and limited field of observance of the sensor make tracking a challenging task. To overcome these challenges the focus of this thesis is on using multiple modalities such as audio and video for multi-target, multi-modal tracking. Particularly, this thesis presents contributions to four related research topics, namely, pre-processing of input signals to reduce noise, multi-modal tracking, simultaneous detection and tracking, and interaction recognition. To improve the performance of detection algorithms, especially in the presence of noise, this thesis investigate filtering of the input data through spatio-temporal feature analysis as well as through frequency band analysis. The pre-processed data from multiple modalities is then fused within Particle filtering (PF). To further minimise the discrepancy between the real and the estimated positions, we propose a strategy that associates the hypotheses and the measurements with a real target, using a Weighted Probabilistic Data Association (WPDA). Since the filtering involved in the detection process reduces the available information and is inapplicable on low signal-to-noise ratio data, we investigate simultaneous detection and tracking approaches and propose a multi-target track-beforedetect Particle filtering (MT-TBD-PF). The proposed MT-TBD-PF algorithm bypasses the detection step and performs tracking in the raw signal. Finally, we apply the proposed multi-modal tracking to recognise interactions between targets in regions within, as well as outside the cameras’ fields of view. The efficiency of the proposed approaches are demonstrated on large uni-modal, multi-modal and multi-sensor scenarios from real world detections, tracking and event recognition datasets and through participation in evaluation campaigns

Multi-modal particle filtering tracking using appearance, motion and audio likelihoods

ABSTRACT We propose a multi-modal object tracking algorithm that combines appearance, motion and audio information in a particle filter. The proposed tracker fuses at the likelihood level the audio-visual observations captured with a video camera coupled with two microphones. Two video likelihoods are computed that are based on a 3D color histogram appearance model and on a color change detection, whereas an audio likelihood provides information about the direction of arrival of a target. The direction of arrival is computed based on a multi-band generalized cross-correlation function enhanced with a noise suppression and reverberation filtering that uses the precedence effect. We evaluate the tracker on single and multi-modality tracking and quantify the performance improvement introduced by integrating audio and visual information in the tracking process

Multi-speaker tracking from an audio-visual sensing device

Compact multi-sensor platforms are portable and thus desirable for robotics and personal-assistance tasks. However, compared to physically distributed sensors, the size of these platforms makes person tracking more difficult. To address this challenge, we propose a novel 3D audio-visual people tracker that exploits visual observations (object detections) to guide the acoustic processing by constraining the acoustic likelihood on the horizontal plane defined by the predicted height of a speaker. This solution allows the tracker to estimate, with a small microphone array, the distance of a sound. Moreover, we apply a color-based visual likelihood on the image plane to compensate for misdetections. Finally, we use a 3D particle filter and greedy data association to combine visual observations, color-based and acoustic likelihoods to track the position of multiple simultaneous speakers. We compare the proposed multimodal 3D tracker against two state-of-the-art methods on the AV16.3 dataset and on a newly collected dataset with co-located sensors, which we make available to the research community. Experimental results show that our multimodal approach outperforms the other methods both in 3D and on the image plane

INDIVIDUALIZED COGNITIVE DECLINE AND THE IMPACT OF GUT MICROBIOME COMPOSITION.

The U.S. population is aging at its greatest rate in history. An older average population will increase the number of age-related cognitive issues. Elucidation of factors that contribute to decline with age and methods to prevent or decrease the incidence of cognitive dysfunction in the aging population is vital to offset the impact of the age shift. Validation of tests to identify and predict decline is the first step, but must be paired with an increased understanding of the inter- and intra-individual differences that influence cognitive decline. One difference, gut microbiome diversity, changes within the person across their lifespan and varies among individuals. An individual’s gut microflora can significantly influence gut-brain communication, brain function, and behavior. The study was focused on identification and prediction of cognitive decline using CANTAB and visual ERP as well as exploring the relation between gut-microbiome diversity and cognitive performance. Participants underwent tests to evaluate cognitive decline over time: the MoCA, a CANTAB battery for behavioral cognitive assessment, and an electrophysiological evaluation via a passive oddball paradigm and an active detection task. The role of microbiome diversity in cognitive decline was investigated, ERP measures were validated against CANTAB measures, the predictive relation between MoCA and future cognitive outcomes were characterized, and the utility of ERP PCA factors and CANTAB outcomes to predict future ERP and CANTAB performance were shown. Three CANTAB measures (RTI, SWM, and RVP) were independently confirmed to significantly relate to selected ERP measures in both the active detection and the passive oddball tasks. Baseline MoCA score and change in MoCA score significantly predicted outcomes in the CANTAB battery and ERP tasks at follow-up. The study also included the design and implementation of novel methodology with two-step temporospatial PCA to successfully predict future performance on ERP with baseline performance on the same task, which, to this author’s knowledge, is the first known use of this method for this purpose. Finally, significant relations between gut-microbiome diversity and healthy cognitive function were revealed, where lower microbial diversity significantly relates to poorer cognitive performance on both behavioral (CANTAB) and electrophysiological (ERP) measures.Doctor of Philosoph

Research & scholarly achievements, July 1, 1985 - June 30, 1986

Annual summary of scholarly activities at UCF for the period from July 1, 1985 to June 30, 1986. This report highlights the contributions to scholarship by the Faculty during this period, including books and monographs, articles, creative works, presentations, grants and contracts

Artech 2008: proceedings of the 4th International Conference on Digital Arts

ARTECH 2008 is the fourth international conference held in Portugal and Galicia on the topic of Digital Arts. It aims to promote contacts between Iberian and International contributors concerned with the conception, production and dissemination of Digital and Electronic Art. ARTECH brings the scientific, technological and artistic community together, promoting the interest in the digital culture and its intersection with art and technology as an important research field, a common space for discussion, an exchange of experiences, a forum for emerging digital artists and a way of understanding and appreciating new forms of cultural expression. Hosted by the Portuguese Catholic University’s School of Arts (UCP-EA) at the City of Porto, ARTCH 2008 falls in alignment with the main commitment of the Research Center for Science and Technology of the Arts (CITAR) to promote knowledge in the field of the Arts trough research and development within UCP-AE and together with the local and international community. The main areas proposed for the conference were related with sound, image, video, music, multimedia and other new media related topics, in the context of emerging practice of artistic creation. Although non exclusive, the main topics of the conference are usually: Art and Science; Audio-Visual and Multimedia Design; Creativity Theory; Electronic Music; Generative and Algorithmic Art; Interactive Systems for Artistic Applications; Media Art history; Mobile Multimedia; Net Art and Digital Culture; New Experiences with New Media and New Applications; Tangible and Gesture Interfaces; Technology in Art Education; Virtual Reality and Augmented Reality. The contribution from the international community was extremely gratifying, resulting in the submission of 79 original works (Long Papers, Short Papers and installation proposals) from 22 Countries. Our Scientific Committee reviewed these submissions thoroughly resulting in a 73% acceptance ratio of a diverse and promising body of work presented in this book of proceedings. This compilation of articles provides an overview of the state of the art as well as a glimpse of new tendencies in the field of Digital Arts, with special emphasis in the topics: Sound and Music Computing; Technology Mediated Dance; Collaborative Art Performance; Digital Narratives; Media Art and Creativity Theory; Interactive Art; Audiovisual and Multimedia Design.info:eu-repo/semantics/publishedVersio

Involvement of Cognitive Resources in Sensory and Sensorimotor Functioning with Age

Epidemiological research indicates a link between hearing loss and poor mobility (Lin & Ferrucci, 2012; Viljanen et al., 2009). One explanation for this association is cognitive compensation, wherein older adults compensate for hearing loss and declines in mobility by recruiting higher-level cognitive resources. A growing body of research using various approaches demonstrates that cognitive resources are involved in both hearing and mobility with age. Our work complements these studies by using experimental, intervention and modeling techniques to investigate how these domains relate in an aging population. Using an experimental approach, older adults and older adults with hearing loss completed a cognitive-motor dual-task protocol, in which they performed an auditory working memory task and a balance recovery task singly or concurrently. Older adults with mild hearing loss showed disproportionately greater dual-task costs on the auditory working memory task, particularly when auditory challenge was added. Given the involvement of cognitive resources in challenging dual-task conditions and particularly among a hearing loss population, the second study took a cognitive enhancement approach. Specifically, older adults with and without hearing loss were given either simultaneous or sequential cognitive-physical training formats. While sequential training appeared to benefit performance on the auditory working memory task, older adults with hearing loss appeared to improve on this same task regardless of format. To complement these group-wise effects we took a structural equation modeling approach using data pooled from the two previous studies to examine individual differences in hearing and cognition which might influence mobility. An additional consideration was the impact of self-efficacy. It was found that the association between greater hearing loss and reduced mobility was mediated by cognitive status, and that self-efficacy in the hearing domain may be an important contributor to balance confidence. Taken together, the current work points to the importance of cognitive resources in both sensory and motor aging, particularly among older adults with age-related hearing loss. As such, the work suggests that cognitive remediation may be a useful complement to traditional hearing and mobility rehabilitation. Moreover, self-efficacy appears to be an important contributor in understanding the relation between hearing loss and mobility