Multimedia information technology and the annotation of video

The state of the art in multimedia information technology has not progressed to the point where a single solution is available to meet all reasonable needs of documentalists and users of video archives. In general, we do not have an optimistic view of the usability of new technology in this domain, but digitization and digital power can be expected to cause a small revolution in the area of video archiving. The volume of data leads to two views of the future: on the pessimistic side, overload of data will cause lack of annotation capacity, and on the optimistic side, there will be enough data from which to learn selected concepts that can be deployed to support automatic annotation. At the threshold of this interesting era, we make an attempt to describe the state of the art in technology. We sample the progress in text, sound, and image processing, as well as in machine learning

Feeling the beat where it counts: fostering multi-limb rhythm skills with the haptic drum kit

This paper introduces and explores a tool known as the Haptic Drum Kit. The Haptic Drum Kit employs four computer-controlled vibrotactile devices, one attached to each limb via the wrists and ankles. In the mode of use discussed in this paper, haptic pulses are used to guide the playing, on a drum kit, of rhythmic patterns that require multi-limb co-ordination. The immediate aim is to foster rhythm skills and multi-limb coordination. A broader aim is to systematically develop skills in recognizing, identifying, memorizing, retaining, analyzing, reproducing and composing monophonic and polyphonic rhythms. We consider the implications of three different theories for this approach: the work of the music educator Dalcroze (1865-1950 [1]; the entrainment theory of human rhythm perception and production [2,3]; and sensory motor contingency theory [4]. In this paper we introduce the Haptic Drum Kit; consider the implications of the above theories for this approach; report on a design study; and identify and discuss a variety of emerging design issues. As part of the design study, audio and haptic guidance was compared for five people learning to play polyphonic drum patterns of varying complexity. The results indicate that beginning drummers are able to learn intricate drum patterns from the haptic stimuli alone, although haptic plus audio is the mode of presentation preferred by subjects

Detecção de eventos complexos em vídeos baseada em ritmos visuais

Orientador: Hélio PedriniDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: O reconhecimento de eventos complexos em vídeos possui várias aplicações práticas relevantes, alavancadas pela grande disponibilidade de câmeras digitais instaladas em aeroportos, estações de ônibus e trens, centros de compras, estádios, hospitais, escolas, prédios, estradas, entre vários outros locais. Avanços na tecnologia digital têm aumentado as capacidades dos sistemas em reconhecer eventos em vídeos por meio do desenvolvimento de dispositivos com alta resolução, dimensões físicas pequenas e altas taxas de amostragem. Muitos trabalhos disponíveis na literatura têm explorado o tema a partir de diferentes pontos de vista. Este trabalho apresenta e avalia uma metodologia para extrair características dos ritmos visuais no contexto de detecção de eventos em vídeos. Um ritmo visual pode ser visto com a projeção de um vídeo em uma imagem, tal que a tarefa de análise de vídeos é reduzida a um problema de análise de imagens, beneficiando-se de seu baixo custo de processamento em termos de tempo e complexidade. Para demonstrar o potencial do ritmo visual na análise de vídeos complexos, três problemas da área de visão computacional são selecionados: detecção de eventos anômalos, classificação de ações humanas e reconhecimento de gestos. No primeiro problema, um modelo e? aprendido com situações de normalidade a partir dos rastros deixados pelas pessoas ao andar, enquanto padro?es representativos das ações são extraídos nos outros dois problemas. Nossa hipo?tese e? de que vídeos similares produzem padro?es semelhantes, tal que o problema de classificação de ações pode ser reduzido a uma tarefa de classificação de imagens. Experimentos realizados em bases públicas de dados demonstram que o método proposto produz resultados promissores com baixo custo de processamento, tornando-o possível aplicar em tempo real. Embora os padro?es dos ritmos visuais sejam extrai?dos como histograma de gradientes, algumas tentativas para adicionar características do fluxo o?tico são discutidas, além de estratégias para obter ritmos visuais alternativosAbstract: The recognition of complex events in videos has currently several important applications, particularly due to the wide availability of digital cameras in environments such as airports, train and bus stations, shopping centers, stadiums, hospitals, schools, buildings, roads, among others. Moreover, advances in digital technology have enhanced the capabilities for detection of video events through the development of devices with high resolution, small physical size, and high sampling rates. Many works available in the literature have explored the subject from different perspectives. This work presents and evaluates a methodology for extracting a feature descriptor from visual rhythms of video sequences in order to address the video event detection problem. A visual rhythm can be seen as the projection of a video onto an image, such that the video analysis task can be reduced into an image analysis problem, benefiting from its low processing cost in terms of time and complexity. To demonstrate the potential of the visual rhythm in the analysis of complex videos, three computer vision problems are selected in this work: abnormal event detection, human action classification, and gesture recognition. The former problem learns a normalcy model from the traces that people leave when they walk, whereas the other two problems extract representative patterns from actions. Our hypothesis is that similar videos produce similar patterns, therefore, the action classification problem is reduced into an image classification task. Experiments conducted on well-known public datasets demonstrate that the method produces promising results at high processing rates, making it possible to work in real time. Even though the visual rhythm features are mainly extracted as histogram of gradients, some attempts for adding optical flow features are discussed, as well as strategies for obtaining alternative visual rhythmsMestradoCiência da ComputaçãoMestre em Ciência da Computação1570507, 1406910, 1374943CAPE

Action-based effects on music perception

The classical, disembodied approach to music cognition conceptualizes action and perception as separate, peripheral processes. In contrast, embodied accounts of music cognition emphasize the central role of the close coupling of action and perception. It is a commonly established fact that perception spurs action tendencies. We present a theoretical framework that captures the ways in which the human motor system and its actions can reciprocally influence the perception of music. The cornerstone of this framework is the common coding theory, postulating a representational overlap in the brain between the planning, the execution, and the perception of movement. The integration of action and perception in so-called internal models is explained as a result of associative learning processes. Characteristic of internal models is that they allow intended or perceived sensory states to be transferred into corresponding motor commands (inverse modeling), and vice versa, to predict the sensory outcomes of planned actions (forward modeling). Embodied accounts typically refer to inverse modeling to explain action effects on music perception (Leman, 2007). We extend this account by pinpointing forward modeling as an alternative mechanism by which action can modulate perception. We provide an extensive overview of recent empirical evidence in support of this idea. Additionally, we demonstrate that motor dysfunctions can cause perceptual disabilities, supporting the main idea of the paper that the human motor system plays a functional role in auditory perception. The finding that music perception is shaped by the human motor system and its actions suggests that the musical mind is highly embodied. However, we advocate for a more radical approach to embodied (music) cognition in the sense that it needs to be considered as a dynamical process, in which aspects of action, perception, introspection, and social interaction are of crucial importance

Understanding Dance Understanding

&nbsp

The Temporality and Rythmicity of Lived Street Space

Tämä väitöskirja, lyhyesti ilmaistuna, tarkastelee arjen katutilan ja kaupunkiliikkumisen ajallisuuksia ja rytmisyyksiä. Kadut ja muut liikkumisen tilat kaupungissa ovat urbaanin arkielämän tärkeimpiä tapahtumapaikkoja – ne ovat keskeisessä roolissa siinä, miten (rutiininomaisesti) käytämme ja olemme vuorovaikutuksessa rakennetun ympäristön kanssa, miten juurrumme asuinympäristöihimme, ja miten kohtaamme muita ihmisiä kaupunkitilassa – ja näin ollen ovat olennaisessa roolissa elävien, kestävien ja tasa-arvoisten kaupunkien muodostumisessa. Tarkastellen katua mobiilina kokoutumana (mobile assemblage), tutkimus selvittää ja käsitteellistää eräitä keskeisimpiä liikkumisen ja katutilan rytmejä, ja pyrkii tuottamaan yksityiskohtaisen kuvan kaupunkiympäristön toistuvista (mikro-)ajallisuuksista liikkumisen näkökulmasta, mitkä osaltaan määrittävät kaupunkiympäristöä jokapäiväisenä ’elettynä’ tilana. Työn teoreettinen kehys ammentaa useista eri kaupunkien ajallisuutta käsitteellistävistä perinteistä, erityisesti Lefebvreläisestä rytmianalyysistä, ja määrittelee tarkasteltavat liikkumisen rytmit tilan, ajan ja kehollisen liikkumisen erottamattomiksi keskinäissuhteiksi. Tutkimuksen empiirisessä keskiössä on ruohonjuuritason liikkuminen. Liikkuminen, tai mobiliteetti, ymmärretään tässä laajasti (seuraten uutta mobiliteetin paradigmaa) toimintoina, jotka muodostavat merkityksiä, kokemuksia, kuulumisen tunteita, sosiaalis-materiaalisia vuorovaikutuksia, mielikuvia ja (liikkumisen) kulttuureita samalla, kun ne siirtävät ihmisiä paikasta A paikkaan B. Tutkimuksessa on tarkasteltu arjessa toistuvia kävely- ja ajoreittejä sekä liikkumisen tapahtumaa tavanomaisissa katuympäristöissä kahdessa suuressa suomalaisessa kaupungissa eri liikkumisen tutkimuksen menetelmiä (mobile methods) (mukaan menemiseen perustuvia syvähaastatteluita, valokuvia, reittivideoita ja reittikarttoja; videoituja paikkahavainnointeja) sekä jälkifenomenologista tutkimusotetta hyödyntäen. Tutkimusaineiston analyysi – mikä on tarkemmin esitelty sisällytetyissä tutkimusartikkeleissa (#01–04) – tuo esiin, yhtäältä, miten ihmiset (inter)subjektiivisesti hahmottavat, kokevat ja toiminnallaan muokkaavat kadun (ja laajemmin kaupungin) rytmisyyksiä omien liikkumisrutiiniensa konteksteissa, ja toisaalta, miten tilallisen toiminnan ja liikkeen kautta tilassa liikkujat tuottavat ajallista, tai hetkellistä, kadun arkkitehtuuria sopeutumalla tai haastamalla muualta asetettuja rytmisyyksiä. Analyysi tuo lisäksi esiin erilaisia rytmien välillisyyksiä (#01) ja rytmityksen prosesseja (#02), kaupunkiympäristön morfologian vaikutuksia näiden rytmien muodostumiseen (#03), sekä katutilan haltuunoton ajallisesti määrityviä rytmisiä muotoja (#04). Työ esittää, että nouseva rytmianalyyttinen tutkimusote on soveltuva ja hyödyllinen tapa lähestyä ja kartoittaa dynaamisia ja alati muuttuvia kaupunki- ilmiöitä. Arjen katutilan suhteen rytmianalyysi paljastaa erilaisia mikrotason ajallisuuksia (yhdessä makrotason kanssa), joiden valossa katuympäristö näyttäytyy monien heterogeenisten ja samanaikaisten ajallisuuksien tilana. Rytmianalyysi auttaa myös ymmärtämään kaupunkiliikkumisen moniulotteisuutta sekä arjen reittien merkityksiä funktionaalisten tekijöiden ohella, tuoden esiin ajallisten keho-ympäristö suhteiden moninaisuuden kirjoa. Yhdessä ne piirtävät vivahteikkaan kuvan kaupunkirakenteista kartoittaen sekä formaaleja (suunnitellut, ’ylhäältä’ asetetut) että informaaleja (sattumanvaraiset tai rutiininomaiset, ’alhaalta’ asetetut) liikkumisen rakenteita. Ne korostavat ihmistoiminnan jatkuvaa, niin rytmistä kuin kitkaista sykettä, kaupunkikudoksen intensiteettiä. Toisin sanoen, ne tuovat esiin kaupungin ja katuympäristöjen tahdin moninaisuuden sekä ennalta suunniteltuna että liikkeellä olevien ihmisten tuottamana.This dissertation, in short, examines the temporalities and rhythmicities of day-to- day urban mobility practices on the city street. Streets, and other mobility-centred spaces of the city, are the main stages of public urban life – they are essential to how we (routinely) use and interact with the built environment, connect to our neighbourhoods, and encounter other city dwellers – and thus play a key part in the making of liveable, sustainable and just cities. Examining the street as a mobile assemblage, the study probes and conceptualizes some of the key rhythms that emerge from such daily mobility patterns of the street, aiming to draw a detailed picture of the recurring urban (micro)temporalities from a mobilities perspective that partially constitute the ‘lived’ aspects of the day-to-day built environments. The theoretical framework on temporalities draws from various conceptual lineages, notably a Lefebvrian rhythmanalytical framework, and defines the studied mobility rhythms of the street as the inseparable relations between spaces, times and mobile embodied practices. The practical research focus is set on the grassroot-level embodied mobilities. Here mobility practices are understood in a broad sense (following a new mobilities paradigm) as activities that, whilst physically moving people from place A to place B, also produce meanings, experiences, sense of belonging, socio-material interactions, imageries, and (mobile) cultures in the process. Utilizing various mobile research methods (in-depth go-along interviews, participant-produced photographs, route videos and route maps; extensive videoed site observations), and by taking a postphenomenological research perspective, the dissertation examines recurring walking and driving routes, and the mobile event of day-to-day street space in two major Finnish cities. The analysis of the data – presented in four research articles (#01–04)– reveals, on one hand, how people (inter)subjectively make sense of and modify the rhythmicities of the street (and the city in general) inside their own mobile daily routines, and, on the other, how people – through their (mobile) uses of the space – produce temporal, or momentarily perceivable, architecture of the street by adapting to, or contesting, pre-set rhythmicities. The analysis further reveals different mediacies (#01) and processes of pacing (#02) of such rhythmicities, the role of urban morphologies in the formation of these rhythmicities (#03), and the time-sensitive rhythmic modes of appropriating the street through mobile uses (#04). The work proposes that the emerging rhythmanalytical research framework is an applicable and advantageous mode for approaching and mapping the urban phenomena that are inherently caught in a continuous flux and flow. In the case of the day-to-day street space, rhythmanalysis can be used to reveal micro-level (next to macro-level) temporalities that depict the street as a site of multiple heterogeneous and simultaneous temporalities and timings. Likewise, rhythmanalysis, helps us to understand the complexity of urban mobilities and day-to-day routes beyond their strictly functional means, revealing the multiplicities of temporal relations in such recurring body-environment relations. Together, they are able to draw a nuanced picture of some of the key urban structures, mapping both formal (planned and designed, set from the ‘above’) as well as informal (accidental and routine-like, set from the ‘below’) mobility structures of the city. They highlight the continuous, rhythmic and arrhythmic, pulses of human activity in the city, the intensities of the urban fabric. In other words, they reveal multiplicities of the beat of the city and its streets, both the planned and designed as well as the ones produced by their inhabitants on the move

The Psychophysics of Brain Rhythms

It is becoming increasingly apparent that brain oscillations in various frequency bands play important roles in perceptual and attentional processes. Understandably, most of the associated experimental evidence comes from human or animal electrophysiological studies, allowing direct access to the oscillatory activities. However, such periodicities in perception and attention should, in theory, also be observable using the proper psychophysical tools. Here, we review a number of psychophysical techniques that have been used by us and other authors, in successful and sometimes unsuccessful attempts, to reveal the rhythmic nature of perceptual and attentional processes. We argue that the two existing and largely distinct debates about discrete vs. continuous perception and parallel vs. sequential attention should in fact be regarded as two facets of the same question: how do brain rhythms shape the psychological operations of perception and attention

Reconhecimento de padrões em expressões faciais : algoritmos e aplicações

Orientador: Hélio PedriniTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: O reconhecimento de emoções tem-se tornado um tópico relevante de pesquisa pela comunidade científica, uma vez que desempenha um papel essencial na melhoria contínua dos sistemas de interação humano-computador. Ele pode ser aplicado em diversas áreas, tais como medicina, entretenimento, vigilância, biometria, educação, redes sociais e computação afetiva. Há alguns desafios em aberto relacionados ao desenvolvimento de sistemas emocionais baseados em expressões faciais, como dados que refletem emoções mais espontâneas e cenários reais. Nesta tese de doutorado, apresentamos diferentes metodologias para o desenvolvimento de sistemas de reconhecimento de emoções baseado em expressões faciais, bem como sua aplicabilidade na resolução de outros problemas semelhantes. A primeira metodologia é apresentada para o reconhecimento de emoções em expressões faciais ocluídas baseada no Histograma da Transformada Census (CENTRIST). Expressões faciais ocluídas são reconstruídas usando a Análise Robusta de Componentes Principais (RPCA). A extração de características das expressões faciais é realizada pelo CENTRIST, bem como pelos Padrões Binários Locais (LBP), pela Codificação Local do Gradiente (LGC) e por uma extensão do LGC. O espaço de características gerado é reduzido aplicando-se a Análise de Componentes Principais (PCA) e a Análise Discriminante Linear (LDA). Os algoritmos K-Vizinhos mais Próximos (KNN) e Máquinas de Vetores de Suporte (SVM) são usados para classificação. O método alcançou taxas de acerto competitivas para expressões faciais ocluídas e não ocluídas. A segunda é proposta para o reconhecimento dinâmico de expressões faciais baseado em Ritmos Visuais (VR) e Imagens da História do Movimento (MHI), de modo que uma fusão de ambos descritores codifique informações de aparência, forma e movimento dos vídeos. Para extração das características, o Descritor Local de Weber (WLD), o CENTRIST, o Histograma de Gradientes Orientados (HOG) e a Matriz de Coocorrência em Nível de Cinza (GLCM) são empregados. A abordagem apresenta uma nova proposta para o reconhecimento dinâmico de expressões faciais e uma análise da relevância das partes faciais. A terceira é um método eficaz apresentado para o reconhecimento de emoções audiovisuais com base na fala e nas expressões faciais. A metodologia envolve uma rede neural híbrida para extrair características visuais e de áudio dos vídeos. Para extração de áudio, uma Rede Neural Convolucional (CNN) baseada no log-espectrograma de Mel é usada, enquanto uma CNN construída sobre a Transformada de Census é empregada para a extração das características visuais. Os atributos audiovisuais são reduzidos por PCA e LDA, então classificados por KNN, SVM, Regressão Logística (LR) e Gaussian Naïve Bayes (GNB). A abordagem obteve taxas de reconhecimento competitivas, especialmente em dados espontâneos. A penúltima investiga o problema de detectar a síndrome de Down a partir de fotografias. Um descritor geométrico é proposto para extrair características faciais. Experimentos realizados em uma base de dados pública mostram a eficácia da metodologia desenvolvida. A última metodologia trata do reconhecimento de síndromes genéticas em fotografias. O método visa extrair atributos faciais usando características de uma rede neural profunda e medidas antropométricas. Experimentos são realizados em uma base de dados pública, alcançando taxas de reconhecimento competitivasAbstract: Emotion recognition has become a relevant research topic by the scientific community, since it plays an essential role in the continuous improvement of human-computer interaction systems. It can be applied in various areas, for instance, medicine, entertainment, surveillance, biometrics, education, social networks, and affective computing. There are some open challenges related to the development of emotion systems based on facial expressions, such as data that reflect more spontaneous emotions and real scenarios. In this doctoral dissertation, we propose different methodologies to the development of emotion recognition systems based on facial expressions, as well as their applicability in the development of other similar problems. The first is an emotion recognition methodology for occluded facial expressions based on the Census Transform Histogram (CENTRIST). Occluded facial expressions are reconstructed using an algorithm based on Robust Principal Component Analysis (RPCA). Extraction of facial expression features is then performed by CENTRIST, as well as Local Binary Patterns (LBP), Local Gradient Coding (LGC), and an LGC extension. The generated feature space is reduced by applying Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). K-Nearest Neighbor (KNN) and Support Vector Machine (SVM) algorithms are used for classification. This method reached competitive accuracy rates for occluded and non-occluded facial expressions. The second proposes a dynamic facial expression recognition based on Visual Rhythms (VR) and Motion History Images (MHI), such that a fusion of both encodes appearance, shape, and motion information of the video sequences. For feature extraction, Weber Local Descriptor (WLD), CENTRIST, Histogram of Oriented Gradients (HOG), and Gray-Level Co-occurrence Matrix (GLCM) are employed. This approach shows a new direction for performing dynamic facial expression recognition, and an analysis of the relevance of facial parts. The third is an effective method for audio-visual emotion recognition based on speech and facial expressions. The methodology involves a hybrid neural network to extract audio and visual features from videos. For audio extraction, a Convolutional Neural Network (CNN) based on log Mel-spectrogram is used, whereas a CNN built on Census Transform is employed for visual extraction. The audio and visual features are reduced by PCA and LDA, and classified through KNN, SVM, Logistic Regression (LR), and Gaussian Naïve Bayes (GNB). This approach achieves competitive recognition rates, especially in a spontaneous data set. The second last investigates the problem of detecting Down syndrome from photographs. A geometric descriptor is proposed to extract facial features. Experiments performed on a public data set show the effectiveness of the developed methodology. The last methodology is about recognizing genetic disorders in photos. This method focuses on extracting facial features using deep features and anthropometric measurements. Experiments are conducted on a public data set, achieving competitive recognition ratesDoutoradoCiência da ComputaçãoDoutora em Ciência da Computação140532/2019-6CNPQCAPE