An original framework for understanding human actions and body language by using deep neural networks

The evolution of both fields of Computer Vision (CV) and Artificial Neural Networks (ANNs) has allowed the development of efficient automatic systems for the analysis of people's behaviour. By studying hand movements it is possible to recognize gestures, often used by people to communicate information in a non-verbal way. These gestures can also be used to control or interact with devices without physically touching them. In particular, sign language and semaphoric hand gestures are the two foremost areas of interest due to their importance in Human-Human Communication (HHC) and Human-Computer Interaction (HCI), respectively. While the processing of body movements play a key role in the action recognition and affective computing fields. The former is essential to understand how people act in an environment, while the latter tries to interpret people's emotions based on their poses and movements; both are essential tasks in many computer vision applications, including event recognition, and video surveillance. In this Ph.D. thesis, an original framework for understanding Actions and body language is presented. The framework is composed of three main modules: in the first one, a Long Short Term Memory Recurrent Neural Networks (LSTM-RNNs) based method for the Recognition of Sign Language and Semaphoric Hand Gestures is proposed; the second module presents a solution based on 2D skeleton and two-branch stacked LSTM-RNNs for action recognition in video sequences; finally, in the last module, a solution for basic non-acted emotion recognition by using 3D skeleton and Deep Neural Networks (DNNs) is provided. The performances of RNN-LSTMs are explored in depth, due to their ability to model the long term contextual information of temporal sequences, making them suitable for analysing body movements. All the modules were tested by using challenging datasets, well known in the state of the art, showing remarkable results compared to the current literature methods

Directional adposition use in English, Swedish and Finnish

Directional adpositions such as to the left of describe where a Figure is in relation to a Ground. English and Swedish directional adpositions refer to the location of a Figure in relation to a Ground, whether both are static or in motion. In contrast, the Finnish directional adpositions edellä (in front of) and jäljessä (behind) solely describe the location of a moving Figure in relation to a moving Ground (Nikanne, 2003). When using directional adpositions, a frame of reference must be assumed for interpreting the meaning of directional adpositions. For example, the meaning of to the left of in English can be based on a relative (speaker or listener based) reference frame or an intrinsic (object based) reference frame (Levinson, 1996). When a Figure and a Ground are both in motion, it is possible for a Figure to be described as being behind or in front of the Ground, even if neither have intrinsic features. As shown by Walker (in preparation), there are good reasons to assume that in the latter case a motion based reference frame is involved. This means that if Finnish speakers would use edellä (in front of) and jäljessä (behind) more frequently in situations where both the Figure and Ground are in motion, a difference in reference frame use between Finnish on one hand and English and Swedish on the other could be expected. We asked native English, Swedish and Finnish speakers’ to select adpositions from a language specific list to describe the location of a Figure relative to a Ground when both were shown to be moving on a computer screen. We were interested in any differences between Finnish, English and Swedish speakers. All languages showed a predominant use of directional spatial adpositions referring to the lexical concepts TO THE LEFT OF, TO THE RIGHT OF, ABOVE and BELOW. There were no differences between the languages in directional adpositions use or reference frame use, including reference frame use based on motion. We conclude that despite differences in the grammars of the languages involved, and potential differences in reference frame system use, the three languages investigated encode Figure location in relation to Ground location in a similar way when both are in motion. Levinson, S. C. (1996). Frames of reference and Molyneux’s question: Crosslingiuistic evidence. In P. Bloom, M.A. Peterson, L. Nadel & M.F. Garrett (Eds.) Language and Space (pp.109-170). Massachusetts: MIT Press. Nikanne, U. (2003). How Finnish postpositions see the axis system. In E. van der Zee & J. Slack (Eds.), Representing direction in language and space. Oxford, UK: Oxford University Press. Walker, C. (in preparation). Motion encoding in language, the use of spatial locatives in a motion context. Unpublished doctoral dissertation, University of Lincoln, Lincoln. United Kingdo

STUDY OF HAND GESTURE RECOGNITION AND CLASSIFICATION

To recognize different hand gestures and achieve efficient classification to understand static and dynamic hand movements used for communications.Static and dynamic hand movements are first captured using gesture recognition devices including Kinect device, hand movement sensors, connecting electrodes, and accelerometers. These gestures are processed using hand gesture recognition algorithms such as multivariate fuzzy decision tree, hidden Markov models (HMM), dynamic time warping framework, latent regression forest, support vector machine, and surface electromyogram. Hand movements made by both single and double hands are captured by gesture capture devices with proper illumination conditions. These captured gestures are processed for occlusions and fingers close interactions for identification of right gesture and to classify the gesture and ignore the intermittent gestures. Real-time hand gestures recognition needs robust algorithms like HMM to detect only the intended gesture. Classified gestures are then compared for the effectiveness with training and tested standard datasets like sign language alphabets and KTH datasets. Hand gesture recognition plays a very important role in some of the applications such as sign language recognition, robotics, television control, rehabilitation, and music orchestration

Pictures in Your Mind: Using Interactive Gesture-Controlled Reliefs to Explore Art

Tactile reliefs offer many benefits over the more classic raised line drawings or tactile diagrams, as depth, 3D shape, and surface textures are directly perceivable. Although often created for blind and visually impaired (BVI) people, a wider range of people may benefit from such multimodal material. However, some reliefs are still difficult to understand without proper guidance or accompanying verbal descriptions, hindering autonomous exploration. In this work, we present a gesture-controlled interactive audio guide (IAG) based on recent low-cost depth cameras that can be operated directly with the hands on relief surfaces during tactile exploration. The interactively explorable, location-dependent verbal and captioned descriptions promise rapid tactile accessibility to 2.5D spatial information in a home or education setting, to online resources, or as a kiosk installation at public places. We present a working prototype, discuss design decisions, and present the results of two evaluation studies: the first with 13 BVI test users and the second follow-up study with 14 test users across a wide range of people with differences and difficulties associated with perception, memory, cognition, and communication. The participant-led research method of this latter study prompted new, significant and innovative developments

On topics today

This article surveys the state of so-called topic theory today. It charts its development through two generations of topic theorists. The first is constructed around three influential texts: Leonard Ratners seminal book that established the discipline in its own right, Classic music: expression, form and style (1980); Wye Allanbrooks. Rhythmic gesture in Mozart: Le nozze di Figaro and Don Giovanni (1983); and Kofi Agawus. Playing with signs: a semiotic interpretation of classical music (1991). The second comprises significant advances in topic theory essayed through two further pairs of texts: Robert Hattens Musical meaning in Beethoven: markedness, correlation, and interpretation (1994) and Interpreting musical gestures, topics, and tropes: Mozart, Beethoven, Schubert (2004); and Raymond Monelles Linguistics and semiotics in music (1992) and The sense of music: semiotic essays (2000). Topic Theory's role as the soft hermeneutic sub-field of music semiotics (relative to the harder, formalist practices of Nattiezs neutral level analysis) is portrayed here as navigating a number of treacherous polemical paths. These wend their way between referential style (expression) and structural syntax (form); historical reconstruction and hermeneutic construction; and heightened sensitivity to social meanings and imposed acts of creative interpretation. This existence of topic theory in a continuous dialogue between structural formalism and the semantics of expressive discourse is held responsible for its marginal position both to the dominant strains of contemporary postmodern musicology and to the dying embers of formalist analysis. The failure of topic theory to strike a fashionable text-context balance thus highlights why musicology continues to view semiotics with scepticism. Ratner presents his thesaurus of style labelssomewhat dubiouslyas the historically authentic ready-to-hand materials (types and styles) of eighteenth-century expressive musical rhetoric. But it is Agawus combination of this universe of topics with a Schenker-influenced beginning-middle-end paradigm that establishes the hallmark of first generation topic theory on which the first half of this paper focuses. Agawus delicate equation between extroversive and introversive semiosis is essayed as a pivotal turning point in topic theorys ability to transcend the mere passive ascription of rhetorical labels. Out of this equation, expressive meanings can ariseas much from the non-congruence, as the congruence, of signs and structure. Hatten's critique of Agawu for neglecting the full interpretative consequences of his signifieds is the springboard for his more hermeneutically replete brand of topic theory and the emergence of the second generation topic theorists. Hattens use of troping (a kind of musical metaphor), is one of many interpretative tools that are responsible for broadening the arena of topic theorysome of his others being: expressive genres, emergent meanings and markedness theory. These are deployed across a variety of musical parameters as Hattens attention increasingly turns to the prototypicality of topics in their euphoric and dysphoric states. Hattens interpretative work is shown to transcend historical reconstruction to comprise creative interpretation built on a much broader definition of expressive gestures, of which topics are only a constituent part. The article concludes with Monelles expos of the dubious historical underpinnings of Ratners topic theory foundations. This does not render this vibrant branch of semiotics redundant but, on the contrary, charts its future direction as one calling out for far deeper historical investigation and cultural criticism. Monelles enlightening forays into the more replete expressive meanings of such topics as the horse and pianto make this point abundantly clear. The future of topics today, if not musicology in general, is one of cultural criticism