375 research outputs found

    Audiovisual Gestalts

    Get PDF
    This paper presents an algorithm to correlate audio and visual data generated by the same physical phenomenon. According to psychophysical experiments, temporal synchrony strongly contributes to integrate cross-modal information in humans. Thus, we define meaningful audiovisual structures as temporally proximal audio-video events. Audio and video signals are represented as sparse decompositions over redundant dictionaries of functions. In this way, it is possible to define perceptually meaningful audiovisual events. The detection of these cross-modal structures is done using a simple rule called Helmholtz principle. Experimental results show that extracting significant synchronous audiovisual events, we can detect the existing cross-modalcorrelation between those signals even in presence of distracting motion and acoustic noise. These results confirm that temporal proximity between audiovisual events is a key ingredient for the integration of information across modalities and that it can be effectively exploited for the design of multi-modal analysis algorithms

    Multimodal language processing in human communication

    No full text
    Multiple layers of visual (and vocal) signals, plus their different onsets and offsets, represent a significant semantic and temporal binding problem during face-to-face conversation. Despite this complex unification process, multimodal messages appear to be processed faster than unimodal messages. Multimodal gestalt recognition and multilevel prediction are proposed to play a crucial role in facilitating multimodal language processing. The basis of the processing mechanisms involved in multimodal language comprehension is hypothesized to be domain general, coopted for communication, and refined with domain-specific characteristics. A new, situated framework for understanding human language processing is called for that takes into consideration the multilayered, multimodal nature of language and its production and comprehension in conversational interaction requiring fast processing

    NieR: anàlisi i dissertació

    Get PDF
    Treballs Finals del Grau de Comunicació Audiovisual, Facultat de Biblioteconomia i Documentació, Universitat de Barcelona, Curs 2016-2017, Tutor: Sergi HerreroAnàlisi escrit del videojoc japonès NieR, comparativa amb referents literaris i temes diversos que l'obra tracta, a més de la seva intencionalitat i usos innovadors del mitjà. Inclou dos annexos informatius i el treball de fi de grau que parteix del material exposat en aquests annexos

    COMPOSING WITH ISOMORPHIC AUDIOVISUAL GESTALTS

    Get PDF
    This research represents an attempt to explore aesthetic deviations from established audiovisual practices through the composition of audiovisual works based on the use of audiovisual gestalts. Within the submitted portfolio, the visual materials are made from digitally generated 3D graphics rendered with lines and points suspended in negative space whilst the musical side is anchored in post-acousmatic (Adkins et al., 2016) and post-digital (Cascone, 2002) beats-infused aesthetics. The aesthetic deviations were explored through different qualities and behaviors of gestalts that span between the organic and the artificial. Organic qualities were achieved by exploiting physics or pseudo physics-based algorithms within spatio-temporal continuum while artificial qualities were derived from the use of metric grids, step transformations and rectangular shapes. This exploration includes different approaches to the creation of audiovisual material such as 1) exploitation of the motion of 3D graphics for the purposes of sonification, visual effects and temporal distortion, 2) juxtapositions of different audiovisual hierarchies in terms of which medium leads and which follows. This written text accompanies the portfolio of 1) computer generated audiovisual works composed between 2014 and 2018, consisting of three solo works in total length 36’06” and two collaborative works in total length 7’47”, 2) a musical study piece that is 9’29” in length, 3) Stringvencer –a sequencer that utilises physics-based algorithms for temporal deformations of pre-composed musical structures, consisting of one Max patch and two Max for Live Devices 4) two Max externals and accompanying patches for optimising workflow when using 3D models within Max and for connecting points with lines in a 3D space based on the distances between points, 5) media clips and Max patches that support the commentary. The written text presents theoretical models and concepts as well as tools and techniques that lie behind the construction of the works in the portfolio. Following the presentation of the portfolio in Chapter 2, the commentary continues with the presentation of a theoretical framework in Chapter 3. This theoretical framework, which I named the Isomorphic Audiovisual Paradigm, encompasses models of perception that deal with the perceptual integration and segregation of elements within an audiovisual scene. The models are borrowed from gestalt psychology (isomorphism and grouping principles) and psychophysics (binding factors). In Chapter 4 the aesthetic influences and their integration into my audiovisual language are discussed. Chapter 5 focuses on my compositional practice and thinking. The topics of time, compositional indeterminacy and what I call audiovisual hierarchies, audiovisual tightness and audiovisual escapology are discussed. The workflow and the development of a custom-made audiovisual system that represents my working environment are presented in Chapter 6. Chapter 7 discusses the technical solutions to the problems faced in the course of the research

    Listening-Mode-Centered Sonification Design for Data Exploration

    Get PDF
    Grond F. Listening-Mode-Centered Sonification Design for Data Exploration. Bielefeld: Bielefeld University; 2013.From the Introduction to this thesis: Through the ever growing amount of data and the desire to make them accessible to the user through the sense of listening, sonification, the representation of data by using sound has been subject of active research in the computer sciences and the field of HCI for the last 20 years. During this time, the field of sonification has diversified into different application areas: today, sound in auditory display informs the user about states and actions on the desktop and in mobile devices; sonification has been applied in monitoring applications, where sound can range from being informative to alarming; sonification has been used to give sensory feedback in order to close the action and perception loop; last but not least, sonifications have also been developed for exploratory data analysis, where sound is used to represent data with unknown structures for hypothesis building. Coming from the computer sciences and HCI, the conceptualization of sonification has been mostly driven by application areas. On the other hand, the sonic arts who have always contributed to the community of auditory display have a genuine focus on sound. Despite this close interdisciplinary relation of communities of sound practitioners, a rich and sound- (or listening)-centered concept about sonification is still missing as a point of departure for a more application and task overarching approach towards design guidelines. Complementary to the useful organization along fields of applications, a conceptual framework that is proper to sound needs to abstract from applications and also to some degree from tasks, as both are not directly related to sound. I hence propose in this thesis to conceptualize sonifications along two poles where sound serves either a normative or a descriptive purpose. In the beginning of auditory display research, a continuum between a symbolic and an analogic pole has been proposed by Kramer (1994a, page 21). In this continuum, symbolic stands for sounds that coincide with existing schemas and are more denotative, analogic stands for sounds that are informative through their connotative aspects. (compare Worrall (2009, page 315)). The notions of symbolic and analogic illustrate the struggle to find apt descriptions of how the intention of the listener subjects audible phenomena to a process of meaning making and interpretation. Complementing the analogic-symbolic continuum with descriptive and normative purposes of displays is proposed in the light of the recently increased research interest in listening modes and intentions. Similar to the terms symbolic and analogic, listening modes have been discussed in auditory display since the beginning usually in dichotomic terms which were either identified with the words listening and hearing or understood as musical listening and everyday listening as proposed by Gaver (1993a). More than 25 years earlier, four direct listening modes have been introduced by Schaeffer (1966) together with a 5th synthetic mode of reduced listening which leads to the well-known sound object. Interestingly, Schaeffer’s listening modes remained largely unnoticed by the auditory display community. Particularly the notion of reduced listening goes beyond the connotative and denotative poles of the continuum proposed by Kramer and justifies the new terms descriptive and normative. Recently, a new taxonomy of listening modes has been proposed by Tuuri and Eerola (2012) that is motivated through an embodied cognition approach. The main contribution of their taxonomy is that it convincingly diversifies the connotative and denotative aspects of listening modes. In the recently published sonification handbook, multimodal and interactive aspects in combination with sonification have been discussed as promising options to expand and advance the field by Hunt and Hermann (2011), who point out that there is a big need for a better theoretical foundation in order to systematically integrate these aspects. The main contribution of this thesis is to address this need by providing alternative and complementary design guidelines with respect to existing approaches, all of which have been conceived before the recently increased research interest in listening modes. None of the existing contributions to design frameworks integrates multimodality, and listening modes with a focus on exploratory data analysis, where sonification is conceived to support the understanding of complex data potentially helping to identify new structures therein. In order to structure this field the following questions are addressed in this thesis: • How do natural listening modes and reduced listening relate to the proposed normative and descriptive display purposes? • What is the relationship of multimodality and interaction with listening modes and display purposes? • How can the potential of embodied cognition based listening modes be put to use for exploratory data sonification? • How can listening modes and display purposes be connected to questions of aesthetics in the display? • How do data complexity and Parameter-mapping sonification relate to exploratory data analysis and listening modes

    SOUND MATTERS: AURAL RHETORIC IN PHARMACEUTICAL ADVERTISING (TOWARD A THEORY AND METHOD OF AURALACY)

    Get PDF
    As scholarly theories and explorations of visuality continue to grow in prominence within the discourse on multimodal communication, a gap remains in the literature regarding the affective qualities of sound in multimedia messages. The following study attempts to address this gap and likewise propose a theory of auralacy (aural literacy) by examining the audiovisual interactions present in three different pharmaceutical commercials. This study begins by examining literature regarding Gestalt perceptual theory, with emphasis on its origins, historical developments, and current status in perceptual research. The literature shows that perceptual wholes may be formed visually, tactilely, aurally, or even intermodally. After reviewing this literature, a holistic methodology is presented for the interpretation of multimodal messages. Gunther Kress and Theo van Leeuwen`s Reading Images: The Grammar of Visual Design is invoked to help analyze on-screen movement of visuals in the pharmaceutical commercials, and Leonard Meyer`s principles of pattern perception--derived from Gestalt theory and set forth in Emotion and Meaning and Music--are brought to bear on the advertisement`s soundscapes. The affective qualities of each soundscape are examined, including music, narration, and sound effects. A visual strategy for presenting audiovisual interaction, dubbed as the movement analysis, is used to help foreground the aural mode and display patterns of movement and shape. This study concludes with a discussion of auralacy as an important tool for understanding contemporary composition and communication, as well as potential directions for further developing the movement analysis

    A Gestalt Theory for ‘Disorder’: From Arnheim’s Ordered Chaos to Brambilla’s Entropic Art

    Get PDF
    The article revisits the concept of entropy in art as discussed by Gestalt psychologist and art theorist Rudolf Arnheim. His discussion of artworks and their reception as complex dynamic fields where the forces of entropy and orderliness counter and complete each other, are brought into dialogue with newer approaches, from the perspective of complexity theory and neuroscience, to the dynamics of perception and to entropic processes in the brain. I will argue that even though Arnheim’s observations can still be valuable for contemporary art criticism they need to be updated as they tend to overstate the tendency for order as well as the visual aspects of reception in the expense of multimodal and embodied aspects. In light of these observations, I will discuss contemporary cases of ‘entropic’ art through the moving image works of Marco Brambilla, their aesthetics as well as the ‘structural themes’ arising and the Gestalt processes involved in their reception
    corecore