Spatiotemporal Granulation

This document introduces a novel theory and technique called Spatiotemporal Granulation. Through the use of spatially encoded signals, the algorithm segments spatial and temporal information, producing grains that are localized in both space and time. Well-known transformations that are derived from classical granulation, as well as new manipulations that arise are discussed, and outlined. As a means to reassemble the grains into a new configuration, we explore how granulation parameters acquire a different context, and introduce new methods for control. We present findings and limitations of this new technique, and outline some potential creative and analytical uses. The viability of Spatiotemporal Granulation is demonstrated through a software implementation titled Angkasa

A Compositional Interface for Generative Audiovisual Systems

With a digitally implemented generative system for artistic purposes, controlling a non-trivial number of parameters is one of the difficulties in the process of temporal composition with the system. Unless done manually for every keyframe in the duration of the work, a kind of interface that facilitates the process is necessary. This research proposes an interface for temporal control on parameters of the generative audio-visual system. The proposed interface operates in `state space' formed by a simplex shape of arbitrary dimension with the barycentric coordinate to define the locations inside. Each vertex of simplex represents a reference point in the parameter space of the system. In this manner, regardless of geometric relation between reference points in parameter space, the artist can locate and use any mixed state of the reference states by simple interpolations. This decoupling of the relative coordinate of the states from the specific values of the parameters makes an abstraction for the artist to perform the manual design of the temporal structure efficiently. The state space is a connection between the generative system and the temporal control. With the proposed interface, three artworks are introduced as proof of concepts. Reconstruction is an installation piece, experimenting with virtual geographical shapes. The Reconstruction is a proof of concept that utilizes the state space model for its automatic control, hinting the possibility of a state space model for connection between the generative system and manual composition. Balanced Movement is a fixed length composition, using dynamic equilibrium from the stochastic system as the primary material of the visual. Balanced Movement utilizes state space to define its manual temporal composition. Wavefront is an immersive audio-visual performance using a mathematical abstraction of an ocean wave as visual material. In Wavefront state space is used to spawn entities in performance time dynamically

Audiovisual Reproduction in Surrounding Display: Effect of Spatial Width of Audio and Video

Moniaistinen havaitseminen perustuu informaation yhdistämiseen eri aistikanavista siten, että yhdistetty aistimus tuottaa enemmän tietoa ympäröivästä maailmasta kuin aistimusten käsitteleminen erillisinä. Tämän seurauksena vanhat laatumittarit yhteen aistiin perustuville järjestelmille eivät toimi arvioitaessa monimutkaisempia audiovisuaalisia järjestelmiä, ja uusien laatumittareiden kehittäminen on tarpeellista. Tässä työssä audiovisuaalista havaitsemista tutkittiin immersiivisen audiovisuaalisen näytön avulla. Näyttö koostui 226 laajasta videokuvasta ja 20 kaiuttimella toteutetusta 3D äänentoistosta. Tutkimuksen tavoite oli tarkkailla kuulon ja näön vuorovaikutusta, kun kuvan- ja äänentoiston avaruudellista laajuutta rajoitettiin. Subjektiivinen laatuarviointi toteutettiin käyttäen diskreettiä näytteenhuonontumaskaalaa (DCR) havaitun laadun heikkenemisen arviointiin neljän eri videosisällön kanssa, kun äänen- ja kuvantoiston leveyttä rajoitettiin. Tämän lisäksi osallistujilta kerättiin vapaita kuvauksia heidän antamiinsa laatuarviointeihin vaikuttaneista seikoista. Osallistujien yksilölliset taipumukset kokea uppoutumista arvioitiin ennen koetta kyselylomakkeen avulla. Tulokset osoittavat, että videon leveys on määräävä tekijä arvioitaessa havaittua laadun heikkenemistä. Myös äänenleveydellä oli merkitystä, kun videonleveys oli suurimmillaan. Taipumus kokea uppoutumista ei ollut merkittävä tekijä laadun kannalta tässä tutkimuksessa. Videosisällön merkitys oli vähäinen. Vapaille kuvauksille suoritettu rajoitettu korrespondenssianalyysi ehdottaa huonoon havaittuun laatuun vaikuttaviksi tekijöiksi äänen väärän tulosuunnan, rajoitetun videonleveyden ja puuttuvan tärkeän sisällön.Multimodal perception strives to integrate information from multiple sensorial channels into a unified experience, that contains more information than just the sum of the separate unimodal percepts. As a result, traditional quality metrics for unimodal services cannot reflect the perceived quality in multimodal situations, and new quality estimation methods are needed. In this work, audiovisual perception was studied with an immersive audiovisual display. The audiovisual display consisted of a video screen with field of view of 226 and 3D sound reproduction with 20 loudspeakers. The aim of the study was to observe the crossmodal interaction of auditory and visual modalities, when the spatial widths of audio and video reproduction were limited. A subjective study was organized, where the overall perceived degradation of the stimuli was evaluated with Degradation Category Rating in four different types of audiovisual content. In addition, free descriptions of the most prominent degrading factors were collected. The participants' individual tendencies to experience immersion were screened prior to the experiment with a questionnaire. The results show that video width is the dominant element in defining the degradation of a stimulus. Also audio width had an impact when the video width was at maximum. Individual tendency to experience immersion was not found to have significant impact on perceived degradation in this study. Slight content effects were observed. Constrained correspondence analysis of the free description data suggests the reasons for highest perceived degradation to be caused by wrong audio direction, reduced video width and missing essential content

Exploring the role of large-scale immersive computing environments in collaboration between engineering and design students

In order to solve the engineering challenges of today, multidisciplinary collaboration is essential. Unfortunately there are many obstacles to communication between disciplines, such as incongruent vocabularies and mismatched knowledge bases, which can make collaboration difficult. The silos separating disciplines, created through focused educational curriculum, are also a large barrier. During their education, designers and engineers are encouraged to employ specific methods unique to their discipline to share ideas with their peers. In many cases, however, these methods do not translate between disciplines, making it challenging for two groups to exchange ideas and perspectives effectively. There are, however, some tools that have emerged to help bridge the gap between designers and engineers. Currently, the most pervasive solution to these challenges is Computer-Aided Drafting (CAD) software. This software is used by both engineers and designers, allowing both groups to design and evaluate models in a common medium. This makes it decidedly easier for these two groups to collaborate with each other. However, CAD has its own limitations. Navigating in a three-dimensional environment with two dimensional input devices is unnatural and imposes an additional physical and cognitive load on the user. Desktop screens also limit decision-making capabilities due to their small size and the potential to create distorted impressions of size and scale of models larger than the computer screen. Large-Scale Immersive Computing Environments (LSICEs) improve upon the benefits of CAD software. They provide users with the ability to not only visualize their designs three dimensionally, but also allow for natural interactions with 3D models and the ability to view a design as the designer had intended, in true scale. This can improve the ability of users to collaborate in a number of different ways. The natural interaction interface allows students to focus on sharing ideas with their collaborators. Additionally, the common medium makes it much easier for the two groups to communicate with each other, eliminating one of the main obstacles to interdisciplinary collaboration in education. This research seeks to gain a better understanding of how design and engineering design students use LSICEs to collaborate with peers, both within and outside of their discipline. Two studies were conducted. In the first study, two different classes of students used a LSICE as a tool during their design process. The first class was a design class that utilized the LSICE as a part of three design projects throughout the semester. The second class was a sophomore engineering design class. These students also used the LSICE as a part of their design process, however these students used the virtual environment over the course of a single semester-long design project. Students were given a short survey at the end of their experiences in the virtual environment. From this study, some interesting results emerged. Both groups of students indicated that the virtual environment was a benefit to their design process, regardless of background or time spent in the space. Statistical analysis of the students\u27 responses revealed no significant differences between the two groups of students. The final study brought engineering and design students together to complete a design review task within the LSICE. This study was conducted in order to evaluate the role that LSICEs play in facilitating collaboration between engineers and designers. Upon conclusion of the design review, students were given a survey to gather information of their perceptions of the virtual environment in visualizing designs, communicating with their peers and interacting with designs. From this study it became quite clear that students find LSICEs to be effective in facilitating communication between disciplines. Additionally, the majority of students commented on the positive effect that the natural interaction interface had on their ability to evaluate the design. Throughout each of these studies, common themes emerged between both groups. Student responses show many perceived benefits to LSICEs which have the potential to inspire student-driven interdisciplinary collaboration. Participants found that the environment improved their ability to communicate, whether it be with peers within their disciplines or when working in interdisciplinary groups. Students also found that interacting in the environment in a natural way improved their ability to make judgments about spatial relationships among components. The results from this research are quite promising. Providing students with collaboration tools that support natural human interaction with CAD models of real size has the potential for greatly improving a student\u27s educational experience. Manipulating full size CAD models encourages students to visualize the size and shape of the final design before it is built. Seeing the designs in full scale allows everyone on the team to experience the design and provide their input into the design discussions. This research continues an effort in academia to leverage cutting edge technology to improve student learning by providing unique opportunities to interact with peers in design teams, promoting graduates who are well equipped to work effectively across disciplines to address the challenges of today

Biometric Data Art: Personalized Narratives and Multimodal Interaction

Biometric technology has brought enhancements to identification and access control. As more digital applications request people to input their biometric data as a more convenient and secure method of identification, the possibility of losing their personal data and identities may increase. The phenomenon of biometric data abuse causes one to question what their true identity may be and what methods can be used to define identity and hidden narratives. The questions of identification and the insecurity of biometric data have become my inspiration, providing artistic approaches to the manipulation of biometric data and having the potential to suggest new directions for solving the problems. To do so, in-depth investigation of the narratives beyond the visual features of the biometric data is necessary. This content can create a close link between an artwork and its audience by causing the latter to become deeply engaged with the artwork through their own stories.This dissertation examines narratives and artistic explorations discovered from one form of biometric data, fingerprints, drawing on insights from various fields such as genetics, hand analysis, and biology. It also presents contributions on new ways of creating interactive media artworks using fingerprint data based on visual feature analysis of the data and multimodal interaction to explore their sonic signatures. Therefore, the artwork enriches interactive media art by incorporating personalization into the artistic experience, and creates unique personalized experience for each audience member. This thesis documents developments and productions of a series of artworks, Digiti Sonus, by focusing on its conceptual approaches, design, techniques, challenges and future directions

Multi-Listener Auditory Displays

This thesis investigates how team working principles can be applied to Auditory Displays (AD). During this work it was established that there the level of collaboration and team work within the AD community was low and that this community would benefit from a enhanced collaborative approach. The increased use of collaborative techniques will benefit the AD community by increasing quality, knowledge transfer, synergy, and enhancing innovation. The reader is introduced to a novel approach to collaborative AD entitled Multi-listener Auditory Displays (MLAD). This work focused upon two areas of MLAD distributed AD teams and virtual AD teams. A distributed AD team is a team of participants who work upon a common task at different times and locations. The distributed approach was found to work effectively when designing ADs that work upon large scale data sets such as that found in big data. A virtual AD team is a group of participants who work upon a common task simultaneously and in separate locations. A virtual AD team is assisted by computer technology such as video conferencing and email. The virtual auditory display team was found to work well by enabling a team to work more effectively together who were geographically spread. Two pilot studies are included; SonicSETI is an example of a distributed AD team, where a remote group of listeners have background white noise playing, and use passive listening to detect anomalous candidate signals; and a geographically diverse virtual AD team that collaborates through electronic technology on an auditory display which sonifies a database of red wine measurements. A workshop was organised at a conference which focused upon ensemble auditory displays with a group of participants who were co- located

The Drowning World : The visual culture of climate change

A challenging question today is how to understand and act on climate change. Previous analyses of the public outreach of the climate sciences have concluded that the urgent communication of climate change is inadequate. It is foremost the invisibility of carbon dioxide and the lack of a tangible relationship between current emissions and future effects that have been seen as the main challenge to visually represent. The Drowning World questions how the communication problem is articulated, and the analysis focuses on the supplementary images that come with this scientific communication, including cover images to reports, backgrounds to diagrams, or graphic design elements. The conclusion is that even if the scientific images might fail to communicate the complexity of the climate issue, the supplementary images, and the way the story of our changing world is told, manage to bring a feeling of change with them. Images of water are especially recurring, as are projects that use immersive environments like virtual reality, and these representations compete for attention in the media noise of modern society, a world that “drowns” the viewers in auditory and visual stimuli. Thus there are many reasons for the title of this thesis – The Drowning World

Three Etudes for Symmetrical Cuboctahedral Speaker Array: An Exploration of Aural Space-Time Symmetry

Strategies of musical symmetry can partition an aural space in a manner similar to the way that scales partition an octave. In an exploration of a 12-channel diffusion system, I will compose three musical pieces expressing various methods of designing symmetrical based aural spatial premises with the goal of creating an experienceable spatial harmony. To achieve this, I will first design a cuboctahedral speaker array capable of representing multiple spatial harmonies. Second, I will develop a notation system that symmetrically visually represents the spatial harmonies for compositional intentionality. Finally, I will program a Max MSP software patch capable of integrating the array and notation for playback experiences and compositional interpretations. After these processes are outlined and described, I will then compose three etudes to demonstrate diverse methods of compositional strategies; they are dedicated to, and inspired by the works of Terrence McKenna, Nicola Tesla, and Carl Sagan

Rhythm, Timbre and Space as Musical Expression

This dissertation comprises a collection of works that address aspects of soundoccupying space. Those aspects refer to inner components of timbre and rhythm andhow these may influence the perception and projection of sound. These works alsoaddress the interaction of physical or virtual spaces and spatial attributes of sound inregard to rhythm and timbre. The composition of timbres serves as fundamental basefor the musical material contained in the works, using sound synthesis and extendedtechniques as a paradigm for articulation and instrumentation of sound in vertical andhorizontal layers. Rhythm is not only a parameter that articulates time; it may also bethe parameter that articulates musical material. The relationships of timbre and rhythmserve as expressive substance for the elaboration of immersive soundscapes that aremolded by time and memory