Composition with complex data : a contribution on the mapping problem through practice-based research

Composition with complex data is a field of computer music composition/interactive art that uses extra-musical data from various sources like stock exchange data, weather data or seismic data. Despite the fascination that one can have in his exciting field of composition, there is still a lack of data management applications for artistic use. Hence, one is generally forced to create one’s own applications, at the expense of the time that should be spent in the artistic side of the work. As the technology can be part of the work but never the sole constituent, we decided to develop a toolbox, DataScapR, which allows artists to work easily with data, helping them in focusing on the artistic side. This toolbox will help the user to quickly advance beyond the technical development and focus on the artistic side of the project. This project consists of four components: a theoretical framework for sonification art, a state of the art and discussion on mapping techniques, the development of a sonification toolbox for composers who wish to use complex data (more specifically stock market data) as the source material for their music and a series of works as case studies to show the capabilities of the toolbox. Bringing theory and practice, art and technology together, this project can be seen as a practice-based one embedded within a theoretical framework.Nesta dissertação, abordo o tema da sonificação em contexto artístico. Sonificação, a tradução de dados en sons, é um largo campo de investigação e existe todo o interesse na exploração deste domínio num contexto artístico. Esse projecto visa contribuir para a criação de um ramo teórico para a arte da sonificação e, em simultâneo, apresentar uma aplicação que facilite o uso de sonificação em contexto artístico. Com efeito, actualmente, existem poucas aplicações que permitam usar sonificação na composição musical de uma forma acessível. Por esta razão, um compositor pode ter que aplicar um tempo considerável no desenvolvimento de uma aplicação própria o que muitas vezes, pode ter um efeito prejudicial na criação da obra em si. Se o compositor tem que aplicar demasiado tempo no desenvolvimento tecnológico pode correr o risco de considerar a aplicação como sendo a obra em si, o que não é o caso: pode ser uma parte mas nunca a obra. Para combater o pequeno leque de aplicações acessíveis criei uma caixa de ferramentas, DataScapR, desenvolvido em Max, e disponibilizada para o público em geral. DataScapR é um projecto aberto: o utilizador pode estendê-lo livremente e adaptá-lo às suas necessidades. Todos os patches são comentados extensivamente para facilitar a sua edição e extensão. O uso prático de DataScapR é exemplificado através de Através de estudos de caso demonstrando que a sonificação pode ser uma prática interessante integrada num contexto artístico. Nesse projecto de doutoramento foco um tipo de dados especifico: dados da bolsa das acções. Isso vem dum interesse pessoal e no dinamismo inerente à bolsa. Sempre considerei a bolsa fascinante e penso que pode ser interessante para usar os dados para sonificação. A dissertação consiste em quatro partes. A primeira parte aborda questões teóricos: procuramos uma definição de arte de sonificação e integramos essa prática no contexto da composição. Tratamos da questão da natureza e definição de dados e como eles podem ser aplicados na música. Depois de construída uma base teórica, descrevemos o estado da arte. Nesse segundo capítulo descrevemos obras que usam sonificação como componente importante da própria existência e discutimos os diferentes métodos de mapping. Seguidamente, discuto o software existente bem como a necessidade duma nova aplicação. No terceiro capítulo apresento DataScapR, um dos componentes práticos do doutoramento. DataScapR é uma caixa de ferramentas para sonificação de dados da bolsa de acções. Assim, apresento os três módulos que permitem usar dados em tempo real e dados históricos. Os métodos de mapping são explicados e a estrutura interna dos patches é apresentada. Finalmente, no quarto capítulo apresento as obras realizadas usando DataScapR: For A Fistful Of Data (flauta de bisel), 4D Brokers (instalação), Vapourwaves (instalação), Mirage (obra sobre suporte). Para cada caso, apresento a obra, discuto a sua estrutura, os mappings utilizados e as questões técnicas e termino com uma avaliação da obra. No final do capítulo concluo com uma avaliação geral das peças. Na discussão final realizo uma avaliação do trabalho feito e aponto direcções para trabalho futuro. A dissertação é da caixa de ferramentas DataScapR, de quatro estudos de caso e dois blogs: datascapr.wordpress.com onde o DataScapR está disponível e sonifcationart.wordpress.com onde discuto vários projectos de sonificação. Esse projecto de doutoramento mostra apenas uma das posições possíveis em arte de sonificação e, por esta razão, deve ser considerado como uma abertura para novos caminhos a explorar

Gesture-Controlled Interaction with Aesthetic Information Sonification

Information representation in augmented and virtual reality systems, and social physical (building) spaces can enhance the efficacy of interacting with and assimilating abstract, non-visual data. Sanification is the process of automatically generated real time information representation. There is a gap in our implementation and knowledge of auditory display systems used to enhance interaction in virtual and augmented reality. This paper addresses that gap by examining methodologies for mapping socio-spatial data to spatialised sanification manipulated with gestural controllers. This is a system of interactive knowledge representation that completes the human integration loop, enabling the user to interact with and manipulate data using 3D spatial gesture and 3D auditory display. Benefits include 1) added immersion in an augmented or virtual reality interface; 2) auditory display avoids visual overload in visually-saturated processes such as designing, evacuation in emergencies, flying aircraft; computer gaming; and 3) bi-modal or auditory representation, due to its time-based character, facilitates cognition of complex information

Responsive Sensate Environments: Past and Future Directions Designing Space as an Interface with Socio-Spatial Information

Abstract: This paper looks at ways in which recent developments in sensing technologies and gestural control of data in 3D space provide opportunities to interact with information. Social and spatial data, the utilisation of space, flows of people and dense abstract data lend themselves to visual and auditory representation to enhance our understanding of socio-spatial patterns. Mapping information to visualisation and sonification leads to gestural interaction with information representation, dissolving the visibility and tangibility of traditional computational interfaces and hardware. The purpose of this integration of new technologies is to blur boundaries between computational and spatial interaction and to transform building spaces into responsive, intelligent interfaces for display and information access. INTRODUCTION Rather than the traditional computer aided architectural design and information communication technology (ICT) integration into architecture, this paper looks designing computer-aided architecture, i.e. spaces and structures enhanced by embedded sensor technologies and responsive (computational) building intelligence. Architecture's responsibility to society could be viewed as designing a sympathetic environment for human experience and interaction. Emerging sensing technologies and intelligence research illuminate interesting opportunities for designing this experience. RESPONSIVE ENVIRONMENTS Responsive environments include sensate spaces, enabled by spatially-and sociallytriggered devices, intelligent and smart houses (utilising video tracking and data capture), networked sensor environments, pervasive mobile computing solutions and ambient visual and auditory displays. This paper briefly reviews the benefits of extant responsive technologies that have developed since last century until th

Application of Optimization and Simulation to Musical Composition that Emerges Dynamically during Ensemble Singing Performance

31 pages, 11 FiguresThis paper presents and tests a new approach to composing for ensemble singing performance: reality opera. In the performance of such a composition, emotions of the singers are real and emerge as a consequence of their interactions and reaction and to a dynamic narrative. This paper gives background and motivation for the form, based on three key concepts, incorporating the use of technology. Then proposed techniques for creating reality opera are instantiated in an example, which is performed and a behavioral analysis done of performer reactions, leading to support for the feasibility of the reality opera concept

End-user Development of Sonifications using Soundscapes

Designing sonifications requires knowledge in many domains including sound design, sonification design, and programming. Thus end users typically do not create sonifications on their own, but instead work with sonification experts to iteratively co-design their systems. However, once a sonification system is deployed there is little a user can do to make adjustments. In this work, we present an approach for sonification system design that puts end users in the control of the design process by allowing them to interactively generate, explore, and refine sonification designs. Our approach allows a user to start creating sonifications simply by providing an example soundscape (i.e., an example of what they might want their sonification to sound like), and an example dataset illustrating properties of the data they would like to sonify. The user is then provided with the ability to employ automated or semi-automated design of mappings from features of the data to soundscape controls. To make this possible, we describe formal models for soundscape, data, and sonification, and an optimization-based method for creating sonifications that is informed by design principles outlined in past auditory display research

Pulsed Melodic Affective Processing: Musical structures for increasing transparency in emotional computation

Pulsed Melodic Affective Processing (PMAP) is a method for the processing of artificial emotions in affective computing. PMAP is a data stream designed to be listened to, as well as computed with. The affective state is represented by numbers that are analogues of musical features, rather than by a binary stream. Previous affective computation has been done with emotion category indices, or real numbers representing various emotional dimensions. PMAP data can be generated directly by sound (e.g. heart rates or key-press speeds) and turned directly into music with minimal transformation. This is because PMAP data is music and computations done with PMAP data are computations done with music. This is important because PMAP is constructed so that the emotion that its data represents at the computational level will be similar to the emotion that a person “listening” to the PMAP melody hears. Thus, PMAP can be used to calculate “feelings” and the result data will “sound like” the feelings calculated. PMAP can be compared to neural spike streams, but ones in which pulse heights and rates encode affective information. This paper illustrates PMAP in a range of simulations. In a multi-agent simulation, initial results support that an affective multi-robot security system could use PMAP to provide a basic control mechanism for “search-and-destroy”. Results of fitting a musical neural network with gradient descent to help solve a text emotional detection problem are also presented. The paper concludes by discussing how PMAP may be applicable in the stock markets, using a simplified order book simulation. © 2014, The Society for Modeling and Simulation International. All rights reserved

The Archetypal Market Hypothesis; A Complex Psychology Perspective on the Market’s Mind

The thesis introduces the Archetypal Market Hypothesis (AMH). Based on complex psychology and supported by insights from other (mind) sciences it describes the unconscious nature of investing and how it shapes price patterns. Specifically, it emphasises the central role of numerical archetypes in price discovery. Its ontological premise is the market’s mind, a complex adaptive system in the form of collective consciousness which originates from the collective unconscious. This premise suggests that investing involves more than cognition and reaches beyond rationality and logic. Among others, the thesis clarifies the affective impact of price discovery: it is not only what we can do with prices, but also what they can do with us. Numbers receive their affective powers from the numerical archetypes. They preconsciously create order in the mind by facilitating the dynamics of symbolic mapping as the mind attempts to make sense of what it senses, bridging the imaginative with the real. This autonomous and often dominating impact of the numerical archetypes manifests itself: • in individual consciousness via numerical intuition, and • in crowd consciousness via participation mystique which underlies intersubjectivity. The thesis will argue that both are supported cerebrally. The collective intersubjective nature of the market’s mind and its symbolic expression via prices make it an exemplary phenomenon to be researched because the archetypal dynamics are strongest in such spheres. The PhD’s goal, as part of the AMH proposition, is twofold. First, to formalise theoretically the concept of the market’s mind, in particular the collective experience of market states, generally known as market moods, and how these shift as a result of herd instinct. Second, to propose a framework for further empirical research to show that representing market data in a non-traditional way, based on Jung’s active imagination and similar techniques, can improve investors’ understanding of those states. If successful, the method (including bespoke software) can complement analytical investment research methods currently used by investor

Artech 2008: proceedings of the 4th International Conference on Digital Arts

ARTECH 2008 is the fourth international conference held in Portugal and Galicia on the topic of Digital Arts. It aims to promote contacts between Iberian and International contributors concerned with the conception, production and dissemination of Digital and Electronic Art. ARTECH brings the scientific, technological and artistic community together, promoting the interest in the digital culture and its intersection with art and technology as an important research field, a common space for discussion, an exchange of experiences, a forum for emerging digital artists and a way of understanding and appreciating new forms of cultural expression. Hosted by the Portuguese Catholic University’s School of Arts (UCP-EA) at the City of Porto, ARTCH 2008 falls in alignment with the main commitment of the Research Center for Science and Technology of the Arts (CITAR) to promote knowledge in the field of the Arts trough research and development within UCP-AE and together with the local and international community. The main areas proposed for the conference were related with sound, image, video, music, multimedia and other new media related topics, in the context of emerging practice of artistic creation. Although non exclusive, the main topics of the conference are usually: Art and Science; Audio-Visual and Multimedia Design; Creativity Theory; Electronic Music; Generative and Algorithmic Art; Interactive Systems for Artistic Applications; Media Art history; Mobile Multimedia; Net Art and Digital Culture; New Experiences with New Media and New Applications; Tangible and Gesture Interfaces; Technology in Art Education; Virtual Reality and Augmented Reality. The contribution from the international community was extremely gratifying, resulting in the submission of 79 original works (Long Papers, Short Papers and installation proposals) from 22 Countries. Our Scientific Committee reviewed these submissions thoroughly resulting in a 73% acceptance ratio of a diverse and promising body of work presented in this book of proceedings. This compilation of articles provides an overview of the state of the art as well as a glimpse of new tendencies in the field of Digital Arts, with special emphasis in the topics: Sound and Music Computing; Technology Mediated Dance; Collaborative Art Performance; Digital Narratives; Media Art and Creativity Theory; Interactive Art; Audiovisual and Multimedia Design.info:eu-repo/semantics/publishedVersio