Data Sonification in Creative Practice

Sonification is the process of data transmission with non-speech audio. While finding increasing acceptance as a scientific method, particularly where a visual representation of data is inadequate, it is still often derided as a ‘gimmick’. Composers have also shown growing interest in sonification as a compositional method. Both in science and in music, the criticism towards this method relates to poor aesthetics and gratuitous applications. This thesis aims to address these issues through an accompanying portfolio of pieces which use sonification as a compositional tool. It establishes the principles of ‘musification’, which can be defined as a sonification which uses musical structures; a sonification organised by musical principles. The practice-as-research portfolio explores a number of data sources, musical genres and science-music collaborations. The main contributions to knowledge derived from the project are a portfolio of compositions, a compositional framework for sonification and an evaluation framework for musification. This thesis demonstrates the validity of practice-as-research as a methodology in sonification research

Live coding machine learning and machine listening: a survey on the design of languages and environments for live coding

The MIMIC (Musically Intelligent Machines Interacting Creatively) project explores how the techniques of machine learning and machine listening can be communicated and implemented in simple terms for composers, instrument makers and performers. The potential for machine learning to support musical composition and performance is high, and with novel techniques in machine listening, we see emerging a technology that can shift from being instrumental to conversational and collaborative. By leveraging the internet as a live software ecosystem, the MIMIC project explores how such technology can best reach artists, and live up to its collaborative potential to fundamentally change creative practice in the field. The project involves creating a high level language that can be used for live coding, creative coding and quick prototyping. Implementing a language that interfaces with technically complex problems such as the design of machine learning neural networks or the temporal and spectral algorithms applied in machine listening is not a simple task, but we can build upon decades of research and practice in programming language design (Ko 2016), and computer music language design in particular, as well as a plethora of inventive new approaches in the design of live coding systems for music (Reina et al. 2019). The language and user interface design will build on recent research in creative coding and interactive machine learning, exemplified by the Rapid Mix project (Bernardo et. al., 2016, Zbyszynski et. al., 2017). Machine learning continues to be at the forefront of new innovations in computer music, (e.g. new sound synthesis techniques in NSynth (Engel et. al. 2017) and WaveNet (van den Oord, 2016)); the language will seek to integrate models based around these new techniques into live coding performance, and also explore the efficacy of live coding as an approach to training and exploiting these systems for analysing and generating sound. Existing live coding systems and languages are often reported on, describing clever solutions as well as weaknesses, as given, for example, in accounts of the development of Tidal (McLean, 2014), Extramuros (Ogborn et. al, 2015) and Gibber (Roberts and Kuchera-Morin, 2012). Researchers are typically reflective and openly critical of their own systems when analysing them and often report on its design with wider implications (Aaron 2011; Sorensen 2018). However, they rarely speculate freely and uninhibitedly about possible solutions or alternative paths taken; the focus is typically on the system described. Before defining the design of our own system, we were therefore interested in opening up a channel where we could learn from other practitioners in language design, machine learning and machine listening. We created a survey that we sent out to relevant communities of practice - such as live coding, machine learning, machine listening, creative coding, deep learning - and asked open questions about how they might imagine a future system implemented, given the knowledge we have today. Below we report on the questionnaire and its findings

A short history off-line

Emerging technologies for learning report - Article exploring the history of ICT in education and the lessons we can learn from the pas

NEUVis: Comparing Affective and Effective Visualisation

Data visualisations are useful for providing insight from complex scientific data. However, even with visualisation, scientific research is difficult for non-scientists to comprehend. When developed by designers in collaboration with scientists, data visualisation can be used to articulate scientific data in a way that non-experts can understand. Creating human-centred visualisations is a unique challenge, and there are no frameworks to support their design. In response, this thesis presents a practice-led study investigating design methods that can be used to develop Non-Expert User Visualisations (NEUVis), data visualisations for a general public, and the response that people have to different kinds of NEUVis. For this research, two groups of ten users participated in quantitative studies, informed by Yvonna Lincoln and Egon Guba’s method of Naturalistic Inquiry, which asked non-scientists to express their cognitive and emotional response to NEUVis using different media. The three different types of visualisations were infographics, 3D animations and an interactive installation. The installation used in the study, entitled 18S rDNA, was developed and evaluated as part of this research using John Zimmerman’s Research Through Design methodology. 18S rDNA embodies the knowledge and design methods that were developed for this research, and provided an opportunity for explication of the entire NEUVis design process. The research findings indicate that developing visualisations for the non-expert audience requires a new process, different to the way scientists visualise data. The result of this research describes how creative practitioners collaborate with primary researchers and presents a new human-centred design thinking model for NEUVis. This model includes two design tools. The first tool helps designers merge user needs with data they wish to visualise. The second tool helps designers take that merged information and begin an iterative, user-centred design process

Hyvinvoinnin muotoilu

Designing for Wellbeing consists of 12 projects which represent actual services or processes in the cities of Helsinki, Espoo, Kauniainen and Lahti. Projects address different dimensions of wellbeing, focusing in particular on municipal wellbeing services and patient-centered health care solutions. Designing for Wellbeing highlights new working methods in design, such as service design and the opportunities it provides for municipal decision-makers and the general public using the services. The projects are aimed at finding ways of encouraging people to adopt healthier lifestyles and helping designers and municipal decision-makers to design more pleasant and healthier environments. Examples of the services include redesigning the Villa Breda service home for the elderly in Kauniainen to include cultural services and social events for today’s active retirees, developing the environments and practices in psychiatric care units in Helsinki, reinventing the suburban neighborhoods in Helsinki and Lahti, designing better online services for basic health care and creating smoke-free public environments

An exploration of the conceptual relationship between design aesthetics and Aristotelian rhetoric in information visualisation

This study explores the conceptual relationship between design aesthetics and Aristotelian rhetoric in the context of information visualisation. Aesthetics and rhetorical theory are traditionally studied as separate discourses, but conceptual links between these fields are identified, specifically in terms of communicative goals and strategies. This study therefore compares selected theories on design aesthetics and Aristotelian rhetoric in information visualisation in order to ascertain whether a combined framework may be feasible. Although information visualisation is traditionally practiced from software engineering disciplines, this study frames the practice within the broader field of information design. The democratisation of the field of information visualisation and the emerging practices that emphasise the aesthetic value of visualisations is explored. In order to understand what is meant by the term ‘aesthetic’, a variety of both classical and contemporary views on aesthetics theory is investigated. Even though the term ‘aesthetic’ is not defined, a broad understanding is created by identifying the main conceptual themes in discourse. A specific focus is placed on understanding aesthetics in a design context, since there are many misconceptions about ‘aesthetics’ in this context. The idea that aesthetics relates to the communication of artifacts is explored, which provides a point of departure in linking aesthetics and rhetorical theory. The communicative nature of information visualisations is thus explored in relation to visual rhetorical theory. Aristotle’s three rhetorical appeals, namely logos, pathos and ethos, form the backbone of the visual rhetorical analysis of visualisation artifacts. The aesthetic and rhetorical theories explored throughout the study are compared by applying them to Charles Joseph Minard’s seminal information visualisation of Napoleon’s march to Moscow. This comparative analysis considers the traditional divide between aesthetics and rhetorical theory but identifies sufficient conceptual links between the discourses to suggest that a combined aesthetic-rhetorical framework for information visualisation may indeed be practical. Lastly, the wider implications and potential value of such a combined framework is considered within a broader design context. CopyrightDissertation (MA)--University of Pretoria, 2012.Visual Artsunrestricte

Soundscape Generation Using Web Audio Archives

Os grandes e crescentes acervos de áudio na web têm transformado a prática do design de som. Neste contexto, sampling -- uma ferramenta essencial do design de som -- mudou de gravações mecânicas para os domínios da cópia e reprodução no computador. A navegação eficaz nos grandes acervos e a recuperação de conteúdo tornaram-se um problema bem identificado em Music Information Retrieval, nomeadamente através da adoção de metodologias baseadas no conteúdo do áudio.Apesar da sua robustez e eficácia, as soluções tecnológicas atuais assentam principalmente em métodos (estatísticos) de processamento de sinal, cuja terminologia atinge um nível de adequação centrada no utilizador.Esta dissertação avança uma nova estratégia orientada semanticamente para navegação e recuperação de conteúdo de áudio, em particular, sons ambientais, a partir de grandes acervos de áudio na web. Por fim, pretendemos simplificar a extração de pedidos definidos pelo utilizador para promover uma geração fluida de paisagens sonoras. No nosso trabalho, os pedidos aos acervos de áudio na web são feitos por dimensões afetivas que se relacionam com estados emocionais (exemplo: baixa ativação e baixa valência) e descrições semânticas das fontes de áudio (exemplo: chuva). Para tal, mapeamos as anotações humanas das dimensões afetivas para descrições espectrais de áudio extraídas do conteúdo do sinal. A extração de novos sons dos acervos da web é feita estipulando um pedido que combina um ponto num plano afetivo bidimensional e tags semânticas. A aplicação protótipo, MScaper, implementa o método no ambiente Ableton Live. A avaliação da nossa pesquisa avaliou a confiabilidade perceptual dos descritores espectrais de áudio na captura de dimensões afetivas e a usabilidade da MScaper. Os resultados mostram que as características espectrais do áudio capturam significativamente as dimensões afetivas e que o MScaper foi entendido pelos os utilizadores experientes como tendo excelente usabilidade.The large and growing archives of audio content on the web have been transforming the sound design practice. In this context, sampling -- a fundamental sound design tool -- has shifted from mechanical recording to the realms of the copying and cutting on the computer. To effectively browse these large archives and retrieve content became a well-identified problem in Music Information Retrieval, namely through the adoption of audio content-based methodologies. Despite its robustness and effectiveness, current technological solutions rely mostly on (statistical) signal processing methods, whose terminology do attain a level of user-centered explanatory adequacy.This dissertation advances a novel semantically-oriented strategy for browsing and retrieving audio content, in particular, environmental sounds, from large web audio archives. Ultimately, we aim to streamline the retrieval of user-defined queries to foster a fluid generation of soundscapes. In our work, querying web audio archives is done by affective dimensions that relate to emotional states (e.g., low arousal and low valence) and semantic audio source descriptions (e.g., rain). To this end, we map human annotations of affective dimensions to spectral audio-content descriptions extracted from the signal content. Retrieving new sounds from web archives is then made by specifying a query which combines a point in a 2-dimensional affective plane and semantic tags. A prototype application, MScaper, implements the method in the Ableton Live environment. An evaluation of our research assesses the perceptual soundness of the spectral audio-content descriptors in capturing affective dimensions and the usability of MScaper. The results show that spectral audio features significantly capture affective dimensions and that MScaper has been perceived by expert-users as having excellent usability