Gesture-Timbre Space: Multidimensional Feature Mapping Using Machine Learning & Concatenative Synthesis

This paper presents a method for mapping embodied gesture, acquired with electromyography and motion sensing, to a corpus of small sound units, organised by derived timbral features using concatenative synthesis. Gestures and sounds can be associated directly using individual units and static poses, or by using a sound tracing method that leverages our intuitive associations between sound and embodied movement. We propose a method for augmenting corporal density to enable expressive variation on the original gesture-timbre space

DIGIT - DIGItal sTeps

Apesar da grande evolução tecnológica sentida nas últimas décadas no domínio dos média no sentido da automatização dos processos de pós-produção áudio este não se fez sentir na realização de foley de som, que continua a ser um processo manual que assenta nas capacidades técnico-interpretativas de um artista de foley. À imagem em movimento, este último, oferece significado sónico através de uma manipulação expressiva de objetos, fazendo coincidir a sua performance com a imagem em movimento. De reconhecido impacto expressivo, o sound foley é um processo moroso e que exige muitos recursos materiais, e assim sendo foram desenvolvidas recentemente propostas digitais para a concretização destes sons, e mais concretamente para a produção do som de passos. No entanto estas soluções apresentam limitações em (1) exaustão das propostas no caso de recurso a bibliotecas de som, (2) limitação do controlo MIDI no caso do recurso a plugins para a produção do som de passos e (3) alguma falta de realismo no caso do recurso ao áudio procedimental.Nesta dissertação, de forma a dar resposta às limitações levantadas anteriormente apresento o DIGItal sTeps (DIGIT), um sistema digital interativo para a concretização do som de passos num contexto de foley de som. DIGIT baseia-se no conceito de perfil acústico de um gesto produzido pelo impacto da mão numa superfície captado por um microfone de contacto. Através da descrição do perfil acústico destes gestos e do seu mapeamento para parâmetros de controlo de síntese concatenativa, pretende-se a realização de uma pesquisa automática numa bases de dados composta por sons de passos pré-gravados, e a posterior escolha do segmento mais adequado à descrição acústica do gesto. DIGIT potencia a expressividade e exploração do gesto como forma de recriação do som de passos em ambiente de pós-produção áudio de um filme ou jogo, dando um seguimento à prática tradicional do foley de som, que através da exploração táctil de objetos físicos, e reconhecimento das suas qualidades acústicas, os adapta às necessidades temáticas e narrativas da imagem em movimento.DIGIT apresenta vantagens em relação às soluções anteriores na medida em que utiliza a descrição do perfil acústico do gesto (nas suas múltiplas dimensões) como controlo do sistema visando desta forma contornar as limitações impostas pelo protocolo de comunicação MIDI. DIGIT dispõe ainda de um motor de reprodução áudio granular que permite uma alteração das características originais de um som de forma a criar mais possibilidades a partir de um pequeno número de amostras. Esta funcionalidade contribui para uma diminuição do tamanho da base de dados de som de passos pré-gravados que compõe o sistema ao contrário das soluções como bibliotecas de som ou plugins.Uma particularidade do sistema DIGIT em relação a soluções já existentes é o facto de se poder trabalhar por camadas dando a possibilidade ao utilizador numa mesma instância do software poder compor som de passos mais complexos.Despite the great technological evolution felt in recent decades in the media field towards the automation of audio post-production processes that did not occur in the practise of sound foley, which remains a manual process that is based on the technical and interpretive skills of a foley artist. In the moving image, the latter offers sonic meaning through an expressive manipulation of objects by matching its performance with the moving image.With recognized significant impact, foley sound is a lengthy process and requires a great deal of material resources, and therefore have been recently developed digital proposals to achieve these kind of sounds, and more specifically for the production of footstep sounds. However these solutions have limitations in (1) rapid exhaustion of proposals in the case of the use of sound libraries, (2) MIDI control limitation in the case of the use of plugins to produce footstep sounds and (3) a lack of realism in the case of recourse to techniques of procedural audio.In this dissertation, and in order to respond to those previously raised limitations I present DIGItal sTeps (DIGIT), an interactive digital system to achieve footstep sounds in a sound foley context. DIGIT is based on the concept of the acoustic profile of a gesture produced by the impact of the hand on a surface and captured by a contact microphone. Through the description of the acoustic profile of these movements and their mapping to concatenative synthesis control parameters, it is intended that DIGIT performs an automatic search in a database comprising pre recorded sounds of footsteps, and subsequent choice of the most suitable segment according to the acoustic description of the gesture. DIGIT enhances the expressiveness and operation through the use of gesture as a way of recreating the footstep sounds in an audio post-production environment of a movie or game, giving a follow-up to the traditional practice of sound foley, which through tactile exploration of physical objects, and recognition of its acoustic qualities, adjusts them to the thematic narrative of the moving image.DIGIT has advantages over previous solutions in that it uses the description of the acoustic gesture profile (in its multiple dimensions) as the main system control in a way to circumvent the limitations of the MIDI communication protocol. DIGIT also has a granular audio playback engine that enables a change of the unique features of a sound in order to create more possibilities from a small number of samples. This feature contributes to a decrease in the size of the sound database of prerecorded footstep sounds that comprise the system unlike the solutions presented by sound libraries or plugins.Also a feature of DIGIT system in relation to existing solutions is that it can work by layers in a same instance of the software giving the user the possibility to create a more complex sound

DESIGNING A TAXONOMY FOR VIRTUAL MUSEUMS FOR THE USE OF AVICOM PROFESSIONALS

Full version unavailable due to 3rd party copyright restrictions.This thesis aims to go beyond the concept of so called ‘virtual museums’. In this work I will attempt to trace a new definition of the term ‘virtual museum’ providing the concept with renewed dignity, comparable to ICOM’S definitions of museums and other existing definitions of the concept. To do so the main part of this thesis is about creating a meta­model of taxonomy capable of including all the experimentations that have taken place in the field of ‘virtual museums’ in the last 20 years. In this direction I have investigated the concept of the museum as a medium as described by McLuhan and other thinkers, both within and outside the field of museology. The discovery of an unabridged work by McLuhan on technology in museums endorses, and opens a discussion on how technology is intended to be used for the communication of heritage. Another aim of this thesis is to investigate how museum professionals can deal with the new role of Information Technology in communicating heritage. In this thesis I intend to respond to the need of museum professionals both inside and outside ICOM for definitions and clearer understanding concerning the following questions ‘What is a virtual museum? Can it be comparable with a ‘real’ museum? What different kinds of virtual museums can be discerned in past experimentations? Can they be included in a taxonomy? How does this change the day to day work of museum professionals in accordance with the new technological potential for the communication of heritage

Media activism on and off line by Italian adolescents and young adults

From several researchers it appears that Italian adolescents and young people are grown up with commercial television which is accused to contain too much violence, sex, reality shows, advertising, cartoons which are watched from 1 to 4 hours daily. Adolescents are also great users of mobile phones and spend a lot of time to use it. Their academic results are below the average of Ocse States. However the widespread use of communication technology and social networks display also another side of adolescents who engage in media activism and political movement such as Ammazzateci tutti!, Indymedia, Movimento 5 Stelle, Movimento No Tav. In which way does the world economic crisis -with the specific problems of Italy as the cutting founds for school, academic research and welfare, the corruption of political class, mafia and camorra organisation induce a reaction in our adolescents and young people? Several researches inform us about their use of internet in terms of spending time but, more important, how internet, and the web 2.0, could be an instrument for their reaction? What do they do online? How they do it? Which is the meaning of their presence online? And, has their online activity a continuity offline? The research aims are: 1. Trough a participant observation of Social Network profiles opened by 10 young active citizens, I would seek to understand which kind of social or political activities they engage in online as individuals and which is the meaning of their presence online. 2. To observe and understand if adolescents and young people have a continuity of their socio-political engagement online in offline activities and which kind of experiences it is. 3. Try to comprehend which was (or which were) the significant, learning experiences that convinced them about the potential of the web as tool for their activism.All’origine di questa ricerca ci sono due affermazioni, la prima di Al Gore, premio Nobel per la pace nel 2007, ha affermato che “La primavera araba ha avuto internet come strumento” La seconda è di Jacques Gonnet membro del gruppo di studio francese Clemì: “… [esiste un] legame fondamentale tra democrazia ed educazione ai media: (…). La scuola, luogo d’iniziazione ai valori di una società, si inscrive e si rivela, grazie ai media, in una visione politica del mondo. L’educazione ai media diventa un’iniziazione alle pratiche democratiche, (…)” La dichiarazione di Gore mi ha indotta a riflettere sull’attivismo dei giovani italiani, a fenomeni quali “Ammazzateci tutti!”, “Indymedia”, “Movimento 5 Stelle” “Libera” il Movimento “No Tav” ecc. Quella di J. Gonnet su quali possono essere le esperienze formative relative ai media per i giovani italiani. La scuola? La famiglia? I pari? Tutte queste? Altre? Le domande che guidano questo mio lavoro di ricerca sono state: in quale maniera la crisi del nostro paese induce una reazione nei nostri adolescenti e giovani? La rete quanto è utilizzata per dare forma a queste reazioni? Molte ricerche ci forniscono dati sull’uso che i ragazzi fanno di internet in termini di tempo dedicato e attività svolte ma, come internet può essere uno strumento per il loro impegno? Cosa fanno i ragazzi online? Come? Come la loro attività online si traduce nella continuità offline? Avvalendomi di strumenti come l’osservazione di profili su Social Network, l’osservazione in presenza di eventi quali manifestazioni e dibattiti, l’intervista e il questionario, lo studio della letteratura esistente, tenta di dare una risposta a queste domande

Time, movement, urban space. “Places of transit”: an urban asynchrony?

The aim of the research project is to examine the spatial transformation of “places of transit” for migrants in 21st century Europe. The aim is to analyse the relationship between these places and the contemporary city, and how the latter is being transformed, or should be transformed, taking account - or not - of their existence. Transit places are part of a short timeframe, with a normatively limited duration; their existence, however, has an impact on the morphology of the city over a (more or less) long period. Rather than considering them as “out-of-the-way places” or as “high places” (usually in the media), let’s start from the premise that, like the other elements of the urban jigsaw, they are places that (de) structure, organise and shape the city and the lives of those who live there. Characterised by a specific temporality, one in which exile becomes waiting, these places are “intervals” because they are as much perimeters obeying specific social and spatial dynamics as they are “pockets of time” covering ways of living in this territory that are out of sync or poorly synchronised with the “urban pulse”. The question of time is therefore a key factor in understanding the relationship between these places. Bringing space and time together here amounts to “discovering” what constitutes a mesh, paradoxically not very visible and yet the matrix of urban space, where the nodes are called chronotopes (rhythms), urban continuities (duration) and spatial resistances (memory). This is my line of enquiry. Based on five case studies (Calais, Lampedusa, Lavrio, Lesbos, Amman/Zaatari), this thesis offers a viewpoint on the dynamics that govern the integration, or otherwise, of “places of transit” into a network that extends beyond them both spatially and temporally.Le projet de recherche entend questionner la transformation spatiale des « lieux de transit » pour personnes migrantes dans l’Europe du 21e siècle. Il s’agit d’analyser les relations entre ces lieux et la ville contemporaine, comment cette dernière se transforme ou doit se transformer en tenant compte – ou non – de leur existence. Les « lieux de transit » sont inscrits dans un temps court, à durée « normativement » limitée ; leur existence a toutefois un impact sur la morphologie de la ville selon une (plus ou moins) longue durée. Plutôt que de les considérer ou comme des « hors lieux » ou comme des « hauts lieux » (médiatiques le plus souvent), partons du postulat qu’ils sont, à l’instar des autres éléments du puzzle urbain, des lieux qui (dé)structurent, organisent, nervurent la cité et la vie de ceux qui l’habitent. Caractérisés par une temporalité spécifique, celle où l’exil se fait attente, ces lieux sont « intervalles » car ils sont tout autant des périmètres obéissant à des dynamiques sociales et spatiales particulières que des « poches de temps » qui recouvrent des manières, désynchronisées ou mal synchronisées par rapport à la « pulsation urbaine », de vivre ce territoire. La question du temps est dès lors une entrée déterminante pour lire la relation entre ces lieux. Agréger ici l’espace et le temps revient à « découvrir » ce qui constitue un maillage, paradoxalement peu visible et pourtant matriciel de l’espace urbain où les nœuds se nomment chronotopes (rythmes), continuités urbaines (durée) et résistances spatiales (mémoire). Ceci est ma ligne d’horizon. À partir de cinq cas d’étude (Calais, Lampedusa, Lavrio, Lesbos, Amman/Zaatari), cette thèse propose un point de vue sur les dynamiques qui président à l’intégration, ou non, des « lieux de transit » dans un maillage qui les dépasse et spatialement et temporellement

Claves estéticas de la ciencia ficción cinematográfica (2014-2019): la representación de lo otro a través del diseño de producción audiovisual.

[ES] Esta investigación de Tesis tiene como objetivo principal analizar un determinado corpus de producciones de la ciencia ficción cinematográfica reciente. Estas películas parecen compartir unas claves visuales que, lejos de mantener los parámetros tradicionales del género, abordan otras referencias estéticas relacionadas con lo familiar, lo orgánico y lo nostálgico. La investigación comienza con el planteamiento de los objetivos, el marco teórico y las metodologías de trabajo y avanza con la delimitación del corpus y la justificación del mismo. A continuación se profundiza en el género de la ciencia ficción y en el lenguaje visual como herramienta fundamental para abordar la cultura visual reciente. Gracias a esta exploración, se propone una línea temporal de la evolución estética de la ciencia ficción cinematográfica mediante la que definir las corrientes mayoritarias del género, detectando a su vez los giros y propuestas alternativas más relevantes. Al realizar este ejercicio se localizan una serie de producciones de los años sesenta que, por su discurso en torno a la nostalgia y la subjetividad parecen inspirar al corpus actual. A partir de esa información se aborda el análisis del corpus, desde sus elementos conceptuales y narrativos -relacionados con la estética de la dirty scifi- a su expresión formal, precisada en tres claves definidas: lo natural -en un género que tradicionalmente rechaza lo orgánico-, lo espiritual -en tramas definidas por la razón- y lo físico y lo íntimo -que vuelve la mirada del afuera y del futuro al aquí y al ahora-. Por último, al profundizar en estas claves, se exploran determinados elementos que actúan como figuras simbólicas de cada relato. Elementos referidos en este análisis como artefactos que representan visualmente la propuesta conceptual -o novum- de cada película y que, como construcciones diegéticas, permiten ciertas reflexiones sobre el modo en el que la sociedad que las ha diseñado se relaciona con los conceptos de memoria, nostalgia y futuro. Finalmente, las conclusiones dan cuenta de aquello que -previamente planteado en los objetivos iniciales- ha quedado resuelto, de qué preguntas se han podido responder y de qué otras han surgido a partir del análisis realizado. Como resultado, la investigación propone un recorrido que abarca las generalidades de la ciencia ficción cinematográfica y su cualidad simbólica para detectar un impulso estético que, si bien se intuye en la contracultura de los años sesenta, asume hoy una presencia compartida. Esta perspectiva trata de ofrecer al lector un desarrollo explicativo equivalente al desarrollo investigador, de forma que le resulte posible descubrir esta tendencia y reflexionar acerca de cómo sus motivos visuales podrían evidenciar el carácter de su contexto creador.[CA] Aquesta investigació de Tesi té com a objectiu principal analitzar un determinat corpus de produccions de la ciència-ficció cinematogràfica recent. Aquestes pel·lícules semblen compartir unes claus visuals que, lluny de mantindre els paràmetres tradicionals del gènere, aborden altres referències estètiques relacionades amb allò familiar, orgànic i nostàlgic. La investigació comença amb el plantejament dels objectius, el marc teòric i les metodologies de treball i avança amb la delimitació del corpus i la justificació d'aquest. A continuació s'aprofundeix en el gènere de la ciència-ficció i en el llenguatge visual com a eina fonamental per a abordar la cultura visual recent. Gràcies a aquesta exploració, es proposa una línia temporal de l'evolució estètica de la ciència-ficció cinematogràfica mitjançant la qual definir els corrents majoritaris del gènere, detectant al seu torn els girs i propostes alternatives més rellevants. En realitzar aquest exercici es localitzen una sèrie de produccions dels anys seixanta que, pel seu discurs entorn de la nostàlgia i la subjectivitat semblen inspirar al corpus actual. A partir d'aqueixa informació s'aborda l'anàlisi del corpus, des dels seus elements conceptuals i narratius -relacionats amb l'estètica de la dirty scifi- a la seua expressió formal, precisada en tres claus definides: allò natural -en un gènere que tradicionalment rebutja l'orgànic-, allò espiritual -en trames definides per la raó- i allò físic i íntim -que gira la mirada del fora i del futur a l'ací i a l'ara-. Finalment, en aprofundir en aquestes claus, s'exploren determinats elements que actuen com a figures simbòliques de cada relat. Elements referits en aquesta anàlisi com a artefactes que representen visualment la proposta conceptual -o novum- de cada pel·lícula i que, com a construccions diegétiques, permeten unes certes reflexions sobre la manera en què la societat que les ha dissenyades es relaciona amb els conceptes de memòria, nostàlgia i futur. Finalment, les conclusions donen compte d'allò que -prèviament plantejat en els objectius inicials- ha quedat resolt, de quines preguntes s'han pogut respondre i de quines altres han sorgit a partir de l'anàlisi realitzada. Com a resultat, la investigació proposa un recorregut que abasta les generalitats de la ciència-ficció cinematogràfica i la seua qualitat simbòlica per a detectar un impuls estètic que, si bé s'intueix en la contracultura dels anys seixanta, assumeix hui una presència compartida. Aquesta perspectiva tracta d'oferir al lector un desenvolupament explicatiu equivalent al desenvolupament investigador, de manera que li resulte possible descobrir aquesta tendència i reflexionar sobre com els seus motius visuals podrien evidenciar el caràcter del seu context creador.[EN] To delimit and analyze a certain corpus of productions of recent cinematographic science fiction is the main objective of this Thesis research. These films seem to share some visual keys that, far from maintaining the traditional parameters of the genre, approach other aesthetic related to familiar, organic and nostalgic references. Research begins with the statement of the objectives, the theoretical framework and the working methodologies and it progresses with the delimitation of the corpus and its justification. It then delves into the genre of science fiction and visual language as a fundamental tool for approaching recent visual culture. Thanks to this exploration, a timeline of the aesthetic evolution of cinematic science fiction is proposed to define the main currents of the genre, detecting in turn the most relevant turns and alternative proposals. In carrying out this exercise, a series of productions from the sixties are located which, due to their discourse on nostalgia and subjectivity, seem to inspire the current corpus. Based on this information, the analysis of the corpus is approached, from its conceptual and narrative elements -related to the aesthetics of dirty sci-fi- to its formal expression, specified in three defined keys: natural -in a genre that traditionally rejects the organic-, spiritual -in plots defined by reason- and physical and intimate -that turns the gaze from the outside and the future to here and now-. Finally, by delving deeper into these keys, we explore certain elements that act as symbolic figures in each story. Elements referred to in this analysis as artifacts that visually represent the conceptual proposal -or novum- of each film and that, as diegetic constructions, allow certain reflections on the way in which the society that has designed them relates to the concepts of memory, nostalgia and future. Finally, the conclusions give an account of what -previously raised in the initial objectives- has been resolved, which questions have been answered and which others have arisen from the analysis carried out. As a result, the research proposes a journey that covers the generalities of cinematic science fiction and its symbolic quality in order to detect an aesthetic impulse that, although intuited in the counterculture of the sixties, assumes today a shared presence. This perspective tries to offer the reader an explanatory development equivalent to the research development, so that it is possible to him or her discover this tendency and reflect on how its visual motifs could evidence the character of its creative context.Cuenca García, C. (2022). Claves estéticas de la ciencia ficción cinematográfica (2014-2019): la representación de lo otro a través del diseño de producción audiovisual [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/19024