Sensing and mapping for interactive performance

This paper describes a trans-domain mapping (TDM) framework for translating meaningful activities from one creative domain onto another. The multi-disciplinary framework is designed to facilitate an intuitive and non-intrusive interactive multimedia performance interface that offers the users or performers real-time control of multimedia events using their physical movements. It is intended to be a highly dynamic real-time performance tool, sensing and tracking activities and changes, in order to provide interactive multimedia performances. From a straightforward definition of the TDM framework, this paper reports several implementations and multi-disciplinary collaborative projects using the proposed framework, including a motion and colour-sensitive system, a sensor-based system for triggering musical events, and a distributed multimedia server for audio mapping of a real-time face tracker, and discusses different aspects of mapping strategies in their context. Plausible future directions, developments and exploration with the proposed framework, including stage augmenta tion, virtual and augmented reality, which involve sensing and mapping of physical and non-physical changes onto multimedia control events, are discussed

Interactive intelligence: behaviour-based AI, musical HCI and the Turing Test

The field of behaviour-based artificial intelligence (AI), with its roots in the robotics research of Rodney Brooks, is not predominantly tied to linguistic interaction in the sense of the classic Turing test (or, "imitation game"). Yet, it is worth noting, both are centred on a behavioural model of intelligence. Similarly, there is no intrinsic connection between musical AI and the language-based Turing test, though there have been many attempts to forge connections between them. Nonetheless, there are aspects of musical AI and the Turing test that can be considered in the context of non-language-based interactive environments–-in particular, when dealing with real-time musical AI, especially interactive improvisation software. This paper draws out the threads of intentional agency and human indistinguishability from Turing’s original 1950 characterisation of AI. On the basis of this distinction, it considers different approaches to musical AI. In doing so, it highlights possibilities for non-hierarchical interplay between human and computer agents

Cybernetification II: toward a sixth ecology

Digital infrastructure and learning algorithms, digital industry and automated services, digital society, humanities, and health converge with the Anthropocene. This mix has given birth to extraordinary challenges; for architecture as discipline, in its role as cultural heritage and as materialized co-evolution of man and machine. Despite of re-modelling and re-designing work-flows to integrate data necessary to arrive at a structurally and energetically ‘good’ piece of architecture, or rethinking an IoT (Internet of Things) supply chain management from planning via production to construction of a building or city and its performance monitoring, the digital has impacted our 20th century our long-standing relationship to architecture, technology and nature. This chapter suggests first ideas of a sixth ecology – influenced by Felix Guattari’s “The Three Ecologies” , , Reyner Banham’s “Theory and Design in the First Machine Age” and Benjamin Bratton’s “The Stack” . The Sixth Ecology describes one of and for a dynamic relationality across systems; multi-parametric, functionally adaptable, morphologically changing, cybernetic. One, where man, machine, technology and ‘nature’ merge – or speaking for architecture: designer and computer, builder and robot, planner and construction factory, construction site and biologically grown habitats. In 1958 Gilbert Simondon states that “culture has become a system of defence designed to safeguard man from technics. This is the result of the assumption that technical objects contain no human reality.” , , Vaucason’s 18th century automaton the defecating duck , describes a cornerstone for the beginnings of humanification of mechanical technical objects. The rise of digital technical objects since the 1980s and the disappearance of their atomic materiality in a wireless world describes a step further. Humanification of technology crosses the border of co-evolution and co-existence towards coalescence. My research on the ‘sixth ecology’ and concepts such as ‘netgraft’ and ‘neurotecture’ has been influences by the work conceived computational architecture studio Codes in the Clouds which I ran between 2009 and 2016 at DIA, Dessau International Graduate School of Architecture.Digitale Infrastruktur und Lernalgorithmen, digitale Industrie und automatisierte Dienste, digitale Gesellschaft, Geisteswissenschaften und Gesundheit konvergieren mit dem Anthropozän. Diese Mischung hat außergewöhnliche Herausforderungen hervorgebracht; für die Architektur als Disziplin, in ihrer Rolle als kulturelles Erbe und als materialisierte Ko-Evolution von Mensch und Maschine. Trotz Umgestaltung und Neugestaltung von Arbeitsabläufen zur Integration von Daten, die notwendig sind, um zu einer strukturell und energetisch "guten" Architektur zu gelangen, oder eines IoT (Internet of Things) Supply Chain Managements von der Planung über die Produktion bis zum Bau eines Ob Gebäude oder Stadt und seine Leistungsüberwachung, das Digitale hat unser 20. Jahrhundert unsere langjährige Beziehung zu Architektur, Technologie und Natur geprägt. Dieses Kapitel schlägt erste Ideen einer sechsten Ökologie vor - beeinflusst von Felix Guattaris "The Three Ecologies", Reyner Banhams "Theorie und Design im ersten Maschinenzeitalter" und Benjamin Brattons "The Stack". Die sechste Ökologie beschreibt eine und für eine dynamische Beziehung zwischen Systemen; Multiparametrisch, funktional anpassungsfähig, morphologisch veränderlich, kybernetisch. Eines, in dem Mensch, Maschine, Technologie und "Natur" verschmelzen - oder für Architektur sprechen: Designer und Computer, Baumeister und Roboter, Planer und Baufabrik, Baustelle und biologisch gewachsene Lebensräume. 1958 stellt Gilbert Simondon fest: "Kultur ist zu einem Verteidigungssystem geworden, das den Menschen vor der Technik schützen soll. Dies ist das Ergebnis der Annahme, dass technische Objekte keine menschliche Realität enthalten. ", Vaucasons Automat aus dem 18. Jahrhundert, die entkräftende Ente, beschreibt einen Grundstein für die Anfänge der Vermenschlichung mechanischer technischer Objekte. Der Aufstieg digitaler technischer Objekte seit den 1980er Jahren und das Verschwinden ihrer atomaren Materialität in einer kabellosen Welt beschreibt einen weiteren Schritt. Die Menschwerdung der Technologie überschreitet die Grenze von Koevolution und Koexistenz zur Koaleszenz. Meine Forschung über die 'sechste Ökologie' und Konzepte wie 'Netgraft' und 'Neurotecture' war beeinflusst von der Forschung, die ich in den Jahren 2009 bis 2016 gemeinsam mit meinen Studenten am DIA, Dessau International Graduate School of Architecture, durchgeführt habe

Cybernetics in Music

This thesis examines the use of cybernetics (the science of systems) in music, through the tracing of an obscured history. The author postulates that cybernetic music may be thought of as genera of music in its own right, whose practitioners share a common ontology and set of working practices that distinctly differ from traditional approaches to composing electronic music. Ultimately, this critical examination of cybernetics in music provides the framework for a series of original compositions and the foundation of the further study of cybernetic music

History of Computer Art

A large text presents the history of Computer Art. The history of the artistic uses of computers and computing processes is reconstructed from its beginnings in the fifties to its present state. It points out hypertextual, modular and generative modes to use computing processes in Computer Art and features examples of early developments in media like cybernetic sculptures, video tools, computer graphics and animation (including music videos and demos), video and computer games, pervasive games, reactive installations, virtual reality, evolutionary art and net art. The functions of relevant art works are explained more detailed than is usual in such histories. From October 2011 to December 2012 the chapters have been published successively in German (The English translation started in August 2013 and was completed in June 2014)

After dualism

Prepared for discussion at a conference sponsored by the Gulbenkian Foundation on Challenges to Dominant Modes of Knowledge: Dualism, SUNY Binghamton, 3-4 November 2006.This talk concerns issues of time and dualism in academic theory and the real world. The dualism in question concerns people and things and the habit of thinking of them as distinct ontological orders. In contrast, I explore various projects of ‘ontological theatre’—projects that variously thematise nondualist engagements and stage them in a variety of fields including brain science, the science of complex systems, psychiatry, management, politics, spirituality, the arts, music, architecture, and so on. The examples are largely drawn from the history of cybernetics, and locate the engagements in question as constituted in reciprocal becomings in time

Vector synthesis: a media archaeological investigation into sound-modulated light

Vector Synthesis is a computational art project inspired by theories of media archaeology, by the history of computer and video art, and by the use of discarded and obsolete technologies such as the Cathode Ray Tube monitor. This text explores the military and techno-scientific legacies at the birth of modern computing, and charts attempts by artists of the subsequent two decades to decouple these tools from their destructive origins. Using this history as a basis, the author then describes a media archaeological, real time performance system using audio synthesis and vector graphics display techniques to investigate direct, synesthetic relationships between sound and image. Key to this system, realized in the Pure Data programming environment, is a didactic, open source approach which encourages reuse and modification by other artists within the experimental audiovisual arts community.Holzer, Dere

Modes of Interaction in Computational Architecture

This thesis is an enquiry into the importance and influence of interaction in architecture, the importance of which is observed through different modes of interaction occurring in various aspects of architectural discourse and practice. Interaction is primarily observed through the different use of software within architectural practice and in the construction of buildings, façades and systems. In turn, the kind of influences software has on architecture is one of the underlying questions of this thesis. Four qualities: Concept, Materiality, Digitization and Interactivity, are proposed as a theoretical base for the analysis and assessment of different aspects of computational architecture. These four qualities permeate and connect the diverse areas of research discussed, including architecture, cybernetics, computer science, interaction design and new media studies, which in combination provide the theoretical background. The modalities of computational architecture analysed here are, digital interior spaces, digitized design processes and communicational exterior environments. The analysis is conducted through case studies: The Fun Palace, Generator Project, Water Pavilion, Tower of Winds, Institute du Monde Arabe, The KPN building, Aegis Hyposurface, BIX Façade, Galleria Department Store, Dexia Tower, and also E:cue, Microstation, Auto-Cad, Rhino, Top Solid and GenerativeComponents software. These are important for discussion because they present different architectural concepts and thoughts about interactivity within architecture. The analytical processes used in the research distinguished and refined, eight modes of interaction: (1) interaction as a participatory process; (2) cybernetic mutualism; (3) thematic interaction; (4) human-computer interaction during architectural design production; (5) interaction during digital fabrication; (6) parametric interaction; (7) kinetic interaction with dynamic architectural forms; and (8) interaction with façades. Out of these, cybernetic mutualism is the mode of interaction proposed by this thesis