The poster was presented at the SIGGRAPH '16 Special Interest Group on Computer Graphics and Interactive Techniques Conference, Anaheim, CA — July 24 - 28, 2016.

We describe a digital design process that interfaces real-time data with 3D modelling and 3D printing techniques. Digital Impressionism is a platform for exploring new material possibilities, by 3D modelling physical and digital objects, affected by the invisible forces that act upon them, using a 3D pointcloud as a medium. We describe an experimental project run with our MA and PhD students, incorporating real-time audio data to manipulate 3D physical hybrid forms

Ferrarello L.

Walker K.

Laura Ferrarello

Kevin Walker

Crossref

The form of sound through hybrid materials

The poster was presented at the SIGGRAPH '16 Special Interest Group on Computer Graphics and Interactive Techniques Conference, Anaheim, CA — July 24 - 28, 2016. We describe a digital design process that interfaces real-time data with 3D modelling and 3D printing techniques. Digital Impressionism is a platform for exploring new material possibilities, by 3D modelling physical and digital objects, affected by the invisible forces that act upon them, using a 3D pointcloud as a medium. We describe an experimental project run with our MA and PhD students, incorporating real-time audio data to manipulate 3D physical hybrid forms

Ferrarello, Laura

Walker, Kevin

Royal College of Art Research Repository

The Form of Sound through Hybrid MaterialsLaura Ferrarello, Kevin Walker*Visiting Lecturer, Head of the Programme Information Experience Design, Royal of Art Kensington Gore,Figure 1: Danila Luppino, pointcloud modeled through sound (Left).Kotaro Abe, manipulation of pointcloud through sound (Last on right). AbstractWe describe a digital design process that interfaces real-time data with 3D modeling and 3D printing techniques. Digital Impressionism is a platform that explores new material possibilities, by 3D modelling physical and digital objects, as affected by the invisible forces that act upon them. Using a 3D pointcloud as a medium, we describe an experimental project run with our students in our programme incorporating real-time audio data to manipulate 3D physical forms, resulting in new static and dynamic shapes with what we call a hybrid materiality. The modeling design platform of Digital Impressionism sees materials as composites, which substance becomes physical through the digital interference the environment applies to digital forms. Through dynamic modeling processes we describe how data enables a new hybrid tectonic made of composite shapes and materials. This poster introduces the project, discusses related work, describes our methodology and results, then describes the implications and next steps.Keywords: 3D printing, Data, Hybrid Materiality, PointcloudConcepts: • Cross-computing tools and techniques; Design; performance; Experimentation;1   The Tectonic of Digital 3D Modeling via Environmental DataBig data constitutes the intangible landscape we dwell everyday. Likewise many other fields design employs data as guideline that helps shaping digital forms. In addition sensors reflect digital behavior to physical space to implement the physical environment of digital information [1]. The “material language” the crosses both realms - physical and digital - is the pointcloud, for its capability of transforming intangible information in tangible ones through 3D coordinates.    Indeed Digital Impressionism employs pointclouds to give intangible data digital materiality, which substance takes form through the process of 3D scanning, digital modeling and 3D printing. In this poster we describe the process through which environmental sound data affect, and infect, digital forms. The process of design and making creates hybrid tectonics, i.e. the architecture of hybrid materials and forms. In the example illustrated in this poster we used ambient sound to 3D model pointcloud. Through 3D digital design process we interfaced natural entropic systems with its digital simulations. Under the supervision of human creativity we aim to enable responsive and interactive tectonics that reads ambient data – sound in this particular case – in digital space. 2   Digital ImpressionismThe Digital Impressionism project began in 2015 with the aims to return the artist’s hand back to 3D modeling, and to integrate the design process with a continuous flow of live data to inform the creation of form, by focusing on materials and the tectonic process of making. The design platform aims to form hybrid materials via the human perception of materiality extended to the digital space. The project looked at the coding literature currently available in digital design, which mainly employs the keyboard as interface; engineering software like Autodesk Ecotect, which allows to analyse the environment but it doesn’t leave any room for interactivity; voxel 3D printing for its possibility of understanding materials as made of units. 2.1   Interactive Voxel Tectonics    *e-mail:{laura.ferrarello, kevin.walker}@rca.ac.uk.Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear  this  notice  and  the  full  citation  on  the  first  page.  Copyrights  for components  of  this  work  owned  by  others  than  the  author(s)  must  be honored.  Abstracting  with  credit  is  permitted.  To  copy  otherwise,  or republish,  to  post  on  servers  or  to  redistribute  to  lists,  requires  prior specific  permission  and/or  a  fee.  Request  permissions  from permissions@acm.org.  ©  2016  Copyright  held  by  the  owner/author(s). Publication rights licensed to ACM.SIGGRAPH 2016 Posters, July 24-28, 2016, Anaheim, CAISBN: 978-1-4503-ABCD-E/16/07DOI: http://doi.acm.org/10.1145/9999997.9999999Within the context of the Information Experience Design, a programme launched in 2012 with the intention to integrate information with experience through design [2,3,4,5], we set up an experimental lab to test how the physical world can be real  time interfaced with the digital one through the environment. Human creativity and perception of materiality drives this process; interactive Pointcloud is the medium that enables “entropic collaboration” between human and machine [6], as it translates and extends human tactile perception of material in digital form. In order to translate hybrid materiality in physical form we are investigating into voxel materials and voxel 3D printing, which allow combining different kinds of units (voxels) in a single composite shape. Similarly, by considering data as voxel units, Digital Impressionism enables new (hybrid) materials, which are assembled between the physical and the real under the supervision of human perception of physical materiality. We are aiming to open new design territories in art, moving images. We consider our relation with humans and machines as an oxymoron, which juxtaposes the contradictory processes of flattening (digital physicality) versus depth (physical materiality) [7]. 2.2   Process and MethodologiesFigure 2: Sound waves video Virginia Koutla captured and distorted by environmental sound (Left). 3D printed banana distorted by sound waves, by Francisco Norris (Right). We began with project [8] with Microsoft Kinect depth camera, which scannes an existing real-world object (a vase in this case); the resulting pointcloud was imported into 3D modelling program Rhinoceros via its algorithmic editing plug-in Grasshopper. Real-time data on the vase ambient lighting conditions was exported as triangulated mesh (vase.OBJ) and imported in ZBrush. The mesh went under human digital sculpting, texturing and colouring, before 3D printing and spray painting the resulting transformed vase as a new, hybrid object. We found that Grasshopper does not deal with real-time 3D pointcloud data very well when asked to create a mesh; Grasshopper, indeed, required substantial manual input for mesh construction and repair. We thus switched to Processing. As an initial dataset, the previously scanned vase was used. The pointcloud vase (vase.CSV) was modified by real-time audio transmitted by the computer microphone to Processing via the audio Minim library. This phase of work took place over three months, involving five MA students and one MPhil student, all from different backgrounds, to experiment with the system. Using the OpenKinect library to import pointcloud data using the Kinect, Processing was used to generate 3D objects via the OpenGL graphics library, which were then exported in .OBJ format, along with a .CSV file of 3D locations, which was then imported into CloudCompare software to be turned into a mesh for 3D printing. The process was kept open as a way of thinking through making. Second-year MA student Joanne Harik began with the pointcloud data of the vase; she imported the sound of breaking glass into Processing and applied this to the .CSV file. The result was a sonic shattering of the form, mirroring the real-world phenomenon of sound waves shattering glass. First-year MA student Virna Koutla worked with sound input only, experimenting with staccato, percussive sounds versus sustained vocal sounds, then manipulating the resulting waveforms. (The animation can be seen at http://bit.ly/1ZF5XUc.). First-year MA student Francisco Norris scanned a banana for its simple shape and flat colour. He then printed a version of the banana distorted by the audio data. MPhil student Caroline Yan Zheng is researching tangibility and interactivity in relation to human emotions, coming from a background in fashion. In this project she experimented with giving perceivable qualities to emotions. She first set the Kinect to record the position and movements of her own body, then introduced recorded sounds of human emotions – laughing, hailing, quarreling, whispering and crying – to observe how the various frequencies of each sound shifted the points representing her body in different directions. Ker Siang Yeo took a more took a more narrative approach; he used the sound of hummingbirds making the nest as modeling parameter of the pointcloud.	 3   Dynamic Design Processes as Drivers of Sustainable Eco-SystemsAs experimental project we are testing and evaluating process and results. For instance Processing is not a comprehensive 3D modelling program, although the libraries gave us the flexibility of navigating different kinds of information through different interfaces. Kinect, with a pair of low-resolution infrared depth cameras, also results in messy pointcloud data. We need to test the voxel 3D printing to give shape to hybrid materials. The project can expand in many different directions: environmental data can be interpolated in creative and artistic practice that employs sound and moving image interfaces (AR headsets, sensors, drones, etc). Digital Impressionism aims to embody technology in human thinking. Technology is understood as medium that establishes creative and sustainable relationships between humans and machines. By embracing the physical and digital environment Digital Impressionism aims to establish a flexible, feasible and complex domain, which sees information implemented when traveling along the creative journey. We envision dynamic sustainable processes that create eco-systems via tangible media. Because of the interactive and dynamic components moving image is not employed as documenting media, but as dynamic and creative tool. We draw inspiration from the early Silicon Valley spirit of hacking, deconstructing and constructing our own machines; the difference now is that we can now draw on and collaborate with existing digital tools and artificial intelligence. We are thus looking at new multi-material 3D printers and 3D bio-printers, which work dynamically though digital reality headsets. Indeed for us forms are dynamic composite systems that performe in relation to the surrounding conditions. Via Digital Impressionism we aim to challenge intelligent behaviour, which includes sensors or smart materials.  Digital Impressionism incorporates traditional artistic methods with a critical perspective, hacking current machines, algorithms and data in order to improve our collective knowledge, in the spirit of early hackers [9][10]. A key aspect of what makes us human is the ability to make. Indeed the human factor has a pivotal value in the project: human emotions are vehicle to tangibility highlights, like identity and subjectivity.  Our decision-making process operates through the value and meanings our being projects to our personal, social and physical contexts [11]. AcknowledgementsThanks to IED for giving the project the space to operate. I would like to thank IED students Francisco Norris, Virginia Koutla, Ker Siang Yao, Caroline Yan Zheng, Joanne Harik, Danila Luppino, Kotaro Abe, Kim Yip Tong and Kelly Spanou. I would like to thank Juan Pablo de La Vega for helping smoothing out the technical aspect of the project and Michael Pecirno for the help at the early stage.  References1. HORAN, T. A. 2001. The Paradox of Place. In Communications of the ACM 44(3): 58-60.2. WALKER, K. AND CRAFT, B. 2013. Data, representation and aesthetics. In ZHOU, C. (ED.) Information in Style . Beijing: ADPP.3. WALKER, K., HOGG, S. AND DI DUCA, D. 2013. Mechanics of wonder: scripting visitor experiences with technology . NODEM conference, Dec 2013, Stockholm.4. WALKER, K. AND CLAISSE, C. 2014. Big Dada: From visualisation to experience. Death of the Desktop workshop. IEEE VIS 2014.5. WALKER, K. AND FASS, J. 2015. De-Computation: Programming the world through design, NORDES 2015: DESIGN ECOLOGIES , ISSN 1604-9705. Stockholm, June 2015.6. FERRARELLO, L. AND FAIRBROTHER, W. 2015. Material Oxymorons. Tangible Means: Experiential knowledge through materials. In Design Research Society conference, Kolding, DK, 25-26 Nov 2015.7. FERRARELLO, L. 2016. The oxymoron of touch: Tactile perception of hybrid reality through material feedback. In BROADHURST, S. AND PRICE, S. (EDS.). Digital Bodies: Creativity and Technology in the Arts and Humanities. Palgrave, in press.8. FERRARELLO, L., PECIRNO, M. AND SPANOU, K. 2015. Digital Impressionism. Proceedings of DADA: Digital Architecture Design Association conference, Shanghai, Jul 2015.9. WALKER, K. 2012. Hackers and Slackers: The New York New Media Underground in the Early 1990's. (Amazon CreateSpace)10. LEVY, S. 1984. Hackers: Heroes of the Computer Revolution. Doubleday.11. WALKER, K. 2010 Designing for meaning making in museums: Visitor-constructed trails using mobile digital technologies . PhD thesis, University of London.

The form of sound through hybrid materials

Abstract

Similar works

Full text

Available Versions

Crossref

Royal College of Art Research Repository