The history of the development of sound on film offers us lessons for the development of sound and music for interactive artforms. Now that technological developments have enabled almost unrestricted importation of audio into interactive platforms, the time has come for us to ask: is the content of the audio good enough

Deutsch, Stephen

English

Bournemouth University Research Online

  Harnessing the Power of Music and Sound Design in Interactive Media. Stephen Deutsch   Digital Content Creation  Earnshaw, R & Vince, J (eds.)   Springer Verlag, 2001 http://www.springerlink.com    Abstract The history of the development of sound on film offers us lessons for the development of sound and music for interactive artforms. Now that technological developments have enabled almost unrestricted importation of audio into interactive platforms, the time has come for us to ask: is the content of the audio good enough?  Article For the first fifteen or twenty years of its history, cinema produced works which interest us today primarily as historical documents. Earliest movies were astonishing because they moved at all, not particularly for the content embedded in the movement. Whether as the historical record of Lumiere, or the conjuring tricks of Mélies, early film’s real content may have been the technology itself.   It wasn’t until Griffith, Eisenstein (and others) developed a cinematic language which was unique to that form, that artefacts begin to speak to us today. From that point until the arrival of sound in 1928, cinema developed quickly, offering a sophisticated language of image, photography, lighting, montage and gesture by which audiences could read the "content" of film.  The arrival of sound changed everything. It overwhelmed the industry. In 1928 Hollywood produced over 150 silent film feature dramas. In 1931 in produced one (Chaplin’s City Lights). Because of the cumbersome nature of sound equipment, the noisiness of silent film sets, the clatter of the camera, the sound engineer came to dominate production in sound’s early years, even having veto over the director over when to say "cut!". For the audience, the fact that the films spoke at all was the main attraction, it took some time for people really to care what was said, and longer to care what was heard. The interactive media industries are today probably just past this point. Back in around ‘92/93 when the first digital sound entered the games world - albeit with almost no memory, and thus with few options - the fact that a character spoke at all was the issue, not what s/he said. Scriptwriters weren’t needed -yet.  The arrival of sound in film, back in 1928 provides us of a maxim which was true in the early days of silents as well: Those who control the technology of a new medium control its content as well. As the technology spreads, the control of its content dissipates.  To a lesser extent other later technological developments in film determined content; Film, threatened by tv, introduced Cinemascope, which insisted on epics (a chat in a dining room is not really its meat); The IMAX process demands wideshots and little camera movement (fast pans can result in nausea), etc.; 3D needed for things to jump from the screen.  In many respects, interactive moving pictures are at a stage of development analogous to cinema in 1915 or 1928. The technology is still important, and the technologists still rule... but a demand for improved content, in the sense of narrative, dialogue, music and sound is being heard more loudly,   In 1941, Orson Welles almost instantly invented many of our modern notions of cinema sound (he also invented several cinematographic conventions about image and narrative, but that’s for another paper). He was able to do this because he knew very little about how movies were made. This enabled him to disregard the technologically led problems and solutions. He was not the prisoner of the industry’s notions of what was possible.  He did know something about radio, however, and it was with the fusion of radio and cinema that the real "sound film" came into being, thirteen years after the introduction of sound onto moving images.  Back to the maxim: Those who control the technology of a new medium control its content as well. As the technology spreads, the control of its content dissipates. What emerges very strongly is that this transfer of power involves a shift in values as well as processes. Those who follow are less inclined to pay service to the tenets which determined developments prior to their arrival.  The short history of electronic music offers us a model here. Post-war electronic music (we have here to ignore, for convenience, developments in electronic instruments prior to WW2, notwithstanding their contribution) very quickly split into two camps, analogous perhaps to the initial schism in film between the Realists (Lumiere) and the Conjurers (Mélies). These two camps were musique concréte and electronische musik. The first concerned itself with the reorganising of "found sound" (sound recorded in the home, the factory, the street, etc.), and was in many respects reminiscent of its stylistic antecedent, Dada . The second, based upon the notions of technologically led art with a technological aesthetic, held sway intellectually and politically, and was a dominant form of avant garde music during the period 1950-75. To make such music, these practitioners needed to be totally au fait with the technology, which initially consisted of oscillators, filters, ring modulators, reverb units and tape recorders. The music was predisposed (both technologically and aesthetically) away from tonality and harmony. It was directed towards structuralistic and colouristic systems of composition which most untutored listeners identified with Sci-Fi. This identification was solidified by the feature film, Forbidden Planet (1956). Electronic music studios were set up at great expense and run by these composers, notably Karlheinz Stockhausen (WDR) in Cologne and Pierre Boulez in Paris (IRCAM). It became a priesthood of a new aesthetic, blown out of the water in about 1980 by the availability of the DX7 FM synthesizer and the portastudio. With these tools anyone could make electronic music and determine the content for themselves.  Interactive moving pictures - games -. sound and music. In some respects sound is being competently used at present, if not as imaginatively as one might expect. That sounds can be attached as objects to movements, gestures, events and locations is well developed, as is the process of altering those sounds as a result in changes to environment and perspective. The model is of a "filmic’ reality (which many people mistake for actual reality), a reality which seems appropriate to the user. But the use of sound here is often too literal, sound effects rather than sound design - a pale imitation of the sound design used in film. The continuing weakness in this area is due to the unnecessary constraints in the production process, based upon old methods of operation. These constraints place sound near the end of the chain of production.   This is why any systems which enable scripting of the sound to lie in the hands of a sound designer are highly significant — together with a wider recognition of the power and importance of sound within the rest of the development team. It is also essential that sound be placed near the beginning of the planning process.  But music offers us a different set of issues.  Very many users of games software, given the choice, turn the music off. It would be a wise move for games manufacturers to commission a survey as to how many users actually do turn the music off, or would do so given the opportunity. That music is given such low a priority by so many might prompt developers to ask questions about what the music is actually for, but few do - except in the most rudimentary way.  There are two types of music for moving pictures: diegetic music, music which is part of the action (the characters in the film are meant to be able to hear it), and (not surprisingly) non-diegetic music, music which is in the background, part of the film making process, similar to the editing, lighting, camera movement, etc. The first type is not really an issue here. This second type is worth considering with particular reference to interactive moving pictures.  Non-diegetic music has a number of different functions, not always in the background. In most film, the appearance of any music at all is a signifier of emotional content. When the music comes on, we are told that the scene is invested with emotional significance and is different from those scenes which contain no music. In most films, this effect operates regardless of the kind of music used.   The most common type of this kind of music is what one can call "narrative support music". It tells the audience things the director wishes them to know. Sometimes the signals can be crude (and most films used such before the 60s); in Spellbound (Hitchcock, 1945) the music tells us that Ingrid Bergman will fall in Love with Gregory Peck the first time she lays eyes on him (the lighting helps us here as well). Many games continue use this type of musical semiology. More sophisticated films, which have scripts allowing for ambiguity of motive and character, allow the music to aid this ambivalence. Narrative support music accounts for about 80% of all moving picture music. It is effective and done well, almost inaudible to the conscious ear. The trend interactive media is toward this approach. But there is a problem.  Non-diegetic narrative support music is composed to the linearity of the film. It is so constructed so as to provide a vector to the scene it accompanies, guiding us along a path which culminates or dissipates at a pre-determined time. Most film music is composed to picture, with the composer using SMPTE technology to lock the music to the events on the screen. Linearity is so crucial to the process that to transfer this process to a non-linear medium, is fraught with difficulties. Most games music which attempts to mimic film music is unconvincing because of this difficulty. Some composers have used loops (of varying lengths) as a way of filling the space, but such techniques are incapable of proving a convincing vector to an environment because of the indeterminacy of the duration. My own compositional research is suggesting that one can create such music, but the compositional processes are difficult, not technologically, but musically, and without wishing to insult anyone here, beyond the imagination and technique of most composers, including those engaged in popular, film and concert music.  There are other types of non-diegetic music. The second type of music can be called "ironic", where the music and the images tell us different stories, often causing us to distance ourselves from the lives of the characters on screen. This is a very cinematographic gestural language and its imminent widespread use in interactive environments, especially those which are ‘goal oriented’ (games) seems unlikely.  A third use of such music is "iconic" or "referential". We are most of us familiar with the use of Richard Strauss’ Also Sprach Zarathustra as an iconic metaphor for space as a result of its use in Kubrick’s 2001, despite the fact that the piece was composed about 60 years before the first manned journey into space. Interactive Media uses this technique, especially in games (the love theme from Tchaikovsky’s Romeo & Juliet comes to mind when love is in the mix), but these fragments become stale when listened to on repeated playing.   We need here also to remember that a film is likely to be seen by most people a few times at most, and those viewings separated by some considerable time. This interval allows the audience to forget the music, or at least not be over-familiar with it when next the film is seen (exception here the music for David Lean’s Dr. Zhivago and Lawrence of Arabia both of which allow some of us to become sick of the music long before the end credits roll). An interactive media package such as a game will be visited many times, especially at the lower levels, making most music intrusive and boring at best. What is needed is a musical language which becomes so integrated into the atmosphere of the game, that its disappearance would be seen by the user as detrimental to the game-play.  Another way music can be used is irrelevantly. Jaque Tati is the master of this technique in such films as Mr. Hulot’s Holiday and Mon Oncle. Here the music takes the place of the atmos track. Several CDRoms, especially those for children (Broederbond comes to mind) use this technique, but like "iconic" music such music in an interactive environment is likely to bore.  The last way is a combination of the "narrative support" and "iconic". This is the cartoon technique, where every action has a musical sound, and the score follows exactly what we see on screen. Most of us have spent sufficient time with Warner Bros. and Tom & Jerry for us not to dwell too long here. Some games use this effect, but it wears thin quickly. This isn’t used much in games, but is fairly common in children’s CD-ROMs.  New artforms and new inventions usually mimic the forms available at the time of their invention. The first automobiles did look like "horseless carriages"; the first electric light fittings resembled candles, oil-lamp or gaslight fixtures, our current computers are a hybrid between the typewriter and television.   Similarly, the content of the new technological artforms often mimic earlier genres. Early films were theatrical performances played to an unmoving camera; recordings were souvenirs of performances, trying to capture (in classical music, at least) the acoustic world of the best seat in the concert hall; and early television was radio with pictures. In most cases (classical music being an interesting exception), eventually the form begins to influence the content. It is at this time when "outsiders" are most likely to make an impact. Their imagination propels the artform into new directions because they can see (often more clearly than the practitioners) where the new form might go, and, they are unencumbered by notions of what is "possible".  Most of today’s practitioners of interactive media music and sound grew up in the industry. Many were amateur musicians who had a talent for computing. Early technology in this area allowed for only the most rudimentary musical gestures, produced by the computer equivalent of the "Stylophone". In such a work, small looped patterns a la PAC Man were the norm. As the technology developed, more space and power has been allocated to sound and music, and just lately, many "in house" composers simply do not have the skills (and perhaps, alas, the musical talent) to take this exciting form further. Composing for film (and there aren’t even that many excellent film composers) requires skills and talent analogous to driving a racing car; few can do it. Composing for Interactive Media requires skills and talent analogous to flying a space shuttle. The next generation of composers will be required to function as sound designers as well; to be able seamlessly to mix sounds and musical gestures into a far more meaningful soundscape than even the best product can now offer.    From  Earnshaw, R and Vince, J (eds) Digital Content Creation.  Springer 2001 All Rights Reserved 

Harnessing the Power of Music & Sound Design in Interactive Media

Abstract

Similar works

Full text

Available Versions

Bournemouth University Research Online