Pioneers on the Ludomusical Frontier: Sound Art and Spectromorphology in Games   1 comment

I recently presented a paper at the 2014 Ludomusicology Conference, which this year was held at Chichester University in Chichester, UK. I’m presenting it here on this blog for all to read who couldn’t make it or otherwise are interested in reading it.

After finishing the conference, I now realize that I attempted to tie together a lot of topics and ideas, somewhat less successfully than I initially thought, and definitely barely touched upon some points. Still, I’m happy with the ideas and arguments that I brought to the table here. Please comment if you’d like any clarification or have any questions or comments.

For further reading, I highly recommend reading Denis Smalley’s seminal paper “Spectromorphology: explaining sound-shapes”.


In a time when the video game community still struggles to convince a significant portion of the population of the lasting artistic importance and vitality of video games, video game music also often struggles to be taken seriously in academic circles. While video game music has come a long way since its inception in arcade machines and simple 8-bit chiptunes, composers and sound designers tend to underutilize certain technologies and techniques; were they to be incorporated more comprehensively, the results could mean a new chapter in game music and sound.

One concept of particular interest involves blending the interactive diegetic sound world of games with non-diegetic background music in ways that blur the lines between these two often-discrete ideas in a way that only interactive media can accomplish. Take the Silent Hill series, for example. The radio in the first few Silent Hill games acts as a sort of sonar, amplifying in volume and intensity, warning the player as monsters approach (while also occasionally serving some narrative purposes, especially in Silent Hill 2). Whereas some games (FTL and Mass Effect to name two) will transition to an alternate battle version of an area’s music (and Silent Hill does this at times), Silent Hill uses the diegetic sound of the radio to serve this purpose and raise the tension while also combining with, and contributing to, the surreal industrial electronic aesthetic and atmosphere of the music proper.

(see: 4:27)

To consider a more specific example, at one point early on in Silent Hill 2 the player enters an apartment building and has to solve a puzzle involving unlocking the face of a grandfather clock. As the player enters a separate room in the apartment, after having heard virtually no sounds other than the character’s footsteps, the occasional dialog, run-ins with monsters, and the associated radio, a strange, regularly-pulsed, mechanical sound (even somewhat clock-like…) begins to fade into the texture, hitting its peak as the character reaches into a mysterious hole in the wall to retrieve the key to the clock, during which the sound ceases. As soon as the sequence ends, the sound resumes, before fading away when the player leaves the room.

Moments like this serve to disorient the player subtly. The sounds are not actually emanating from any in-game objects, so they are not diegetic sounds per se, yet they do correspond to a specific location in the game that has a clear narrative connection to what may be considered a sort of source of those sounds. Especially since the player initially doesn’t know what they’ll find in this gross and rather frightening hole in the wall, they may not even be consciously aware of the connection between the disturbing sounds they’ve begun hearing, the location they’ve arrived at, and the puzzle item they’re about to uncover.

Similarly to Silent Hill, Rich Vreeland’s game January puts forth an interesting thesis on diegetic sounds and music, but through a markedly different aesthetic and with the accompanying notion of composing through coding. January is a simple game in which the player controls a man walking outside in the snow, catching snowflakes in his mouth. Each snowflake triggers a note or chord, with certain snowflakes of varying shapes shifting the chord, scale, or tonal structure. The result is peaceful, eloquent, and serenely surreal. As a composer, Vreeland supplies the overarching structure through which the player performs and, to an extent, composes; the player can choose to avoid snowflakes that might change the tonality if they are appreciating the current scale, or seek out one of the scale-changing snowflakes if they want some variety. Whereas most musical performance requires a level of skill on the part of the player, in January, even non-musical players can be part of the musical performance.

January’s blurring of the diegetic boundary is rather similar to that of Silent Hill, despite the markedly different tone of the games. Whereas Silent Hill builds a horrific and frightening tension through its use of disorienting, ambiguous sound sources, January creates a decidedly more peaceful, surreal, almost childlike environment through its use of pleasant, simple waveforms organized in a rather simple tonal structure. Still, January, too, uses sounds that clearly do not literally emanate from the world objects with which they are associated and by which they are triggered (snowflakes do not make synthy diatonic tones when they hit one’s tongue).

As Vreeland stated in a blog post about January, “Using code in games to actually create music, note by note, on the fly, is an underutilized idea that has potential to add huge value to games that aren’t even focused on music, by offering an extra sense of authorship to the player’s experience.” Live generated and/or processed sounds are already widely utilized in sound art installations and concert pieces using software like Cycling ’74’s Max, with varying degrees of interactivity. The potential to do this with games is certainly there now and, as Rich Vreeland mentioned, underutilized at this time.

Obviously, not all composers can or should be expected to have the requisite knowledge of computer programming languages, middleware, and other general audio implementation expertise required to make a soundscape of music and sound design that is so tightly intertwined through direct coding. However, even just fostering a basic understanding of the systems being used to implement sound and working closely with developers, programmers, sound designers, and/or audio implementers to work out ways to implement sounds in unique and effective ways can have a drastic effect on the quality and vitality of a game’s sound world. Having an ongoing dialogue between team members can foster a back-and-forth creativity and inspire new creative solutions that might never have occurred to either party on their own. Composer Austin Wintory has commented on his work with sound designer Steve Johnson in the game Journey, in which the musical tones used by players to communicate with each other and trigger certain mechanics were created through close collaboration between team members to ensure a unified soundscape across the game’s world.

When the composer acts as the creator of both the music and the entire game world’s sound design, even more of this enormous potential can be seen, as the composer becomes the creator of the entire sound art composition that accompanies and integrates with the game. Take Playdead’s Limbo as an example. Martin Stig Andersen’s score and sound design in Limbo demonstrates what an awareness of different levels of diegetic sound can make.

(see: 4:19)

Although the sound in Limbo consists largely of ambient background noise and environmental elements, its subtle music enters at key moments in the narrative and puzzles. One such moment occurs early on, when a group of hostile figures begin chasing the player character in the direction from which he had just come, through the traps he had painstakingly avoided. When the player approaches the beings, a subtly dissonant droning tone enters the soundscape, and further sounds enter as the chase intensifies. As his adversaries succumb to the traps, the music is accented by a resolution of the drone and the emergence of a new prolongation of the resolving chord, which continues as the player proceeds to the next puzzle and eventually fades away into the diegetic mechanical sounds of said puzzle.

Even the sounds themselves, the levels of individual sounds, and the ways they blend together into the whole of the world suggest a unity that would be exceedingly more difficult to accomplish were it not the work of a single person. The more musical sounds, quite often single droning notes or simple chords, often seem to be emerging from and coalescing in the environment of the game itself rather than outside of it as most game music does. The blurring of the diegetic line is again effective in fostering an interesting, dynamic world that subtly frames the narrative in a way that doesn’t manipulate the emotions of the player so brazenly.

An awareness of spectromorphology can add even more interest to a game’s overall sound world. Spectromorphology is a method and set of vocabulary put forth by Denis Smalley for hearing, analyzing, and understanding music that is not limited to traditional Western common-practice musical ideas of tonality, rhythm, and instrumentation. It is primarily focused on, but not strictly limited to, electroacoustic music and sound art. The word spectromorphology itself is rather self-explanatory: spectro-, referring to frequency spectrum, and -morphology, referring to the way this spectrum changes (or does not change) over time. So put quite simply, it is the way sounds change over time. As just one example of the extensive language Smalley develops, he describes the three structural functions of sound events: onsets, continuants, and terminations.

(from Smalley’s “Spectromorphology: explaining sound-shapes”)

The language that Smalley uses is very visual and spatial by necessity, applying physical descriptors to abstract aural concepts. “Emergence,” “passage,” “arrival,” “disappearance,” “streaming,” “flocking,” “motion and growth processes,” “spectral space”… even at a most basic level, this language of visualization and spatialization makes it a perfect candidate to be applied to multimedia, and hence to video games.

As Smalley has pointed out and Manuella Blackburn has elaborated on, although the language was developed from a listener’s perspective to aid in a multi-layered understanding of what is being heard, knowledge of this language can inform composition, playing into or against expectations and common tropes. Martin Stig Andersen’s sounds in Limbo often tend to follow this pattern of spectromorphological attention to detail. The previous example from the game, even without the context of its game, can stand alone when the various elements of sound design and musical elements come together. Within the context of the game, they create something new entirely. In interviews, Andersen has expressed an awareness of what he calls the “seamless transitions between realism and abstraction.” He states further:

“What I found interesting in relation to audiovisual media was that soundscape and acousmatic music together embraces the entire continuum between representational and abstract sound, in this way dismissing the traditional dividing line between sound design and music.”

Similarly, in Denis Smalley’s paper on spectromorphology, he discusses intrinsic-extrinsic relations in sounds, referring to what he calls source identification in the context of extrinsic qualities; i.e., what Andersen calls representational sound. Spectromorphology encapsulates these ideas and more to create an all-encompassing methodology for hearing and analyzing music that doesn’t fit into the traditional molds of Western music theory and notation. This theory has not traditionally been applied to music and sound for video games, but it can be used quite effectively.

Furthermore, the player’s role in this adds another interesting layer. An analogy could be made to music for the concert hall. In the same way that most musical performances in the concert hall require performers to bring the music to life, games require the player for them to be experienced. Concert hall music is performed primarily for the enjoyment of the audience, while games are to be enjoyed by the player. In games, the player, therefore, serves the role of both the performer and the audience.

In this way, games as a medium (together with their sound and music) have something in common with interactive sound art installations, although they do differ in a few ways. For one, video games, especially narrative ones, generally take place across a linear path. In this way, video games are more similar to traditional music than installations. Vreeland’s January is an obvious exception to this rule; it truly is an installation of a game, an eloquent set piece that serves a singular artistic function.

Interactivity provides both a challenge and an opportunity to create a unique work of art from a musical and general developmental perspective. To not take advantage of the tools, techniques, and opportunities at hand is to miss out on a significant path of artistic potential. Games with more traditional static background music separated from a distinct layer of diegetic sound effects certainly have their place and will continue to well into the future. Not every game has to tread onto this newer path. However, the potential road that games can take goes far beyond the restraints of genre tropes and questions of artistic value.

Blurring the line of diegetic sound, creating music through direct coding and/or close collaboration with programmers implementing sound, and application of spectromorphological thought processes can all be taken to even higher levels than have been demonstrated here and used to create games and game music that push through current trends of the medium that treat music as a separate entity from the game and its sound world. When more abstract options are considered, the possibilities are expanded even further. Games like the recent Polyfauna, from Radiohead and developer Universal Everything, begin to go down this path of purer abstraction, but again, it can be much further explored and expanded. As Igor Stravinsky has said, “music, by its very nature, is essentially powerless to express anything at all.”

Contemporary art music is at a crossroads. Although the idea of the relevance and livelihood of art music being on the line is certainly nothing new, that doesn’t make the idea any less relevant today. Video games don’t suffer from this problem; games, especially experimental indie games, are doing better than ever commercially and popularly. Game audiences, especially in the budding independent game community, are open to new and exciting experiences that contemporary art and music so often present. If more composers from the world of contemporary art music, especially those with experience in the (to some) alien world of electroacoustic music and the technical know-how and coding that often goes along with that, were to join game development teams, challenging contemporary art music would benefit from a greater audience, and games and game music would benefit from taking a step closer to achieving their artistic potential and perhaps begin to be taken more seriously by a greater section of both the scholarly and general communities.


Posted 12 April 2014 by jonbash in Games, Music

One response to “Pioneers on the Ludomusical Frontier: Sound Art and Spectromorphology in Games

Subscribe to comments with RSS.

  1. Pingback: What Makes the Score to "Mad Max: Fury Road" Work - Designing Music NOW

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: