The Incredible World of Sound Design for Video Games

Of all the aspects of video game production, sound design is the one that takes my imagination further away. This happens because I know almost nothing about what happens in the magic sound designers and producers do in our beloved industry.

To better understand everything this world has to offer, I had the chance to talk with Pablo Schwilden Diaz, CTO at Demute, about his job and my curiosities about this topic.

So, let’s dive in.

First,

General Overview

What is sound design for video games? Does it sound different from other sound designs? (pun intended). In this first part of the interview I tried to make Pablo explain to me the skeleton of sound design for gaming, the ins and outs, and for the first question, I wanted to know about the fundamental aspects of it to create immersive experiences for players. Pablo said that “the most essential is to always try to look at things from the player’s perspective. It’s easy to get lost in the technicalities of game sound design and lose track of the bigger picture (…) To try and find the best experience, I always think back to the goals of video game audio such as Information, Immersion, and Interest. Any sound created has to fit at least one of those goals. Does the sound convey any information to the player, and if so does it convey the correct information? Does the sound help immerse the player in the experience or does it take them out of it? Does the sound create interest and curiosity in the player or does it bother and confuses the player? If a sound doesn’t work towards these goals, then it must be scrapped and started again.

So, here you have it folks, information, immersion, and interest! Sounds incredibly reasonable to me! I mean, who doesn’t recall that scare jump playing a horror game or the angelic music when you start to face another boss arena on a map or the music that slowly builds to keep you interested in a quest! Well, but if it wasn’t for Pablo I wouldn’t put the pieces together so it’s time to keep exploring this world.

Let’s go to,

Sound Effects and Foley

First things first, time to understand exactly what is the process behind the creation of sound for video games, does one start by the score, or by the ambient sounds? There is so much to do that I can’t possibly guess what is what. So, let the expert explain. Pablo says “The first step is to conceptualize the sound. That might sound obvious but many people forget the importance of this step and regret it later. This includes gathering some basic information such as: what the sound represents, what context will it play in, what’s the goal of the sound, etc. The more info we have here and the more decisions we can already make, the easier the rest of production will be.”

Of course! Prepare and plan first. How could I forget this part…

But it doesn’t stop there. The next steps are, as Pablo explains, “Then we wait. Seriously, it might sound weird, but more often than not, it is important for us to have a finished animation, VFX, or level to be able to create the best sound for it (…) Depending on the production and the amount of work, that waiting period is the perfect time to prepare material. Let’s say that we are working on a new creature, and we know from the concept phase that we will need a lot of material from a specific animal (let’s say a llama). While the modelers and animators work on that creature, we can go look for llama recordings or create them ourselves. This way, once we start sound designing, we already have all the necessary materials and are ready to work!

Once we have prepared this material and the necessary content is completed, we create the sounds needed. This will usually be done inside a DAW (Digital Audio Workstation), such as Reaper, by combining, processing, and editing different audio sources (recordings, synthesizers, etc) into one sound. When we are satisfied with the sounds we built, we export them as individual files following the format conventions decided during the concept phase.

After that comes the implementation phase. This will vary greatly depending on the tools and game, but the idea generally stays the same. Implementation is the step where we will define how a sound should play. Things like volume, reverb, looping, and the number of sounds, are all defined here. We are kind of setting the “rules” of how the sound should react to the player’s actions. Sometimes, this can be very easy (think a menu button sound) or very complex (full dynamic mixing system with parameters and runtime rules for reverberation). More often than not, implementation is not just set and forgotten. We usually come back to this step quite often in the production and sometimes we work on individual sounds, sometimes on bigger groups of sounds that should react similarly.

Finally comes the programming phase. This is when we set, in the game, when a sound should play and set the necessary parameters. Things like saying that the menu button should trigger a sound when clicked or telling the audio system the value of the volume slider in the game settings so that all sounds can play at the appropriate volume. Game audio programming is a whole discipline in itself and includes a lot of things like loading audio data in memory, handling CPU usage, and making sure sound spatialization works.

Normally, after all these steps, the sound conceptualized in the beginning should be playing back in the game (after some bug fixing, most probably). An important factor, however, is that these steps don’t necessarily happen sequentially during production. Many of these can be done in parallel with proper planning (again stressing the importance of the concept phase).

Well, Demute is a company that does all of this, after the importance of the concept phase they do the waiting (not a lazy waiting as you could read above), the production of sounds using different media, the implementation of those sounds with the animations and after that something that I thought was not the responsibility of a sound department, the programming phase, that seems to me one of the most important parts, otherwise in Mario we would have the “uh-uh” sound not when we press the button to jump but maybe when we would fall making it an “uh-uh” sad, oh so very sad!

What next? Maybe, the footsteps of characters, the first thing that comes to mind! Let’s go!

Footsteps

According to Pablo, the approach of making sounds for this part of video games is similar to other sounds, always commencing with the context and goal phase and then pursuing said goals, because different approaches to creating these types of sounds come with different goals. For example, “Are we playing a city builder with many different characters on screens walking around? Then we will design the footsteps to sound discreet (not only in volume but in frequency and dynamics too) and make sure they work well when played in high quantity (so enough variation to avoid jarring repetitions). Are we playing a third-person stealth game where the footsteps of our character give information to the player as to how much noise they are making? Then we will concentrate on making the footsteps sound full and precise to fit their goal.

Technicalities

To close up this part of the interview I tried to understand the technicalities of sound creation and sound mixing. According to Pablo, The most important tool we use daily is our DAW (digital audio workstation). It’s the software that lets us construct a sound by combining, editing, and processing different audio sources. If we were to do a sculpting analogy, the DAW is the tools and the audio sources are the clay, stone, or any other sort of material we sculpt with those tools. The DAW we use here at Demute is Reaper.

To gather audio sources to work with (our sculpting materials) there are many different techniques. We will, for example, record sounds in a controlled environment in our studio (usually referred to as “foley”). We can also go outside with a portable recorder and record sounds from our environment (cities, nature, groups of people, cars, etc). More often than not, we will actually buy sound libraries created by more experienced recordists than ourselves (for example, US-based recordists usually have way more experience recording guns than Europeans). We can also create synthetic audio material to work with. These are computer-generated sounds that are created from basic electric shapes and signals. This can get very complex and usually demands a lot of work to get exactly what we want, but is essential to create sound material that is “extra-natural”. Have you ever had the chance to record a dragon or a spaceship? Me neither, so we rely on artificial sounds to help fake those sounds”.

Ambience and Environmental Sound

At this point in my conversation with Pablo, I was curious about how sound immerses players in the gameplay, how sound can dictate a player’s movement and actions, and how sound can construct an environment. My first question was directed at immersion, how they do it with sound and not overwhelm the player so much they feel lost. Pablo says that for the immersion to be correctly implemented, one must consider two parts design and mixing.

“On the design part of things, we have to think about what is the role of the ambient sound and how to build something that fits the uncontrollable nature of video games. The best example is something like a radio on a level. It will be very tempting to decide we want to play some sort of radio broadcast sound, because that’s what radios do right? However, how are we going to handle the possibility of the player just standing next to the radio for hours? If we loop the sound, the player is going to notice that something isn’t right, but we also cannot create hours of radio content. This extends to any kind of environmental sound. We need to think of what really needs sound and how to shape that sound to be as transparent as possible while still giving the information it needs to give.

On the mixing front, we will mostly have to think about when an ambient sound is important. It’s the same in movies. If you focus on a horror movie, you will notice that any time a jump scare is about to happen, the ambient sound slowly fades out to create an oppressive silence. Ambient sounds are very important to give an atmosphere, but also need to go away when relevant. Therefore, if we want to create working immersion, we have to spend time setting up rules about how ambient sounds will work and react to the player’s actions.”

Sound and mixing sounded perfect but one question popped into my mind. Could it happen that even with meticulous design and mixing of the ambient sounds, a player could be misled by them?

“The particularity of audio is that it is always present and can hardly be filtered out. This means that it can act as a powerful guide to lost players. A well-known example of this is present in multiplayer games, where the sounds a player makes will inform the other players of their location and distance. However, “with great power comes great responsibility”. Players will rely on the fact that the information they are hearing is correct. What could be worse than hearing an enemy shooting at you and then realizing there’s no enemy or danger?

An important fact to remember when thinking about this is that players did not work on the game like we did. Meaning that a lot of our assumptions are going to be invalidated when a new player picks up our game. We should keep an open mind as to what guides and what misleads players and make sure to take feedback from them because nobody plays the game the same way and nobody will play it like we did.”

Remember players, audio is powerful!!!

Voice Acting

Now, let’s dive into the topic of the voice acting sub-topic.

Well… who doesn’t remember exactly the voice of their favorite video game characters? This happens because two things merge: amazing gameplay and narrative and great voice-acting and audio mixing. So, let’s try to understand better the process of voice-acting for video games.

When working with voice actors, what are the key considerations to ensure their performances align with the game’s overall sound design?

I’d say there are two things to take into consideration when thinking about aligning with the overall soundscape of the game. First, is technical consistency. Having voices recorded in different sounding environments will make it way harder to make the player believe they all come from the same virtual space. The second one is context and projection. If an actor’s voice lines have to fit into a scene with a lot of shooting and explosions, we need them to be shouting at the recording stage. If we don’t, then these voice lines are going to be very hard to integrate into the overall mix.

How do you handle voice direction to maintain consistency in tone and quality across different characters and scenarios?

We ensure that there’s always a voice director at every recording session and that the voice director knows not only about the technical specifications but also about the game. Then it’s a matter of using our ears and taking the time to make sure everything is ready before we start recording.

What challenges do you face in integrating recorded dialogue with other in-game sounds to ensure clarity and immersion?

The most common challenge is making dialogue work in a lot of different contexts. Sometimes the same dialogue line needs to work in a quiet environment and a very noisy environment. Ideally, we’d try to have two different recordings for such a case, but it’s not always possible, so we have to find ways to make it work through dynamic mixing.”

Music Score

The last aspect that we will cover about sound design is music scores! The music scores! Oh, the memories… the score when Artur Morgan dies, or the one from Tetris, so far apart in time of production but so incredibly accurate on the gameplay that they were attached to.

A music score can transform a video game, iconize a story, can be something that will endure more than the game itself.

So, I asked Pablo a few questions about music scores and how to implement this part into a sound design job for a video game:

How do you collaborate with composers to ensure the score complements the sound design and enhances the player’s emotional experience?

To get the best results, we need a close collaboration during all of production for music. It is a very powerful tool and thus requires special attention. Usually, we will have long meetings with the game director and the composer, or musical director, to make sure that we are aligned exactly on what the music should provide and its place in the narrative/gameplay of every situation. We write lots of documentation and iterate a lot, until we have something that creates a satisfying experience.

Could you explain the process of dynamic music implementation and how it reacts to the player’s actions within the game?

Dynamic music implementation is this idea that the music should follow the player’s actions without sounding like a very bad dj randomly changing tracks every minute. How to make this work greatly depends on the game, but it mostly relies on decomposing the music in different elements (be it different instruments, different parts, different intensities) and turning them on and off “musically”. By “musically”, I mean in a way that makes musical sense : respecting tempo, harmony and generally making sense. Obviously, this can be easier or harder depending on the music style and the game genre. Free jazz and ambient music are way more flexible in the kinds of transitions that work compared to rock or pop. The ultimate goal is to make the player feel like the music has been specially composed for their unique playthrough.

To finish this amazing exchange of ideas with Pablo, there was only one thing left to ask, about the future of sound design and future projects for him and Demure:

Future of Audio Design for Video Games

What were some of the best projects that you’ve worked on with Demute and ones that you can’t wait to work on? What are some emerging trends in-game sound design, and how do you see the future of sound in video games evolving?

We have worked on more than 60 projects by now, so claiming any of those to be the best would be hurtful to the others! From small indie projects like Please Touch The Artwork to huge games like Stellaris, we take a lot of pride in everything we do. We have a couple of games lined up in production that we can’t wait for people to hear and give us feedback on, but I can’t talk about them yet, unfortunately.

The latest trends have mostly focused on spatial audio (bringing more layers of immersion to players with great sound systems at home or in VR) and user-generated content. Obviously, AI is in everyone’s mind right now and it is indeed slowly being integrated in the tools we use and the way we work. We are already using it for dialogue editing and programming for example. I personally can’t wait to see how we can continue working towards less menial work in audio allowing for more space to make creative choices and direction decisions.