Recently, I had the chance to design a small audio module for a space-themed WebVR simulation.
Many people don't know that browsers support a remarkable set of tools for creating and transforming audio files.
We wanted our space simulation to be as immersive as possible. The user experience was all about communication. So we needed immersive audio to match our VR.
Let's see how we can create immersive audio for our interstellar radio transmissions, using only the browser!
Here's what our original, clean audio sounds like.
Here's what it will sound like when we're done!
In this article, we'll focus on getting the audio right. Later I'll share how to use the Google Daydream controller to create a great, immersive experience so players feel like they're using a real sci-fi Commlink!
First, we need the audio to feel like it traveled a long distance. You're in space, after all.
I first implemented WebRTC (real time communication), a powerful tool for streaming audio between clients.
WebRTC can be fussy. We also found it felt too immediate for our purposes. Once connected, the browsers would send a continuous stream of audio to the other user.
We wanted discrete messages like a radio transmission. The user, holding a Google Daydream controller, would push and hold a button to record a message. Once they released the button, the message was finalized and sent to the other user. It made the controller feel like a sci-fi commlink. WebRTC is great for streaming, but felt like a poor solution for this push-to-talk functionality.
Instead of using WebRTC, consider using a fast database solution like Google's Firebase. Firebase is a fast, real-time database that you can monitor from a Google dashboard. Each time a user recorded a message, we would transform it to a binary string and send it up to Firebase. The other user had a Firebase listener checking if any new messages appeared in the database.
Once the new message was in the cloud, Firebase sends it to the other user automatically. Although this is probably not a common use for Firebase, we were satisfied with the results.
Sending audio this way provided two cool advantages:
-
It's slower than WebRTC. So it organically introduced a delay between when the audio was recorded and when it was received by the other user.
-
It keeps a record of all messages that had been sent by both users in small, easy-to-use audio files rather than a giant recording of the continuous stream from WebRTC. Since players recorded with push-to-talk, all the audio on our server was signal, rather than lots of dead air or noise from the WebRTC stream.
For immersion, we can't have the crystal clear audio that WebAudio typically provides. We wanted our radio transmissions to sound like the classic audio from the Apollo space missions.
To get this effect, I used Tuna.js, an audio processing library that wraps the WebAudio API. The vanilla WebAudio API has everything you need to transform audio: overdrive, filters, and tons of other effects. But libraries like Tuna.js make your life a lot easier!
The WebAudio API uses a series of connected nodes to process audio. You can define the nodes you need, then chain them together between the audio source and the audio context that plays the processed audio. Here's a great tutorial for how to get started.
I used filter and overdrive. A bandpass filter removes frequencies above and below the band you define. This 'hollows out' the sound, making it more closely resemble a radio's limited range. Then I added light distortion to the audio with an overdrive node. This made the audio grainy and crunchy.
After overdriving the audio, the signal was a bit too loud. I used a gain node to lower the volume.
Here's the final module to process the audio.
import Tuna from 'tunajs'
/*
Takes in the audio context and audio source declared by the WebAudioAPI. You'll
need to set get access to the user's microphone and set up an audio node.
Use this tutorial to start: https://www.html5rocks.com/en/tutorials/webaudio/intro/
*/
const processRadioTransmission = (audioContext, audioSource) => {
var tuna = new Tuna(audioContext)
// Filters out high and low freqs
var filter = new tuna.Filter({
frequency: 440,
Q: 80,
gain: 0,
filterType: 'bandpass',
bypass: 0
})
// Distorts audio
var overdrive = new tuna.Overdrive({
outputGain: 0,
drive: 1,
curveAmount: 0.65,
algorithmIndex: 2,
bypass: 0
})
var gainNode = audioContext.createGain()
gainNode.gain.value = 0.05 // Careful with volume here!
// Chains WebAudio nodes together to return processed audio buffer
filter.connect(overdrive)
audioSource.connect(overdrive)
overdrive.connect(gainNode)
gainNode.connect(audioContext.destination)
// ^ After lowering volume, pass to destination for playing
}
export default processRadioTransmissionWe're ready for final touches. The audio now sounds great. Let's add the famous quindar beep at the end of each radio transmission to signal that the transmission has ended.
const playAudio = (dataArr) => {
const audioArrBuff = convertDataStreamToAudioArrayBuffer(dataArr)
// ^ converts raw data into an audio array buffer so that audio node can play it
const source = audioContext.createBufferSource()
// ^ prepares audio context to receive array buffer with a new buffer source
// Registers an event listener to play 'Quindar Beep/Tone' at end of transmission
source.onended = () => {
// When audio source has stopped playing...
NASABeep && NASABeep.play() // bullet proofing for some smart phones that do not support all parts of WebAudio API
// Hides UI indicator if Pilot
toggleHUDIndicatorVisible(transmissionIncomingIndicator, false)
if (audioQueue.length > 0) playAudio(audioQueue.shift())
// ^ Call playAudio again if multiple messages are queued for playing
}
processRadioTransmission(audioContext, source)
// ^ Transform raw audio into a 'radio transmission'
// Transforms ArrayBuffer into AudioBuffer then plays
audioContext.decodeAudioData(audioArrBuff)
.then(decodedAudio => {
audioSourceIsPlaying = true
source.buffer = decodedAudio
source.start()
// Displays UI indicator if Pilot
toggleHUDIndicatorVisible(transmissionIncomingIndicator, true)
})
.catch(err => console.error('decodeAudioData threw: ', err)) // your preferred error handling here
}This function handles audio playback for our radio transmissions. It converts raw data from Firebase, processes the audio using the chain of Tuna.js/WebAudio nodes that we defined. Then it plays the transformed audio back using our audioContext.
When the audio source has finished playing, our event listener ('onended') will trigger separate audio node to play. This audio node holds a small file with NASA's famous Quindar beep. When our message is finished playing, the Quindar beep will sound.
That's about it for this step! Let me know if you have any questions.