Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
let rtcConnection = null;
let rtcLoopbackConnection = null;
let loopbackStream = new MediaStream(); // this is the stream you will read from for actual audio output
const offerOptions = {
offerVideo: true,
offerAudio: true,
offerToReceiveAudio: false,
offerToReceiveVideo: false,
};
let offer, answer;
// initialize the RTC connections
this.rtcConnection = new RTCPeerConnection();
this.rtcLoopbackConnection = new RTCPeerConnection();
this.rtcConnection.onicecandidate = e =>
e.candidate && this.rtcLoopbackConnection.addIceCandidate(new RTCIceCandidate(e.candidate));
this.rtcLoopbackConnection.onicecandidate = e =>
e.candidate && this.rtcConnection.addIceCandidate(new RTCIceCandidate(e.candidate));
this.rtcLoopbackConnection.ontrack = e =>
e.streams[0].getTracks().forEach(track => this.loopbackStream.addTrack(track));
// setup the loopback
this.rtcConnection.addStream(stream); // this stream would be the processed stream coming out of Web Audio API destination node
offer = await this.rtcConnection.createOffer(offerOptions);
await this.rtcConnection.setLocalDescription(offer);
await this.rtcLoopbackConnection.setRemoteDescription(offer);
answer = await this.rtcLoopbackConnection.createAnswer();
await this.rtcLoopbackConnection.setLocalDescription(answer);
await this.rtcConnection.setRemoteDescription(answer);
@slorenzo

This comment has been minimized.

Copy link

@slorenzo slorenzo commented May 14, 2020

I found this workaround very interesting because I'm facing an acoustic echo issue. I'm working only with the Web Audio API. Do you think it could be solved without using RTC connections? Thanks in advance.

@alexciarlillo

This comment has been minimized.

Copy link
Owner Author

@alexciarlillo alexciarlillo commented May 14, 2020

I found this workaround very interesting because I'm facing an acoustic echo issue. I'm working only with the Web Audio API. Do you think it could be solved without using RTC connections? Thanks in advance.

The only way I could think to do it only using Web Audio API is to essentially write your own audio analyzer and echo cancelling algorithm and run your audio through a script processor node. I don't know efficient that would be compared to using Chrome's AEC. I figured the people developing AEC for Chrome are much smarter than me and I should just use what they provide given that is also likely to be more performant than a JS script processor node (I don't know that for sure - just an assumption). I think it really depends on the source of your audio and the cause of the echo as well as what processing you are doing in the Web Audio API. I have a feeling the AEC using the RTC connection will not work very well if you are making significant changes to the input (e.g. adding delay, reverb or other effects).

@nick-parker

This comment has been minimized.

Copy link

@nick-parker nick-parker commented Jun 1, 2020

Does this mozilla example allow you to set echoCancellation true in chrome?

https://developer.mozilla.org/en-US/docs/Web/API/Media_Streams_API/Constraints#Example_Constraint_exerciser

I implemented your loopback approach in my project but it didn't kick AEC on, and so I started wondering whether AEC worked at all on my system. That mozilla example refuses to set echoCancellation: true on chrome for me but works perfectly in firefox. Also, firefox seems to have better echo cancellation than chrome even with echoCancellation: false, which is odd. It does improve with it on though.

@alexciarlillo

This comment has been minimized.

Copy link
Owner Author

@alexciarlillo alexciarlillo commented Jun 8, 2020

Does this mozilla example allow you to set echoCancellation true in chrome?

https://developer.mozilla.org/en-US/docs/Web/API/Media_Streams_API/Constraints#Example_Constraint_exerciser

I implemented your loopback approach in my project but it didn't kick AEC on, and so I started wondering whether AEC worked at all on my system. That mozilla example refuses to set echoCancellation: true on chrome for me but works perfectly in firefox. Also, firefox seems to have better echo cancellation than chrome even with echoCancellation: false, which is odd. It does improve with it on though.

Are you applying echo cancellation to the source track before passing it through the loopback or after? The constraint should be applied to the track being fed into the loopback.

I agree FF's AEC seems to be better and this workaround is not needed there (but doesnt seem to hurt anything either).

@nick-parker

This comment has been minimized.

Copy link

@nick-parker nick-parker commented Jun 10, 2020

I ended up not applying the constraint at all because it defaults to true in chrome, and I seem to be getting good echo cancellation, although it kicks out intermittently under circumstances I haven't fully isolated yet.

For me the issue was that the output of the loopback needs to be connected to an unmuted Audio element. I was playing it via WebAudio's destination, which Chrome doesn't seem to like.

I suspect but don't yet know that my problem with intermittently losing it is related to the fact that I handle multiple calls with multiple Audio elements, and Chrome is upset about that. Probably what I ought to do is mix them all in WebAudio and then dump that out via the loopback->Audio element route so Chrome has one obvious sound stream to apply AEC to.

@JamesMcMahon

This comment has been minimized.

Copy link

@JamesMcMahon JamesMcMahon commented Jul 8, 2020

// setup the loopback
this.rtcConnection.addStream(stream); // this stream would be the processed stream coming out of Web Audio API destination node

Does anyone have an example of this working with an AudioContext? I am having a doozy of time pulling a processed stream out of an AudioContext.

@yousefamar

This comment has been minimized.

Copy link

@yousefamar yousefamar commented Sep 24, 2020

This seems to work everywhere except on (surprise surprise) iOS Safari under a mobile data connection. With a minimal working example, nothing but silence out the other end. It works on WiFi though. Any ideas why? I'm assuming it's something to do with the loopback connection setup, though it's not immediately obvious why, as there are no errors or unexpected states. We'll disable it for that edge case for now, which is maybe ok anyway if the echo isn't an issue on iPhone (or if they have hardware echo cancellation)

@alexciarlillo

This comment has been minimized.

Copy link
Owner Author

@alexciarlillo alexciarlillo commented Sep 24, 2020

This seems to work everywhere except on (surprise surprise) iOS Safari under a mobile data connection. With a minimal working example, nothing but silence out the other end. It works on WiFi though. Any ideas why? I'm assuming it's something to do with the loopback connection setup, though it's not immediately obvious why, as there are no errors or unexpected states. We'll disable it for that edge case for now, which is maybe ok anyway if the echo isn't an issue on iPhone (or if they have hardware echo cancellation)

I would check what ICE candidate it is selecting when not on WiFi. It may be selecting a candidate for loopback that is incompatible with being on a cellular signal. In this case I think you can attempt to filter out problematic candidates - I believe in our case we currently do this with IPv6 candidates on the loopback as they were causing problems for some users.

@yousefamar

This comment has been minimized.

Copy link

@yousefamar yousefamar commented Sep 24, 2020

Thanks for the suggestion @alexciarlillo. So after debugging this properly, I found that it was indeed an ICE issue (state would get stuck at "checking" or end up "failed"), and the actual candidate was probably IPv6 as you said (though it was showing as a .local) and that was the only option. This whole workaround is kinda nuts anyway, so rather than mess around too much with that, I decided to instead add a little check to see if we could set up the loopback ok, and if not, just short-circuit it and use the input stream instead. It would be used as (await isLoopbackOk(rtcConnection, rtcLoopbackConnection)) and I also added a 2 second timeout for each so it doesn't block forever waiting for the state to change. It should never take that long anyway for two local peers I imagine.

const isLoopbackOk = async (input, output) => {
	const goodStates = [ 'completed', 'connected' ];

	if (goodStates.includes(input.iceConnectionState) && goodStates.includes(output.iceConnectionState))
		return true;

	if (input.iceConnectionState === 'failed' || output.iceConnectionState === 'failed')
		return false;

	if (!goodStates.includes(input.iceConnectionState)) {
		const state = await new Promise((resolve, reject) => {
			input.addEventListener('iceconnectionstatechange', e => {
				if (goodStates.includes(input.iceConnectionState) || input.iceConnectionState === 'failed')
					resolve(input.iceConnectionState);
			});
			setTimeout(() => resolve('failed'), 2000);
		});
		if (!goodStates.includes(state))
			return false;
	}

	if (!goodStates.includes(output.iceConnectionState)) {
		const state = await new Promise((resolve, reject) => {
			output.addEventListener('iceconnectionstatechange', e => {
				if (goodStates.includes(output.iceConnectionState) || output.iceConnectionState === 'failed')
					resolve(output.iceConnectionState);
			});
			setTimeout(() => resolve('failed'), 2000);
		});
		if (!goodStates.includes(state))
			return false;
	}

	return true;
};
@alexciarlillo

This comment has been minimized.

Copy link
Owner Author

@alexciarlillo alexciarlillo commented Sep 24, 2020

Yeah this workaround is indeed nuts. I was extremely surprised when it actually worked, and even more surprised how many people have adopted it at this point 🤣

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment