rtcloopbackhack.js

let rtcConnection = null;
let rtcLoopbackConnection = null;
let loopbackStream = new MediaStream(); // this is the stream you will read from for actual audio output

const offerOptions = {
  offerVideo: true,
  offerAudio: true,
  offerToReceiveAudio: false,
  offerToReceiveVideo: false,
};

let offer, answer;

// initialize the RTC connections

this.rtcConnection = new RTCPeerConnection();
this.rtcLoopbackConnection = new RTCPeerConnection();

this.rtcConnection.onicecandidate = e =>
  e.candidate && this.rtcLoopbackConnection.addIceCandidate(new RTCIceCandidate(e.candidate));
this.rtcLoopbackConnection.onicecandidate = e =>
  e.candidate && this.rtcConnection.addIceCandidate(new RTCIceCandidate(e.candidate));

this.rtcLoopbackConnection.ontrack = e =>
  e.streams[0].getTracks().forEach(track => this.loopbackStream.addTrack(track));


// setup the loopback
this.rtcConnection.addStream(stream); // this stream would be the processed stream coming out of Web Audio API destination node

offer = await this.rtcConnection.createOffer(offerOptions);
await this.rtcConnection.setLocalDescription(offer);

await this.rtcLoopbackConnection.setRemoteDescription(offer);
answer = await this.rtcLoopbackConnection.createAnswer();
await this.rtcLoopbackConnection.setLocalDescription(answer);

await this.rtcConnection.setRemoteDescription(answer);

I found this workaround very interesting because I'm facing an acoustic echo issue. I'm working only with the Web Audio API. Do you think it could be solved without using RTC connections? Thanks in advance.

I found this workaround very interesting because I'm facing an acoustic echo issue. I'm working only with the Web Audio API. Do you think it could be solved without using RTC connections? Thanks in advance.

The only way I could think to do it only using Web Audio API is to essentially write your own audio analyzer and echo cancelling algorithm and run your audio through a script processor node. I don't know efficient that would be compared to using Chrome's AEC. I figured the people developing AEC for Chrome are much smarter than me and I should just use what they provide given that is also likely to be more performant than a JS script processor node (I don't know that for sure - just an assumption). I think it really depends on the source of your audio and the cause of the echo as well as what processing you are doing in the Web Audio API. I have a feeling the AEC using the RTC connection will not work very well if you are making significant changes to the input (e.g. adding delay, reverb or other effects).

Does this mozilla example allow you to set echoCancellation true in chrome?

https://developer.mozilla.org/en-US/docs/Web/API/Media_Streams_API/Constraints#Example_Constraint_exerciser

I implemented your loopback approach in my project but it didn't kick AEC on, and so I started wondering whether AEC worked at all on my system. That mozilla example refuses to set echoCancellation: true on chrome for me but works perfectly in firefox. Also, firefox seems to have better echo cancellation than chrome even with echoCancellation: false, which is odd. It does improve with it on though.

Does this mozilla example allow you to set echoCancellation true in chrome?

https://developer.mozilla.org/en-US/docs/Web/API/Media_Streams_API/Constraints#Example_Constraint_exerciser

I implemented your loopback approach in my project but it didn't kick AEC on, and so I started wondering whether AEC worked at all on my system. That mozilla example refuses to set echoCancellation: true on chrome for me but works perfectly in firefox. Also, firefox seems to have better echo cancellation than chrome even with echoCancellation: false, which is odd. It does improve with it on though.

Are you applying echo cancellation to the source track before passing it through the loopback or after? The constraint should be applied to the track being fed into the loopback.

I agree FF's AEC seems to be better and this workaround is not needed there (but doesnt seem to hurt anything either).

I ended up not applying the constraint at all because it defaults to true in chrome, and I seem to be getting good echo cancellation, although it kicks out intermittently under circumstances I haven't fully isolated yet.

For me the issue was that the output of the loopback needs to be connected to an unmuted Audio element. I was playing it via WebAudio's destination, which Chrome doesn't seem to like.

I suspect but don't yet know that my problem with intermittently losing it is related to the fact that I handle multiple calls with multiple Audio elements, and Chrome is upset about that. Probably what I ought to do is mix them all in WebAudio and then dump that out via the loopback->Audio element route so Chrome has one obvious sound stream to apply AEC to.

// setup the loopback
this.rtcConnection.addStream(stream); // this stream would be the processed stream coming out of Web Audio API destination node

Does anyone have an example of this working with an AudioContext? I am having a doozy of time pulling a processed stream out of an AudioContext.

This seems to work everywhere except on (surprise surprise) iOS Safari under a mobile data connection. With a minimal working example, nothing but silence out the other end. It works on WiFi though. Any ideas why? I'm assuming it's something to do with the loopback connection setup, though it's not immediately obvious why, as there are no errors or unexpected states. We'll disable it for that edge case for now, which is maybe ok anyway if the echo isn't an issue on iPhone (or if they have hardware echo cancellation)

This seems to work everywhere except on (surprise surprise) iOS Safari under a mobile data connection. With a minimal working example, nothing but silence out the other end. It works on WiFi though. Any ideas why? I'm assuming it's something to do with the loopback connection setup, though it's not immediately obvious why, as there are no errors or unexpected states. We'll disable it for that edge case for now, which is maybe ok anyway if the echo isn't an issue on iPhone (or if they have hardware echo cancellation)

I would check what ICE candidate it is selecting when not on WiFi. It may be selecting a candidate for loopback that is incompatible with being on a cellular signal. In this case I think you can attempt to filter out problematic candidates - I believe in our case we currently do this with IPv6 candidates on the loopback as they were causing problems for some users.

Thanks for the suggestion @alexciarlillo. So after debugging this properly, I found that it was indeed an ICE issue (state would get stuck at "checking" or end up "failed"), and the actual candidate was probably IPv6 as you said (though it was showing as a .local) and that was the only option. This whole workaround is kinda nuts anyway, so rather than mess around too much with that, I decided to instead add a little check to see if we could set up the loopback ok, and if not, just short-circuit it and use the input stream instead. It would be used as (await isLoopbackOk(rtcConnection, rtcLoopbackConnection)) and I also added a 2 second timeout for each so it doesn't block forever waiting for the state to change. It should never take that long anyway for two local peers I imagine.

const isLoopbackOk = async (input, output) => {
	const goodStates = [ 'completed', 'connected' ];

	if (goodStates.includes(input.iceConnectionState) && goodStates.includes(output.iceConnectionState))
		return true;

	if (input.iceConnectionState === 'failed' || output.iceConnectionState === 'failed')
		return false;

	if (!goodStates.includes(input.iceConnectionState)) {
		const state = await new Promise((resolve, reject) => {
			input.addEventListener('iceconnectionstatechange', e => {
				if (goodStates.includes(input.iceConnectionState) || input.iceConnectionState === 'failed')
					resolve(input.iceConnectionState);
			});
			setTimeout(() => resolve('failed'), 2000);
		});
		if (!goodStates.includes(state))
			return false;
	}

	if (!goodStates.includes(output.iceConnectionState)) {
		const state = await new Promise((resolve, reject) => {
			output.addEventListener('iceconnectionstatechange', e => {
				if (goodStates.includes(output.iceConnectionState) || output.iceConnectionState === 'failed')
					resolve(output.iceConnectionState);
			});
			setTimeout(() => resolve('failed'), 2000);
		});
		if (!goodStates.includes(state))
			return false;
	}

	return true;
};

Yeah this workaround is indeed nuts. I was extremely surprised when it actually worked, and even more surprised how many people have adopted it at this point 🤣

Just in case anyone finds this, here is some context on this: https://bugs.chromium.org/p/chromium/issues/detail?id=687574#c60

What do you mean by "Web Audio API destination node". Is this just the final node before going to the context.destination ?

@weepy I am referring to: https://developer.mozilla.org/en-US/docs/Web/API/MediaStreamAudioDestinationNode so you run your audio through whatever series of nodes you want in Web Audio, and the last one is a media stream destination node. This node will then provide a regular audio stream of the processed audio that can fed into the peer connection.

@alexciarlillo, I've tried your workaround to overcome this issue (electron/electron#27337) that was observed on the Electron framework (which has Chromium inside). The problem is that on Electron echo cancellation is currently not supported for desktop audio capturing. So if A calls B and shares the screen including audio, B will hear the own voice when talking, because it is not cancelled out from the microphone stream on A. Unfortunately your workaround did not help. Can you tell if it should work in that case at all? In an earlier post you mentioned that echo cancellation must be enabled for the stream that's being input into your code. But that's the problem. Echo cancellation cannot be enabled in this case. My understanding was that your loopback - workaround would achieve exactly that (adding echo cancellation to a stream which is not echo-canceled for some reason).

Thanks!
Bernd

@berkon Unfortunately it does not work this way. All this does is re-add echo cancellation to a stream which has been passed through WebAudio and thus lost it's echo cancellation (but all the input data to apply cancellation still exists). Chromium does not consider screen capture audio for echo cancellation so this wont work. Echo cancellation works based on running the algorithm on an input audio stream and output audio stream. In this case there is no "input" to be considered as Chromium totally bypasses it for screenshare, and thus no echo cancelation from that source can be applied to the output stream. It looks like there are patches to workaround this in the issue you linked to though.

@alexciarlillo can you give me some demo which can run as webpage?

@liyoubdu sorry I do not have time to do this right now (I am just returning to work after paternity leave and have much to catch up on). But if you do come up with a single page working example please do share it back here for others to reference.