GStreamer Echo Canceller

For a long time I believed that echo cancellers had no place inside GStreamer. The theory was that GStreamer was too high level and would never be able to provide accurate enough delay information for any canceller to work. With fairly simple test, I could quickly confirm that the reported latency is often off by a period (generally 10ms). This isn’t strictly GStreamer’s fault and is not in any ways catastrophic for general playback experience.

With the apparition of WebRTC in browsers, it most likely became apparent that to be cross-platform browsers needed to have their own canceller. That’s exactly what happened in libWebRTC (former libjingle, used in both Firefox and Chrome to implement WebRTC). They implemented an echo canceller that accept an approximate delay and this changes everything for GStreamer.

At Collabora, I recently had the opportunity to implement this WebRTC Audio Processing based echo canceller. The main motivation was that the canceller on the hardware DSP we had didn’t work due to a hardware bug. A lot of those boards had been produced and no rework was possible. To save these boards, we decided to try with a software echo canceller. Even though it was using a fair amount of CPU, the experiment was a success. I have then clean-up the code and the new elements are now available in GStreamer Plugins Bad.

How does it work ?

Echo

The first step is to understand what is the echo. In a phone call with loud speaker, what happens is that your microphone records both your voice and the far end voices. The side effect, is that you are sending to the far end listeners both your voice along with a bad copy of their voices a moment before (the echo). To avoid this echo, you need to monitor the far end stream that you are playing back and “subtract” it from the recorded stream. In practice, it’s much more complex work, since the signal is deteriorated by the speaker and the microphone. You also need to figure-out the delays and hint the canceller, otherwise you may end-up with a terrible startup time or it may simply not work.

The implementation was greatly inspired from an experiment Olivier Crête did in 2008 using Speex DSP. I must admit, I never really understood his way of synchronizing the streams and literally ignored pretty much all the code that wasn’t GStreamer specific. The design works this way, you have a DSP element (webrtcdsp) that process the recorded stream and a probe element (webrtcechoprobe) that analyses the far end stream (before playing it back). Due to WebRTC library limitation, those two elements will transform the input buffer into chunk of 10ms. This is done using GstAdapter help. On the probe side, we push buffers in the adapter with timestamp transformed to running time. This time, plus the pipeline latency, gives us the moment in running time when the buffer should be heard by the microphone. We then synchronize the far end data against the recorded data and then let WebRTC Audio Processing library do it’s magic. A simple way of testing the element is by using an echo loop.

  gst-launch-1.0 pulsesrc ! webrtcdsp ! webrtcechoprobe ! pulsesink

Without the canceller, this pipeline would create a lot of echo, and probably end with loud feedback if your microphone volume is high enough. With the canceller, you should instead ear only one echo. It behaves a bit like a sound monitor but with too high latency and the side effect of fading in and out monotonic frequencies. After all, this is not what the algorithm have been design for. Try it in your real audio call application, that’s where you will most likely get the best results.

Before I conclude, there is a good reason why I called the element DSP rather then AEC. WebRTC Audio Processing is much more then just an echo canceller. In fact, it implements a wide variety of filters, noise suppressor, voice activity detection, etc. Currently we enable of subset of it, but I’m definitely looking forward enabling more (if not all) features from this library. I also encourage contributions. This works was only possible because of the great effort Arun Raghavan have put into extracting the echo canceller form the WebRTC project, create a standalone library usable by all. If you are interested about what cool feature could be added in the future, have a look at Arun’s blog about beamforming. And last one, thanks to my colleague who had to suffer me speaking with my computer listening to my echo for few weeks.

4 thoughts on “GStreamer Echo Canceller

  1. Hello Nicolas,

    Great work, and thanks for this post! i will test you echocanceller on very short notice. Regarding the picture in your blog, i think you need to swap the symbols for audio sink and network sink.

    Regards,
    Johan Kleuskens
    The Netherlands

  2. Hi
    I’m using GStreamer for capture, send & receive over UDP. But when I try to send and receive from remote android app, on my hardware it creates a lot of noise and echo. I want to implement echo canceller in my hardware. Below are my command:

    Sending:
    “gst-launch-1.0 -v alsasrc ! audioresample ! audioconvert ! audio/x-raw, format=S16LE, layout=interleaved, rate=8000, channels=2, endianness=4321, width=16, depth=16, signed=true ! udpsink host=$1 port=5001”

    Receiving:
    “gst-launch-1.0 -v –gst-debug-level=4 udpsrc port=5003 ! audio/x-raw, layout=interleaved, rate=8000, format=S16LE, channels=2, endianness=4321, width=16, depth=16, signed=true ! audioconvert ! autoaudiosink”

    Can you please tell me how can I implement echo canceller in above commands. ?

  3. Hi,

    thanks for your interest. You seem to need a bit of assistance, notably because you are using some GStreamer 0.10 syntax here. My blog is of course not a support forum for GStreamer, I would like to invite you to ask your question at gstreamer-devel@lists.freedesktop.org , which is a mailing list dedicated for this kind of question among GStreamer developers. You’ll need to register here in order to receive the reply.

    https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Leave a Reply

Your email address will not be published. Required fields are marked *