gaudio

뒤로가기back

Integrating the Audio AI SDK into WebRTC (2): Methodology for Building a Testing Environment

2023.06.26 by Dewey Yoon

Integrating the Audio AI SDK into WebRTC (2): Methodology for Building a Testing Environment for Effective Integration Development

(Writer: Jack Noh)

 

As outlined in the previous post (Integrating Audio AI SDK with WebRTC (1): A Look Inside WebRTC's Audio Pipeline), WebRTC is a substantial multimedia technology, encompassing audio, video, and data transmission, among other aspects. Even considering the audio segment alone, it encompasses various modules (APM, ACM, ADM, …). It's a testament to the high applicability of this technology. In this upcoming post, I plan to share the methodology behind creating a 'robust testing environment' — a critically significant step when incorporating an Audio AI SDK into WebRTC.

 

WebRTC Audio Processing Module

 

Among the audio modules in WebRTC, which one is most suited for integrating a noise suppression filter? You may have already inferred from previous posts, the module most compatible for such integration (indeed, specifically designed for this purpose) is the Audio Processing Module (APM). Primarily, the APM is a collection of signal-processing based audio filters constructed with the aim of enhancing call quality.

 

The Audio Processing Module chiefly functions as an essential module imparting effects on audio at the client-side. This module is designed with the purpose of assembling filters that elevate the quality of audio signals, making them suitable for calls. These filters, also known as sub-modules within the APM, carry out various functions.

 

Here's a brief introduction of some prominent sub-modules in WebRTC:

 

  • High Pass Filter (HPF): This filter operates by removing low-frequency signals to isolate high-frequency ones, such as voice.
  • Automatic Gain Controller (AGC): This filter maintains a uniform level by automatically adjusting the amplitude of the audio signal.
  • Acoustic Echo Cancellation (AEC): This filter eradicates echo by preventing signals from the speaker (far-end) from reentering the microphone.
  • Noise Suppression (NS): A signal-processing-based filter that eradicates ambient noise.

These sub-modules are strategically used in both Near-end Streams (where microphone input signals are transmitted to the other party) and Far-end Streams (where audio data received from the other party are outputted to the speaker). To illustrate this, consider the example of a video conference scenario.

 

Initially, the audio signal inputted into the microphone is processed through the High Pass Filter (HPF) to eradicate low-frequency noise. Then, to prevent discomfort from abrupt loud sounds, the signal passes through the Automatic Gain Controller (AGC) which automatically adjusts the signal's amplitude. Following this, the Acoustic Echo Cancellation (AEC) comes into play, preventing echo by stopping signals from the speaker from reentering the microphone. Finally, the Noise Suppressor (NS) eliminates any remaining ambient noise.

 

Visual representations often simplify the explanation process compared to textual descriptions. The function ProcessStream() depicted in the diagram below signifies the processing of the signal stream captured from the microphone, a pathway often referred to as the forward direction. Concurrently, ProcessReverseStream() exists to handle the processing of the reverse stream, which aims to render audio data received from a remote peer via the speaker. Each of these processes can be understood as an effect-processing step within the Near-end Stream and Far-end Stream of the APM.

 

Let's revisit the aforementioned process flow via the following diagram:

 

(WebRTC Branch: branch-heads/5736)

 

It's crucial to note that the two stream processing stages are not independent. This is because, in order to mitigate the echo—a situation where the voice of the other party enters the microphone again—the signal needs to be analyzed within the ProcessReverseStream() before being passed onto the ProcessStream() for echo cancellation. (This is the role of AEC as discussed earlier!)

 

With an understanding that the structure of WebRTC's APM is as described, my intention was to integrate it with Gaudio Lab’s superior audio separation technology, GSEP-LD. At first glance, it seems logical to replace NS with GSEP-LD, but is this the optimal strategy?

 

From a signal processing perspective, this might seem plausible. However, as the SDK we intend to integrate is AI-based, it's not possible to conclusively determine this. Positioning it before AEC might enhance the output, or conversely, placing it at the very end might yield superior results.

 

In response to the question 'Would the optimal integration location coincide with the location of the existing noise removal filter?' numerous additional queries arise:

 

‘Will there be any side-effects with other Submodules?’

 

‘What about simultaneously using NS and GSEP-LD?’

 

‘Is it necessary to utilize the HPF filter to eliminate low-frequency noise?’

 

‘How would the efficacy of GSEP-LD alter depending on the operation of AEC?’

 

 

Upon analyzing WebRTC's APM, it became clear that several elements need to be carefully evaluated during testing. Now, let's delve into strategies for robust testing of these scenarios.

 

Note that the integration of another SDK as a Submodule into WebRTC is not particularly challenging. This article primarily discusses the establishment of a test environment for the WebRTC Audio pipeline and will not extensively cover this topic, but here is a brief overview:
WebRTC's APM is scripted in C++ language. Initially, a Wrapper Class is devised to manage instances and states, mirroring the functionality of other Submodules within the APM. Subsequently, the Wrapper Class that you wish to integrate is embedded into the actual APM Class, and is managed in a similar fashion to other Submodules.

 

 

Leveraging APM CLI for Efficient Integration

 

Building upon our comprehension of the APM module, we now aim to establish a rigorous testing environment to derive results from GSEP-LD. For this, creating a Command Line Interface (CLI) capable of running the APM independently proves effective for obtaining file output results (Indeed, we need to actually hear the output!)

 

We have identified two methods for harnessing the APM CLI for efficient integration:

 

1) Method one involves using WebRTC's open-source resources directly.

 

2) Method two makes use of an open-source project that exclusively features the WebRTC APM.

 

 

1) Utilizing WebRTC's Open Source Directly

 

WebRTC's open-source resources can be directly accessed from the following link. The WebRTC project offers comprehensive guides on build procedures for each platform. Upon downloading the source code, installing the necessary software according to the guide, and ensuring a successful build, you are then equipped to use the APM CLI!

 

By checking the build path, you will be able to locate audioproc_f , a file mode testing tool for APM. With audioproc_f , you can feed a WAV file into the APM and obtain an output that has been processed through the audio filtering effects of the APM. To verify the APM output results using default settings, execute the following command:

 

$ ./audioproc_f -i INPUT.wav --all_default -o OUTPUT.wav

 

The testing environment can also be customized. For the next step, we will set the stream delay between the Near-end and Far-end to 30ms and activate the Noise Suppressor. Here, 'stream delay' refers to latency introduced by hardware and system constraints. Notably, to improve the performance of AEC, accurately specifying the stream delay between Near-end and Far-end is crucial, making it an important parameter to be cautious of during testing.

 

./audioproc_f -i INPUT.wav --use_stream_delay 1 --stream_delay 30 --ns 1 -o OUTPUT.wav

 

2) Leveraging an Open Source Project Solely Featuring WebRTC APM

 

The first method indeed offers a substantial level of testing. By modifying the source code of audioproc_f , a robust test environment can also be established. However, within the WebRTC project, there exists a range of additional, nonessential multimedia source codes (such as those pertaining to video, data network source codes, and other modules like ADM, ACM for audio only), thereby presenting a disadvantage due to their redundant presence. Consequently, we contemplated the possibility of devising a leaner test environment focusing exclusively on the APM, the essential component. In the course of this consideration, we stumbled upon a valuable open source project which enables us to isolate and structure tests solely for the WebRTC's APM module (Special thanks to David Seo for suggesting the idea!). This constitutes our second method.

 

Here is the link to the open source project employed in the second method. This particular project provides the opportunity to separate the APM of WebRTC and assemble it using the Meson build system.

 

A brief interjection on Meson: Meson represents a forward-thinking C++ build system, capable of constructing codes more swiftly compared to other build systems such as CMake, primarily due to its user-friendly syntax. Furthermore, it offers straightforward handling of test cases and management of test architectures. To compose a unit test in Meson, you first generate a test execution file encompassing the test code, and subsequently, inscribe the test code in said file. Afterwards, you define the test with the test() function and instruct Meson to carry it out. For instance, you could draft a code as follows:

 

test('test_name', executable('test_executable', 'test_source.cpp'), timeout: 10)

 

This code specifies a test named test_name . It conducts the test by operating test_source.cpp, a test execution file constructed from the test_executable file. The timeout parameter assigns the duration for the test to be carried out, expressed in seconds. If the test fails to complete within the stipulated time, it is deemed unsuccessful. It's noteworthy that the test execution file is capable of receiving arguments, thereby allowing for the application of different values to accommodate various scenarios.

 

Having covered the simplified process of crafting test codes with the Meson build system, the subsequent step involves orchestrating the tests for integrating GSEP-LD. The subsequent image offers a representation of the test design we assembled (albeit it being more abridged than the actual test we conducted).

 

 

 

First, we partitioned our tests into two primary categories. As you may know, there are two types of streams in WebRTC: Near-end and Far-end streams. We designated these as Group A and Group B, respectively.

 

The principal distinction between the two groups lies in the operation of the Acoustic Echo Canceller (AEC), responsible for echo removal. (When only the Near-end stream is active, the AEC is turned off, whereas it is enabled when both the Near-end and Far-end streams are utilized.)

 

Each group also has a unique configuration for the input files. In Group A, the signal consisting of speech through a microphone with added noise suffices as the input. However, Group B requires a simulation that mimics a scenario where the output signal from a speaker (i.e., the signal produced by the other party) re-enters as an echo. The echo_generator serves this purpose. The input signal for Group B is formed by summing the echo originating from the echo_generator with the signal transmitted through the microphone.

 

Subsequently, we controlled the integration point and operation of GSEP-LD. The integration was managed at the forefront of the processing (Pre-processing), at the existing noise removal filter's location, and finally at the Post-processing stage.

 

It is essential also to examine the potential side effects with other submodules, isn't it? Let's explore the side effects concerning the Noise Suppressor (NS). To validate the side effect with NS, we toggled its status and set the parameter controlling the degree of noise reduction. It's worth mentioning that the noise reduction parameter in NS can be set anywhere from Low ~ VeryHigh. However, employing an excessively high value may lead to significant sound distortion.

 

Finally, we categorized the noise types present in the input file. We performed tests for a variety of noise types, including noise typically found in a café, car noise, and the sound of rain. Despite this degree of detail leading to a substantial number of test branches (owing to the multiplication of each independent branch), the test functionality of the Meson build system allows for easy generation using just a few for-loops. Below is a pseudo-code depiction of the Meson build file.

 

 

The test code is then complete. Using Meson commands to execute the prepared test code enables us to cover all designated test branches and obtain respective output results.

 

The test went well and creating the output

Check metrics to be ensured the test went well

 

 

Conclusion

 

In a project like WebRTC, which encompasses numerous technologies, integrating a third-party technology presents a host of considerations. One must understand the overall flow for selecting an integration point and anticipate the integration across a multitude of environments. These environments involve the location of integration, the operating environment, and the potential side effects with pre-existing technologies. The details discussed above take into account relatively straightforward cases. Nevertheless, we are still presented with: Noise types (5 types) x AEC On/Off (2 types) x NS settings (5 types) x GSEP On/Off (2 types) = 100 scenarios!

 

These 100 scenarios require consideration. However, in practice, the need to test an even wider variety of noise types, consider additional submodules beyond NS, and account for platform-specific differences (Windows, MacOS, etc.) results in an exponential increase in the number of test cases. Furthermore, should the GSEP positioning need to be controlled across a broader range of locations for testing purposes, it becomes rather confusing.

 

In this article, we have shared our experience of effectively managing complex scenarios, utilizing a more streamlined Command Line Interface (CLI) environment for testing. Our learnings have demonstrated that by harnessing the capabilities of the open-source projects that isolate WebRTC's Audio Processing Module (APM) and employing the testing feature of the Meson build system, a relatively straightforward environment construction is feasible. If you wish to integrate specific filters into WebRTC in the future, we hope this article provides some insights and assistance. Thank you for taking the time to read this article.

 

+) For those eager to integrate Gaudio Lab’s unique audio SDK into WebRTC and develop a service, you are always most welcome!

+) Thanks to David Seo for letting me know about WebRTC's audio popeline, and for the idea of using an open source that stands alone as a WebRTC APM! 

 

 

 

pre-image
Integrating Audio AI SDK with WebRTC (1): A Look Inside WebRTC's Audio Pipeline

Integrating Audio AI SDK with WebRTC (1): A Look Inside WebRTC's Audio Pipeline (Writer: Jack Noh)   Curious About WebRTC?   The MDN documentation describes WebRTC (Web Real-Time Communication) in the following manner.(It should be noted that the MDN documentation is essentially a standard reference that anyone engaged in web development will inevitably encounter.)   WebRTC (Web Real-Time Communication) is a technology enabling web applications and sites to capture and freely stream audio or video media between browsers, eliminating the need for an intermediary. In addition, it permits the exchange of arbitrary data. The series of standards composing WebRTC facilitate end-to-end data sharing and video conferencing, all without requiring plugins or the installation of third-party software.   To simplify, WebRTC is a technology that allows your browser to communicate in real-time with only an internet connection, which eliminates the need to install any extra software. Services exemplifying the use of WebRTC include Google Meet, a video conferencing service, and Discord, a voice communication service. (This technology also gained substantial attention during the outbreak of Covid-19!) As an open-source project and web standard, WebRTC's source code can also be accessed and modified via the following link.     Understanding WebRTC's Audio Pipeline   WebRTC is a comprehensive multimedia technology, encompassing diverse technologies such as audio, video, and data streams. In this article, I aim to delve into aspects related to WebRTC's audio technology.   If you've had experience using a WebRTC-enabled video conferencing or voice call web application (e.g., Google Meet), you might be intrigued to understand how the Audio pipeline is structured. The Audio pipeline can be separated into two distinct Streams. Firstly, 1) the Stream of voice data captured from the microphone device and transmitted to the other party, and concurrently, 2) the Stream that receives the other party's voice data and outputs it via the speaker. These are respectively referred to as the Near-end Stream (sending the microphone input signal to the other party) and the Far-end Stream (outputting the audio data received from the other party through the speaker). We'll take a closer look at each Stream, which consists of five steps, in the sections below.   1) Near-end Stream (Transmitting Microphone Input Signal to the Receiver) Audio signals are received from the microphone device. (ADM, Audio Device Module) Enhancements are applied to the input audio signal to augment call quality. (APM, Audio Processing Module) If there are other audio signals (e.g., file streams) to be concurrently transmitted, they are integrated using an Audio Mixer. The audio signal is subsequently encoded. (ACM, Audio Coding Module) The signal is converted into RTP packets and dispatched through UDP Transport. (Sending)   2) Far-end Stream (Projecting the Received Audio Data from the Sender through the Speaker) Audio data in the form of RTP packets is received from the connected peers (multiple Peers). (Receiving) Each RTP packet is decoded. (ACM, Audio Coding Module) The decoded multiple streams are merged into a single stream by an Audio Mixer. Enhancements are applied to the output audio signal to augment call quality. (APM, Audio Processing Module) The audio signal is eventually outputted through the speaker device. (ADM, Audio Device Module) The names of the modules responsible for each stage of the process are noted in brackets on the right in the preceding descriptions. WebRTC exhibits this level of modularization for each process.   Here are more detailed explanations for each module: ADM (Audio Device Module): This interfaces with the input/output hardware domain, facilitating the capture/render of audio signals. It's implemented using APIs tailored to the respective platform (Windows, MacOS, etc.). APM (Audio Processing Module): This comprises a set of audio signal processing filters designed to boost call quality. It's primarily employed on the client end. Audio Mixer: This consolidates multiple audio streams. ACM (Audio Coding Module): This executes the encoding/decoding of audio for transmission/reception. The aforementioned process can be visualized as shown in the following diagram.     As previously described, the audio pipeline in WebRTC is notably modular, with its functionalities neatly divided.     Enhancing WebRTC Audio Quality with Gaudio SDK   Gaudio Lab houses several impressive and practical audio SDKs, such as GSA (Gaudio Spatial Audio), GSMO (Gaudio Sol Music One), and LM1 (a volume normalization standard based on TTA). The idea of developing applications or services using these SDKs, thus delivering superior auditory experiences to users, is indeed captivating.   (Did you know?) Gaudio Lab boasts an SDK that fits seamlessly with WebRTC – The GSEP-LD, a noise reduction feature that operates on AI principles. Interestingly, it offers real-time functionality with minimal computational demand (and provides top-tier performance on a global scale!)   We often endure discomfort due to ambient noise while conducting video conferences. To alleviate such noise-related concerns, WebRTC incorporates a noise suppression filter rooted in signal processing. (As a point of interest, WebRTC already contains filters beyond noise suppression to improve call quality!) This noise suppression filter forms part of the previously mentioned APM (Audio Processing Module).   Imagine the potential improvements if we replaced the conventional signal processing-based noise suppression filter with Gaudio Lab's AI-driven noise suppression filter. However, despite the eagerness to instantly substitute the existing noise suppression filter with GSEP-LD, it is crucial to proceed with caution. Attempting hasty integration (or replacement) in such a complex, large-scale project can generate complications, as raised by the following considerations:   Does the original performance of GSEP-LD measure up well? → It is important to verify the quality of the original performance. Could there be any side effects with the existing signal processing-based filters? → It is advisable to check the effects while managing other filters in WebRTC. Does the optimal point of integration align with the location of the existing noise reduction filter? → Various points of integration should be tested. Can performance be guaranteed across diverse user environments? → This requires a wide range of experimental data and consideration of different platform-specific settings. If one dives headfirst into the project solely driven by enthusiasm, they might find themselves overwhelmed by the questions mentioned earlier, causing effective integration to become increasingly challenging. To circumvent this, the primary step entails building a 'robust testing environment'. The larger the project, with its many interconnected technologies, the greater the emphasis on this requirement. However, establishing a robust testing environment is not an easy undertaking. In this article, I have discussed the audio technology of WebRTC. In the next article, I will share my experience in establishing a solid testing environment within WebRTC with relative ease. This allowed me to significantly boost my confidence in the performance as I was able to integrate GSEP-LD into the WebRTC audio pipeline.      

2023.06.26
after-image
Synchronizing lyrics? It's Directly Handled by AI: An Introduction to GTS by the PO

Synchronizing lyrics? It's Directly Handled by AI: An Introduction to GTS by the PO (Writer: John Jo)   Before we dive into the details of GTS, let me introduce myself.   Greetings! I'm John, the Product Owner (PO) of GTS at Gaudio Lab. I work as the PO of GTS at Gaudio Lab by day, and chase my passion as a jazz vocalist by night. I lead and sing for the New Orleans marching band SoWhat NOLA (shameless plug here).   Do you agree with the phrase, "We listen to music with our eyes 👀"?   What I’m actually referring to, is song lyrics. GTS plays a pivotal role in the 'real-time lyric display' feature seen in today's streaming services. Here, AI autonomously syncs lyrics with their corresponding music.   In this discussion, my goal is to shed light on the inner workings of GTS and explore how this AI product, jointly cultivated by Gaudio Lab and myself, is revolutionizing our world.     The Untold Story Behind 'Real-time Lyric Display'   Currently, most music streaming services offer real-time lyric services.     <Here's a snapshot of a real-time lyric service - just like this!>   Until recently, were you aware that the synchronization of lyrics and melody for music streamed online was manually done by individuals?   Unfortunately, the practice of manually synchronizing lyrics comes with its own limitations, some of which may seem familiar:   It is a time-consuming process. Syncing requires listening to the song in its entirety. If lyrics flow at a rapid pace, as in rap, or if the language isn't your native tongue, the processing time escalates significantly. The quality isn't always consistent. As I mentioned earlier, tracks that are harder to sync take longer to process. This makes it near impossible for those handling multiple songs in a day to perfectly process each one. The music market is flooded with thousands of songs each day. It's only logical for music streaming services to invest more in labor to ensure proper lyric syncing. Factor in management costs, and it quickly becomes a daunting task. It concerns artists as well. I personally experienced difficulties with lyric syncing during my album release. When I released my debut album two years ago, Spotify, an international streaming service, didn't provide real-time lyric services. As a result, I had to painstakingly use a manual sync service. In contrast, I recall the sense of satisfaction I felt with the smooth syncing on local streaming services that had incorporated GTS.     Let's dive deeper into the revolutionary AI solution known as GTS that resolves all these issues at once.   [GTS Concept Diagram]     GTS, short for Gaudio Text Sync, is a tool that [utilizes AI to automatically synchronize lyrics with the audio of a song]. For instance, given an mp3 file of BTS's new song "Take Two," and a txt file with the lyrics, GTS automatically generates a timeline for the lyrics.     If you're curious about how GTS can cut down time and cost, here are your answers:   GTS only takes about 5 seconds to process a single song and it covers a wide range of languages, including English, Korean, Chinese, and Japanese. Regardless of the song being processed, GTS creates accurate and considerably faster timelines than human capabilities allow. Even high-speed tracks from artists like Eminem pose no problem. For streaming services facing similar challenges, adopting GTS could be a game-changing decision (that will also positively impact your business!) GTS not only provides synchronization results, but also offers a detailed 'Sync Result Report' for each song. This report specifies the accuracy of the synchronization and checks if any lines of lyrics have been improperly synced. Users can use this report to effortlessly review and correct any discrepancies in the lyrics or synchronization. (And it could even get you off work sooner!) Today, most Korean music streaming services provide real-time lyrics created through GTS to their customers. Foreign streaming services have started to take notice, and we're continually holding intensive meetings with them. We're setting our sights on international expansion. Full speed ahead! The surprises don't end here. GTS, which started with creating timelines for each line of lyrics, is now capable of generating timelines down to each word. The R&D team is currently working on advancing the technology to create timelines on a per-character basis, just like you'd see in a karaoke session.   As the saying goes, seeing is believing, so here's a comparison: Let's take a look at the lyric syncing capabilities of Company G, a leading Korean music streaming platform, and NAVER VIBE. I've selected a track titled "Still a Friend of Mine" by one of my favorite bands, Incognito.   First, let's take a look at Company G's real-time lyrics. You'll notice that the synchronization is off by almost a complete line.   Many of you might have encountered this frustrating and inconvenient situation. It's difficult to navigate to a specific part of a song just by clicking the corresponding lyrics.     Now, let's take a look at VIBE, which has incorporated GTS technology.   With accurate synchronization, the music-listening experience is greatly enhanced. (Plus, VIBE has an exclusive feature allowing you to see native-level translated lyrics!)         GTS is revolutionizing the world in numerous ways. E.UN Education: Facilitating the enjoyment of music for those with hearing impairments. It's instrumental in music education services for those with hearing impairments. Through GTS, these individuals can experience music via vibrations corresponding to each syllable. GTS is actively being utilized by E.UN Education in Korea, who are gearing up for the launch of such an application. By determining the beginning and end points of each phrase in a song, GTS assists hearing-impaired individuals in discerning each line of the lyrics. This innovative use of AI technology is a true embodiment of Gaudio Lab's mission to leverage AI for the betterment of society.     [Picture taken during a meeting with E.UN Education]   CONSALAD: Indie musicians can now effortlessly generate lyric videos!   At CONSALAD, lyric videos are created (refer to picture below) and distributed to promote indie artists' new songs.   Here, GTS is utilized to synchronize the lyrics in these videos, illustrating how Gaudio Lab's technology assists indie musicians in their promotional efforts.       These cases are testament to Gaudio Lab's dedication to 'delivering exceptional auditory experiences through innovative technology'.     However, the journey of exploring GTS's potential doesn't stop here. The applications of GTS are endless…   GTS can be applied wherever there is a need for synchronization between text and sound. In OTT or videos, where synchronization between closed captions and audio is required. In audiobooks, where synchronization between audio and text is essential. In language learning, where synchronization between audio and text is crucial. Furthermore, GTS can be incorporated into a much broader array of applications than you might think, and I'm always open to further exploration and discussion.     In conclusion,   The first project I undertook when I joined Gaudio Lab was the commercialization of GTS. Today, GTS is widely used in most of Korea's music streaming services, and it's gratifying to see how it enhances the music experience for many users.   My next goal is to speed up the implementation of GTS in a multitude of music/OTT streaming services around the world.   If you are interested in enhancing the auditory experiences of users worldwide, don't hesitate to reach out to us at Gaudio Lab!        

2023.07.07