Lightning Talk: Enabling the Audio Modality for Language Models - Eustache Le Bihan, Hugging Face

This talk focuses on how to integrate the audio modality into language models within the open source landscape. The speaker discusses the prevalence of automatic speech recognition and text-to-speech models observed on platforms like Hugging Face, highlighting the growth in the use of these technologies. They explain the role of audio language models, which utilize audio encoders such as Whisper Large V3 and consider the essential characteristics of audio data, including its time dimension. The speaker elaborates on challenges like sequence length in audio processing and presents advanced techniques for generating audio tokens and creating streaming architectures. They conclude by addressing audio integration in the Transformers library, emphasizing the importance of enabling seamless model distribution and performance efficiency.

Hi everyone, thanks for being here. I'm Ash. I'm a open source audio machine learning engineer at hanging face and uh my objective for today's talk is that you get a general idea of how we can enable the audio modality for language models. So let's first talk quickly about audio in the open source landscape. So if you go on the ginkace hub so the platform we have to

share models data set etc. and you look at the models. So the 2.7 million models you can find there public models you can find that they are splitted in tasks that match specific use cases and if you check those tasks especially uh for audio uh so you have those I'm not going to read them all and um so this is internal data we have I'm not showing

the y-axis because it's not public but you see the the usage of the different uh tasks for audio and you can see that what's really driving the industry still it's clearly automatic speech recognition uh Maybe you've seen also a lot of text to speech releases lately and it's super clear here. We'll see that starting Feb 2025 it really started to rise. Uh so now let's focus a

bit more the idea on audio in the transformer. So the auto auto reggressive parading landscape and how we can do it. Uh so audio language models they are basically vision language models to the extent that if you take uh the prefill prompt for so this one is for quenu to audio for example you do like for like you would do for vision you just uh tokenize and

embed your text and you take you get embeddings and for the audio you just take uh some pre-train audio encoder it's been widely adopted that this would be whisper large vtory but it can be something else and you find a way to project this uh either simply linear or something more complex to the embeddings to the text embeddings then you just interlive and concat them you pass

them to some pre-trained text decoder and just like that you've done your prefield you can start generating um so we've seen that this type of architectures they match the audio text to text it's pretty obvious now we inter leave audio and text and we generate text with it uh but if you set the prompt for the prefield to something like transcribe the audio You see that actually

the audio text to text models can serve to do automatic speech recognition and what we're seeing this year is really that automatic speech recognition is becoming a subtask of audio text to text. Um so there's been quite a lot of releases that adopt this paradigm with a lot this year and from different actors. uh so we just said that audio language models are basically VLMs but are

they really? Um so VLM what is it? It's you take an image you take some text there's no time dimension and uh and the image can be seen as a fixed size input. Well actually for audio uh as I'm speaking you can see that time extends and there's a time dimension that needs to be taken into account and that's the main big difference even if they uh

they adopt the same architecture. Uh so interlu embedding from text and audio. It's pretty different from vision encoders and audio encoders because the encoders uh they they uh they encode um audio that's represented with a time dimension and this gives a problem which is that you have of course growing audio sequence length and all the problem we know when we have sequence lens that grow. Um let's

talk a bit more in detail about uh audio encoders. So as I said for a lot of the the impactful releases uh they all started from a pre-chain whisper large v3 encoder uh which is uh pretty uh inconvenient because whisper large v3 for the ones that worked with it has a dense non-causal attention which means that you're fixed with 30 seconds input for the audio uh and

that's super inconvenient and the way that has been tackled for example for vaural order models is that you actually get more close to a video language model to the extent that you would chunk the audio as you would split frames in a video, batch them, pass them to your audio encoder and then uh reconcate in the embeddings. And if we go back to the interle text and

audio embeddings and rather than interle them, we just uh overlay them with a slight delay and we sum the embeddings and we pass them to our text decoder. We get a token, we can embed it again. And if we get the next chunk of the audio embeddings we have, so imagine you're receiving a new chunk of audio, you re-mbed you get a new chunk and you sum

it with the embedding that you've just generated. you can have an auto reggressive generation and just like that you've built uh you've built the streaming architectures we've seen the last year so cute were the ones to do it um so kudos to them French uh French cool lab um and it's been also adopted in in var real time uh the the tiny detail here is that uh

we cannot absolutely not use whisper encoder uh because we need causality for that um so there's different tricks uh that has been adapted on the wisp code architectures that are causal convolutions with convolutional cache and same thing using attention masks that are causal. So notice I said that voxal use whisper large it was the first vaura release not the vaual real time that's the streaming model um

so now we've just seen in general how we do audio in how can we do audio out so audio is a continuous u it's continuous right uh but with our parading we can only generate discrete tokens and that's where the audio tokenizer comes in uh I don't have much time to talk about it but it's super interesting I I suggest you check it out and the idea

is that you have Uh likewise like same API as you would do for a tokenizer. That's why you call it audio tokenizer. You take some audio, you uncode it. So you get a sequence of discrete tokens. The difference is that at each time step you have a new dimension. Um and you can decode back into text. Uh the the detail now is that so you you need

to multi generate multiple tokens at each time step. Uh so for that you can use multiple elements. But the tiny detail is that uh subsequent tokens on a time step they are sub they are dependent. Um so to add to to have this depend this dependency baked in the model you can apply a delay pattern that will map this dependency or you can have something else which

is using a deformer. So it's a a second autogressive transformer that will generate at each time step the the new dimensional uh audio tokens. Um so yeah so now we've seen generally how we can do audio in the auto reggressive landscape. Now we're going to talk how we do that in the transformer landscape. So transformer I'm talking about the the library. So I guess you're a bit

familiar with it. But the main idea of transformer is that it it's the modeling reference. uh it's the library that's going to hold your pytorch module uh for the model that's going to be able to run the forward and you integrate the model once in transformer and it will be available in the other libraries uh serving libraries like VLM as lang etc. Um the big bet of

the pet the the transformers library is that it's it's touch only. There's no jacks. There's no tensorflow and it's an opinionated framework. It's not meant uh for agents to read. It's really meant uh for for engineers and developers to understand the code and it comes with different um uh a specific philosophy which is that every model needs to have all the information in the same place. there's

no there's really little inheritance in the library for one model you get one file. So how does that link to what we saw just before? Um we need to be able to enable fast integration so that the model is available everywhere and uh to do that uh the idea is that we pick an architecture that's becoming standardized as we see in the audio text to text architecture

and we integrate it cleanly once we make sure it's well done. So here you see quickly the the vaural integration. It's only inherits two classes. The the pre-trained one that allows you to load and push from the EB and the generation mixing that allows you to generate and just like that you're able to do generation but also to use the model in serving back serving libraries. Um

and so what what why I'm saying that you can integrate it cleanly once and not replicate your work while we don't do inheritance is because we have a tool a specific tooling that's called modular that allows you uh to define it looks like inheritance. You can define the model and it's really mapped to the way uh the model actually developed. you start from an architecture and you

change things so it's super visible directly on the modular file and the tool is just will just unravel the full modeling file directly. Um so specifically now uh to audio what we have in transformers is uh so the input for audio are spectrograms uh maybe you're familiar with it it's basically for your transfer for your transform for now they are called feature extractors they will soon become

audio processors but the P is not finished yet um so they are touch so you can do fast batch cuda for transform etc. Um and the audio tokens that we just discussed before with the audio tokenizers, we've introduced a special class as pre-rain audio tokenizer. The idea is that we want to replicate what's happen with text tokens but with audio. Um so for a specific model you

can set in its config which audio tokenizer it is using that's integrated somewhere else. And just like that if you wrap this in a processor you can prepare the inputs for your model to generate on. So you can in the inputs do the OJ tokens will be directly uh prepared also the labels if you're training it etc. Um so now let's conclude um so this was a

quick talk but the idea was really for you to get a general idea of how the audio is looking right now in the open source landscape also in the in the research landscape and in the transformer library landscape. Uh thank you a lot for listening to this. Uh I invite you if you want to learn more about how you can distribute these models in in the web

to go to Joshua Joshua conf later uh to Joshua's talk later sorry and I'm happy to answer any question if you have uh some this was not um uh completely on everything that happened so feel free to discuss more about Any questions? Could you clarify how you said you could still have tokens being outputed? So, you had this Can you explain what that multi is? >> Of

course. Yeah, that that was tricky to present quickly, but the idea is that so at every time stamp um the difference with text is that you can have multiple tokens and actually at every time stamp the next token it's so it's called codebooks. They define the difference you have from the first token. So they are in inter interdependent. So if you go from an ident state instead

of generating just one token you can generate multiples with multiple elements. So just imagine that you have eight element you take your input your your ident state and you output eight distributions from which you can sample but you do your sampling we you don't bake the dependency between the tokens. So the idea is that so so that the model can understand the dependency either you apply the

delay pattern which is the next token you're going to going to generate it at the next step and the one after at the next step and the one after at the next step so that it has knowledge about the the the first token so the first codebook and and and so on. Either that either you use since we know about uh generating with dependencies you just use

a new transformer that will start from the hidden state and generate auto reggressively taking the the token that's been generated and going on again >> pleasure >> uh so it's u so the cool thing about it is that actually you can first it's trained with a fixed size. So 32 for Mimi from QAI and uh and the cool thing is that you can select uh you you

are not forced to use them all. So for example CSM from sesame which is a texttospech model they they used Mimi but they used only the eight first Yeah. In terms of benchmark, do you have do you know which one better? >> Um it's uh it depends on the so many factors that it's difficult to just say if it's one that's better than the other and mainly

on how it how it's trained. So I think that I think that you can get the same performances from both. Uh the dev former approach is maybe a bit more computer inefficient in my opinion. Other questions? Okay, thank you very much for listening.