Keynote: Stream Everything - Moving from Request input to Streaming input - Patrick von Platen

In this talk, Patrick from Mistral discusses an innovative approach to streaming in language models (LMs) by drawing parallels to human sensory processing. He emphasizes the importance of continuous input and output for real-world applications, particularly focusing on live transcription using their Vox Real Time model, which allows for real-time speech recognition by processing small audio streams. The speaker explains the architecture behind Vox Real, highlighting how it combines audio and text embeddings to generate output while simultaneously receiving new inputs. He explores the challenges of aligning audio with generated words and introduces the concept of adding multiple modalities, such as video, into the streaming architecture. Additionally, he addresses potential applications in text-to-text streaming, particularly in translation scenarios where real-time processing is beneficial. Patrick concludes by discussing practical considerations and future directions for implementing streaming models in various industrial contexts.

Hey everyone. My name is Patrick. I work at Mistral in the in the research team and I also I lead the open source efforts of Mistral. I think it's been roughly 4 years now that I I've been contributing to the ML open source ecosystem previously at Hugging Face. Um and this is actually my first PyTorch conference. So also very very happy that I can can speak here.

Great. Okay, today I want to I want to talk about streaming. It's not the fancy agent or L talk. It's it's a bit different. And I want us to to take a mental step back from how we understand LM applications today. So when you think ChatGPT and how the LM is used, you give a chat completion request where all the input um is is contained in one

request. You process with LM and then you stream the output um back. So instead, we could also think about a a framework where we stream the input instead of getting all the input at once. And to to motivate to motivate the the idea, let's maybe take a look at how how we humans process the world. it's a very simplistic explanation, but we we can understand I guess

like our our brain is um audio and and vision is streamed in and we we take an action at every time step um while new audio and and vision is being streamed in. And also, we can understand our actions as being auto regressively kind of conditioned on previous actions all the while we get more more input. Um so so that design um um like like it's it's

important I think to to see it that we generate output while new input is coming in. And why is that important for LMs? So if you want to have a flawless real-world LM or general or like real world machine interaction, you you need streaming. You need to be able to have an LM application that takes continuous input and gives continuous output. Some classic applications autonomous driving or

AI in warehouses slash robotics. And also one I think easy to understand example is live transcription. Here in this talk to to explain the streaming architecture, I want to focus on on live transcription or ASR. want to talk quickly about our Vox Real Time model which is something we're very proud of because it's um I think one of the few um true streaming real time applications or

true true real time models that is out there. we built this model heavily inspired by the research done by by Q AI. So I think they've championed that a bit. Um the paradigm. And essentially what is different to previous speech recognition application like Whisper is that with Vox Real we don't have to take a whole 30 second input audio chunk, but instead we can take very small

audio stream inputs and we generate outputs while new input is coming in. So exactly how we we humans would would process the world. So you can see that in green we we have audio and the LM receives not only the audio input but also the auto aggressively generated text embeddings. And at every time step you take new audio in and you generate output. So this application is

essential to have true speech recognition. Otherwise what you have to do is you have to chunk your audio into smaller and smaller pieces and and you always would get problems when you when you look at a very low latency. Great. Okay, so if we take a closer look at how the architecture is actually built, it's it's quite simple. So, you have your word embedding and you have

your audio embedding. And what we've done for for Vox Real Time is that we've just combined the two vectors into a hidden embedding and then we feed the embedding to our causal LM. Causal LMs are very good at auto-aggressively generating the next token. And all we do here is we essentially combine what was generated with input that is coming in. Cool. One problem with is that you

need to be sure that your generated output is one-to-one aligned in in time with your incoming input. And that can be tricky in audio, right? Because I can speak very fast or I can speak very slow. and I need to be sure that every audio frame is is aligned to my generated output so that I can feed in the the together. And here in this example, you

can see a sentence like I am wondering wondering might be cut into two tokens and there might be silence before wondering. So, I cannot just predict the one word embedding wondering for one So, what we do here is instead, we teach the model to also predict silence or padding tokens and we predict the the word wondering actually once it's been fully spoken. And so, once the full

input has been processed by by the LM. One other difficulty is that um it's quite difficult to to predict the word right away, right? There has to be some latency so that the model can look a bit into the future. One example is if you if you want to transcribe for example the the audio let's say San San Francisco, it will be much easier for the model

to to predict to transcribe the word San once it's heard Francisco as well. So, what you can do is you you essentially induce a latency to the model. Um you can see this here in the in the slide is that um we we essentially have three frames that are just delay tokens. And we will predict the transcription of the very first audio token I only after in

in this case, I think it's 240 milliseconds. So, this allows the model to essentially look into the future um for for 250 And um we've also noticed with Vox for real time is that this is a very nice parameter you can adapt according to your use case. So, the lower you you set the delay the faster is the model, the lower is the latency. Um the higher

you set it, you can also set it something like 2 seconds, then you're essentially in in offline transcription um Um but then you get much higher performance. Um with with Vox for real we've seen that um a delay of of roughly 480 milliseconds um gives already state-of-the-art speech Um which was actually very very very nice. Great. Okay, so I think now we have everything we need um

to be able to kind of combine every input stream with them auto-regressively generated word embeddings. here you you can see the application um or like the the detailed the architecture when we add the induced as well as um as as stretching um the the audio text alignment so that we can fit a a long audio sentence with a with a single the Great. Um okay, and I

think now that we've explained a bit how we can match these two, um it's also I think quite simple to make the mental jump towards uh let's just add another modality. Um because if we if you align um audio with our generated text embeddings, um and we just add these vectors in in latent space, um we could also just add another embedding, for example, video. video will

have probably the same um kind of time frequency as audio, so we chunk it the same way, and then we can also add it the same way. Um and this is a very nice framework that allows our LM to essentially process a in unlimited uh number of um of modalities. Okay, um and now it's where it becomes interesting. So, and now I think it's going to be

a bit weird, but we can also think about actually having text-to-text streaming applications. And this is not going to work um for for a normal um application. So, when you have a a question request, for example, it's going to be very hard to stream an output once you've received only the first um word, right? So, if my request is, "How many people live in Paris?" I'm I'm

it's I I I can't really teach the LM to produce an output when I've only seen how. I I need the full information. here I have to process the whole request, and only once it's been processed completely, only then I can produce. So, it's very difficult to actually put this in in streaming. when we look at this, um we see it's uh it's problematic. We we cannot

just predict something from from All right, but uh there are some applications also in text-to-text where it might actually make a lot of sense to do streaming, um and I want to show them here now. It's so when you think, for example, about translation, right? In translation, you can think about it in use cases where maybe you don't have audio, you do chat translation, and you want

to have that live. So, while people are typing, you automatically translate. And here, you can actually stream um output while you're processing input. Um in translation, you usually only need to know the context of one sentence, maybe a bit more, but but not much more. You don't need to know the whole um article that you want to translate. In this case, we could teach the model the

input word by word. And then, um the model will be taught that once the sentence is finished, it can start streaming the output while no input is coming in. So, in this example, you can see that once I've reached the end of the sentence, and I start processing my next sentence, so in this case, um what did I write? Dinner? Dinner. Um I um I start um

translating um the the word of the previous sentence, Paris. So, it's an induced delay here again, um but the model could be could be taught to do that. Um Great. In um So, what once we have that, then we could actually combine all modalities into the the streaming architecture, and we we stream in word, audio, and and video, um and generate output word embeddings. Um so, there

are some I think so, some fun applications you could think about in in industry where this might make a lot of sense. Now, some practical considerations with this paradigm. Um So, so, so, we have more of a um just this interesting idea to talk about because in practice, you're going to run probably into some problems when you want to do streaming for text-to-text. The first thing is

um how do you pre-train? Because if you do streaming, you always kind of need an input-output mapping. Um if you need input-output, it's it's very hard to to to pre-training because pre-training usually just you train on raw data, um where there's no labeled um no no no labels available. Um similarly, there's a lot of applications um and I'd say most applications that are used nowadays for um

text LLMs where um the input and the output is not going to be aligned um in in time space. So, when you think about um you you copy-paste your humongous error log um into ChatGPT, um you want to get a short answer while your input might might be a thousands of lines of uh of text. So, this going to be extremely um difficult for for streaming. But,

um the very nice thing with streaming is um in this approach is is very simple to combine all modalities. We we can combine obviously all modalities um in in the current paradigm where um we have the full request at once, but then we always have to think about where do we put the image? We put it before the text, after the text, between the text? What do

you do with the audio? If you want to do audio, image, and text? Um when we do streaming, it's it's very simple because whatever comes in is going to be combined in the same um time interval and then processed. Also, what is very very fun with a streaming is that it allows us actually to do early stopping. It's also quite nice, right? If you um would do

live translation and you notice that the model goes off rails right in the beginning, you you can stop it instead of having to wait until the model has processed your your very long article. Um and then also, there's a lot of industry tasks um where you might need real-time systems, right? Where like text might be streamed in in some uh console and you want to get outputs

right away. Um and and lastly, um even though it's very hard to do pre-training with streaming, um we can adapt pre-trained LLMs very um very very nicely um from um full request to output to um streaming the the request input to output. All right. Um but even even now um with our current models, the the idea of streaming input is still um considered very useful and Because

um what it allows us to do is to do a faster prefill. Um I think if if you work a lot with VLM, for example, you know that if you have a very long input, VLM does automatic chunked prefill. So, your whole input sequence is not processed at once because it's going to take too much activation memory. Instead, the input is is chunked into pieces, and VLM

will one by one process the chunks. So, if you allow your input to be streamed, um you could actually um run there the input faster cuz as soon as you finish the first input, um you you can actually start um prefilling your your KV cache. So, this explained here. Um if you if you have an input stream of um how how many um only part of the

sentence can already be processed by the causal and then you essentially grow your KV cache with your um with your input. And that might um actually already be useful for applications where um you have part of the input before all of it, so you can use the LM as fast as possible to not lose time. yes, and finish the the talk, um I just want to throw

some some some links out here. Um what when we released the Vox Real-Time, we worked quite a bit with the VLM team, and we added a um streaming input API as well as a real-time API um that you can try out, and um it might also be very um interesting for for future models that have full streaming capability. Well, that was actually very exact on Thank you

very much.