Lights, Camera, Inference! Video Generation as a Service With VLLM-O... Ricardo Noriega & Doug Smith

This talk presents VLLM Omni, a powerful inference server designed for multimodal model serving, which has gained attention for its flexibility and performance improvements. The speaker, Ricardo Noriega, discusses its architecture, which allows integration of various models like auto-regressive and diffusion models through custom pipelines. He explains how VLLM Omni supports multiple modalities including text, image, audio, and video generation. The speakers also highlight practical applications and demonstrations, addressing the challenges and limitations of the current models, including issues with the quality of generated content. They encourage developers to participate in the project as it rapidly evolves, emphasizing the importance of community collaboration.

First of all, thank you very much for for attending this session lights camera inference. I'm very excited to be here as a Spaniard and as a European I'm I'm very glad by torch has finally come to to Europe. And Yeah, the the the world of AI is changing very fast as you ever you know, as everybody knows. Like two years ago we were really happy with 100

tokens per second. And now if your model cannot see and understand the world and render it back in 4K, you are behind the curve. So we're going to to explain in this a prototype that we've built video generation service with the LLM Omni. My name is Ricardo Noriega. I'm a principal software engineer in the office of the CTO at Red Hat. I'm working in different model architectures

and I'm also part of the review board of the official by torch block. And I Nice. Yeah, thank you for coming. My name is Doug Smith. I am coming from Vermont, which is our own anglicized version of La Mont Vert, which would be the Green Mountains. And I am working on the AI engineering team at Red Hat. Specifically on the LLM. I've been working on this for

the last approximate year, but with a long history in open source projects going back to the early 2000s. Looking at our agenda today, first we're going to talk about what is VLLM Omni. You may be familiar with VLLM, which is a inference server that is built heavily on top of PyTorch itself. We're going to take a look a little bit about uh, v l m omni in

action and see kind of like what we were able to push out of it and we're going to talk about just how fast this project is moving with a genetic coding tools and the community velocity this stuff is moving at crazy speed. Um, we're going to look at some of the gaps here like there it's there's a lot of them. Um, but then we're going to show

you how you can actually run it yourself and talk about how to get involved in the project. Thank you. Yeah, it We are trying to make this uh, session like informative in the technical uh, side but also as a first time contributors to to the project and what are our experiencing experiences there. So, what is v l m omni? Um, v l m omni was released uh,

by the end of of last year and it has been uh, an exciting project with a lot of traction and it's getting uh, a lot of attention. So, I've basically borrowed this this is light from the v l m omni community and they they do meet ups and presentations and I I love this slide because it's very um, you can get the uh, the full picture of

the project in in one slide. So, the goal of v l m omni is basically doing easy, fast and cheap omni modality model serving. Um, with v l m v l m is focused on traditional auto regressive models and where the input out there and the output is just text. An auto regressive model is uh, those models that generate one token at a time but you have

feedback that token into the input so it knows how to continue the the output generation. But for right now that the world is also demanding to generate other kind of artifacts like image, audio, video, and this is what Omni VLLM Omni is focused on. So, we have to run not only auto regressive models but also non auto regressive models and things like diffusion diffusion models. So, one

of the key aspects of VLLM Omni is the architecture. It's all about um building pipelines and and stages because these models are very different. Even in the same kind of category like text to image or text to audio or whatever. Um these models are completely different from one another. So, VLLM Omni is capable of allowing the the developer to integrate their models using pipelines and and stages.

in in this kind of pipelines we have different encoders like vision transformers, speech to task to text encoders like whisper for example. Then there is always an LLM backbone or like a core auto regressive model that is basically used for the understanding of the request of the of the user. And finally we have them the modality generators or basically the models that are going to generate the

the artifact that that we want like uh it could be diffusion traditional diffusion models based on convolutional neural networks or it could be the diffusion transformers uh uh, different decoding heads and and so on. some key features of the project is uh, basically simplicity. There is a seamless integration with the Hugging Face libraries like Transformers and Diffusers. And also it supports different API endpoints based on the

Open AI API specification, right? Like uh, chat completions, image generations, image edits, audio, um, so it's very easy to to use from the developer perspective and also from the production uh, perspective. the the key if if I have to highlight one aspect from vLLM Omni is the flexibility. Um, it has an abstraction layer uh, that allows the developer the model developer to to integrate it into the

inference server. And it's basically writing um, data structures for your stages in the model uh, for the auto regressive part, for the diffusion generator, and you know, having this kind of heterogeneous uh, landscape with these models, it's um, a step forward. So it's very cool. Um, we inherit the the optimization techniques from vLLM but also, you know, for the auto regressive part, for the LLMs, but we

are also adding new optimization techniques for for the Diffusers. And that's how of course that has landed into um, performance improvements and and so on. Uh, as a future roadmap, this is changing so fast that if you go to the repo in GitHub, there is an issue for the roadmap of every quarter. And so there you can see like the latest features. There is There are two

modes of operations, um offline inference and online serving. So, the offline inference we maintain a Python interface for for developers. So, you can do like batch offline batch inference with models like Quan Omni. And as you can see in both sides of the of the slide, the Python code is basically the same. You can use as example, right? Quan Omni to generate text or audio or whatever.

And Quan image to to go from text to image. And the final interface that you have to use, you build a data structure of your input and the the interface or the method that you have to use is just Omni.generate. So, it's very easy to use different models and have the same developer experience. And for the online serving part, vLLM Omni has a fast API based server,

um where you can launch the server the same way that you do in in vLLM. It's vLLM serve, but you have to add just the the Omni flag. So, it's very very easy. And from the client side, in this case for example with Quan Omni, you can use the chat completions API. We We all know the chat completions API. And in the payload, you put the model,

the message like the prompt. But also, tune or or fine, you know, yeah, fine-grain tune your your inference using the sampling params list. And it's part of of the payload of the of the message. Uh if you want to use a text-to-image model like when image, uh the way you launch the the server is the same. Uh the model name uh dash dash Omni. Um from the

client side, we can use a different API endpoint like the image generations. And in the same curl command, you can put the prompt, uh the size of the image to generate, and the seed. And it's interesting because the the uh the output the generated output is basically a base 64 encoded uh chunk of text. So, you have to decode that uh base 64 text and put it

in a PNG image, One of the main focus of the community is uh to have uh great model support. Um back in December 2025, um um Alibaba released the Qwen 2.5, I think. Uh Qwen Omni uh 2.5, and uh vLLM Omni had day zero support of of the model. So, it's one of the main focus of um of the communities. vLLM Omni supports Qwen Omni, Qwen Image,

Bagel from ByteDance, uh Flax. So, there are a lot of a lot of models, more than 30 models. So, it's a a great experience to play with with all of Finally, I wanted to talk a little bit the multimodality models. Uh as I said, it's a kind of a wild space because every model is different. If you go if you go to Hugging Face and you check,

you know, the directory structure and how all the configurations, every model is different. But we have kind of classified them in three categories. Uh the first category we call like thinker-talker. it It allows the user to do like any to any, you know, any artifact to any artifact. And you have multiple encoders for for these modalities. And then there are usually two auto regressive models, two LLMs.

Of course, served by vLLM. And one is in charge of doing the reasoning of the request. And the other the embedding generation to pass to the generator, like a diffusion transformer, for example. The second category is just one auto regressive model. Sometimes there are like two variants here, one doing prefill and just prefill phase to create embeddings that will be passed to the decoder. And the other

is with prefill and decode. So, the understanding of the request is part of the of the embeddings. And the denoising process is is done by the decoder. And the GLM image, for example, is is one of the models that follow this architecture. And finally, what we call the unified architecture is is like the new trend in the multimodality space. One example for this is the Bagel model

from Bytedance. understanding phase and the generation phase, they both share the same KV cache. So, it's it's a bit more difficult to decouple in stages, but I think it's more performant. And finally, one question that we get a lot is how does vLLM Omni interoperate with vLLM? Right now, it's a as we call a creative and strategic method, but basically it's doing some monkey patching of the

code. Uh, you can see in VLLM Omni / patch.py. And it's basically going through all the VLLM uh, modules like the engine core, uh, tokens, uh, the M rotary embedding, all these modules and overriding the the class with their own implementation. So, that's basically a bit of an overview of VLLM Omni and now we are going to see it in action. All right, cool. So, this is

a a short demonstration of kind of two modes of VLLM Omni that you can do right now. The first of which is to Can it Will it play? Um, is a just straight up uh, text to video, right? So, just input text, video out. Weird results assured. Um, and then next kind of mode that you would do would be Um, would be from your image to video,

right? So, you start with an image, you give it a prompt, and then you're going to get something that is based out of that. And again, weird results assured. Um, when we first started looking at this, there was no ComfyUI interface, which is kind of a like community darling for a way to interact with especially visual inputs and outputs for generative AI. Um, but yeah, quickly just

kind of like what we love about VLM Omni, especially from a perspective of like trying to build product on top of it is that it is built on top of VLM. And VLM is built on a really stable, well-understood, well-respected stack such as PyTorch itself, right? So, this is something that from the perspective of going to from making something, you know, a cool open-source project to something

that you can actually deliver to your customers, this is a really important and really stable type of stack. We love that about it. Um, for LTX2, which we kind of focused on for this talk, we love that it's open weight and we love how capable it is for an open-weight video model. Incredibly capable. Um, from a open-source people perspective, we don't entirely love the like licensing model

and that I wish I could change, but regardless, we're happy that these weights were opened up and were available to the public. And credit to the um uh the folks from the uh LTX2 community, um when we went to their Discord to like chat about with them, they were like happy to to discuss it and I'll talk a little bit more about the kind of open-source community

side in a moment. As much as it is really capable, um it also does have a number of limitations and kind of the number one one is this is sort of like a non-video video. Um, this is what we call the slideshow effect. And at least in my testing which I have to admit and you will notice by the videos like these were sort of like tech

guy POC type of videos and not necessarily tuned for like a like Nike pop-up shop or whatever like you're going to get your like interactive like cool experience at the Nike pop-up. Um, so I didn't do the kind of stuff that you might see if you go on Reddit and go to the stable diffusion sub or something like that and you're going to see like highly tuned

like highly cherry-picked like awesome stuff. Like these were like the real things that we experienced when we went to use it and the slideshow effect was like a major one that I thought with and at least for me I kept having to simplify and simplify and simplify my prompts and it seemed like the more complex the prompt I had the like worse result I was actually getting.

And another thing you're going to get is you're going to get especially as the videos get longer you're going to get these like spaghetti limbs and bodies and stuff where it seems to be like kind of ever expanding. Music is like not as good as I wish it would be like if you've played with Sora sometimes you'll get something you're like oh that's actually like pretty bumping.

You'll often get pretty weird stuff and there's also like kind of a limit but I think that we see that like across a bunch of a bunch of video models including the like closed source ones. when we put together this talk that it was like I don't know a few days after um uh LTX 2 was dropped and it was just an entirely green field. There was

nothing out there. There was no video generation endpoint. Um we had just recently contributed an image generation endpoint. And in that time, since then, like an entire stack has been built there. And I am used to open source projects where you come up with an idea, you POC it, you go to like 15 meetings in a row, you argue about getting consensus on it. So, that was

kind of like my mind frame going into that. And by the time I got that material collected to do that, I had noticed, oh wow, all of this stuff is actually implemented. Um so, yeah, today we've got LTX 2 support. It's in there. They like beat us to the punch. If you're going to contribute something, I say make a pull request. Um don't wait for the consensus.

See what happens on the pull request. Um there's the text-to-video endpoints. Um there's image-to-video through the both through the video endpoint. There's ComfyUI nodes. There's like a like custom ComfyUI nodes like directly in the VL M repo. And um also there's a async video serving endpoint now. The like first iteration you had to like wait for the generation. Um there are a number of gaps. And the

first one I wanted to kind of talk about was infrastructure-wise. So, Ricky mentioned that there's this sort of like offline serving, online serving. In my opinion, when you go to actually deliver this, I think that the online serving is what people are going to want. They're going to want these stable API endpoints. Um and I think that there is some big questions about how this is actually

going to work once you've got this deployed across a number of models across a number of node instances and how you're going to get the right request to the right machines to get those responses. So, like if you have one machine, it's serving uh LTX2, you have another one serving, say, your Gwen 35 Omni or whatnot, how are you going to route that incoming request to the

right thing? Um this architecture is under pressure with this like rapid changes that are happening. Um one of our uh great colleagues, Alex Brooks, pointed out a number of kind of issues that we're seeing. Um the one that really stood out to me is that in the name of expediency, people are doing things like copying portions of Diffusers code, getting it directly into the code base, and

then we're going to start to see a further and further drift. That's going to build up tech debt to the max. Um there's also a weird arc split with Okay, we have omnimodality models that are in stages, but we've also glommed on a lot of like other types of modalities in a discrete fashion. So, for diffusion and like LTX2, it's basically like frame-by-frame um kind of a

thing. We are seeing that uh this is like a little weird. Can we see like a better unification of this? Um and lastly, um it's kind of a Schrödinger's platform to me right now. So, one of my like great mentors in open source once pointed out that open source is a wave, and at some point we have to like pick a distinct moment of time on that,

and then we have to figure out how we're going to support that and have like a longer term arc for that. Like we can pull like head of main branch and debug that as engineers, but when we go to deliver that to a customer, like that isn't ideal. With the speed at which Omni is moving, it's hard to pick that point in time and also to get

the like correct CI signal from it to know that like, "Hey, this is a legitimate release that I can like potentially like deploy in production for a while." Would take additional testing on top of what's there. It's like really expensive to do and it's mostly donated resources and some Linux Foundation funds to to do that. All right, with the minute that I have left, um first I've

got for you a GitHub gist of kind of like all the pointers and how you could go and run this stuff today. You could go and make um even weirder videos than I created um and we will also post this deck um on sched so you can get the links as well later. Um and I'll give everyone just a second with that before I um pull up

the last slide, but the last slide is to say like it's a great time to get involved and also I feel like we're kind of entering a new era with open source software. So I would say if you want to get involved, um just go for it, make pull requests, make issues, see what happens and there is also uh like weekly meeting. Um there's two meetings, one

is in Chinese, the other one is in English. Um this is mentioning the English one um which is a like a really unfortunate time for Euro time zones, which I want to try to fix because I have a lot of people on my team in European time zones and I want a more of a global perspective. With that being said, thank you very much, appreciate it. Appreciate

the time.