Kubeflow in Cloud Native AI: Orchestrati... Johnu G, Valentina S, Antonin S, Alexander P & Michael Z

This talk explores Qflow, a platform designed for cloud-native AI orchestration, specifically focusing on agentic AI and LLM operations. Valentina Rodriguez, the principal architect, along with other maintainers, discusses how Qflow serves as a comprehensive foundation for AI platforms on Kubernetes, highlighting its ability to streamline processes for data scientists and AI engineers. They explain various Qflow components like the Spark operator, model registry, and the new notebooks 2.0 feature that improve model building, automation, and deployment. Additionally, the presentation covers the unified Python SDK that simplifies access to these components and introduces enhancements related to model training and inference. By focusing on flexibility and user experience, Qflow aims to adapt to the evolving landscape of AI development.

All right. Hello everyone. Welcome to our session. Uh this is Qflow in cloud native AI orchestrating the next wave of uh agentic AI and LLM ops. So this is in the maintainer track. I'm Valentina Rodriguez. I'm a principal architect at READ. I work on AI. I'm also a product maintainer at Qflow. And here with me I have >> Hey, I'm Jonu. I lead initiatives within Newanics. I'm

maintainer of a couple of uh CNC projects like the cubeflow case serve and on gateway. >> Hi, welcome everyone. My name is Stephan. I'm a software engineer and a maintainer for the flow SDK and the cubeflow trainer. >> I'm Alexander Pearlman. I'm a distinguished engineer at Capital One and I'm also a Qflow pipelines maintainer. >> Hey everyone. Uh I'm Mike. I'm a lead MLE at Capital One

and I too am also a >> All right. Uh so how many of you are using Qflow currently? >> All right. How many of you have heard about QFlow? All right. So I guess my yeah everyone is trying to learn about Qfl. So you are on the right place. Uh so usually in this um yeah this is very special session for us. So we are bringing uh

maintainers and users to really uh walk you through the value of cubeflow, what Qflow is. Um so I will talk about this in in a second but also we will talk about the different components. So what is really Qflow is a foundation of tools for AI platforms on on Kubernetes. So, but it's only not it's not only about LLM ops, it's also about Genai. And we do

have a lot of use cases in our website that you can explore. Um, and how it works is really we want to make things simple for data scientists, AI engineers, platform engineers, but also be portable and be scalable. So, this is this is a QFlow overview. As you can see there are different components because we want to provide the different tools for all the personas in order

to build those models in order to build those AI workflows. So um yeah we see uh we have the spark operator trainer kive kerve uh we have also pipelines model registry as well and uh not and notebooks of course. So how all this works together? So I have here how you work so how we can work all together. So you may start with a data preparation with

a spark uh then you move into uh feature store where you can use for uh features um and then you also work with notebooks. So the idea with notebooks is allows you to start like really building and testing your models. Um, of course we want to automate this and this is where pipelines plays an important role in order to automate all those functionalities to build and deploy

my model. Um, we also uh do work with model training. So you can train the model, you can also fine-tune it and also we want to ser that model. So we have a caser that is a component that we can ser that model. Um and yeah all of that is also part of the AI development life cycle when you can uh finish and completing that model development

but also you can start for gen AI. So with that um I think I mentioned all the tools I forgot about KIV of course. So Kive is for uh have a parameter optimization. So after you train the model you can fine-tune those parameters and optimize it. So this is like a high level overview. So what I just described is all the different components that you can use

either independently or you can use it all together. So depends on what is your need as a data scientist or as an AI engineer you can use all the different tools. So I will start talking about notebooks. notebooks is we are providing that uh specific component so the data scientists can build we were saying right the models but what is the idea the idea is that you

can run this on kubernetes it can be portable but also you can access all the tools that you need on the same environments but also you can share all those environments across your organization and you can access all the compute resources that you need in a standardized way so that's the that's the whole idea with notebooks So, how many of you are using notebooks right now? Great.

So, have you heard about notebooks 2.0 that is coming? You probably heard some of this, right? Thanks. So, yeah, we were talking about notebooks 2.0. Uh, the community's been working for a while already because it's a lot of work involved on this. So, it's really a shift on the way that it works. So a Qflow within the community as AI evolves we want to evolve within and

the whole idea is to make things simple. So simple for platform engineers simple for data scientists. How many platform engineers we have in the room? All right. How many data scientists? All right. The rest AI engineers, developers. All right. Um so yeah if you are a platform engineer you start thinking okay how can I provide the tools for the AI uh ML engineers right or data scientists

that they want to start building. So the whole idea is you will provide the tools but you can control who access what. So we are bringing more customizations in terms of Kubernetes like with customs resource definitions. So you can control who access what the GPU uh resources you can also control the runtimes the images all of that will be all control. So the idea is to really

simplify that experience for data scientists. So data scientists only focus on building the models. So we are moving kind of the abstraction complexity everything Kubernetes to the platform engineer operations and the a IML practitioner will be able to choose from different images to use those remote environments to build the models but also they can access those templates to access those uh complex GPU resources when they need

who are provided by the uh the platform engineer. So it's really a combination between moving the complexity into the platform engineer but also simplifying the use for the and now yeah so cubeflow provide a single unified Python SDK that gives access to all the cubeflow components and that makes it easy to use all the popular a IML libraries and framework like pytor deep speed jack on kubernetes

but it also provide a local execution mode that you can use to get started locally and then you can you can um connect to your remote cluster later on without changing your um your code basically. So it um that the SDK include a client for cubeflow trainer kib. We've been working recently in the latest release for a client for um the spark operator for the model registry

and we are working for a client for cube pipeline and we've been working lately on publishing a dedicated website. So if you want to check it out at sdk.kflro.org you you can get more information about the unified SDK. Yeah. Thank you. Um so now on the on the other components from um the cubeflow platform the model registry it provide a central and intuitive interface for managing your

AI models. Um it comes with that um web UI that you can see on the left hand side um where you can um get access to all your model metadata version and uh artifacts. Uh as I said before we've recently added a model registry client to the unified SDK. Uh we've been working on um getting a deep integration with KSEL so that you can um deploy models

directly from referencing model from metala in the model registry and um you can keep track of the case of inference services in the model registry records. Um and the community is working on integrating with six store for secure sec secure supply chain by signing and verifying models. Um in addition to the model registry uh we are working on the cube hub um to meet the larger need

for LLM ops and agent AI with a model catalog that you can use to integrate external model sources like phase and uh as well MCP catalog so you can manage your MCP integration for your agent um the cubeflow trainer so that's the component that enables distributed model training and finetuning um in Qflow uh it provide also advanced scheduling capabilities with the integration with Q Apache unicorn and

um and those type and volcano for example uh it also provide a distributed data cache so can you can stream with zero copy tensor to GPU nodes using Apache arrow and apache data fusion um and we've recently in the latest version we've been um adding more um more runtime like jacks flux for HPC type of workload XJ boost um on the control plane side. So for those

who are familiar with the cubeflow the cubeflow training operator of v1 the cubeflow trainer v2 provide a single um kubernetes API the train job API um whether you use um pytor dax you you can only use that one unified API and it clearly separate concern with a training runtime API that platform administrator can use to configure the training runtime and there is also a a training client

in the unifi tryflow SDK. So you for for the data scientist or the AI practitioner, you don't have to do any EML uh and they can just do use the Pythonic API from the Cubeflow SDK. We've recently in the latest version published Elm chart for those who are using Elm compared to customize and we are working on a dedicated um website for for the flow trainer. Um,

one of the major highlight from the latest um, release is the ability to keep to keep track of the training job progress directly in the train job API on the control plane side. So for example, you can use a cube cutter and and gra the train job and you you get to get the real time progress of that train job. Um, there is there has been a

session for those who are interested from Bijit Dumal and and Rob Bell yesterday. I've added the link there. Um and it's important because it defines a contract between the runtime and the control plane. So if you see like the pod, it uses that contract to publish in real time. It pushes the the train job status to the uh training controller and then the training controller publish the

update the train job status on the control plane and that enables a lot of new functionality and new feature. uh obviously real-time progress tracking on the train job status but also on the kib that we'll see in the next slide for for hyperparameter optimization using the train job API we've recently contributed inface transformer a PR for a cubeflow call back uh so you with zero code basically

it guides cube for train um a gface transformer with cube for trainer and and we'll be working on the training UI uh where you'll get like a progress bar for example for your training jobs uh integrating with transparent container checkpointing which is a new working group uh in Kubernetes that I started where you could do intelligently checkpoints based on the runtime metrics h as well as uh

support for pytorch elastic train job with um by directly triggering the um elasticity using realtime metrics um so cubeflow optimizer as known as kib that's the component that provide kubernetes um native believe autoML functionality like hyperparameter um neural network search um early stopage uh we've integrated the new crypto trainer train job API with with um with Katip for for doing hyperparameter experiments and um there is a

client uh for trainer in the Unifi SDK and you can get more information on the link below um and the Kflow spark operator so data processing at scale with Apache Spark. Um, recently we've added support for the the the version 4 of Spark. Um, it provides a a Kubernetes native API, the Spark application custom resource definition for batch job. And we've recently been working with a new

CRD, the Spark connect to enable interactive um development. And there has been a lot of like new feature uh in the latest version with the Q integration for spark application and the integration with notebooks and jupyter enterprise gateway. So that's a way to run pispark um from your notebook but remotely on the Kubernetes cluster and um and in the latest version of the crypro SDK we added

a spark client and as well you can find more information on the link below and on that oh yeah and um we plan also to provide a MCP integration for for Cubeflow uh where you can use natural language to actually um use the the functionality from the platform. So it's just an example as we'll be starting with the flow trainer where you can um with your agent

integrated with that MCP integration uh drive the the fine-tuning of a model using natural language and we the plan is to start with the flow trainer and to roll out to all the and on that I'll pass to Ju I'll talk about the inference platform. So it's a big topic but again in interest of time I'll keep it short if you have questions we can take it

offline we can talk for a week um of various inference features but now on a high level um queso is a kubernetes native u hardware agnostic inference platform serving your predictive and generative use cases so the key point is um we are not opinionated you can bring different frameworks that of your choice and get it plked it supports all the native kubernetes abstractions like autoscaling networking etc

So for example um on the networking layer on the gateway side you can have you can bring any networking stack which is gateway API compliant um uh envoy envoy gateway to a lot of scaling kada uh standard HPA on uh the uh runtime side on generative use case it can be vlmdglang on the predictive side it can be xt boost u scikit or any other popular ML

platform and we have very specific geni integrations as well llmd the on via gateway and for LM cache for KB cache management. I'll talk a little more about that in the next one. So from case side we are focusing a little more on the genai use cases um uh like in all the releases that we actually had. So we have a standard uh standardized interface for OPAI

compatible protocol. Uh which is a front interface for users to interact. Uh model cache for um uh ensuring that uh we have uh uh lesser code start time. Uh KV cache offloading to uh offload your KV cache to multiple tiers including CPU and storage. uh KV cache aware routing to actually like send requests to the right part where um uh your request can be served better for

the better TTFT. Uh you can bring your uh model files from storage of your choice. Um and it supports all different accelerators. It's not opinionated at all like it can be Nvidia AMD GPUs or even u uh Intel AMX uh or any other accelerator that you have. Uh so recently we have added a new CD specifically for GNI LLM workloads which is the LLM ISVC. Um so

this has better expression for LLM specific workloads. Uh it's a separate Helm package itself for now because we didn't want to uh have a conflict with the existing inference service. So it's a separate helm package separate controller but again over time we plan to migrate that entirely onto this even for the single node inference as well. So it has um LLM uh workload aware autoscaler support which

is WVA um workload variant autoscaler coming from LLMD and the underlying or orchestrator is the leader worker set from So getting everything onto a single slide I think this is just one slide that um I want you guys to take it back home. Uh so on the top you have the envoy gateway. Um this can be any gateway API compliant networking stack. So ju just as a

reference we have taken oni gateway which provides a unified API for uh external clients to connect to um uh be it uh openai or any any other protocol that is supported. Uh it has uh token level uh uh rate limiting, token level usage metrics uh intelligent routing in form of uh KV cache aware routing obserility for your LLM workloads be it MCP routes or standard LLM um

uh LLM request and it has support for gateway API inference extension for picking the right endpoint picker. um if you follow that uh the idea is to basically figure out what's the right VLM instance or the runtime that I have to forward to based on the KB cache and other u metrics that we have. Uh so it provides a standard interface you can have the standard uh

GI uh endpoint picker or LLMD has something called u inferenceuler for doing that any implementation that uh uh follows uh gateway API inference extension. Uh on the left you have single node inference VLM uh standard single node VLM or SD lang of your choice. On the right we have the new C which is um LLM is providing distributed inference. So you can have multi-node inference, you can

have pre-filled decode um uh kind of a disagregated uh inference as well um which can run on multiple GPU types uh using KV cache external KV cache managers like LM cache. So it can be any KB cache manager um be it Dynamo or any other uh frameworks that you have. And on the vertical you can have different autoscalers for which has support for Kada and HPA uh

model cache controller uh GP scheduleulers including support for DR and uh different model registries that you want to host the model. Okay. So, uh, QFlow pipelines is a Kubernetes native workflow orchestration engine for machine learning workloads that ships with a very accessible Python SDK and web guey. Uh, so Mike and I are QFlow pipelines maintainers, but we're not the only ones. I just want to take a

second to give a shout out to the other maintainers who could not be here today and they're listed on the slide. Um, they deserve to be here just as much as we do, if not more. Um, so we're going to break this up into two sections. First, we'll highlight some exciting new capabilities that were recently enabled and then we'll talk about some stuff that's coming down the

dump pipeline. I'm sorry. Uh, so KFP relies on an object store to pass and persist artifacts such as data sets and models. And we recently replaced Mino with Seaweed FS as the default object store for KFP. Um, as some of you probably know, uh, Mino changed their license to AGPL in 2019, which kind of locked us into an older release, and recently they moved their project into

maintenance mode. So, seaweed has a more permissive Apache 2.0 license. It's more well-maintained and provides a much simpler operational model. Above all, it elegantly solves for multi-tenant artifact isolation by restricting access to namespace specific subpaths in the context of a single bucket which or PVC which dramatically reduces complexity and simplifies uh operational overhead in addition to improving security. So we recently uh enabled uh Kubernetes custom resources

for pipeline definitions and pipeline versions which let basically let you deploy and manage uh resources in a Kubernetes native way using cube control aroc or any other githops framework that you use. It's often and it's completely backwards compatible. The API SDK and dashboard all continue to work as is and changes made through either API are essentially reflected in the other. Um on the horizon is support for

runcrs as well. So there's this new uh embedded artifact path parameter and notebook component uh decorator which provide a really nice convenient streamline mechanism for injecting logic from your local development environment directly into components in your DAG in a way that optimizes UX, streamlines iteration and simplifies testing for end users. Injected notebooks are actually automatically executed at runtime using notebook client with paper mill style parameter injection.

All KFP containers now build as m multiarchchitecture manifests for both AMD 64 and ARM 64 which enables native support for AWS graviton AMR as well as Apple silicon. And uh we also have this uh improvement. DSL collected now works for both parameters and artifacts, not just parameters. Just subtle distinction that I won't go into, but uh this is a long requested feature. You can basically collect all

the outputs from a parallel 4 task into a single downstream task. And under the hood, the backend uses BFS traversal through iteration DAGs to resolve and aggregate outputs from each iteration. And uh Mike was the contributor of this one. Um, okay. >> Hello. Yeah, I'll be talking about some of the new stuff coming down. Um, we, uh, for those of you that don't know, uh, in V2,

every component in your DAG will actually spawn two pods, which is kind of resource intensive. So, great members from the community kind of hopped together. uh this one in particular, Anton, he ended up deciding to go with uh leveraging an Argo executive plugin. And essentially, we're just going to have one additional pod that will handle all the work that the previous secondary pods for each component um

would perform on any any changes for the actual component runtime pod. essentially instead of having two pods per component for every workflow, you're just going to have one additional pod per workflow. Um, but yeah, great work from him. Um, also, uh, those that aren't familiar with KFP at all, uh, it's kind of built off of Google's, uh, Google's initial, uh, implementation of it. And they previously were

using MLMD as the backend for any machine learning metadata. Um, which they have their own schema and database usage. So a great person uh the other maintainer humare he opted for us actually deprecating it and providing a native supporting uh schema that will be uh in the back end as well as uh we'll have front-end uh updates for that. But yeah, that's to come down the pipeline.

Um, and then there's also this plug-in architecture where essentially instead of having so many contributors come in and try to, you know, blow up the the back end and trying to supply all these features, you know, it it becomes very hard for us maintainers. uh one of the things that they they opted for in a proposal was uh having this this plug-in that you can enable where

any local deployment. So if you're deploying it on EKS or you know you have your own setup, you deploy pipelines, you can also deploy a plug-in server that can actually hook into um life cycle events uh for the pot. So every component can now access whatever plug-in server you would want to set up. So you could collect any metadata that you'd want to you know store uh

be it for component inputs and outputs or yeah any runtime events. Um and with that uh another thing that's coming down is uh MLflow native plugin. So uh MLflow has great experiment tracking. It's a great open source package for those of you that don't know. um a lot of great people from Red Hat are going to work on that integration and yeah no it's uh something to

look forward to um and then another thing too is with v2 there was a separation of packages uh python packages in particular um it's as maintainers it's actually kind of burdensome to maintain all these different versions and have to deal with patches between them if any breaking changes occur from you know any PRs that come from us or the community So, one of the goals for the

next release that we're going to do for the back end as well as the front end is uh actually consolidate it back to one package and uh yeah, hopefully it'll be easier for everyone's build environment and also we're going to be migrating to a more modern uh Python build um because I think previously it's we're still relying on the setup.py stuff and yeah, we don't want to

keep going that way. Um and another thing too is uh the Qflow pipelines repo actually was storing components maintained uh by other companies or you know thirdparty contributors. Uh we've actually extrapolated that out gave it its own repository and yeah we're going to house all the components there. It is also available to the community to just contribute any uh any components that they feel like they're wanting

to um as well as just have anybody use them. So yeah, with that I'll hand it back to Valentina. >> Yeah. All right. So we are almost at the end. Um so an announcement um just a couple of days ago we uh we have a new release. So each uh component that you heard has its own release cadence and then uh what we are doing is uh

we also put them all together in what is called the Qflow AI platform which we call the GitHub is um the manifest repository uh and we it's a it's a way that you can install all together. So we do that uh in a cadence uh by quarter or six months depends. We just change that to a calendar uh release. So that was released yeah on the 26

on March. So yeah check it out. We have a lot of uh updates. So usually what we do is we add a lot of uh security and scalability um solutions on the manifest but also we integrate all the different updates from the different uh components. So with that um the last thing is you heard about uh all the different components and what uh you know how they

work. one of the the features that are coming such as notebooks 2.0 that is coming. Um so you may be wondering okay I want to explore how I can get started. So these are like great ways that you can get started. You can just watch some videos. You can join our community. On our website there is the technical documentation that it has a lot of how-tos and

technical guidance about how to install how to use her how to use it. Um but also there is listed uh our meetings. So you can join each working group meeting or our community meeting on this Friday. We also will have the outreach meeting where you can join just to learn what's going on. You can just join the meetings just to hear like hey what's going on? How

I can contribute what's happening. So if you are you know new we invite you to join us. if you want to contribute or if you just want to explore or just use the product as an end user, we are always welcome you. And the last thing uh so on Thursday we we are at the Pro Pavilion from 10 to 2 pm. So if you have any also

any other questions that we don't have time to answer today, I will invite you. We will have amazing demos uh from some of the teams that are over here. No pressure. Uh well, thank you so much and then yeah, if you have any questions. Yeah, any questions. >> Thank you for the presentation. And just one small question, can you update the uh presentation in the schedule application?

So, we have the full slides. There's a few missing. >> Uh yes, yes, yes, yes, for sure. Yes. Any other questions? >> All right. Thank you so much.