Project Lightning Talk: Still Burning GPUs On Debugging? Scale AI In One Line - Anna Kramar

This talk covers the challenges data scientists face when scaling AI workloads, particularly when transitioning from local environments to Kubernetes. The speaker, Anya Cramer, a software engineer at Red Hat and Kubeflow SDK maintainer, introduces the Kubeflow SDK, which offers a set of unified Python APIs designed to simplify this process. By using Kubeflow, users can easily run and scale their training functions across multiple GPUs with minimal changes to their code. The session highlights features such as local and containerized backends for development, hyperparameter optimization, model versioning, and planned enhancements including workflow orchestration and observability integration. The speaker also shares the project's growth, noting significant interest and downloads of the Kubeflow SDK.

Hi everyone. Uh, today I'm going to talk about how you can um make easier to run any AI workloads at any scale. And with that, my name is Anya Cramer. I'm a software engineer at Red Hat and I'm Kubeflow SDK maintainer. So, instead of just introducing the project to you, uh, let me start with introducing the problem that most data scientists are facing um and how we

are solving it. So, imagine you're building your training function and you run it locally, it works and now you need to scale it. You need to run it on Kubernetes. And that's where you start facing all the infrastructure issues with pods, CRDs, YAML files. You need to debug your pod scheduler and so on. So, you end up spending a lot of your time on debugging instead of

building or improving your AI model. Um, and every fail costs a lot of money because of the GPU hours you spent. So, um is there a better way to run AI uh on scale? There is. We are building Kubeflow SDK for that. So, Kubeflow SDK is a set of unified Python APIs um to run any AI workloads at any scale. You can access that with pip install

Kubeflow. Um, and let me show you how that works. So, here's you have your training function and you can run it locally using trainer client. Um, and then whenever you're ready, you can scale it and distribute it across different GPUs uh with few modifications. So, first of all, for local execution mode, we are using the backend config, which is local process backend config. That means that your

training function will be running as a sub process. For distributed training, we are skipping this part because we are using Kubernetes backend um as default. And then we are providing um training function for both cases, but also for distributed training, we are providing the number of nodes and resources per node. Uh so in total, we are going to distribute our training function across eight hundred GPUs. So

we just added a few lines and we needed we didn't need to make any modifications to our training function. Um so this is my favorite part is that you can change your environment variable really easily with just one line. So as you've seen for quick prototyping, you can use the sub process config. Um you can also change it to the um containerized backend uh such as Docker

or Podman uh for your local development as well as whenever you're ready to scale it, you can use Kubernetes backend. And you didn't need to change any um function or code in your training function. You just changed one line and that's it. Um so that's one of the cool features we have in the Kubeflow SDK. Uh we have more. So obviously training a client as you've seen

for fine-tuning or distributed uh we after training, you probably want to optimize your hyper parameters. For that, you can use hyperparameter optimization with optimizer client that is already available uh in Kubeflow SDK. Uh also for model versioning and artifact storage, you can use model registry client. And we just recently added a spark client for um distributed data processing um on Spark. So everything is straightforward. Feel free

to use it. And we have many more features coming in in Q 5 SDK. So, we already working on including pipelines client for workflow orchestration and also feast client for feature store. As well, we already working on integration with open telemetry for observability and MLflow for experiment tracking. We also planning to build MCP server so AI agents can't operate the Q flow. And we have more coming

in. But also before I wrap up, I just wanted to quickly mention that we recently hit 140,000 downloads for recent months of Q 5 SDK. We are super excited about that. Thanks everyone who contributed to Q 5 SDK. And that's the project that is growing fast. while we are scanning these QR codes, one of them is for Q 5 SDK website. Another is the blog post that

we recently published for release. Feel free to reach out to us throughout this week. We have a Q 5 booth on Thursday B8B. We also have a maintainer session tomorrow. And you can reach out to us on the Slack channel on the CNCF called Q 5 ML experience. We also have a bi-weekly meetings. And thank you so much for having me. Awesome. Thank you very much.