Bridging the Hardware Gap With Code Harnesses on the Hugging Face Kernels Hub - Ben Burtenshaw

This talk covers the integration of coding agents with custom kernels to address the growing demand for optimized machine learning operations. The speaker discusses how agents are increasingly used to tackle complex problems closer to hardware, particularly through the use of custom kernels in PyTorch. A key aspect of this integration is the establishment of a standardization layer within a hub that allows users to pull pre-built kernels without the need to compile from source. The session highlights the importance of optimizing for hardware-specific operations to improve performance metrics such as compute, memory, and overhead. It also addresses the concept of agent skills, which guide the generation and management of these kernels, and how users can easily incorporate optimized kernels into their PyTorch workflows. Lastly, the talk emphasizes community-driven contributions to the kernels hub for shared learning and development.

Hi everyone, I'm Ben from Hugging Face and the talk that I'm going to give you today is titled Bridging the Hardware Gap with Coding Agents. And is we're seeing that people are using code agents more than ever. I think everyone's experiencing that. And they're tackling harder and harder problems. They're kind of going deeper into the stack, closer to the silicon if you will. And we're not getting

the impression or at least I'm not getting the impression that people are tackling the same problems just faster, right? People are looking for harder problems. Uh and they're using agents together. And that's kind of the context that I want to talk about custom kernels. Because custom kernels have been an extremely popular task to use code agents on. If you look at leaderboards and recent hackathons and papers,

you'll see that. So, there are two main threads to this talk. One, we can use code agents to write custom custom optimized kernels. We can use them in PyTorch. However, we need a standardization layer and we have that in the hub for custom kernels with native PyTorch integration. So, these two threads run through the talk, basically. massively connected and important. So, in short, because if we want

to go ahead and and generate all of these kernels and optimize models for every single type of hardware, we're going to need a standard layer in order to distribute them and use them and pull them into our PyTorch code. So, for a while, custom kernels were seen as this kind of unattainable goal of uh for coding agents, right? It was seen as something beyond what the humble

agent could achieve. But if we look at uh hackathons, as I said, like GPU mode projects or papers like kernel bench, we'll see that more and more kernels are being generated in DSLs like like CUDA, but also in more niche DSLs and kind of on local hardware. And we've also tried this out ourselves and I'll share some resources for you if you want to try it out.

And and that's really great and really fun and and it's amazing that the community is getting uh interested in such challenging machine learning tasks. But it raises the question, how do we use these kernels? How do we distribute them and how do we get them into PyTorch, basically? So, the TLDR of the whole thing it is this. That we can pull kernels from the hub and use

them in PyTorch and we don't need to build from source. So, if we're generating kernels with agents, we can pull them directly into PyTorch and the hub becomes our distribution layer, our standardization layer. This is what it looks like on Hugging Face. It it's a repo with the same standard as a with the same status as a model or data set. The kernels are defined in a

a standard which I'll go into. They're attached to authors or organizations just like models. There's metadata that exposes which hardware they run on uh relevant to your own configuration and your own profile. But I'll come back to all of this as I work through the integration. So, let's start from scratch. I'm sure many of you are are familiar with with kernels and and how they work, but

let's start from zero. When you use a PyTorch model on a on a GPU or any accelerator, the actual work is executed through a through a kernel. These are low-level hardware specific functions that are designed to interact with that with that accelerator and they'll typically change from one generation of accelerator to the next. And different models will use them in different ways. Then the question comes, how

do we optimize these kernels? So, when we think about efficiency in machine learning, there are three main attributes: compute, memory and overhead. The compute is the time spent doing math, right? The matrix multiplication. The memory is the time spent moving tensors around, typically from one type of memory to another. And the overhead is everything else, the Python environment, the PyTorch dispatch. And most people would assume that

compute is the bottleneck here because it's kind of um yeah, using up all these flops. However, that's not necessarily the case and often memory is the main bottleneck. If we take a GPU like a H100, it can do a petaflop per second of computation, but memory bandwidth is about uh 3 terabytes a second. So, that's a 300 to 1 ratio. What that means practically is if your

operation uses fewer than 300 flops per byte of data, it is a memory bound process. And the GPU will be sitting idle waiting for tensors to return. So, many of the core operations that we use like layer norm or GELU, they're all memory bound on most devices. And even if the GPU was a 100 times faster, the data wouldn't be ready. Fortunately, there are solutions to to

this problem in most use cases and from from science and from industry. Uh and what they generally do is increase the arithmetic intensity of any operation. So, instead of reading data and performing an operation and writing it back, they read data less times and perform more operations at once before writing it back to a slower memory. In short, they they spend less time so they spend less

time computing the tensors. Or they they they keep the GPU warm, let's say that's something that's often said. By batching work together. So, flash attention is the is the poster child of this. It makes an O of N squared problem into an O of N problem. Uh and it's used in um yeah, almost all machine learning libraries using LLMs. As I said, it's a poster child. However,

on some systems it can take 2 hours to build from source uh with procedure that's specific to the install matrix for hardware, for CUDA versions, for Python versions. and this is is a common experience for machine learning engineers, right? This process of installing flash attention and other comparable custom kernels. Whereas if we have them as a binary, we can install them in a couple of seconds. And

that's what the the kernel hub brings. To use a kernels hub, we use a library named kernels, which I'm sure you've already seen. And it has a a unified and predictable structure. So, we have a a build.toml, like a like a Python project even, which will define the hardware that the kernel works on and the generation of hardware that it works on and any kind of software

compatibilities. There's a C source and a torch exterior file, so you can have torch bindings and your DSL code in separate directories and it will always be consistent. uh it is reproducible because it uses Nix to build kernels on different hardware, so you don't necessarily need access to the given in order to run the It has native PyTorch compatibility as I showed you earlier and so you

can either pull kernels in a function or register them as decorators. It's also compatible with torch.compile. You can share with the community on the hub and just by pushing through the library. And yeah, if you want to go check out on GitHub, there's a repo huggingface/kernels. So, that repo has two halves to it. The first half of it is kernel builder, which is a tool for building

kernels for your respective hardware. So, this uses Nix to create reproducible builds of your DSL code and your your PyTorch bindings. Most people will use kernels, the other half of the library, which is a is a Python library for pulling these kernels and using them in in PyTorch. It handles downloading the right wheel. So, because we have a standard, we know the location of the right wheel

that we should be using on a given hardware. yeah, if you want to check out kernels, the best place to go is is docs/hf.co, docs/kernels and you'll you'll check that out. And this is what these kernels look like. So, they have predictable structures, they have a build.toml, uh as I said, and they have um the C source and flake.nix. And all of this is pushed to the

hub and mapped to a a UI. Which as I showed you, looks roughly like this. what we have here is a representation of that same YAML uh and then we have a a readme just like a model, which can be populated with a kind of custom spec and then a representation of the metadata for the given hardware that the user defined. On the right-hand side, we have

like the the torch version that's that is defined in the same build.toml, uh the OSS that's defined in the build.toml. So, as you distribute these kernels, you have this representation so the end users know um what your kernel's for. So, let's take a look at that inside So, the the most straightforward way of using a kernel from the hub is to um pull it directly into PyTorch

code. So, in this example, we don't need to use the the functional example. We can just use the we pull it from the hub and we refer to the repo ID, so in this case kernels community activation. And then we stop using the functional GELU and we just use the kernel like a a module and we can operate with that. So, that's the most basic way to

get started. Um and then we can also use it as a as a module. So, in this example, we have two main functions. We have use kernels forward from the hub and kernelize. So, we we use kernels forward from the hub to refer to a specific operation, so in this case silu and mol. And then at the end we'll use kernelize. So, we define our module just

like normal and we have a forward without any change to it. And then once we've defined that module, we wrap it in the kernelize and the kernelize function will replace all uses of silo and mole with this kernel from the hub. And how does that that work? Before that, we define a a kernel mapping that goes from silo and mole to specific kernel layer repositories referred to

on the hub. So, let's say we have a layer repository for CUDA that we want to use and another layer repository for Rockem that we want to use, then we can define those separately and otherwise we'll fall back to a native PyTorch. Likewise, in transformers, many of the model authors are already doing this. So, we can refer to use kernels and we'll kernelize those modules and those

models as we use them in transformers. So, in many cases, if you want to use this, you can just use transformers and refer to a specific model. before we go back to agents, let's just take a look at that from a from a high level again. So, because this is really where the the hardware gap is being bridged between architecture, model architecture compatibility and and hardware. So,

if you take a given model released at a a given point in time, what you'll see are optimizations for hardware at that point in time. You might also find certain compatibility across hardware vendors that are related to the model author or were motivated by partnerships or community interest. Invariably, this doesn't cross time and it's not maintained perpetually. And it doesn't cross vendor boundaries. So, the end user

has to work to set up these kernel optimizations. So, what we end up with here is a a shared resource on the hub to share with models. So, in effect, we have a a set of kernels that are aligned with a model architecture and then aligned into Let's now take a look at agent skills. Uh and this is an example of how we used agents to generate

kernels. if the easiest way to to describe skills, and I find that they're an extremely hyped topic at the moment and probably um uh yeah, have more hype to them than they necessarily deserve. But in simple terms, they're really just prompts in file format or context in file format, you might say. And so, because of that, they have all the advantages that we know and love from

files. We can load them at certain points in time. We can version them. We can source control And so, the agent can do the same thing. It can progressively disclose parts of of the skill at certain points in time as it needs them. They're also a way of taking a zero-shot task and turning it into a few-shot task. So, we can give the agent specific examples of

processes that we want it to follow, like instructions and scripts that we want it to use or configuration. And it will follow these and not generate new or or hallucinate new alternatives. It'll instead rely on these. So, in the context of kernels, we can give it examples of AMD kernels and and CUDA kernels and we can give it scripts and utilities to to benchmark those and to

distribute those. In practice in practice, they're markdown files and and scripts and any kind of example or configuration in a directory. At Hugging Face, we're focusing on packaging skills within projects so that the skills are maintained by the project authors and not necessarily separately. So, just like documentation or code itself, the skills are updated with versions of the project. And we think that that's the best way

to do it. We have some skills that are outside of projects, maybe because they're more experimental or they're more high-level. But the core skills to work with a library are maintained within the project by the authors of the project. So, this is what the the kernel skill looks like. We've added it into the CLI with as as with all the projects, so you can install it with

kernels. So, if you're running kernels anyway, you can just add So far, we have skills for AMD and Nvidia, but we're open to contributions more. And you can add more skills if you want, like if you have custom skills for a specific DSL. That's quite good to combine with kernel skills because it focuses more on distribution and packaging for the hub and not so much on the

DSL side. So, let's say you've got a a CUDA skill, you could add that in with these and it would work quite well. Or maybe a metal skill or something like that. and so, yeah, these mainly focus on hub integration and benchmarking and packaging. So, we tried these out we tried we wrote a blog post on this. I'll share a link at the end. And we tested

this with Qwen3 8B and and a a diffusers model LTX video. In this example, we increased throughput by 94% which was a huge optimization for this model on that hardware on H100. This isn't a a state-of-the-art optimization of that model on that hardware by any means. It was just wasn't supported by that model. And so, in many cases, that's those are the kinds of optimizations that end

users can make, not necessarily pushing the needle, but adapting models to the use case that you intend to use them on or gaps in the compatibility matrix that don't exist for for one reason or another and to kind of optimize your workloads with Uh and I'll just give you share a a short video on on what that looks like. So, as I said, you you can do

kernel skills add and then you uh refer to your specific agent. In most cases, you can use uh uh you don't need to pass an agent because Codex and Open Code are compatible now. But if you use um cloud code, then I think you still need to. You'll see that it's a directory and it contains a skills definition. You can go and check that out on the

blog post. You can then start up your agent and refer to that given skill depending on the agent. So, in Codex, it's like a a bang notation and then the skill name. In all of the examples that we've done, we've been pretty verbose with the prompts that we pass, even though the model has a lot of information in the skill, it it's best to refer to it.

Here, we skip to the end and you can see that the agent has performed the task and it's pushed a hub a kernel to the hub. Uh and it's returned a snippet for you to use in in PyTorch code. And then we also because I didn't have a H200, uh I can also run the kernel on Hugging Face on jobs. And so, that's what we did there.

We exposed it by MCP. And and this is my kernel that I published on the hub for Qwen3 8B on H200 architecture. sorry. Okay. So, that's the the overview of how that works. So, just a couple of takeaways really. If you want to write kernels with agents, the best thing that you can do is to run this kernel skills add command and you'll get these skills inside

your agent context that you can use to push your kernels to the hub. It won't give you so much around the actual optimization itself, but it's a really good base skill that you can use with others. If you want to use your kernels or other kernels inside PyTorch code, you can import the kernels library and the get kernel method or the module method that I showed you

earlier. And then you can distribute all of these kernels on the hub as repos and this will be coming out soon. If you want to go deeper onto any of these, I'd recommend checking out hf.co blog. There's a number of blog posts that I and colleagues have recently published about this, like learn Hugging Face kernels in 5 minutes. But we've also got more agentic-focused blog posts where

I generate kernels using the skill I just showed you, where we train models with agents and various other things. And they're a great place to go and check Cool. Are there any questions? It's one there in the middle. I think you need the mic. Can you talk about the safety mechanisms that you put in place for the kernels? Yeah, so the hub has built-in safety mechanisms for

models. And kernels use the same safety mechanisms as for binaries as as models do. Um So, if you want to check that out, you can go to the docs. We've got a page on security and safety and and how that works on for each repo and the checks that we do and the checks that you can do to optimize safety for your hub. It is a pretty

detailed problem for various different frameworks, yeah. But it's a it is a is a key issue. Thanks. Um I have a question about um what kind of kernels like the does the maybe the community need or in general, like let's say I have a kernel that I've implemented. And what does it take for it to get into to the repo? Like are there some specific kernels that

that are needed or Yeah. Yeah, it's it's a really interesting question. I guess it's a question that I ask myself a lot and it probably relates to specific use cases. So like I I kind of said in the talk, if you're working at the frontier of of model and hardware, the demand for kernels is mainly tackled by industry. But if you're on the fringes of those things,

then there's a lot of interesting and important kernels for specific use cases. So like let's say like local models, getting local models to work on specific devices, like laptops, maybe on metal, maybe with AMD chips, and optimizing them for specific use cases. These kinds of things are really interesting and and really important. Uh and you can with the kernels hub, you can share those kernels like on

the hub independently. You don't necessarily need to build up a project with a status of say flash attention and kind of maintain that and and solve every problem. You can really just focus on like like one optimization that works, let's say on a given generation of a given manufacturer, and fix that and then share it and and and from there kind of work out. So it means

that yeah, we can tackle smaller problems on the fringe. I I I think that's the kind of real win for the community from kernels hub. Thank you. Cool. Thank you. >> [applause]