Keynote: The Hub as Infrastructure. From Open PyTorch Models, to a Safe and Perfor... Lysandre Debut

In this talk, Nander, the chief open source officer at Hugging Face, discusses the development of the Hugging Face hub as an infrastructure for making open models widely available. The speaker provides a historical overview, highlighting the transition from the initial implementations of the BERT model in TensorFlow to the establishment of the Hugging Face hub, which now hosts 2.74 million public models, 1.44 million of which are compatible with PyTorch. He explains the ease of using PyTorch for tasks such as fine-tuning and developing model adapters. Nander touches on the user demographics, noting the significant increase in independent developers and the involvement of Fortune 500 companies in utilizing the hub. The talk also emphasizes the importance of security, particularly with the introduction of safe tensors to mitigate risks associated with model downloads, and outlines future developments aimed at enhancing model deployment across different hardware environments.

So, I'm Nander. Uh, you've seen me a bit earlier today. I'm the chief open source officer at HuggingFace. And today, I'm here to talk about the hub as infrastructure uh and how we help make open models available to many. I think to start things off, it's best if we take a look at our shared history with PyTorch. Um, we've been working with PyTorch for a very long

time now. I wouldn't say we've been there since day zero, but I will say um we've been here since day one. Probably maybe you recognize this repo. It's the original repo behind the transformers repo today. And we can see that the first release or the first commit was done about eight years ago now. It was originally an up for re-implementation of the bird model which was originally

only in TensorFlow. uh and originally we only had like four or five checkpoints within this repo uh to use the BERT architecture. Quite rapidly we saw the arrival of many other checkpoints that then we would merge directly within transformers. We wouldn't have a hub or anything until a bit later. So we would accept PRs with model cards. We would merge them. We would host uh the checkpoints

themselves them themselves sorry on our S3 bucket. Um and and that was about it. A year or two later, we started with the very first version of the hugging face hub that I'm sure not many of you here know. It has evolved into this that maybe more of you know now. Um, and one number that I want to take a look at today is the 2.74 uh

million models that we have right now on hugging face. Those are only public models. There's a large number of private models as well. But out of this number, 1.4 4 million of them are PyTorch compatible. And so PyTorch compatible doesn't mean that you can sorry that you'll have to download the model, convert it to use with with PyTorch. It means that it runs PyTorch out of the

box that is very easy to use within PyTorch and very often it's with safe tensors that it's built. And one of the reasons why we see so many PyTorch checkpoints on the hub is because PyTorch is very um malleable. It's very easy to work with. So you can fine-tune, you can do adapters, you can do quantizations and so you end up with 240,000 PET models or adapters

which are very easy to use. Same with fine-tuning which is also very simple to do. I think a good thing to take a look at is the evolution of the usage across the hub since 2022 to today. In 2022, we had about 70% of users that would use uh that would be coming from industry. So those are labs, companies, entities. Uh about 17% were independent developers and

13% were researchers, students and so on. Since then it has evolved significantly. The number of independ independent developers grew significantly um and so that's why we see a lot of usage within quantizations um within adapters and so on. But it doesn't mean that the industry part shrank. Uh if we take a look at the amount of companies that use hugging face today, it's still massive. It grew

actually from 2022 and we have 30% of Fortune 500 companies who are using PyTorch checkpoints on the hub right now. And so I know what you're probably all thinking is that is that if we have 2.7 million checkpoints on the hub, um how many of those are actually being used? Because I upload checkpoints on the hub every day. I'm pretty sure no one uses them. Um, and

actually this is partially true. Indeed, the top 200 models account for 50% of the downloads on hugging face. So these are the gemas, the llamas, the mistrals. Um, and there's a long tale of usage afterwards. But actually out of this long tale of usage, 50% of all models on hugging face, so the top 1.4 million models still get at least 200 downloads. So while 200 downloads isn't

necessarily like wide usage, it still shows that there is a lot of interest in all models. There's a lot of trial and errors. There's a lot of testing going around with those models. And so with so many models on the hub, one key thing that we have to think about is how do we make sure that this is secure? Out of the 4.47 47 million revisions or

like model uploads that have happened on the on the hugging face hub. We've seen that about eight to 9% can contain malicious files. And so we've been working with uh those entities at the bottom that I've highlighted to identify to scan the models on the hub to just ensure that um they're safe to download and if not so that we can notify the user, hey this is

probably unsafe, maybe you should be careful about it or just block the download altogether. And one of the ways where we patch this is with the safe tensors library. So safe tensors allows you to save just just tensors to ensure there's no possibility of remote code execution happening um and no denial of service attacks either. And so that's why we're super excited today to move safe answers

to the PyTorch Foundation. We really think that in order for it to become ubiquitous for it to become adopted by the field at large it really needs to be worked on by many many more people not only on the technical side but only on the also on the governance side. So um in terms of direction and strategy um I'm going to go quickly over the road map

uh and also what you may expect to change and what you may expect will not change. So of course the governance will have a formal steering committee now uh stewardship it's not going to be led just by hugging phase but by the community as a whole uh so community ownership and we're going to uh look at how we can integrate directly within PyTorch as well. So having

safe tensors be a serialization option directly within PyTorch. What will not change though is the format itself. You should not expect any breaking changes at all. If you're using safe tensors today, you can use the same safe tensors tomorrow without any changes. The pace of iteration, if anything, it will probably speed up with all with all the contributions that we're expecting and not slow down. And new

data types is a is a bit of a teaser into our road map um which is the following for the following few months. So with um layer loading and checkpoint loading in machine learning, you have a lot of possible optimizations that you can do to really optimize it across your different workloads. And one of them is the device aware loading, making sure that there's no CPU staging

happening, making sure there's that there's no additional copies that need to happen. So this is something that we want to focus on and that we want to enable for also existing and future devices. Quantization and data types is one important component right now in ML. Um I don't think anyone uses FP32 only right now. So if the ecosystem uses some additional data types, some additional quantization types,

it's important for us to support it as well. Support it as well within the safe tensor framework. Distributed training and inference as well. Uh tensor parallel pipeline parallel can really be optimized if you ensure that the loading part of the checkpoint isn't messing around everywhere. and security and community of course is uh the key point that safe tensors tries to bring forward. We've had safe tensors audited

in the past, we're planning on doing it again as we continue adding some additional features. And so knowing all of this, knowing the hub, knowing what we've done on security side, we can somewhat say that distribution is solved. Now this is of course a work in progress. I wouldn't say I wouldn't say it's completely solved. But one thing that we've been taking a look at right now

is execution. And so how do we work on making sure that execution is as well handled? Deploying your model on a specific hardware is actually not super easy right now. Uh there's a lot of different targets that you can optimize for and usually if you're going to optimize a given model to a given hardware is going to result in some uh additional very specific code. Now of

course there's a lot of DSLs and there's a lot of nice tooling that makes this much much simpler. Um, but we're trying to work here on the infrastructure that allows doing it and that allows using this in a much simpler manner for our users. This is the ideal scenario. You're using a dense model on a Nvidia GPU for example. Uh, you want it to be optimized. So,

you just download or use the kernels that are optimized for it. You only need one version of flash attention that works for your hardware and you don't need the additional um kernels for as you're using a dense model. If you're switching to an AMD machine, uh, then it's the same. You just use the available kernels and you don't need any additional kernels to be installed. And so

this is the original statement of the kernels library that we've released last year, which allows you to build some kernels, upload them to the hub, and then use them locally uh by mentioning which kernels you trust um and just using them within your models to directly speed them up. Um, one thing u that we're going to do now to really help this move forward is that today,

actually starting today at 9:00 a.m., so probably 10 minutes ago, we've released kernel pages on the kernel hub uh to allow you to browse different kernels specified to thank you to to allow you to browse kernels on the hub directly to see the hardware versions and the software versions with which they're compatible. And so this is one example of the SGLang project pushing one of their flash

attention specific kernels that they're using within SGLang right now as well. If you're interested in learning more about this, we have Ben Burton Shaw uh from Hugging Face that's going to talk specifically about how to leverage agents to build kernels to then use directly within your models a bit later today at 3:25. So, we've been building uh everything around hugging face with PyTorch for the past eight

years. I think uh it was the right bet and we're ready to continue with it. Thank you very much.