Write Once, Run Everywhere with Pytorch Transformers - Pedro Cuenca, Hugging Face

This talk discusses how the Transformers library serves as a reference implementation for various downstream libraries, emphasizing the transition to a PyTorch-only framework to streamline development and enhance user experience. The speaker highlights the importance of modularity in the codebase, allowing for easier contributions while maintaining a singular file structure for model architectures. They introduce new abstractions like an attention interface and the implementation of one-click support for model loading across different platforms. Additionally, the speaker touches on the impact of AI tools in contributing to the codebase and the necessity for rigorous review processes to maintain high-quality standards. Future goals include expanding support for multiple libraries, ensuring swift and efficient access to models across the ecosystem.

We're going to talk about how Transformers is the the reference implementation for a lot of downstream downstream libraries. Whenever a new implementation comes in Transformers, it's taken from there and and and ported or adapted to a lot of different very different frameworks and libraries. And we have been we have been experiencing that and we are actively helping that happen and accelerating and accelerating that. So, our goal

is that any new model added to Transformers becomes immediately available or as soon as possible in in all the other downstream And a big reason why this is happening and a big reason why we are accelerating this is because we decided to go PyTorch only. We removed support for TensorFlow and and Jax. And this is not only about recognizing that PyTorch is the most used framework in

the community. It's also about giving us tools to develop faster and to incorporate abstractions in in an easy way without having to complicate abstractions too much. So, going PyTorch Sorry. Is one of the of the key components of this of this strategy. one of one of our design goals is to make and one of the reasons why I believe that all these downstream libraries use Transformers code

is because we are very careful about providing code that can be easily understandable. We are very adamant about the fact that a single architecture resides in a single file and in that way a practitioner can understand the architecture just by reading the file top to top to bottom. That's something we started from the very first moment when we started doing the Transformers code base. And this is

something that has really helped drive adoption of the of the library. However, many years have passed and there are a lot of components like I don't know rope or KB catching or attention interfaces and and so on that repeat more or less from model from model to model and they add a lot of craft to the modeling to the modeling code. So, we found that PRs to

Transformers tended to be large and slow to get accepted because they had all these components interrelated and the reviewers had to review thousands of lines of of code. So, what we did was something we call modular, which is a trick whereby the contributors can focus on the on the diffs, but we still provide the full one file model implementation so that our contract to give an standalone

implementation for practic for practitioners still holds. So, that's something we built. We also built a a few abstractions, but we try to do them carefully. sorry, I'm getting a call. My son my son is calling. Sorry about that. We are building a couple of abstractions, but we try to adopt abstractions very very very carefully so that they don't get in the in the in the way of

of modeling files. One of these abstractions is the attention interface. And with by adopting a single interface, a single signature method, it unlocks a lot of interesting use cases that we we will see in a moment. The same thing happens for parallelism where parallel behavior is defined in configuration files and you don't have to update modeling code to take parallelization into into account. the first two are

mostly intended for contributors to the to the code base, but we also have tools for the for the users of the code base. Nowadays, people can, for example, download um hardware optimized kernels for their devices and they're they're using. So, we are aiming towards a near zero installation experience for for users and and optimizations. On this line, uh please do not miss Lysander's talk to tomorrow in

which he'll reveal something something about about this. Okay, so the modular system is about making it easy to contribute to And it looks like inheritance, but it's not quite inherit Here, for example, um you can see that the that the author of the All Model is is expresses that the norm is the same as the as the as the Llama one. And a few lines below, um

the code block says that uh it uses similar attention to Llama attention, but there are some some differences to it. This is not uh there's nothing All Model and Llama. This is This is based on the fact that the code looks alike. So, we reference code blocks in the library that were already there. So, this looks like we have tooling to translate this simple sparse uh implementation

to a full fledged modeling file with many Um as I said uh we are super fortunate that Transformers has been adopted by by many um many libraries in the in the ecosystem and many and many frameworks as a as a reference. Uh but this means that they have to wait until the Transformers implementation is ready and then port those implementations to their to their own to their

own frameworks. we have tried to improve this somewhat by creating closer integrations with some libraries like some important libraries like vLLM and SD Land uh where the purpose is that they can use the Transformers implementation, but they can still um bring the unique characteristics of the of those of those libraries. So this is possible thanks to the closer integration with them. And the next step is what

we call one-click support, which is agentic methods to help um perform these conversions efficiently in a way that also provides confidence to to the reviewers that that the code is is correct. Uh to give a bit more context um these are um some of the libraries and frameworks that use Transformers as a as a reference. They cover all the spectrum of of tasks from from serving like

uh vLLM or SD Land to training and quantization to on-device execution like MLX or llama.cpp. can we do better than just wait for those libraries to to to port the models from Transformers? As I said uh we have been working on some closer integrations and for example it is possible now in vLLM to load a Transformers model just in in a few lines of code and expressing

that the back end execution back end is going to be Transformers. The same thing is possible for SGLang. We can download a Transformers and use a Transformers model in in SGLang. We can start an OpenAI API compatible server with with a single command in vLLM for an LLM or a VLM. And we can do the same with with SGLang. And the trick here is that the model

implementation comes from Transformers. But it gets augmented with all the cool features that vLLM and SGLang provide. How does it work? It's This happens when when model co- Hello. This happens when model adhere to a to a particular contract. for example, they use attention interface signature and attention interface method with a given signature. And they declare that they adhere to this to this process. So, what this

allows is to have a register of attention alternative attention implementation so we can swap them at at run time. For we can use SDPA or flash attention from PyTorch. Or we can bring the implementations from vLLM or SGLang when when the model loads. This is way simpler than having to port the models to those libraries. At the left side, we see a vLLM partial vLLM implementation where

you can see that it uses classes from the vLLM framework. And the right hand side is much simpler. There's nothing to do. VLLM registers their own implementation, their own attention implementation as one of the possible options in Transformers, and then it can load a model and use that attention implementation. Okay. Does this matter when agents can now write code and they can do everything for us and

and we are free to no longer write code? It's true that we have seen a lot of um PRs open to the Transformers repo. I think that's because one of the most most popular GitHub repositories. So, there are lots of well-intentioned people that wish to become Transformers contributors. So, they have an agent that that can open PRs. So, they ask the agents to open PRs. And those

agents do open PRs. This is not exclusive to Transformers. We are seeing the same in other domains such as the App Store, for example. Everybody can submit an app an app. So, many people do submit submit apps. So, should we be accepting those PRs and get done with it? The answer, unfortunately, is it is it is it is not. It's I think it's completely okay for some

code bases uh where the code where the code looks like does not matter a lot, but Transformers is not one of those of those of those libraries. And the reason is what I said before. And the reason is that we are very careful when creating code and we design code to be read by humans. we try very hard that a human reading an an architecture implementation can

understand it top to bottom. So, we need to keep that because that's and the basis of all the use of Transformers across the the down all the other down down stream libraries. Um as you know agents are very willing to help and they offer to refactor things and they offer to bring best practices from the average software engineer project. But those best practices are not applicable in

the most part here because they break all these hidden contracts all these hidden basic contracts that all the community knows about. so we can't accept those PRs. But we think we have to learn how to how to work in this new environment and we think we have to understand what open source means going forward and how we are going to keep collaborating together. So we have been

doing a couple of experiments. One is that the release notes for the Transformers repository itself it's now LLM generated. And this is saving Lisandra a lot of But it's not about the time it saves him. It's about we can achieve better quality with the LLM generated version than than without. And and and the reason is um that we are sometimes biased and we tend to focus more

on the latest trend and we forget details about PRs. If you see the release notes for Transformers you'll see there are dozens of PRs for for any given release. So we may miss details and the and the LLM is very thorough and very comprehensive and and covers all that. So we only decided to release this when we thought that it was better than what we were doing

manually. And I have to say that the release notes from Transformers the manual ones are very very good quality. But these are a little bit better. Another experiment is Niels, who is a very well-known contributor in and he has written a lot of ports for Transformers models. He focuses on computer vision. So, he has done some experiments to use Codex to contribute to contribute models. And it

works. It works because he's an he's an expert. He knows what a port involves. He knows how to prompt the model. He understands when the model is going off. And he knows how to steer him how to steer to steer the model back into track. And he knows how to determine whether the code that the agent produces is is is looks okay and and and and is

actually correct. So, this is a great tool for him as an expert to contribute models to to to We want to extend this uh further. And we have been experimenting with Transformers to MLX conversion skill, which we are releasing today. And the goal here is very simple. We want to capture all the knowledge about the Transformers code base, all the knowledge about the MLX the MLX code

base, and all the process to convert from one to the other. So, that the goal is that the user issues a simple prompt like convert this architecture to MLX. And the skill knows how to do a lot of stuff. It knows how to find models on the hub and that conform to that architecture. It reads the configuration files for those models. It understands the the architecture points.

It focuses on the salient architecture novelties and then it brings and tests for those specifically. It runs long sequence generation, which may uncover subtle bugs in robe on what not and it does a lot of things to make sure that not only the code works but also it adheres to the standards of the MLX code base and also correct because multiple multiple tests are are are passed.

And when all these succeeds it offers the user to open a PR and the PR includes a comprehensive report with with all the things that the skill tested for. In the in the in the top section you can see for example a layer by layer comparison of logits between the MLX implementation and the original transformers implementation. That's something the skill did on on its own. There are

robe tests, there are long sequence generation tests, there's a many things. And the important thing is that this is not only a tool for contributors who must own the PR and must understand it like Neils does in order to engage in a in a review cycle but it's also a tool for the reviewer as well. The PR is long but it's not just a wall of text

explaining what it did. It's also data with all the tests that were done to ensure that that the implementation is correct so we believe that code bases like transformers and MLX will still go through the standard review process and we want to use agentic tools to support In terms of testing we also have a separate test harness separate it's a non-agentic test harness when a when a

PR is open to the MLX code base using this skill it downloads the models and it tests them following some procedural test classical tests so to say not LLM generated ones. And we do this to provide additional confidence to the reviewer that the test results, for example, have not been hallucinated. And also to provide reproducibility. Anyone can download those tests and run them in their own machine.

And it also provides documentation. Everybody can see what the skill did and how it worked in real life bypassing some some tests. And of course, it makes sure that it adheres to priority idioms. What's next? We want to extend this. We have been focusing on MLX LM, which is about language models. Just to restrict the scope to something manageable. We want to extend the same idea to

MLX VLM and we are going to be working closely with with Prince, who is the author of the of the package to to make this happen. We are also going to work on Llama CPP. We'd like to at least. It's a big project, but I think that limiting the scope of what the agent can do can can achieve great results, for I'm particular particularly interested to see

if all the image pre-processing code, which is very different the way it happens in Llama CPP versus the way it happens in Python code in in Transformers or PyTorch code. The code is quite different. And there might be subtle differences in the results. So, I think this is an area LLMs could could be could be helpful. And so, that's something we want to explore. And with these

experiments, our goal is to get closer to this ideal of once a model is available in Transformers, it's available everywhere else very quickly and very very easily. And that's it. Thank you.