Build PyTorch to Understand PyTorch - Vijay Janapa Reddi & Andrea Mattia Garavagno

This talk introduces Tiny Torch, an educational project designed to help learners build their own machine learning framework from scratch using PyTorch. The speaker emphasizes the importance of understanding the underlying mechanics of machine learning, as opposed to merely using pre-built layers and training scripts. The project aims to educate a new generation in efficient ML practices to address rising energy consumption from data centers. Tiny Torch's learning path is structured into four progressive tiers that cover foundational concepts to advanced topics such as transformers and model optimization. By recreating milestones in ML history, learners gain practical experience while engaging with essential coding challenges. The open-source platform promotes community involvement and provides resources for learners and educators alike, ultimately striving to create knowledgeable AI engineers.

Hello everyone. I'm Andrea Mattei Garavagno from the University of Genoa in Italy and I'm here to present build PyTorch to understand PyTorch. But before starting, there is a little story. In theory, we should present this together with Professor Vijay Janapa Reddy, which is the inventor of this uh project. But unfortunately, there were some issues and so we thought, "Okay, you can pre-record a video. I can just

play it and then talk after you." Then, this morning I discovered that there was no video at all. So, you have to stick with me. I'm sorry, guys. But uh besides that, now I can start telling you about Tiny Torch. Tiny Torch is an educational project that lets you build your own ML framework from scratch, line by line. This is the philosophy. Don't import, build it. Because,

as you know, traditional ML education teaches you how to combine together pre-baked layers, you do your own training scripts and you're done. You have your ML application. That's I mean, easy. Or easier than building your own layers from scratch and doing your own training receipts. And by doing this, you create a near irreversible understanding that at this time is critically needed cuz the electricity consumption by region

from data centers is expected to ramp up by a lot in the next 4 years. So, we have to do something about this. Or at least, if we would don't do anything, we are projected to pay the highest bill. But if we teach new generation how to build ML framework from scratch, we let them know all the tiny nasty details of ML. And maybe we can try

to lower our projection and pay a lower bill. And that's the aim of this presentation and of this effort by Professor Janapureddy. And we do this while recreating ML history through milestone, starting from the first Frank Rosenblatt neural network, going through CNN, attention, transformers, up to MLPerf benchmarks, so that we take the whole ML history in this project. And we do this while using PyTorch syntax, so

that you can have a smooth transition from an educational framework to a production-grade framework. You are not wasting your time by doing this. The learning path is a divided in four progressive tiers. We start from the foundations, where we build the tensor class, our own autograd, the layers, and the training loops. Then, we move on with the architecture tier, where we focus on convolutional neural networks, attention,

transformers, and generative pre-trained transformers. And after that, we start thinking about optimizing things. We learn how to measure, how to profile workloads. And then we start to act. We reduce the memory consumption, the computational consumption, and we go on and we conclude this path with a torch Olympic, which is a capstone project that in my opinion is perfect for ending a semester of teaching. So that you

can have your end there after this. And this is for everyone. No No machine learning experience is required. Just a little bit of Python and linear algebra. And this can be for students who are willing to go in deep into the rabbit hole of what under the hood of machine learning. And it is also for instructors who maybe want to transition in a to a new way

of teaching ML systems. And also for self-learners seeking to a career change. without further ado, let's dive ahead into tiny torch. The workflow is this. You install your application and you set up with a few common CLI commands and then you start progressing in your journey, which is divided into modules. You do some modules and after some time you approach a milestone, an historical milestone, and you

run through it recreating ML history from scratch. And then you repeat the process. You iterate over and over until you reach the capstone project. And as you can see, the installation is easy. You just have to copy-paste a bunch of commands in your CLI. And we support Mac OS, Linux, Windows. there is plenty. then, after running those commands, you get this nice CLI screen where there are

a bunch of commands suggested. We can, for example, start a module, look at the system health because you can break things, and that happens. And we can have a just a look at the status of your progress. So, let's try to do this. We run Tito module status. And after a while, we see that there is a list of modules. There are 20 modules. We also have

a nice progress bar, which is void right now because we have just started. And we see that the first module, the tensor module, is ready to be start. So, we can go ahead and run Tito module start 01 to get into our first module. Here is the command. And after some time, we see that a lot of stuff happens. And after that, a Jupyter notebook is started

on your browser. So, we can have a look at this. Every notebook starts with an introduction that states what you are going to do and give you the basics to uh face those this module. And after the textual introduction, we have the code, the actual code. We give a code skeleton to guide you through the development of the ML And as you can see in the first

module, we are implementing the tensor class, which is the foundation that is needed everywhere. And there are some methods of this class like the addition, which is already done because come on, you're not interested in doing addition with tensor. maybe you are interested in doing matrix multiplication, in developing your own algorithm to perform matrix multiplication. And that's what you have to do. Here there are some comments

that guide you through the process. So there is some suggestion, there is the approach, and then you can start typing. there are also some unit tests so that after you have write so you have wrote some code, check if it is good. And if it is bad, you can change something, check again, and continue until you pass all the unit test and you can proceed. Anyway, this

is just an example of method that you don't do matrix multiplication. You also have to do other stuff in every notebook. And after you complete your notebook, you can type title module complete. that all the running all the unit test are run to check your solution. And if your solution is good, it is exported to the real code base, the code base you are building. And after

that, we do some integration testing so that we check that everything works well. And if it is the case, we update the progress bar. And after this, congratulation, you have completed your first module. And you can also share your progress with the community. And compare in this little rotating globe. And as you can see, we are almost at 700 members. And consider that Tiny Torch launched in

late December. So, in my opinion, this this is a good start. And after running into some modules, we meet the first milestone. So, if we type tito module status again, we can see that some something changed. For example, the is not void anymore. And we have completed the first three modules. So, we unlocked two milestones. And we can proceed to run the first one. Milestone are prompt

experience your code is used to recreate ML history. So, in this case, we are using the code you developed in the first three modules to reproduce Frank Rosenblatt neural network. Here we two clusters of points which are clearly linearly separable, and we want to classify them with a neural network. This is the by setting weights randomly. And as you see, we got really lucky because 90% of

the case we guessed the right class of the point. But this is just by chance. There is no learning here, just forward propagation with random setting. And these are the weights that have been used. And as you can see, here there is the diagnosis. You have been lucky. And we suggest you after this to run this script multiple times because the random seed is not fixed. So

every time you get a different outcome. And uh at the end you can always share your progress with your mates. And these are This is just the beginnings. Those are just the basics. We There is a clear progress. Then we implement autograd, layers, training loops. We move on. We do convolutional networks. Attention layer, transformer, GPTs. So we arrive up to the state of the art for education

at least. And then we move further by start profiling things, applying quantization, acceleration techniques. And we end, as I told you before, with a project. So here are some highlights from our demo For example, here from the GPTs we have the forward pass to implement, which is something maybe you are interested in. And in the related milestone, your code will be used to build a small GPT

able to reverse text sequences, remarkable for an educational project in my opinion. It's not easy to reverse sequence with small GPTs. And you are doing all of these with your own code that you have built. And then we can also have another look at the MLPerf notebook in which we have to update the KV cache in this case, another hot topic. you can speed up 11 times

the execution of your small GPT. In the case of GPT-2, the KV cache was just 1% of model size. this is a a good trade-off that most everyone is willing to accept for this. And you are doing this with your code. I want to stress this concept. So, the key takeaway of TinyTorch are that this is an educational framework for building better ML system because we want

to pay lower electricity bills, not higher. we do this while using PyTorch syntax so to a production grade framework. So, here I have a call for action for you. As you saw before, a lot of people is using ML framework to build their application, but very few are actually building ML frameworks. So, I want to you to try to transition from the group on the left to

the group on the right while trying TinyTorch. Also because you guys are using a tiny every day, so you know it very well and you can give us really valuable feedback. And also, maybe you can try to catch some bugs that are always there, of course. This is just a preview. It it hasn't been tested in classroom yet. We plan to test it in the next winter

semester. So, there is plenty of time you can engage. It's also you can contribute. It's open source. You can contribute to the GitHub. Every PR is welcome. So, please take a look at it. But, we are not done yet. Here there is more than tiny torch. Tiny torch is just a small part of a larger ecosystem where we have the machine learning system textbook from Professor Vijay

Janapa Reddy who which contains all the ML theory needed in tiny torch. So, it can be paired well and this is all open source. You can find it on a GitHub. And we have also hardware kits for physical AI for people who want to join the community. There is a a community online doing a lot of fancy projects and showcasing them online with Here there is also

a QR code that you can scan to access all these resources. please take some time to do it. and this is because it's not just about energy efficiency. It's not just about lowering our electricity bills. There is There is also reliability. There is also scaling. Cuz right now we have these amazing stuff in our hands and we're trying to see what are the limits, what we can

do with this. So we are rushing to build applications. But no one is caring about reliability, about scaling. So we are in a situation where we need new expertise. We need people that knows not only to build AI application, but how it works under the woods. Also hardware, also system engineering. knowledge. Knowledge that can be used by an AI engineer. We are trying to push AI engineering.

Something similar has been done in the by the NATO alliance with this science committee in 1968. They had the same problem in the past. There was a digital electronic coming out. Programmable logic. You could do a lot of new stuff and everyone was rushing doing new application of that, but without worrying about reliability, about efficiency, about scaling, just trying to have the next good product. So the

NATO alliance asked for an effort to make new professionals, new professionals able to build reliable, efficient, and scalable systems. And this is how software engineering was born. all of these material, we are trying to do with the same thing. We are trying to promoting the AI engineer. if you want to help us to get involved, a QR code is always available in the slide. So, you can

scan it. And here there are some highlights of the community. As I was saying to you we do applied AI workshop to help educator teach ML using our material. And specifically, we do it in the global south. disseminate tinyML kits to university to help new professionals. And we do show and tell where we welcome everyone to talk about their new ML project that they have developed thanks

to these material. there is a QR code for everything, but also everything is linked. You can from the other websites all these And the community is going going fast. As you can see, we reached more than 20K stars on GitHub. So, we are very happy about this. now that's that's up to you. If you guys want to to help us, you can do that in many way.

So, thank you very much for your attention. That's all from my part.