Orchestrating Thousands of GPUs: Engineering Patterns for Large-Scale Model Training - Krishnaswamy

This talk delves into the engineering challenges of training large language models, specifically Llama 3.1, which boasts an astonishing 405 billion parameters. The speaker recounts their initial skepticism upon discovering the scale of Meta's model training, which required 16,384 GPUs and took 54 days to complete. The session explains the fundamental role of matrix multiplication in deep learning and how it can be efficiently parallelized across multiple GPUs, highlighting the importance of NVLink for high-speed communication between devices. The speaker elaborates on the necessity of massive computational resources, the significance of redundancy to handle hardware failures, and strategies like sharded data parallelism to manage memory constraints and optimize performance. Ultimately, the talk illustrates how meticulous engineering and pioneering techniques enable the training of expansive models in a feasible timeframe.

So last year on the same time, I joined a team to do some kind of a model training. And then the team was very excited about the recent paper published by Meta about their model and how much they use and everything. I thought, okay, what's the big deal in it? Because I actually come from a different background and then I've actually seen lot of uh microservices, lot

of infra. I have done seen a large scale of distributed application and everything. I thought what is it going to be right and why people what people are talking about it? And this is the first number they they showed. Okay, can you guess what's this number? >> A million? >> Yeah. So they used somewhere around 16,384 GPUs. Then I was thinking, okay, which means it's a real

single uh if you know about the PCIE chips what you put it on the thing, right? They used 16,384 chips. The first question occurred to me is, right? Why would one someone even need so much? What is it the need for it, they did it in 54 days. Then I was thinking, why would it even need 54 days to run this, right? And how would it keep

running for 54 days? the the world or what we live in is, right? It's all about um microservices. We deploy it and you don't care if it goes, it restarts and then comes back. But here it needs to run continuously for 54 days. If something goes wrong, then everything has to restart. It's a continuous 54 days of run and then they achieved it. And on that they

had 419 hardware failures, which means one of the hardware on that number actually is gone wrong. And then they have to replace another GPU chip and then continue the training and then they achieve this one model which is Llama 3.1 405 billion parameters. And then this is this all when I joined this particular person how does this even possible right? So all these numbers I've never seen

in my past. Right? And I think I hope a lot of you know not seen right a lot of people are not even seen this because this is a pretty new area right? I think if somebody claims that uh I know all about this uh it could be only what? One or two years? Three years? All these things would have been within that particular period people would

have just know got to know about all these things. So nothing to worry about it but this is a real number. Okay? Um So I just did a mean uh time between failures it comes out somewhere around three hours. So every three hours there is a hardware failure. And they have to go and fix it, solve it and then try to rerun the training. Okay. So that's

the number and then we're going to understand how why you need so many GPUs and how they did in 54 days and everything but before we go there right? I just want to give you some basic so that everybody comes on the same page so that we can actually build on that particular thing. people talk about neuron right? Single neuron is this. Uh you can think of

it as some kind of a weight and then you're going to get an Y output. It's just like nothing like you need to multiply all these things on with the bias. That's what you're going to get in weight right? This is a single neuron. And a layer many neurons same input. Right? So when I say neuron you know what is it? When I say layer you understand

what is it? Right? A layer consists of multiple neurons in it. Just a second. So, mathematically you can represent that using matrix, right? So, whatever I shown you, it's all represented using a single matrix multiplication, which is called Y equal to uh weight into input plus bias. That is a standard thing. So, this is the single matrix application. So, underneath they could do a lot of things,

but everything boiled down to the single matrix multiplication equation. Right? Uh and where one very interesting thing is, right? So, this matrix multiplication is a very fascinating thing. So, you you need to multiply that and this, uh you'll get the C. And then uh if I do this, you'll get that. So, what is very interesting in this? Actually, you can actually uh it can be done simultaneously.

What I mean by that is you you can pick one, you can pick one, you can pick one. So, you can actually split this into multiple smaller pieces and you can actually do it parallelly. That is a very interesting aspect that you need to pay attention to. Right? So, why I'm stressing on that is layers in that, right? It all boils down to the single equation. Whatever

you say, right? Dense layer, attention layer, convolution, back pass, everything is just this matrix multiplication. I'm so I'm simplifying it. Yeah, if you do you don't have to understand a lot of math behind it, but everything just simply boils down to this. Okay. So, in this entire thing, we're going to see some scale of what the what is the number of C's? So, how many billion parameters?

>> 405 >> 405 billion, right? Which means it's going to be so many matrix multiplication everything. And uh the reason what you can do about it, right? Actually, you can split this into multiple smaller matrix, and then you can actually assemble it back. Right? So, once you're done with that, you can actually assemble it back. So, that is the most beautiful thing about this matrix and max.

You can actually split this like however you want to split it, and then do the calculations, and then put it back. So far, so clear? So, this is called embarrassingly parallel. Uh if you are in the HPC world, this particular term called embarrassingly parallel will be pretty common. And uh this is this is a you can go and read more about it. This is Right? This particular

problem, yeah. So, now we saw that it's a multiplication matrix multiplication problem. Correct? Even though it's a model is a model training and everything, fundamentally it's just a matrix multiplication. I'm just simplifying it. There could be more to it to have this sake of understanding of what's going on. I'm just simplifying it to matrix multiplication. Right? So, what's the right machine? What I mean by right machine

is right, we all know CPU. Right? And why there's a need for even GPU? Right? That's the first question I was saying, "Hey, why you need so many GPUs, man? Why can't you do with CPUs and then get with it, right? It's very easily and cheaply available, and why you need even GPU?" Right? That's the first question I asked because I come from a different background, and

these people are saying, "No, no, we can't do it in GPU." I really want to understand why they can't do it in GPU. So, let's uh I'll go back to little bit. I'll just quickly talk about CPU. What is CPU? It has all these things. It's just a branch predictor, out of order engine. It's It's an internal L2 cache and everything, right? So, and on the GPU,

sorry, on the GPU everything is it's called CUDA cores, and this is the number of CUDA cores which cores which H100 has. Okay? Sorry, sorry. It's going a little fast. So, it has no branch predictor, no out of nothing. It's a very simple thing which can actually do a single uh floating point operations per second which is like add, multiply, or fused multiply add A into B

plus C. Whatever we saw there, right? So, somebody's actually devised a chip which can exactly do A into B plus C. That is multiplication. So, why you need that, right? Um Let's understand a little more. Uh there's something called a wrap. You don't have to go much deeper into it. I'm just throwing so that all these things will be very clear for you. So, there's one wrap

which has 32 core in a lockstep. Um and what that means I'll show you. Right? So, it So, one streaming uh multiprocessor has four wraps in it. Uh So, in total right? In total, you have 16,896. Cool. So, let's do a simple matrix multiplication using CPU. What happens? You have to do like this. Like uh typical you'll do it in a loop and then you get the

result. Uh and then uh you have to do 15 more, right? So, if one is doing, the other 15 has to wait. Or if there's a multiprocessor thing or multithread, you can actually split. You can actually still do things. Let's see how different it is in GPU. In GPU, I'm just showing 16, right? So, everything will be done on one cycle. So, if you just give that

thing, it'll be done in just Right? So, this is a simple the race I'll just call. So, CPU will be done like this. In GPU, it'll be just like that. It's very simple. And if you want to show the real the thought process comparison, right? Or uh throughput comparison, see this this is a CPU which is like a 4 teraflops per second. And this is 67 and

with H1 tensor core, it'll be 900 teraflops per second which is like 240 times faster than your CPU. Latest CPU on Earth, it is 240 x faster than This is you can see it like this like a few CPUs. CPUs are like a few PhDs solving hard problems. Our GPUs like they don't know anything. They'll simply do A plus B into C. That's it. Nothing fancy. Cool.

So, now you know why you need GPU. Clear, right? So, we can Okay. Now, let's understand, right? So, now we know that for model training everything is a matrix and you know you need a GPU, Now, still something is not pretty clear saying why you need how many CPU GPUs you need when you start 16,000 something, right? So, why you need so many, right? Let's let's start

to see that. Okay. So, this is the number of layers in the Llama 3.1 405 billion parameter, which is like 126 transformer layers. Okay? I'm just going to zoom into one of the layer. Okay? And then see what is in it. So, one layer has six uh weight matrix multiplication per layer. So, these are the different six things you need to do, which is like a Q,

K. This is like attention block and everything. So, you don't have to pay attention to all these things. That's This is just I want to show that it has some six weighted matrix per layer. Okay? So, now I want to zoom in further what is inside a Q, right? Sorry. Okay? So, this is Q and one Q matrix has 216 million parameters. That is the scale of

the matrix which is inside a Q on a single layer. Clear so far? Clear? So, still So, if you remember our earlier thing which is like embarrassing parallel, now this can be split. The same metrics can be split into multiple smaller pieces, which we can actually do it faster using GPUs, and you can assemble it back. So, you can actually do the same for all six metrics

in every layer. Uh now, let's go back and uh do our math. So, how much compute is needed? So, one metrics, you needed that and the multiply by that by six metrics, you need 1.6 billion per layer, right? So, you have you need that much compute. Uh So, you need somewhere around 3.8 into 10 to the power 24 flops per second to do this entire thing. So,

what that even means, right? Uh one H100 can actually do 990 teraflops per second on the marketing, uh that's what they can actually do. So, I'm just do a rough math on that, right? If you use single H100 machine to do the training, it will take 1,200 years. We are only doing matrix multiplication, but even to do that matrix multiplication on a single H100 chips, which is

240x faster than your CPU, it But, that's not the real one because you cannot achieve 100% of 100% utilization of the GPU. You can actually even if you achieve this 35 or 40% practically, that is very good number in the GPU training world, which means you need 3,500 years on a single H100 chip to train that particular model. Okay. So, what's the one way, right? But they

did it in 54 days. Isn't this itself a fascinating thing knowing about all these things and then thinking that how would even they would have done this in 54 days? And this is just a matter of actually publish all this number. Everybody would have been doing the same thing. They are not topics doing it. Google would be doing it and then they could be using different techniques

and all those things. But just look at this sheer from the engineering perspective, this looks like a very fascinating. This is like going on some uh going and looking at uh visiting Taj Mahal and then people say that hey, it look took so much here to build this Taj Mahal and everything, right? It's something similar to that, right? So, how did even they do it? Um So,

that is what I'm going to talk more into it saying how they even achieved this. Right? Uh Any questions so far on this? Can you move? Okay. So, now we all understood why you need so many GPUs. Pretty clear? One H100 GPU has only 80 GB memory in it. The RAM. Okay? But to train this particular thing, you need 2,100 GBs of RAM. And now you know

why people are going behind RAM and why RAM price is all crazy, right? It's all because for GPU training and everything, you need a lot of GPUs, a lot of RAMs. And this is it. So, this is all the numbers and then you need this. You cannot fit this in a single GPU. Not even possible. Okay? So, which is 10x of that, right? >> Okay. So, for

the full training actually you need 4,860 GB of RAM. Without that you cannot do this So, yeah. This is this is just to fit the training you also need speed. You remember it can only do if you have one GPU you can only do a particular speed, but if you have multiple GPUs you can actually exceed otherwise you'll be doing it under under 1,000 years. You need

to paralyze it. So, how we're going to paralyze it? The embracing parallel you remember right? You can actually split the matrix, give it out, do the matrix multiplication, assemble it back. Looks fascinating interesting idea, right? Okay. What is happening? Okay. Okay, cool. we need 16,384 GPUs. I'll just use the laptop which is this this is not working properly. Okay. So, it is actually solving three So, we

we need so many GPUs for three problem. So, one is it doesn't fit. It is too slow. It will break because any GPU can go down. Let's say if you're only going to say hey, I'm doing a proper calculation you're ordering for a so many number of GPUs any GPU can actually go down. It can actually can malfunction it will do it. If that GPU goes down

you cannot continue with your training. You need that to be in place even to continue. Which means you need some kind of a redundant GPUs also to be available so that if something goes wrong I can actually replace it with other GPU. So, I'll I'll reiterate everything so that we'll go further. we understood how many Uh first we saw embarrassingly parallel that we can actually split the

matrix. We can do the calculation. So, then we saw why we need so many GPUs, why GPUs, and then why we cannot fit everything. It's clear, right? Now, we also understood saying that now we need more machines, and then we need to split this matrix and then we need to do things. Now, this is where this is going to get more interesting. Okay, inside one machine, okay?

What's going to happen? Okay. Okay. Okay. So, in a GPU or in a in a physical machine, right? You have GPU and you have CPU and you have a GP. Because that's how you connected. You you put a GPU card on a machine where you have also CP. And all the device will communicate via CP, right? So, if if anything needs to be transferred from one particular

one peripheral to other peripheral, you need CP in picture. Without that, you cannot do it traditionally. Yes or no? Right? Okay. But that's not Okay. So, in in the in in terms of GPU, that uh desirable. I'll tell you why. So, in in terms of GPU, you have something called NVLink, uh which actually does a 900 GB per second. Uh to put into the context of the

picture, this is what it is. So, if I do PC transfer, it will be 120 GB per second, but if I use NVLink, it will be 900 GB per second. Why is this even needed? Why am I even talking about this? >> PCIe connectivity. >> Yes. So, you are going to split the matrix into multiple GPUs, and then you need to open the result back. Which means

GPUs are going to talk to another GPU and then going to coordinate things. And if you're going to run it on a slow PCIe based thing, it is going to slow you down. You need a much much higher bandwidth which can actually do things so that the results are come can come back really So, you need something like an NVLink. Without that, you cannot do it. And

the next thing is, right? In a typical H100 machines, you have eight GPUs, and all eight GPUs are connected with each other. Because any GPU can should be talking to each other. It's not like one way saying I'll G1 only be talking to GPU 2 G2 3, right? Anything should be able to talk to each other. Then you need so many connections, which is like a 28

connections. To avoid this connection, there's something called uh NVLink switch. So, that also you need to put it into the device so that the things can be much faster. And then you can actually achieve this particular speed. So, this is inside one machine. Uh so, GPU internally, it can actually do within within GPU, it is actually 3,500 GB/s, but if you use NVLink, it comes to 900

GB. This is what it is possible as of today, which means we're already 3.7x What a real GPU can actually do is being degraded just because of one device talking to the other device. Here comes the interesting more interesting part, right? Which is like now you can actually do it in a single H100 machine, you can under achieve 900 GB/s, but you need 2,048 of them. Which

is like a one machine to the next machine. The only thing we know is, right? You can actually connect some connector, which is like a USB-C or Ethernet connection. Whatever you can actually connect. But the moment you connect that, even that is also going to go via your CPU. The moment it goes via CPU, it is going to come down even Right? So, let's see how it's

being done across machines. If you do it like this, this is a typical thing how we do it. So, the packet will come like packet will go like that. So, if you do that, it will be lot of wastage uh and for this particular training scale, this is not going to help help at all. Right? So, you need something called RDMA, which is like a remote direct

memory access. So, what that will do, application A or can actually talk to the other memory in the other machine directly without even involving your So, this is why you need RDMA. cool, pretty cool. So, we have built everything, right? So, now we understood that in a single H100 machine, even within GPU GPU is going to be slow. Then we understood that we need multiple GPUs and

then we need to talk to each other using the other machines. We need something like a GPU direct. Okay? So, here is a full bandwidth hierarchy. So, GPU internal, you have this speed. That's the speed within a within the actual memory or the within the actual GPU card. And this is between two GPU cards. And this is machine to machine. And that is standard ethernet. So, you

can actually see how it is actually coming down. So, I think the real uh engineering or real uh advancements actually going on this and to match this so that all the training can actually happen in that the much faster rate. But also if you think of it, right? You cannot do much about this because you cannot eliminate TCP. It because TCP is everywhere, right? And you have

to live with that limitations of TCP and everything or people needs to do even if you find a new protocol, how do you even roll out to all the billions of devices already which is running out there? How do you even make it compatible or you need to build a new protocol and then just keep it within this particular training thing or something. You have to do

you have to do something and even if you do that, then all your application needs to talk use that protocol. Your Linux needs to understand, the kernel needs to understand, everything has to understand. So it's going to be Yeah, I think it will evolve. Maybe after another 10 years or something, right? You might see that hey, it will all match. But yeah, as of today it's not.

Or at least as of this happened last year, it was not the case. I don't know the even the real numbers what's happening nowadays there are much better machines from Nvidia which are like in GB200, GB300. So this is a little more older one. remember this because we are going to use we go we are going to come back to this or connect back to this inside

later. Uh one GPU you can actually achieve uh with inside one GPU you can actually some 3 GBs or sorry, 3,350 GBs and inside one machine you can have that achieve 900 GBs. Across machines 50 GBs. Yeah, remember this. Do you think everything is done? Uh in the sense uh we figured out that uh within a machine we need a NVLink, NVswitch, RDMA and now we can

actually connect all the problems and everything, right? So we can actually connect all the machines everything and now we are ready to go. How many of you think that we are actually done now? Like we are pretty good and we can actually do the training. All the puzzles are solved. >> Are you talking about the state or 60? >> Yeah. So, the next bigger problem is the

coordination problem. So, what is the coordination problem? You have matrix. But who is going to distribute saying hey, this GPU you do this particular matrix multiplication, this GPU you do this matrix matrix matrix multiplication. Who is going to do that? The first thing itself is right, you need to figure out find out each other. Yeah. So, if you ever talk if any of you are coming from

DevOps world or infra done something right or even you need to know the other IP addresses. In this case, it's going to be 4,996 machines, right? So, then you need to configure each and every machine saying that hey uh here is a configuration. Even if you just think about doing the configuration, right? You need to do the configuration in all the 4,900 machines saying that here are

the other peers. Here are the other peers. If something goes down and if a new IP come pops up, then you need to go and rechange This is what will happen, right? So, let's say if you're doing something like a model training if you're running for so many number of days and when a GPU failure happens and all this matrix multiplication will be lost. Which means you

have to redo this entire But people can say that hey, I can do a checkpointing. So, if you do frequent checkpointing, what will happen is right, remember? So, again you're going to involve disk here. Which means all the GPU all the matrix whatever have, right? You're going to put it into disk. Which is going to slow you down further. your training days is going to be extended

even beyond 54 days or 56 days, which is you need to find optimal number how frequently you need to do a checkpointing so that you can actually see how this can be done in a faster way. If something goes wrong, I can actually recover from that particular checkpoint. I can actually re-initialize all those things. I can actually continue from there. So, there's something called asynchronous uh checkpointing.

You can actually do it and uh and you can achieve some Yeah. So, then if something goes wrong, uh you can actually delete that. You can actually replace that. You can load from that checkpoint. Actually, you can resume the training from that particular checkpoint. But, remember you cannot do a very frequent Go go on. So, uh you read from the uh memory and then you uh asynchronously

start writing it. So, you don't you don't wait actually for it to complete. Asynchronously start writing it like how you do it in the Node.js. You just request and then comes back like the same asynchronous thing. You can actually put it out. You can actually do >> If one GPU fails, then you have to go to the phase isn't >> Uh if one GPU goes down, then

you need to go to checkpoint of the previous checkpoint. You need to reset the entire process in between before that happens. >> And >> No, we already uh uh what we did right? We did the math saying we have only 80 GB of RAM. So, whatever is possible to fit within that, we already gave all those metrics. We will overload that. You don't have any more space.

So, the way it works right? I'm actually over simplifying things. It has a lot of things. You know gradient descent, And then once everything happens, everything needs to give back that final gradient descent value. It needs to propagate that value to all the GPUs. And then it needs to set that value and then it will start the new cycle. Which means every all the final value needs

to come back to all the GPUs. It is not easy. Yeah, somehow they wired everything together and then they're doing this, man, if you ask me. if you ask me, right, who would be the platform engineer, infra engineer for these GPUs doing the training model, right? I mean, you're not even able to think of there. Think so, for even just for a simple application things like it

is going down like one one application goes down or databases uh doing a wrong behavior and everything. They think about this, it a lot of so many small small small things, anything can actually go wrong in this. >> Let's talk to my team in this test flight. So, what after we found this one credit? >> But there is no single formula, man. If you ask me, I

cannot give you the single formula. It's all depends on how many parameters what your model training on, what's the GPU training on, what's the disk speed you have, a lot of things. If you If you're just using an simple HHD it's going to be really really slow. I'll say you cannot do a lot of faster checkpointing. If you use NVMe disk which is next to the CPU,

then you can actually do a much more faster, There's no right answer to it. >> So, like how but someone is taking the model, so how >> Maybe they would have done all those things, they would have learned by this, and then they know all those things. And uh they are publishing these numbers like uh DeepSeek and everybody, they are openly publishing it out. Meta is also

publishing it out in hugging face. A lot of people are figuring out pushing out those metrics. You need to go on read from that and learn from that. I'll go ahead and talk about the next thing. So, we kind of figured out that all the all these puzzles now. So, still few more puzzles needs to be actually solved to see the full picture of how they actually

achieved it. Okay. Uh I think we are going to go into Yeah. We are not yet ready. So, the reason is the fundamental question is this, right? So, 80 GB 4860 GB is needed. You cannot fit this in. Correct? So, somehow we need to split this. Now, the prob- So, we have actually seen uh we solved all the puzzles like whatever has been or the challenges we've

been thrown at it, right? We somehow challenge kind of solved it. Now, comes the next problem saying, "How are you going to break this down?" This 4860 GB how are you going to break this down? So, that you can actually use 80 GB and then you can actually solve So, this is very simple, right? Hey, just split the training data whatever the data you have and then

just to all the GPUs. You do the train, you get it back. Very simple idea, no? Do you think this will work? Why? >> Look at the pins. >> Yeah, you cannot fit it. Even this simple idea of, "Hey, let's split it and do it." will not work. So, because what you're actually doing it, right? You're You're using the same model, but different data. Model is the

one which is actually causing you this particular GB of memory, right? And if you're just going to split the data alone, if you're going to give it to uh GPUs still right? So, what has happened is that it has actually you have actually exponentially overgrown the RAM requirement to 311 TB from just 4860 GB. Just simply you're saying that, "Hey, all GPUs load the same model. I'll

just split the data and give it to you. If you just simply do that, then it will become like that. So, which means you already made it redundant. Uh so, you cannot do this. What's the way to solve it? In one So, one interesting thing is, right? Instead of uh splitting the data, you need to split the model itself. So, in in GPU zero, you can actually

do layer one and two. In GPU one, you can actually do three and four. And in GPU two, if you do this, right? This is nothing but your fully shard data parallel. So, people will call about a lot of parallelizing technique. One is like a data parallel, a lot of different parallel techniques. This is that exact thing which is called um uh fully sharded data parallel, which

does this. You shard the layers, and then you do the Okay. Here is the next idea. We already saw that uh Okay. Uh we can actually split the weights of the matrix. So, this is like a uh we we already saw that embedding technique, right? You can actually split the matrix, and then if you split it uh column wise, you don't have to communicate with needed each

other. Because all the uh results can be within stored within the particular GPU. But the moment you split that uh matrix into row wise, So, if you if you do a row wise split, right? You need to coordinate back. That was a thing you were you were asking, right? Why you need to coordinate back? If you do this kind of a split, if you split like this

in a matrix, you can actually do the competition locally. But, if you split like this, give the result back. Right? Um so, if you split like this, column parallel, it no communication is needed. But, if you do uh this kind of a all reduce, if you do that, then you need a to communicate back. So, the moment communication is going to involved, you're going to hit the

next problem of uh your NVLink NV switch speed. And then, if it's going to be across GPUs, you're going to hit the other problems. If you're going to split the matrix. Okay. So, this kind of thing like uh the previously what you saw, right? We actually the first model, we saw how to split the layers. In these things, we're actually splitting the tensor itself. You're parallelizing this

is going to be tensor parallel if you do this Still, right? So, tensor parallel, you cannot achieve You cannot do it in all these because you can actually only do it in a single machine. You cannot do tensor parallel across multiple GPUs. Because you need that NVLink switch NVLink thing so that will be faster. The moment you cross that, it is going to come and haunt you.

It's going to be slower. You'll be wasting a lot of GPU cycles. I'm not saying it's not possible. You do it. But, if you look at your GPU utilization, it will be really really slow because you're going to sit and wait idle for the other GPUs across the things to data to come back to you to do that. So, what's the better way of doing it? Simply

think of like an assembly line uh where you can actually split things across multiple GPUs. Uh you can actually split this 128 layer into some kind of 16 and then you divide that and then say hey eight layers per stage and then you do the which is something like this. So even splitting and then giving it right, you cannot it's not an easy problem. So let's say

if these are like these are all different GPUs and then different time and then you're actually allocating it right. So even if you do some of your GPUs will be waiting for the results to be computed. Because you would actually split it and then you've actually given the you've actually done the embarrassing parallel technique you did the math specification however you want to do it but because

of the latest and everything right, you will see these gaps where GPUs will be sitting idle for the actual result to come back from other GPUs which is running on the other GPUs on the other nodes. Right? So how do you solve this? There's something called the micro batching through which you can actually solve We can do something like this which is called a micro batching like

any optimization problem, you can say that hey I have so many GPUs, I have so much that needs to be done. You do the work load scheduling problem that you can actually think of it and then you can schedule saying hey I split this work into multiple smaller batches. I'll do these things when it wait I'll do this operation. Like that you need to start filling in

the GPU time and then you can see how it can be utilized and then to achieve this. It's not yeah you And if you do that uh you can actually reduce that bubble to 8.6% from 70%. So now we'll move on to the final thing. Maybe I'll show it like this itself. Pretty much more easier. So, whatever we did with an assumption all the layers we actually

we split the layers, correct? And then we did it because of memory fitment. But, there are some kind of a model which needs a different thing saying all the input is needed. If you want to do something like that, then there are I'm just showing you something saying it's not easy. Uh, so there's something called a router where it comes and it goes to the different nodes

and then it comes back. And then, that is something called expert parallelism. Like, you have lot of experts which can actually models there are there'll be experts which will be doing individual calculation and different GPUs and then you get that back. That is an expert Um, yeah, so that's another thing which you need to worry about. So, here is the full picture on how they achieved it.

So, the first thing is uh, whoever did this, they actually used all the three techniques whatever you saw so far. The first thing is they split the entire model training into whatever is possible within the node, they did they did the horizontal split. Then, they did the vertical split as well. On how to do it. Then, finally They used all the three techniques which is like a

tensor parallel using NVLink they did it. With pipeline parallel they use InfiniBand and then they use data parallel cluster wide. And this is not the only way to do it. So, whatever meta did it, right? And then Deep Seek later came in saying that hey uh you did it in 60 million I did in 5.6 million. You remember uh after this came in the thing got crashed

because the only thing they changed it right here they used tensor parallel they used expert parallel. That's the only thing they changed. They used mixture of experts rather than the tensor parallel and then they were able to achieve it at a much much reduced uh number of GPUs here. 16,384 there it is 20484 GPUs only. pipeline parallel data parallel 54 days of single checkpoint and then 495

days if something goes wrong they can use this PyTorch to get it back and then they'll restore it from the checkpoints. This is in 5 minutes. Get it going and they did the entire thing. Yeah, this is I just added this. So, we all do this thing with the ChatGPT saying hey you ask questions, right? But remember this. They mean they the model has been trained using

all these techniques and we are simply chatting with it. But that's the fascinating engineering behind it. It's not magic actually, it's engineering. If I think humans are very good, man. I think if you just throw a problem they will engineer it and then they'll get it done. Because all those things, right? If you think of it, right? I just didn't put in one thing. It The GPUs

are used in the gaming and then movie industry. Correct? And then it was also doing this matrix multiplication. Somebody saw it and then hijacked it. And then we are retrofitting on top of that, right? Whatever is what there and then we're using that AI with that particular limitation technique and then we people are trying to make best out of it, right? Because if you're paying for a

GPU and if you're buying it out, right? They won't utilize it to the maximum extent. And people are figuring out day in day out saying how can actually engineer it to take the maximum money and value out of And then with that that engineering, they did all these It's really really fascinating one.