Sponsored Session: Fault-Tolerant Training: How We Build Rel... Cyril Konkratenko & Maurits de Groot

This talk addresses the challenges of GPU cluster failures and presents strategies for minimizing such issues. The speaker, representing Nebius, discusses the reliability of GPU setups, noting that while single GPUs perform well, scaling to larger clusters can significantly increase the frequency of failures. They categorize different types of failures, such as out-of-memory errors and software bugs, and stress the importance of measuring operational efficiency through metrics like mean time between failures and goodput. The presenter also outlines various approaches taken by Nebius, including pre-deployment checks, health monitoring, and the integration of advanced orchestration solutions like their custom Kubernetes operator. Ultimately, the session emphasizes the need for resilience in GPU clusters by using proactive mitigation techniques and fostering continuous improvements in performance.

Thank you for joining us in the last couple hours of the uh of the conference. It's always stuff doing a presentation near the end. Um what we're how clusters fail and what you can do to not make them fill as often. Um we are with Nebius. My name is Mitz. With me I have my colleague Kil. For the folks that don't know Nebius just yet. I'll give

you the 15-second uh pitch, but I I'll keep it brief. It's a sponsored talk, but we want to talk about the technical stuff. Nebas is a GPU provider. We are an AI cloud. So we full of GPUs and all of the services you need to make use of those GPUs. So brow compute orchestration on top of it. Things like object store to uh to get your data

from somewhere and then you can start to train or fine-tune your model. Um with that being said, we have a lot of GPUs. We do see a lot of failure when looking generally at GPU great. They're sort of reliable at single GPU scale. if you start to combine lots and lots of them, you're going to have some uh some challenges uh some technical challenges. So in um

in this paper, Meta has described that with one of their clusters. I think this is an H200 cluster where they a large cluster. If we have about a thousand GPUs, they'll fill about three times a day, which is already quite often. And if you have a cluster in which it's the case that you're using all of these GPUs on your same training run and you just do

it in a sort of naive way, then failure of a single GPU might set you this uh this gets worse with time because the probability of one GPU failing is more annoying if you scale up. So if you go down to 16,000 GPUs, then you have a failure every two hours or so. And if you go to massive massive scale then every 15 14 minutes um you're

out of can be very counterproductive. So we need to talk about some ways where you can mitigate this. When we say failure failure is not a single thing. Things can fail for all sorts of reasons. For the purpose of this talk we want to use at three main categories of failures. There there's a lot of failure around GPUs and this is from the same uh it's linked

at the bottom. highly recommends reading up on it if you haven't already. For now, failure can occur. Um when when something fails, we see the symptoms of it. We might see out of memory. When you see out of memory, it's almost always the fault of the programmer because maybe you try to fit in a model with a different parallelism strategy for try to fit a small model

in a or sorry a large model in a small GPU and things run out of memory. That that could happen and this is quite easily resolved. you might need to tweak some parameters. Sometimes uh it happens that you get a uh GPU driver error which could be an issue on the firmware level. Uh in this case your codes could be flaw GPU itself might be technically working.

Uh the hardware itself is correct but just uh the software the firmware of your GPU might need to be tweaked. Maybe there's a bug or something like that. Sometimes you have some errors which are physically located to the hardware. There could be some sort of cosmic array which managed to there's something wrong maybe it's overheating and you get an error like that and then hardware failure which

also happens. The trickiest things are the last two at the bottom where you see uh let's infamous and CCL timeouts where there is something which timed out. It might be PyTorch. It might be nickel pack. It's just not working. And these are the most annoying to debug. Um because it could be your fault or the programmer's fault, not your fault. It could be somewhere on the hardware

or on the infrastructure. The orchestration might be doing something odd. Um so, but still we're not doing something productive. What we want to our GPUs not only work, but work doing something productive. And it's very difficult to talk about it without defining what does good look like. So there there's multiple ways of measuring it. For now, I'm just going to stick to these two. Um where we

have the meanwhile between failures. So how long does it take for my GPU cluster on average case one of these faulty behaviors we talked about in the past? And this is taken into account that you're not rerunning every 10 times with the out of memory error because then you're skewing the numbers a little bit. But for the other elements, it's uh it's relevant to know how good

am I doing or maybe not as good. What could I do to improve? So when we define the meantime between the amount of GPUs we have available in our cluster and then um multiplying it by the time these GPUs are actually doing something something operational and then you divide it by the number of failures. So if you have you see the the slide there are some numbers

just to illustrate the point. It's simple, it's easy to understand. flawed because it doesn't actually tell if we're doing something which is useful with the GPU. So there's been another measurement put out which is called a good boot and I kind of like goodput. It's a bit more elaborate to measure because you're not only looking at the failure itself but you're saying okay I want my GPU

to doful and useful compute can be time spent on inference or maybe do a training run. Um, and if I look at the time I spent on doing something useful, I divide that by the total time. So that's the time spent on doing something useful as well as the time you are actually not doing something as useful, your GPU might say it's idle. You don't want to

have idle GPUs because that's but it could also contain things like creating a checkpoint, which is very useful when something goes wrong, but the act of making a checkpoint itself will not get you closer to your objective. So checkpointing good practice. I'm not saying that you shouldn't do it like do checkpointing look at asynchronous checkpointing but the time you spend on checkpointing is time you're on computing

gradients it's probably what you want to do with it so when we look at it we can say okay we have this useful compute and it's a bit difficult on the screen so bear with me maybe I have some time where it's idle which is this bar here which is slow contrast and then we we take a checkpoint every so often which goes a bit down because

this time is useful. This time not helping for our gradients. We have some useful time again, another checkpoint and some calculation again. Um then maybe an interruption occurs or probably at some point in time at some scale an interruption will occur because we can't get rid of these unfortunately. We can minimize them. When this interruption occurs there is um the time you have till your last checkpoint.

It takes some time to recover. That's the meanantime to recovery. How long does it take to get new compute? Ensure that your the state you saved in your last checkpoint is there again and that you can do something useful again. So we some time which is being lost here. The more often we checkpoint the lower our goodput might be but the lower the amount of work is

which we might lose. We we call this time between the uh interruption and the restored state the mtr or the meanantime to recovery. something happened, we are recovered from whatever happened. Um, there's a lot of things we can and we do at Nebus and you can do as well to help to uh improve all of these metrics, but this is at least the metrics we are interested

in optimizing. At Nebus, we we do quite a bit to help you get started with this because we have a lot of GPUs, but we also have a lot of we've seen a lot of failures and we have a on mitigating these failures. So things we are doing at is being very very elaborate before giving you the GPUs because a GPU might be faulty but if we

detect that before giving it to our customers then we're not going to use the GPU in our operational clusters. So we have um a multi-stage acceptance test which will circle besides that we look at health checks. Hell checks are very useful because it's very cheap to maybe run some quick tests before starting your job knowing that at the point of start all GPUs are reporting um network

is configured as we expect we see the right number of links these checks can be fairly can save a lot of headaches later on and I'll link to a blog post in the end and there's a line by line description on what kind of checks we run and you could maybe incorporate in your own workflow as well some of these checks are active checks we have some

passive checks as well which just run more passively um just to ensure that we're catching some things um situations. Besides that, when a node fails, as they will, again, I'm not going to make this disappear, we have some nodes on standby. So, if you purchase a,24 compute nodes, we actually secretly have a bit more to uh to give to you when this failure occurs because when this

note able to swap this out quickly, how we solve this is that within our platform, we do virtualize the nodes. We do not virtualize the Nvidia interfaces. So we pass through the GPUs so that you get bare metal performance while we could quickly switch out nodes. So if a node goes down in your cluster, we can substitute it with another one. Um and since we have that

node available already, we're not capacity constrained because otherwise somebody had to go into the data center, pull out the node, repair it, put it back um as you would with bare metal which takes a lot of time and that's that's ours. We can do it in minutes. Last thing I wanted to uh to touch upon all failures are created equal. Maybe you get some indication, some XIDs

errors that this node is not performing as it should be, but it hasn't crashed. It hasn't failed. It's just not performing as it should be. Maybe the terminals are off. Maybe you see some errors which are non-critical errors. At that point in time, we um we isolate this note. We'll like let it keep on performing the way it does. make it finish maybe the epoch so that

you can go towards a place where you have some useful results on it but not not schedule new jobs on it and if possible uh sense like some emergency like maybe it's good to make a checkpoint right now so that you can hook into that and not uh not see happens we can offload the note and try to figure out what's going on with this node before

you are actually using it in your compute statement if that um doesn't work out uh sometimes you have a critical failure then um isolating this might not happen but these are things you can think about when you are working in a larger cluster ensure that you uh you have resiliency resiliency is something which is not at a single place it's all across our stack so I'll hand

over to Mike to my colleague Kil who'll talk about that okay thank you so yeah I'm going to tell you what we actually do on like many levels of our so first of All like yeah as you can imagine in the cloud provider we have many different layers of the stack we have our hardware infrastructure on which we build our virtualized platform after that uh like before

you run any actual workload you will have the scheduleuler level like slurm kubernetes and then you actually have your on each of these levels we try our best to to make your life as easier as possible by eliminating issues. So we start first with uh the hardware part of course. So like once uh we're building the data center like before we actually deploy the virtualized platform on

it uh have uh a lot of uh quality control in place but we will also like do everything according to the Nvidia's bringup guide. So we will run a lot of tests on bare metal like uh compute and uh infinite band uh like storage anything uh after this uh like we here we we're arriving at the level of our virtualized platform and here I represent it as

our managed kubernetes service. So basically like before before we continue I want to tell you that yeah we are a virtualized platform but like we don't virtualize GPUs or like infinib band so like whatever matters for performance you get the bare metal level uh uh but uh at the same time virtualizing uh CPU and and and uh the compute part allows us to be more flexible and

provide you a better experience. Uh so at this level uh quite a number of uh health checks uh like mainly these are passive health checks which allow us to see to check that computer instance is behaving correct correctly. We also have a lot of uh uh Nvidia specific stuff like topology detection DCGMI uh discovery check if like the the driver is correctly loaded the kernel model modules

are present etc. And after that we arrive at the schedule level. So uh here uh we we're talking about soperator. So s operator is a a solution which we developed in nibbute. So this is our own take on slurm. So basically as operator is a slurm kubernet slurm operator for kubernetes and uh it is designed in a way to provide you like the standard slurm user experience

which I think all most of you are familiar with and like and at the same time it gives us and your s sur if you would wish to to manage it by yourself uh uh the way to manage it as any like kubernetes uh CRD. So it's uh it's best of both worlds. You have users get people who maintain it get ease of uh like scaling and

maintenance from Kubernetes. Uh at this level we also run a lot of uh health checks and the advantage of integrating it in our own in-house solution is that uh you can actually run active health checks and they represent perm workloads. So uh you're going like we're going to run uh something like GPU fryer, nickel tests, uh DCGM diag, uh among other things and uh like these will

be scheduled in between your workloads. So we're not going to obviously mess with your but they will be scheduled at the same storm cluster. Well, it is obviously configurable to the user. If you don't want them, we can disable them. Okay. And here we're arriving at the last uh layer of the stack is the actual distributed workload. So there's actually have been a significant development over the

last years starting uh new frameworks which can benefit from fault tolerant uh infrastructure. So here we take an example of torch fft. I don't know if you're familiar with that, but it's a framework uh which is based on torch titan and allows you to recover from uh interruptions without checkpointing. Uh it uh it treats the the world change sizes uh like as a membership change and it's

not going to to make you restart the whole the whole workload. Uh recently we also get uh PyTorch Monarch I think it was the last uh PyTorch last year PyTorch conference uh like to my mind is a step forward which allows you to implement custom behavior in case of failure and uh Nvidia is also uh keeping fall tolerance in their mind. they integrate uh the the possibility

to shrink communicator of Nikl in case of failure instead of completely like re recreating a new communicator. Okay. So well we do a lot of uh different things on different layers of stack but uh these uh things well they mostly run on the compute nodes but as you can imagine this is not enough to check to to to all of this is covered by our comprehensive uh

layer of observability which covers every every layer of the stack. So I mean yeah if we go uh from top to bottom yeah of course you have your workload and if you desire so you can uh like this is completely expose uh like nickel inspector logs or us uh for us to be able to proactively uh uh proactively replace uh your um your faulty nodes or help

you detect the strugglers. Obviously, you also have a bunch of other issu uh detected by uh active or passive health checks. For example, maybe your workload is doing something strange. Maybe uh uh it has some unusual pattern of like using uh uh I I don't know uh infin like the the way you're loading the GPUs is different. So we can we can detect that obviously uh there's

also uh the infinite event fabric level issues because I mean while we like from the node side we can see a lot of stuff we also have observability on the infinity band switches and like to run it on every node if like one switch is faulty we can just proactively remove the whole the whole pod or scalable unit from the from the cluster and yeah of course

There are a bunch of other things happen like yeah if you worked with hardware you know this list keeps going. So before we we wrap things up yeah just wanted to tell you that um we actually uh are able to achieve pretty good uh MTBF numbers. we uh were able to to achieve uh over 160 uh,000 of GPU hours between meantime between failures and something which we

particularly proud of is the our meanantime to recovery is around 10 minutes. So basically once we detect that the node is faulty in 10 minutes you get you get the new one integrated on into your back in your Uh yeah if you if you're interesting to in learning more of uh more stuff about our reliability and um uh resilience efforts. Yeah, please uh please visit our blog

post and of course yeah we are always uh looking for for new talents to join us. Yeah would be help happy to answer any questions if if you have It's very weird because it's we can hear each other clearly. >> Hello. Uh so why do you put training specifically in the title of the talk while like in the keynote we saw that 70% of the workloads now

inference and uh do do you mind like is training more important for your customers or inference and how much are these tools that we presented here also translatable to like mixture of experts inference serving. Yeah. >> Yeah. Yeah. Sure. I mean I would say training obviously is a very good showcase of resilience because once you have like one node is down your whole job goes down typically

but uh like that particular use case which you mentioned uh like uh mixture of experts inference with like spans multiple nodes. I think basically any distributed workload would benefit from this. But yeah, that was just just an example. Yeah, also maybe to add to that um the scale on which inference happens and the scale on which training happens is slightly different. You can have and with disagregated

serving it can be complex but having a thousand systems in your inference system and then caring if one of them fails because with disagregated inference it's easier to have some resiliency from the framework where you can lose some prefill or decode nodes. That's a different focusing these challenges um might appear more often due to the nature of having more nodes. Does that make sense? Any other questions?

Going once. Going twice. All right. If there's no more questions time, thank you so much for your attention and enjoy the rest of the