Lightning Talk: Avoiding CPU Throttling: How Go 1.25's Container-Aware Runtime Fix... Adarsh K Kumar

This talk focuses on addressing throttling issues in Go-based applications when the Go max procs are not correctly set. The speaker, a principal engineer at Rapido, explains how containers and cgroups work in managing resource allocation. He details how Go versions prior to 1.25 did not properly account for container resource limits, leading to inefficient concurrency. With Go 1.25, the Go runtime now considers cgroups bandwidth limits to set Go max procs, which enhances performance. He also discusses alternative solutions for those not upgrading to 1.25, including using the auto max procs library from Uber, ensuring better handling of CPU resources in Kubernetes environments.

Hi everyone, good evening. My name is Adarsh. I work as a principal engineer at Rapido. And today we are going to talk about understanding a problem that we face, basically a throttling issue in Go based applications when Go max procs are not set. Before that, we are organizing a KCD in Cochin, India. If anyone is from the region, please do tune in. It's going to happen on

April 18th in Holiday Inn. Cool. Containers. Containers is a way of isolating processes, right? And it is built using basic Linux constructs like namespaces, which basically limits what you can see. Cgroups control the resources that you can use. Chroot basically giving you a file system of your own if you from a directory, right? And that's about containers, Go. Docker is built on Go. Kubernetes is built on

Go. We have a lot of cloud native applications built on Go. But Go until 1.25 was not container aware. We'll try to understand a bit more about how concurrency happens in The basic unit of concurrency in Go is a goroutine. They are similar to an OS thread, but they are lightweight and it starts with a 2 KB stack. And they're multiplexed into n number of OS threads,

right? So we'll come into that number n in a minute. And the Go runtime which takes care of memory management, scheduling, etc. It's compiled into the binary. It doesn't have a separate runtime unlike Java or other languages, right? Cool. Last topic here before we get to the actual problem and how Go 1.25 solves. If you are not in a position to upgrade to 1.25, we have an

alternative. Resource request on CPU basically helps us figure a node on which you can schedule your pod. CPU limits us basically what is going to control the maximum amount of resources that you can use. How does CPU limits work, right? take an example. We have an application. It takes and it have an API which takes 200 milliseconds to execute. The container have a limit set of 400

millicore. End of the day, this limits even though we are specify we are specifying it from the point of view of cores or CPUs, it is time, basically the amount of time you get on CPU to execute. By default, so Cgroups is what is going to restrict the Cgroups is what we have what Linux uses to restrict the amount of CPU time that you have. If you

go above on memory, you get OOM killed, but CPU again it's a time based thing, so you get throttled basically, right? So there is a default accounting window of 100 milliseconds. That is where it's called the PCF period. That is the window in which you'll keep the CFS completely fair scheduler keeps keep a tab on your usage and it basically it throttles you. How the quotas managed

is basically whatever limit you are setting into period seconds. So let's say in this case we get 40 of CPU time in that period of 100 milliseconds. We'll take a quick example. We have a 100 millisecond periods and we get 40 milliseconds. So we said it'll take 200 milliseconds of CPU time to actually execute the API. But what happens is it ends up taking up 500 milliseconds

because it keeps getting throttled. Now this is assuming only this is the only thing that is running, but there could be other things running on the process as well. By default, how many scheduler threads goroutines are multiplexed into n number of operating system threads? How many scheduler threads will Go create by default? That is based on Go max procs. By default, when the runtime bootstraps, it until

1.25 what it used to do is it used to make this call runtime.NumCPU. But runtime.NumCPU take the logical CPUs available on the node. Take an example quickly. So I have a Kubernetes node which is having 24 vCPUs. I have a pod with a container with a limit of two. Go max procs would get set to 24 because on the host I can see 24 nodes, which is

what the runtime.NumCPU would return. And the result of that is your application would get throttled, right? Now what solutions do we have? Go 1.25 which came almost 6 months or a little bit ago finally made the fix. What it does now is it looks at the Cgroups bandwidth limit that is set and accordingly set the Go max procs. There is a catch here though. If you are

going with fractions, which ideally you shouldn't, you should try to figure out with a perf test what would be the ideal CPU limit and request required. If you go with fractions, it does a seal, so it goes to a higher number, so you'll still get even if it is on Go 1.25 with this And the other good thing is with in place pod resize that has come

in the recent version of Kubernetes. One other thing that happened is keep periodically checking what is that CPU limit based on the Cgroups bandwidth and it keep updating Go max procs. This again happens in Go 1.25. All of this will get disabled if Go max procs is manually set. Now if you're not in a upgrade to 1.25 yet, yet the other option we have is to use

auto max procs from Uber. All you have to do is add the library and add an equal import and you'll be having the same effect as that. And that's it. Thank you.