The Missing Half of Performance Profiling: Understanding Memory in Cloud Native Syste... Dom Delnano

This talk discusses the often overlooked aspect of performance profiling, specifically focusing on memory management in cloud-native systems. The speaker, who has extensive experience with eBPF and works at Gimlet Labs, traces the evolution of performance profiling tools from the early days of computing to modern sampling profilers that operate continuously across cloud environments. They detail how understanding memory usage is crucial as applications grow complex, especially in Kubernetes, where memory usage patterns can lead to performance issues like oom kills. The session explores various memory profiling techniques including allocation sampling, in-use profiling, full heap dumps, and object introspection, illustrating how tools like PProf, Parca, and Pyroscope are utilized to diagnose and optimize memory usage effectively. The speaker emphasizes the need for effective memory profiling as we transition into architectures relying heavily on device memory and novel protocols.

Hello, KubeCon. Thank you for joining me today, and I'm excited to share more about the missing half of performance profiling, understanding memory in cloud-native systems. Before we jump into things, I wanted to give myself a brief introduction. For the last 6 years, I've been working in the eBPF space. Um most of that focused on the CNCF Pixie project, which does observability for Kubernetes. Um I've also worked

at CrowdStrike and some other companies uh applying eBPF in other areas as well. My day job is working as an engineer at a startup called Gimlet Labs, where we're building the first multi-silicon AI inference cloud. Um and you can find me at my socials on this slide. Since the beginning of time, computing's oldest enemy has been debugging slow programs. In the early days in the 1960s, you

would take your punch cards, give them over to the operator, the operator would then run your program at some point in the future. Hours or days later, you would get the printout, and if there were issues there, you know, you'd then have to debug and figure out what was going on. Moving on to the days of personal computing, I'm sure we've all had that experience where Sophos

or some security scanning software runs on your computer or you run something expensive, your fan kicks on, the Mac spinning pin wheel all shows up, and your computer grinds to a halt, and you're left with what's going on. And even today, despite all the sophistication in the modern cloud environments, with how distributed and complex things have gotten, a timeout error like this is often extremely challenging to

debug. There's so many areas to look at, and so despite our software getting more and more complex, slow programs have always been something that's been challenging. As any good engineer, what this forced us to do was form the uh area of performance profiling. And specifically, we started kind of in the CPU profiling area. So, looking back to the 1980s and 1990s, we had this tool called gprof.

It worked at the scope of a single program. The way you instrumented the program was through recompiling it. You'd have to provide this {dash} pg flag. This was done ad hoc since you had to recompile the program, and the way the instrumentation worked, it actually, you know, added a whole bunch of stuff to the binary, and so the overhead was very high. Moving on to the 2000s

and 2010s, uh sampling profilers became a thing. Tools like perf. This changed the scope of these profilers from a single program to a single host. And this also meant that you no longer needed to instrument your binaries. You could just run the tool whenever you needed to. So, they were still ad hoc, but this brought down the overhead tremendously. From there, we now have the continuous profilers

that we have today that were kind of initially started in the 2010s, but now are widely available throughout the cloud native ecosystem. This significantly improved the scope. We now have these profilers that can look kind of per program, per host, or even amongst all of your entire fleet. Um and really the main difference between this and the previous sampling profilers is that we got this always-on usage.

You no longer needed to, you know, realize something was slow and try and catch it in the act. You just had this available informa- this information available to you at your fingertips. So, with that, we solved all of computing's performance problems. Not exactly. I'm sure many of you have seen an issue like this before where your Kubernetes pods are running just fine and then memory usage continues

to creep up and then all of a sudden they hit the pod limit and Linux oom kills the processes. These then restart and their memory starts off, you know, at a normal like much smaller level and it takes a while for that to creep back up. But, the difficult piece here you don't really know just from seeing memory use, is this a memory leak? You know, is

the application requesting a sane amount of memory? It might just be that its working set size should be over 2 GB. But, you really can't tell from just this static memory usage metric. So, what we're going to be talking through today is the tooling available that allows us to really peek inside of this and get a much better understanding of how program memory works, the tooling available,

and some of the tradeoffs associated with it. So, with that, we are going to first get a shared understanding of how programs use memory. So, we have this sample C program on the left-hand side and what we're going to do is kind of step by step go through the program and understand how this changes allocations on the stack and allocations on the heap and that will form

our basis for when we look at these tools, how they help us understand that type of usage. So, the very beginning, the main function gets called. That pushes a frame onto the stack where we have this integer also included as part of that allocation. The program calls the process function, which takes in that size argument, so that n equals 10, the argument itself, and the variables inside

that function definition also get allocated on the stack. From there, uh we hit the first heap allocation, where the we initialize a certain number of char bytes. And in this case, we're doing 10 because that's what the process function was given. Um then the next allocation happens, where the span struct also is allocated 10 times. And then from there, we call free, which deallocates the Uh it's

likely not zeroed out, but the memory allocator from that point forward knows that it can reuse and repurpose this memory. One other thing to note is that even though these are this heap and the stack are viewed side by side, these actually are one contiguous address space, and so they actually grow towards each other, and you know, in rare cases, the heap could actually, you know, collide

with but so for the purposes of this example, if we assume that that span structure is eight bytes, this function allocated 80 bytes for that span struct, and 10 bytes for that char array. And so as we look at these memory profiling tools, um a lot of like the kind of main concepts that they provide is you're we're going to look at total allocations, which is just

like the cumulative bytes. It'll also usually provide some form of like object or struct representation, because that helps kind of tie it back to the code. Um and then there's also in use metrics, which are basically like the net from what you've allocated against what you've deallocated. So, for this example, if we sum our bytes, we allocated a total of 90. There were 20 objects because we

had 10 spans and 10 instances. And the net in use was actually zero because this function freed everything that it allocated. Now that we have that shared understanding, let's look into some of the memory profiling tactics, use cases, and kind of rough overhead for some of these. So, the first we'll look at is called allocation sampling. What this means is when you have a memory allocator that

supports this, there's going to be some probability associated with It's not going to keep track of every allocation, but there's usually some threshold. So, like every 512 KB allocated, it's going to sample the next allocation. And so, what this allows you to do is this identifies allocation hotspots. And it also is really helps to track down really bad offenders of memory use because if you allocate something

over 512 KB, it's always going to be sampled. But if there's a number of smaller samples, it will only happen if those small samples happen frequently. And the other thing that these mechanisms do is they capture the call stack when the sampling occurs. So, that not only gives you the amount of data allocated, but it ties it back into your application code. And so, you can see

here on this right-hand side, we have that this Go application had its handle request function and JSON unmarshaling identified as part of this recent And so, basically, this gives you the ability to have kind of like temporal data as in time like it you know, these occur at specific times. And so, this is something you could graph and look at in a nice Grafana chart. And this

type of profiling is often seen in tools like Go's PProf, gperftools, and TCmalloc, which are part of Google's like larger umbrella of of PProf. Next up, there is the in-use profiling. This is actually usually paired with allocation sampling profilers. Um and it works very similarly in that you need to track the allocations, but on the garbage collection or deallocation path, when a certain chunk of memory is

deallocated, the sampling piece will check to see if this is memory it had already been keeping track of. And if it was part of that, then it will count that as we've reclaimed it. Um this is so that when you're comparing you know, when you've sampled the memory being allocated, you you need to make sure it's only opted in if that was memory we were already keeping

track of. And so, again, this is low overhead cuz it's sampling. Uh the use case of this is this allows you to detect leaks. If you're allocating memory, but you're never freeing it, unless it's expected that that stays, you know, allocated for the lifetime of the program, there's probably some type of logic error there. And so, this gives you that nice tie back to the source code

because it samples the call stack when when these occur. Um the next memory profiling tactic is doing a full heap dump. And so, unlike the last two things that we looked at, this actually has extremely high overhead. Um and what this provides is basically an entire view of the program's memory at that particular point in time. The reason why this has such a high overhead is because

in order to look at the entire memory of the program, you basically have to stop it while you do the heap dump capture. Otherwise, there's risk of corrupting the data. And so, this is kind of popularized by some of the older tools like Valgrind. Um JVM has heap dumps. Probably your programming language of choice has something here. But they're often difficult to use because of their high

And this final one that we'll look at, which is kind of a newer concept, is called object introspection. So, for many programming languages, you have container types. You have lists, you have dictionaries. with some of these programming languages, they usually have heuristics to determine whether they can allocate small containers on the stack, and whether larger container sizes then later get moved to the heap. basically by introspecting

these objects, if you know the distribution of how large these containers are, you actually can make smarter data structures to make sure that your container only allocates on the stack, and you avoid heap allocations depending on what your trade-offs are. The difficulty here is that if you don't know your container's distribution in your production environment, picking some value for this is probably going to result in worse

performance. But so, that's what these types of tools aim to help is making it so that you can understand where this distribution lies, so you can make a smart choice. And now that we've sort of carried covered that background, this sort of sits in the middle. It's not low overhead because you do have to kind of parse these data structures at runtime, the use case here is

that you can almost completely avoid heap allocations for certain types of workloads. Um and this was actually a tactic that uh Meta open-sourced an experimental project for also called object introspection. And so if we look at this diagram here, what we're looking at is if we imagine that we have a thousand instances of a container type, and each of those containers has some different size that matches

this distribution. If we were to arbitrarily If we were to store this in C++'s STD vector type, um it by default stores everything on the heap. So we would get a a thousand of heap allocated integers. If you compare that against Now that we know the distribution and right around that 20-element mark, we know that if we used a small vector, which is from Facebook's folly project,

um you essentially can get 91% of that data allocated on the stack and only 9% of it will be heap allocated, which for certain use cases can really speed things up. Now that we've gone through those tactics and understood, you know, some of these principles about round what projects uh provide for memory profiling, let's give pprof an introduction since that's come up as a kind of example

umbrella project. So pprof itself is is kind of a broad thing. The first component of it is that it's a profiling data format in protobuf, and it supports CPU profiles in addition to memory profiles. I would say it's sort of becoming the standard. OTEL has its own profiling format, and it it well with Protobuf. But it essentially allows you to have this data collected from one tool

and you, you know, can visualize it or apply it in some other tool. Um the instrumentation piece of Pprof is size-weighted sampling. And so that is that uh sampling allocator and in-use profiler, those first two examples that we talked about. The kind of third part of Pprof is that it also allows you to visualize these And so that's how we get this nice diagram here that you

see on the right-hand side. What this shows is this is a heat identifies the actual function calls and how much cumulative memory they've allocated throughout the lifetime as this uh profile was captured. now that we've gone through and understood some of the concepts from these memory profiles, let's take a look at some of the tooling that exists in the cloud-native ecosystem and how these tools provide those

types of profiling. So first up, we're going to look at Parca. And the profile format that it supports is Pprof. So you essentially give it the ability to scrape your Pprof data and it will happily ingest that and then allow you to view that in their flame graph visualizations or their time series views. So typically this is done by exposing an HTTP endpoint similar to how the

Go Pprof tooling works. Um and what this provides is you get allocation sampling and in-use And on the right-hand side here, we can see what the sample Parca configuration looks like if you had an application that had this type of endpoint. now this depends you the programming languages that they support are go and that's because Pprof is extremely well well done in Go's ecosystem. You can also

add Rust support by switching the memory allocator. And then there are Python and Node.js integrations that allow you to also get these similar endpoints. Moving on to the next project, we have Pyroscope, which is built by the people at Chronosphere This actually doesn't have like an import input profile format and the reason is is their integration works through a push model. You essentially install their SDK in

your application and that automatically does this sampling for you and it sends it to their back end, which is later visualized. So, what this SDK integration looks like is what you see on the right hand side here for this Go application and it tells it you have to explicitly say which types of profiles you want to opt into. And as you can see these match very closely

to the concepts that we've been talking about. There's the allocated objects, allocated space, and the in-use equivalents. And so, like we talked about, this gives us that allocation sampling and in-use as it's no surprise, Go is supported here because Go has that native tooling built in and they also support Java and .NET. Looking at what this type of visualization actually gives you from we get that temporal

allocation view like we were talking about, so you can see over time how these allocations are happening. Um if you were this is currently the allocated view, but if you saw the in-use objects, you know, we would expect your program to be at a relatively stable baseline, Otherwise, that shows that you have code paths that allocate data but never allow it to be reclaimed. And then on

the bottom half, um a lot of these profiles, because they capture the call stack at these allocation sites, you can actually generate a very nice flame graph to show you where your allocation heavy code is. In this case, uh the kind of widest towers of these flame graphs make a lot of sense. The encoding JSON piece right here is turning some binary representation into text. And similarly

with the log res logger, logging is a lot of string templating and building strings. So, this kind of shows you which pieces of your code are the worst offenders for those types of allocations. And finally, we have Pixie. So, Pixie is actively working on memory profiling support. It's not available right at this moment. Um we are planning to support that P Prof endpoint ingestion similar to Pa-

to Parca. Um the actual collection and ingestion piece already works, but we are just need to tie it into our flame graph visualization. But, what we do currently use this for is Pixie as an observability tool stores a lot of data. And so, people often want to know, you know, what is Pixie's memory doing? And so, we've built up this tooling because we've needed to optimize and

figure out our own memory footprint. And so, we have this PXCLI that lets you kind of query Pixie's data sources, and it natively exports P Prof. You can then use the P Prof CLI, like we've been talking about, and visualize that diagram, and that gives us that nice representation on the right. And so, that's actually been how we've debugged and uh improved Pixie's memory over the last

6 to 8 months by doing this exact type of investigation. One area that's Pixie's doing something a little differently is that we are looking into this object introspection use case. So, Pixie has the ability to deploy dynamic instrumentation, meaning you can there's some out of the box stuff that we collect from eBPF and from the network, but we also have the ability to probe programs and kind

of calculate metrics and then provide that as part of our data analysis and processing. one thing that we are prototyping right now is being able to visualize those runtime distributions of container data types. And this is largely motivated by further optimizing Pixie's memory footprint because for many of the network traffic and other things that we intercept, it's just a lot of data. And so, knowing exactly where

you can squeeze additional allocations from becomes extremely important. Um and this will also just be available to other, you know, native applications initially to understand their container sizes. And so, this diagram here is showing what a pixel script looks like for a given container type that has a mainly very, very small number of elements, but seeing that distribution is how you can really tweak the trade-off between

the stack and heap allocations. So, now that we've stepped through those cloud native projects, let's take a look and kind of broadly uh compare these tactics and also what they provide and their overhead. So, the first technique we talked about was allocation sampling. And that basically tells you the total number of bytes or total number of objects per call site. It gives you your code that does

the most allocation heavy work, so that you know where to fix frequent garbage collection or fix high CPU from those And typically low overhead and as we talked about Pyoscope and Parca support these use cases well. There's also the in-use profiling which tells you how many bytes are currently used by the different pieces of your function calls. And what the purpose of this is mainly to track

memory leaks. And this is also typically done in a sampling manner so that it's low And as we discussed Pyoscope and Parca widely support this. The third tactic is the full heap capture. This does the the full stop the world snapshot of the process. This is really only needed if you need a detailed point-in-time memory snapshot um because it has to completely pause the runtime of your

program. not recommended unless this is like very specifically what you need. And finally, there's the object introspection piece that we talked about that Facebook kind of pioneered and Pixie is exploring where you can measure for data structures that have small strength small container-sized stack optimizations. You can understand the distribution of those so that you can purpose-built assign these data structures the right size to save all heap

allocations. And this kind of sits in the middle in terms of the overall overhead. So, recapping performance profiling, where we are today is that CPU profiling is mature and easy to adopt with the available tooling. For memory profiling, you want to prefer sampling-based memory tools, otherwise they won't be fit for production. The particular uh size-based sampling that PProf takes is low enough overhead while still gives you

good signal on problematic allocations that that seems to be the latest and greatest. And additionally, PProf has basically standardized these CPU and memory profile formats. And finally, in terms of the broader memory profiling ecosystem, it's still extremely fragmented due to memory-specific requirements. Many of this tooling still has tons and tons of overhead. I'm sure as, you know, this tooling continues to build, the those things will change

and and things will standardize in the same way that CPU profiling has. The final teaser that I'll leave you with is that this is sort of just the beginning. Cloud applications today largely use host CPU memory. But, in the world of AI applications, we sort of have totally different memory We're moving to a like an architecture where device memory is starting to become very important. These weights

of the model and key value caches often are significantly larger than typical heap sizes. RPC is sort of moving from typical microservice traffic to actually remote direct memory access. So, we sort of have processes writing directly into remote host memory. So, memory is sort of becoming a much bigger part of these new architectures. this tiered memory is going to start to be commonplace. And so I really

see understanding and being able to measure memory use is only going to be more important as we have more GPU memory, more accelerators that have SRAM based architectures, and more protocols like CXL that allow you to actually access memory from another host. With that I want to thank you for attending my talk. Um you can find the links to the Pixie project here and our documentation. And

I'm also happy to answer any questions if you have them.