Ballroom F Sunday Mar. 08 - SCaLE 23x

In this session, Nom Levy, the field CTO at Ground Cover, discusses the importance of profiling as a critical pillar of observability, particularly via eBPF-based techniques. The speaker emphasizes how profiling can provide insights into resource allocation, focusing primarily on CPU profiling while also acknowledging memory profiling's significance. He elaborates on the concept of ad hoc and continuous profiling, allowing developers to gain insights into how applications behave over time. Levy highlights the accessibility of profiling tools, the challenges faced historically in operating systems compatibility, and the advancements brought about by the OpenTelemetry project, which has recently standardized profiling as a signal. He also discusses the role of eBPF in enabling zero instrumentation, allowing engineers to gather performance data from applications without requiring changes to the application code or production environment.

wipes. Hello. Hello. Shut that off. Say hello. >> Hello. Hello. Yeah, >> good. I think it's best to put this like in your back pocket on your belt if you have a belt. >> Yeah. No belt or pocket. Whatever you're comfortable with. >> That's great. That's great. And then if you want to shut it off, just tap it to shut it. I think actually no. Hold it

for one. One, two, three. Awesome. >> I need to see or body can hear me. Yeah. Awesome. Cool. Uh yeah, I think we'll make it timely. So, I'll just uh go ahead and start. Uh in the spirit of efficiency of uh this conference, this actually might take less time than one hour. So, hopefully I'll be able to give you back uh some time for the rest of

your day. Uh nice to meet you everyone. My name is Nom Levy. I'm the field CTO in uh ground cover. Ground cover is an observability company. Uh we do EBPF based tracing and metrication of applications. Uh today I'm going to discuss uh the fourth hopeful the hopefully the fourth pillar of observability uh which some of you may had experience with in the past called profiling uh and

how it is becoming quite uh powerful and even more accessible than ever. And it's a really good time to engage with profiling more and more. So profiling is a very powerful concept. It is becoming very accessible, much more accessible than it used to be. And we're going to tap into how it became accessible. And most importantly, soon enough, if things are going the pace that they are

going, uh it's also going to become much more frictionless. let's talk about why profiling is powerful before we discuss that it's accessibility. So a small 101 on profiling. Anybody here had experience with working with profiles in the past application profiling? Yeah, it's uh it's one of those pillars that is I would say the most intimidating of them all. Uh very hard. It's a bit hard to consume.

We're going to tap into that as well. But in general profiling give us a lens into how resource application how resource allocation uh is done by our application and by the third party the application is using uh not just CPU but also memory. In this session I'm going to mainly discuss CPU profiling which is the I would say the most the more popular pillar although memory power

memory profiling is also a very powerful technique. When we talk about profiling there is a concept of ad hoc profiling. So for those of you who developed in Golang for example and use pro then we can ad hoc create profiles on a running application and there is the concept of continuous profiling basically doing profiling on running applications even in production for a long period of time and

see how our application is behaving throughout the life cycle and it gives us fairly good granularity. uh we can get basically up to the function call level when we're uh looking at our application from the lens of profiling. Why we profile application? So what kind of when we discuss why we do things better a better way to think about it will be what kind of questions we

try to answer when we profile application. So example questions is the most basic level when we talk about CPU profiling is where do we where our application spends most of its CPU in in which procedures. H another pillar another type of question and this is also going more to memory profiling is where do my application is doing allocations and deallocations of memory throughout the life cycle. This

is where stuff like garbage collection activation kicks in when we profile applications. And finally, and this is kind of like the the the the issue that everybody's looking for and very rarely it is always it is the issue is that we try to blame the garbage collector for why our application is slow. TLDDR usually it's not the GC fault, it's how you use the software runtime that

you are working with. Uh but that's very like people sometimes look into a profile and they see like this big chunk of like GC allocate free. Yeah. >> Can we check? >> Yeah. So a very this is a great question. Profiles also gives us access to Cisco operations and we can see how much time and depending on the profile configuration we can also see how much time

our application is idling waiting for uh sys calls to fulfill. Uh so also network operations, file system operations and yes um few words and terminologies just to set uh the level straight that you might hear those terminologies. I think that especially when we're talking about profiling, uh I can tell you that even me getting ready for this uh presentation really there are no dumb questions and sometimes

things are uh being communicated as simple while it's not necessarily simple and it's actually very crucial to understand the basic uh the basic terminologies in order to have a good understanding of what we do with profiling. So when we investigate profiles, usually this this is the picture or the kind of visualization you're going to see when we're looking into profiles. We call this flame graphs. Um there's

a lot of work around how uh how you visualize and how you color flame graphs. It's a whole philosophy and I'm going to tap into that once we're going to look into some examples. The way we profile applications and we time operations is by walking the stack or wind unwinding the stack as calls are being done. So in the profile picture that you can see here for

example as we go up is more and more function in nested function calls within the original let's say wrapper call. And basically profiling is the act of walking through that stack up and down and time the procedures and how much time they took. And another terminology another word that's going to be center to this presentation it's not related to profiling specifically just a big enabler to the

accessibility revolution and the frictionless revolution that wiling is having is ebpf. Anybody here heard about ebpf or used? Awesome. Wow, that's like so the the amount of heads hands being raised when we speak about EVPF in these conferences gets higher every year and this is a great news. Uh so we're going to tap into that and uh great. So now let's talk about What is accessibility? And

sorry that I'm like it looks like I'm dumbing down the conversation but I think it's very important to understand what questions we're trying to answer together here. So accessibility at least the way I look at it is how easy it is for me to implement a pillar or a signal into my application and the other side of this coin is how easy it is for me to

consume this pillar. how easy it is for me to get conclusion and actionable items because generating signals in applications yes sometimes it cannot it can be hard but why go through this if I'm generating signals that end up in a dashboard that nobody is making sense or more importantly taking actions according to those signals that I was generating so let's talk about implementation in of obser of

profiling profiling is kind of of a fragmented field. Um, it's runtime sensitive. So, every software runtime has its own especially binaries uh and and software runtimes that are not h compiled uh just in time like rust binaries or CPP binaries they have their own issues when it comes to uh profiling. Uh it's a pillar that people here usually I assume that some um a lot of people

here have have experience with traces and with metrics and with logs. Those are pillars that usually appearing in a single or a unified on a very few observability systems within an organization. While profiling was kind of like the advanced pillar that only a very small portion of the engineers in a company had the ability to even work with to work with and to implement. So it was

never really contextualized with the rest of the pillars and with the rest of the signals that application was generating which is bad. We want contextualization. We want to look at a metric in the light of the profile that we were we had. want to look at a profile in the light of a trace that we got. Um, also there were big challenges when it comes to operating

system compatibility with profiling in the past. There was a tremendous work in operating systems in Linux specifically in making it more profiling friendly. By the way, the and I'm I'm going to mention it in the end. There is a person that I highly suggest looking into his blog and he's going he's probably one of the uh most amazing engineers of our time especially around performance engineering named

Brendan Greg. He's one of the main contributors to EVPF and also to the concept and uh making profiling an accessible uh technique for engineers. He speaking a lot about the compatibility journey with profiling in his blog. highly recommended read. So these are solutions that are implementing profiling today used to at least. So elastic search para is an open is a is a popular uh profiling project and

graphana spyroscope which is a backend for profiling and we're going to touch it as well these days and hopefully some of you also heard about it. Anybody here heard about open telemetry? Awesome. Open telemetry is pretty revolutionary project that basically has the mission of aligning the way we generate and ship telemetry from our applications regardless of the vendor. One of the I would say the core issues

of observability historically speaking is that the concept of observability were gen were evolving in proprietary landscapes uh in proprietary vendors and it made the action of adoption and moving from one solution to another very hard. Open telemetry kind of like basically won in the last three years I would say got super mature and it is now pretty much the universal and the go-to framework when you want

to to make your application generate signals and to ship them to the vendor of your choice that you are using at this time at this point most vendors if not all of them are open telemetry compatible and those are not are in serious troubles going forward so Far open telemetry standardized the way we ship and we generate and ship logs distributed traces and metrics but profiling as

a signal was never standardized. A year ago I believe it was a year ago maybe it's almost two years ago now the signal of profiling is a separate signal was standardized by the open telemetry project. It is still in beta. The the the I would say that the ecosystem around profiling open telemetry is still very much experimental. It's something important to have in mind. But there is

the acknowledgement that profiling is a signal that needs to be standardized and it has an experimental support in open telemetry collector. And I'm going I created also a lab for you to try out and I'm going to share it in the in the end of this conversation. Why it's important that we'll have a standardized ecosystem for generating signals. First of all, it allows us to be portable

in an era when there is all everending innovation. Our ability to adapt technologies into our code is something that is very important from decision making that will make sure that our decision have a long-term outlook, a long-term runaway. We don't want to adopt proprietary technologies to our applications especially around instrumentation for example without thinking about well is this going to be a troubles soon if I want

to for example try out another solution another uh point and for me it's not less important than the actual maturity of the project in any given point in time is how referable the project is the moment we have as a software engineers As software engineers a reference that all other software engineers are using it's much easier to adopt technologies it's much harder to adopt SDK just relying

on the docs of the vendor and not relying on examples from open source solution that tens and the dozens and hundreds of other organizations are adopting themselves. This is a very powerful primitive to have the entire industry across different companies with different challenges aligning to the same terminologies. Basically, we are all all of a sudden speaking the same language while in the past we're all speaking different

languages. And another very important thing about open telemetry that across the software runtimes across the different configurations the semantics are aligned. There is no different semantics or environment variable or configuration when we talk about open telemetry in CPP compared to C# compared to Java. That concept into open is called semantic convention and it's very crucial for the universality of the project itself. So let's do a small

10. Open telemetry also referred as hotel. Let's do a small 101 of how pipeline looks like. So when we talk about telemetry generation especially about uh the pipeline part of it not the consumption part of it which is not universal still uh there are three steps we receive data we modify it and we export it and that's the same terminology that we will use in the open

telemetry configuration. So this is an example of a of a simple open telemetry Open telemetry define configuration will define a receiver other than the spec and the collector binary itself. Open telemetry also define a protocol which is called OTLP open telemetry line protocol which is very important because as long as we ship data also from open telemetry SDKs or from open telemetry solutions they are all consolidating

to the same endpoint. They are not speaking in different protocols. Less ports less configuration all align to the same Then we define a processor. A processor is where we uh define h mutations on the data that is being received. For example, and my talk is going to focus on Kubernetes environment doing profiling Kubernetes environment. In Kubernetes, we have the concept of enrichment. So we get data signals,

but we want to enrich them with the meta data of the pod namespace service name. So processor and open telemetry is where we are able to do this kind of enrichment. In processor step we can also do drop and filtration and static absertion upsert of values of our choice. And lastly we're going to do an exporter which is where to ship the data after the rece the

reception of it and the processing of it. It's cool about open telemetry you can ship the data to multiple exporters. uh so and then last step you can see here that I'm defining the service where basically I'm creating the chain of receivers processor and exporter and this is very cool because I can create multiple pipelines that every pipeline can for example decide to take only a set

of receivers run through a set of processors and ship to select exporters and each service pipeline can really uh ship to different I can drop the data in a bucket for long-term storage but only a subset of it and then I can ship it to my observability vendor with a different retention period and so forth. So it's very it's very declarative kind of like the same philosophy

that we have today behind Kubernetes and very very flexible. This was the config the the part of the collection of the data. But what about the generation of data? So usually when we talk about open telemetry and an observability about signals in general we used to had to do something called instrumentation. Instrumentation is the act of mutating our code inserting code into our applications to generate signals

from within. The pros are obvious. What are the pros is that we are able to control where and what is the scope of the signal that we're generating and the engineers know their application. the best the best hopefully right theoretically speaking problem is is that the same pro is also the biggest disadvantage of this method is that we rely on engineers to generate the data. So in

a sense when we observe applications we want to have a an angle of I would say um zero trust to the signals that we generate. If we are bounded by what engineers are doing in their applications then potentially we have blind spot in our application. Think about it like security related aspects. If an organization would say that they are secured because engineers are not clicking on suspicious

links because they swear they don't click the on suspicious links in their mail. That's probably very bad security. Good security is having um malware detection, antivirus, fishing protection elements that are not assuming that they should trust the users or the uh or the actors. Same kind of posture will be super beneficial for observability as well. So how do we achieve it? This is going to I'm going

to discuss it soon. But this is where basically the challenges and the state of implementation ends. Now let's talk about consumption. We were talking about accessibility. These were the challenges of implementation and how open telemetry unblocks them. Now we have the ch we need to discuss consumption. So as I mentioned profiling is a pillar that is hard to consume. Like I can give you an example. Uh

most people always look at the flame graph. Half of them at the beginning think that the x- axis is time which is never is and then everybody's okay so what's the where where where things are going what does it mean uh or they see that the most time is spent on the GC on the garbage collector and they will be like okay then what do I do

like it's not it's not my code right I just I just allocated an array like how can how can it be so slow, how why the garbage collector takes so much time. So it's hard to understand and so far only few humans in an organization could make decision based on it. But we are now in the a in the era of agents and we have a superpower

that can take the same profiling data and help us even if not remediate it completely at this point but can help us see through the complexity of profiles. Now will I tell you that this is that agents read profiles perfect? Of course the answer is not. Uh this is not LinkedIn. I'm not going to tell you the end is near because I was able to figure out

the profile but it's definitely something that can help you go through the hurdles and to give you context about profiling as you look at the at the data. So in gen generally speaking going forward the assistant of AI or the ability going forward of AI to potentially even get conclusion from profiles causes leads to the bar of using profiles and make benefits out of it much lower.

So if we solved if we solved implementation or like not solved implementation but the very very least implementing profiling now is a very is is an effort that seems to be a long-term beneficial effort. There is no vendor locking when we are talking about implementing profiling anymore and the consumption of profiles is only going to get easier. Then maybe it's a very good time in software engineering

to introduce more profiles and to push organizations to use this pillar that can provide huge benefits when it comes to performance engineering. The last remaining question so how we actually put this all together and make it a frictionless experience going forward. So let's speak about eBPF for example for a second. For those of you have not are not familiar with eBPF, eBPF is a kernel sandbox that

basically allow us to load programs that gives us visibility into running applications and to running operations within the kernel. The program is loaded agnostically to the user programs that we're getting data from. The notion is that we basically hook operations in the kernel and in the user space and then we're able using specific functions to exfiltrate data from those running procedures. As Brendan Greg, one of the

ushers of the EBPF revolution said put it once. This is like putting JavaScript in the kernel. This is basically the times that we live in. Here's a small example of an ebpf hook. So the way it works, we basically load a program that is decorated with the point where we want to hook. In this case, you can see that I'm loading something called a krobe that will

hook a sys call do unlink at unlink is the sys call that fulfill the delions of files on the disk. And basically what is going to happen here is that I'm loading one program that every time unlin cys call is being fire invoked this program is going to run and the program under it is called a k is a k probe hook basically hooking the return from

the sys call. So when we're back from the kernel to the user space after the unlink sys call was fulfilled this second program is going to run within the probe. I have a set a limited set of instructions that I can run and as a result also and this is the sandbox aspect of ebpf. have a limited set of finely tuned and very well thought of functions

which are called EBPF helpers that I can use within the probe. In this case, what you can see in the example here is that whenever I do an unlink whenever an an unlink sys call is being invoked, I'm going to print using a special print helper that ebpf of the sandbox provide us to show which process can be the rm command for example um is in is

is invoking the sys call and for which file name and as The sys call is being fulfilled and I'm the sys call is done and I'm going back to user space. My unlink kret probe is going to run and then I'm going to print the results of it. Um this is by the way a fairly old example of a like this okay probe is the I'm not

sure if it's the old school way but uh now we also have new primitives in eBPF for hooking kernel um events. uh we use F entry and F exit now which are a bit more performant although K probe do does allow an extra flexibility um it's there are tons of other there are a few others probes that we can use with eBPF eBPF can also use to

shape network traffic this is the famous for those of you who are interested in uh network performance using eBPF u meta are probably the leading organization of implementing ebpf at planetary scale Every packet in meta goes in h goes through an ebpf program these days. Uh and we have open source project like ITO and uh selium uh that are leveraging ebpf to do network shaping and we

are also able to do probes on user level applications as well. What does it mean that we can do it in terms of like even I would put it as like a business objective. What are we unblocking using this technique? So ebpf unblocks the a technique called zero instrumentation. Basically we can now generate signals about applications without requiring engineers to modify the applications themselves. This is a

huge primitive for any organization. I can now generate signals about applications that are being developed by tens of thousands of agents running from engineers that are looking very very briefly at the output of their code or for their codeex or cursor or whatever uh generated code and without relying on whether the code used instrumentations, I can still generate signals about potentially and surface new behaviors that those

applica these applications are presenting if having zero instrument I would tell you that when we even started ground cover 5 years ago we and we're still big believers that observability cannot rely on instrumentation as a whole it cannot be like a strategy for organization to assume that all the signals will be generated by engineers this was five years ago today with AI with aentic develop velment. This

is the amount of software that is introduced into production which much oversight is unprecedented. We're basically shipping 100 times more code in some environments with much less attention to the code being shipped. the ability to generate signals without relying on instrumentation just as a concept without even talking about profiling is pretty much essential to us guaranteeing reliability of the software that we ship. So PBPF uh we

were able to do signal generation for metrics and for traces. So we can do APM, we can generate spans, we can generate metrics about latencies, about HTTP endpoint calls about the SQL transactions. Um and now finally we can also do it for profiles for profiling. So remember all the vendors I uh I I I named earlier Elastic and Graphfana and uh Parka. basically we are all and

the biggest ground cover is not doing profiling so I can't take the uh the credit for that but the other vendors that were invested in profiling in the past are kind of all consolidating into the open telemetry project as well uh actually I would say that elastic did the extra mile of even contributing a patented code that they do for creating profiles for stripped binaries which is

pretty cool um And basically having this in open telemetry like having in open telemetry an eBPFbased pillar that generate a the fourth signal of observability in a universal way that is going to be compliant with the spec that every other observability vendor and every other pillar in your organization is aligning to is basically like amazing. Amazing news. Amazing news for everybody who's interested in observability. So, and

the then the last benefit of it obviously is that we do not need even instrumentation to get started with it. So, how do we start? So, we discussed the um open telemetry collector and the back end. Um this example is going to be based on Kubernetes. Uh but we're going to set up a pipeline together which contains an EVPF agent. It's going to ship to an open

telemetry collector like we introduced and it's going to ship the data to a profiling back end. We're going to do this all in Kubernetes which is just very easy to uh deploy applications to once you get through the hurdles of Kubernetes themselves. But after that it's pretty uh straightforward and if you're already running applications in Kubernetes you're going to just get immediate value out of out of

this. So let's start with dissecting the pipeline one by one. Start with the agent, right? We need to emit the signal. So, we need to deploy the open telemetry eBPF collector. Few things to know about open about EBPF uh agents in general. EBPF agents are privileged. When you hear about a technology that is based on eBPF, most probably like 99% of the time, just because I'm hesitant

to say 100% of the time, you will need to run it as a privileged container with access to the host system. Access to the host system in Kubernetes means that we need to give it access to the process name space. We need to give it explicitly the security context privilege true or the capabilities like CIS like CIS admin and we need to mount the host volume in

Kubernetes. We need to mount the node the node volumes itself into the pod file system. So the container has access to the entire proc name space and the entire cis the entire cis subsystem within the running host. Next we have the collector. Not sure if the text the text here is too small. So I'll just go through what is it says there. We're going to have an

OTLP receiver because as we mentioned open telemetry BPF profiler is an open telemetry project. Hence it complies with the OTLP protocol which is fantastic. We're are running this in Kubernetes. When we do stuff from EVPF, we have no idea of the notion of pod of a namespace of a service name. We need to somehow enrich it into the incoming data. Open telemetry has a really cool processor

called Kate's attributes Kubernetes attributes. Basically you tell it what is the associ association enchor within the incoming data and then it will request from kubernetes the rest of the metadata. The profiler the ebpf profiler in open telemetry knows the container ID. The container ID is good enough association and core for the processor to complete the rest of the information. And lastly we need a backend. So back

end is still the proprietary part. This is your vendor that supports shipping uh profiling data. In this case, I took the what was the simplest uh open-source uh way to get started with a profiling back end, which is Pyroscope. Now, the back end for profile is going to be a database. When we speak about databases in Kubernetes, we would usually want to attach them to a disk

and make sure that the controller that deploys those the the controller that deploy the application is aware that this is indeed a database. So in Kubernetes, this is called the stateful set controller and this is the controller we use to deploy database and data sensitive We have all this in place and how this is going to look like. So this is basically what you're going to look

at once you're going to port forward the back end or access the back end itself. You're going to get immediate catalog of all your applications in the Kubernetes environment. And for each application you're going to press you're going to see a flame graph. Now the flame graph depending on the back end as I told you coloring is a big philosophy um a matter of conversation in uh

in flame graphs. Pyroscope in out of the box will show you based on the package. So each color here represent a different package. You can also just do the U based on the nesting or based on the length of the uh CPU time being spent. And uh there are other techniques. So for example, Brendan Greg H is a flame graph generator allows you to generate a flame

graph that is going like for example for Java applications going to show you when this is like Java code when this is JIT and when this is like the CPP JVM underlined code. So you can decide depending on like what makes sense to you when you're looking into the application. You can decide on the U itself and again I get this information about how much time is

being spent in the application without doing any changes. So you can take any development cluster that you have running applications in your application is going to appear immediately in the Pyroscope UI and you are going to be able to look into the profiles. This is a good now we were talking about how promising this is entire thing is but it's important to say this is still an

experimental signal and an experimental EBPF agent still early days not necessarily in the sense of stability but in the sense that there are a lot of changes being uh a lot of changes that are breaking changes happening in this pipeline. However, it is still all happening within the unified ecosystem of open telemetry. The other challenge is symbolization. So, I didn't explain symbolization quite thoroughly when we began.

When we do profiles, we want to have meaningful names in the flraph like what is the name of the function. Sometimes binaries when they shipped into production are going through the act of stripping. Basically, we take the symbols out in order to keep the binary small in production and fast. In CPP, for example, we are stripping away the dwarf data. The dwarf is where we have the

debug symbols in uh in binaries these The current philosophy, if I understand the spec correctly, and the plan is to allow the backends to allow you to load um symbols. So you ship the uh profiles with offsets and then there is going to be like a resolver that is going to resolve those offsets to uh the actual symbols. It is important to say this is a strictly

a problem for stripped binaries and for in and specifically Golang for example which is probably popular here even though it's a compiled binary Golang does not has this issue like with Golang in open telemetry BPF collector specifically they are like special casing go because you can always extract the symbols um and then nojs python all of those again no issue interpreted languages uh no issue Let's look

into some examples now. So ground cover for example I'll tell you one of the challenges that we constantly face. Um we run the so we run an observability back end at the customer premise. We don't run it in our cloud. Basically our UI component in the cloud is a very simplifi I would say to put it simply a very it's a it's a fancy proxy. We get

a request and we ship the we delegate the request to the customer back end and then we serialize the request and then we ship the data back to the browser or to the client that is requesting the data. Not a lot of things that can go wrong, right? But for us for example, what we discovered when we turned profile in for the first time and not now

but earlier when we looked into some of the scaling issues when we got a lot of like requests going on in the in the solution we we saw that we spent like a lot of time compressing data. So all of a sudden we understood that we can like potentially gain like really cool wins just by looking into better compression algorithms or better uh compression libraries that are

focusing on uh zero copy or introducing some novel techniques or potentially even look to be on the lookout for bugs. We had like a very nasty it was we we got we we saw it through a memory profile but like two years ago we had like a very nasty uh gzip library bug and that we by moving to a fork that introduced a fix to the allocation

process we were like cutting down the UI service memory by like 40% like something crazy saved a lot of money with AWS this day this that day um so but this kind of operations like for example looking at your application and all of a sudden just realizing wow like compression I never like gave it too much thought you know you use the standard library like I I

never thought that maybe moving to deflate from gzip or try snappy can do magic to my AWS billing or to my application performance in general and all of a sudden the peripherals of your applications are being shine like are getting like this light that was very uh very uh important for us. The other thing and is blocking operations. So it's not appearing quite well but as one

of you mentioned here. So for example when we delegate work to the customer back end we are basically in a non-blocking we we we don't have a reason to block. We're waiting for a response from the server. With profiling you can also opt in for looking into the time spent idle. So it's e a very excellent pillar and a very excellent tool to understand if your application

is idling and to let to to to shine light on areas where like oh maybe I can introduce some concurrency in this procedure and stress the CPU and make like more meaningful work. Um for example for us one of the areas where this is very uh interesting to look in is middleares. So the concept of middlewarees for those of you who don't know HTTP middlewarees is that

I have my endpoint that I the engineers are uh implementing an endpoint that returns for example amount of logs between the endpoint that is returning the amount of logs and the request coming in to the HTTP server. There are a bunch of middleares that are crucial before I let data out. Right? I have security middleares I have rolebased access control middleares. So I need to make sure

that the the user requesting the data has access to this data. I'm checking a database. So all of a sudden before I'm even hitting the logic of the application of the endpoint itself, I might have a query to the DB uh that is taking a lot of time. So yeah, obviously we cache but when cache is being invalidated, how much time I'm spending on the query itself,

it's also important. And when you do tracing like general tracing sometimes you can miss those kind of operations or you see that the middleware took a lot of time but you don't necessarily going to see within the span depending on your instrumentation again depending if the engineer put it in the instrumentation you might not get visibility into what made that middleware specific middleware running for so much

time. So that kind visibility that is not relying on the quality of our instrumentation and that shines light on areas that intuitively we're not thinking about when we're doing performance tuning is crucial for us delivering the best performance possible. Another very cool thing for ground cover and for people who use your EBPF is that all of a sudden we were able to see the overhead of our

own ebpf probes. So told you that we have so we remember the yeah the I really need to zoom in into those examples. Sorry about it. I will explain what's going on. Um so we have here what you see here in the uh the above the light blue um party of the flame graph where it started with ASM there is this big purple chunk. This purple chunk

is a read of is an SQL uh read operation. This is the we use the uh jax cpgx driver for postgra queries in Golang. when we do the read operations like the way our solution works every time we do we see in not in just our applications but in customer applications a DB read operation our EVPF sensor up probe because this is still happening in the user

mode. So I showed you K probes which are kernel probes. This is a probe that we load um to the SSL library call. h ask for sometimes from the CPU to take away the uh payload the the basically the SQL query that was being shipped to the SQL driver. This is how we provide visibility into how much time SQL query took and give visibility into the SQL

query itself. For our customers, it's very important and for us as a company, we cannot do these kind of things without giving the utter assurance that we're not going to steal a lot of CPU time. Now, eBPF basically kind of like already makes it hard for vendors to uh hurt the application performance themselves, but all of a sudden have this tool that allows me to check the

performance and the latency that I inflict on SQL calls myself. uh which is very powerful for h let's say that assuming that you're going to use ebpf tooling on your applications. So up until now we discussed the peripheral such as like the gzip library that you're using your applications are probably also behind the scene being blocked or not being blocked by your probes and k probes and

ebpf code that is being loaded. most of the time it's going to be fine. you're going to see these days a lot of ebpf kobs and new pops in your profiles because for most of the CNIs in Kubernetes in networking for example these day leverage ebpf um but this is again one another this is again another example of how much we don't know about how our application

actually behaving in production at scale can do much more so we talked about the third party effects uh we didn't dive into kernel uh here. So um you can also see in profiles how much time you spent in the Cisco call before application time and not just in the user mode and there is also the concept of memory profiling. So EB or the open telemetry eBPF profiler

is limited for now for CPU profiling. You can also for example in go you have the pop of library where which allows you to do memory profiling as well and then you can see how much allocations are being done and how much memory is being allocated by specific functions in your application. That's another great uh visibility pillar that I suggest trying out. Probably the easiest way to

start at least that I'm familiar with with memory profiling is with Go because it's a tooling within the Golang ecosystem. It's literally one import that you add. Yes. Unfortunately, it relies on instrumentation and then you get this fancy web web UI and it's very easy to uh to generate uh profiles from your applications as well. Starting to conclude here. Um profiling data especially with the example that

we just seen really complements what we call the three traditional pillars of observability. uh we call them MLT and metrics, logs and traces. There's a huge debate. I'm not going into it whether there are three pillars, there are no pillars. It's a it's a hot topic on LinkedIn. I'm going to stick with the uh legacy definition at least for the data type perspective. So we used to

work most of us in terms of people who instrument code with metrics, logs and traces. Profiling is definitely a pillar, a new signal that complements and empower this data. It really allows us to start observability flows such as the two that I wrote in the slides here. So for example, I can start with everybody looks at latencies of their applications and then they look into spans. Now

knowing that I have profiling data, continuous data and I see that I have the span of this specific middleware takes time. Now I can start adopting the mindset of okay let's go to look at let's see the profile data and see why this middleware I can also start from the other way around. So I can start with my suspicion from the profile file and add instrumentation or

add look at the traces and have a much better uh metrication or instrumentation in place. So for example, I can decide that maybe it's not a good idea to generate a span that covers the entire endpoint fulfillment, but maybe I will generate a span just for the quering part and the span for the compressing part. So I have it in a more convenient way in my system.

But for me at least and at least from what I see from as an observability vendor perspective, we at least some of us really try to push for the dream of a single pane of glass. I believe that the observability is not about getting data from a specific data point that your application is generating. It's really crucial especially in environments like Kubernetes to understand how a specific

data point plays along other data points that happens during the same time. It's never an investigation of a single data point that gives you all the answers. It's the ability to understand what happened. What was the node CPU and memory while my application took five seconds to process a request or whether the node was in an unready state why there was a huge memory spike while my

application was sending 500 error codes. It's always a matter of context and our ability to streamline profiling into the same data lake, the same pool of the other pillars that we have and not make it a tool for the advanced user in the for the power users in the company is probably a huge factor in reducing meanantime to resolve and to have better understanding of how applications

are behaving. So let's speak about AI. So I had to do this try and because uh this is like you know this is the call of the hour and I open JGPT Atlas and I opened it on Pyroscope. I just told him told him it's it's very hard these days to stick to the English grammar of it when it comes to agents. But I told it to

tell me what's wrong with this profile. So it's actually did kind of a good job. Now it's my fault that I gave him a very specific profile in time where I think we just I just started the EBPF profiler. So the most of the time was spent on uh hooking the running applications. The at least from what I understood and I I I actually didn't know but

this uh the first call that it's written there the aggregation precore handler is related to K-pop loading. H so I think that I I was like very enthusiastic. I just launched the profile and then I was seeing a lot of time being spent on the EVPF hooking procedure. But as I went down, it kind of nailed what I kind of knew about this application. I it it

pretty much nailed down that I need to look into the gzip implementation and the JSON encoding implementation and suggested to look into operations that are putting more emphasis on zero copy and more performance and making sure that I'm not uh double encoding or gzipping unzipping before I'm shipping the uh the request itself which pretty amazing at even at this point. So together and then I was like

I was thinking okay EBPF kind of like unblocks generating signals without human intervention and then AI from the other end and agents are able to analyze the analyze the signals without human intervention with very minimal orchestration or guidance. this like plays along really well and this allows me from I wouldn't say anymore like a software engineer perspective but like from an operator perspective to take this to

to to take this uh method and to leverage it while I'm writing software. One of the things that we are like that at least now I'm trying when I'm writing code is to connect with MCP to my solution to my observability vendor which is my case myself and when I'm generating when I'm letting the agent generate the code I'm asking for it to rely on the data

coming from production. Now this is un this concept regardless of profiling. That's if we zoom out for a second. If you think about what is missing for agents more than anything else when they are generating code today, it's not the code. We do 1 million context window these days. Context is no longer the issue of the code. The only thing missing in the context is how this

application actually behaves in production. How this application actually plays out. what alerts the engineer and the business have on this application. And all of a sudden without even relying on the agent to generate instrumentation data, I I can make the agent potentially write much better code, much more senior code by leveraging data points about how applications actually behave, which I think is, you know, we talk a

lot about harness and how to optimize code generation these days in agents. This is in my opinion at least the next 10x for agent efficiency when it comes to the quality of the code being generated. What I want you to remember potentially from this session. So profiling adoption becomes easy or more straightforward or more h something that you can rely on for the long term because we

are now using standardized tools. It's only going to get better from here. like this is a very good time to introduce something and then get happier every two weeks because the releases are kind of like huge. It's not like they're it's less about regressions, more about oh wow, they introduced this. Oh wow, they cut we still had the oh wow it's now takes twice less memory. Wow,

it's like we still have a lot of wow moments. So it's our applications are doing a lot of stuff that we are not aware of and ebpf together with agents but ebpf as a deterministic technology that is already mature and available really unblocks a human agnostic observability. Not in the sense that human are not needed but in the sense that humans are we are not relying on

the engineers to generate signals that are crucial to understand the applications. And as I mentioned during the talk I also want you to take away to go to the Brandon Greg blog from here and dive into the wonderful world of profiling as well. This QR code uh is will take you to a profiling lab that I created for you. Uh the idea was simple because uh by

the way these stuff are not simple unfortunately today so it took some time but the idea is to hopefully if you have a running kubernetes environment with to run a single command of cubectl apply and you're getting the entire lab up and running without any configuration. It's going to deploy it in its own namespace. It's going to load the demon set. It's going to do the Kubernetes

enrich metadata enrichment for you hopefully and let me know in the issues. I made it a public repo. Um if it's not working out of the box in Kubernetes, let me know. I'll try to make it very uh convenient for you. Yeah. Any questions? what steps. >> So if you will go to the open telemetry ebpf collector library the the GitHub repository you have um you have

the ability to run the agent locally on your machine and it should work as well. uh once it will generate the data I would still recommend shipping to so I I don't have this lab working I will just give you the outline you probably want to run docker that will run the pyroscope and the open telemetry collector as containers and then just run the ebpf collector locally

on your machine where the go application is running at that point you're going to get your profiles you're just not going to get the nice name uh you are getting from the ebpf Um the tags that you will leave are the you will have is the P ID and the process name. So you are going to be able to find your application still just not going to

be like fancy. Okay. Yes. the so running EVPF is not like just compiling the program. There is a tooling around wrapping and loading the applications. Uh there are other repositories that I suggest using when you're trying to package a binary that flows evpf program. Uh I believe that um isovalent yeah if you will go to ebpf.org or the there is like an organic foundation around eBPF uh

is valent I believe it's either s I think it's uh selium provides like a packaging for ebpf programs and >> ah I don't know to answer like what potential issues you have there >> oh it's a Mac yeah you need to run a Linux uh virtual machine on a Mac and then like run it uh Yeah. Yeah. It's a there is it's mostly for Linux this technology.

Um there is like a an ebpf for Windows. I don't believe it answers to the same spec. I'm less familiar with this project. And yeah, it's not supported for Mac as well. Uh yes, behind you there was another question. Yeah. >> I don't I don't think Okay. So I think it's a good thing because I want engineers to still be very involved with the code and I

do not believe that eBPF is going to solve it's not going to make them u unneeded or like make the the familiarity with the code unneeded. I believe in eBPF as generated data as a compass for engineers. I want EBPF. I want to have a layer for observability at least. I believe that I want the ability for engineers to think from scratch about their data points but

I also want to produce some hints regardless of how confident they are feeling with the with uh how their application actually is running. uh for example we are not not doing open telemetry in ground cover and like I really believe in instrumentation as a culture. I just think that instrumentation as a culture is much more efficient if it is being built on a bedrock of signals that

are decoupled from the instrumentation effort itself. If it makes sense Um, no. I I don't think it's about making them not understand what they're planning. I think that I can be only certain to a specific level about how I'm familiar with the project, especially when we're not talking about low-level projects where engineers have incredible intimacy with the code. And to be f like in terms of at

least what I seen in reality, software is not especially in the agent era, software practically speaking is not being developed by people that are looking for being intimate with their code as much as possible. It's just simply not the way modern software engineering unfortunately works. I wish people uh to be like more like Lionus Roaltz for example who is famous about not adopting CPP into the kernel

because he wanted to understand just by looking at the code how the oper what is going to happen behind the scenes. This is potentially the the the the most the the pinnacle of how software engineers need to be familiar with the code. Unfortunately, potentially n I would say 80% of the code written today is generated by people who are just very amazed that cloud code got it

the code oneshotted. They wrote a post to LinkedIn about it and all of a sudden people are being reliant on the code that a lot of software engineers very gave very little thought of. So I want to hint them and give them like this nudge that hey there are potentially some stuff that you want to dive deeper into. This is at least my my take on it.

Uh I'll be happy to take this conversation after if you if you wish. >> Yes. EVPF basically lets you depending on how many EBPF program you load and um to to get almost ability to hook software in a lot of meaningful places across both user application um calls and also kernel application calls regardless of how many applications are running. Um the notion of like if it's it's

I would say usefulness is a question of what you're looking for like it's depending like what what would you like EVPF to tell you about? Yeah, you can definitely with EVPF generate signals like we start with the profiling example like tell you what where CPU is being spent the most time in the application as well itself. Yeah. Um any other questions from that side the crowd? Okay.

Uh awesome. Thank you very much. We're just the time really appreciate everybody uh taking the time to attend and uh have a great rest of your day. one. That's good. All right. >> yes. >> Okay. Hello, my name is Shannon and today we have Amamir Jacobe from Sawmill's uh AI who will be presenting making OTTL go four time 40 times. Go ahead. Let's give a round of

applause to Amir. All right. So air I work for Smills. Before SAS, I was VP engineering at New Relic. I managed the uh incident management group and learned a lot about over there. And I'm going to talk to you about um a story that we have here that we that started in Sill's back like startup. We started about two years ago, year and a half ago. Um

and and and one of our first customer we migrated them from no it doesn't matter and we uh had a like had an issue like they they started telling us that their observability are missing logs they have they have they have crashes they have no crashes no spikes they're just like nothing but they're missing logs. Uh we're going to talk about the incident we're going to talk

about how we investigated that uh a deeper dive into OTTL and how it uh how it works, how we uh fixed that. We did a bunch of very very cool, my optimizations to OTTL that are still not released, still not part of uh OTTL, but maybe benchmarks and what we learned along the way. Thank you for joining me on this journey. Um so what is Sono doing?

We're doing something very similar to what we just discussed. were um coming in between the uh agents not AI agents though I know it's like very in right now but you know good old agents that collect logs metric maybe traces and uh we um connect to them and from them we uh we integrate to them and send the data uh back to data dog new reel open

search open like like any vendor you can think of whether it's open source for closed source. Uh we analyze the data um at your premise and send it back to its destination and you can make decision and reduce your cost uh by using the product and it it's quite significant. So we rely very heavily on the uh collector. So open telemetry for those who doesn't know is

an open source uh that started as an SDK and an API of like definition of something called OTLP which is the open telemetry uh protocol and it's define how it started with traces define how traces logs and metrics uh should should look like and there are a lot of SDK uh all around that can uh export uh those metrics there's those telemetry data in general out in

a very nice and standard way. It's actually the second most active CNCF project after Kubernetes. So, um if you're here, you probably know it. So, the collector itself is the Swiss Army knife of open telemetry. If you want to collect logs, you can use it just from a machine. If you want to analyze and drop logs, you can use it. And we're going to talk a lot

about OTTL. OTTL is sort of like the the language that allows you to do a lot of the stuff that makes the collector awesome. So we're going to talk about that a lot. The uh collector is comprised of three main parts. There's also other things but basically receivers which allow you to process the like receive and collect the data either from other sources or from a file

or wherever you want. And then processors you can filter, transform, batch, h do things like memory limit and anything else you want. It's very easy to extend and then you send it out. So you know input you do something output very standard and processors are very interesting because they got to process every every log record of telemetry data. So you know a lot of processors mean a

lot of processing. So our constraint as a company we're basically our main operation is we're running first at the at the customer premise right so we're running let's say in your Kubernetes cluster and that mean and and we're there to help you manage your cost we're there to help our customers figure out what's going on in their telemetry data and and help them reduce it. So we're

not there to increase their compute or memory or any other other things. So that that constraint made it very very critical early on in our journey as a company. Uh it makes it very important to us to really understand how the collector operates behind you know behind the scene. So what happened? What were what was our incident? So we installed our collector at a c at a

customer. Everything looked good. You know, we we at at least at least in our eyes, we process about about 40 terabyte a day for for a cluster that wasn't necessarily a single collector. It was a few uh but one cluster 40 terab terabytes a day. Um we migrated most of their transformation from that vendor and those was were over 300 very complex regaxes and Grock and and

and you know regular transformations and quite honestly I don't think the people that designed the collector really thought about that like so many transformation and filters in the collector so it so it was pretty intense we we even needed to uh figure out how to load all the configuration file because it was so so large that you couldn't put it in a config map. So we had

to like write something that would load it from S3. Anyway, every log record needs to get through all of these OTTL expression and it looks fine for us. Like we're cool. It's working. We even did did some performance test before and it all all looks good, right? But it wasn't really good. We're starting getting, you know, calls about like, look, there are logs missing. We can't see

everything. We're we're not understanding where it's going. And we're telling them, no, that the collector is fine. It's something else. And it took us a while to to understand that like something is really really wrong with the collector and we don't have any visibility into why. Uh we we we want to go over the next slide, but basically there's a lot of metrics. the collector is great

at the metrics you get from the collector are from the collector. So if you start processing a lot of data in the middle and let's say you have four four and you know you have eight go routines that are receiving data in in a loop and trying to process them and all of all all eight of them are in the middle of processing data and there's a

lot of traffic coming in at the same time. You're going to find yourself hanging. the the processor will will start the the process I mean will start hanging and you wouldn't really be able to uh process the data and the collector wouldn't know because it basically don't accept it doesn't know that it has incoming traffic that's supposed to to come to it right let's see what else

I'm missing here yeah so basically rejected connection before the the receiver could even see them That's that's the bottom line here. So, what did we do to to actually get visibility? So, sidecar is is a really uh nice pattern that that I I personally really like. H it allows you to take, you know, in a a pod and you have you have your main container and you

can attach as many containers as you want. So what we've done, we actually already had another container that sent telemetry data about the collector because we don't want to contaminate the working of the collector. So it's kind of funny. We have we're a company that is doing telemetry like observability for observability in a sense and then we had another level which is doing observability on the observability

of the observability you know. So yeah and and what we added on top of that is an haroxy sidecar that would sit in front of the collector and would analyze all the incoming connections to that collector and would let us know whether incoming connection are actually accepted or not. So that actually you know open our eyes because suddenly we we were shocked to see that the collector

is actually rejecting most of the connections and we had no idea that that's happened. There's no there's like zero visibility into the incoming connection into the collector. So that was amazing. Another thing that I really like about HR proxy is that you can actually configure quite simply uh fallbacks. So the first thing we did is say okay if the if the collector is not able to process

any data just send it to the vendor. So we started moving to um a methodology of of like identity functions of like saying always receive the data and send it in the same format. So you can basically do a fallback or a circuit breaker if you want and that was really awesome. So we had we had the rejected connection. We had a lot of latency. I think

the 80 percentile was about 10 seconds which you know is is crazy. Really we had a few really bad days back in the time. See pretty much what I just said. So when we started looking into what a collector is doing, we assume and and and and we're right that there were too many OTTL expression basically which again you know title of this co of this So

we had too many OTTL expression we weren't able to uh accept connection because there were no go routines available to accept the uh the request. RPC rejected at the socket like before there was no really available receiver. The collector had no idea. We had no visibility. H so the H proxy allowed us to to to first gain visibility into Then we started going deeper. So you know

we uh basically um took a lot of the logs from from that pro from from that customer and started doing some low testing and puffing profiling the uh the collector and quite fast we saw that the OTTL EVA was the um culprit and more specifically when you look here you'll see that the allocation and um value comparison and resolving path were were part of the you know

high were the one that took the most time basically. So allocations and let's talk about that. So we'll we'll do a deeper dive into how TTL works, but basically every uh OTTL evaluation means going through an a and abst abstract uh syntax tree and the abstract syntax tree is generic which makes sense right. So it it get types of any and then it it needs to box

them. It needs to like takes a string and box it and boxing means a location. Allocation is not good for us. erh resolving path like traversing the a is also quite it's actually very efficient like I tried different a uh engines are not OTTL and then and and OTTL was really really efficient so I I don't want to like say that it's not it's it's really built

really well but still you need to traverse a tree and and if you can avoid that that's better lookups over um generics also not that efficient because it means boxing again any boxing when you go over 300 filters which is what we had and you have a location you get to like 600 to 1200 allocation per and you know when you have a lot of a lot

of record per second just too much and and I didn't even start talking about what it means for uh the garbage collector that needs to do something with all those shortlive object So we had a cycle where we had more location, more garbage collection which slowed down the throughput in a way that's very hard to see. Uh we got more back pressure, more retries. There's abs syntax

tree overhead which which really impacted CPU over some some you know open source that's very hard to you know OTTL is very deep part of open telemetry collector. So we didn't really know what to do at the beginning like the first thing we did is start checking whether some of the you know just okay so let's remove some of the filters. So we tried doing that we

remove some of the filters we removed some of the try and simplify some of the regress but eventually we ended up with about 200 filters and we still had the same problem. So we h dug a little bit deeper into OTTL. So what is OTTL? I think I already gave you, you know, a hint, but OTTL is the DSL that's built into the collector. It allow us

to do things like filter, transform, and route. It's declarative. It's it's very nice. You can attach it to the YAML config file or any any any like YAML representation. It allow you to uh not dynamically but without compiling the um without any compilation to attach and manipulate those log traces and metrics and go into all of their fields dynamically had a lot of interesting built-in functions. You

can check if things are you know their types. So basically it's um it's it's a language and you don't need any custom code but um you need to do interpretation and that was the main issue that we had right it's still when when you load OTTL it goes through a phase of evaluation so it doesn't stay as as OTTL it's been translated like I said before into

an A but still not very uh still not not optimized as it can Okay, so this is how a log record looks like when you translate that into a JSON. So you got the body, you got some, you know, fields there, you got the attribute, the level, the status code, you know, I think pretty standard. And here's an example of like filters. So checks that you got

to check if like you want to manipulate the body first, you got to check if it's a map, right? Right? And then if you want to check something inside, you can do things like contains and go into attributes. If you don't do the check at first, you're going to get an error. The error can be ignored or not. But it's always nice to check and you can

you can do you can complicate it. You know, add between you can do nested fields, you can call regaxes or gro very very nice. H all of this is translated into a a tree. I'll show you an example pretty soon. Uh and you got to evaluate a lot of it. H like we had about 3 million evaluation a second. So when you when you start and look

into uh the profiling we saw that simple kind of comparison or basically had one allocations and took 57 ncond sounds like seem very very fast but it's not can be much faster three allocations here and very complicated had like four allocation and and you know like a lot a lot it takes a lot of time so the the reason is partially uh the AD and and quite

frankly things we didn't get to which is the reax. The reax is also terrible. The go reax is so no it's really not efficient. It's going to be a call out about it pretty soon. H how how we translate OTTL into an actual thing we worked on the collector works on. So the there's an expression it's been parsed translated into a tree h and then every time

a lo a new log message arrives you got to walk and traverse the tree. So you're going to visit you're going to go and turn the expression attributes level equal error and then you got to go and check the um and traverse it. So you start with the end and then you do the comparison you know and then I'm sure you you you under you're here and

you understand what I'm talking about but basically it's it's a lot of operation all this traversing even though it's it's quite efficient those those expression are static but and the data changes but because they're static you know quite honestly we can just compile them that was my first when we got into this project my my first impression was oh cool so as it kind of remind me

of my classes about you know languages and that's very nice you can take an a and basically turn it into code right so what go go goes nice it's actually has this library called gen so let's just translate it into into code and that's what we've done we actually wrote a a parser that takes a and translate it into code using a library called gen that took

us took first few weeks h we got into um straight go code so there was no a walking the compiler is very gracious you know it optimize you you get very optimized code h I don't think you can get any better unless you go into the assembly or into the you know interpreted like language that h is being compiled by go and the only small very very

not small issue that that specifically static language like go has is that it kind of compile it into an so plugin and then you got to do a plug-in open and load it and there and and when you start to dig into the the plug-in framework that go has it's very very um limited so what do I mean by limited when you compile your code you got

to compile it exactly with the same packages and versions that you h used initially in in your collector. So you compile a collector and then you got to know exactly what version you use when you compile it. That's the first. And second, when you load a plugin into Go, it kind of loads into memory and that's it. You can't unload it. You're stuck. So if you want

to load and dynamically change OTTL, it's you're going to have a bad day eventually. Like you're gonna keep loading plugins until you run out of memory. you're going to lose data to a So, you know, we kind of got stuck. So, we've done that. We solved the problem. I'm going to show you some some numbers pretty soon, but it wasn't good enough for real production. So, it

worked for one customer. That's nice, but it's not scalable. And that was we fought the end of the journey. I didn't release it into open source because I didn't feel comfortable uh with that solution. Seemed to to be too limiting. and and let me know if you actually want that release. That's that's fine. I can do But but for about six months, we got stuck. We said,

"Okay, we're not sure if we're going to get any other customers that have this kind of problems." But, you know, we we did. We were running into additional customers and we need to do this oneoff solution for them. compiling, loading a loading a plug-in, getting you know, excellent results in term of in term of performance. But but we uh we got stuck. Uh what do we have

here? We have a little bit about how the evaluation look like, how we parse the condition yeah, I think a lot of things I'm talking I already talked about. So I'm going to go over. So a function that being um generated you kind of skip everything on the left. So over the left side you see how we walk over the a and we go into a boolean

expression. we need to walk over every leaf and every part of the um syntax tree in comparison in code you basically go and get an attribute you know you do some validation and and and and com and comparison but it's very very straightforward you know it generated code it doesn't look good but but it's and and it turned from from this thing into you know 57 ncond

with two allocation into eight nancond and zero allocation and that just for a simple expression when you go into more nested trees it becomes quite a nightmare in term of a location and speed so we're talking about about 7x just for this simple expression so I talked about the trade-off before but let's go over them again so we got like between three to 12x speed ups near

zero location there's still small location that we didn't work on optimizing but I I I quite quite honestly believe that you can get into zero zero locations. So it's really nice but again you goth another thing that you got to be dependent on sego which is not that nice if you want to be uh without any you know you're in docker you don't really want to be

uh you want to be headless you don't want to have any dependencies so with uh with plugins you have to use uh sego so you got to have some C library around and you got to have some Linux there and you got to compile that was another issue by the way compilation takes a lot of time, you know. So, we we what we wanted to do initially

was to load the collector has some code there in the do in the Docker container has like a an initialization step where it compiles the the OTTL expression and then load it into the collector. But that took a few minutes. So, and also you're very limited, you know, running in a container in Kubernetes in a customer's side. So it's like what you're going to they're going to

pay every time to load a container that like doesn't make sense. So you have to build an infrastructure where you compile it somewhere else and you need to know exactly what collector version they have and then you need to deliver it. So it's it's a nightmare again. So seems like a very good solution but not really. So what did we do? H we came up with a

very nice idea, very neat idea uh called um closure combinator. so what what do we do now? So we can walk the a and and and by the way, I'm not going to take credit for this idea. This is all thanks to uh coding agents. I think Codex, I don't remember which one that I used. I don't remember when around October maybe. So before the the new

models are even better. So maybe there's better ways to do it. Who knows? Just you know prom way. Um so before that we've done we walked an ST node resolve the path we need to box values turn them into interfaces which means and do some type assertion compare and return true false uh based on the comparison that we're doing if we're doing if we're talking about filtering

it can also do transformation of course and now what we do is we uh basically parse parse expressions and the the idea with closure is you take every leaf and every part and turn it into a function that you can compose. So you can think about it like functional programming like you have a lot of small utility functions and at load time you compose them together and

then instead of traversing a tree you basically already have a function that calls a function that calls a function etc etc etc uh until you actually get the result and and and you know surprising enough it turned out to be uh almost as good as the compiled version and in some cases which which honestly surprised me. I didn't expect it to work so well and to be

so elegant. So again we moved from five steps into a single step but without any heavy lifting right the only thing we needed is basically to build a lot of small utility functions which nowadays when coding is not as hard as used to be is is is not such a pain in the ass. It's like we we were able to do it. And so a deeper dive

into how it actually looks like. So you have your code on the like like an example on the left which is turned into so you got to get a key, right? So you you're building a getter and then you're building a small function that can do um comparison B for string and you need to have the same function for every type that you have. And of course,

you're also going to have a generic one that does boxing for the case where you need to compare objects that you're not familiar with, but you can always optimize it. You know, if you if you see that you're you're dealing with a type that you never met before, h you can support it as well. So, and eventually you compose everything together. So in this case you take

is error and compare it with uh do an end over the over 400 and the over 400 is make int 64 greater than. So it compares you know get code and with four uh with 400 and and and you know you can go go over all the the the small composition that we have the the nice composition and you get into the same relation to an a

like the same representation that's what I looked for. So all the the the composition of the a is done at the the the generation time. I guess you it's not really compile but very similar and at runtime it's just function. So there's no a you don't need to assert types. You don't need to do any allocation. So boxing again you got to do uh you you you

got to deal with that. The way we've we've dealt with boxing is we turn that any translation into many many small functions that allow you to h do uh basically get you extract the value uh based on the and you have all of that when you generate the code. So that wasn't that big of a problem and then you your composition becomes very type specific. So this

is an example of comparison for string. So when we're doing the compilation or the generation, we're basically trying out with the thing that we know are are usually uh uh the one that are going to occur. So so string in the case of uh the telemetry data that uh we usually see is the type that uh that we usually that that that come first. So we try

to do a comparison for that and then we move uh we move to the next one and that's only for the generation right in the actual runtime it always know what the type's going to be okay and and then the uh also uh path resolution so you know attributes and can also be nested because you can have map that points to with with a key that points

to a value that's a map and so on and so forth. So all of that can also you using some some uh code that can go through that because again in OTTL expression you kind of represent where you want to go. So you can build the retrieval function using composition. So here's the before and here's the after. We got from about 200 500 255 ncond for a

location. Just an example over the things that we you know ran and for a location into a zero location and when you look at the actual heat map you see um that you basically get into the actual retrieval. So you know the thing that you got to do because you eventually end up you you process telem 3 data you got to do go and retrieve and do

comparison and do those checks you got to do that in go um so we reduce about 27% of CPU overhead there's no more a uh traversal there's no allocation um when we're talking about more complicated expression and all the composition uh we got into about four 40x uh improvements without any garbage collection pressure. When you talk about the breakdown of the um expression that we have simple,

you can think about simple as just a simple comparison without any ending between the the more complex is a lot of expression that has a logical operation between them. very complex also contain um regaxes and gro and things like that and then of course you have nested path that that basically means you got to retrieve you got to check the type and and things like that so

that's even more uh um complicated and here the interesting thing is the closure one was actually I I ran this benchmark and updated it yesterday and it really surprised me that co that closure was more uh performant than the codegen I didn't dug into why so I can't tell the only interesting the only thing that remain for us is basically regexes. So what about regex? So when

you when you're talking about h the go reg is not that efficient. It's it's really nice, but Intel released something called hypers scan a while ago that uses the specific cmd acceleration uh instructions and and it's really really fast. And then uh in on on ARM you have vector scan. I didn't try it yet, but I really believe that if we uh if we really want to

take the collector over the edge when it comes to performance, we should uh invest in um supporting these uh two you know form of of of algorithms or libraries and and and potentially we can get like 50 hex. So if your operation is very dependent on regaxes that's something to consider. I don't really have a good solution for Grock yet. Grock is also a nightmare when it

comes to performance. So I guess my next presentation is going to be about Grock or something. So what we eventually end up shipping is a closure combinator. Uh we're in the process of also releasing it in open source. I didn't start the conversation in the uh open territory committee and the collector meeting that they have every week about about any of this. So there's still road to

to go but on our personal like sills contribution is going to be released in the next few weeks. H so if you want you can use it and again the best trade-off between the compiled one is the speed up but in my opinion it's not just that it's the uh you know you don't really need any compilation you don't really need to to to know exactly what

is the library so all the the surrounding all the mess you need to to manage is gone. you you you basically have very speedy uh load up initialization that's not very different than the evaluation of OTTL. What we learned you know always start with measuring the HA proxy we should have thought about that in the beginning but eventually we added that we we now ship it in

every um any distribution collector. I really recommend putting something before the collector so you get some better visibility in in term of compute and memory. It's like almost zero. It's really worth the the hassle of having another sidecar and configuring an HA proxy. So that's you know a good tip. I was surprised by the allocation and boxing and as I I I actually assumed that you know

the OTTL and A is like the best we can get to but I was surprised that that it's not and that's actually not that efficient. uh compilation is always the best but when it comes to go uh which is a very static language when it comes to you know they don't really like loading things loading code uh it it didn't really turn up and and you know

safety is over always better than speed in most cases even here. >> Yeah. Yeah. So, so we saw that the latency wasn't good, but what I mentioned was the P80 P90 percentile. So, most of the time we thought everything was fine and we assumed wrongly that the latency came from sending data, you know, outbound. we we we we wrongly assume that there is high latency because of

the exporter not because of the uh valuation because it like doesn't make sense usually in a something like the collector you would assume that that IO is going to be the problem and not and not you know CPU like it doesn't make sense right like you think about the collector like okay what's going to be the bottleneck here it must be IO so we're like okay so

there's latency because those guys you know downstream and not don't know what they're doing or something like that and and and and that was the wrong assumption on our end. >> Yeah, exactly. When you use something like HA proxy, you basically say I'm going to use an external process that is not dependent on any uh CPU or memory to to really understand what's going on. And and

in our case, we we were very curious about the IO, you know, networking. So it gave us visibility and also allow us to I think the the best thing for our customer was the fallback mechanism. So while we were figuring this out, we could actually tell them, look, yeah, we have an issue that we're not really processing and reducing your cost, but at least we're not losing

any data. So not great, but you know, better than than where we started. Um so so the C we we figured out that it's the CPU when we started doing the the P profiling like we started profiling it uh on on our machines. So, so we uh yeah, so what what we've done to figure out whether you're asking whe how how we figured out that CPU was

the issue, right? So what we've done was we uh basically took a lot of the telemetry data from a with with a permission of our customer and then we built a sort of a benchmark testing library and we checked what was the issue. to actually send data downstream. And so, oh, and like P-p profofiling in in you see it in a second. If you um do some

some P profiling, some profiling to the processor, you basically very fast. You see those flame graphs that that show you exactly where the issue is. So we we done that and and immediately we saw that the the culprit was actually the uh OTTL evaluation and not the um IO of sending something somewhere. Um the the the interesting another interesting bit is we didn't really see high CPU

like when we looked at the collector we didn't really see high CPU and and you know what to this day I don't really understand why there's no we like it got stuck but the CPU was never an indicator that it got stuck on evaluation. I I don't know why. Yeah, exactly. You could and and and and it's very very hard to get to zero fallback like to

zero in our case like getting into zero fallback became our target and it's very hard because even even if you're set like we can have an an entire talk about how to do autoscaling to the collector which is also very hard like how do you do autoscaling? Do you use a sending queue for example? Yeah, like sending Q is another big lesson in general. I'm saying that

we had when it comes to production like taking the collector into production uh great because you you you want to have the ability of of of like have some queue that would allow you to batch not batch but but but like store data before you send it somewhere in case there's some issue, right? So so that's another very important thing. But yeah, you you basically wouldn't know

if if you're losing data. You you you might be losing data now and you don't know it. Yeah. Any other questions? Every time you do what? Yeah. So, so you're asking Let's see if I understand question. Every time we change filters, we need to do this closer com combinator again. Um yeah, basically yeah every time and it's the same thing with OTTL in general like when you

define um some OTTL expression right and you put it in a transform processor or filter processor or anything else and you load the collector usually people don't really load it dynamically they turn turn off the collector and turn it back on and what happens is there's an initialization phase where you take it only happened once when the processor loads where you take OTTL turn it into an

A and then the A is there and when traffic starts coming you basically uh walk the A over every log message so a new log message arrived and then you you like the collector traverse the tree um the the the what we're doing is something very similar we're taking the um OTTL and instead of translating it into an A we do another step so we still do

the same a thing because what like it's very helpful and then we take the a and we translate it into a function that we compose on top of another that's like closure combinator that's the name of the this pattern it's we didn't invent so yeah you got to do it every time for uh fortl when thetl changes yeah Yeah, but you know what? Like it's you're in

when you're doing uh plugging and you're loading code, you you're you're the the the trick that the plugging authors did is you load it in memory. you load it in memory into the process as if it's part of the original binary. So there's no Yeah, exactly. Yeah. So, so that's why it's like native code and that's awesome, but it also have a lot of limitation and nobody

really thinking through right now about what's going to, you know, I don't know. I but I don't I'm not aware of any active um work to make it better. Yeah. Yeah. That's that's why I'm here talking about it. Yeah. Because I agree. if I I I didn't see anything like that in Go. Yeah, you're welcome. Any other All right, guys. Thank you so much for coming. Yeah,

I'm a mirror if you