Schema Inference and Automation: A New Era for Telemetry Mana... Nicolas Takashi & Arthur Silva Sens

This talk covers schema inference automation, focusing on a tool called OpenTelemetry Weaver, which is designed to streamline the process of defining and managing telemetry schemas. The speakers, Arthur Silva and Nico Takash, discuss how the tool aids in generating metrics from declarative schema definitions, enabling automatic contract testing and various forms of automation. They emphasize the need for observability by design, where telemetry is treated as an API, helping to eliminate common pitfalls in observability practices. The session includes a live demonstration of how to use Weaver for generating metrics and ensuring compatibility with telemetry data, thereby promoting best practices within observability strategies.

Hello everybody, thanks for being here. Our talk is called schema inference automation. Although we do have inference in the title is not AI related. I'm sorry if that's all your expectation. That was not intentional. Uh But anyway I'm Arthur Silva Sense. I'm a software engineer at Grafana Labs. I'm also a Prometheus maintainer. I lead the Prometheus special interest group in Open Telemetry community together with Dave David

Ashpole. And hello folks. My name is Nico Takash. I'm observability stack leader at Coralogix and I'm Perseus maintainer and I've been working in some observability related projects at CNCF like Thanos, Prometheus, and Open Telemetry as well. Thanks for being Yeah, but we are not talking about Perseus or Prometheus today. We're talking about a tool called Open Telemetry Weaver. Uh who here has heard about Weaver before? Oh,

that's actually More than 30 people. That's cool. And have you besides hearing about it, have you tried? Okay, that's a little less. >> it's I mean I I mean I tried myself. Uh I think uh we [snorts] watched a talk uh in Observability Day in 2024 if I'm not mistaken, which is called Observability by Design. Sign. Um this is the QR code for the recording if you

would like to understand more what is Weaver. And the concept of Observability by >> Yeah, yeah, and the concept of Observability by Design. There's also another talk that happened yesterday during the Observability Day uh by Benedict and Sebastian. Say hello. Yeah. All right, yeah. Sorry. [laughter] yeah, this is the link also for the their talk. They showed how um Observability They talked more about a schema, not

necessarily about Weaver, but how Shopify is using uh schemas to to automate a lot of things in the Observability space. And uh that brings me one question. I Who here likewise Shopify has a team of 20 20 engineers? Is that correct? Only working on Observability. >> Only for Observa- Observability. One, two. Three. With Shopify, yeah. Okay. Yeah, I I think like in reality the Observability by Design

is great, but uh we are going to talk about how to go into this place when you are not You don't have like a superstar team. That's how That's how it bad, but that's not what I meant. Uh well, so okay. Uh what is Observability by Design? So, it's when you treat uh your Telemetry like an API. Oh, you have a document that declares how things look

like and you could from this declarative uh file, you can do all sorts of things like contract testing, you can do code generation, you can do You can do a lot of stuff and we will not talk about all the things, but like Okay, I I think I messed up. I'm just going to move it on. Uh so, is something that Weaver is created to. Weaver is

a tool managed by the Open Telemetry organization where it receives schemas as input and it outputs all sorts of things. Besides uh generating stuff, it can also do the what we said about contract testing and other kinds of automation. This is one example of how a schema looks like. We have a metric called HTTP request duration seconds. Uh it has uh uh brief, which is a description

of this this metric. We have units, examples, and from this single file, we can do all sorts of things. Uh we can, for example, rename our metric and we maintain backwards compatibility in the back end. We can do policy uh enforcement. For Prometheus conventions requires that the unit is units are uh suffix in the in the metric name. And this uh and Weaver can enforce that in

your organization. If you have other kinds of very custom uh uh policies that you need for your org, uh Weaver is a design for this. You can do instrumentation, you can do dashboards, you can do alerts, like literally anything that you imagine. I mean, inside of computer, of course. Uh you can [laughter] do uh So, basically, this is heaven, right? Like we once we have the schemas,

we can But like it's not easy to get there. Like Observability by Design is the dream, but in reality, it's not how things really work. We get into them in our org by we have an incident and then we desperately create a new metric because we'll need in the future or we we see somebody else doing uh Observability and we believe that the way they do they

do also works for us and we just copy. Yeah, we copy paste. Uh and also by accident sometimes, like we we import a new dependency, we create a new we we use a new infrastructure piece in our uh architecture and all of a sudden we have thousands more metrics, more spans, more logs that we didn't expect uh to get in our databases. Uh so, as I said,

I'm a Prometheus maintainer and I wanted to try how to do schema driven Observability inside Prometheus. What I mean by that is like Prometheus has a whole set of metrics, literally hundreds of metrics about their own internal stuff. Like we have metrics about database, the scripting manager. We have I don't know. We have >> Observability >> Yeah, we have hundreds of metrics. And I I don't want

to write this whole YAML file by hand. Like this is too much. I >> It's error prone, right? Yeah, it's error prone. It takes a long time and I I was talking with the Weaver maintainers and maybe we could create something that uh makes this easier. And this is how we implemented a new command called Weaver registering for. And my colleague Nicholas will tell you all about

it. Cool. Thank you very much, Arthur. So, folks, it's demo time. And so, it's a real demo. It's live demo. So, please pray with us. Um so, but before going to the demo, I want to like Let's get back into the the way the time that we had to develop a APIs, right? Before the Open API specification, the Open API movement, all those things. I was there.

Probably most of you were there as well. Like we joined with our colleagues, front end engineers. We got to a room. We would agree on a specification, right? Fields, names, and And we we move away. Each one implementing your stuff. Like back end engineers defining data models, exposing the API, front end engineers developing the UI expecting that the API is returning that fields in a certain way,

right? And when we try to integrate weeks later everything would break because back end engineers, they decide to change the the data structures, forgot to update the front ends. The front ends now require something different from the back end and start forgot to talk to the uh back end And this is one of not intentional, right? Like just a gap on the process. Like um we have

no fashion ways of generations of client SDKs, no automated documentations, nothing. And like after the Open API specification, many different companies start doing this, like generating the Open API specification. Uh generating client SDKs, documentation, all those things, right? Now there is coherence, right? Between back end Yes, it's easy, you know, you like it's not even perfect because like not perfect, but it's way better than it was

in the past. And if we think a little bit, Telemetry is not very different than that, right? Because we write client instrumentation on our application as we need as Arthur was just mentioned. Um we push to production. We look into the Telemetry. Weeks later, we thought, "Okay, this Telemetry is not working as I thought it would be working. I want to change it a little bit a

little bit." And you just change it. Nothing is validating that. And then you see a dashboard not showing any information, an alerting that you are expecting to be firing not firing, a system that should be scaling not scaling just because a metric has changed or span or whatever. And this is what we are trying to address today. This is what we're we're going to show you how

we can move from zero to one and go into this direction of Observability by Design or Telemetry first, you name it. But in in real systems, okay? Or kind of real systems. um let's go to the to the code editor. What do we have here? Is a very tiny Go application, the application doesn't really matter. It's it's meaningless. Uh what really matters here is this application is

exposing metrics in a way that we are doing that today, right? Let me just close all those things. Spoiler. So, this application is exposing metrics like this. I don't know, is good enough for everyone? You need to zoom in? I think so. Cool. So, it's a handwritten client using Prometheus client go lang. We have a mix of business and infrastructure metrics like per source total is just

a business metric here. HTTP request total is an example of infrastructure related metrics and probably this is what we're doing in production today, right? In real systems we are writing the metrics or the response. Like we're using metrics here, but this is true for any other kind of telemetry. Um and that thing, so we need to move from zero to one and generate the schema that I

had to show you in the example in the beginning. From but doing that manually, it's boring. We could probably ask if AI to do that nowadays, right? But it's still far from optimal. Uh probably I cannot predict all the different combinations of telemetry. Like if you think as an HTTP server, based on the user user input, you might have like 500, 200, um 529, whatever. You You

have different uh status codes, So, the idea is to achieve that from live telemetry. To do that, let me get back to slides for a second. We are using the schema inference. That's the command that I had to just mention. Like Weaver is receiving uh OTLP payload, inspecting the payload, and generating the hash history .yaml from the live telemetry. And we're going to simulate that right To

achieve that, we need to translate the metrics that's being exported by the packet sorter, which is Prometheus format. So, we need to translate this into OTLP and push to Weaver. We're using OTL collector to help us. We are basically using scrape uh Prometheus receiver to scrape the packet sorter, translate it to OTLP, and push. And we have this load generator. What does it means? Um as I

mentioned to you right now, based on the user behavior, you generate different combinations of telemetries, The load generator here is simulating this user behavior. So, giving different inputs to this packet sorter uh application in a way that can generate many different uh combinations of telemetry like 500s, 200s, um I don't know, 420 429 and so on. So, let's see this in in practice. So, we run make

schema and run infer schema. Like this make is just one abstraction because we need to spin up a collector server, a Weaver server, and the application load the test. So, we just packed everything in a single command, So, we just wait a few seconds. Demo gods, please. Mhm. Be with us. Okay, nice. It's It's better than nothing. So, let's see what's happening right now. We just generated

this YAML file. if we look here, HTTP request total is not here, so it's demo, right? So, it's duration is here. Oh, it's a requests total. It's not here because I'm not typing properly. So, HTTP requests total is here. Parse souls total is another metric. It's also here. This means [snorts] that all the telemetries this application is generating is actually now on the on the bring us

from zero to one. Like now we are in the place that we can start leveraging the other commands that Weaver is is providing us like the generate that we can generate client SDKs from that. We can generate documentations for that, right? Observability by design style arts now, once we have the schema. So, [snorts] what is the new one, right? Whenever we want to change a metric right

now, we want to change on the YAML schema and then your process generates the client SDK, updates the documentation, eventually updates your dashboards and your alerts and all those things. But before we move into this automated process, we need to have something that give us confidence, Because like we already can generate the client SDK for Prometheus, for example, for this instrumentation. But I don't have anything in

making sure that the contract stays the same. So, the code generation it's producing the right But we are lucky enough because the Weaver project has another command naming live check. And the name is hardly engineering. This command is just doing what it's it's saying that it's doing. It's doing live check. So, you have an OTLP payload and a hash history and Weaver is asserting that the live

telemetry and the and the and the hash history, they are compatible. The telemetry defined in both are compliant, okay? If If we compare Weaver registry infer and Weaver uh registry live check, like both are receiving OTLP messages, infer will create the YAML for you and uh live check will make sure that the message that we was received complies with the YAML that we had before. So, if

we run live check right after infer, should succeed, right? >> Yeah, 100% and this is what we're going to see right now. So, let's get back to Um we have make live check, which is basically doing the same thing, starting up the application, the collector, the load generator, and everything. And since we just created the this hash history, we are not expecting any breaking change here because

it's like fully compatible, So, as we expected no violations, the the the contract it's 100% compatible. But let's do another thing. Let's go to our go code. We are changing from seconds to milliseconds. Right? So, we're going to run live check again. What do you expect to hear after? I hope we see a failure, otherwise I'll be very embarrassing. Yeah, yeah. Let's go. Cool. So, a few

more seconds and now we're changing. So, we changed the the source code, but we did not we did not the hash history, right? And we can see now that we have a violation. Uh that's cool. Like HTTP milliseconds metric did not exist on the hash history, which is very true. And folks, this is so cool. Um let me get back to this. Like we have this command

in in CI as well. Like if you have GitHub actions, you can just run the exactly the same thing and you can just block your PRs. And what I want to do right now, folks, is stop here with you and reflect for a moment because let's imagine that you have like, I don't know, hundreds of code of line codes changing, okay? And you have a single line

between hundreds that's changing the metric name, but by accident accidentally it was not intentionally, right? You give a prompt to cursor or cloud code and they change it, but you didn't saw it. >> [laughter] >> Uh code review didn't get that, you know? Um and you don't have this process. Like you just merge it and you you don't have like alerts firing, you don't have dashboards working,

or even worse, you don't have systems like KEDA not scaling your applications and causing a a potential production outage, right? So, this very dangerous. And this kind of process is bring us some confidence to to to the observability um space, right? now we have all this confidence, >> How would be the the right way to do it? Like now we we we're good enough, right? Like the

right way to do that is like now we are still using manual client instrumentations, but we already have confidence. We see that we have something asserting what we are our telemetry and our contract. So, we are ready to move into automated world where we can generate the client instrumentations. Let me get back to slides again. How we are in time? We're good. Weaver has commands to allow

you generate artifacts. What I mean by artifacts here it's anything, documentations, typed client SDKs, whatever kind of artifact that you or your organization needs to create from And this is very flexible because Weaver is working based on ginger templates. Please don't don't be scary, but it's just like some old-fashioned ginger templates and this makes you so flexible because you can create a template for basically anything, right?

For example, in in this template we are using Prometheus SDK, but we we could also create this same telemetry in another SDK. I think we have one, right? >> Yeah, we have another template for OTL, So, we basically getting on the hash history, we are generating the client SDKs both for Open Telemetry and and Prometheus. Like just to be consistent, we have another command named make generate.

It's basically generate the clients. Uh and we have it over here on the generated folder. Like there's no hard structure, you choose where you want to place your generated clients, okay? So, if we look this, this is the code that was just generated by the generate command. Uh if we run live check, uh we have instrumentation mode. Oh, I I think we need to explain this. So,

to make this demo easier for us and I guess a little bit harder for you. We build an interface where we pass the environment variable and it based on the environment variable it chooses different instrumentation code. instrumentation code. I think we need to code. I think we need to explain this. different instrumentation code. I think [snorts] little bit harder for you. We explain this. HTTP requests duration

duration seconds. Let's change it to milliseconds again. So, I'm changing now the the hash string. I'm going to run make generate. This is going to regenerate all the clients, And since the clients we they are type it we have like now compilation errors. Complete compilation errors. So, we can come here and say oh, this change it. So, it's now milliseconds, And now milliseconds. If we change the

spec without regenerating the clients, we're going to have breaking change, right? And this client regeneration could be some CI process that you are generating publishing I don't know in some package manager that your consumers are consuming, you Cool. Anything else you want to show? I guess without a demo we're kind of So, what is the process in general folks to adopt open observability by design in existing

systems like you can just do the schema inference. You can have something mimicking user behavior in your application or you could just connect to Weaver in one open telemetry collector in staging or production where you have traffic, Once you have the schema, you are ready to start generating artifacts from these, but before doing that you want to be sure that you are not breaking change. So, you

introduce the notion of the contract testing or the live checking as the usual the original common is saying. And now you're ready to once you have contract testing code generation, any kind of evolution your telemetry could start on your hash registry. You can easily propose oh, I'm planning change this metric. You share with your stakeholders or your teams. People can like think that first before you generate,

Cool. Back to you, Arthur. Just to wrap this up. I'm a Prometheus maintainer, right? So, I I guess I could do some advertising on the Prometheus side. Yeah. Prometheus is [snorts] the Prometheus team is experimenting with making Prometheus aware of telemetry schemas. So, if you have like you are you're already an adopter of observability by design and you're using schemas to transform telemetry from like HTTP request

seconds to milliseconds, the schema will document this transformation and Prometheus will be aware of how your telemetry looks before and after. So, if you query for the new metric, Prometheus will also grab the previous version and merge this all during query time and the other way around as well. If you're querying for the oldest version and you have new version on the database, the query should work

out of the box. There's a talk about that last last year. Yeah, I think Bartek gave a talk about this in KubeCon Paris 2024. I this I feel like this infer command is something that the industry industry needs to [snorts] adopt observability by design. This is what I needed when I tried to move Prometheus code base and I I implemented this in the old fashioned way. I

did not use AI. I actually read the Rust book if you can believe it and I Yeah, so thanks a lot Josh, Jeremy, Laurent, Lyudmila for the very patient reviews on my PRs. I am I literally my first time writing Rust. So, yeah. Thank you. Thank you, folks. And thank you all. Um, microphones are right there if you have any questions. Um, mic's not working, I think.

Uh, thank you very much for the talk and thank you for the work that makes it easier to actually make observability by design possible. And my question refers to the fact that a lot of time you want to standardize certain metric across a number of different repositories. And if you just well, design by end inference, you end up with a number of different registry files overlapping. Is

there a plan to introduce some kind of inclusions or imports that allow for the standardization? That is a very great question. We actually this is on the roadmap. So, infer today it doesn't have any flags like in the CLI. We like we want to So, you pass the as input as well and it will compare what you have with what is coming and whatever you have will

like will be removed from the what is being inferred. So, you can compare like if you do infer and you already have an output input registry, it will and it matches, you see no difference, right? Nothing is in the output, but if it's different then you have the the difference between the registry and you can slowly work towards moving away from one to to the other. But

I think it also helps very much you adopt standard telemetry because you the idea is getting the the live schema once, right? And once you have the live evolve your telemetry for from there. And the fact that you can auto generate instrumentation client, for example, means that you can define the standard metrics for the organization and generate clients from many different languages that you might have and

clients just import. Does that answer? Yeah, I think it does answer to an extent, but then you generate these client libraries for different services and then those different services start adding new metrics, so their registries expand. And then you make a change to your base metric like HTTP request duration. How do you propagate this change then because then your client libraries are already generated, right? I think

the key thing is that you do adopt the generated like the templates and make sure that you slowly roll out this this generated code so you don't depend on manually written instrumentation anymore. Like the the schema needs to be the source of truth and you need [snorts] to roll this out through your organization. Once everything is based on the schema, it's impossible to like the idea is

that a CI should block anything that is manually written. >> [snorts] >> Thank you so much. I really enjoy working with Weaver. I have a question for the template. Do we have a way some guidance about documentation for template or SDK as you suggested because that was missing and LLM helps there but to have some sort of standardization on that templates. I think docs is one of

the greatest areas that could be improved in the Weaver repo sorry. I think if you tried yeah you had to pay. I think the team is doing something very So the the maintainers the former maintainers they created a new repository called Weaver packages I think and Weaver the CLI is being distributed as a container and inside the container there will be a lot of default templates and

I think you don't I did not enjoy writing Ginger myself and I think if >> and I think the if the Weaver team distributes the templates together with the CLI that will help a lot and I think that's the plan. Awesome. That would be great. Do we have time for more? I think we have a last one. Yeah. Thanks for the presentation. I have a question regarding

to metrics attributes or metrics paths in in your example for example you used HTTP request total but if one is going to standardize this it should be HTTP if the developers don't follow these standards then you will have a very divert metrics path. Yeah. Can Weaver solve this problem? Do you want to answer it? Yeah you can take it. I I think the final goal is adopt

the open telemetry semantic conventions right? Like they use dots don't have the suffixes but like I think we need to take a step back like and first we have a source of truth like understand how your telemetry looks like and make sure that you adopt the auto-generated SDKs. Once your org adopts auto-generated instrumentation you can like you have and you have control of the schema you can

switch the the metric name or the attribute name in your schema and make sure that this rolls out to all your org. Yes because like imagine like thinking about Prometheus labels that like you have a type of SDK and you're How you fulfill the path and which label is your decision, right? Like so imagine that you are you have HTTP request total with status code and you're

adopting the semantic conventions. On your templates fulfill both to have some gradual rollouts and later we when you have one rollout with the semantic conventions you can release a new version or if you're happy you're removing the first one you know. So it gives you more control on the on the >> idea is that you expose duplicated telemetry for a while. It's not duplicated telemetry it's just

you have different attributes with the same value but it's like a smooth rollout, right? Because you need you cannot just removing the HTTP the HTTP status code because you're going to breaking everybody so you need to rolling with the with the two two attributes and then after you remove the one. Sorry does does that help? So We we can talk we can talk Yeah we can talk

more. Thank you folks. Okay we are out of time. Thank you. >> [applause]