OpenTelemetry Logs Driving a Major Shift: Events, Richer Data, and Smarter Semantics - Robert Pająk

In this talk, Robert Pajonk discusses the evolution of logging standards within OpenTelemetry, emphasizing the need for better logging practices to reduce noise and improve performance. He highlights the importance of structured logs and event tracing using systems like BPF in Linux and Event Tracing for Windows. Pajonk explains how OpenTelemetry is moving towards making logs a first-class citizen in telemetry, detailing the structure of log records and the inclusion of complex attributes. He introduces the improved APIs for logging that enhance efficiency and encourages the use of semantic conventions for consistency across services. The session culminates with plans to deprecate span events in favor of log-based events, aiming to streamline how developers emit events in their applications.

Hello everyone. So, let me start with a disclaimer. I have This is the next version of this presentation. I have given it twice, but last time was a few few months ago, like six or seven. And so, I had to even be in seven months, I had to restructure it. So, yeah. I will need your I need your help first. Please, I have an ask I need

a cooperation. Could you please all stand up, the people who seen logs like this? Could you please stand up? Anyone of you say seen logs like this one? Like, you know, info level, some dashes, exceptions which tell nothing. Oh, yeah. Okay. Now, I have the next question. Who of you No, please please keep stand up here. It's pretty late. You have been sitting very long time today.

So, you need to exercise a little bit. Okay, so now sit down those who do not see this kind of logs anymore. Oh. Yeah, so I think you're a perfect audience. You can now sit down. So, so thank you for your cooperation. So as you can see, a lot of us deal with logs, which are basically like trashy. So, they cause uh they cause noise They they

add a lot of noise. They also uh they also affect our performance. Because of this, we also pay more more money because we need to store somewhere this kind of logs. And this is why we need better logging standards. And the thing is that there are already systems which approach uh like event tracing or logging in a better way. Uh there are kernel level trace points, uh

user events in BPF in Linux, even Windows has something called event tracing for Windows. Language runtimes have also special mechani- mechanisms and systems for for efficient event tracing. And those kind of systems makes logging structured and extremely fast. And this is what we need. And when I say we, I say I want to say we are the our ecosystem. And I also I want to say that's

OpenTelemetry because OpenTelemetry is a framework for tracing, metrics, logs, profiles, and any kind of telemetry. maybe I should start with introducing myself. Because, yeah. I'm Robert Pajonk. Uh you can find me as palant on GitHub. And I have contributed to logs in OpenTelemetry since November 2023, which probably for end user perspective, if you're waiting something from OpenTelemetry for logs, feels like a lifetime. But from my perspective,

it feels like a few days. And I'm also an OpenTelemetry Go maintainer. I'm working on designing and implementing logs in otel go. And also a specific I'm also working in a log SIG in OpenTelemetry special interest group to improve the logging experience. and I work at Splunk. A company which embra- embraces logging. And yeah, and it gets it it pays me for doing open source. And a

disclaimer, I'm not an English native speaker, but if something is not clear, don't blame my English. Probably I'm just saying some un- nonsense. Uh okay, let me start with the structure of the presentation. My goal is to tell you why open term in logging OpenTelemetry is no longer something which is only about bridging logs from your libraries, the logs which I sent you uh sorry, which I

have shown you just before. It's about have making it a first-class citizen. So, I'll tell about the data structure and APIs for logging and also the semantics that we're trying to drive. So, let's me start with the syntax. What is a log record in OpenTelemetry? first of all, it consists consists of some timestamps, when when some something happened, and when it was observed. Then, because OpenTelemetry is

also about correlation trace correlation, we have also the trace correlation uh fields so that the magic happens that we can correlate logs and logs with uh with the tracing. We have the event name, which tells you what uh which tells you I'll get to it later, but basically it it's a name which tells you what happened What what is what is the record about? We have a

severity number, which is like a logging level. We have a body, which is a human representation human readable representation. Uh we have also some additional attributes. Resource attributes, which tell you what the software has What software has emitted this log record? And instrumentation scope, which tells you unit of the software which has emitted this telemetry. Like a component. So, in OpenTelemetry, this is just a JSON for

the sake of the presentation. In OpenTelemetry, logs are not JSON because these are just strings. So, this is just a like a domain model. And in OpenTelemetry uh s- logs are not strings. This is structured log. It does carry the correlation to context. And I also want to add that recently in in OpenTelemetry, we have also added the support for complex attributes, which means that attributes This

is an example of a GenAI uh based on a GenAI semantic conventions. And you can see that attributes can contain complex data like maps, arrays. And those data can be uh homo- heterogeneous. Uh so, no more flat things like like uh serializing stuff to JSON and then deserializing on your backend. My clicker stopped working. Oh. No, it has not. And the thing is that these complex attributes

are now included in all signals. So, this is one of the things which the OpenTelemetry logs is change is making a shift and a change in the whole OpenTelemetry OpenTelemetry ecosystem. And you can read about it more in the in a blog post in open-telemetry.io. And let's go back for today let's go back to the topic about the event name. So, we need a distinction between something

which already have the existing legacy stuff that was emitted through logging libraries, and something which conforms to system semantics. So, we have an event name which defines you the well-known structure of your log record. And the when you when you use it, you should also follow some semantics. So, some common some attributes to make sure that this this is some kind of versioned and you make sure

that uh it it you can predict that it will have a consistent structure. And there's also more about it on a blog post called OpenTelemetry Logging You, which is the which which also describes the background and context why we have decided that the events is just a special version of a log The next thing which is changing is that right now OpenTelemetry logging is not about the

bridging. It's also emitting logs directly from the from the OpenTelemetry APIs. Thanks to it, we have no format conversion. We can reuse the attributes that you're using in tracing and metrics. And yeah, you can use these maps. You don't have to rely just on strings. So, it makes your your code base that emits telemetry more consistent. And it's very important for instrumentation library authors. So, they do

not need to depend on third-party logging libraries like log4j. it can be also useful for application developers that want to simply have a between logs that kind of are just like a debugging salt, or from something which actually matters and should, for example, fire alerts. The next thing which I wanted to talk, which is kind of new, is the enabled API. before, each time you wanted to

reco- you want to bridge or emit a record using OpenTelemetry, it was always you you had to always just emit it. And it was causing a wasted CPU cycles and memory allocations when you had to put some expensive attributes. But right now, we we added additional enabled check. And you can say that is like measure twice, cut once. So, you do not need to pay for this

additional overhead. And the reason we also added it because some tracing even tracing systems allows allows to make it very efficient, the thing which I mentioned previously like event tracing user events, ETW. Because then you are able to have almost zero location log emission. Because if you're not subscribed, you're not interested in receiving some events, then why paying the cost? And we added it first to the

logging, but we also added this functionality of enabled also to the metrics API and tracing API as well. And one of the biggest changes which is coming to OpenTelemetry is a shift. We are trying to transition from using span events to log events. Right now in OpenTelemetry, there are two there are two possibilities on how you can emit Sorry, you can emit events. And two ways is

much too many. The first one, which is since for years, you have span events. So, when you're creating a span, you can you can also add an event with a name with attributes. And this is the way that most people are often use when making logging using OpenTelemetry. But right now, we also have the possibility to emit events via logs API. And these events are more rich

because you can also have this the severity the severity level. And the problem is that giving two possibilities is not really great because then it splits like users to ask which one should I use? How should we evolve semantic conventions? And all of this kind of questions which are hard to answer, but also we The thing is that sometimes you have some something which is not a

connected with a span. So, that's the reason why we are trying to shift to log base events. So, what is changing? We are working towards deprecation of the API for adding span events. And we want to prefer and guide new users and new instrumentations to emit log base events using logs API. At the same time, we really are aware of all the users that are currently using

span events, and we want to make sure that we provide good migration story for the end users that still rely on span events or use the span events API. So, they they can still rely all depending on the use case or depend on the model that the events are still recorded on the span events or to make it possible to to transfer the span events into the

log events. And we want we are starting to evolve the semantic conventions of OpenTelemetry so they start to use log based events instead of span events. These are big changes. And we are just deprecating the APIs on the OTLP layer, so we are not removing anything. We do not want to break the end users. We want to provide a migration story for you. And there's a blog

post which we I think published like a week ago. And we also have a GitHub issue that if you see any problems with the migration, if you see any dangers of you're unhappy or you are happy, please provide feedback. So, we got we mentioned the structure, but now I want to a few words about the semantics. So, one of the most important thing that the OpenTelemetry brings

is that it tries to give you a some guidances to name the things in in the same way so different teams follow the same patterns. And thanks to it, you can create common dashboards between applications and you're able to correlate stuff between multiple services written by multiple teams. So, that's why OpenTelemetry semantic conventions comes in. And semantic conventions consist of I will say two main areas. One

is the registry of attributes. So, you have cross-cutting concerns like attributes like code code line number, error type, service name, which is common across many signal types. You have also signal specific attributes like local UID or IO stream. And there are some some domain or technology specific And one of the things that the semantic conventions also say that it provide guidance how to emit events, how to

structure the event, what what levels of severity levels you should use, what attributes are recommended, how you should how you should report errors. And also domain specific events, how they should be structured, when should you emit it or not. And it's not just about naming, it's mainly about the behavior. So, quick summary. Logs in OpenTelemetry are not just but these these are these are complex structures that

carry context and type And these complex attributes are already available in all signals. Maybe not in all the languages, but in OTLP, they are for even years. And more languages are adding support for emitting those. And for OpenTelemetry events are logs with a name and well-known structure that follow the semantics. And you can start using you should be start using OTel logs API directly, especially when you

want to follow some conventions. Enabled API has been added to all signals in places where performance matter for you. And yeah, probably the most recent change which is coming deprecating of span events and transitioning to logs And please follow the semantic conventions. They tell you how to use events. They tell you how to name attributes. It will make your SREs or your future yourself a more happy

person. The goal is that our vision is that we want to make OpenTelemetry logging for you performant, at the same time providing you very rich data, and possible to integrate with efficient efficient logging systems. Thank you all for listening. This is a QR code for the presentation. >> [applause] >> Now, go forth and log responsibly. >> You can find me on GitHub as palaret. And here's the

microphone for asking questions. We have 13 minutes. I went too fast, so please take your time and ask questions. Thank you.