Hello World, Meet the Spanimals: Observability for Beginners - Tiffany Jernigan & Matthias Haeussler

This talk provides an introduction to observability, focusing on the key concepts and components that make it essential for modern software development. The speakers, Tiffany Jernigan from Grafana Labs and Matias from CGI, utilize interactive games to illustrate the distinctions between logs, metrics, traces, and profiling data, personified through fun characters. They explain how these concepts work together in an observability architecture and introduce OpenTelemetry as a framework for collecting telemetry data. The discussion transitions into a real-world example application that generates animal facts and images while demonstrating how observability signals are collected and visualized. Throughout, they emphasize the importance of correlation among logs, traces, and metrics to diagnose issues effectively in distributed systems.

Hi everyone. Um, welcome to our talk. Um, if you who out of curiosity, who here is like a beginner or newer to observability? Okay. So, maybe less than half the room. So, I'm definitely curious as to like what people's thoughts are afterwards and if they have ideas that, oh, you should have done this because people have opinions in many spaces and this is one of them. But

yeah, so basically as you can tell from the screen, um we're going to be talking about more of the like intro level things of observability. So yeah, hi, I'm Tiffany Jernigan. Um I am a developer advocate at Graphana Labs. Um hi, I'm Matias. I'm a VP expert at at CGI. And um we've just made the discovery that we've pretty much been in this very room exactly three

years ago. So this is where we did our first talk together. And I think we should take this photo again. Don't you think as well? So come our guests. >> All right. So now back to the topic. Now we know it's just right after lunch and people sometimes are a bit um um it's a bit harder to like concentrate and focus. So we we start with a

little game. Tiffany's just asked um who is new to the part of obsibility. Who in here is already experienced in observability? Okay. And who is not going to raise their hand no matter what I'm going to ask. Okay, good. Uh I just wanted to check if you're all there. So now we're going to do a little interactive game. The first of the round we um I'm going

to play directly with Tiffany and then I want you to interact and those with some more background in observability are definitely uh invited to answer. So, um, Tiffany, um, what do you think who that is? >> I think that's Tracy. >> That is Tracy. Correct. So, um, as I said, we're going to do a light introduction before we go into the more technical uh, deeper topics. So,

this is Tracy. And if you if you know Tracy by now, you should also know that Tracy is not coming along alone. Tracy is coming along with a couple of spannies which are kind of like similar to Tracy like little Tracy's um and they're normally coming around together. So what Tracy does it's it's basically she's jumping from one point in a large leap to the other while

the spannies normally take like smaller leaps um and like the compost uh jumps off the um of the spannies is basically what the trace you can do in one step. So now we're going to go for the next round. >> Okay. So now that you kind of have an idea of how this game works, do people want to try the guessing the name of this one? No,

>> nobody. >> Do you want to try then? >> Yeah, I might I think this might be Logy. >> So, you might be seeing the theme now a little bit here. Um, yeah. So, say hi to Logy. Um, Logy is well for Logs. So, basically, you can see Logy likes to chat a lot. If you ever have been around a parrot, you also know they love to

chat a lot. Um, so there can be different things that it can be about. There can be different types of formats. Um, the trick with logging is to kind of just like, you know, filter out the information that is being talked about instead of just like all the things that are happening. >> All right. Um, next round then, uh, any volunteers to guess who that might be?

>> Yes, >> Metri is very good actually. Um, it's not exactly. We called it metric. So I know this is hard to hard to guess but I I I mean we'll probably switch it to Matri after this talk. I realize now this is a lot a better kind of an idea. So why the elephant? Why ele? So um as most people know elephants have a big brain.

They can memorize a lot and that's exactly what what eleics does. So he's really into numbers and gauges and values and is is able to um to memorize all that and and and collect all that information. So on to the next one. Okay. So now, right now we've seen something to do with like metrics, we've seen logs, we've seen traces. Does any one have an idea as

to what this animal is even related to? >> Yes. Okay. Do you want to try guessing the name? >> I just want to repeat for the others. Is that profiling? And this is all exactly correct. So we now need a name for the profiler. >> We did this one a little bit weird. So we understand. Um, this is flamy because of flame graphs. Um, so basically it

can see things that others can't see. So like details that you might not be able to see. We'll go into what some of that means a little bit later, but right now this can give you a little idea just to visualize how things are. >> All right, we're getting somewhere. So now to the to the final one. Um, this is not directly related to one of these

called so-called signals. Um does anyone want to take a guess? >> Distributed tracing. >> No, this is uh distributed tracing is a bit far. This is more distributed tracing is more we had under the umbrella for Tracy. So this is one is collie. So why do we call it collie? Um now if you see all the traces, bannies, metric and flamies in there and this is basically

how they appear in the wild. Often very distributed, not very well structured and organized, often not correlated. So colleague kind of has that task um to collect and group them and um we in a second we're going to see what we all going to map those two. So this is kind of like to get this all started have a bit of an um like analogies between this

spanimal world and and what they really represent. So um as we mentioned traces or tracy do kind of the end to end journey where whereas the spannies are like the breakdowns in the in the middle element metrics for the for the metrical part for trends and rates and and all the kind of um of um kind of statistical data so to say >> like if you're used

to like hey how much CPU uh memory uh how many times did this function get called um just things like that counters etc. All right. Do you want to go for the next rest? >> Yeah. So, um, and then logs. I mean, even if you aren't specifically in the observability space at any point, you've probably seen something giving you like, hey, warning like this version is out

of date or error X thing doesn't work or just like some info messages. Um, so just kind of like events of what's happened. Maybe it's things like, hey, this user logged in or this function that you set up, something failed. Um and then we also have profiling. So blame me which I guess if we followed the theme it would have started with a P. So feel free

to suggest new names for us. Um so this is like going through your application be like hey where is time being spent. So things like um how much CPU is being used, how much memory is being used. You can use it to figure out things like hey maybe you have a memory leak somewhere or hey this function is only supposed to take like a millisecond but for

some reason it's taking two minutes. Obviously something is going wrong there. And then we have Collie which is our collector. Um so we'll kind of go into that a little bit more in the like open telemetry space um afterwards. But like basically it's going to collect and route your telemetry data. You can do some sort of processing etc. Who here has um done something with one of

the things on this chart? >> Okay. Good. >> Okay. And uh who here has done something with open telemetry? >> Yeah. Good, good crowd. All right. So, um, yeah, I guess you can continue in a second. So, the question would now be like transferring a little bit from the from the word of metaphors into like a real world application. We kind of built a a demo application

that we going to look into in a moment and trying to like walk through all the various signals that we're just covering now. Um, >> yes. So it's it's real world in a sense of you are actually it's something you can actually follow along and do the chances that you'll be making this exact application in production is probably extremely low. Um but yeah basically um the first

step like component I guess that we're paying attention to is we have a fact generator. So that fact generator can go and take in some sort of animal um and then it will talk to open AI and then it'll give some sort of fact. Um then we also have the image generator which will um t it'll take a prompt which I is coming from the fact generator

which we'll show in a bigger diagram of everything but basically we have our image generator. It also talks to open AI. Um I as he told me I named my image database application terribly because it's not the actual database but it puts stuff into our Postgress database and reads things out of it where we can see things using time leaf. I'll fix that one day. >> Yeah,

we we're going to refactor it to image back end after this talk. We didn't want to break it just like 5 minutes before the talk. >> Um so then we also have a runner. So basically what is happening overall here um is that you have these individual components. You have your runner that is what you're going to talk to. It will go and talk to your give

that animal. So say if we had like an elephant or a goose or a goat and pass that into the generator. Um then for the fact generator that fact will be used to generate some sort of image. It will put that into uh it'll use the image database app which is not a database um that uses that uses spring and Java unlike the other ones which are

Python and then um put that into a Postgress database. All right. Yeah. So we have also like different interfaces how to talk to that application. But in general I mean if you build such an application it would not be mandatory to like cut all the things out into distributed components. I mean the reason why we did that in this case of course to like resemble a polyglot

distributed application and also to show in the end that like the the obsibility information that we can get out of it kind of correlates um in the end. So that brings us kind of back. Um did you want to do that slide? >> You can go for it. >> Okay, I'm not going to. Um so in the end all of those components in one way or the

other um will generate logs, we'll generate traces, we'll generate metrics and so on. So then we're going to show of course um how Collie will come and collect all of those and in turn of course um not only being the site for collection but also being the site for exporting that information so that in the end we have um observability dashboards for looking into traces u metrics

logs and um and profiling information. And now we're going to do a bit of a hard cut and like moving away from that animal world into like a real world. So we mentioned open telemetry before and then um of course collie represents the open telemetry collector that we display here on the left that collects all the information and can send those things to various backends like we

put a couple of open source things in there like JGO Prometheus and open search but since I'm speaking with a Grafana person uh we going to do some uh demo using the graphana stack I mean it's important to know and that is one of the important features also in open telemetry that this is not bound amount to like a certain vendor. Um it can be configured individually.

So um you can basically configure like the data to send it to that and in turn also where the data is being exported. We're going to show you different configurations later on but we're going to we're going to start with this one. >> Out of curiosity who here has used graphana in some way. >> Oh wow. Okay. Um yeah so for those who have not just to

make it a little bit more clear. So like um the different icons that you see on the computers those are so like tempos for tracing low keys for logs there's mimir uh for metrics and pyroscope for profiles but those are like the backends and then you have graphana which is what you're using to visualize some of you may have used it just for like creating dashboards etc.

And then open telemetry has more than just having a collector of course but like Matias was saying um like one I mean it is open source for one actually everything that we're talking about here is um but just being able to do things like switching out different um like between Jerger and Tempo if you want to if you want if you need to put it to both

without having to learn something entirely new for every single tool that is there to set it up because that can be a lot even with like both within like say tracing or within metrics but also the entire uh like landscape of your telemetry signals. >> All right, so that basically brings us to the end of that um spanimal kind of intro presentation. So we're going to stop

the presentation at this point and switch over to the um to the live demo. So in order mostly people probably going to be used to seeing Grafana in the dark mode. So we put everything into like a >> Say what? >> Oh no, you just keep walking away from your microphone. >> Okay. I was I wasn't sure where you were pointing at. And uh but we we

basically put everything into light mode to make it easier for you to read. Um >> can you read the text the font size on here or should we make it bigger? Raise your hand if it's good. >> I see a few hands in the back so that the people must have good eyes. >> I guess raise your hand if it's not >> Okay. All right. So what

I'm going to do >> glasses as well. >> What I'm going to do now is um we don't have a fancy UI. We want to do some commanded interaction. So I'm I'm going to run a curl command against that application uh and say since we just talked about elephants please generate um anything of like on the theme of an elephant. This is naturally going to take a

bit of time because it's using the open AI API basically leaves the machine the network and and then goes to the internet trying to collect the fact and again based on that fact generate an image. So what we seen here and this is this is not a setup this is really generated. So elephants are known for their incredible memory. What I just said they can remember and

recognize individual elephants they have met and so on. And down here we're going to get >> you can go to the other the front end. >> I I will just start with this. Now um the model we are using might not be so elaborate but you can get to the point that this is an elephant and this has basically just been generated. I'll do another one. um

maybe a cat. And in the meantime, we can switch over to the to the dashboard and see if any of those things can be displayed there. So, we're going to since we're waiting here, um I'm going over Where did it go? Too many things open. Where is >> It's in the same one as you open the >> Uh that's right. Okay. So, okay. You can go ahead.

>> Demos are hard. >> It It is not recorded. It is really live. So, um, >> okay. Is it being slow? I'm just going to go to explore. >> No data. >> What's going >> H? >> You looking for a trace? >> As in it's not showing up at the moment. There we road. Okay. All right. So, um yeah, there's multiple different tools have different ways to

be able to actually go and view things. Um and depending on how the setup is, it may take more or less time for it to actually show up. Um after it's like it takes time for things to be able to get collected and then being able to go and see them in general. Um you can change whether how you're doing that and also depending on what you're

doing, maybe you're pulling versus pushing. Um so this is uh drill down which is in graphana. Um, basically it's just an easier visualization way to be able to look at like your metrics, logs, traces, and profiles. Um, if you have something super specific, then you can do it in regular in explore. But like, so for instance, if I click on here, I can see the trace name.

Um, we so like when we were talking about having traces, so what we have here for this top one that's going all the way across, which is like 23.6 seconds, um, that this whole entire thing is our trace. And then you can see that within each trace within the trace you have like these little parts. So like for instance I have this post right and inside this

post you can see all of these ones are taking the same amount of time etc. But like each one of these little parts is considered a span and together all of these and like the amount of the time this takes and the amount of time that this part takes you can see that together they take up the entire time there. Um so like first we I was

mentioning that we have the runner. Um the runner then goes and talks to our fact generator and then and actually can people read this? Okay, maybe. Okay. Um so then we have the runner again as I was mentioning and then that goes and talks to the image generator and then depending how you set up your application this will look different like for instance because uh the fact

generator and image generator are being called with just separately they have their own endpoint. Um you can see each of those are in like in the same line. Whereas the image generator actually goes and calls the image database from within it. It pushes it to there from it calls the endpoint for the like the back end um there. So you can see that actually it's within there

as well. And you can see things here like it's actually talking the database. You can see that there's an insert. You can go and and look in here and there's like different things like span attributes. Um there's resource attributes which you can look into a lot more afterwards. we can see things like hey I'm using Java um I'm inserting into Postgress etc and things like that there

basically there's just a lot of information that you can end up finding about how things are talking to each other and how long they're taking as well as like being able to for instance if this one has any logs we can go and look that one doesn't specifically but if we were to just go do we want to just >> yeah take a bit more >> there

we go Yeah. >> Yeah. Um, so I mean just to to wrap this all up, I mean if if you if you look at the the traces here, we know only collected two because we only done two calls. Um, it's pretty obvious that um, this is like the time for generating the fact the take generating the image takes a lot longer. Um, we can basic invoke a

couple of more things. I also have like a uh I do the code in a second. Um, there's also a health check. So you can see this responds really quick. it doesn't generate anything. And if I generate a goat, um we have a built-in um problem so to say that it takes a consumes a couple of seconds more um because it it's um it it listens particularly

uh onto that um onto that goat string. So if I go in here again and and refresh this uh we can now see we have three traces. This is that get command for the health and you can see this only took like 82 milliseconds. And now the the calls to the database suddenly have like a bigger percentage as in in within the calls. So that of course

gives you the ability in your real world application to identify like slow running components or components that take really long to respond and drill down on an error. I think the goat might have finished now. So um if we if we refresh the few at this point um it's not there yet. >> So it's it's there now. Um, so this one took a little longer and now

you can also see I mean this is we we see it probably quicker because we built it but normally this runner component did not take a lot of time. Now this suddenly has this 5 seconds delay in there. That's like an error that we like put in to make that visible. Most likely your double the debugging won't be as easy as that. Um but this is the

way how of course you can easily identify and spot that. And once you click on it, um, you can then say, I want to have the logs for this span and it's correlated to the logging information that we put into the application at this point. So we kind of put in a question. Do you really want to get a fact for a goat? Okay. Well, here it

is. Um, this is basically um is what is being being spit out if you have any logging information at this point. So it's it's correlating the various signals basically within that dashboard. And then I theor the theoretically built in a fake animal. And then you can see here that it goes and says animal doesn't exist. And then if we go back to traces and see if it

showed up yet. Still sometimes it takes a bit of time but there we go. Should be >> it's bottom one. >> You can see from the time. >> Yeah. So we can see here uh that basically you have this error and then you can go and like see an event and we can see hey um the animal stomal fox doesn't exist um and then like there's an

exception. So that can help you be able to be like oh hey there's something clearly wrong here and um where is that happening and you can see at least I mean this part it's in the runner so it's the first thing it never even gets to the other parts whereas for other applications you may have something working perfectly fine and then suddenly you hit an error and

then you can start diving into that more. >> All right so since we touched logs and and traces quite a bit I mean logs of course you you cannot only access if you like relate them to any kind of trace. So there's this kind of like you see the logging of all the four applications in here and that's normally also the tricky thing because like each application

logs in a different way. They might have different conventions and formats and and it's it's kind of difficult to get something meaningful out of that. We have basically the same thing here. But once you can correlate it to like actions that happened within the call. Um it's it's much easier to to pin down basically the root cause of a problem. Now to like complete the picture uh

I'm going to do metrics first and you can say a few things about about profiles. So I mean metrics are not the most exciting thing. Um it's just basically a collection of of the data. We you can see currently we are picking up 50 different metrics in total. Um they all build very beautiful graphs. So if you um want to narrow it down, you can basically type

whatever you're looking for. So if I go on DB, these are the the database metrics I'm collecting here. There are 13 out of 50. Um if I'm more interested in whatever for example the the Java agent would pick up then I have the Java components here. So things like garb garbage collection time u memory consumption or invocation rate whatever else that will help you um to get

a get a better grip of your system and see uh and see if the things are performing in the way that you want them to perform. And then also like if we just take a quick jump back when we were looking at the traces um there depending on how you have it set up like when I clicked on something you can see that there's logs maybe there

aren't actually logs for that thing but um some tools uh have the ability to be able to like you have say the metrics that he was just talking about you might have the logs that we're just talking about the traces and then just like having a way to actually be able to correlate them together um can be super useful. You can also like from traces for instance,

you could also have metrics specifically in there as well. But like if you're looking through and you're like, "Hey, this thing is taking so long. Why is this taking so long?" Oh, hey, let me go look at the logs at this point in time instead of having to go search that up manually. Um, so like right now we have things Yeah, like for instance, how long is

something taking? We have uh different like metrics for the uh things for like the database for our application. At the moment we didn't make any like metrics specifically for it but you can create metrics for something that's happening in your application. Um then we also have So um at least starting from this you can see like things are kind of more in like the metric side of

thing. you have your graphs, you can see, oh, hey, like um so this is using the EVPF profiler that um is so so to be able to get um both the Java and um Python profiles for that. Um there's different ones depending on what language you're using, etc. If you're using a langu langu one specific to your language um so basically like the way that this is

you can see that there's uh different services that we have in here. Um, for how this one works, it's splitting it up by language. So like for the Java one, we have one application, so that's fine there. Um, with how things are right now at the moment for this one, um, if we look, we have Python that currently is putting all of them together, but there's different

things you can do. Um, right now the profiling I think it's as of like yesterday or just this week, it has um moved to the next stage um in for um the open for open telemetry. Um, so it is a lot newer than the other signals in open telemetry and just it's newer for people in general I feel especially based on um what I've seen in general

with people raising their hands and such but um so who has looked at a flame graph before okay so not a lot of people all right so like for instance if I were to let's go yeah let me just find the Java one here and if I go to the flame graph the warning about the plugin is because I don't have the plugin for if there or

something AI related. But um basically you can see things like how much uh CPU time is consumed and then this thing lovely thing here is considered the flame graph. Um I first heard about it like quite a number of years ago through Brendan Gregs. So if you want to see like where some of the initial conversations about flame graphs came about, you could just probably look up

Brendan Greg and flame graphs and some of the blogs will show up for that. Um but basically you can see like the for this it's for CPU. So you can see like the total um you can switch it to uh for depending on the uh language and what you're able to get you can like switch it to memory etc. Um but it basically gives you an idea

of like how much time things are taking. So, say if some we didn't I made an example that kind of messed with things a bit, but um it didn't it did not make it into this round. But basically, if you look and see something that is taking way longer than you expect um for the amount of like say the amount of CPU, the total time that's being

used there, um that can help you figure out what's going wrong there. And then some of these tools have the ability to actually connect like say GitHub for instance and be able to see like per line from your GitHub code like how much CPU or how much memory is this specific thing taking. So basically you can use these different components all together to help get you a

big picture of what is happening with your environment. >> All right. So I mean that kind of uh brings us to the end of at least the um the explanation of the four signals that we have been talking before. So we have a bit time of remaining. So what I could quickly show is also the aspect of like the collector architecture. So this unfortunately doesn't look as

cool as a flame graph does. So I'm kind of jumping real quick. This is our repo and I'm also going to share the link to that in a in a second if you want to play around with that as well. Um basically the um the open telemetry collector is being configured by such a collector file and the one we're currently using um is is this one. So

on the one hand as I said you have some receivers that's where like all the your data sources can can emit the um the the the information to and then you have the exporters and as we said here we export to Prometheus, Tempo, Loki and and and Pyroscope. Um if you wanted to take a different approach you can I have an alternative configuration here. So in this

case um I only have one signal um receiver open. So this one would listen only on gRPC and in turn it would export like the locks towards JGO and the um and the matrix to Prometheus. This is also basically an important aspect. Um you can have all of the four signals but maybe you're not going to need all four of those. So that also means um you

don't have to like fill those gaps. you can configure the the collector in a way to only have those kind of signals addressed which which are really irrelevant to you and um yeah with that I think we have about one minute remaining I would say um is there are there any questions is there something you have heard in terms of obserability and want answered I mean we

try to cut it down to like the really basics since this is a beginner's talk um I hope for the beginners it was still able to follow that concept of a flame graph which is a bit more advanced but Is there anything else you want to ask and see? >> And then there there's microphones in front. And in the meantime, um like depending on what tool you're

using, like an idea, one of the good ideas about it is to be able to put bring everything together. So like um at least on Graphana, I don't know about all the other specific things, but you can create a dashboard and then you can pull in like, hey, here are my traces uh for this thing. Here are specific metrics I care about. Here's my flame graph. And

be able to have a visualization of all of that. Um, if I'm just going to throw this out there and not go into explaining it all, but like if you want to be like, "Hey, make me a dashboard that does blah blah blah instead of figuring it all out." Like if you're you can use an MCP server. So like Graphana has a open source MCP server, so

you could create things um and not have to learn how all of it works, which also helps you understand like hey, what traces are failing, etc. And you can learn more about observability doing >> All right, any questions? Okay, well we're still going to be around. I mean um Grafana also has a booth. You can definitely find Tiffany there. Um maybe one two last things. Um we

have the the link to the repo. Um and you we have it somewhere and I also have another talk in a in a good hour from now. So at 4:00 I'm also speaking about my experience at teaching university with the help of CNCF toolings. If this is have any interest of you please feel free to stop by. And for now, I would first of all like to

say thanks for coming and and and listening. >> Where's it? German keyboards are hard. Uh so yeah, it's just Tiffany Fay/an Animal Effects. Um I'll push another thing in so you actually see examples of profiles. And our to-do is to actually have a read me. But um in the meantime, this is exactly what we're running. Thank you.