Ballroom F Saturday Mar. 07 - SCaLE 23x

In this talk, Jacob Coffee discusses the significance of observability in Python infrastructure, focusing on best practices for monitoring systems effectively. He emphasizes that while monitoring tools are essential, the real challenge is understanding system outputs and eliminating unnecessary alerts that create noise. Jacob shares insights from his experience with the Python Package Index (Pip) and the critical need for actionable alerts that do not disturb team members for non-urgent issues. He highlights the importance of community contribution and collaboration in maintaining the Python ecosystem. The session covers personal anecdotes, illustrating how observability evolves within a small team and the necessity of maintaining infrastructure reliability as Python serves users globally.

Okay. Hello. How are you guys? How are you doing? So, my name is Shannon. Um, we are here today with Jacob Coffee. He's going to be going over thoughtful observability and moni monitoring the Python infrastructure. So, let's give a round of applause for Jacob. >> Hello. Thank you. How are you beautiful people doing today? Super happy to be at Scale. Uh, this is my first time at

scale and I've had a really great time meeting a lot of you in the hallway. Um, not that I've skipped any talks, but the hallway track is always my favorite track. Um, so, um, most of the things I've done this week is try to convince people to contribute to the Python ecosystem in one way or another. That's there's lots of ways to get plugged in. Whether you're

super coder or you just want to work on docs or do design work, there's tons of ways that you can get plugged in. Things like python.org you can contribute to or the CPython runtime, you can contribute to that directly. um other open source projects like Lightar, which is one that I help maintain. Uh tons of options. So, um I've also been trying to convince people to come

to Pyon US this year. It's in Long Beach, so it's super close. There's no reason you shouldn't come. Um it's the the biggest Python conference, I think. Yeah. Uh in in the US, but probably the world. Um and if you ask me, it's the best conference in the world. Um, so I've been trying to convince all of you to come and maybe if you're a a person

that can sponsor us or have a booth there or if you want a community booth, if you're on a community project, we'd love to have you there. So, if we talked earlier in the hallway about any of this, uh, congratulations. You've been upgraded to the extended version of my sales pitch to Pyon US. U, I'm Jacob Coffee. I'm, uh, staff at the Python Python Software Foundation. I'm

also a maintainer of lightar which is an awesome web framework written in Python and also a CPython triagger. Um today I wanted to talk about something that may sound tedious on paper or boring to some uh but it keeps me up at night especially when I get 3 a.m. pages. Uh observability. So uh not the buzz word but like the practice. And I won't go super in-depth.

Uh I tried that earlier a few weeks ago when I was thinking about what I was going to present because there's lots of things you can drill into um and I did not think that that would be uh super beneficial. So we're going to do a sort of high level and maybe some best practices uh that you can think about and take back to your organizations or

your projects. So um when infrastructure starts failing the real question is not usually uh do we have monitoring because things are actively failing. It's like do we understand what our systems are telling us and does our observability that we do have at the moment uh give us like the shortest path to resolution and if not what do we need to change? Um quick question I guess before

we start. How many people have been paged by monitoring before? Okay. And keep your hand up if that alert actually helped you diagnose the problem quickly. Okay. Okay. One more question. Uh keep your hands up if you just silenced the alert because it was bogus or not time urgent or it was 3:00 a.m. and you spent the night prior indulging a little bit in too much wine.

Okay, perfect. Yeah, the ratio is pretty much what I was uh why we were talking about observability today. Um but first we can talk about where it started for me. Uh it's not really with Python but just with responsibility and with the feeling that if something breaks it's on you. So I was born in the CIS admin world. Uh lots of staring at logs and fixing dis

space late nights if a server went down at 2 in the morning. That was that was on me and I was still am a big fan of self-learning. And sometimes your job does not really require you to or give you access to because corporate people, you know, they want to lock down things so you don't you don't get access to all these things. You don't get to

do the things that give you the on the job experience. So for me, that's where my home lab came in. My home lab taught me that there's no SLA more punitive than a partner during streaming hours. Uh 59s is is not aspirational. It's like the bare minimum especially when a new season of Telaso comes on and you can't stream it because you you've messed up the VLANs

or something else. So next for me um I started this CIS admin job and I'm ADHD is super lazy and I'm like how do I script away all the clicks? I was not familiar with programming or open source or anything like that at the time. Um, and so I got into PowerShell to automate. We were Windowshop at the time to automate all this. And then Python entered

the picture eventually. And as Brett Cannon said, I came for the language. Python was great. It was easy to jump into. There were tons of resources online. The community was great. Uh, so Brett Cannon said, I came for the language, but I stayed for the community. And, uh, you'll hear that at Python conferences all around the world because that is an amazing statement, but is super true

as well. I've always cared about keeping things running. Um, okay. This is a peacock. It has eyes. Uh, the story behind that in Greek mythology, Argus Bonopes is allseeing giant. He had 100 eyes all over him and he could see everything everywhere all at once, which is a great movie by the way. You should really go see it. Uh, and yet he was killed in his sleep.

Something snuck through the cracks. And so I'm going to take that and with some liberty and say that seeing everything is not always understanding. And I think that's a big problem in observability. So as the story goes, Zeus uh had a had a maiden that was not Hera, his wife. Um and one day uh Hera found out it was Io. And um when when she found out,

Zeus transformed her into a cow. And um Hara was smart enough to know what was going on. So she had Argus take Hara, the cow version or Io, sorry, and um take her away and watch over her. Uh Zeus didn't like that. He wanted Io back. And so what he did was have uh Hermes, which is one of the other Greek deities or or people, um go

and lull Argus to sleep. Now, how do you do that? Because he has eyes. He can see everything. You know, how can you do it when when half of his eyes are are sleeping and resting, the other eyes are awake. So, it's like 24/7. His observability is like his thing. But, uh, Hermes was able to lull him to sleep with some music and eventually kill him. So,

this is a problem in observability to me. Now, I can show you what we're actually dealing with. The Python infrastructure is is not a side project. It's the backbone of uh global modern software development. When Python uh systems go down like Pippi, um whole nations see things. Governments notice um projects from all around the world. So it's important that we can see what we're dealing with and

and know how to get it back up quickly. So part of the Python ecosystem like I just mentioned is Pippi. It's Python package index. If you ever pip installed anything, um, that's what you're hitting. Well, pieces of that are ours and then technology partners like Fastly help us to front some of that load. Uh, it's the global dependency backbone. We also have docs.python.org, the reference documentation. It's

a good way to jump in, learn more Python if you're not familiar with it, especially if you're a student here today. Um, millions of developers consult this daily. We have our mail system that is like a coordination layer. one of the coordination layers uh where PEPs and decisions get debated endlessly and decisions get made and then we have And this is the place it's Python's home. So

it's the place for all Python information, but we also host uh Python builds there. Uh for so when you download Python, you're probably uh downloading from that service. Uh there's tons of other services that empower Python developers and the CPython team that that produce the binaries that you use um on all of this runs on infrastructure that a handful of people maintain. So, um, we have 3.4

billion plus package downloads per day. Um, I want you to sit on that for a second. That's that's a lot of traffic. That's not like a vanity metric. Every one of those downloads is someone or some system, some CI building something, whether that's a startup or a research project, Fortune 500, um, a state government, and all that flows through infrastructure that we maintain. Our daily downloads have

grown pretty well since September of last year from around three billion uh weekly or week weekday downloads to now I think around 4.5 um during the week. We have a big drop off during the week in that's why we have this this average here. Um notably we also host um a large store of of packages uh for free. So that's a great service uh for the community,

but companies host there too. Uh Pippi handles over 1.6 trillion requests per year and along with all of the other ancillary services that run Pippi. Uh we push 300 terabytes of data a day, sustain a lot of requests every second. Uh and one cool thing about this to me is that Pippi handles more traffic per second than Google does Google searches per second. Um, we found this

out a few weeks or a month ago at Fosdom, uh, after digging in a little bit and that was pretty mind-boggling to me. And all of this flows through infrastructure maintained by two paid people now. Uh, the rest are volunteers. We we really rely on volunteers. You may know that other big languages have the backing of gigantic companies and Python is managed by volunteers and and the

Python Software Foundation is a nonprofit. So, it's a lot of give and take there. Um, this is a little snippet of how many projects as of uh yesterday that that Pi uh Pippi hosts. So, 75 750,000 projects, tons of releases, way too many files, and over a million users. Uh, so all of that to say, there's no quiet hour. When it's 3:00 a.m. in California, it's peak

hours in Europe. when when Europe sleeps, u Asia picks up. Python infrastructure serves developers across every time zone, every continent, and there's literally never a moment when um someone is depending on this being up. So when Pippi slows, uh CI slows, builds fail, deployment stall, developers open issues, they want their their services working. uh blue sky and other mediums like that light up with why is Pipi

broken? Uh when yeah so when it fails people notice it's not a niche infrastructure and it's the foundation of a big chunk of the industry. Um so our obser our observability is not maybe not be what it needs to be. Um and and I'm going to take you through what we um do to are doing to fix that. Uh, real quick, small team, big service. So, the

team behind this, I'll show you a little bit of reality. The Python Software Foundation has about 13 staff now. That's the entire foundation that it's not just engineering. Um, it was two. Now, it is just me. By the way, we're hiring for an infrastructure engineer. So, if you'd like to come work with me and help me on observability or any other of the fun infrastructure, uh, come

apply. Um, but that's it. So there are two security engineers um tons of innumerable volunteers just all over the world and one to maybe soon two one of you uh infrastructure engineers. this is part of the open source model. Python is open source. People donate their time and their expertise and their weekends and without them none of this is possible. So, this is a really big shout

out to the volunteers everywhere in in any open source uh system. Oh, and we're hiring. Just told you that. So, uh with a team this small, every alert takes away from real work. Every notification pulls someone away from building. For me, uh context switching is not easy. So, uh when I when I see a page, I I really want it to be actionable. uh because I'm I'm

fixing something or planning for Pyon US, tons of other um feature work, we don't really have the luxury of of this much noise, but with a system with with systems of this scale, um we we really need to lock in and we can't afford to page someone at 2 am for something that does not require human intervention right now. Uh that's not really efficiency. That's like keep

the lights on mode and it's not great. Um, so this is kind of the honest part, the part where I tell you maybe what we got wrong and um, if I told you that I stood up here and told you that we had it all figured out, I'd be lying and you'd learn nothing. So, we had an alert that fired constantly, uh, every day, multiple times, um,

and and you couldn't just silence it for a whole number of reasons. uh it was monitoring a a deprecated service and it could have been made it static ages ago but instead of simplifying the architecture and dealing with the politics of whatever was behind that we had just been uh pressing the duties and slap some monitoring on it and add some uh synthetics checks and things like

that. Oh, and a dashboard. Everyone loves a good dashboard. Uh it felt like we were on top of it, but we weren't really fixing anything. uh we were just watching and responding. So, and what happens is eventually you stop reacting. You get a page for a thing all the time and you're like, "Oh, I can I can just do this later. It's not that important because it

paged me an hour ago or yesterday or or whatever." So, your brain, the human brain just adapts and the constant noise becomes background. Hello to all my fellow ADHD friends. So that alert that fires every single day, it becomes the one that you stop reading and when the real problem shows up, you miss it. So the noise, as I just said, trains you to maybe ignore alerts.

Uh maybe if you're on a big corporate team, you have more people, you can manage these things, but as as a small team, uh it's like signal erosion. And it's like a silent killer of observability because your moni monitoring can be technically perfect. You can tag all of the things in your Kubernetes architecture. You can put all of the century things you want on top of it.

Make all the data dog dashboards, all the metrics collected, all the thresholds set. Uh it's still useless if you have people that have learned to ignore it because it's too noisy. So we made some mistakes. Um and so these are some of the principles I would like to work towards. Now, um, as I said, I like to so we're we're in some busy times. We have PYON

coming up. It's our biggest thing uh for the year and then existing infrastructure and other duties uh to support our volunteers. So, that demands a lot of my time. So, did I mention we're hiring? So, monitoring everything is is super easy. Uh, you add metrics, you add your dashboards and alerts. The tooling really makes it trivial. There's tons of great options. You can spread out across different

services. Um, and it feels productive, but like like, hey, look at all this data. I I have it. Um, but I think the better question maybe is deciding what not to monitor. Uh, that's hard. That requires discipline. Um, and it means looking at a metric and saying this doesn't this hasn't earned its place. It's not actionable or it's just noise. Maybe we should delete it. And actually,

maybe that's kind of framed wrong. like what if we did instrument everything because you want to have the correlations and and like references to like oh this was done because of this other thing over here and you know all the other fancy things that we have but we want to adjust our alerts so that it is most impactful uh and useful for us and that I think

is the actual skill um of a lot of observability. So there's a distinction that sounds obvious but it gets violated constantly. Dashboards are for understanding like how's my system doing? You can look at them when you choose to and they give you context and trends uh in the big picture. Paging though is is for action. It means someone must act like right now and and it interrupts

whatever they're doing. So there's a human cost every time it fires. every page should be worth waking someone up at 3:00 a.m. If it's not, it's like a dashboard metric, not a page. Uh, and a lot of times we blur this line and that goes on for for too long. So, don't make the same mistake. So, if it does not require human action, it's not a page

period. put it on dashboard, send like a weekly digest, log it, but don't interrupt someone's sleep or dinner or time with their kids or their weekend when they go play volleyball unless something is like actually broken and a human can fix it right now. Um, so and also you want to be mindful of your team members and how might they be affected if you just add all

of these telemetries and then you put your alerts and now you're pinging your entire team uh and annoying them. We want to monitor what users feel. Uh big latency issues which might lead to failed downloads and timeouts or broken builds. Um I think these map directly to someone having like a bad day and that is something that is actionable. Or the CPU is at 80% or or

90% whatever disc is at 80%. That's fine. Trending upwards. Yeah, we could we could do some checking, but we don't need to uh page infrastructure on vanity metrics. Um I think high utilization is not inherently a problem. Uh I think the question is better framed as is a user affected right now. If the answer is no, um don't page. It's a recurring theme. If you haven't picked

up on that, um it might just be we can fix this by the end of the day or the end of the week uh or put this on on a project for later. I think a a good practical test to apply is like a 90-day question. Has anyone looked at this dashboard in the last 90 days? Has this alert led to a meaningful action in the last

90 days? Uh if the answer is no, maybe delete it. Like really delete it. Uh you can always recreate it if you need to, but unused monitoring is not neutral. It's very cluttered. And if you've ever tried to navigate data dog uh dashboards when there are hundreds of of projects, uh it's a lot of noise. And if you're not great at navigating those tools, then that's just

uh making things harder to find. And this is really hard discipline in observability is be willing to to to throw things away or or take the alerts off or adjust your thresholds. And one of my least favorite things about these big systems, as I've just mentioned, we have a bajillion dashboards spread across Graphana, um, Data Dog, Sentry, Pingdom, tons more. And so, finding the right thing to

do is often a pain point So, a little bit on our tool chain. Um, it's not a product demo, but I do love these services. Uh, let me touch on these. Uh, Sentry, we use for error tracking and performance monitoring. uh it knows when the code breaks and it gives us nice stack traces. Data dog for infrastructure metrics also for APM uh it's like a big broad

system overview for us and then graphana for uh customer dashboards. Customer for us is the community. So we we house some community projects and we want to give them access to these dashboards so that they can uh dig in when they need to and then we have a lot of uh pingdom and uptime monitoring with centuries uh and data dog synthetic checks and cron monitoring. So seeing

what users actually see from the outside and we're transitioning more to this this uh synthetics check and uh cron monitoring lately and so far it's proved very fruitful. Again though you don't want to hook up all of your alerts and spam the team. So you you want to check that and also make sure that you cut down on the overlap. You don't want um five tools doing

the same things. You each tool should earn its place. um one of them should go maybe if if two tools are doing the same thing. So we've we've talked about the tools and alert and noise. Um and now I want to talk about the like most powerful observability system we have not that we should rely on on people but one that no vendor sells and no dashboard

can replace and that's community. Sometimes the first signal is not a dashboard, an alert. It's a person. Someone opens a GitHub issue or pings us on Slack or Discord. Someone posts on uh Discourse, which is another medium that we have. Um why is Pippi acting weird? Um friction reports are like observability data. Uh but some of the most valuable observability data that you can get um because

it comes with context that no dashboard can provide from people. So a big big thing for me coming into Python is we have people that have been there for decades. uh there's a lot of knowledge to be had from people that have worked there and know why this system is done in this way and a veteran team member can look at a graph and say oh well

we had this like CDN issue last March or you know that context the human memory is not dashboardable if that's a verb it is now um nothing a runbook can capture um and this institutional knowledge is uh a great form of observability ility and it walks out the door though when when people leave. So you want to be able to document what you can also recognize that

some of the most valuable signals you can get um lives in people's heads and and people move on. Uh life changes but a good thing to do is uh create succession plans and knowledge dumps so that you can set up yourself and future team members uh for success. So um I can't do this alone. Nobody can. That's not a weakness. That's the model. Open source often runs

on shared responsibilities and shared understanding uh shared ownership. And the moment one person becomes a single point of failure for knowledge uh or for action system is then considered fragile. Uh we need more people who understand how this works and why it matters and people who are willing to step up. This might not apply if you're in a corporate setting. You've got a big budget and you

have four people on your team. You can spread the load. It's great. But for a community project like ours, um the people are are a a key pillar. All right. Um I've told you about our challenges and our mistakes and I'm now realizing that I've flown through my slides. So this is going to be really awkward on about 10 slides, but I want to invite you into

this work. So if you want to get involved, um the coordination happens in a number of places. Pyon US. Um we have um summits uh sprints, hallway conversations that turn into PRs. You can get plugged in. You can also join the discourse uh in mailing lists. Don't recommend that for getting started. Um there's a lot of back and forth there, but it's a good place to get

decisions made. And GitHub, you can just look up one of our repos and and get plugged in there. Uh where the work actually happens, do the code reviews. Uh it's a big heartbeat of of who we are and you don't need permission to show up to any of these. You just kind of show up and we're welcoming you Um Pyon US, uh I would be remiss if

I do not invite you all to this. It's in Long Beach May 13th through the 19th where we're going to have summits and development sprints, maintainer meetups, and community conversations. Uh if you ever thought about contributing to Python, whether that be the the infrastructure that we manage or a big project that you might want to work on, uh some of your favorite open source projects, their their

maintainers will be there at the uh conference during sprints and you can work side by side with them as sort of an on-ramp to get plugged in. And you don't have to be an expert. You just have to show up. So if you care about Python, if you're here, I'm hoping you do. Uh I'm um you're and you're at a Linux conference watching a talk about Python

infrastructure, show up and contribute, ask questions, file issues, be persistent, um write code, review code, write documentation. It's not really marketing. It's just a genuine invitation uh to come contribute. Really would love to have you here. Um so some principles on this monitoring what users feel, not what the systems report. Dashboards are for understanding and every alert should really be worth having people context switch or wake

up. Um, also remember to delete what no one has looked at in 90 days or used or maybe just cut the alerts off. You can still have it for later if you need to reach for it. Um, the first signal is often human, not a metric and noise is the enemy. Uh, for me especially, uh, silence is a feature. Um so Argus had 100 eyes and still

wasn't enough. You can observe everything but observing everything monitoring everything is not really observability. Uh but we have uh something that Argus did not uh some backup. So if he had someone to back him up probably wouldn't have died. Uh we have a team uh if you work at a company um or you know like me an incredible community uh people who care and show up and

notice uh when something feels wrong before any dashboard does. Um awesome. Well, I hate to say this, but we're 25 minutes into my 40minut talk. So, uh I would like to invite you for questions. If you want some deeper deeper things, we can I do have some extra things that I didn't think I would get to. Um, some cool things about one of our our data dog

uh infrastructure is this concept that uh it's called notebooks. And I'll just read this off. Data dog notebook in their own terms is a collaborative web-based document where you can pull in all the great things that Data Dog has, collaborate with other people um at the same time. It's kind of like data dog and Google Docs had a baby. So, uh here we have uh a a

a problem with uh latency and it ended up that we needed to uh just add an index and but people are able to reference this and tune the alerts better. Um and okay, yeah, I want to go back here. Awesome. Uh any questions? >> Does anyone have any questions? Yep. >> Give me one second. >> If people are interested in volunteering, where where specifically would they go

to to get started as a volunteer with the PSF? >> With the PSF, uh you could literally just show up if you are interested in Django. Python.org runs on Django, but there's also tons of other things uh like CPython. you could get plugged in, but it's a bigger surface. So, there's different modules in in CPython, the things that you might import from the standard library. So, you

could be an expert in that or if you just want to write documentation or you want to hook into the infrastructure side and and help us with our builds and uh observability there. There are lots of places you can get plugged in. Uh I would say go to GitHub um specifically to to to get plugged in. Yeah. uh any projects uh recommended for getting started. I think

um one of the best would be maybe python.org. And I'm not being biased here, but I would I would really appreciate the help, but it's it's not a very complicated app. It's just a big Django app and and there's different pieces you can get plugged in and you' just be interfacing directly with me uh working on on features or bug fixes or helping me figure out what

things we don't have tracked yet in metrics. Uh lots of observability features that we can can add that we don't yet have. >> Yes. What's your involvement? >> Um, so what what is the pattern for onboarding a new a new thing or an existing thing into getting metrics that don't already have them? Right. Um, I would say like the the basics we want to make sure we

do like synthetics checks, make sure that we can hit the endpoints that we need to. Um, that's that's a good start. And also like some latency tracking things that make sure you know users aren't negatively impacted. That's like a good a good baseline. >> And are you using a consistent framework? So you're using data dog. Do you have your own sort of wrapper around those tools in

order to make sure those are all provided to you in a consistent fashion given that you're a volunteer organization? Uh it it's a little bit all spread out right now. Uh again, like I said, we're hiring. So, if if someone wants to help us uh tag properly the things that we have uh because it's it's decades of infrastructure that have have maybe been rewritten or it may

be a thing that we have that's still on Ubuntu 12, not pointing any fingers. Uh, but there's lots of room for like improvement on what we actually ingest and and how we do that. >> Thanks. >> So, I just wanted to get your take. I I wanted to push back slightly on silence as a feature. Mhm. >> If you have an alert that hasn't paged anybody for

years, that's great. Unless maybe the code is no longer emitting the metric the alert is looking at or, you know, something like that. >> So, you know, how do you get confidence that your alerts will actually go off when they're supposed to? >> Oh, that's a good question. Um, I think a a good thing you can do maybe is is set your your thresholds lower than you

would expect to do like for a real alert. Maybe maybe I want to alert when my disk space is is 95% full, but to make sure that it works properly, I can set it, you know, to 50% or something like that. Same for latency or other tracking. >> Yeah. Another another approach I've seen is uh every alert should probably have a corresponding dashboard. So you can go

look at past data and you know has this come close to alerting in the past >> even if it didn't go off and then hopefully that'll mean that it's it's a viable alert. >> Uh any more questions? >> Perfect. >> So you're using a data dog and a fairly tight budget. Uh any tips for kind of balancing getting all the data you need but not exploding your

bill? >> Um I have no context of this. So I I know of people that use Data Dog and they have hundreds of thousands of dollars per month bills in Data Dog, but uh Data Dog and Sentry give us their services for free as a nonprofit. So it's so so one of the big things I was talking to someone uh about this earlier is that our current

infrastructure we don't really have tuning for um uh um billing. It's just like oh we get this for free so we can just dump everything into it. And that's great if you have the money for it or if you're like us you get it for free. But something that I would like to work towards is uh mind being mindful about our our setup and optimizing so that

we're not you know if if someone ever decides like oh we can't sponsor you anymore that's going to be a huge bill. So I want to get ahead of that uh and this something that the future infrastructure engineer that we hire for uh will definitely have to work on. >> Any more? >> Anyone else? Okay, perfect. Well, thank you all. Uh, I'm very sorry that I did

not meet my 40 minute time, but I tried my best. The um the the idea of one minute per slide is not true if you're very nervous and talking super fast. So, >> all right, let's give one more round of Thank you so much, Jacob. Anybody? Yeah, not too much stack. Hello. Hello for me. Is that all right? That sounds better. I think I'm good. Appreciate it.

>> Nice to meet you, Philip. I'm >> XP Paul. Hello. How are you guys? Are you guys ready? Are you guys excited? Okay, so today we have XP Paul Vhill who's going to be talking about migrating to open telemetry. So give a round of applause to XPaul. >> Okay, take it away. >> Thank you very much. >> I'm ex Paul Vhill. I'm currently the director of DevOps

engineering at KEXY. I have been with KREX for about a year and a half. I got started in software development and open source technology way back in the late 1900s uh while I was still an undergrad here at Caltech and I took a part-time summer job typing for a developer who had repetitive stress injury at the dot incubator here in Pasadena called Idea Lab. Over the last

few decades, I've worked for small and scrappy startups for growth stage aspiring unicorns, midsize solid businesses, and well-known behemoths, some of which are mentioned here. I've had the privilege to manage tens of thousands of physical servers, designed and grown data center infrastructure from the ground up, supporting billions of page views, hundreds of millions of unique users a month, and dynamic support for dozens to hundreds of unique

applications at the different uh companies that I've been a part of in a variety of different languages. Over all that time, I've had a ton of different roles, a ton of different titles. I don't really care what you call me. I've been doing the same thing for a long time. DevOps, sat they're all I think when I was at Yahoo, I was a senior technical Yahoo and

that was probably the best title that I ever had. Uh at KRExy, our software, uh I'm sorry, Kraxy is the commercial real estate exchange incorporated. it. Uh we are a commercial real estate transaction platform. We empower millions of users per month to close more than $600 billion in deals and over 7 trillion in property value since 2015. Today I'm going to share with you an amazing tale

of an intrepid hero, a merry band of engineers and their many mighty feats together. This is not a getting started tutorial. This is not a vendor comparison tale. It's not even a we nailed it like success story. Raw raw. This is a story of a messy realworld production migration warts and Some of you may be thinking does it got any sports in it? Are you kidding? Are

you kidding me? It's got logging. It's got tracing. There's metrics wrangling. We've got dozens to hundreds of dashboards and monitor. We've got vendor lock in and revenge. There's servers and containers, functions, and apps, databases, and cloud services. There's even budgets and deadlines. True love. Okay. Well, this is not a kissing book, but it does have a poignant look back at the lessons we all learned along the

And if you're here, you all probably already know that open telemetry is an open- source standard for instrumenting code and transmitting data to observability backends. In the olden days, aka before my time at Krexy, there were at least at least two previous observability platforms in use and to move to each platform to each new vendor. We had to or they had to uh develop a new set

of proprietary interfaces, roll it out to the different applications, do whatever customizations. So, let's say that we're on the third iteration here, possibly, maybe more. I don't know. The company's been around for 10 years. Rumor had it we had to get off the old platform in a rush and somehow got it done in under a month. And if you believe that, I've heard that there's some prime

oceanfront property in New Mexico you might be interested in. In the newest platform, we already had an hotel interface available to us and we had some interest already in sidestaping the vendor lock in that goes along with that. But our mighty team was very small and many competing priorities. In fact, at the time our tale begins, we had just built some new vendor um integrations. As time

went on, I was with the company for a few months. A few months later, over the course of about five months, we noticed that our observability spend had more than doubled. Not because we were being careless, but because we were growing. We were adding ingestion pipelines. We were adding real user monitoring. We had compute clusters coming online triggering per instance monthly fees. We had log, noisy, verbose

services that frankly were doing exactly what they were supposed to do, generating a lot of Upwards of billions of events and terabytes of data per month. And here's the thing about this kind of pricing pressure. It doesn't just cost money. It costs decisions as well. Engineers start asking, "Should I add this log line?" before they ask, "Do I need this log line?" you get a little bit

of observability anxiety, the fear that understanding your system better will trigger a billing event. That's the opposite of what you want. We had several instances where the developers were trying out enabling new integrations, trying to gain introspection, and my team's instinctive response was concern for the increase in cost. We started huddling to try to see which levers we could pull, to tweak our retention policies, to offload

data to cold storage, to rehydrate it later, anywhere we could shave a few nickels and dimes off of the the bill if we really needed it. And we started looking at our upcoming contract renewal with anxiety. There had already by the time I joined the company been an effort to add open telemetry to some of the new services that they were um playing with. One developer had

started a local development project wanted to deploy into a shared new a shared dev environment. I remember pouring through the open telemetry io the vendor documentation trying to understand how to enable the hotel endpoint how to figure out which protocol he had been using how to accommodate this new configuration in our pre-existing infrastructure as code standardized self-service orchestration framework another one of my teammates deployed a pipeline

with an additional an initial custom log forwarding serverless function because they didn't have the support that they needed to forward the events into our vendors platform. Uh by the next iteration of that one, we had deployed an a persistent vendor specific log forwarding service in each of the VPCs in order to capture that need. uh we had uh partly out of curiosity, partly as a hedge, partly

because we knew it was ultimately the right way forward and we have a compelling business motivation in terms of the costs. And we found in the experimental stages that the open telemetry SDK the the only things that you need to change to swap the backends are the endpoint and in our case we had uh some off headers that were different between our old vendor and the the

one that we are about to start uh talking about. Everything else stayed the same. Everything traveled with us. It was this is the fundamental promise of open telemetry. When you actually see it work in practice, maybe it will change how you think about your vendor relationships. You can stop thinking about the platforms that you're locked into and think about the backends as pluggable infrastructure, making a migration

conversation So we identified this new vendor an EB an eBPF uh based observability platform with a bring your own cloud model. Uh the last part is important. Uh they would spin up the ingestion infrastructure inside an AWS account in our own organization. So, we own the data. We own the storage. We're not shipping our telemetry out to someone else's SAS endpoint and hoping it stayed there. And

I think this is maybe worth pausing on. One of the quieter costs of traditional SAS, I guess the last 10 years of SAS observability is data sovereignty. You ownership of your traces, your logs, your metrics. they live in someone else's house, under someone else's policies, under someone else's security posture. Owning your own observability data is philosophically consistent with owning your own infrastructure. And it's practically important when

you have your security team, your compliance team asking questions about where's our data, how do we know that there's nothing leaking in our logs, etc. Another benefit of the BYOC model is that all of your metered usage costs, your storage, your network, your compute. This all becomes more transparent and ultimately more efficient because there's no middleman taking a percentage or adding a little bit of profit on

you can actually see what it's actually costing you to host your data. See what it's actually costing you to consume that data and receive that data. And of course, since all of these metered costs are now back in house, they also factor back into any savings plans or volume discounts you might have or any other cloud incentives that may be available to you. Okay, so we had

some exposure to the hotel technology. We had a possible business motivation to switch vendors working as kind of a forcing function, but uh we still needed to make sure that the vendor was actually going to be a good fit. And there was some reason to think that it might not be. We are not a Kubernetes shop, at least not yet. And this particular vendor's target demographic is

primarily Kate's customers clusters and workloads. the other claim to fame that they have the leveraging of the extended Berkeley packet filtering. Uh it works well if you're running directly on Linux but not so much when you're running on um ECS Fargate serverless. Uh we also had u several missionritical Microsoft Windows instances where the ebpf is still very experimentally available but not quite where we would have needed

it to be to be able to leverage that. So I started with a short list of the things that I did think that we would be able to do. The first of which is proof that we could actually port an existing hotel implementation over seamlessly. We had a recently built uh prototype for a a vector database service that was already using the open telemetry libraries. We had

a self-service orchestration that enabled it. We were able to update the self-service YAML specification, change the um the old vendor service name to the new vendor service name and add the authentication headers and magically redeployed. And it worked. It was amazing. We were so impressed. Uh I I was actually a little concerned in the middle of it. we had like a wrapper class around the hotel initializer

and I wasn't sure if if just adding anel exporters header was going to pass all the way through but there was no there was no challenges no friction uh everything worked out of the box no code changes required then I needed proof that we could actually port over our container logs a huge amount of our data was coming directly from our apps and I didn't want to

have to redeploy the apps in order to test this PC. I wanted to see what I could do to ship log data into the platform. And luckily, they had u docs to use fire hose. This is not the most costefficient way of doing it, but I set up a fire subscription to the e the cloudatch log groups and uh shipped the data through that to the hotel

endpoint. we needed to see the data, make sure that we would be able to get logs into the system and um uh we need to get the signal first and tune the delivery second. So, we weren't worried about what that looked like. We needed to make sure we could get our Windows metrics via the open telemetry uh collector, the the there's a a package you can download

off of the site. And uh we had to tune the configurations for the Windows specific parameters to collect the host metrics to collect uh enough enough off of the system to show that this was a potentially viable path for us. Uh the Windows receiver configs are a real test of how complete our implementation would be. And then we needed to get those traces and spans working for

the real user monitoring. uh I had to pull in one of our front-end developers to help us evaluate whether or not that was actually going to be viable. Uh we it was relatively straightforward. It did require a a vendor specific package, but the old uh run was also a vendor specific package. we had some minor issues with ensuring that we had uh the encryption and security and

the authentication uh pieces working, but those all got ironed out pretty quickly. Uh this uh was as it turns out uh an incredibly naive approach to this particular PC, but it was an initial baseline to see if this was even a possibility for us to consider it because I I I did have quite a bit of skepticism towards it. Uh the proof of concept in calendar time

took maybe about 10 weeks. um the vendor side of the setup, uh provisioning the BYOC environment, uh standing up the ingestion endpoints, wiring the collectors, all of that was fast. It was impressively fast. Like it took me longer to get the account provisioned uh over the weekend with the necessary access controls. Uh I handed off to them the IM roll on Monday morning and it was ready

to go by noon. I was like, what? So the delays that we had to make that draw out to full 10 weeks was on our side which is worth admitting plainly because I think teams often blame vendors for PC delays when the real bottleneck is internal prioritization. I think the first two weeks uh were almost up before I ever spent any significant time with the documentation or

made the real first attempt to get logs into their system. And after the next couple of oneweek extensions, because that's kind of how uh worked, uh I could start to kind of see the path forwards. I knew this was going to be a possibility. And so I brought more of my team in for a consultation on this and I got an earful. In my fixation on seeing

if the open telemetry path was available to us, I had neglected a whole list of other considerations. One engineer told me that if we were going to be leaving a premium platform, we should just roll roll our own visualization front end and skip having a vendor altogether. Uh, another one told me it was a perfectly acceptable increase to just pay your observability vendor on a sliding scale

at yeah 10% of your overall cloud cost. That sounds like a good number. As if. I have to say I was tempted by the first suggestion. I'm an old school guy. I remember configuring SNMP dammons, oids by hand, getting discio metrics to my MRTG servers. I have nostalgia for ngiosk for cacti. These are things I could see myself running my own observability but we're running about 15

developers per S and our crew and core infra core infrastructure services require additional care and feeding beyond what our normal application load requires for our commercial real estate customers. And we're swamped as it is. And it's nice to have somebody else responsible for some of the baseline services, all the bells and whistles that come with it too. Uh I did incorporate team feedback for the remainder of

the PC at that point, leveraging the demands uh to obtain further vendor engagement. So we were we exported monitors to JSON, handed the JSON to the vendor and let them do the implementation of our monitors on their system. Um uh we were we had a lot of interaction with them to massage the uh alerting and notification templates. Not not really open telemetry related stuff just part of

the um larger uh big bigger picture of a vendor migration which I think is um a a key part of any consideration of moving to open telemetry. Our vendor was uh scrappy, young, and hungry, and responsive to all of our bug reports and feature requests. And uh by the end of the PC, we had a pretty clear picture of what worked and what didn't. Uh some of

the service integrations that we relied on were not yet available on their platform. Um there was some synthetic monitoring that was unavailable on the new platform that we kept on the old platform for the time being. We had some other limitations that were relatively minor. Uh I do want to name uh this directly because I think it's important at the time of the final PC demo. I

felt like I was still getting some pretty serious push back from my senior engineers. Uh and their concerns about feature parody were legitimate. These guys are smart guys. I totally respect them. They weren't like just being resistant to change and stuck in their ways or anything like that. They've they lived through incidents. They know which tools, which dashboards are important. They know what they rely on when

things go wrong. And this put me in a bit of a bind because I had to make a judgment that overrode the recommendation of my engineers. And I am accustomed to leading by consensus. I'm accustomed to like we all power out. We decided this thing. And here I felt like I was I I had to make an executive decision that was a little uncomfortable. I had to

I wasn't 100% certain, but my instinct was that the cost trajectory that we were on made this decision urgent in a way that didn't give us the luxury of waiting for full consensus. I had the support of my own leadership and I had to trust my instincts. And this was an opportunity for us to improve our overall posture in the long run. And this also marks for

me a milestone in my leadership journey. I've I've made executive decisions before, but this one felt a little bolder. So after back and forth, after mitigation planning, after documenting the gaps and committing to a path to close them, we uh signed the contract. Uh and that's when the work real work began. So about this time we had some staff turnover. Uh you never expect the Spanish Inquisition.

One guy left, another came on board. He helped with some of the migration. Then he left a few weeks later. There were other departments in the company that were seeing sim similar behaviors. Very weird, not at all disruptive. And I mentioned this not to complain, but because I want to be realistic about the conditions under which migrations actually happen. They don't happen in ideal circumstances. They happen

when you're losing people, when you're hiring people, when you're fighting fires, when you're maintaining the system you're trying to migrate away from simultaneously. We poured through the company's GitHub or looking for all references for our soontobe former vendor and we found existing references in the codebases to the previous vendor that existed before this one. We're like uh we knew we had dozens of these uh often net

apps that basically all needed to be upgraded simultaneously as part of our monolithic monor repo. they all shared the same orchestration code. So if we made a change to swap out the container uh sidecars or the layers in between them, we would end up with a mess. We needed additional developer time to help uh deploy the zero code auto instrumentation within the um service tasks. Uh we

had there were issues. We had it the logs going to two places at the same time. We had duplicate logs showing up in the new system. We had sensors deployed to the Linux hosts. We built packages around the Windows services so that we would be able to have some restartability on the services when they went down or when they were disrupted for some reason or on a

reboot. We ran into discrepancies on metrics nomenclature where our old vendor monitors were dependent on a particular u name for the metric coming off of the host and the default hotel names were different. One of the guys put some extra thought into our expected deployment strategy and decided rather than shipping straight from the apps to the vendor managed OTEL endpoint uh we would actually deploy the sidecars

again and I ultimately agreed with this decision but it did mean us deploying our own custom sidecar based on the open- source open telemetry container that exists. We no longer needed to add the off headers into the container apps. We just shipped directly to the sidecar with a unoff hotel and then let the uh collector serve as an aggregator to ship it to the vendor endpoint. We

ran into miscellaneous performance bottlenecks, scaling issues, etc. Uh we belatedly implemented SSO and folks ended up in the wrong workplace. things go things go wrong and we recover from them. The most urgent objective after signing the contract of course was reducing spend on the old platform and it took us actually about six weeks uh before we started seeing the decline uh in the spend uh and it

dropped even more precipitously about a month after that but it took us time to get the roll out done. uh even today I think we still have some residual services that never migrated off the old platform. But even with those costs plus the metered usage on the new account plus the flat licensing fee that cost amvertised over a per month spend, we're actually still under what we

were spending uh before. And we're able to onboard more integrations to more services without worrying about oh it's going to cost us like so much more to add more logs. So, our migration took longer, involved real disagreement, real uncertainty, real loss of team members at inopportune times. Uh, it involved making a call that I wasn't 100% sure I was right about and sitting with that uncertainty while

we built the evidence to confirm or refute it. Um, I can tell you now that we're running at a fraction of the cost that we were a year ago. uh our telemetry data lives in infrastructure that we own. We have developers who are invested in and improving their own and we're not locked into any vendor and in a way that prevents us from making a decision if

the right reasons arise and of course the portability is what open telemetry gets you. uh not just a standard protocol, not just SDK compatibility, it's genuine ability to treat your observability backend as a choice you made rather than a trap you're in. Um the migration wasn't easy. the architecture. Uh it led us to um owned data vendor portable instrumentation self-service um uh integrations that still work with

the um stack deployments that we have. This is an extra an architecture that I uh definitely feel comfortable with um and would build again from scratch if I had and I've reached the end of my dialogue. So, I hope some piece of this story has been useful to you or interesting at least. Uh, I want to thank my team. They did the lion's share of the work.

Um, I want to thank the Krexy leadership including my VP of systems engineering who is an organizer here at scale, Chris Smith. Um, I want to thank the Foothills Community Toastmasters Club for the leadership and public speaking opportunities that I've been doing for the last couple years. I want to thank Palumi for our self-service infrastructure. I want to thank the wonderful team at our new observability vendor

uh, ground cover who have supported our journey to open telemetry so thoroughly. I want to thank all the people at open telemetry for providing resources that allowed us to conduct a crash course in self-education and to all of you and the rest of the open source community. Thank you. Are there that be different? >> So the question was if you you have dozens of NET applications with

zero instrumentation, zero observability. What would I prioritize as a DevOps versus as a software engineer? That is a great question. I think if you have no visibility at all, you need to identify what are the most important things. So, uh, if it's a web service and you have a dependency on your business model for page views and impressions, you would want to make sure you've got some

visibility into the amount of traffic that you have coming into the system. And I would, you know, prioritize that. Uh, probably very specific to whatever your your use case is. Um, I will say that, um, in my experience, um, where I care most about observability tends to be performance and knowing when we've when we've gone off the rails from status 200 to status 500, when we've got

database uh, CPU time at at 100%, you know, maxed out kind of thing. So, I I would make sure that you kind of have those basics first. uh and then focus on what serves the business needs. >> I'm an ops guy, so yes, I I don't want to be woken up. I don't want to be surprised. I don't want to have to be working six hours late

tonight trying to figure out what the heck happened. I want to have that that data available to me uh if and when a problem arises. You're welcome. Any other questions? Um in the hat. >> So where we are now uh we have a number of services that are sort of residual in the old platform. We have uh reached out to the developers. We again we need time.

We have to collaborate. S DevOps folks cannot instrument all of your code for you. We we cannot write all of your unit tests. We cannot design your uh software architecture. We can help implement and and provide the the tools and the basic standard patterns for uh the developers to do their deployment so that they can have that the introspection that they need. where we have created traces

um um in our platform uh to track uh all the way through the system when a user has uh an open session through the web front end all the way through to the databases. Um not all of the trace IDs make it all the way through. Uh that's uh that gets a little bit of attention. Um, in the uh past several months, I think we've been averaging

about one uh sprint demo per month where one of our developers has uh brought up something new that they discovered about working with the platform uh and some new trick or new technique for achieving um visibility into the per performance characteristics of the of the application. We have some very um um high value uh transactions uh when real estate auctions are closing for example and like we

have to we have to keep those services running uh even when there's a ton of traffic coming at them with people refreshing constantly to see what the the latest thing is. So there's a huge need for visibility and transparency in uh in our platform and anywhere that we can facilitate that is a high priority for us. >> Uh I think I saw you had a question. hotel.

>> Right. So in our case, we have some data going. We uh I think anyone who's been doing this long enough ends up with multiple dashboards and mo multiple single panes of glass to see what's going on in the system. Um the idea behind open telemetry as a sort of like uniform standard for how to port between those different ones uh I think is what gives its

uh gives it gives it its um um I don't know power majesty the the excitement of open telemetry at this point is that you don't have that vendor lock in and when it comes time that there's some other catastrophic outage or you your site has grown big enough that you need to be cloud agnostic and failing in between then you would have more uh portability in in

that case. Um but uh I I can't say that I um I don't know that cloudatch has the uh same level of well I actually I'm not familiar with X-ray either. So, so in terms of traceability and spans and the ability to to to track the user the user session all the way in, It's It's better for you. >> Well, good luck to you and I look

forward to hearing if if you have successes. There you go. There you go. Uh, I think I saw a question over here. uh so the question was is there a risk of having the control plane separate from the data storage plane and the why do we need a separate control plane in the first place. In this particular vendor's model, the u the bring your own cloud model

leverages a multi-tenant solution where they have a singular SAS uh UI service uh that reaches back to the data sources that live in the customer accounts uh separate. So it's a it's a single um software as a service front end uh but your tenant uh data is stored um in your own uh VP in your own VPC in your own account that does not belong to them

that you that you share with them. Does that answer the question? >> The control plane is the UI. It's it is the the front end to the entire observability platform for them. um the customer cloud in this case this is just the VPC that is containing the hotel endpoint uh the um telemetry databases the uh storage backends etc. Uh and then that that is part of a

larger uh customer cloud of ours that includes our actual production um development Uh no. So the open telemetry endpoint lives inside the uh customer cloud um private data plane. Uh it is exposed as a public interface. Uh so there's encryption and authentication that goes along with it. But the events and the metrics and telemetry being generated from our actual um product uh services are being shipped uh

directly to that endpoint and the control plane is really only the front end. This is not a vendor the the vendor did not provide this diagram. I I I realize now that I didn't uh provide an attribute for it, but I'm pretty sure I found it like on a medium.com article. Yes, sir. so uh the question was how do we get to AI SREs uh given the

current architecture that we have basically I don't know that I believe that we're going to get to an AI S sur certainly having context knowing how how the architecture is designed knowing have having introspection and ability to reach all of these things. Uh there is uh MCP servers available for the vendor so that you can query into your observability data from your um agents. uh I I

we certainly leverage uh AI throughout the process for our own uh poll requests and code reviews uh as we're making changes to our self-s surface infrastructure as we're making changes to the uh applications themselves. Um but the overall architecture and design I mean I I you c you can ask AI to give you a project plan right and then it will give you a an outline and

a structure that may or may not be relevant to your precise needs and then you have to uh evolve it. We we currently at least tend to think of our agents as junior devs that we're trying to train and hopefully they will get better, but they need a lot of supervision and I I imagine that will be the case for the foreseeable future. Did that answer your

question? Not at all. Not in the least. I didn't commit to the >> uh how we get to an AI SRE from uh analysis I I don't I don't know that I'm qualified to answer that question. I think it's great it's great to have these conversations. I think it's important uh for all of us who are in this industry and I'm sort of assuming that you're in

the industry as well to be asking relevant questions like this. uh so as a reiteration I How do we get to a full um artificial intell artificially intelligent uh site reliability engineer uh given where we are now? Given that we have MCP servers, given that there is some amount of context that's required and domain knowledge in order to make the leap to having a fully autonomous uh

agent who's able to manage your infrastructure and troubleshoot your problems and plan your projects and do your data migrations. The question is how do we get there from where we are now? And uh I have not personally done the thought exercises. Uh about uh 15 years ago I came across an IBM paper on the road to autonomous infrastructure and it laid out like a five-step plan for

developing a an a data center that was manageable in the sense that it had defined services. that had defined uh mechanisms and actions that could be done upon those services to increase, scale out or scale down that you would have um some sort of a programmability component to it where you're actually able to interface with all of those actions and and be able to programmatically make those

decisions. moving to the next stage where you're able to have metrics around the kinds of inputs that go into making a decision to scale up or scale down and so on and so forth until you get to building the whole stack where it's able to detect for itself. Oh, I need more servers. Let me place a procurement order and have somebody ship me some more servers. Somebody

still going to physically, oh, now a robot is going to physically deploy the server into the data center. And now the robot is going to manage the power and we're going to be monitoring temperature concerns and u the observability of the infrastructure informs the AI's ability to uh self-provision and to self-regulate the services that are needed within it. I imagine that you can probably extrapolate that to

this next set of uh challenges in terms of trying to reach the kind of goal that you're talking >> That's a great question. So the question is has this journey to open telemetry impacted my uh view or my ability to do service discovery uh and service registry and uh the kinds of orchestration um components that go to detecting the presence of services in the environment. And I

would say that that for us uh still has actually literally been on my mind recently because we have self-service provisioning for our developers. So they can create whatever stacks they need with limited sets of profiles of the kinds of services that they can provision. they're they're allowed to provision a self-contained database or a memory um cache or an EC2 an ECS uh container and they can specify

their own set of parameters and we on the operations team don't necessarily have any visibility or awareness of those services possibly until they've already gone into production and so obviously we are interested in making sure that we have kind of a standard set of profiles for all of our services and a standard level of quality for all of our services. Uh I won't say that this particular

migration has uh moved the needle on that. Uh I think that that still belongs to our orchestration >> Let me let me come to you. Sorry. >> Sorry. I actually have two questions. Uh the first one is is do you really feel nostalgic about Nagios? >> Do I really feel nostalgic about Nagios? I think that there's something interesting about the human brain that has memories and when

it dwells upon things from long ago, there is a certain amount of pleasure that goes along with it. I think it would be fatuous to say that I did not on some level have some amount of nostalgia for those good old days. >> Uh and then the the second one kind of ties into that last question. uh which was in the early part of the talk you

talked about getting buyin from uh your engineering staff uh and after this kind of process completed did they feel like the kind of concerns or the objections that they had initially either bore out or did they feel like what we actually got in the end was worth the the transition and they were still able to kind of do those do those kind of That's a great question.

I almost put some of that into the presentation and I kind of decided against it. So many of us are familiar with the idea of the bastard operators from hell, the cynical system admin who is grumpy about everything that happens. And I didn't want to portray my engineers in that in that light. There's a certain truth to those kinds of stereotypes. And I'm not saying that that's

necessarily the case here. Uh I will say that the the team uh focuses on collaboration and teamwork. They're they are pleasant to work with. They do work uh willingly and happily. They're not like trying to sabotage it or anything like that. Uh that does not prevent them from offering snark. And um I think the comments that I've heard the most often was uh they feel like they're

the QA team for the vendor that we've migrated to. Uh uh I think in the course of uh six or nine months maybe we've filed 45 uh tickets with the vendor. Um again we're not their typical use case. So, um I don't think that's a a huge um um amount. Um uh and I don't I don't I think uh as I mentioned uh at the very beginning,

this company has had at least two observability platforms before this migration began onto the fourth one. And I'm I feel reasonably confident these guys who have been with the company since the beginning are familiar with with the arbitrary need to move to a new observability platform. I feel the same way about ticketing systems. Like please God, don't send me to another ticketing system. I don't care which

one it is. Whatever we have now, let's just leave it at that. Um but you know, there's something nice about a green field. This is great. I really appreciate all these questions. I I had no idea we would have any kind of dialogue. This is awesome. Joel, >> we're using Azure and you know the monitor that has built-in logging and it's expensive. So like limit the time.

Were you using that in this savings or were you already out of that into something else? >> like were you did you have your own I guess this because I see here you're using your own VPC your database so you're obviously not using uh like AWS is monitoring stuff here or are you >> uh so again we do have some of the metrics that are being gathered

through um uh the Amazon the cloud services uh monitoring service um we do have some monitors that are that have been conveniently configured there and they will continue to live there and we will regularly see the last seven or 14 days of um metrics in the system for those. >> Um but we also had the third party premium class uh observability platform with all the bells and

whistles and you know a completely different pane of glass with a different kind of purpose and intent. Um more for the developers and I think largely oriented around a uh a happier user interface for the logs. >> So you went from that one to here. >> Uh yes. And so in this one we have the the it's a different it's another different visualization front end but our

back end is now portable. >> More portable I should say. >> Cool. I think I saw another hand go. All >> Wow. >> So, what was your biggest decision point with picking a vendor like um versus open source like dealing with a Graphana or something like that? >> Um I'm lazy. That's uh it's nice to have uh vendors that offer bells and whistles and like their own

set of monitoring. Uh once you have a monitoring system in the old days of naggios, now you have to have a ngagios that monitors the naggios and you end up with this sort of like well how how many turtles down do we go? Uh there's a a a higher level of uh expectation of kind of service isolation for the core infrastructure service monitoring that has to be

kind of separate and it's nice to be able to offload that to someone else. I think also the feature set of the vendors the if you've if you've been on any of the big observability platforms they have all kinds of uh fun expensive semantic analysis and u uh u UI features within the you know graph zooming and updating the whole page all at once and there's there's

all kind of >> nice to have things if you premium >> So, we got someone in the back. >> Hi. Hello. So, uh open telemetry comes with its own set of semantic conventions related to how you should name various attributes and spans or how metrics are supposed to be named. For example, uh when you do migrate to these things, how often do you uh see yourself violating

some of these conventions or thinking that oh this doesn't make any sense? H I don't know that I so so the main the main uh conflict that we encountered was based on our previous vendor uh tagged the the names of the metrics with like system CPU instead of host CPU. I don't remember what the specific uh difference was. Um, for us, the inconvenience of having to update

all of our either update all of our monitors, which are looking for one or the other, or modify all of our default uh open telemetry config uh files to specify to to rename those metrics by some other name. Um, that was kind of the biggest inconvenience. Uh I can't say that I know too much about uh the uh spans and traces because I generally defer that to

our developers. so I would I would have to ask them if that has been a problem for them. I it hasn't specifically been a problem for >> Thank you. >> Does anyone else have a question? Okay, let's give a round of applause to XP Paul. Thank you so much, XP Paul. >> Thank you guys. Really appreciate it. >> Thanks. Thanks for the questions. That was awesome. Ow.

Wow. Wow. Okay. Hey, there we go. >> You want tape? >> No, my skin's so oily. It'll come right But thank you. >> No, this is hooking real good. >> Okay. No, it's I'm good. I'm fine. Thank >> Oh, on the pack. >> I mean, 80 80% maybe. >> I'll check. >> Okay. Thank you. I mean, yeah. No, it's about it's about Yeah. 35 minutes and then

there's a demo in the middle. >> Yeah. And then we'll do questions at the >> All right. Thanks a lot. >> Matt. >> Yeah. Joel. >> Phip. >> Phillip. Okay. Thank you, Philip. interview. >> No, I'm looking at some platform stuff, It's pretty rare. >> Yeah, >> a lot of platform. >> for sure. >> A lot of people don't know what S is, >> yeah they are.

Yeah they are. >> What was your name? Shannon. Nice to meet you. you know, People have questions in the middle, that's okay. You know, we'll do a main Q&A at the end, but if someone has something they want to ask in the middle, that's totally fine, too. >> Yeah, you're going to introduce me. >> Perfect. Thank you so much. Okay. Hello. How are you guys? My name

is Shannon. Hope you guys are doing good and having a great day. So, today we have Matt Davis here. He's going to be talking about metrics as music and open-source symphony. So, let's give a round of applause to Dave. Oh, Matt, sorry. >> Thank you. Thank you very much. Hello everyone. Uh, welcome. Today I want to show you an avantguard way to observe the system. This talk

is about an application called Montever, which is a composer from the 1600s. The app pulls metrics in key value formats from endpoints that can be configured dynamically. So as it's running, you can add more endpoints or remove them. Then it produces patterns from configured maximums and those patterns can be played as music. So Montever is open source and it leans on open source. Even the synth has

open- source hardware and a lot of DIY in it. So today we're going to try to understand a little bit about how systems can be observed and I'm going to show you a little bit about how AI used in a joint cognitive system can enhance our work. So don't worry, we're not going to leave the synth playing until the end. We'll get to that. Um but first

I do want to talk about systems. So there's the story that systems thinker Russell Akoff told in his lectures that I think is relevant to how we think about observability in SR. We are often presented with parts of a system and told to monitor those parts. So here's how the story goes. Imagine that you have a team of expert engineers and their job is to collect the

bestin-class pieces from every automobile on Earth. every good automobile, every good steering wheel, the best brakes, the best shocks, the best engine, the best transmission, the best of all of those things from every car there ever was. And they put it all in one room. And now you tell your engineers, "Okay, now make me the best possible car from all of these parts." Well, the engineers come

back to you and you know what they say? They say, "Well, we can't do it." And and I ask, "Why? Why can't you do it?" Well, the parts don't fit. this is our challenge in observability, seeing our system through the haze of all those pieces. Um, there are two definitions of observability up here on the slide and Montever is of the second definition. So, this talk is

not about changing the way that we observe systems. This talk is about ways to enhance our understanding of the system by finding new dimensions of witnessing it. So the app I'm going to demo today is just one method of seeing differently. Understanding the system as a set of signals experienced from a completely different perspective. I want everyone to look up at this ceiling right now. See this

gritted ceiling? Um, when I was in high school, we had one of these gritted ceilings on our ceiling, but it had the actual tiles and, you know, it's not like this. So, it was one of those drop ceilings that had the grid of square tiles in it. And the art teacher decided that he wanted to do a project to paint this painting on the ceiling. So, every

student in the art class got a piece of the ceiling. We all took home one of the 2 by two squares that comes out of the drop ceiling. And I remember where mine was. Right here where the blue rectangle is over to this side. And our teacher told us a trick. He told us to help ease the expectation of not being able to draw a thing, we're

going to turn it up upside down. So, we turned my piece of the painting upside down. And then I turned the piece of the ceiling upside down. And now we could copy it. So instead of trying to draw two incomplete people, I'm just drawing shapes. It's just a different way to kind of turn the system over. When I was in music school, I was given Leonard Meyer's

musical analysis technique to understand and research. What he did is he successfully brought together two things. this Gestalt theory of psychology which is focuses on like how we how we uh inter how we interfere patterns how we um receive patterns in our brains, how we perceive them. Um and he took that and he took it as a model to measure tension and release in music regardless of

the style. And he called this the law of good continuation. So in music good continuation is written into our pattern recognition pathways. You think about chord progressions in rock music or jazz music. Those are tugging on the law. You want to get back to the dominant chord, right? Uh we want to keep the good continuation vibes going. Our brain does. We need that last note of the

melody to end where our brain thinks it's going to end. So what Leonard Meyer did is he took some terms from poetry and he choose he used those terms to describe the music. This is what we're looking at up here on the slide. The bar, if you can look at the top, you don't have to read music to understand this, by the way. Just look at the

bars and the little like parentheses that look like they fell over. So, that's an accent followed by a non-acent. The bar is the accent. And in his musical theory um technique, they mean the same thing as they mean in poetry. They mean accents and non-acents in the poetry. Um, so accents are where the law of contin of continuity is challenged. When an accent happens, our brains want

the law of continuity to bring to being bring us back to the ground. And you can tell in these two pictures how there's different levels of those accents. Um, they happen in hierarchies from individual notes to like entire phrases and entire operas. Anyone ever seen this line? Just raise your hand if if you know this line. Couple of people. Okay. All right. Anyone want to volunteer to

try to say Go ahead. >> Oh, very good. I like that. Anyone else? So, I'm going to show you where the accents are in that line. That's a line. That's the first line from Richard III. You wouldn't want to start a play with a non-acent. So Shakespeare put an accent at the beginning of the sentence. That's called a trochi. Each one of these groupings is called a

foot in poetry. Now is the winter of our discontent. You were pretty close. I think you actually were spot on. Very good. In the software that we're talking about today, uh, in Monteverity, this is called a pulse. So a foot which is a a up one pattern we call a pulse in Montarity. In fact this is what um Leonard Meyer's pulses look like. This is what his

accents look like and what they were named. You can see in the Mozart example on this side on I guess at the left side those are trochy patterns. So accent non-acent accent non-acent. If you ever heard of waltz think of a waltzum bum. Those are the accents. Uh similar in the Beethoven except the accents happen in a different way So Meyer took the idea in poetry and

and applied it to music to understand the law of continuity in music. And something to remember as we think about this is that the law doesn't say that it has to stay at the ground. The law just means that our brains want to pull us back to the ground. That's what that means. into like a return state or the language that Leonard Meyer used was state of

goodness. Okay, so I was wondering about this and I was thinking I wonder if I could apply this to incident metrics because I'm an incident safety nerd and I think about this stuff all the time. And then I wondered, well, wait a minute, could I generalize it to any metric? It made sense to me that all I needed in my application was a flow of numbers, some

flow of metrics and then some way to delineate where the accidents were. Um, kind of like this. You get a little trochi on the fix big spike and then like some four little IMs, you know, little spikes going up and nothing and up and then nothing and up. We can measure these patterns as accents and I thought that was really cool. So, it happened like this. I

lost my job in June of last year. I'm still unemployed, still looking for uh SR and platform work. And I made it my personal mission to get better at programming, get better at software engineering, not just coding. Uh so first I'm going to build myself an observability tool because I'm that kind of nerd. But I wanted to challenge myself uh to make it as professional as possible.

Make it employment worthy work really like I'm working on trying to find a job is what I'm thinking about here. But secondly, I wanted to learn to use LLM's AI effectively in the kind of work that I do. and I don't get a lot of reasons to practice that when I'm unemployed. So, I kind of came up with a way to practice it, to build something. This

isn't a vibe coded. I want to make sure that's just said right up front. This is my code, but I'll show you in a little bit how I used AI to help me out. Third, can this even work? Harmonic analysis. I call it harmonic accent analysis. Does it even work? Does it make things interesting? Um, you know, sometimes late at night I was thinking to myself, this

is a dumb idea. But then when I got into this project, I was like, you know what? I'm going to make this happen. And if all of this is true, if I can pull these metrics, if I can pull these accents out of the metrics, can I play the system back as music? So, the app is called Monteverity. Montever was a late Renaissance composer. he came up

with this new idea and he helped develop this new idea called Boso Continuo. So I don't know if there's anyone in here that likes like producing electronic music or knows some Okay, a couple people. Cool. So like when you write a baseline first, you get out your 303 and you're like, I'm going to write my baseline now. That's Basso Contino. You're doing the baseline first and then

you fill in all the harmonics on top of it. This dude invented that. and he helped an entire era of music come from it called Baroque music. He recognized this himself. He was a badass. And he said, "I'm going to call this seconda practica," meaning it's a second practice of harmony coming from the Renaissance into the Baroque. This was actually heretical. The church didn't like this at

all, by the way. So why why observability? Um, you know, if we could do something, this is in my head, if we could do something like distill any output, so incident metrics or direct metrics or aggregate metrics or results from open telemetry or anything like that, we could start to form an understanding of what our good continuation is for our system, right? What is the good continuity

in our system and when is it headed for trouble? Will it let will it let us hear what's going on? This is a little preview of one of the uh UI elements of Monteverity. And you see those pulses going around the circle. And I'll tell what that means in a little bit. But when I began seeing these kinds of patterns emerge as different parts of the system

were put together, I began to see how this could be an opportunity to get something else that we want, which is more information for less data for more periods of time, which also costs less, by the way. Okay. So, I'm going to turn on the music now. Turn it up. See? Can you hear my voice? Okay. Over that. Awesome. Actually, let's dub that out a little Okay.

So, what are we listening to? We're listening to this app. It's running right now on this laptop um behind the presentation software. And this is kind of what it consists of. It's a Golang app. It has an extendable plug-in architecture. So I can pull in different kinds of metrics like floating point integers even exponential values which are very very common in stuff like Prometheus and then I

can take those in I can transform them and then put them up on a UI. This what we're looking at is a D3JS web UI. There's another terminal UI I'll show you in a little bit. So to give you kind of an overlay of what this symphony is coming from. So we're making sound with a Euro Eurorack modular format. This is this is a free and open

format. You could go build your own module from scratch using the Euro Euro format. Uh musical instrument digital interface. This is another old thing. There's actually two versions of this and we've never gotten to version two. Version one is what everybody uses. that's how the notes are getting from numbers on the computer through this interface right here into what we hear in the synthesizer. Again, non-commercial. Anyone

can use MIDI and you don't have to pay somebody for it. Here's something to think about. This is the configuration of what we're hearing. So, you've got some scales in there. I'm going to show you right here. Yeah, I've got the pentatonic scale configured. You could configure any number of these scales, but Monavity will only play the intervals. These are intervals right here between each of the

notes. Those are the black keys on the piano. So, if you look at piano and you play those five black keys, that's what a pentatonic scale is. And so that's what we're hearing right now. There's some other interesting stuff in there that I'll get to in a little bit. This is the MIDI interface that we're listening to. This is an open-source hardware MIDI interface. It's programmable. It

has Circuit Python on it, which is really, really cool. Right now, it's using a default patch. Circuit changing the commands. Again, this is where the numbers actually become MIDI nodes is in this interface in the circuit Python. So, circuit Python sees a number come in and if it go if it's a certain thing, then it'll say, okay, this is the pitch of the note and I'm going

to send this out this port. Oh, this is the beginning of the note. We call that a gate and that comes out this port. And so, when you hear those pitches change, that's what's happening. Montever is changing the pitches. Unfortunately, Winter Bloom has kind of ceased operation, but you can still go buy this or you could still go get the open source hardware specification and build this

yourself. Uh, it's fun to program other stuff on it, too. I programmed some cool LFO clock divider stuff. What are we listening to right now? Now we're listening to a Moog, not open source, but very very amendable to doing people doing clones, but this is an original Moog. And we're hearing I can even turn the the bass pitch up on the Moog and kind of change it

around. That's kind of cool, isn't it? I like that. So, imagine that you're in a room. It's a dark room and you've got all your monitors. It's a monitor wall, you know, and you're seeing all of your observability right there and all of a sudden and you have this playing and then all of a sudden the whole thing changes. You wouldn't even have to look at anything

to understand that. Um, let's go back down. One of the other things that I have working on here is another piece of open source hardware. It's the music thing modular radio music. Uh there's the URL. By the way, this slide deck is up on the scale site. So any of the URLs that I show in here of all the open source stuff, you can get in the

slides from the website. Let's turn on the music thing modular and hear what it sounds like. So that's not a pitch, is it? All I'm doing right now is I'm triggering with that gate. That's this one. I'm just triggering a sample to play. So the sample is playing at the same rhythm as that pitch is going, but it's not changing pitch like the pitch This is a

schematic of what we're So what's happening is I have another little I could point this at any number of different kinds of metrics. I could point it at a Prometheus metrics endpoint. I could point it at something that's issuing JSON blobs and pull a key out of that. Uh and I can configure the delimiter um like Prometheus metrics. I I think Prometheus has an equal sign, but

some some platforms don't have an equal sign. they just have a space and you can configure that. Uh, net data is the other one I'm thinking of that I tested with. But here I wanted to be able to have a reliable way to play. And so I I wrote my own little thing. It's called um Toad Lester. And Toad Leester creates metrics. That's all it does. It

just creates metrics and it serves them at an endpoint. And that's what we're hearing. So we're hearing Monteverity pulling the metrics from the endpoints. This isn't an actual metric we're hearing. It's just a go metric that I thought people would be familiar with. So imagine you have this metric in the Monteverity configuration. You would say at 500,000 I want to trigger an accent. And so if we

have like you know four different values in series coming from our metrics endpoint you can see this one's over 500 this one's not this one is and this one's not. Montever detects those accents. You remember the troes in the poetry and the music. That's what that's what we're hearing is processing the metrics, figuring out that it's a Trochi, and then playing it out through a plugin that's

a MIDI output plugin that is what I described before, transforming the numbers into notes. I think what I want to do is I'm going to show you a still picture of what's going on and then after we get through the slides and I take the keynote down, I I'll show you the web browser and I'll show you the live interface and the terminal UI that I'm using.

But this is basically what it's going to be. It's the the the primary rolled up in a these inter inner circles. This is 60 seconds as they go around this circle and then after 60 seconds they move out to this circle and they go around for 10 minutes. There's a way that you can see what those metrics are coming in. If you can so you can notice

toad leester is the the endpoint and two different endpoints are actually defined here. There's a randomizer and then these these uh linear metrics and these linear metrics come in as exponentials. They come in as floating points. They come in as integers and Monteverity transforms them all into integers so that it can figure out that max and then easily do the pattern recognition. This is a place where

you can dynamically I mentioned that dynamically change the configuration of the endpoint. You can see one of the that's the randomizer endpoint configured there at the top. I'm going to turn this down a little One thing that I'm just uh demonstrating here is just how well that I was able to get to a performant piece of software um by leaning a little bit on AI. And I'm

going to tell you a little bit about how I did that. Um I don't have more than one slide on resilience engineering, I promise. But it's important to think about when we're thinking about working with AI. Uh AI is right smack in the middle of this intersection of people, technology, and I employed a joint cognitive system, which is how we find success in complex uh by building

an illustration for this slide. Now, I didn't ask the LLM to build the slide. I also didn't ask the LLM to build the presentation. I asked the LLM, well, you can see what I asked it right there. I don't know if you can read that, but I just said, "Hey, I need a I need a slide graphic. This is what I want it to look like. I

think it could be done in an SVG." And then immediately, this is what I got. So, I didn't change the prompt at all after I got this image. Um, but something that was interesting and something that I learned through our joint cognitive system is that Claude can't put trademarked images. And so when I got this image back, Claude told me, I can't show you trademark images, but

I created faximiles of them. And so that's why you see these weird non- logos in this illustration. So in this way, I've learned something about the technology. I've learned something about operating in concert the AI and me in our joint cognitive system. This is the place I use the language models where they supplement and enhance my abilities as a human in doing technical tasks. Not replacing me,

but forming a joint cognitive system with me and then we get stuff done. A couple of recommendations that I have about LLM use like this. I specifically tell the LLM, when you give me code fixes, I do not want the full sheet of code. I don't want the whole file. I only want you to show me the specific code to fix so that I can go fix

it myself. I actually have those in my personal preferences in Claude. What I tend to get a more learning type of environment with Claude this way. Um, but the other thing that I do is when the LLM recommends an algorithm or like a full ass method, the number one thing I do is I test it. Here's another great open-source resource. Um, I think I think it was

about three years ago that my good friend Bobby likes Linux was helping me with some Golang interfaces actually and told me I should go learn on this Gitbook site and that's what I did. Test-driven development. I tell y'all, it opened a door for me that I didn't know existed. So, my brain greatly prefers typed languages, but I also get overwhelmed really easily. I'll get real buried in

the code and not know where to turn. Um, the tests give me a way to control that. They give me they give me a way to stop working and I love that. Uh, this kind of behavior actually leaked into other parts of my life. You know, um, when I think about test-driven development, I think about other areas in my life where I get super overwhelmed and just

thinking in that way has really helped. Um, learn go with tests. This is an excellent website. The link is right there. Uh even if you're a Go expert and you've never tried Learn Go with tests, it's just a really really great educational resource. I really like it. So what I'm going to go through now is a mermaid flowchart that you can follow if you'd like. Um, if

you go to GitHub and you go to uh m oda maroda montaverity and you look for the monteverity.made file, uh, you will see this you will see this outline. Um, just like I did with the overlapping AI image logo thing, I asked Claude to read my codebase and draw me a top level flowchart and it really sucked. Um, but it had all the elements I needed. So

after that first version, I took it out. One of the cool things though is that I had never heard of Mermaid and Claude just created this mermaid flowchart for me and I'm like, whoa, what Mermaid? That's this is really cool. And so like once Claude turned me on to Mermaid, that's the last time I talked to Claude. I was just like all over the Mermaid docs and

learning how to edit this and and that's what we're going to see. So I'm not these are very full of words. I'm not going to sit here and read all the flowchart for you, but I'm going to I'm going to focus on like what I did with our joint cognitive system to help me out in the configuration area. Um, I did a lot of asking what what

could be tested. Um, one of my problems is really finding the right things to test. Um, knowing what you really need to be testing and being able to build like very voluminous um, property based tests. More than once um, I told the LLM, "Hey, I need a test that has all of this data in it." and bam, it would just do all the data. Um, building complex

JSON is the same way because then I would have like a bunch of endpoints that I wanted to see if the configuration worked. So, I just asked the LLM, build me a JSON with this endpoint as if I were pulling the metrics. Um, endpoint testing is actually something I've done a great deal of. So there wasn't much around fetching data, but again testing really helped. Um the

LLM turned me on to several new things this way. Um like I mentioned, Mermaid um and then other just like go libraries that I wouldn't have really known about. Uh really helped on the ingest and transform part of Monteverity. So this is where we were in kind of the middle of the flowchart. And so what's happened now is Montever has pulled and this is happening every so

often. You can configure the pull interval in the configuration. I think we're pulling right now every four and six seconds. That's what we're hearing. Montever takes that data and then it transforms it. It might have to be a rate. For example, there's a calc, you know, a calc rate plugin that will calculate the rate instead of the climbing counter. Um, there's also a JSON key plugin that

you can just give Monte the JSON key and it'll pull that out of the JSON blob for you. Um, so a lot of parallel stuff happening here. Multiple endpoints can be uh reached at the same time. multiple metrics from those endpoints can be pulled from them and there was a lot of parallelism. Uh LLM really helped me with race conditions. Um analyzing my code alongside the output

when I ran go run trace or go run race. That's hard to Stack traces. LLMs are really good at And it helped me learn a lot. that helped me learn about threaded operations. I really enjoyed it. Um, when I got to something and I was something I repeated a lot in my prompting was that I'm learning how to use Golang. And when I just when I gave

the LLM that suggestion, the suggestions came back in a very kind of like, well, this is what you would do in a simple case and this is what you might do in a more complicated case. And I'm not real good at math. Um, you know, the LLM helped me run through this algorithm that I was building and rubber duck the algorithm and make sure I was thinking

about it correctly, making sure that I was thinking about um, which side of the data I needed to look at and stuff like that. And eventually, one of the weird things that happened is that I've been interviewing for jobs and I have been giving my resume to the LLM, right? And on my resume, it has some of the languages that I've learned like Erlang. And when I

started doing some of the algorithm work in Golang, the LLM recognized that I was thinking like an Erlang programmer. And it made that connection for me. I was doing some head. If you know Erlang, there's this idiom of head tail where you take the head of a list and you leave the tail. Um it's like the minm max of a slice decor de declaration in Golang and

I was using that a lot and the LL noticed this and met yes this algorithm is very much like Erlang I can tell you're pulling on your Erlang roots and I was like that kind of really surprised me. when the harmonic analysis is done that means that Montever has found an accent. It groups that accent into a pulse. If you remember the Shakespeare at the beginning each

of those foot each of those feet is one pulse to Montever and then Montever shoves all those pulses into a moving window. And so you get these pulses. Some are trochies, some are imams, and they kind of alternate sometimes and mix each other up. Um, this is one place where I again would really be able to lean on the LLM to help me with testing because I

would want to get sequences of metrics that I wanted to test. And so I needed a specific pattern to be created from, you know, pulling those metrics. And the LLM was able to help me with that. and create those tests. Um, sequences of input that would definitely turn out as a troy or definitely turn out as an I am. And I mentioned I'm bad at math. I

really have a kind of dyslexia for symbols and numbers. And so in this way, the LLM was really beneficial for me. I was able to check my math to the LLM or check the math in this function. There were several times where the LLM told me, and I was not surprised, "Oh, you're thinking about this backward. It's actually the other way around." Uh, and you know what?

I probably would have spent a week on something like that. Um, so for me, that's really helpful. Um, let's look at the TUI. So, this is how Monty Verde started. It was a terminal app. I wanted to push myself. I wanted to learn something new. And I was inspired by this book by Julia Evans called The Secret Rules of the Terminal. If you don't know Julia Evans

cartoon um instruction books on Linux, they are the best. And I was like, hey, I'm I'm going to learn some stuff about the terminal. I'm going to do this. And that's what you're looking at here. We'll look at it a little bit when I take this down, but this is actually what we're hearing is what you see up here. Here this is this is a screenshot from

the terminal UI of of this demo uh set of metrics and you can see how this is the metric integer going up. You can see the metric integer going up and then it stops being above the max here and then again it comes above the max over here and it goes up and then it goes back down and then it goes back up and then it goes

And up here you can see them going off, on, off, on, off, on, off. So I have these metrics right at the top. I really only wanted that very, very top peak. And then what you see down here are the patterns being recognized. There's actually these last two rows are not down here because there aren't enough here for the software to do the algorithm. But the top

three lines if you can see this is a representation kind of an abstraction of the numbers up here the metrics coming in into the darker red is no accent the lighter red has an accent. So what about outputs? Learning the terminal UI was completely unfamiliar territory for me, but I learned so much about both Golang and the terminal in the process. But I haven't done web design

in a long time. Uh I did web design as my first kind of technology job way back in the other century and I gave a talk on Java at internet world. I was really into front-end applet design and graphic design and all of that. But then I I got jobs into data centers and databases and data and infrastructure doing S sur work and I just moved away

from the front end. Um and you know it's hard to keep up with that stuff. Uh Claude was my JavaScript tutor. I'm gonna be straight up honest about that. You know, providing basic starting points for me again with that configuration in my claw that says, you know, give me snippets, don't give me full things. I'm learning and you know going in, you know, give me like a

basic CSS for this. And so I'd get a basic CSS and then I'd go in and just change everything. And it really helped me get back on that bicycle. It really did. Pretty soon I wasn't going to the LLM. I was like, "Oh, I remember this JavaScript." Yeah. So, working with the LLM about specific problems to solve is the way that I approach the joint cognitive system

of working with an AI. Give me a basic structure. Then I would explore that code myself and then make changes to it. and I might paste a section of it back to the AI, say, "Is there anything wrong here? Uh, could you make any corrections or suggest any corrections?" But then I would go make those edits myself. The plug-in architecture is something that came about because I'm

thinking about how do I make it play music? And I didn't want to mess up the algorithm stuff that I worked on in the terminal UI, you know, long hours on figuring out the algorithms. I wanted to get that to a good place and then set it aside. And so I decided to create a Golang interface that is the plug-in architecture for the software. So uh you

can see up here in the very top these are the two output plugins that will work. The input plugins by the way are the JavaScript and the uh the rate calculation plugins. Um you can configure this with Badger DB. So it will write to a Badger DB database. Uh and I haven't played with it too much. I did configure it and it's working but it was kind

of like okay how do I do this? and Badger DB is very very well defined. So um I learned to write my interface by using Badger DB as kind of the reference architecture and then I built the MIDI interface, the MIDI output interface. Um so uh there's some kind of weird things about the Go MIDI library that I was stumbling on. So the LLM really helped me

with that. like in order to send MIDI commands through the MIDI interface, it actually uses a callback function directly in uh directly in the in the slide or in the uh strct. Sorry. So, I had to wrap my brain around that and the LLM really helped me do that. I love that little D. That's a great Montever has its own Prometheus endpoint. When I first built Montever,

uh, we were I was checking its own metrics. Um, which can be a little bit hard to do to get, you know, the results that I really wanted it wanted out of it. Um, but the Prometheus metrics are there, including like, you know, I I showed the heap size from the Golang thing. All of that stuff is there as a Prometheus metric, but the really interesting stuff

I knew would be done via open telemetry. So I this is me just you know doing reps on doing instrumentation in hotel while I'm trying to get a job as an S sur and uh and sent you can see over here this is graphana cloud this is one of the um the trace views this is the duration right here each of these um separate traces are being

shown and I think I don't remember you recognize this from the Shakespeare slide I Um, that was actually from Graphfano, that example. the LLM helped me think about where to put the traces. Um, it helped me out a little bit considering how do I how do I want to wrap my handlers to be able to do auto instrumentation, which is a really cool thing because you don't

have to do anything. You just wrap your handler at the top, your main router in Golang and then you get you get automatic uh uh telemetry from your web front end or your API. But then I had Monteverity specific functions and so that was the place where I was like hm what makes sense for me to go which function to which function how deep does it make

sense to go things like that. So what is next? Having the pattern status show up as traces in open telemetry would actually give a really cool level of confirmation that things are working, but it would also provide some historical benefit, some historical visibility. Um, Montever doesn't, unless you have the Badger DB plugin, uh, enabled, it doesn't hold on to the data. It only, it has an hour

window that's configurable. You saw that in thein file. And it drops the data after an hour. Uh, it would be cool actually if we could send that data to a model and do a neural net on it. um create an output plugin that would kind of hand off the historical pattern recognition to an AI or something like that. Uh and maybe this is kind of like I

this is where I don't know what this what this could do. You know, if we got something like pattern recognition flowing and then we had a bigger thing that can do better pattern recognition, could we eventually get to the point where we could actually see the law of continuity, hear the law of continuity in our metrics, and then maybe even see when the system is about to

fail. I think that's like the, you know, that's what we all want to know is how proactive can we be? And I think this is a really cool avenue to explore. What if we could import existing KV time series? What if you already had a bunch of metrics and you're like, "Okay, I want to get all the accents out of this big time series." Um, it would

be cool to be able to import and replay that. Um, there's a lot of work that I want to do on the MIDI output. When I describe the MIDI uh interface, I only describe two aspects of it, pitch and gate, which is the trigger and the length of the note. There's other stuff. So, you could do things like there are control values, continuous control values in MIDI

that you can send out to like a filter. In here I'm doing the filtering with um a a module called maths ironically. And something that I really want to do is I want to do some performance testing. this can run as a Linux container and I have this running at rainbowq.net and uh it's not doing any MIDI. So the the output plugins are not enabled in the

Linux container version, but the idea there is still the same. Um just we're not using the sound as the output, but what if we could scale that out? Could this be scaled? And then how would that be scaled? You know, I don't really know a lot about it doesn't take a lot of CPU. Um but it it is very it can be memory hungry and so I

haven't really figured out those levels yet. Um, it's like scaling to multiple instances behind a load balancer. That would be something really cool to do. So, this is all the slides I have. And I wanted to show some of the uh symphony of players up here on this slide. These are actually a collection of some of the things that I talked about, the open and free hardware

formats and a lot of the dependencies that are in Monteverity. Um, yeah, I have Prometheus and Open Telemetry in there at the same time. The Go MIDI library, which is uh which is just really fun to work with. Um, and all of these other things that just kind uh are dependencies and they they float around what Montarity does. one last thing I did work with the LLM

to create this word map. This was another case where I went, "Okay, create we create me a word map." I actually gave it the go.mod file. I said, "Take the go.MD file. I want a word map of all the high-level open-source resources that are in my mod file." And so that's kind of what I got. And then I did some of the replacement for like the winter

bloom soul that's at the top. So now what I want to try to do, does over here. Um, let's get a mic, please. >> On my way. Oh, I was coming to you. >> Uh, first, this is awesome. >> Oh, good. Thank you. >> Um, so I have, if I understood correctly, you have to kind of tell it which uh metric series it should observe to do

the harmonic accent analysis. And so what if I have a bunch of metrics as one does, have you also done some research or have some thoughts about how could I plug that into some real production metrics? How do I pick the right ones or is there a way to just like you see what I mean? >> Yeah. Yeah. Yeah. The the way that I answer that is

um thinking back to that that good continuity like what are the aspects of my system that are going to tell me a little bit about my good continuity. Um maybe this is an aggregate metric that you bring out of a bunch of stuff or maybe it's the result of of a trace of an open telemetry trace or maybe I've even thought this could be SLO results. This

could be error budgets. This could be any of those kinds of things. Anything that makes sense that that it's helping us understand the system. Um, and and people look at that from different ways. I like to look at it from a user perspective, which is why I think doing putting SLO into this would be really cool. Let me see about Oh, yeah. Okay. So, what we're looking

at right where's my pointer? So, here are the live metrics coming in down here. Toad Lester is that metrics app that I wrote for this. And you can see uh when it hits the max, the n the percentage turns red. When it's nearing it, it turns yellow. But when there's no accent, it's green. So what you can do boy, you can get on this screen and pick

a metric and it shows you the same information. Uh, so I kind of tuned this so we wouldn't have to mess around with it too much, but you what you would do is you would look for this metric. Um, and then change it in the configuration down here. Just change it right in the JSON. You can validate JSON from here. And then you can send it um

directly to the configuration endpoint. And you can do this in a curl command too. The other thing that's interesting about this is uh the plug-in interface. So this is showing you um the the the characteristics of the plugin. Uh these are basically looking into the future of what notes are coming. Um what notes do I have left to play? Uh, this is one of the improvements that

I'm talking about with the MIDI interface or the MIDI output is can I build in some kind of configuration so that the end user can um say, well, I only want so much of a backlog of notes to play and that kind And you can see down here at the bottom, I don't know if that's too easy to read right here, but it's telling me about my

Winter Soul MIDI interface down here. And it's even showing the configuration, the scale, my pentatonic scale, um, defined in intervals. And intervals is the keys on the piano. Last but not least, this is it. This is the interface So what I described What the notes that we're hearing are really only the orange and blue ones right here in the middle. This is going around every 60 seconds.

And something we can do is we can turn on some of these features down here and you should be able to see when something transitions. I can also put a grid across the whole thing. Something that I like to do is I can make my pulses be uh organized in That changes the way the patterns appear. I thought it was really cool to be able to Hey,

maybe I can I can change make it like a Tempest game. Anyone ever played Tempest? can change around. And the idea here, make the make the pulses fatter is being able to see different kinds of metrics I can even select them. Oh, this is an interesting pattern. This pattern right here. Does it match any other patterns? Oh, it's kind of interesting how that pattern matches this pattern.

I can also select rings and rings will tell me what the metric is down there at the bottom. It does have some built-in help too. Where is that help button? There it is. Basically describing what I did in this And then down here at the bottom, the last feature of the UI is that each of the patterns lights up. If you watch the bottom only when they

get processed when they get recognized by Montever engine, it'll blink the pattern that it recognized and then and it's colorcoded too. So you'll see the the troes and the IM come up separately. The other thing before I show you the terminal UI real quick is that these purple patterns out here are actually what's called an amphabro. And the amphib is no accent accent no accent. So it's

a big bell curve. It's what an amphib looks like. So the outer rings, the purple is actually pattern recognition of the inner rings. So out here, these are telling me you just had a pattern of a trochi of an I am, a troy, and then an I am. That that's a big hierarchical amphib just like we saw in the Mozart and Beethoven examples where Meyer had the

note by note accents and then the phrase byphrase accents. And that's the idea is that we would be able to kind of like extract all of these accents. Maybe we would even get to a bigger point where we're making even more abstractions of the second dimension of what we're recognizing. Just trying to find different ways to discover things about the system. And lastly, I want to show

you where it Let's see if I can move this over there. There we go. So, this is the same screenshot that I showed you before, except it's live now. I can hover over each of these and you can tell at the bottom what the the metric is. You can see what the metric source is down here. I can click on it and I can get a raw

value. That's what this number is. It's the raw value that's being measured out here at the front. This is the front of the histogram. And this number changes when a new number comes up here. There's another view to this. This is only I tell you when I first saw this happen, I jumped out of my chair. I couldn't believe that the harmonic analysis had actually worked. I

mean, I was laughing. I just couldn't believe it. And then I created a way to view it and it's been playing for long enough that it's got some of the This is excellent. What we're seeing right now are just the TR. Let me get This is all If I hit I, you can see the I ams. This is interesting how the engine kind of ran together. Like

if we go back, you see these big accents because I decided I want the accent to trigger real low. And so the patterns look real long. I can look at a trochi way of looking at it. One big old accent like that. And then you can see I could even switch back. Do I am look any different from trochis? And can I look at a next level

up? This is the amphibros. So you saw those purple things. Sorry, it's green here, but this is what the purple things on the ring were. It's the higher dimension. So it's taking the lower patterns and it's making that big bell curve pattern out of them. So that's what we're seeing. If the display were longer, and you can configure it to be longer in the M file, we

could probably see the amphro the other side of the amphro, which would be a non-acent like Back to pulse view. Thank you very much for listening to our beautiful symphony of open source. I really appreciate it and I'm glad that everyone was able to come and sit and help out. Thank you. >> Thank you so much. Let's give him a round of applause. >> Any other questions

over here? Oh, I was going to ask um is it okay if some of us can take a closer look at the um synthesizer? >> Oh, yeah. Yeah, yeah, yeah. Come on up, please. There was another question over here. Yeah. >> Um I was going to ask, does it always sound this good? And also, in your experience, has Has it been more beneficial to do the the

audio than like looking at graph charts for your observability? >> Oh. Oh, yeah. I I can't I wish everyone could have been in the room with me when I made the first sound. I I just couldn't believe it. And that was kind of my real goal was to get to that point to make the sound. And it the the sound part of it was kind of the

culminate the culmination of everything else. Excuse me. So having the sound actually helped me think about the algorithm itself. Am I being too greedy about you know where I put the point of the accent in where Montever is measuring the whole thing? um should I make the window longer? Something that I'm contending with now is like um buildup um in the MIDI output because MIDI is serial.

You get one thing at a time. That's how MIDI works. So when you got when you're getting dozens and dozens of metrics that are coming in, that doesn't quite transfer well to the MIDI part. So it's helping me retroactively think about like efficiency and stuff like that. Yeah. Any other questions? You're welcome to come up here and check this out. Um, and I'm going to be like,

you know, packing up, but please come on up. Thank you again. testing. Testing. Okay, that works. Okay. Uh, hi everyone. Um, I'm Tiffany Jernigan and you may notice that Nathan is not here with me. Uh, the poor guy got sick yesterday with a high fever and had to cancel last night and this morning effectively. So, yeah. Um, if you're here to listen to observability things and MCP,

awesome. If you weren't here, well, hopefully it's interesting enough to you anyway. Okay, so I'm going to start off by going into things on the observability side and then shift over to the AI, agents, MCP, etc. side of things. and we'll be asking along the way to just kind of gauge where people are effectively. So, seeing that everyone is here for visibility, I'm guessing that everyone has

at bare minimum heard about it if they haven't used it in some way, whether it be for personal things or for work. So, just to make sure that like everyone is level set since there are probably people in different levels here, um I just wanted to quickly go over some things. So basically like what is observability? One of the definitions of observability has been that observability is

the process of making a systems internal state more transparent. So like what are some things that come along with that? Depending on who you ask um some people love this term, some people hate it. Um who here has at least heard of someone say, "Oh, blah blah blah the three pillars of observability." And one moment apparently this one fell out of my ear. So, um yeah. Okay.

So, who likes that terminology because it helps that make sense to them? Okay. Who hates it? Okay. About the same. Okay. Okay. Um, so if you had heard that term, it was usually like the three pillars of observability. You would hear about people talking about like things like metrics. So who here has used metrics in some sort of way? Most of them. Okay. So what about logs?

About the same. What about traces? Actually quite a lot more than I expected. Um, and then depending on who you ask, some people now say, "Hey, actually there are four pillars of observability." So, who here has used profiles? Okay, if people are around, there's way fewer people. Okay, so like in practice, a bunch of the things on here are what you might be seeing when you're looking

at observability. So, there's things that you're also dealing with. Hey, maybe I'll be going and creating alerts based on some sort of data. Maybe like there's incident management, maybe SLOs's. So I'm going to just kind of go into a bunch of the different um pieces that we have here. But first I'm going to mention these uh database backends that we have. So since the majority of the

things I'm showing through the demo or through slides are from using graphana plus these ones. So for metrics there is mimeir and then of course you have prometheus in there. Um we have logs. So with that is Loki, traces is tempo and profiles is Pyroscope and you may be using other tools which totally works too but everything that you'll be seeing here is using those. Okay. So

this is kind of like a viewpoint of the four different pieces that we have here. The four telemetry signals that I was just mentioning. So some things that you kind of need to know are like what each of the different ones are and then like how the fact that they also can work together. Some people might talk about hey I'm just using my metrics and that's all

I need. Or maybe I'm just using logs maybe or I'm using logs and metrics okay cool don't need anything else or I'm using traces and there are people that say hey with traces I don't need anything else because I can get my logs and my metrics from my traces. So depending on what you're doing and who you're asking, things might be different there. So kind of based

on just looking around the room, metrics and logs are the um ones that are most used. Um so like just to like kind of quickly go over some of this part seeing that the majority of the room has used some of these things. Um so we have metrics and that's basically asking like is something happening? So you have like some sort of name that you might have

uh some of the common metrics that you might deal with even if you're not specifically doing something like I'm observing my system in a way like say if you do go look at your machine way back in the day or even now and be like okay how much CPU is being used how much memory is being used what's just kind of like what's the storage that I

have on here you may have used DF for instance or like hey you're using a thermostat what's the temperature now and there's just different things there and those are some of the common things you might be used to for metrics. Um so this is uh what if you're using Graphana and Mir this is kind of just like a dashboard basically that was created to show off different

metrics and what that could potentially look like because at least for most people um just having numbers isn't the easiest way to look at something. You want to be able to visualize that and see how things are changing perhaps over time, whether there's some an outlier that you're not expecting and things like that. Okay. So then we have logs. So that's basically like what is happening. So

they help you prov help provide context to what is going on. Maybe you have something like hey this person logged in. Maybe it's somebody you wanted to login, but maybe it's someone you didn't or it's a log message from your application saying, "Hey, uh, Spring boot spring just started up and um, I was unable to connect to my database." Things like that or like your application just

started and then it stopped. So, if you're playing around with either for fun or for some like professionally in some way with Docker, maybe be like, "Hey, my container crashed. Why did it crash? Look at the logs for that. So, and then this is just like a visualization of what that could potentially look like. Um, so maybe you have some like debug, maybe some info and just

things to tell you more about what's happening with your system. So then we have traces. So that's asking, hey, where is it happening? So basically, you go and follow some sort of request. Uh maybe you I decided I'm going to talk to my application via some front end that I have and then that front end goes and talks to say maybe OpenAI or some other tool out

there and comes back and then it talks to some sort of backend and maybe that talks to a database. So basically it's following from the beginning to the end of that whole chain and I'll show you next of kind of a visualization what that looks like. So for instance, this is three parts of the same thing just um split up a bit. So who here like has

heard of a trace before? Okay. So for the ones that didn't raise your hand, if it's because you don't know or instead of the I don't feel like raising my hand, I'm going to just kind of uh summarize. Basically it's a representation of a single transaction or operation as it moves through various uh components or services that you have in your system. So basically it's like a

visual way to see the journey of a request. So like this first part is just having like my trace that I have overall. Then we also have a thing called a span. So you can see that I'm just kind of opening things up more and uh every all the pieces down below are basically the building blocks of what your trace is. So for instance you might be

using like there's the Pyroscope right go with your bike handler. You have an HTTP get. So there might be different uh applications that are being touched along the way as you are making your request. And then like for instance this is a example of a typical data service. So basically you can see that you're going and talking from one p part to another part. And then you

can see things like how long is something taking. You can the numbers at the top even though they might be hard to see are compare um coming from what is in the uh diagram down that you have below. So just kind of going through and then everything that goes and comes back. So it can help you if for instance you're like hey this thing was only supposed

to take five milliseconds and oh no now it's actually taking 40. And then you can go look into see more things of like why that could be happening. Um so there's a few different pieces to traces. So they have structure and semantic meaning which helps you to be able to query things. Um, so we have things like we have the name for what our span name is.

Then we also have a duration. So that's like the difference between the end time and the start time of a span. So like how long is that thing taking? Then we have some sort of status which is very helpful because hey if I want to specifically find all my traces that have errors you are able to go and do something like that or maybe something isn't set

right. And then after that we also have like the kinds. So there's things like server, if it's client, consumer etc. So there's like different also there's different types of attributes. So you have your span attributes. So you can choose to add things like um what ht like thinks about htt method. What URL am I using? But if you overdo it just like with logs, it ends up

becoming not so useful. And then there also resource attributes. So maybe that'd be like your container ID or your Kubernetes cluster name for instance. Okay. So and then these properties they are called intrinsics and basically they are fundamental to a span. Okay. So then we also have profiles. So which is a lot newer and sorry I'm going to drink water here and there because this thing here

is quite warm and drying. Okay. So that basically like helps you see like how much like where time is being spent in your systems. So like for instance am I using x amount of CPU or memory at this part of my code. Um maybe you could see things like uh hey I'm not doing maybe there's some issue with like garbage collection. Um, and then like it can

help you like expose inefficiencies like depending on what you end up having a setup. Um, maybe you can be like, "Oh, hey, this is using way more CPU than it actually should be doing and be able to figure out what is potentially causing that." And then who here has heard of ebpf? Okay, starting to become more and more nowadays. Um, yeah, so uh it ends up uh

it leverages ebpf. So, and then there's different uh languages and how the amount of data that you can get from each of the different languages. So, uh some of them like you could do there's like Java, Go, etc. Like if you're using um JavaScript, you have like your wall CPU and just things are a little bit different for that as well. And so profiling has been around

for a while in general. Um but then people started talking about okay what about continuous profiling constantly be able to check that and then that there came about a bunch of things such as like pyroscopes. So like who here has seen a flame graph before? So more people um but yeah this is just like an example flame graph. We have our top table. So like if you

were just even like on your machine um not looking in some one of the like this specific type tool for profiles you can be you might be familiar with a top table in general and then we have the example of like the flame graph on the right and you can and then if you can dig you can dig into things to see more about um what what

utilization is being taken. basically we have all those different pieces but by themselves they aren't exactly use just like they're useful by themselves but they're a lot more useful if you use them together which I was kind of mentioning before. So there's different ways to be able to connect between them. So for instance like I could have a visualization of like my metrics. You could see maybe

it's going up and down and then you could have exemplars where you can go and I can click on that and I can have it set it up so I can link to hey let's take me to my trace or I could be in my trace and be like hey I want to go and look at my logs for this specific trace or I want to go

and look at my profiles for instance. so that all these different things can work together to help you figure out thing like are things working the way you expect and if not why are they not. So this is a older slide um that doesn't have profiles specifically in there but like there you have your metrics that or you have you want to be able to collect your

metrics you want to be able to collect your logs traces profiles etc but it might be a lot to be like hey let me go and uh make my own tooling to go do that most people nowadays probably would not do that ver I mean there's definitely going to be like people, new companies, new people creating their own things of course, but it probably doesn't make sense

for every single individual person to go and create some way to uh get your data. So basically um that's where things like maybe people are using who here has used something like uh for traces like Jerger or Zipkin for instance. Okay, a few people. So, like if you're doing that, maybe you need to figure out, hey, how um how do I use Jagger specifically? What do I

have to set up there? Or what do I have to do specifically for um Prometheus for my metrics? But all these different tools that you had to like fully figure out setting things up if you decided if one tool got deprecated at some point in time, there wasn't the easiest way to move to another tool in that space for instance and things just like could be different.

So that is where uh open telemetry came along. So basically um who here has used open telemetry in some capacity? Okay. So yeah basically um I actually until was told the other day I didn't actually think about breaking up the parts of open telemetry and what the different parts mean but basically the idea is that we have all these different telemetry signals. We have all these different

tools out there, but having a way to be more standardized so that people can one have it be easier where you don't have to learn something spec like all the specific things for every single tool that is there in that you're wanting to use. Of course, there are going to be things that you do need to learn for each one to an extent like say maybe a

tool has its own specific query language. So maybe you're trying to figure that out depending. So basically having that and having it be open source and being worked on can result in just like more ability to share across the different things and make it be easier for each person. So kind of like what so basically open telemetry like it provides a bunch of different things. So you

have tools APIs and SDKs. So there's like language specific APIs for instrumentation. You have SDKs that implement data handling and processing. Um there are optional components that you could have such as a collector which I'll go into a little bit more but like for like routing, exporting, processing etc. And so it just basically helps standardize things across metrics, your logs, your traces while keeping your backends pluggable.

Okay. So I mentioned instrumentation earlier but in a way this is a little bit different of how I'm talking about it. So like um when we talk about instrumentation we are talking about like okay how are we adding some sort of visibility into your application code. So yes you might just have your metrics you might have your logs you might have your traces but you need to

actually be able to get all of that. So maybe you might directly uh if you weren't using open telemetry I could be like hey I'm just going to send my traces directly to my back end. You could do that for all of them if you wanted. Um, but if I'm using open telemetry, there are a few different ways to go about doing that. Um, who here has

heard people either say like zero code instrumentation or auto instrumentation or manual instrumentation or something along those Manager on my team and no one else except one other person. But um yeah so basically um it's again it's so it's not basically just about tracing but you it's there for you have it for metrics you have it uh for things like logs um profiles is less so um

because it is currently being worked on so like we have all these different types of instrumentation if you want to like hear a lot more about the specific ones you can go to that YouTube it is a community call that um Lud Miller here who does a talk after me. Um and I and some others at Graphana did um basically like talking about tracing at a just

just diving into things uh tracing in general um a little bit on tempo itself because I mean it is a graphical call but like it's more of the high level stuff and um next Thursday if you want to know more about sampling we're going to go into that as well. Um so basically like the high level idea is like with zero code or automatic instrumentation is where

you don't actually have to go and modify your code. So maybe for like Java you have some sort of agent that you're attaching um when and you're running your uh jar and then you don't have to go and be like here let me go add this specific thing. However, if you are using manual instrumentation, that's where the SDKs come in and like maybe the amount of things

you get from auto instrumentation aren't what you're looking for or maybe it's not enough where you're looking for. Maybe there's too much of things you don't specifically want. So then you can go and add things to your code to be able to do that. And then you could also have the hybrid between the two of them. Um so then we have like collectors. um depending some people

say hey you don't need one some people are very admit that you do um so it depends on like what your case is so it's basically uh like on the open telemetry side of things at least it's a vendor neutral telemetry pipeline so like your collector will receive data from your services it'll do some sort of like processing that you might want to do and then export

it to whatever backend that you are using so like this separation lets you like instrument once and then you can change your backends without uh having to rewrite your applications. So there's the open telemetry collector. So it's um basically the one of the standards that people use especially if you're use like when you're using open telemetry um I have a docs link there. There's also Graphfana alloy

which is also open source um and it is compatible with open telemetry while also integrating with other observability ecosystems like Prometheus. Okay, so basically to kind of just like recap. So open telemetry focuses on generating and transporting telemetry. So it standardizes like how applications produce metrics, logs, traces and send those through a telemetry pipeline. So what open telemetry doesn't do though is like it doesn't go and

annual annualize analyze or like visualize your the data that you have. So like it's not going to go and create uh dashboards. It's not going to go and create alerting for you. Um it's not like things like investigations. So that's where you need something else as well. Um so at least in the graphana ecosystem um that's where we have graphana and so basically it allows you to

act visually act like you can act on a visual way um for the data that you're collecting. So just who here has uh used graphana Okay. Wow a lot of people. That's cool. for this one here, I've never actually had a video thing in a slide before, but Nathan edit it. So, if we'll see how that goes for me. Not well. Okay. Um, so basically, it was

it's just like a short little video that just um from I don't actually know when um but just the different things that you could potentially be doing with Graphana. So, I'm going to skip past his videos. Um so, like part of the idea is like this big tent approach. So meaning that things like integrate with a wide range of data sources rather than just needing like some

specific single like proprietary backend. Um the goal was to make it so that uh users can go like you can query, you can visualize, you can alert on data regardless of kind of like where you're storing it. So um basic I think I showed this image earlier for with um when I specifically talked about metrics but basically you can visualize uh you can the data that you

have you can create um you can create diagram like dashboards with the different types of charts. This one specifically is just related to metrics but you can add things for profiles you could add ones for your traces etc to be able to look there as well. Um, so basically to be able to like monitor your infrastructure and your application performance. Um, so I'm going to since I

don't know how videos work apparently, um, there's a bunch of different things that you can also add. Like for instance, um, this one is not what most of us probably are creating dashboards for and data, but um, it's kind of cool seeing things for like especially when it's with space. Um, so it's not just like, hey, what are my like logs, etc. for um, an application I

have? There's just a bunch of different use cases there. Okay, so now that I've covered things for observability to just kind of make sure everyone is on the same page, um, I wanted to start talking about the AI side of things. Um, and specifically like LMS and how they can play a role in Um, so who here has used some sort of like LM experience with I

don't know what some of the uh AI assistants like maybe use chat GBT maybe use claude cursor a decent number of people okay so like um who here is um like newer to like what to LLM and AI Okay. Okay. It seems like a lot of people are pretty familiar. So, I'm just going to do a really quick recap. Um, especially for the folks that did raise

their hand there. Um, so like at a high level, um, this image here, um, is like what a base model like LM what a base LLM model looks like. Um so in this case specifically um it's an open source llama 2 model. Um you can also think of like some of the proprietary models that are out there that you've used with gg from like uh maybe like

there's claude opus 4.6 or some of the other things out there. Um so it's not exactly like chat gbt or claude though it does like you can generate things you can ask it questions. Um, but there's a lot more things that end up being helpful based on like how much like fine-tuning that you can end up doing with it. Um, so I'm just going to dive into

things a little bit more. So like this is a specific example. So um, ele behind the scenes basically are just like next word or like token generators. So taking in like the previous context of like hey you have a conversation you have some sort of prompt um and then outputting the most probabilistic next word. So who here has heard of openenheimer just out of curiosity. So, um,

as you probably know by now, like, so in order for these LMS to have ever been useful in the first place, they needed to be trained on some on some amount of data, right? If just like a newborn does hasn't been trained on learned anything. So, it's not the same as like any of us who have spent a lot of time learning things. That's the same case

for these models. and they basically have been fed like a huge data set of things on the internet like for example this Wikipedia article for instance and so like um the more that you end up training it the more that you can end up going and getting out of it. Um so basically hey if you give it specific words like the more uh like the the little

example that was beforehand um based on that it can try predicting like what the next word is in the sequence. So like for instance like the uh red words that were highlighted here um are some of the words that might contain a lot of information. So like for example if your objective is to predict like the next word presumably your parameters have a lot to learn on

this knowledge. So for instance like you like you know things like uh when was he born or like when did like what did he do like what like who is he and things like that and based on those types of things you can end up learning more about uh the world. um, I was just being a dork because there's, uh, a nice fun movie called How to

Train Your Dragon and, um, Nathan had put How to Train Your Chat GPT, so I feel like it needed to be there. Um, so, uh, this training basically like leads to generating a base model that you can think of sort of as compression of the training data set. It's not exactly useful yet. Um, and it requires amount of fine-tuning to turn from like just generating random documents

from the internet um, to actually being a helpful assistant and actually helping you out based on your prompt versus just like here is like a random document from the internet because like when you're using Google early on, it would give you a bunch of different documents that you can go and look at. But we need to go another level past being able to do that and actually

having it be able to tell you things from those. So that kind of leads to Loom today. So like as most of you have like tried out things like chat GBT or some other tool um that is LM based to like maybe figure out some an answer to a question maybe used it to generate some sort of image. Maybe you've used it to help you with your

coding. So like these LMS are they're they're great. They're super useful. Um they gave developers a way to access the context of the internet with based on the training data snapshot. Some of you may have noticed at some point you would ask some questions but like the most recent information you were getting was maybe like 2023 for instance because it didn't have anything newer than that. Um

and it was doing that using natural language. So like hey like me specifically talking to it could like the wording that I would use when talking to maybe like a friend or a co-orker etc. Um, and then it would be you'd be giving it like a predefined context and instructions and then you would get some natural language answer kind of just like if you asked it like

hey what what is the day today and what are all the events that are important in history that happened today and it could give you those things. So then like relating things back to observability um we found some like initial promising applications of LLMs. So, uh, this is from Pyroscope, um, a little while ago, and basically you would go and feed in your profiling data to the

LM, and it would go and explain, um, different things about your uh, flame graph. It would analyze it. Basically, you weren't going and just like asking it questions or going into specific details, but it's just like, here's my data. Please explain it to me. And so, they have like context from the internet. So it would go and look at the internet and be like based on my

knowledge of profiling etc. here or may or also like you can have it like based on the code that you might be feeding in as well. Um and it'll give you information about that. So basically it would help like lower the barrier to understanding profiles especially if you hadn't used them before and you're like hey I have this giant like colorful thing going on. I if I

click on it maybe I'll see that something came like I c I'll see some sort of like function package or what not but I don't know what any of this means but that will help you do that. Um but they only kind of knew exactly like what they've seen in training like you're passing in for that instance like your data but like it couldn't actually interact with

it. It couldn't actually do anything there. Um it didn't know anything for instance about your dashboards. um it couldn't go and query your telemetry, your metrics, your logs um until things moved forward and then there became things like that people started talking about uh with So to kind of like talk about like that agents so basically like what are they? So like they are um LM powered

systems. Um they're able to do more reasoning on the different things. If you were looking at like say cursor for instance, you might if you have um uh the be able to ask questions, you can see that at the bottom there might be a little thing that says agent. Um it has the ability to go and you can ask it questions, but it also has the ability

to do something like you could be like, "Hey, um I have this issue with my code. Please fix it for me." Um, and it could actually go and do that. And so basically it was allowing you to extend things past like the base model and you could have it so that it would have external knowledge as well um with tools. And that's where we'll be talking a

little bit about MCP. So basically like it gave LLM like the agency to complete more complex tasks. It wasn't just talking to a chatbot and having it just give you an answer back. Um it would do things like deciding like hey um what do I specifically need to do? Um what tools do I need to call? For instance, if I want to create a dashboard, I need

a tool that is able to do something like creating dashboards. If I want to do something with um say if it's specific to graphana, if I want to do something with traces, I need tools that are specific to my tracing for tempo or something specific to Pyroscope. So it would need to know like what tools that it actually needs and then like what order to be able

to call those in and it would adapt based on like the more information that it was getting. So basically like based on the task that we give them, the LM can take some sort of action. It can get feedback from the environment that it has and then decide what to do next and just keep you can see that it just keeps going until it actually finishes the

task. Whether that task is actually like what it finished and gave you is what you were looking for is a whole different story, but at the very least it finishes at a point when it thinks it has given you what you were looking for. Um, and we wanted to see kind of like where does that play out in the graphana world. So like um some things that

were kind of like missing. So, like we've talked about like LMS, we've talked about agents, but like once you try to use them in real systems, there's um some like issues that you might end up running into. Um so like the access to your context isn't standardized. So um there might be like different uh tool connections that could be custom. You might have different security boundaries. Um

that's kind of where the next part of the story kind of begins. So like um around like 2022 um so LMS became a lot more like widely accessible. Um they were again trained on large data sets but that is data is static. It didn't know things like your internal documentation. It didn't have anything about maybe like customer records or the current state of your system. Um they

also didn't inherently connect to like APIs to your databases or different SAS tools that you might be using. So like while they can generate language well they are lacking a real-time context and operational awareness. Um so that's where things led off to uh retrieval augmented generation or as a lot of people like to just say is rag. So who's heard of rag before? Okay so like most

everyone who's used a in some way. Okay. So yeah, in 2023, um, rag basically became like the dominant pattern there. Instead of like going and retraining your models, we would like retrieve relevant documents and insert them into the prompt at runtime. So basically, this would allow it so that LMS could go and answer your questions using private or like domain specific knowledge. Um, it made reli like

it made it so that reliability was greatly improved. um but it's still like focused on going and like reading information but they could now reference some sort of external knowledge. Um they still couldn't consistently like act on systems in like any sort of structured way and that's what led to like the next phase of things. Um since I was just talking about agents in general um I

wanted to just like mention that who here is of the folks using graphana who's used graphana assistant okay so um yeah basically uh for graphana assistant you can ask questions like hey um what traces are having errors or what are my services etc um and then it would give that back to you because it's running inside of graphana it has access access to everything that you have

there in that context. Um but say if for instance you are using uh cursor cloud etc. it's not directly connected to graphana. It's not you don't just go log into graphana and have all that access and that it can go and get that information. Um so basically it it was kind of clear that like um as agentbased systems were growing that like ad hoc tool integrations weren't

exactly scalable. Um, so every model to tool connection was custom. So MCP or model context protocol basically introduced a standardized way for models to discover and use tools in a structured context. So that kind of moved us from having just like static models to granted models. So rag and then to capable agents that could um interact with real systems in a consistent way. So that's yeah going

with that basically like the topic of MCP. Who here has like used an MCP server at some point? Okay. Um things like who has used one for observability? So is it more like stuff like maybe like GitHub's MCP for instance? Okay. Um so yeah, basically there was the proposal for having a standard. So basically it's designed to make it easier for AI to interact with different tools

and data sources. Kind of like you could think about it maybe like uh USBC for AI apps. Um it goes and simplifies development and usage. So like if you are like a service owner or MCP server developer like one way of defining functions or as a user um maybe like interoperable with different MCP clients like your IDE or your CLI or chat clients. So maybe like cloud

code um for app developers um like basically each MCP server tells the client which tools are available with instructions on how to use them. So basically developers would define like some sort of tools. So maybe there are like API calls the different functions custom actions and you give the LM a choice to use the tool if it helps answer things. So basically it allowed it to go

past just like what does like what's the context based on what's over on the internet and like you could talk to an API that had private context. You could do more than just be able to have a chat response and for example like maybe uh you could create a PR in a private repo. Um and then basically like things started off as like super vendor specific and

like custom integrations. Um which could end being a lot of work. So at least for Graphana, we have uh an MCP server that exposes our core tools to LLM. So um and this is also open source. Um so you can find it on GitHub. Um so basically it made it so that you could interact with like hey I have my um data and I want to be

able to query that. So you could be like um what metrics do I have or like what yeah what errors do I have etc. And it would you could be in cloud you could be in cursor etc. And you could actually get that information from um what you're running in graphana. Um and then over time thing more things are being added to it. um you can see

what's there. Um currently everything for like setup and whatnot is inside of the GitHub, but soon there will be full on uh pages of documentation separate from there to make it easier to uh learn more about it. Um so basically like now that LM can finally speak graphana and interact with your observability as needs we can actually like answer things more like so for instance you could

be like hey I like I'm having this active incident and you could use graphon MCP to like see hey what tools are available to help me figure this out. It would go and be like there's a tool that's for incidents and then you can get responses based on the context that you have running. So this could help make it be a lot easier to uh analyze what

is going on. Um if you want to try out uh Graphana MCP uh you can go to the GitHub there and um set that up. You could either like install a binary which is what's happening there or you could use like a pre-built Docker image and then you configure your MCP client in say cursor cloud code etc to use it and then you would uh create and

I'll show you some of the stuff in the demo but like uh you would create a token um in graphana in order for it to be able to actually pro be able to do anything with uh what you have and then you'll be able to see hey what are the different tools I have and then based on that you can ask things. Okay. So I am going

to hope that demo things work. Okay. So let's first get rid of can. Okay. So moving screens does not. So I am going to move things to the other screen if I can. Actually I'm just going to exit out this. Okay. Sweet. Okay. So um let me it's it's never beautiful um in for visualization of things um but yeah so uh first I will go over here

if we go over to we have graphana and then mcp graphana so basically everything for setup is in here um there's a main readme and then there also uh which you can see is here and there's a bunch of like things for easy setup etc. But so basically we have like this example specifically uses UV. You don't need it for it's not something required in order to

use graphana mcp in general. It's just for this quick setup. But um if we you can see that you have things like your graphana URL. You have this graphana service account token and then you have like your MCP graphana whether that's you're running as a binary or directly on your machine or if you're using a docker image. Um, and then it has other things for like if

you look in the docs folder. Um, if I scroll further, um, there's different clients in here and it can tell you how to install things there. If I look here on cursor, um, if I go to I'm just going to go to my settings and then my tools. Um, we can and tools in MCP, we can see that I have graphana here. When I was mentioning the

different tool that there are tools, you can see different ones here. such as like creating an annotation, creating alert rule. Um things with dashboards, um there's stuff with Prometheus, etc. And then if we go and take a look in here, um we can see I just this is what I have here for GitHub, which I don't have currently working. So I have the one here for um

Graphana MCP. So you can see that like there you would be giving the service account token and then I have my stuff just running on docker on my machine using docker and compose. So then I just have host docker internal and then port 3000 for my graphana URL. So if we take a a look at my uh index.js JS. Um it is a pretty uh s like

simple application just for that uh that one of the people at Graphana created um yes just to kind of like show things for this. Um so basically for instance um this is it's not using open telemetry for the metric side of things. Um it's using Prometheus and you can you have a metric for instance like HTTP request total. um there's um duration in seconds we are collecting

metrics. So what the application all it really does is it's going to generate a number um 70% of the time it's like if it's point like for there's 70% of the time you have 10% etc. Like based on what uh random number you're getting it's going to determine if you are going to get a 200 a 400 a 404 or a 500. So nothing special for that.

Um, and then based on that, it will send you like, hey, here's a hello world, bad request, not found, etc., etc. So, if we go over, let's see if Okay, so we go here and we go to my page, it just like right now, the last time I ran it, it just said hello world. If I uh refresh it a few times, I'll get like bad request,

maybe some other things, and then not found, etc. If I go over to here, um, this slashmetrics, it's just like the endpoint that you get maybe used to with like Prometheus and you can see like all the different uh metrics that you can get data from. Um, if we go into uh, graphana over here. So this I just have on port 3000. Um, for instance, I could

go over here and I could go to explore. I could go to Prometheus. Um, and then in here you can see a bunch of the different metrics that you might have. So like if I want to go look at my um my data for my request duration, you can see the different metrics that we have. Um, so like if you I wanted to if you want to

do that, you could go for each specific metric that might not make the most amount of sense. So if you're doing things with metrics, it probably makes a lot more sense to um create some sort of So if I were to decide to go and do that myself, um I could go to dashboards. I could go create a dashboard, add some sort of visualization. I want Prometheus.

Um and then basically you would need to find every specific thing you want to add, like what metric do I want to add? Um you could do either code or builder depending on how you like setting things up. And then there's a specify like um like for your x-axis, etc. Like if you're really good at dashboards, you can probably do that super fast. I'm sure some people

here have made many dashboards over time. Um but it can be made easier. Um so for instance like if I were to um talk to so here since I have a graphana MCP running um or connect I have the graphana MCP server that's connected um with my cursor. Um, so for instance, I could be like, um, hey, uh, create a dashboard based on metrics in my code.

And I'm just just so it doesn't some every so often it tries to ask the internet and I don't want it to run off and inquire the web. Um, so I'm just going to say just because it's a demo and I less likely for things to go wrong. Um, using So basically you can see that it starts thinking about things. It's looking at my index.js. Um I

have like my data source.l and it's it knows that I'm actually I'm using Prometheus. Um so then it's going through and like finding what is is existing there looking and creating a dashboard. Um so it's saying that it's working on creating a dashboard with panels for my app's metrics. So like request rate, latency, histogram, status, breakdown, etc. Um, it seems like it's still starting working on that,

but let's see if we go over here what things are like at the moment. Okay, so it's still There we go. So, if we look here, we can see that we have a new dashboard. And if we go in, there's like things like request rate. We have our P99 latency, um, error rates. Um, some things are, um, just having the numbers, and then obviously the red usually

is a not good thing. Um, but just like it can go and create things like that. It may not be specifically what you want, but that's where you can go and iterate through and be like, "Okay, please add this, please remove this, etc." But the amount of time, for most people at least, that it took for it to go and just create this dashboard is a lot

less than what most individuals would end up needing to do for creating that. Um so basically like I could be go and say hey um like uh please add latency metrics um to my hopefully when like earlier so yeah is looking through what I have there and figuring out like how what it can add um depending on whether it feels like being nice to me at the

moment. um when I was running it earlier it would be like okay here let me add these things to your code and since you have this um I'm going to also go and uh add all this stuff to your dashboard as well which was quite neat um while it's going through that um I wanted to cover a few other things and then show you a little bit

more what so that way it's doing two things at once. Um, so if we go back to PowerPoint, okay, so there's also a Tempo MCP server. Um, so this unlike Graphana, it's not built into Graphana MCP. It's its own MCP server that is running inside of Tempo. And so basically you can ask it things like, hey, what traces have errors? Um, there's a blog and uh there's

two different blogs there that kind of go into it. one of them mentions cloud for it, but it's not just specific to cloud. Um, so for setting up that one um in your tempo configuration, you would have um your MCP server enabled. Um, so one of the pretty cool things there that they did, um, is that we, so if I look back into cursor, if I look

at, let's see, if I look over here, I have tools. I there's a separate temple one. I just currently have that one disabled. Um, but basically, they changed uh the Graphana MCP server to be able to go and actually proxy the Tempo MCP server. So, like if you're building your own MCP server, that's something that you can consider doing as well. That is pretty cool. Um, I

could be like, "Hey, um, uh, what are my services?" And then like, oops. Okay. So, it's taking its time. It's looking at graphana. It's looking at tempo. Um we can that's not exactly what I was looking for but um so yeah this we can see that we have the demo app and then we have tempo all and basically you can be just like um what traces are

having errors and you can ask just like a bunch of different things to find out more information about what is happening and you don't have to learn like the specific query language. you don't have to learn how the specific tool works with if you have a bunch of different tools out there, you would interact with it the same way for all of them. And that can help

you figure out things a lot faster. For instance, like it can tell me, hey, uh, this trace ID is having an error. So, if I go over back here and I go over to my traces and I put in that and I search, um, we can see I found one with an error. And then you can look like look into more information like operating system type. Um

that's using Node.js. You could have it. In this example, I don't have logs enabled. Um but you can see things like I have an internal server error and just find out more information that way. Um and then if I just go back here to finish the last um part. Uh there we go. Uh yeah. So these are the link at top was um what I've been using

as the demo. Um right now this tracing stuff isn't pushed into that. It's a PR. Um there's two other uh projects that are quite flushed out that have profiles, logs, metrics, etc. You would need to enable the MCP server since they're not currently enabled in those. Um but you can try those as well. Um if you want to play around otherwise um you can go to uh

play.garfono.com graphfana.com if you just want to look in general with how things are working with graphana. Um I'll share these slides online but like basically there were just a few things for like covering of like what we learned with uh Griffon MCP. Um like for one of the biggest lessons was like how sensitive models are like to design like are to tool design. So like tool descriptors

matter how they're written directly affects like how the model selects and uses them. um context window pressure is very real as the number of tools grow. We can we saw there were like 64 or something like that u or 67 um and then like as a result some tools are now disabled by default um and they can choose what you want um and then like looking at

this if we were to like revisit things um we'd consolidate probably more aggressively fewer composable tools with parameters instead of just like having it narrowly scoped. Um and then like looking forward maybe we do something like have like a hosted version um after things like authentication challenges are addressed. Um there was also with Griffon MCP um Joe who created it liked implementing it like directly because it

did not require like a middleman MCP proxy allowing us to just point claw directly to tempo for and he learned that like a key lesson was to present the data to the LM as simply as possible. So like he created a new uh endpoint that would have the data in a way that would be easier for an LM to be able to read which is going to

save time and money. So that is pretty neat. Um there's a bunch of different uh observability MCPS that people have created and that's a link that I've seen some of the other ones that exist out there and you could create potentially one for your own um system. But yeah, thank you so much for coming to this talk. Uh if you have questions um I will be at

the booth tomorrow and like till two o'clock and then if you want to learn about more on like open telemetry and things like that stick around for Lud Miller's talk after this. So yeah, Check check check. >> Hello. This works. Hello. It It's working, right? >> No. >> Okay. I I'll scream then. So, we are about to get started. Let's just give people one more minute. I'm

so happy, Tiffany, you gave the talk before me, so I don't need to explain what a telemetry is anymore. >> Who's here heard about telemetry? Okay, what about Prometheus? Okay, same people. Okay, that's fun. Let's get started. Hello, my name is Shannon. How are you guys today? Are you guys excited? >> It's good. So, today we have Ludma Mova. She's going to be talking and going over

your telemetry has a story and write it down. Take it away. >> Yeah, thank you. So, how many of you uh woke up paged at I don't know 2 a.m. to only to learn that they have no idea what is this alert about? What this metric means or like how what you if it's if it's even a problem. Okay, this makes two of us uh many of

us. Uh let me ask you a different question. Uh, have you ever built some dashboard or alert and it never worked or fired just because somebody broke the telemetry? They changed the schema. Okay. Yeah. Again, many of us. So, I'm glad you're here. Let's fight it together. Let's make it better. My name is Lumil Malova. I'm with Graphana Labs. I work on up until I'm a technical

committee member and I maintain semantic conventions. Um okay so if maybe you know Jerassi if you heard of open telemetry he's a active contributor and he likes to say that there is a lot of bad telemetry out there. Well, not of it is bad, but most of it is not documented. And even if it's good, like being undocumented means that people cannot really use it. So, let

me let's let's see how people actually document stuff first. Um, and maybe it's too small, but I'll we'll go through it a little bit. And I'm not trying to dig at the project who wrote this documentation. I really appreciate they did it. Um, but let we we're just looking at the exhibits what people in the industry do. And they have a metric name. It's percent sedular task

submitted. And it's the name is dynamic and will be resolved at runtime. Good luck with that. Um, but they documented the metric and it's actually useful and they have a list of keys, tags, dimensions, attributes, labels, you you name it. But essentially this is the dimensions of the metric and there is a description and they even say which possible values this label can take. Okay, let let's

take a look at another example. So this is a popular messaging broker and these are their client metrics and you can see for example they have a producer node request latency average it's gauge it has label node ID and it has description pretty awesome let's play it out um uh like well sorry let's summarize then the matrix they have a name they have a description they have

attributes or labels, tags, whatever. Um, we will use this example of storage client operation duration throughout the the talk. It's just an example. Don't think it's some sort of real thing. Um, okay. So, we break down this metric by some important uh things. Upload duration is different than download duration, right? So we kind of want to leave it as an attribute so we can later on break

down by upload and download and we want to know where we were uploading to. So we record server address or report as the end point where the storage leaves. Um okay so this is the metric 2.5 milliseconds whatever upload. Oh, it's upload duration. Upload 2.5 milliseconds. Is it good? Is it bad? Like what what's missing exactly? Yeah. So if we do the percentiles, if we see uh

what this actually is, we would see that 2.5 was P50. So 50% of my requests are completing under 2.5 and P 95 95 of 95% of my requests are completing under 300 So what's missing? Okay, so if we filter down to successes only, turns out that it's not 2.5 P50, it's 270 milliseconds P50. Um, so the metric that does not have indication of error and success is

meaningless. And if you see a metric and you don't know what it is, if you build your dashboard, maybe you don't have errors. So you build your dashboard without errors. and should that like your your your dashboard becomes meaningless. So metrics without good design and the good documentation are going to lie to you. Um they make no sense. Okay. So what should have the what how should

we define this metric? So first we need units. Every metric should have a unit. Like this throughput of 500 doesn't make sense. you need to know bytes, requests, kilobytes, whatever. Um, the metric should have a type. I I I actually cheated. There was type in all of the metrics that were uh in our examples. Uh, but this one, if it's a duration, it should be a histogram.

The histogram allows you to record not like just one data point, not an average, not a minimum, not a maximum, but the distribution. And later on at the query time you can kind of see uh what was the distribution of your request rather than meaningless P50. P95 is meaningless in the same way. It's just one number. You need to know the distribution. Okay. And like it's nice

to know the types of the attributes. Uh it's not important for Prometheus people. They only accept strings. But we are modern observability stack and open telemetry. pun intended. Um, uh, we prefer typed attributes and for example, server port is an int. If you pass, uh, whatever gibberish string, we wouldn't like it in open telemetry. Most importantly, we want error type or some other error categorization and lack

of it, lack of this attribute on the metric means it was successful. And from metric like this, you can derive throughput, you can derive latency, you can derive error rate. There there is one metric to rule them all. Um, okay. Done with metrics. Let's talk about logging. It's my favorite signal. People use it however they want. They write human readable messages. They write everything. Uh, I like

structured logging. Uh, because you don't need to parse it after. So, um, here we are trying to design something based on structured logging. Let's say instead of recording a metric I want to record each start and each end of something my um I would say okay the download started there is a time stamp there is some context there is trace ID span ID this allows us to

correlate logs for the same um together and we can say okay give me all the logs for that operation give me all the download start and fail here we have three different events to describe what happened. Start, fail, and end. And this is how people write their logs. The method, you enter the function, you start, you write logger.info, something started. Then at the end, you write something.

And in the middle, you write logger error, something failed. Um, it sucks even if it's structured. Uh well imagine you want to find you you didn't record the metric you didn't record the span you you want to find the duration something took you can grab you can query now you want to aggregate it and you want to find okay all failed operations with this property now you

need if your back end allows you want to take logs table join it with logs table by the trace ID span ID find the event names and whatnot. Here is the query. AI wrote it for me. I barely read it. Please don't write it. You should never write it. Well, if you have to do, but wouldn't life be easier if we decide designed our logs differently. When

you operation ends, you log all the context together. You never need to join logs tables log table with itself. uh you put the duration there, you log, you can log the download start, but the severity like debug because it it doesn't matter anymore. It only matters if you go some somewhere deep. Um and guess what? It also results in lower telemetry volume because instead of uh duplicating

the context and writing two log envelopes, you write just one. and there is no duplication. Okay, so I hope since you came here, it wasn't too hard to convince you that designing and documenting telemetry kind of matters. Uh if you need some convincing, we can talk after. Uh and from now on, I want to talk about how we do this. Um and like the answer is kind

of obvious like how it's just a schema. Let's define schema. Uh what do we do with schemas? We validate them. We validate backward compatibility. We do code generation. I don't know. Uh there are so many things we can do if we have a schema. Um and we we already did it. We went through this in open telemetry. And since Tiffany already talked about open telemetry, I'm not

going to repeat her just a few words that uh this is the vendor neutral framework. For the purposes of this talk, we're going to focus on the instrumentation and collecting telemetry uh and we'll talk less about other uh the instrumentation produced uh the important part they export it like open telemetry SDK experts telemetry over OTLP open telemetry protocol it's just the wire format and there is much

more interesting things that happen inside application in EBPF instrumentations in other different components of open telemetry but for a second let's just focus on the OTLP it's the format many backends support that ingestion uh you just send data and it runs there. So what is under the hood of OTLP? it's schema uh well the the OTLP exists in gpc flavor. It exists in HTTP JSON flavor. Obviously

for the readability HTTP is better but essentially it says okay the span has this properties right it has tree ID span ID parent span ID name timestamps kind and attributes this is the strict schema all spans in the world that you create with open telemetry zip key jagger they pretty much have this properties uh and the similar schema exists for open telemetry metrics which is somewhat similar

to videos but not exactly but anyway so there is a data model that is strict all telemetry of this kind should conform to the schema uh this is boring uh because like all logs can conform to some schema we don't like most of the logs created by humanity I don't like most logs created by humanity anyway um so how you feel in details in the schema this

is what we call semantic conventions. It's like you using up on telemetry wire format you design HTTP spans in a certain way. If you create an HTTP span or HTTP metric of some sort, it always looks like that. Um so this is the opposite of the previous example where you see the values of things matter not the the the top level properties and um For example, the

name of this span, this is HTTP server span. It has certain pattern uh that makes it useful when you let's say you want to do top end operations. you grew by this uh the attributes you put uh like HTTP request method this string it's the well-known constant and whenever you record this attribute you record it in this way on spans matrix on HTTP non HTTP data okay

why do we need conventions and um this like obviously anyone can report HTTP P request in whatever way they want. What bad can happen from it? Nothing bad, nothing good e either, right? Uh like you if everybody reports things in the same way, uh we should be able to build dashboard once and everybody can share it. We can provide some standard experiences like this one. If you

look here there, it understands. It also understands semantic conventions. You see there is a get 200 on the top. It came from attributes because it recognizes these attributes. It understands they are important. And there are other things here that are driven by semantic conventions. Um so how we document this stuff and let's take a look. This is documentation and we'll scroll to HTTP. There are some notes.

Uh we'll look at and again there are some information how to record span names, status. Um here is the where the fun stuff starts. This is the client span. It has stability level. It has again some information about what it represents. It represents a try, right? uh it's kind of important distinction and it says okay it must be client there is status information and it has a

list of the uh and they we also say if it's required or not and we provide examples and there is the description and type so if you are implementing HTTP semantic you know how to do this and there are maybe aund 100 H different HTTP clients that need the instrumentation. So it's important to have one standard for them so that they do it uh coherently. Um but

um it's also documentation for consumers. So as a consumer of telemetry, if I'm creating an alert, if I'm debugging, I can always come here and I can understand what it means. And the important part is like for example if we click on the server address this is something that's defined um there is some document with beautiful pictures on how to populate it in different places but essentially

this is something common I would use this attribute when I'm talking about HTTP gRPC generative AI I would use server address to describe who am I talking to when I'm a client okay, early on we realized that it's absolutely impossible to uh just have markdown for this. It's not documentation, right? It does not scale well if we change the server address definition somewhere. we need to change

like the description of it uh in 10 20 100 places and it's not humanly possible AI or not AI uh there was no AI at that time uh but essentially it's not humanly possible to maintain this documentation without having uh some way of dduplicating and um expressing things in machine readable way. So we did it and we are keep doing this and at some point it became

um uh a requirement to write these things in in YAML. Uh so if we look into HTTP client span actually the markdown you've seen it's it's most of it is generated uh and there are some blobs of text written down here that appear in the markdown. they are like less structured. Uh but if you look into let's say attributes they are pretty much formally strictly defined. Um

and see here we are referencing this attribute. We are not defining it here because attributes are shared across different metrics and different uh spans. And I'm I keep looking at Ronald because he promised me to tell that Prome sorry that opens telemetry sucks. And I'm going to dig on Prometheus. So like Prometheus used to have attributes labels that are parametric I wish they did. Uh by the

way the thing where I'm showing you it's even though I'm talking about open telemetry, Prometheus uses it too these days. they are actually documenting their metrics using the same schema, the same tooling and I'm super excited that we can collaborate between projects on this and um share cool stuff. Okay. Um and from now on we are pretty much done with the theoretical part. This is the demo

part. Uh and I have a nice little convention that created here. It uses a new syntax that we are rolling out. It's beautiful. It's very concise and I'm going to show Oh, the QR code. Um, sure. Yeah, this is on the demo is on GitHub. You can play with it later. Uh, it should be mostly functional. It uses one build from my local machine and I'll fix

and all pretty much everything I'm talking about you can you can play with it. It's out there. Uh, okay. Uh, so let's design our storage client duration metric. Here it is. So it has unit, it has instrument, it has stability. So uh people really have a hard time when we change something in semantic conventions and we break them and like we we do our best to stabilize

things, but it's awesome to be able to express stability so that at least there is a clear contract what you can rely on and what you cannot rely on yet. So we put stability everywhere just in case. Uh okay. And it has a bunch of it references a bunch of attributes as we've seen before. And where are they defined? And we also define spans u and events.

We'll see it in action. Uh so okay. So this is the reference to storage operation name which is defined in the same file. They don't have to be. You can use as many files as you want. Each file just needs to conform to the schema. Um, okay. So this is where the storage attributes are defined. The this guys are not defined in this registry. So we call

like the all the conventions defined in in uh about something as a registry. The open telemetry conventions is one registry. This is our own registry. So, how come where did this attribute come from? Uh well, let's take a look at the manifest. So, there is a dependency in this registry and it mentions open telemetry conventions as uh something it uses and there is a very important thing

here called schema URL. So, when you export telemetry, it comes with schema URL. So, it's part of the payload. So you can receive telemetry from someone I don't know you a platform team and you your customers or teams within the company and they sent some and you can actually go and understand who sent this who created this and why is it so bad uh or maybe it's

awesome and you want to reward them. Uh I don't know maybe you want to well I should stop here. Uh okay. So the the schema URL tells you what registry it is and which version. Uh and this is somewhere you can major version bump. So you can communicate your breaking changes in this way. You can also communicate your stability in this way. And what's not possible today

but we are actively working on it. that will be possible in the next couple of months is that when you hit this URL you would actually get your definitions your actual telemetry. Imagine um us in graphana I cannot wait for it to happen. Um in our UI we can show tool tip explaining what this metric is when we show this metrics on the on the dashboard. Uh

and we can show the summary we can generate dashboards not just with MCP but with just deterministic ways because we know how the telemetry is defined. Okay. So this is a registry. It knows about dependencies. um it can get um attributes that are common ones in open telemetry from this registry. let's let's break it. So um I'm going to say okay my um storage bucket is not

storage bucket anymore. It's storage bucket one. Um something should break. Uh, and oops, sorry. and I'm going to do check conventions. And I'm going to put a hook here. I would not explain to you what happens, but we will see it later. So, it's going to give me some report saying that the storage bucket is not resolved for something. yeah. So it failed. So the tooling uh

that we use is validating the integrity of conventions within and if you reference something that doesn't exist, it will complain. If you put a type that is wrong, if you do something that does not align with the schema or is not properly connected, then it it will Um, but like if you want to show this as a documentation for your Oh my gosh, I broke it all.

Uh, if you want to show it as a documentation for your CEO, probably he won't be happy or she won't be happy to see this YAML as the documentation. So, you probably still want to generate docs from it. And just coming back to make sure. Yeah. we can generate docs and this is the place where you see uh tool called beaver for the first time I like

who who have heard about weaver before two people. Wow I'm glad I'm here. So uh viewer is essentially that tool that allows you to work with the schemas and it supports so many things. It supports validation. It describes this language um and it can generate dogs. So let's go generate some dogs. Okay, it will think for a moment and this is our dogs generation script. It just

runs Weaver. So, I promised to fix it. Uh, it's currently using version on my computer because we didn't release the changes that we made so far. Uh, and we give it the registry. We say generate markdown. This is our new stuff, the V2. Uh, and this is the output folder. And then I'm formatting it. Uh, I have a joke about it. I'll keep it for now. Uh

okay. So it produced something. Where where is it? I hope it did. Yeah, here we are. Documents. Uh so let's take a look. I don't know. Metrics. Uh I'm sorry. I I something wrong. Um, okay. So, uh, this is just the m the the markdown completely generated from the YAML we had. It's pretty much the same what we've seen in semantic conventions. A little bit less reach.

Uh, but guess what? You can uh render it however you want. How does it work? Uh ideally you don't need to think about it and you won't need to think about it because we will have templates hosted in a central place and you can reuse existing templates but today it goes into templates folder and there is a markdown target here and it's a bunch of ginger who

used ginger before okay on a scale from zero to 10 how much you hate >> Yes 10,000. So what why we why we are formatting after doing things because whites spaces in ginger is just pain in the ass. So you'd rather just let it write whatever you let it write and your uh formatter uh prettier whatever can make it nice. Don't waste your time on ginger templates.

So this is all the awful ginger. Um to be fair I at some point in my life I wrote most of it but uh I didn't write this. I just asked AI to adapt what I had uh to whatever changes we've made and it made so much better. You never need to write Ginger. You just ask AI to write Ginger for you and it's the perfect use

of Bible coding. It doesn't hurt anybody and you don't need to waste your time on this awful stuff. Anyway, uh stop with ginger uh blaming and but all of you have seen in semantic conventions is powered by it. Uh and it's pretty powerful. It can do so much you would not need to write any of it if you are happy with the default templates we will provide

out of the box. then this are docs. The docs are awesome. But what do you think about quad generation? Not AI. Fully deterministic. Um well, if you can generate markdown, why can't you generate let's say Python? Of course you can. So what can we do? And it uses all the same stuff, right? It's just Ginger. Uh and uh let's generate some Okay, it's going to do things.

This is a warning. Don't just fine. So here is our generated code. Um I'm going to start with how this code is used rather than what is generated. So this is my handwritten Uh this is my storage client and like for example this is upload operation. It's intentionally complicated. It's a multi-art upload. So you actually care about the duration of the whole thing and you want to

instrument this Um okay. So the all the instrumentation happens in this helper method because um I don't want to spread the instrumentation code and uh make it uh make it like create some noise in the actual business logic and it will it will do all the things for us. it will do the spans and metrics and also it will record exception events. Um so this is my

span. Let's jump into my span. This is generated. It came from the code. It's not it doesn't just start a span for you. It gives you the strongly typed well in Python but strongly typed API that tells you which attributes should be provided at start time and which ones you you will some of them you will provide later but essentially it requires you to give a span

name. It knows which kind to generate because uh it's part of the schema. it knows which attributes you're supposed to give it and it knows to take them here. Uh and then it starts it uses upon telemetry API to start a span and normally like if I wrote this without code generation I would like write something like this and you just pass a bag of attributes who

knows what's there. it's hard to review, it's hard to write. And this is the convenience API that kind of hints you uh at what you need to provide to it. Um same with metrics. Uh so we can just generate the metric name so that you don't mistype it when you report it. But we can also generate the dock string. Of course, we can do much more. We

can hide it all under the again strongly typed uh structure that would take specific attributes and it would tell you which are required which are not required. It can optimize things for you more fun when we report exceptions. I don't know if you've seen but there are multiple properties of exceptions you can report. You can take you can report this just the message the type the stack

trace and some other things. Uh so how do you record when exception happens? What exactly you record? Ideally all three things but maybe you drop stack traces at some point because they are too verbose or maybe you shorten them down. Anyway, so this thing understands that we are reporting exception and it takes the exception Python based exception type as an input and it separates it into multiple

uh attributes and we will have more logic in open telemetry saying okay don't populate stack traces on debug logs for example you can configure it it's extremely hard to go and update hundreds of instrumentations that we have to do this but if it's code generated it's trivial. We can also optimize the how and write very complicated code here to make it super efficient. We don't want our

users to do this but if we code generated then it's right once and reuse everywhere and it just one quick code generation script brings all this performance optimization to everybody. So, this is quad generation. I've shown you a demo, but now this is the the my favorite feature. I'm sorry. I'm so excited about it. Okay, you've got all this telemetry. Maybe it's documented, maybe not, but you

don't know if it complies with the schema, even if the schema exists. or you just got an awesome application from a sister team and you don't know what it produces uh how bad this telemetry is, what's in there. So we can actually run beaver and it will listen to telemetry and it will validate if it's actually conforming to that schema. And there are many ways you can

you can write it. For example, you can use it as part of your CI/CD. Uh you would essentially start the Vaver. It will listen to stuff during your unit integration tests. You would stop it at the end and it will give you a report uh and it will tell you about violations that are that it found. Let's say the attribute is missing in the metric or maybe

you're using some deprecated stuff. Um and so on. uh the in this demo we're going to emit our TLP logs. So V will receive telemetry and if it if it sees something bad it will notify you through logs it will write log to your back end saying that I've seen this bad thing and you can run it continuously in production and just pay attention you can even

alert on this. Uh so let's see how it finally start demo. this is my demo script. I also need to start load test and we'll go to my logs. I'm sorry. I'm for some reason not getting what I'm supposed to be getting. Huh? Finally. Okay. So, uh I'm going to zoom in probably. Uh there are two services running in my demo. The one is the demo itself

that uploads and downloads stuff. It reports some exceptions that we've have seen when operation fails. And we've see a service called Vaver. Uh and if we query for it, we would see uh this uh this is uh the attribute is missing. It's violation. It's coming from the service. Uh the language is Python. Uh and it comes from a span with name upload demo. So I can kind

of go back to my code and I can find that span uh terrible violator that uses attribute that's not documented. Of course, we wrote it by hand and the constant name is wrong. We could have used storage object key not storage object and if we used the con the constant autogenerated by our scripts we would uh never run into this problem. Uh it can do much more

but I don't think uh time to talk about all of it. So okay so we have a beautiful schema. We have tools to validate it. We have tools to document stuff and generate code. The moment will come when you need to change something right. And um let's change something. I don't like the name of this metric. I think storage client operation duration too would be much much

better. And I'm going to maybe create another attribute for something very interesting. And it will be called storage. Maybe I'll put some characters here. Oops. I'm sorry. I'm not used to this keyboard and I should have figured it out. wonderful. Um, so what we just did, we added some crap and we also added a metric. Uh, but we removed the metric. There used to be storage operation

duration. There is no more metric like this. If we generate code from this and god forbid the people who use this generated code don't update theirs, they will run into runtime exception. There is no metric defined for this anymore. Uh and if somebody starts reporting this metric, it will break everybody who relies on the presence of the previous one. So this is the time where if you

have to do a major like if you have to do a breaking change you also do a major but you need to know when it happens. So the check convention story. So let's let's see what the check would bring. And we'll we'll take a look more at the the parameters because they are important. But here it is. It says that this metric no longer exist in the

registry. Meaning this is a breaking change. So how would you handle it? You can deprecate it. We have schema allowing you to deprecate it. You don't remove it. But we actually can notice when you remove something and we can help you not break people by uh running this check. So how does it work? let's get back to our make file and here it is. So this is

a viewer registry check. We provide a list of policies and we provide the registry and we provide the baseline. So this is just my main branch of this repo. I could provided the tag uh of specific version so it's more stable. Uh but for the demo purposes this is fine. The coolest part and this will happen with templates. Um the coolest part that happened with policy is

that you don't need to write this policies because you can just provide the central ones we created. You don't have to uh for example the the this backward compatibility. If you don't care about backwards compatibility, don't run the check. We have a naming check and my bad attribute get flagged by it. But maybe I didn't save my file. Um anyway so and I'm also providing a custom

policy. So let's take a look at the custom policy. Uh here it is. I created a policy to detect unused attributes and this is called regal. It's like the language and uh there is part of it's part of CNCF ecosystem. you can define your policies and uh run them on top of your data and this is a funky language but it works pretty well uh and it's

used by Kubernetes and um multiple projects and uh companies to uh ensure their data is following uh like this is policy enforcement mechanism so I can write this reg it's just a code AI is pretty good at it again uh but essentially I can say that if um I I have an attribute that's not used, I'm going to flag it. I think it's a violation. You can

think it's a warning. Um but that's the beauty of custom policies. You write your own destiny. I'm just going to make sure that we see proper uh errors for our bad Ah, here it here it is. So, this is unused attribute. I don't like them. And also, it violates our naming conventions because it contains some um the dollar sign. Um again, it's not enforced by the protocol,

but it it's enforced by uh for by open telemetry for itself. You can follow this practice but you don't have Okay. So this allows you to know when your stuff is breaking and you should use SER to uh communicate when you have breaking changes. The schema URL thing has your version. So somebody receiving a new major version at least knows that there was a breaking change. Um

and Yeah, we've just seen the demo for so we are at the end of the talk and let me give you a short summary. Uh, so the telemetry schema is your public API. If you produce telemetry, however you treat your public APIs, you treat your telemetry schema, you don't break people. Uh, you make it nice and beautiful, easy to use. uh you enforce its quality and backward

compatibility with some tooling. Uh and we in open telemetry have language for this and tooling weaver to actually make it happen. Um if you use Prometheios you can just use the vaver. If you use something not open telemetry or prime medius, maybe you can leverage some concepts and if you are happen to use VER, we would love to learn about your use case. It's a very active

project and uh we are very interested in what you like, what you don't like, what you find useful, what's missing. Come say hi. We have CNCF, we have a Slack channel on CNCF Slack. You can find us in the repositories uh in open telemetry. We have uh meetings um every Monday 7 a.m. Sorry, every Wednesday 7 a.m. Pacific. Um and if you want to contribute, we would

be happy. Uh if you like Rust, it's all written in Rust. But if you don't like Rust, there is still quite a lot of things that you uh could do. For example, the code generation. Um if you want to generate for go you need to first write this very nice idiomatic go code and then tell AI to go generate ginger for you uh to to to match

it. There is more stuff the life check is something that uh can evolve a lot. We have an MCP server so you can chat. You can say okay is there something in semantic conventions that is used to record errors and I wish people uh who asked questions first asked this uh the their clawed with our MCP server attached uh because in a lot of cases yeah you

cannot find it with exact search but if you use uh LLMs they will find uh they will answer most of your questions anyway. Uh yeah and the cool thing we can do we can migrate from one version to another. So if uh we can just uh generate a list of transformations to apply to telemetry. So if you are making a major version bomb but your telemetry is

compatible we can kind of uh transform it into the new version. And this stuff also needs love and it's has some enormous implications on the industry right now that we we we do And that's it. I really appreciate you coming and I hope you learned something today and if you are interested in using it, I I would be happy to help you. >> Hey. Um, so this

is immensely interesting. Um, sort of we're on a journey to do structured logging like this year for a lot of our legacy systems etc. Part of that obviously is defining good conventions. I'm glad I came to this talk because now we have some semantic conventions that I can sort of jump start my folks in like saying hey this is like a good place to start. Um one

thing that the semantic conventions and some of the automation that you showed seems to lend itself to is like this could be used for static analysis of existing code to say like what's your sort of like you have quality gates for test coverage you have quality gates for security stuff. We could say like what's our telemetry coverage on whatever for metrics whatever. Is there tooling that exists

for this now that plugs into this? Is that something that exists? Is that something that's on the road map or >> um by the coverage you mean okay there are this many operations um this >> right so we have this many like API like endpoints right this many calls and then what's sort of our existing telemetry coverage like what log statements are generated what metrics are generated

do they conform to the standards here if the path through the entire application does it hit these different logs what would the projected coverage be so when somebody puts in a PR it's like you know does it meet the quality gate right Oh, versus actually saying, well, we're going to run the output of it, then you have to basically hit your entire surface area, make sure that

every sort of code path that is executed by sort of your like integration testing, and then you're listening to it and seeing the telemetry that comes out trying to catch it all. But you can do the same thing on the static analysis side and say, well, you have a schema, it's well defined, you know where your log statements are basically based on the framework, you know what

a log message looks like, you can see if those things exist in it. you know, this could be pretty easily integrated with like static analysis >> Yeah, that we don't have anything like that, but it sounds pretty interesting. No. Okay, let's give a round of applause You got to hold it for a second. >> You may want to go ahead and talk. Hello, my name is Shannon

and welcome. Today we are going to be talking about open search, the open search path to search and observability observability sorry uh with Enruda and Prattik Shenoi. Let's give them a round of applause. Hello. Hey guys. 6 PM. So, we'll take keep it more talkative and we'll talk more, ask questions, and hopefully wake everybody up. So, how many of you guys know about Open Search? Before we

like jump in, how many of you guys know open search before you jump? Okay, majority. Anybody who doesn't know open search over here, do you guys know solar? You guys know lucine? So all of these are kind of search engine. Lucine is a library. uh for the best way I try to think about these things is uh lucine is a way to do unstructured search or pure

search and open search is a list of recipes that just come together in a server and give you an entire ecosystem of doing applications around it. So in today's talk primarily we'll go over like what is open search where it belongs why do we call it open source all the facts and numbers and any questions you might need to have before trusting a product and then uh

the core things in open search which are search and vectors and we'll focus primarily on observability AI observability agentic observability and how do we observe all of these things and investigate debug and how does the open search ecosystem help you in Everything cool. Uh let's go next. So open search by just basically the numbers of how much goes into the project. Uh the project is at open

github/opensarch- project. Open search- project forms everything part of the open search umbrella. There are more than 130 repositories. There's a lot in there. It's easy to get lost. So we'll go through some of the highlights and high level structure and where everything is located. There are more than 90 partners. We'll see some of the key partners and what they contribute as we go through. Uh we've been

launching a lot and lot is like every six weeks something comes out and we're always like building or launching. We ourselves working in the teams are like confused what are we doing and how fast the features come out and while we do that maintaining security uh maintaining the quality all these things go around and there's a lot of things to also learn and uh participate over there

like download numbers are just staggering like monthly views number of members that contribute on slack collaborate over forums all these numbers overall look as a project that is mature open search. Go ahead. Sure. Perfect. Thank you. So open search as a project started independent in Amazon. it before calling it open search was called open distro and open dro for elastic search then open search and recently 2024

September the project was donated to the Linux foundation now the entire project all these repos and everything around it runs as the Linux foundation project and it's been great to get the community contribute and heavily participate and love the project Uh here are most of the partners and members that work and contribute. There are many features directly contributed by these members. We can go through some of

them as we uh go ahead. The open search project as a whole with its 130 repositories and major ecosystem fundamentally gets divided into three core like shippable products. First is open search core which is a search engine that powers the entire stack. That's the data layer. Then you have a UI layer which is called open search dashboards. uh open search dashboards runs directly connected to open search

where you can visualize do vertical stacks for security search observability and then just base analytics. It supports various based languages uh native languages for search plus SQL and pipe processing languages which is a observability log analytics language. Then you have data prepper. Data preper is a injection ecosystem. Injection processing, map processing, ingestion to metric processing, uh logs coming in, traces coming in. How do you convert them

into service maps, topologies? How do you get these topologies labeled and tags and convert some of them into metrics? So all the views are powered by the processing happening at the injection tier. So most of the use cases for these three platform components fall into search observability. A IML applications is a holistic term but majority of the popularity of open search also comes from it being a

vector database. It's a highly scalable vector database that is used at scale for customers. uh it runs a suit of security analytic tools and a rule engine that is used to scan. Uh today we'll be focusing primarily on observability as a use case and the latest focus on uh the agentic observability workflows through it. search as a whole uh to cover overall it covers like full text

search, hybrid search and any search mechanism that we have come across I think we support it. Uh it's a very mature search engine and coming with vectors going to hybrid and semantic search is the next use cases that are maturing and have been used heavily in the platform. It supports like faceting, geospatial, if you do want to do anything with maps or geo tagging, mapping, map search,

all of the map uh layout views and mapping use cases are highly sophisticated in here as a developer use case. Like you could go in there and customize it, create your own views, have maps on a dashboard, IP maps and other things are fairly powerful and maps work with multiple layers. So you could work with mapping use cases at scale and they're custom to your use cases.

Uh for a IML applications, multimodel search is one of the key pieces. Uh vectors is what powers this entire workflow. It has support for bring your own LLM. You can connect it to any open LLM hosted and you can make sure all your use cases within the engine can be powered with AI workflows. It supports something called as ML commons as a plug-in with which you can

add remote LLMs and configure them with various parameters that you can start using in your primary workflows in UI and the APIs. So as a building block for applications, building block for anything you are doing, uh these things form a foundation for the new agentic From a performance standpoint, as much as we focus on features for the engine, the focus for the engine has always been from

the initial work in 2024 to 2025 when we released 1.0 to 2.12 just improve performance. So the core engine performance has improved quite a bit in that one year. There have been a lot of independent assessments going forward from like 2.12 to 2.17 it became even faster and with the latest updates uh we'll talk about the newer versions of 3.x and 3.0 coming in. Uh 3.0 was

just launched last year. Now from last year almost May till now we are like almost at the cusp of 3.6 launching in a few weeks. These minor versions also bring a lot of performance improvements uh in the core from a performance standpoint. There are also features being contributed uh by partners which fundamentally change the architecture of the system. Like for example, ingestion has always been thought about

in open search as bringing data into the engine. Uh work done in the engine now supports pull based injection. So you can have your data in a pipe and the engine will pull rather than you pushing the engine and now figuring out like where in the stack you need to buffer back off. The engine will just pull. And when you start pulling data you achieve much higher

throughputs. Uh recently we started adding gRPC and protobuff support. Arrow is supported in between for internet communication. Uh reader writer separation is greatly resource utilization on the clusters. So most of this is core to running pabyte scale workloads. So as much as the features are important, I think the engine has improved in the last two years dramatically in the recent months maybe this is from February uh

of such from the years from like uh last November to this Feb query latency has improved a lot more. Uh query types and aggregation types have been kept on improving. uh index rebuilds and migration use cases have been f furthered and if you see the trend for version to version it just keeps on going every version there is a focus on improving the core engine that's the

heart of the system uh let's jump into observability from a core observability standpoint uh log analytics is the foundational piece of the platform for log analytics uh we support the core search languages but fundamentally people use languages like ppl. ppl is a piped processing language. Uh you use a lot of Linux style commands to keep analytics on unstructured data. Uh we have a lot of demos. So

in the demos we'll go through like complete unstructured blobs of data. How do you synthesize that with variable creation at runtime aggregating them and working through use cases and investigative flows. So I'll quickly go through uh interrupt me if you have questions in between here. From an observability architecture standpoint, here's our primary lay of the land. You start with the hotel collector. We as much as possible

align to hotel standards and then allow you for unstructured data and the unstructured commands in ppl to process in case you don't. But the ideal workflows are all dictated with hotel conventions. Uh the first part in can be hotel collector sends data to data prepper which is our injection system which will primarily do service map topology uh metric conversion uh metric from log metric from trace tagging

and labeling and it may write again to Prometheus which is also running outside the stack but a core part of open search now since the last month it's just a new improvement that we started adding Prometheus as a core part of the open surgic ecosystem. You could also use open search uh hotel collector and exporter to get data directly into open search by bypassing the injection components.

If you're just writing logs or you're using Jerger traces, just go straight into open search. You don't need to process them again. And the UIs also are structured to work around uh Jerger schema formats and core uh native formats based on the open search exporter. More and more as we go through the years, we'll see a shift towards open telemetry collector being more and more native in

our ecosystem. That's the signal we've been getting from customers and users who've been using the ecosystem more and more. Uh a layer above it as data goes through the injection pipeline. Open search supports uh semantic schematized indices and unstructured indices. When there's a schematized index, we know how to process it. It is highly optimized. You don't have to do anything and it aligns completely with logical hotel

schemas. When it goes to the physical tier, we optimize it. If you have unstructured data, you choose to structure it and it will work with the programming languages and the UIs just as such. But the default views and workflows, analytics out of the box, those are more and more tailored towards hotel. The layer on top when we build UIs, the stitching layer between the UIs is always

the pipe processing language. This is a language that we use to build our UIs. This is a language that developers, operators, everybody is familiar with. So the human interaction is always the language and the UI interaction to the lower level is also This gets us velocity to build new features and ship faster. So from a open search user experience standpoint, you start with a new open search

shim. If you have been using open search or seen open search in the wild, you may not have seen this. All of these UIs are kind of under feature flags which have been in testing for quite a while. I think these feature flags will be removed in the next major version of open search dashboards. But uh we'll also be talking about an observability stack where all these

pieces come together and in the observability stack these components are pre-wired for all the new workflows out Observability use cases support correlations primarily as a core workflow. Every UI element focuses on how do logs correlate with traces or traces to logs. Uh metrics correlation is just coming up as we release Prometheus. SLOs's SLI will also start coming in uh by April. So this way we'll focus more

and more on correlation use cases and workflows and deeply integrated into the data model. uh pipe processing language as I said this is the core workhorse of human interaction and of our UI interaction it supports lot of complex commands these commands we'll go through in the demo Prometheus uh integration was recently launched open search now can connect multiple data sources multiple Prometheus clusters uh you starting April

there are multiple RFC's open for this is a unified alerting system. So unified alerting will show multicluster alerts from Prometheus and open search together with SLOs's and SLI built on top of it together in a single pane. uh majority of these workflows will also support AI or agentic integrations where you can have a conversational chat connected to the LLM of your choice in the open source. Uh

APM and tracing views are highly specialized views for services debugging dependencies fall trades. Uh we'll run through a demo of the workflows and the topology maps and how they come uh let's start with agentic AI obserability uh and we'll sometime spend time some some time over here to go through the architectural pieces how agentic observability is laid out in open search and how you can connect into

it. So open search today as a platform supports multiple layers of agentic observability pieces. So you can interact with it in all the ways you have been working with agent traces spans quering and debugging tokens writing queries over service maps or trace maps that you can see. Uh so let's jump in like a highle like flow from a complete flow. Uh the first step you need to

do is work with getting data in and this is where open search is starting to start create simple wrappers around open telemetry semantic conventions and open telemetry libraries. These wrappers may dissolve but today these wrappers help in ease of use. So you could get open search semantic conventions as a Python library or a TypeScript library. add a simple decorator to your agent or your LLM workflow and

it will start sending structured traces, eval spans directly into our ecosystem. So the getting started piece this is where it helps. It will start autodetecting many of the formats whether you're using crew AI langraph strands pick one we trying to support almost everything in the industry. So you can just autowire everything and get data in. Second step is normalize. So suppose you're not writing code you already

have agents out there and you don't want to change your code. That's where the geni normalizer comes in. uh geni normalizer is a spec and a project that we have proposed in collector contribution in data prepper we're trying to just move it upstream right now what it does is if you're sending telemetry today in production at any of the systems like ru lang graph any agentic or

lm systems they will just merge it and converge it into genai semantics so the UI tiers have a unified view of the dat uh So whichever format you're using, change your gRPC ports or HTTPS ports for the total collector. They should just start working on the open search platform for analytics. The third part is an interesting bit. Uh this is a tool that we have developed completely

outside the stack. Now the stack is a production grade pabyte scale system. It is heavy. Even on a led laptop, I can run it, but it is still heavy. It is like 2GB, 4GB and it spins up a lot of containers. The observability stack makes it easy, but it's still a heavy stack. So, what we have done over here is proposed a independent installer which will get

you into CLI based agent debugging on your laptop. It also spins up a UI for local You can integrate it in CI/CD pipelines, run evals over it, do trace analytics over it. Most of the tooling you can run independent. So this is how you run it. NPX agent health. It installs quickly runs up. Uh this is a dashboard you see. Oh, I'm have to do something. Give

me one second. And to go out of my presentation mode and go here. So here effectively let me just do this again. I just run one command. It runs, installs and starts a dashboard. Uh this could run in headless mode completely with the CLI. So you can integrate it in your CI/CD systems and do quality kits. It supports tracing and all the trace Its primary purpose is

also benchmarking and evaluations. Here are some sample scenarios how you can run benchmarks. LLM as a judge and various measures that will allow you to align to how are you developing skills or agents and how do you evaluate them. The data over here could be local or you could connect this to an open search backend running in production or you can run this CLI in your CI/CD

pipeline and connect it to an open search cluster in production and you can see unified data from your local runs, CI runs and correlate them with live traces or live workflows. Everything is available at the end in the platform. But this is an easy way for a developer to get started today. Let me jump in. The part four comes in is the middleware that is either open

telemetry collector or open search data prepper. Both of these components work to take telemetry from genai semantics in and convert them into service maps. Service maps are versioned and created as snapshots whenever there is a change. So with any new agent coming up, new service coming up, there's a new snapshot. You could like timeline and time travel between them. You could generate metrics from the service map

or metrics from a trace or a log, align correlation values between them and send all of the structured data into open search. Now these are different structures in open search. uh we have correlation metadata that we keep separately and all of this can be either provided or we autogenerated from the injection processes. So if you're running custom workloads you can just manually add those keys but for

now if you use the flow it just automatically pre-wires itself. The last part is the core platform where you have data analytics everything coming together as a single story. So this is our sort of five-part flow of agentic end to end AI observability. workflows in open search are used to observe agents but agents are also used to improve the workflows of observability where you could use chats,

conversational flows, natural language generation from natural language to queries. So there is nothing that you have to learn just come in front connect the right systems and you can get start working with the entire ecosystem with our natural language. So I'll quickly run over uh this is the SDK oneline hotel setup sigv4 gRPC all you need to do is run the resistor workflows or decorators start with

your code this is in code step two just pip install work with these commands and add it to your code and go ahead and quick start with an agent development workflow where you add workflow commands agent decorators tool decorators and the register function This is how you would start working with the genai SDK. The normalizer gets like open inference, lang chain, crew and many other platforms and

converts them in one flow. The quick start of it this went we went through a demo of agent health. Now all of these pieces in open search until like a week or two ago even if I say things and show a demo if you go home you will not be able to try it. It was very difficult because there were too many components. The 130 repos take

time to come together and there are so many configuration points. You have to be a rocket scientist. Believe me, sometimes I myself cannot get things together. So in the last few months and weeks, we've been getting together the observability stack to solve this problem. The observability stack overall primarily runs as a single line command. Uh the stack lives It would get the workflow over here online. It

would do sample AI agents and the entire open telemetry demo pre-wired as an option. Hotel collector as a front for the stack. So you always bring in data to You have data prepper in line for service map and topology processing. The Prometheus uh workflows are well integrated so you can start with it right off the bat and open source UI is available on the top. So when

you run a this is how it runs at uh you run the shell command it would effectively check through your environment go through asking a few questions. Do you want open telemetry demo? Do you want example services? All of these will get installed and it'll give you two important things. one an endpoint to send data to if you want to get new data in. This will be

HTTPS and gRPC exactly similar to the hotel injection endpoints. Second, it will give you a UI. Just log into the UI and you get your flow. So the next part of the demo uh I'll invite Pratik who'll go through a awesome demo of investigating things and ask us questions. Let me read this here. Hello everyone. So let's get into the exciting part of the presentation where we

are doing the demos. We'll do some live presentations. We'll hope the demo guards are with me and everything works as expected. So thanks any for running the uh observability stack. Here you can see we have some basic o basic o setup and the dashboards is spinning up with all the components and the pieces that you uh saw in the original architecture stack. Um just to revise things,

you have your microservices that are running on your infrastructure. You use hotel collector to send all telemetry logs, traces, metrics. It comes to our data preparer injection endpoint where we get out logs and traces into open search and the process uh traces go as red metrics into Prometheus. And finally, observability uh dashboards uh workspace will showcase everything that you have. Let me go one step back and

show the new flavor of dashboards. Here inside the new flavor of dashboards, what you see is several verticals that we have created for curated experiences. Observability get the gets a curated experience with a set of plugins like anomaly detection, alerting, forecasting, logs, traces, metrics. And for example, if you want to do something in terms of search relevance, creating hybrid search pipelines, use our flow framework, everything comes

under the search vertical. So that way as a user persona who is just dedicated on the observability uh set of things on open search, you get a dedicated workspace. Now let me go inside the observability workspace. Here on the left hand side, I see a good list of plugins and features that we have. Uh today I'll go deeper into our discover log traces and metrics. Uh do

a tidbit of agent traces, agent dashboards and then also dive deeper into APM with respect to services and application map. There are more features I cannot cover in this demo today but feel free to go into our stack website observability.opensarch.org and you'll find the stacker working there. So let's dive deep into services. When I come to the services page, I have my injection pipeline already set up

in the stack. My traces are stored as observability data set. My traces are correlated to logs automatically. And then going further down, I have my top topology stored inside as a service map index inside open search. The third piece is the red metrics. All the red metrics are stored in Prometheus and the dashboard does a seamless job of taking out your topology, matching it with Prometheus metrics

and then showing it in a single pane of glass. Here I'm running the open telemetry demo which is an e-commerce website. Uh full disclosure I'm part of the demo sig and then I love the product there. What they have is a sample uh e-commerce website which will mimic clicks, checkout, payment of telescopes product that are here and then create telemetry out of it. Uh this is the

overall architecture how it looks like internally for the application itself. These are the multiple microservices running and then the best part about uh the open uh telemetry demo is the feature flag service. It has multiple knobs that you can use to mimic an um existing issue in your microser. For example, here for this demo, I have uh switched on the product catalog failure. So once we go

through the demo flow, you'll see some of the product catalog APIs will be So let's get get to the demo now. I'm here in my services page. What I see here in the application monitoring is setup is what are the top fall trade services that I have. What is the thing that is failing with 5xx errors? On the second tab I see here, what are the dependencies

parts of failures? For example, these are the services that are failing. But what are the root cause services? What are the dependency services which are causing the issues? And then below I see the whole set of service cataloges with different uh languages running on different microservices. Once I click onto a particular service, I land into this service overview page. This is the single dashboard that you need

to look at all the metrics that are related to your services. U here I can get the latency by dependency services. I would know what is the issue causing dependency service that is uh hampering the system flow in this particular service. And then further down I'll get request by operations uh availability by operations and fault rate and error rate by operations. These are operations for this particular

service. If you want detailed rate metrics on each operation, you can go into the operations tab and let's look at this particular flow where we were looking at the product failure catalog. If you remember, I switched on that failure uh flag there and I can see the availability for this particular operation has gone down. So if you want to debug this further, what we would do is

I would just click on correlated spans and see a list of spans for this particular operation on the front-end service and actually start seeing the To debug this further, I can just click on the span ID and it would land me into discover traces. This was one of the sidenav items that I told with the new dashboards. What we have built is seamless correlations into different plugins.

So you don't feel um these as segregated products. These are flows that are intertwined and made to help SR made to help SRS uh debug things easier. Now you can see this is a trace map. I can see different services for the user add to cart flow. It went from load generator frontend proxy came to the front end service finally ended on the product catalog service which

we enabled for failure. And here I can see the error. It says the failure uh flag is enabled and that's why this particular thing is failing. If you look at u if you look at the actual flows in a microser architecture you would actually find an error that you have put in a span or you have put in a log. You can also see correlated logs here

but I don't have anything for for this particular service. um otherwise you would see in this particular tab. Now going back to uh services, what you would like to know next is how these services are connected with each other. How these services communicate with each other. So that's where the application map comes into place. I can look at my application red metrics here at an high level.

What is my application health? How is it doing? I can also click on the application to blow up into a service map which is a full-fledged topology of all services and this showcases how each service is doing in terms of the health. It also showcases how each of the services connect to the connected to the other set of services. If I click on view insights, I can

look at a particular node, look at the red metrics for this particular node and and also go into view details that would land me into this particular UI that we saw earlier. Now going back to the application map, I can also group the services by any attribute that I have selected during injection. So here I can get using telemetry SDK as one of the attributes. I can

see all the languages that are there in my microservices. And for example, if I click on go, I would see all the go services that are here. Now let's back let's come back to the scenario where I have my u logs coming in for a particular service and I don't have spans for it. So how would I debug this further? For example the front end proxy in

hotel demo it has uh logs that it emits out and if I click on view logs I would see all the correlated logs for this particular uh service. The issue here is uh because I didn't instrument it well. I do not see the log level here. I don't see if it is an error, if it has a 5xx, 4xx and stuff. So what I would do is

I would jump to discover logs which is our which is our curated experience ppldriven UI for all of uh open search logs experiences. See the demo gods are not with us for this one. to take or just lost one or okay so we see the logs here for all the services what I could do is I could just click on a particular service filter down all my

logs for that particular service and then um what if I want to extract out a particular thing that's there as in a string this is a usual SR issue where you have ingested Everything in logs are strings. I would use ppl here to extract out a particular error code kind of mechanism. For example, here we have the HTTP status inside our log body. I would go ahead

and pull up a saved query for time. Here I click on the saved query. Once it opens, you can see that I'm extracting out the HTTP status from the body field. I can type this here and see all the 300, 200, 500, everything has come out of my body field into a new runtime field. Now, what if I need to do some analytics on top of these

logs on this runtime field? I can go ahead and add a stats query on top to first bucketize these individual HTTP status codes into 2x, 5xx and 3xs and then look at only this particular error code that I wanted to look at to debug my front end proxy logs and finally come up with a stats query to get request count by the attributes of URL path and

the stats class. So here you can see a heat map. uh our UI directly switches from logs to visualizations. When you use a stats query and it understands that you are doing an aggregation, you would rather use a visualization tab rather than looking at draw logs. Here we have the heat map. I can go ahead switch to a bar chart and easily spoton see that there is

this particular product call which is failing. Again the same feature flag that we set up initially is failing on 5xx errors. Um and that's how you root cause anything that is coming from just logs using visualizations building analytics on top of it. We also have easy add to dashboard features and other pieces that comes with regular opens dashboard features here in discover. U the final piece is

discover metrics. So as I mentioned our red metrics are going into Prometheus. But what about the application metric itself? I can come here and just search for a particular front-end metric that I have added as a saved query. What it's doing is it's running pro uh promql queries on the open source dashboards UI connecting to my remote Prometheus and using the same visualization library that you you

have seen across logs and traces uh but with a new style where you are everything seeing from promical query result. So now let's go on to the hot topic which is the agent traces. uh Anie mentioned that we have agent tracing as part of the local agent health setup where it runs on local host but this is more when you have the pabyte scale of agent traces

coming in from different AI components in the topology map we have a combination of your APM views along with your agent maps that are running here I have a small cluster of agents uh which is a sample u workflow where I'm using a travel parameter agent which is calling the veger agent and trying to create a plan uh for my trip. That's the basic uh workflow that

is handling this particular thing. Now if I go to agent traces, I can see this is all using genai semantic conventions as part of uh the UI. It just builds on top of it. U here we can see all the agent traces that are available. U if I click on a particular thing, I can go down and see what are the LLM calls, what the tool calls

and other pieces. uh I can look at spans itself and see if there are any individual errors I want to look at. If I click on an agent um span uh I can load the whole timeline on chain of events on how did the agent took the input, how did it take the input and then come out with the final result. Here you can see the main

input was plan a trip to Paris and then the final output after all the chain of events that it did was it's a great choice and this is a perfect weather to go around. Uh if you uh have worked with agents before what it ends up doing is it at times take up takes up lot of tokens and you don't know why. Uh this is where our

agent trace map helps in to see if there are any loops that are being formed in in your trace uh execution for your agents. You can also zoom into a particular agent and look at its particular input output. What was the sub agent doing? So here during the trip um execution trip plan execution flow uh it asked the weather agent the main agent parent agent asked the

weather agent what is the weather in Paris so that it could plan my trip and the output was weather is cloudy but it's still good enough. So and also the final thing is we have a timeline view similar to regular traces that we u do in APM style tracing. Uh so that that's how you can use open search even to store agent traces. Get a summary of

your spans tokens the latency look at your input outputs at a glance when you are doing this agent explor agent trace exploration. Finally all this boils down to dashboards. I have a pre-built dashboard here which is the core crux of what we inherited and what we have been holding on to with the new dashboard setup. Here we can look uh this is all using the agent traces

and creating dashboards on top of it. I can see what are my LLM requests per model. What are the tool user statistics? U what is the token per model call and then further down there is like what are the product reviews that are coming in from my open telemetry LLM. Um yeah so that's how we have things um also as a custom dashboard from your existing spans.

Finally I would like to give a glimpse of what we are doing in um coming releases. We are building more comprehensive dashboard visualizations. Uh this is a new dashboard flow that we are building with new set of visualizations. So anything that you see in 3.6 and onwards will have these kind of um metric gauges. um and also some part of variables up coming on soon in our

road map and we'll also have combination of staged charts like these so that's it so it's easy to move a cluster of visualizations across that's what we had for the demo um coming on to road map we have some new features that are coming out and that are also contributed by other uh companies like Uber is working heavily on tsdb native support for open search. We have

uh unified analytics developer experience. We have unified alerting that Anie mentioned earlier about SLOs's and u alerting from Prometheus being an integrated uh UI for open search alerting as well. And then finally we have natural language quering and agent observability and lot of additional things that are coming up. We have an observability task assign group that we request all of you to join in. It's open to

all. uh we have monthly meetings where we are running a project board on what is next onto open search. If you have any features that you want to bring in open search feel free to join the tag and let us know. If there is any issue with the existing stuff please let us know as well. Yep. You can find us at GitHub, Slack. There's lot of blogs

that every contributor pushes. There are a lot of maintainers here. And then finally, thanks. Use a QR code to go to our website. Thank you so much. We are open to questions if there are any. >> Yeah, please go ahead. >> Uh did you Yeah, I can ask again. What's the cold start time and minimum memory footprint currently for the latest version of the core? >> I

see the core starts start time for a single node cluster is just seconds I would say. Uh but if you go into multicluster setups there is some time for network discovery between the two nodes. So it can go into minutes if you have like hundreds of cluster I believe like hundred of data nodes for one open search cluster and the core memory footprint for a single open

search cluster is around 1 GB I think that's the least recommended you can go below but you will not be able to take in any production flow on it plus there are it also depends on how many plugins and jobs you run on top of open search open search has a cluster of plugins uh starting from injection pipelines processing pipelines alerting anomaly detection jobs. So as in

when you increase these jobs uh the memory footprint also increases. >> So that's the point I would like to highlight. >> So minimum recommended would be like two gigs. >> Two gigs is good enough. Yes. >> Y if you're using anything for your local stuff, 2 gigs is good start. >> But to add, I think it's such a composable system with plugins external to the core and

modules inside the core. the way you tune it and what you want to do with it, it could get to an embedded system if you want to really really bring it down and what you want to do >> I know there was one of the developer advocates that we had in the team, he was running open search on Raspberry Pi. So it can go as low as

that. It has no plugins. It's just a pure >> Cool. Uh, do we have any other It's less of a question. I I really liked your visualization the trace tree for agents. >> Where did it come from? Like if you can open it and and to walk me through it. >> Yeah, sure. Looking for anything in particular. I mean when we went to agent traces and you

shown this diagram like the the not the GAN chart but uh there's three uh for multi-agentic flow >> I think I stopped my demo >> oh sorry >> that's fine we can take this yeah we can take this offline but uh the visualization we are using internally is echarts that's our core crux and what it uses is the duration of each span or agent span that it

took and showcases it along with the token count which is sum of your input output per agent span almost all of it is just pure genantics stitched together we just put it into our logical representation uh with very minimal processing but mostly it's the logical sorry logical representation coming in is geni semantics and the physical implementation we manipulate it a bit so You can query and stitch

things >> Yes. Yes. Yes. Yes. Yes. Yes. Yes. I do have to thank a lot of the people in the UX team in the company and many other people who've been breaking their head over it. >> A lot of menas working behind this. We just two are present in this stuff but there's a team of people behind this. Do we have any other questions? >> say like

the few things that we've been working on next is primarily going to be focused on multi- aent workflows which are very difficult for us to debug. That has been one of our focuses going next. And then one particular thing I don't know if it'll work and if you have over here but like a hidden kind of Easter egg Just put a comma over here or semicolon over

here and add something. It will automatically convert multi-line and do like multi- data source multi-queries together. So as a human it's not intuitive but once you get it when you type it just add a semicolon add 10 promql queries it just plots everything together. But it is really powerful if you start with an agentic or an AI flow and the AI just spews everything together with comma

separator and everything just works. The AI does not have to manipulate the UI. We don't have any questions. I think we are good. Thank you so much for being a lovely audience. There's a map. I close your phone. check. What?