Harry Kimpel: It's Game Time at DevDays

This talk focuses on integrating observability concepts with an engaging gameplay experience. The speaker, Harry, introduces the idea of observability through the analogy of a car’s cockpit, emphasizing the importance of constantly monitoring performance metrics. He discusses how to leverage New Relic’s Application Performance Monitoring (APM) by implementing it within the game 'Dangerous Dave', created using Python. Through this hands-on session, the audience learns to collect telemetry data such as logs, metrics, and custom events while playing the game. Additionally, Harry highlights the use of Open Telemetry as a vendor-neutral observability framework, showing how it can be integrated into applications for comprehensive data analysis. The session serves as a unique blend of fun and education, providing insights into monitoring software performance.

[Music] ladies and Gentlemen please welcome our next speaker Harry kimple presenting the topic it's game time at Dev days learn observability Concepts and techniques and play a video game at the same time all right um thank you for having me um I hope you all had a good lunch um you know energized because you know we will use that energy to play a little of a game

today um so welcome you know this talk is all about you know playing a a game having a little fun and you know learning observability Concepts at the same time I hope this is interesting um for you my name is Harry I'm based from in in Germany Southern Germany in Bavaria um and I'm glad to be here first time in vus so um you know quite excited

to be here um so when I think about you know monitoring or the bus word the new bus word observability for me it's always about like you know kind of like that car cockpit right where you see your gauges you see how fast you're going you know how many many issues you may have when you need to go to the service again and so on but you

know it's not only in the case of a failure that you need to know what is going on you know you always want to know how am I you know going too fast am I going too slow you know is there any issue do I need to hit the gas station and so on so all of this information information is really important for me to kind of

like observe what is happening in you know my car but also in my for what we hear about is like for software right and so um I'm from New Relic I will show a little bit of you know our um concept of what we can do like in in such a gaming situation how we can monitor that but I have something also at the end you know

you may have heard about the term open Telemetry which I'll also focus on which is kind of like a vendor neutral approach an open source project um that I can also share neic basically is a gigantic you know Telemetry time serious database um that ingests um any type of data into our back end you typically have you know applications you have digital experiences like you know real

user monitoring your apps like IOS and Android apps that you can monitor and observe your environment your infrastructure um and we have various kinds of apis in place that you can also use you know in case there's no standard integration you can also use that to in um you know ingest any type of data so really you know neic pure SAS service you know but you know

gigantic Telemetry database um that you can use um you know various types of Integrations um all right but this is not what we are here um for today um I thought you know it would be quite great to have like a a really interesting approach and you know not the typical you know uh eShop architecture that you can use to observe all your microservices and stuff like

that which is also useful it shows the concept right but what we here today is kind of like I wanted to show um a little game you know maybe some of you who are a little bit older like maybe me remember this this is a game called Dangerous Dave which we adapted um we kind of like you know some people fored it as a python little python

application and we you know adopted that and used that as a to to showcase um you know what we can do with um such an application it is not the typical you know web application with like a web front end so there are some challenges if you know you don't have like a front end you know web front end that serves HTTP requests back and forth so

um but we'll go through this and we will I will show you um you know how to get this um get this done um the objectives so what is it that I want to show you in that little game you know we will use APM application performance monitoring to show you you know how to you know maybe ingest and you know observe different methods of that game

so of course that that game is also implemented it has various kinds of methods and that you can use to you know um show the the screen you know and you know move the player and stuff all of this uh is something people will observe um logs um you know typically you know the Telemetry data that you collect in in an application is you know metrics events

logs and traces um logs is of course also something we I'll use a standard logging framework within that python app that um we also use to send in any um logging information from within that game and then also some custom events um custom events in in The Relic term is basically just a point in you know an event an attribute in time that you you know maybe

want to send in it is typically used in a more you know traditional scenario like imagine a retail application where you know your customers can put something into the cart um and you can add some additional custom attributes like maybe the product ID or some other um things that you um care about and maybe that help you then analyze in future if you have an incident or

you have an issue to observe what is happening and then finally we will you know visualize all of this information in the in the dashboard to show you what this um what all of this looks like um so basically we want to get from from here on the left hand side the game into like a a custom dashboard that will build throughout the um this session as

well this is you know a Hands-On session you know if you want to play along with me um this is the link to the repository um there's some um you know guidance and some read me um as well if you have an oric account you can use that directly um and you know play around with it there's some you know install instructions on how to get started

it is pretty you know only a couple steps that you need to do uh but if you want to follow along this is the link to the repository that I'm using also as part of this this talk talk today all right um why um is there some some manual um you know intervention needed you know typically you know we as neic at neic we also try to

make it really easy to get started so we have typically some guided install you can use the UI to you know do click click click and then you get like some instructions on what it is you need to do with your application your infrastructure what whatever it is you want to instrument um but in this case today you know it is not that easy and I I'll

show you what what I'm um mean by that and then you know if you know the guided install the typical you know getting started guide is not helpful you know we provide also some you know apis some sdks that you can use to inside of your applications to do some some custom instrumentation all right um with that um let's get started um I will switch back and

forth I have you know my visual studio code environment here and my terminal where I will run the application but basically the first thing that I need to do you know this is kind of like a custom you know python backend application so our agent you know will not discover anything automatically so what is it what is it that we need to do so if we install

the agent is basically um in this case it's a pip um package um that you can install we need to register the application inside of my of the the code right so how does that look like um I have the Repository up and running here in my visual St Visual Studio code environment the main file um that is you know most importantly is this mainor fun. pi

and you can see you know I have various branches that I prepared and just to make it um easier so the first thing that I need to do is kind of like importing that that agent um you know it's a it's a package that I'm that we're using and then um there's some form of you know configuration that I can apply in general you know all of

the neur Relic agents or also you know other forms of Agents like open Telemetry they allow some kind of configuration and for us it's the same thing there's this nur. in file that I can use to customize you know what it is I want to monitor you know I can give it like most importantly um a name right how do I want to you know see that

application in my um Telemetry back end and so on um and in general you know we have to use some kind of license key um of course you know you need to send in the data our API expects to um have some kind of API key so we did the the import I you know um added the um the initialized stage with the um in file and

then all the way at the bottom we have a couple um things that I can um do here and this is kind of like I'm registering registering the application which means kind of like that I'm sending a request to my you know Telemetry backend in this case our on Relic um backend application and say hey someone wants to register an application I want to send in Telemetry

and this is the the Target that where all the Telemetry like the logs metrics traces and so on um should go so if I would start this application right now you know it should work uh quite fine so I can you know start it with the Python 3 command you know you can see that um you know similar to what they had on the screen I can

play along with that game you know collect any items um typical you know jump and run game if I oops if I you know collect the Crone I can then go to the door go to the next level so this is basically the the game that we're looking at all right so like I said I mean right now I would maybe see the application you know being

registered but I don't see any um any data and this is where um then the next step comes in where you know we need to actually you know Identify some background tasks and the neic SDK that we're using here has some you know ways on how to you know annotate these um methods in your code and you will see you know some you know at nur. agent

background task which basically tells um the agent that you know hey there's something happening here that we want to um get more details about so going back into my um code here there's a couple you know um methods here you can see you know there's the title screen and some other methods if I just you know now enable um this for example and start up the application

again you can see you know now the the agent you know is starting up it is you know telling me that it is reporting data to this specific entity I can then you know click around um again play the game like I did before great right so now what should happen is that um this specific method that showed title screen is should already show up in my

uh in my back end and let's quickly validate um this so this is my um neic account uh my playground I can see the data driven Dave um entity everything in the noric platform is an entity um so this is an entity for APM so for an application don't worry about this um this red um indicator I have set up some strange alert conditions that are messing

around with me here but basically um this screen then shows the most important metrics that I should be interested in and then if I go into the transactions tab I should see you know it maybe takes a couple uh minutes depending also upon how fast the the Wi-Fi is here but I should see um at least the show title screen as a transaction that I'm observing from

my game in in NE Relic let me quickly validate this um if I do a refresh it should come up um in a seconds okay maybe the demo gods are not me um anyhow let me um let me finish this um I don't want to go through all the the methods in that um in that code to enable it so what I did here is I already

prepared let me get this out of the way I already prepared a custom Branch let me get rid of these changes so get check out um instrumentation so now if I go back into my um file here it didn't change the branch I have oh yeah live typing on stage interesting okay so now I changed it and you will see if I go through this um again

you know you can you know look at the repository it has all the background tasks enabled now let me go ahead and run this um again which will hopefully bring up some some additional data um in my application so let me play around with this you can already see you know then this is also something you may notice it is slightly slower right because I'm doing a

lot of things in as custom tasks and you know you can um slow down your application um quite significantly if you you know over instrument the application but anyhow just for the um purposes of this um talk I let me refresh the screen I should probably go ahead and close this all right so now you can see there's the the show title screen from before and you

know if we wait a couple more minutes we will show all the other transactions that are being identified I can look at the performance over time you know of you know how you know good or bad was the performance how long did it take to execute that task did I see any errors you know the throughput how many you know people or users gamers are interacting with

my app so quite nice um I have a lot of in uh information the reason why I um closed down the um application is because if I go back in here um you know if I just stop the application use control C to you know exit the process you know the collector the agent will not be able to actually send the data that it collected locally on

my machine to our back end and this is um what you can also do is um you can tell the agent that at shutdown of the application I want to do a final push of all the metrics that we collected typically in like a one minute window we will collect all the traces metrics logs and so on and then send it um off to the the back

end and this is a um configuration that you can use to basically tell the agent to hey shut down you know have a graze period and then it um it should go and um collect all the data this is also something if I go all the way back to the bottom of this you know which I enabled here so at shutdown I want to gracefully shut down

the agent and not that it is like killed and not being able to send the data all right um this is this is great so let's um you know we we have some some data we have um a lot of um transactions hopefully um now as well if I go back into my transactions tab we should see a lot of more um and of course I'm not

seeing it Let me refresh so we see should see a lot of you know additional transactions that we see and I can dive deeper into you know what are the most active transactions how many did receive the most um traffic how often are these transactions called and on come on here you go so now you know as you can see there's a print map function show title

screen show interaction I can you know focus on what are the most time consuming methods within my application if I want to switch over and see what I have the highest error rates or what are you know the the ones that received the most um requests so how what are the transactions that you know are being called the most um this is this one here check player

Collision this is basically whenever I move the player it checks for am I hitting a wall or anything in the game right so this is quite nice I have some data already here um which is um looking good so the next step um what we want to do is um I also of course you know in typical applications you know we have a lot of logs right

you know um developers like to you know write out some line of logs you know I'm here in this um code I'm here you know this is some you know maybe exception that I'm seeing I want to you know see the the trace uh the stack trace of this um Trace so I can send in a lot of data and of course also application performance monitoring the

age that we use allows to also collect the logs from typical um logging Frameworks that you use um for you know net. Java for python they all have their typical logging sdks their logging Frameworks and what we do is kind of like we interact with these default Frameworks so that we can collect also the logs from the application how is this done it is basically you know

just a configuration change for me we have seen the nur. any file earlier and for me it's just an a matter of um enabling the collection of the logs within the application so now what the agent does is if you um see any log. info log. error um message within my application I can simply use the the same um you know loging framework that I'm already using

but the agent will now collect all these statements like there's some basic um configuration I can you know tell it to collect only info logs hopefully not any debug logs in production which will be uh probably a mess but also some custom um things like you know the game has started the game has ended and what was the score I um achieved in my game so again

let's go back into the um the code here and as you can see you know there's some you know log statements already prepared in here you know the game ended with a certain score I can collect this score and sent also the this this you know information into my environment and there isn't really something else that I need to do in order to collect the logs like

I mentioned you know in the neic um in file there is a if I can type there is just this configuration I can allow it to collect the logs yes or no and this is an option that I have let me uh did I make any changes to this code I don't think so so let me um check logging check out logging branch which again has enabled

all the logging okay I did change yeah so and what happens now is if I again you know start up the application it will show me you know again some nice information about the game it'll collect you can see in the output here as well I'm you know outputting this into into the console as well you know some sample you know just some sample logging information like

an info warning and error um statement that you can see here and if I now you know jump around I can again you know play that game quite nice okay go ahead let me shut this down one more time and then you know in the the back end oops I'm sorry in my back end what we try to do is always you know put things in context

so I have the application which is collecting you know traces you know or information from my um methods and and transactions but we also you know try to bring all the things together um same is also for the logs right so we have a section here for logs and you can see I'm sending in you know logs and this is um kind of like the different um

uh logging uh levels that we see with different you know um color coding and stuff like that so quite nice I collected all the the logs I even have um do this again I should see also did I end the game I did I should also have seen um a a statement about you know what is the the the score that I received um as we have

seen in the code for some reason I'm not seeing that let me try this one more time excuse me so in this case I only you know maybe couple you know score of 200 and once I end it I um see the statement in here and this is also something that I um you know should see in a second in the in UI all right but this

is you know not it like we don't want to stop here you know we can visualize the logs in in the platform we can even you know um put these also onto um a dashboard to visualize it in context with other types information but you know there's more that we can do in in the game that we um um see here you know like I mentioned earlier

an event like is just a discrete occurrence of something that has happened in the in the code but you know what does it actually mean and what how could that be used in that game that I'm just showing you so let's have a look at um what this um could mean so consider like the following use case I have um you know you have seen I collected

the crown you know I went to the next level maybe I want to collect and um that event event that someone you know entered to the next level and I maybe want to um see what was the score that he received and in this case this is basically you know I'm defining just an event type so this could be you know any custom event type that I

um think would be meaningful for me in this case I'm you know sending in a level up event and I'm just passing in some some Json so the Json just contains the current level that the um player reached and also the score from one level to the Next Level that we um received and then you know at the bottom you see that statement which is a again

like a an agent API call that we can use the Relic agent record custom event I pass in the event type the param the um the the Json and basically assign it to the um application that we registered at the beginning of the um course another example you know if I completing the game similar to what we have seen in the the logging statement where we can

see you know what was the final score that the player received again I'm you know defining the um event type passing in some Json um like the curent level and the total player score and then what's nice is that I can then you know visualize all this information maybe in a custom dashboard along with other data Maybe I want to see you know maybe did did any

Gamers um have a good experience playing the game or reaching a high level if the application was behaving fast and things like that so all of this can be then visualized in a in such a custom dashboard or even like in a table format we'll see this in a in a second in the um in the dashboard so an example um again I have you know prepared

this over here this is the game complete um section if I save this it should Auto format it doesn't like it okay um and this is exactly what we have seen on the on the slide again you know now if I'm ending the game I'm sending in this to um noric as a you know custom data point uh let me remove this and uh where's my get

check out custom events oh I didn't like it ah it's really hard to type here on stage here you go let's bring up the the game one more time and now what we did as well in the game is you know whatever I collect you know such an such an item here I'm also sending this in as a custom event right so I'm collecting you know these

um you know these diamonds I'm collecting hopefully the Crone and so on the heart so all of this is now also registered as a custom event that I can then visualize platform let me go to the next level to for you to see also what the next level looks like okay great I can die okay let's have a look into the the back end so what we

um typically looking at is kind of like this is the the you know a curated UI for application performance monitoring if you use like agents neur Relic agents or open Telemetry we Tred to show you what is the most important metrics that anyone should look at this is a pure SAS service so this is the same for all the customers but um the nice thing is in

here that all of the Telemetry that we collecting is available as um in the in the Raw format so I can for example I could you know the just the query that I had up and on the screen this is kind of like you know from this is you know a language we call nkel neur Relic query language um similar to SQL you can see it looks

pretty similar if you're familiar with um SQL from transaction select count St star um fast ET by name facet is basically a group by in select uh in in SQL so now you can see what are all the and I can visualize this a little bit differently maybe as a bar chart so now I can see which of these transactions that we have seen earlier um had

received the most requests which method was used the most within my game and you can still um again see the check player Collision is one of the the events but if I change this and now um if the gods are with me we should see some um some custom events collected item for example this is the one let me just output this as a table so now

you can see you know all the the data from this custom event that we have specified I can see you know the item that I collected is it an equipment the item id you know is it just a plain item or a trophy um the item score that um is associated with in that game with that um component and also the item type I can see the

the level number and so on so this is the raw data that I have available for me to you know play around put it on a custom dashboard or use it in a way that makes sense um for me uh how much time do I have until U the bottom oh 15 minutes thank you uh all right so all of this you know is you know ideally

is something that I can use in a um in a combined View and how do I do that you know we can you know you know in The Relic we allow you to add basically any type of data this could be a custom event this could be events from your Telemetry like traces metrics from your applications onto such a dashboard we can even you know we've seen

how you can you know also you know look at logs logs for um us is basically just another event type um that you can visualize is also on the on the screen and this is not what I wanted to show yet but um if I go into back into the platform the neic UI what I did and what is part of the um repository as well which

you can use as a um if you want to look at it it also contains this game stats Json which is basically just the Json representation of such a dashboard let me make this a little bigger and so it basically you know is a representation you can copy paste that Json into a neic account and then go into the dashboard section use that to import the dashboard

I already did that um already so now I can look at you know this custom dashboard that we have um created it shows me some high level information like a screenshot of the game nice I can see how much uh how often has the game been played played what was the highest score that I received you know the maximum level that I reached I'm you know probably

not the best player I should have um have a 10 out of 10 but anyhow and then you can see you what was the top score by level um what was the the items that I collected I can visualize all of this in a in a nice way to also show me you know even the logs here so it basically brings all the information together for me

to visualize what is most important for me and in general this is this would probably be maybe something for my you know my my business owners for management team where I can you know build a um custom dashboard like that to show you um you know the way the data in a in a more meaningful way for uh maybe your your other personas that you're dealing with

so what we have seen um so far is kind of like you know our Nur Relic agent how to you know configure this for the transactions for the logging and for the you know some custom events that are important for my for my game but um I also you know um did the effort and and kind of like instrumented that same game using open Telemetry like I

said open Telemetry is an open standard it kind of like is a vandor neutral implementation from you know based on on a cncf project this um QR code will bring you to a different repository that has the open Telemetry instrumentation all of the things that we have just seen for the Trac for the background tasks for the for the logging and for some custom events also implemented

with um open Telemetry in that case you know it um uses some uh slightly different commands to spin up the application so I'm again using my you know license key for the API that we using in a relic and then you know in this is very python specific open Telemetry um allows to bootstrap your application doing install and then I can spin up the application using a

command called open Telemetry D instrument and what you see uh as parameters to that command is kind of like I do I want to capture traces do I want to capture metrics and logs and so on and I can decide and pick and choose where do I want to send each of the different Telemetry information into I could say the logs go maybe into maybe some elastic

the traces go into you know some observability backend so I have full control over what is happening and um I Ed yesterday in the talk where open Telemetry also allows you to have a or to configure a pipeline that runs you know alongside your application which you can then use to you know do some more interesting things like you know I can batch the Telemetry I can

maybe opos skate on the local environment that I'm using where the application runs I can you know say you know maybe in the in that pipeline as well maybe do I want to forward the debug logs or do I just want to you know have info log so this is something that I can use already in that um in that pipeline to Define what is happening and

then again I give it a name and then you know nurel provides an um OTL endpoint OTP stands for open Telemetry line protocol which is basically the um an open endpoint an open API that you can use to send in any type of open Telemetry data and then I'm just using again the python command at the end to you know start up the but this is the

way on how you can use that with open Telemetry so really you know interesting way in my opinion will be the future uh of application performance monitoring or you know infrastructure monitoring because you know the it has a lot of benefits from you know not no vendor login it's vendor agnostic and you can use it to instrument once and use it um and send the data wherever

you think uh it makes sense for you today and with that um that was it I hope you enjoyed it I hope it was a little bit fun to you know use that in a gaming approach um any questions yes we have one question what is the strongest argument to switch from Prometheus kibana to the from free solutions to new okay good question um so you know

some of the open source um solutions that you typically use you know you have to you know manage them yourself you know and you know you have to have infrastructure if you have like an elk stack to for your logging if you have elastic search or any you know Prometheus environment using grafana you know it's one thing to you know basically put this in place but you

have to manage it you have to you know have resources to deal with it it's not like something you spin up and you know don't touch again you know neur Relic is a fully managed SAS service you don't need to take care of um all the things it scales with your applications we have huge customers who send in million of data points per second and this the

the back end just scales um um gigantically so it's a it's a choice right do I want to take care of this in my own premises or do I want to rely on a managed service um like NE Relic other questions sorry uh okay seems that it was really nice and amazing presentation thank you and it was the last uh presentation in this Hall so uh big

applauds to our speaker thank you and a small gift from us oh thank you so much