Flight Recordings and Mission Control Through Thick Clouds | Martin Skarsaune (EN)

In this session, Martin Skarstein discusses various tools and techniques for troubleshooting Java applications. He shares insights from his experiences as a consultant working with Java since its early versions. The talk covers unexpected issues such as performance degradation, CPU usage problems, and memory leaks and outlines practical steps developers can take when faced with these challenges. Martin emphasizes the importance of monitoring through metrics and logs, and introduces the Java Flight Recorder and Java Mission Control as effective tools for gathering performance data from applications. He also addresses the complexities of troubleshooting Java applications running in containerized environments, highlighting tools like Jolokia and Hot IO that facilitate remote management and monitoring in such setups.

[music] >> so much for coming. My name is Martin Skarstein. I work for a company called Kantega in Norway. Uh the good thing about having a strange Norwegian name is it's very easy to find me. I'm I'm the only one at least in our industry that I know of on that name so you can look me up. Uh so I'm a consultant. I do work full time

on on client projects. So I'm just a regular developer. Um I was thinking back. I actually started with Java 1.0 in the college in '97. So it's not 30 years of Java, but uh almost getting there. So I guess it's like uh seeing your children grow up. You realize that uh you're not young anymore. Um so um the title today was kind of I was playing a

bit on the airplanes and the flights and then the clouds giving us all sorts of problems. That was the kind of thing I came up with. And then when I walk into J Spring very first presentation on the very first morning Captain Garrett took this to the next level, right? With the with a hat and a whole slide set. I felt I was in the airport. So

I know so much for being original and coming up with a good idea, but um yeah. Never mind. Uh let's uh let's kind of put ourselves into the situation where we are today. So you kind of you're feeling pretty content with yourself. You just deployed a shiny new feature to production well in time for lunch. And you know, things are looking on the up and up. But

then disaster happens. Uh so today we will focus on some tools you might use for kind of more technical things, not functional things. So, things like unexpected crashes, uh, somehow degraded performance, slow response times, uh, perhaps uh, some increased CPU consumption, or maybe too little CPU consumption that your threads are just hanging around. It could be memory leaks, um, higher footprint, or you leak some sort of

underlying resource like connections or something. Uh, so what do you do? You of course you gather the team and you try to figure out what's wrong. So, hopefully you have some sort of monitoring in place who tells you that uh, things are bad. Uh, then the metrics will tell you that something is wrong, uh, perhaps also give you an idea about the area things are wrong, but

maybe not why or what so much. Okay, so then you look at your version control, uh, and you try to to see maybe some maybe that latest deploy wasn't so great after all. Um, you can choose to roll back. Uh, and then you need to figure out was this just a coincidence in time or was it actually related to that piece of code, but you have not

really fixed the problem if you roll back because you know, at some point you need to to go forwards for security patches or what not. hopefully you have logs, good logs, uh, they are typically more functionally oriented, but sometimes with the good query languages for logs you can kind of sometimes pinpoint where the problem is. But if not, maybe you you add some more logs, you do

some more deploys, you add some defensive code, you make some hypothesis, maybe you do some extra thorough code review. and maybe you try some load testing. might not be the best tool when when things are burning, but some people are good at that. Uh and I've actually stolen this first pictures on the slides are from uh the big hit, The Mysterious Life of Developers. And the reason

I feel I don't feel bad about stealing that is that those guys are actually my colleagues in Oslo. So, you know, walking into the office being an introvert developer, you can become a movie star in an instant viral hit if you don't uh watch yourself. So, but if none of this leads anywhere, uh we have some tools for you. So, we have the flight Java Flight Recorder,

and that's of course then inspired by the black box in in aviation. So, it records a lot of um very detailed information about various things in the Java Virtual Machine. Um this recording can then be streamed to a file or streamed over the network, so that you can sit at the comfort of your laptop and get the recording in there and kind of make a picture of

what was going on based on that. It's suitable for use in production, uh typically quoted 1 to 2% overhead. I find that very acceptable in my case. Uh the cool thing about the Java Virtual Machine is that the flight recorder is present. It's there. You just flip a switch. You don't need to uh you know, recompile your code with some debug flags or whatnot. It's just ready

to hit. Um yeah, so you don't need to do any deploy anything extra. It's open source. It's free. It was uh donated in Java 11, later backported to 8 because it was a a commercial feature before that. Um you can also create your own events and use this mechanism. Um and then you have Java Mission Control. Java Mission Control is now a project on GitHub and it's

released independent of of the JVM releases. Um, it's modular, so you can create reports and plugins of your own if you like. so yesterday there was a very good like what should I say? Com- complementing talk from AWS guys where they used uh, to kind of Today I will look at kind of ad hoc using these tools. They were like more systematic doing recordings and using AI

to figure out whether your application is healthy or not. So, I would encourage you to to check that out if you didn't. some some theory before we get our hands dirty. So, many of these tools are based a subsystem of the JVM called JMX. It's a really good old one, JSR 003, but I think it's aged well. Uh, so it's built around the concept of management beans,

MBeans, and those um, let's say expose some operational information of the system. So, usually it's a system stuff like memory, CPU, um, things like of that nature, or it could be third-party stuff like um, connection pools, uh, caches, things like that, Spring, uh, and you can even expose your own application internal ones if there's um, yeah, internal state that's useful in a monitoring context. And each MBean

has some attributes. They can be static information or it can be like metrics that you can graph and stuff over time. And some operations that you can execute and have happen inside that VM. There is a protocol called RMI over JSR 160 that I will come back to later because there's some some issues with that. So, let's just do very very brief overview. Not really in depth,

but so um this is Java Mission Control. Uh and uh it will list all my local VMs. So, the idea today is like this is how it works. Everything works out of the box on my laptop. And the hope is that we can make the experience uh as frictionless as this. That's kind of a goal. If you manage to get a flight recording, then we're happy. So,

of course we have the PetClinic application as any demo should. Uh we can start a flight for a minute. So, now it will start the recording, stop it, retrieve the contents, and open the analysis in here. And maybe we can just give it a little bit of work. I see we don't have CSS, but I guess for us old guys that's just uh nostalgic, you know, before

all this um single page web application nonsense where you just you focused on getting stuff done. So, I'll just uh click around a bit so it's not completely idle. Let's see. Here we are. So, you see it's counting Yeah. So, that's more of this. I will I will not cover it in depth. You will find many good resources on on what it contains, but I just give

a brief overview and I think it's quite quite impressive the kind of build image you can build up by processing all these events that have been written to to the recording. Yeah, so um there's some standard reports so tell me for example >> [sighs] >> I had the password in the environment variable, not good. Um yeah, I guess the main focus and that was also covered yesterday

is method profiling to see where your time goes. But you can also like see uh timeline of what your threads were up to. Um if there's a problem with threads not doing what they're supposed to. you have garbage collection. runtime compilations and class loading. So there's been some focus during the conference on startup times. So perhaps uh you can also start the recording when the VM starts

with the flag. So maybe you want to to try different things. You can even spy on the processes and stuff on the host where it's running which is cool {slash} scary. Uh I thought I would just um play a little bit under the hood since I like to find out how things work. Created a little helper script just to forget the the remember the commands and not

forget them. Yeah. So um in your user's temp folder there will be a directory called HR perf data and then suffix by username. So on my Mac that is like this. And there are a number of files with the [snorts] numeric names. So if I run some of the commands in the bin folder of my um JDK you will see the same. So it will use this

to discover what processes are running. And um the most versatile command these days is JCMD, which is kind of a do-it-all. So, you can enter JCMD and a process ID and help. So, now I'll show some command line stuff that's kind of a counterpart alternative to mission control. So, you get this list of commands and you can look at the given command. And the list of commands

showing up will be the combination of the JCMD client actual VM what I can do. Um so, that's all very good. Everything is very comfortable. It's available locally. No worries. We're all good. But then all of a sudden we're told, "You know, we don't run hardware software like that anymore. You need to have your application run in a JVM inside a container inside a pod somewhere. Um

so, very simplified picture of Yeah, kind of. So, so there's a number of things that make this harder now. You have all these pods who are kind of transient. Uh you don't know where to find them. They come and they go and they move Uh you don't have direct network access and then uh these last uh these three points with console access, lacking tools, and access to

files. There have been some stuff during going on during this conference. So, first there was a workshop on Monday. And yesterday we went on a mission to bolt all the hatches and lock everything down. Uh so, we completed that mission, but that leaves me in a tough spot. So, we will try to see what we can do. So, thank you, Catherine, for that. and also typically I

I is is is my expression that uh experience, sorry, that uh uh in the good old cowboy days, we tended very wide permissions as developers. God powers for uh permanently. Um with the new uh Kubernetes and stuff, it's very fine-grained, and you have to beg and ask to do stuff. So, we'll try to to see kind of how what we can get away with with how little

permissions. if you come back here, then now I have uh this on my local Docker, we have these two pet clinic pods running. And I want to figure out what's going on with the one of them. Uh so, okay, can I I'll I'll just like in the old days, I'll just log into the server and see what's going on. So, I'll try to attach to it. But,

okay, it doesn't have a doesn't have a terminal. Okay. But, okay, then the uh Kubernetes uh documentation says, "Well, you can you can execute, and you can open a shell." So, you can go inside and peek. But, it turns out that uh we actually removed shell from from the container. So, there's no way for me to to look inside. Okay. Well, this uh JCMD thing, that was

very helpful. Can't I just do an exec directly with that instead? JCMD is not present, either. Okay, let's really simple stuff. Just look at the processes. Oh, no. Uh okay, let's try to see if we can force a thread dump by sending a signal to process number one. We know it's process number one that's running the the Java. Nope. Okay. Okay, we for sure we know it

does have It must have. So, can I run a Java command? this particular thing is a problem that you have this Java tool option set and I've not managed to find any way to unset it before I run the command. So, uh this is not good either. Uh okay, let's say for the sake of discussion that I get the recording. Can't I just copy it out at

least? I'll um keep this on the clipboard for later. Um but it turns out this one is a bit interesting. You need tar in your container for kubectl cp to but then I think I guess you're jumping in your seat saying, "Well, you should do kubectl debug. Why don't you do that?" And we will do that. Uh so, uh what debug does is uh I I will

debug this pod and I can specify that I want to create an image from this. So, that's a richer image with my toolbox intact. And it should target the pet clinic and because Docker is Docker, I can those will share process space. So, I can sort of run beside and spy into it. And here I want to open a shell. Uh so, you see it's creating a

container with this generated name. So, now I can uh so, if I do jps, I can actually see the process number one from the other container. and if I do I can do jcmd. So, let's let's do JCMD Java Flight Recorder start. So, now we have a a recording going on And it even tells me what to do next. I can do Java Flight Recorder dump. file

name like that, and then I can do JFR just to be nice. Okay. So, I do LS temp. Oh. So, what's going on here? Why is there Why is there no recording there? And the the answer is that now now I'm looking in the debug container, not the Pet Clinic container. Thankfully, there's a way around this. So, I can do copy from process number one user root

temp recording to like this. So, now I managed to to copy it into this more usable container. If I had been completed the mission yesterday and run the other containers as a non-root user, you need to do some extra stuff. You need to create uh create a user with the same UID as in Uh and then the uh this Java commands will uh, work because it will

then look for that username as you saw in the very beginning that uh, the usernames need to match, but uh, I've been able to do this just some extra steps. now I could manage then since we have, let's see, this copy command here. And we had I need to specify what container to copy it from. Uh, and that was, uh, I kept that one. So, if I

do like this, I'll get a, uh, copy so now I can do I've managed to to get the recording out of there and into my laptop and then I should of course, um, detached or finish that debug container. Well, this is great, right? Now we've made it so we could just take an early break. there are however few things that your security guys may not be that

eager about because now we have actually had to spin up an arbitrary container inside our production environment. Uh, kind of doing the exact same we were trying to avoid. Maybe a situation would allow it that it it is an option to use, >> [snorts] >> I would like to look at some some other alternatives as well. And you saw also the process was kind of >> it's

a bit clunky. Yeah? You could have some prepared container images with some scripts and some utilities, but it's still kind of awkward. So, then we have this thing called, um, RMI. Uh, maybe I'll just show it and then we can come back to all the problems later. so I will do a port forward into the and expose port uh, 9001. So now I can actually Let's close

this one. Now I can actually connect to the container do just as as if it was a old-fashioned network. So you see that it it forwards the connection here and just acts as as as a proxy. So so that that works. Um, uh, here's the thing with RMI that may might not have aged so well. Uh, it's the call of stands for remote method invocation. And it

was the very first client server thing we had in Java. It's really it's a binary protocol, which means that in practice it makes sense to to do Java clients. It's very complex with like in in the initially it was like you would have one registry server, you would have one server for byte code, you would have the server and the client and the whole byte code thing

meant that the client would just load byte code in over the wire to act as a kind of a counterpart to the object on the other side. it's a bit of a hassle and uh, firewalls and security people may not love it for that reason, but it's less moving parts than the other example and it was kind of more ready to go than than doing this container

insert. Um uh, I wasted so much time over the years coming up with this uh Harry Potter-like incantations to get this thing to to work. But, yeah, we succeeded again. Hopefully, maybe with the AI these days you get more help to get this this stuff right, but it's uh what other alternatives do we have? So, almost 10 years ago, Java Sun 2016, met one of my favorite

Germans ever, Roland. I created a framework called Jolokia, which then attempts to since the JMX is a good concept, so create a bridge so that you can uh use ordinary HTTP towards it and have JSON payloads. Um it's uh you can easily enable it in Spring Boot Quarkus, whatnot. You can also I use a Java agent to kind of attach it to a running process. Um you

have some granular security, whereas RMI uh is more like all or nothing. So, that can be a good thing because there's uh you know, with great power and so on. Um there's a broadcast discovery And actually uh uh the project has has a real uptick lately. So, you could if you see the nice people from IBM in the hallway, they've taken it under their wings and there's

a lot of stuff uh finally happening now, so there's a bit of a momentum. so so we have the Spring Boot application, and we can call the actuator endpoint just to illustrate how it may actually be present already. And here there there's this endpoint here. And that just uh shows you that it's there and gives you some meta information. So, the way it works is you call

Jolokia. In this case, we want to read an attribute. This is the syntax for an MBean. You have this colon and and stuff. And then there's an attribute. And so, you get this response with the attribute value. So, the name is localhost and and the PID. since this is very simple, you could use all kinds of tools because JSON and HTTP is pretty ambiguous these days. So,

for instance, here I can just do curl or monitoring tools or whatnot. So, it's You don't need to write a thick Java client just to work with it. let's see. Yeah. So, I guess now we are one step further. We could Um and I can I can show you here. But the new thing that's been introduced is uh with Jolokia 1. 7, we created an adapter layer

to emulate the kind of the old or the what's called an MBean server connection over Jolokia. existing tools could switch protocol. And it turns out it works well enough for um all or all the functionality I've found Uh so, if we look at this JMX this is JMX service URL was made kind of modular from the beginning and there's a plugin mechanism so you can provide another

protocol. So that's what we've done here. So service JMX Jolokia and then it's pretty much like HTTP from here on out. Uh which which makes it impossible to to work on it as if [snorts] it was RMI or local. Uh do flight recordings and whatnot. But so okay, what does that help us? We Now we have two different protocols. We do have port forward so we could

port forward either RMI or Jolokia. What's the point? And uh so when I was playing with this there is a part of the Kubernetes API that I kind of stumbled across uh that allows you to So here we have on the top here we had the the stuff from before. You have an ingress, you have a service, you have your service here. You serve on port 8080.

Um we thought maybe maybe having the actuator is risky, right? If you misconfigure something, all of a sudden the actuator is visible outside. So instead we could set up Jolokia as an agent on a different port. And uh then this API allows you to talk to the Kubelet's API to have an HTTP call directly to a port uh inside a pod. Because the other problem with ingresses

and services and stuff is or I guess port forward is is pod specific. So it's it's a bit like this. You have a namespace, pods, protocol, pod, port, and proxy, and then the path to that HTTP port within the um the the pod. So, the idea was then to kind of enable this so that since the Kubernetes API is Kubernetes API is um you can use it

programmatically. So, the idea was to scan scan for pods that fulfill a certain pattern. And then try to to use this um protocol then to uh talk directly. And a bit similar to the previous discussions like port forwarding is fine. It's still manual, but it works. But maybe, uh running HTTP over an existing API is slightly cleaner than running arbitrary protocols directly into your pods. So, we

found some some instances. We found the two in my Docker Desktop. We found a couple running in Amazon. And in um AWS and in Google Cloud. Um so, let's just use this that [snorts] protocol. So, it I have all these in my uh Kubernetes config. I have all all these contexts. So, it's a context, and pods from there. So, we can um we can do flight recording

was what it was our goal. It's a bit slower, but the process runs as it should, and then just needs to stream the results back. Yes. So, this this was the idea. There is also um an option here to do heap dump for a different kinds of troubleshooting. Um and uh uh I broke this the other day, so it won't work right now, but I can I

think I can can illustrate it. So, um because I believe this will fail. If it's a local process and you do this, oh, it worked. Nice. Uh so, what's happened now is that it did a heap dump, and then we see the flight recorder coming back. So, the GUI is a bit doing all sorts of fun stuff. But the the the Now we have the same problem

that uh you know, the the heap was dumped into a file within out there up up and out there somewhere. Uh but we get So, if it's local, it will open immediately. If it's not marked as a local process, uh we will get a file dialog here. So, in the meantime, I need to then go into uh this share here. So, I mounted some mounted some cloud

storage within the containers. So, we have this uh this one here download. There it is. Yeah. So, I can I can download that, and then we see Oh, yeah, network is really ripping maybe maybe I'll just I'll just open an old one. So, the thing is I can I can get it down and I can analyze it just like a flight recorder. I will show a different

tool as well. There is some support in the flight recorder uh for this I guess one important thing just to keep in mind for us techies is that And so, the heap dump is very with these tools are very powerful. You can like traverse the heap dump search for stuff and and things. So, you know, there might be credentials [snorts] in there, there might be tokens in

there, there might be personal data in there. So, if your company has to have a strict policy on these kinds of things, uh don't forget about the heap dumps and treat them in a responsible way. okay. Maybe maybe I'll just I'll just open an So, [snorts] uh Yeah, so there's some some stuff here, but there is if you manage to get hold of a heap dump, there's

um there's another tool that's called the um Eclipse Memory Analyzer. So, this is where you can kind of query and stuff into the into the heap. Kind of so, yeah, we have uh managed to get past these things. Uh I thought I would show an example or we're not done, actually. So, um So, this was the idea. As little friction as possible, connect via this API. However,

uh what you've seen today is uh is uh my branch. Uh It's been a long time getting this one uh over the line, and it's been seen some problems lately. So, I might need to reassess a bit how things work. But, um uh this will be the reality not too long. Another one alternative is something called Hot IO. Um what used to be very related to to

or it's coming from the same guys as Jolokia more or less, so it's also uh owned by the nice people at IBM. So, this since Jolokia offers this uh HTTP and um and JSON, you can then create management consoles with different technology. So, Hot IO is is a front end that uh visualizes stuff just on top of of that. there was some problems just now. Let's So,

I believe I might have to do a port forward. It's uh Don't do demos where a lot of stuff relies on network and things. Yes, okay. Um so the idea is that So there are many ways to do this. Like uh Spring Boot uh Quarkus, you can just enable whole tile in your application directly as an endpoint under actuator in Spring Boot. Um, however, I thought uh

a better architecture could be to run it as a separate dedicated application with a Java backend. And then that backend can get the permissions to do network calls to the local port on the other pods living in the same place. And if you don't have an uh ingress to this one, [snorts] that should be like a a separate one from your main main ingress. Um, so let's

just see. I wonder if again discovery is supposed to work, but it only discovers itself. So that's not so Let's just see. I think uh let's do uh demo. Kind of scary to do this on the fly, but let's see if we're lucky. Yes, okay. So what happens here is this shows the process itself. But when I uh should be allowed to connect. >> Yeah, never mind.

But, this also has the has the ability to um do flight recorders, and you can even Hey, there we go. So, it opens a new tab. So, now I'm looking at the pet clinic and not what I owe. And you see there's a spring boot thing here appearing magically. Uh because it detects that there's spring boot, and if you have camel, active MQ, different things, they will

have created different kinds of plugins for it. I can uh start a flight recording. And I can stop it. And I can download it. And the reason I can download it is it's a rich um Java back end. So, when you hit the download, it will uh kind of send several messages to read chunks from the flight recording, 50 kilobits at bytes at a time, and then

stream them back in the response. So, you can have uh sizeable things. So, that's uh Yeah, maybe [snorts] fourth fourth alternative today. Uh this also can be used to show that uh this spring boot thing uh looks a bit broken. At least the loggers part. Uh but there is that's on purpose if I go to uh spring boot, there's some loggers here, and I can execute this

one, and it will tell me that it's forbidden. uh what we have done is for our application, we created a um Jolokia access file saying that uh allow access to certain things. Uh but we uh only allow a few methods because for instance the Spring Boot logger might show request responses in clear text. So, if done a big job of masking all your logs, you could actually

get the clear text JMX which which is an example you might not want. So, yeah, this is an example of um how it could be good to be a bit aware of what the various MBeans does and what you allow them to do. I think that uh more or less concludes it. Let me see. Yeah, some references. Jolokia.org, hotio um dimension control uh there's another favorite German,

Johannes Bechberger. He was he's here. He has uh a very good blog on uh profiling and does a lot of cool stuff in that area. So, let's see if we had any questions. No. Or is Do I need to double check or is it my network down or Just shout out if there's anything. If not, I will um You can uh look me up. Talk to me

in the hallway. Yes, if you uh scan that QR code if it works, then I posted it on SlideShare this morning. That's a very good question. To be honest with you, I I've been mostly working with Go these last years. I saw a need and I had some ideas, but I've not been dogfooding too much to be perfectly honest. But, I have a I have my fair

share of tricky technical troubleshooting, but maybe not with as good tools as those I've been working All right, then. Thank you so much for