The Hills Are Alive with the Sound of Kubernetes - Stevie Caldwell, Fairwinds

This talk, titled The Hills Are Alive with the Sound of Kubernetes, is presented by Stevie Caldwell, a senior tech lead for the SRE team at Fairwinds, who explores the innovative intersection of sonification and observability in Kubernetes. The speaker introduces sonification as a technique that uses non-speech audio to represent data, drawing parallels with everyday sounds that signal changes in our environment. She discusses the potential benefits of auditory cues in monitoring Kubernetes clusters, emphasizing that auditory input can prompt faster response times than visual cues. The presentation includes a practical demonstration that leverages a Go-written controller, Communication over OSC, and SuperCollider to translate Kubernetes events into sound. Through this creative approach, Caldwell encourages attendees to rethink traditional methods of observability in technology, promoting a more engaging way to interact with infrastructure operations.

So, uh welcome to The Hills Are Alive with the Sound of Kubernetes. Uh this is a talk on sonification and observability. Let's see. So, um I am Stevie Caldwell. I am a senior tech lead for the SRE team at Fairwinds. Um as long as I've been working in tech, one thing that I've noticed is that a lot of folks are also musicians on the side. They have

bands and and they make music and things like that. I'm no exception, so I'm going to take this moment to uh shamelessly plug my side project, which is called And Then There Was One. Uh you know, doing music is uh the primary motivator for me uh coming up with this talk. So, I make music in my home in my little uh homemade studio. It's essentially a desk.

And then I upload them to streaming platforms like Spotify, Bandcamp, um and Apple Music. So, there's that. Uh so, as I said, I work for Fairwinds. Fairwinds is a managed services provider for Kubernetes. So, we build and maintain Kubernetes clusters for um a wide range of clients across a lot of different industries um to free up SRE, DevOps, and developer time uh from doing toil and drudgery

things in Kubernetes and doing more exciting things like like something like this. let's see. So, in this talk, we are going to uh talk about what sonification is, the what and the why. Um I'm going to review my demo at uh a high level, the architecture of it um at a high level. And then I'm going to do a quick demo. Um so, let's get started. So,

uh to start off with, sonification is the use of non-speech audio to convey information or perceptualized data. Simply put, it's using sound to represent data. Um this is a still taken from astronomy.com. It's an image of the sonification of a galaxy cluster. Uh you won't see anything this cool in my talk um because this is astronomy.com and I'm just Stevie. Uh and I have a day job.

Um but I wanted to show this to you to show that although I'm doing this talk today is just like a fun little thing that I did, uh there are practical applications for this and there are very serious people and industries that do make use of sonification um in interesting ways. So, um I entered this talk or the idea for this talk a little backwards. Um as

someone who makes music and works with Kubernetes, I wondered if there was a way to marry the two. Um so, and to do something fun with it. So, that led me to thinking um you know, wouldn't it be cool if you could hear your Kubernetes cluster? And then, how would that be at all useful or, you know, interesting? And I I started thinking about how the human

brain reacts to sound. And something that immediately came to my mind was thinking about the common sounds that you have in your home, perhaps. So, like the hum of your refrigerator, your air filter, the ticking of your clock. These are sounds that are constant uh background noise in your home that you practically don't notice after a while. But a change in the tone of one of those

things or in the frequency of it can have you doing one of these and going, "Huh, something is different." And so, the question is, how does that work? Um like how what does the brain do uh that makes that happen? So, this is actually a phenomenon known as the cocktail party effect. It was coined uh as such by uh uh cognitive scientist Colin Cherry in 1953. My

slide says 1963 because I did not do a good job of uh checking myself here. >> [snorts] >> Um but the idea behind the cocktail party effect is that you're at a cocktail party, you're having uh a conversation with one person, there's the general hum of the room, and across the room, if someone mentions your name, you'll hear it and you'll notice it. And how does that

happen? Well, the brain uh and this this uh cocktail party effect uh the science behind it explains that the brain has the ability to focus on a single sound or conversation in a noisy environment while also monitoring background information uh for a for uh potentially important information. Your brain is actively filtering and switching between these different inputs of sound. So, for the cocktail party example in particular,

your name triggers pattern recognition. It has high personal relevance to you, I hope. Uh you benefit from frequent exposure of your name because people say it often um and it has emotional salience in that it is often equated with receiving news or information. Studies have shown that your brain can recognize your name while you're asleep or even when you're under anesthesia. So, why is this potentially useful

in observability? How or why would you involve sound? Um so, there are some reasons that I've put up here on this slide. Uh and essentially, response time is often faster for humans uh from auditory input than visual input. And we process changes in audio uh really well. Uh for example, think of the Geiger counter. Um you know, this is a a sort of ubiquitous thing that we

we recognize that tick tick tick sound that it makes is very effective for intuitive radiation monitoring. Uh studies have shown that people are better at monitoring radiation levels by audio than by visual. So, they react faster to hearing that sound versus looking at the uh visual input on the device and then doing whatever, you know, quick calculations your brain has to make to determine if that's a

dangerous level or if the level's rising or what have you. Um if you've ever spent time in a hospital, you know that the beeping of devices like cardiac monitors, pulse oximeters, and such help the staff keep track of uh any potential changes in patient status without having to be standing there and looking at uh these monitors the whole time. Um audio plus visual also increases memory retention

and reduces cognitive load. I know for myself, I learn a lot better when I'm watching uh a tutorial when I have the video and the uh the audio and the visual in front of me rather than reading uh from a book alone. Um and the other thing is that we spend, I think probably everyone in this room spends a lot of time staring at a screen. And

I'm sure there's plenty there are plenty of you who at the end of the day have made a move like this, right? Your [laughter] your brain's tired, your eyes are dry and gritty, you have headache uh from taking in all that uh visual input and uh adding some auditory component to that can help relieve some of that visual um fatigue. So, the architecture of uh of my

um demo at a high level, uh we start off with a Kubernetes cluster. We're all at KubeCon, so you all know what a Kubernetes cluster is and what it does. Uh and that is our data source. So, the events that are generated are what we're watching using the watch API that our controller uh subscribes to. Um the controller is written in Go. I started off thinking I

would use Kubebuilder because Kubebuilder is sort of like the starting point for a lot of uh ways that you talk to the API server. Um but um Kubebuilder is great for building controllers that enforce desired state, but my need was a little Um Kubebuilder has a two-part pattern where it, you know, it grabs a key uh a name in a namespace, puts it on a queue, and

then it does a second loop or it has a second step where it takes that key and uses it to get information about the current uh state of things versus the desired state. And um I just want to play a sound when a pod fails. an event happens, sound comes out. So, using the controller runtime directly was uh very simple and direct for my needs. Um and

I run this controller locally on my laptop using my kubeconfig to access the cluster. Um it'll send the messages that it needs to send to uh SuperCollider, which is also running locally on my machine. You could containerize this as well, um but that opens up a whole uh new area of some networking questions and where you're going to run SuperCollider and things like that. So, for purposes

of this demo, I chose to keep it pretty simple and just have it running on my laptop. So, the next uh piece of this is OSC. That's a lightweight protocol for real-time uh communication between computers and audio devices. Uses uh UDP over port uh 57120. Messages are structured like URLs. Um and it has virtually no overhead, which make which makes it uh really ideal for something that

needs to have like a fast response time to events. And last but not least is SuperCollider. Have any of you ever heard of SuperCollider? Awesome. Nice. Yeah, so SuperCollider is a programming uh environment for real-time audio synthesis. Has three components. There's a language server, there's a synthesis server, hard to say, and there's also an IDE component where you can load up your files that have these definitions

in them. The definitions that you put in there are synth devs, which are basically sound generators, and OSC devs, which listen for OSC messages. So when a message comes in, the OSC dev triggers the synth dev and makes it generate a sound. And you actually the SuperCollider is actually used by musicians in real life. There are a lot of websites where people share the music that they've

made just using SuperCollider alone. let's talk about the event to sound mapping. So my demo is is pretty straightforward, pretty simple. I am watching just a few events, pod created, pod deleted, pod crash looping, and deployment scaling up. So these are the events that we're going to be looking at in the cluster. And so now we're going to go to demo time, and here's hoping that this

works All right, so we're going to first of all open up SuperCollider. Can you see that? Is that very small? All right. So this is SuperCollider. This is the IDE. I've already loaded up a dot SCD file which contains the OSC devs and the synth devs that I talked about earlier. So again, just to recap, the OSC devs listen for OSC messages and the synth devs define

and generate sound. I also have my controller over here written in Go. So I defined again a couple of things to a few things to watch. So I have this on pod create method that when the controller gets an event from the API server, it will send an OSC message, and this is the URL or path that it'll send along with it, along with some parameters. So

then SuperCollider OSC dev down here, let's see if I can find that. So this OSC dev is essentially a a definition that says if you receive a message an OSC message with this path, run this function. And then this function is linked to a synth dev here. So if we go up, we can see the synth dev is defined up here. So there are a bunch of

parameters that control the sound that's going to get created when that synth dev definition gets triggered. And you can provide the values that get sent here in the Go controller. You can define them some fallbacks in the OSC dev. You can also define further fallbacks in the synth dev. In in fact, you don't actually even need a Go controller to to make music with this. You could

just do all of that definition inside this file and have it just work and make sound. So let's take a quick listen, and this is the part I'm hoping is going to work, to what those sounds sound like to give you just a a little hint. I played this for my kid last night. I warn you that he told me that it sounded cursed. So apparently it's

not it's not incredibly pretty. I'll turn that down so I don't blow up people out here. See if that comes out. You guys hear that? All right, so that is the sound of pods being created. I'll pump it up just a little bit more, and just one more time for you to get that into your head because there's a fun little quiz that happens later. And these

are just more examples of how you can pass in different parameters, so you can play that same pod created sound with a different frequency and duration. And you get some really groovy whale sounds. This is the pod deleted Very sad. It's very sad sound. And then we have some pod crash looping Which is actually oddly soothing for a pod crash looping. And the idea is that it

takes the restart count of the pod, and so it builds up on that each time. And then we have the deployment scaling up will be the last one that I play for It's very hopeful. [music] We're growing. Doing good things there. So those are all of our different sounds that we're going to experiment with today. So we're going to do a little experiment here. And bear with

me so I can make sure I get all this stuff going. So I've got the server started. I've already booted the server, which should just be hitting enter under S boot. Um and free all is just a nice thing to make sure that anything that's currently running stops. Um and we I believe I've already right. So I have to go and evaluate this file. And we should

hear. So that is a a low drone that I created just for the purposes of putting some background in there. And then if I go over to my terminal a small terminal. Oh, actually. Let me make sure that I am connected to a cluster. And if this doesn't work then I have a local cluster, a kind cluster that I can use instead, but this should work. Great.

It might be a little slow. And so if this if this turns out to be painfully slow, I'll run it on the kind cluster. There's just more things running in this actual EKS cluster that I spun up. So And then so I'm going to do the OSC and make sure this is off because it makes a terrible noise when it first starts up that I've not been

able to figure out >> [laughter] >> how not to have it go away. All right. So now So that tick tick ticking that thumping is my controller. It's my attempt at trying to make it you know, again sound musical. I didn't want just random sounds coming out. I wanted some sort of background, right? So we're going to do an experiment. I am going to run some of

these sounds. And I want you all to close your eyes. You trust me. Nothing weird's going to happen. And then I'm going to play some of these events in Kubernetes, and I want you to shout out what you think is happening based on what you just heard. Yeah? Let's give this a go. And yeah, I'm copy pasting commands cuz I'm on stage and I get nervous. Up.

I have a demo already. No. Keep your eyes closed. What is this called? Demo engine X. Oh my gosh. It's still soothing even if nothing is actually happening, right? All right. Let me see if I can try this again. Oh, it's my pods. It's not a deployment. That is not part of the demo. Just FYI. All right. Here we go. What do you think happened there? Excellent.

All right. Excellent. See, you guys are already able to monitor these things without even having to do anything. All right. And here's another one we're going to do. Yep. >> Awesome. And no cheating if you're like watching >> All right. That's our pod deleted. Very good. our last bit we're going to do here. We're going to create a deployment. Don't say it out loud. There are two

things that happened here. Excellent. Yes. We're doing it again. So this is the lovely music and the sounds of of doing this one by one. So what I'm going to do now is I'm going to use Chaos Mesh to try and simulate a slightly more realistic scenario where you have activity happening at once. The ultimate, you know, the actual events will still be pretty simplistic, but Chaos

Mesh can put it together faster than I can type and paste these commands into the terminal. Chaos Mesh Mesh, in case you don't know, is a CNCF project that lets you define chaos experiments using YAML, just like you would a deployment or a service. So, this is So, this is a workflow busy.yaml that I defined to do a few things in sequence and have some timing so

that they would happen, you know, closer together. These can get very complex. You can also schedule these to run if you want a little extra entertainment in your clusters to have some random things happen at random times. And you all can open your eyes if you want. I'm sorry. I realized I didn't tell you that. Um so, it's a nice tool for testing the resiliency of your

cluster. So, let's see if this will run. that's the chaos experiment. And it just keeps going and doing that sort of thing over and over again. It's kind of nice, kind of soothing. So, that's my demo. Kubernetes events turned into music, events that you can hear. Uh this is a very simple demo, obviously. It's a starting point. Right now, again, the controllers watches for very simple events

in the cluster, pod starting, pod deleting, deployment scaling, but this could be smarter. You could hook this into Prometheus to sonify continuous metrics that change over time like CPU or memory, node pressure, request rates, error rates. And that kind of continuous input could take the place of the arbitrary drone that I I put in using um SuperCollider. And then on the audio side, this experiment uses basic

synthesis like sine waves, filters, and reverbs. SuperCollider itself has tons of plugins that you can use to emulate all kinds of real life audio tools like phasers and choruses. You can also hook this up to other external devices and applications like Ableton, Reaper. You can get really creative with this stuff. I will say that a lesson that I learned in doing this putting this together is that

the I thought the controller part of this was going to be a struggle for me because I'm not a I'm not a Go programmer. But it turned out that the sound design is really the part that was really difficult. It's not so simple to make Kubernetes musical. Right? Like as you saw, it's pretty it's fine to generate random sounds, but if you want it to have some

cohesion, for example, if you want it to be in a particular key or you want to make sure that the sounds that you generate adhere to a certain key, that takes some work. And then, you know, shaping that sound, giving it like a reverb trail so it's not like a quick stab, but it trails off a little bit like you heard some of those do. That's all

work that goes into defining that synth, creating that synth def, and that's the part that was really the most challenging for me in doing this talk. with that in mind, I hope this talk inspires you to think creatively about Kubernetes and have some fun with it. Infrastructure doesn't have to be boring. Kubernetes doesn't have to be just YAML and dashboards and firefighting. So, I encourage you all

to go forth and make some noise. >> [applause]