An Immersive and Visual Journey Into Kubernetes Networking - Benoit Entzmann, Feesh

In this talk, Benoît Ensmann, a DevSecOps consultant, explores the complexities of networking within Kubernetes using a 3D game as an educational tool. He introduces fundamental networking concepts by visualizing packets' journeys through Kubernetes pods and discusses the functionality of Cilium and Istio, both established projects from the Cloud Native Computing Foundation. The presentation demystifies how packets are processed across various layers, explaining concepts like MAC addresses, IP addresses, and transport layers in terms of game mechanics. He also delves into more advanced networking scenarios involving service meshes and solutions like eBPF, VXLAN tunnels, and DNS resolution via CoreDNS. The speaker contrasts traditional networking methods with modern approaches, highlighting the benefits of service mesh architectures in Kubernetes environments.

All right. So, thank you. It was I was just on time to set up the the screen. Um thank you all for for coming. It's amazing to to have so much so much of you here today. So, I wanted to show you something different than PowerPoint, but obviously it's not as easy to to set up as a a basic PowerPoint. So, I'm Benoît Ensmann. I'm a DevSecOps

consultant for Fish in Switzerland. And I have a background of two decades in a networking and telecommunication. And I can tell you that when I was starting with Kubernetes a few years ago, um it was I was as confused as anybody with the networking. So, I had this idea to develop a 3D game just to explain the basics and go up to more advanced scenario. And this

is what I'm going to to show you today. Uh just for me for for the information, who is beginner who consider beginner with the networking? All right, good. So, a part of you. So, that's good. So, I hope I will explain it to you with my with my game and you will get it a bit better. uh four four level today, um two with uh Cilium and

two with Istio. Both are graduated uh project from the CNCF. And so, we will see how they operate. And the the idea is to see the the the packet exactly how it goes from one point to another point. So, let me bring you into my underwater world. So, consider imagine you jump into the water. You you are diving and then you see some buildings. The buildings are

the node into Kubernetes and you can see in some of the room there are some light. That's the pod up and running. That's what we like. And so why not start from the pod from the room and see how the packet is processed from one room to another room. So that would be the first scenario. And hopefully the Yeah, so the screen looks good now. So I

will explain to you. So the fish is the packet. The packet is the the layer one is the physical. On the top you see the information it carries. So you see also the the fish has a backpack with uh four colors. That's the four colors of the layer. The information are layered organized into layer in the packet and the the color match what you see on the

top. So layer two, that's the traditionally is the MAC address and everything. That that's but that's not used by Cilium. So I will show you Cilium inside Kubernetes. Layer three is the routing, the IP address. So instead of an IP address I will use some room number. So it's going to be easier to to understand. And layer four, that's the transportation layer. So we have a TCP

packet here. And the layer seven is the payload. So you see you know on the IP packet five, six, and seven are inside the same what we call layer seven and it transport the payload I want to deliver to the to the far end. this is the for the for the for the packet. On the left you have a mini camera that show the view a little

bit from the top. So you have a bit of a better view of the of the the path of the of the packet and on the right you have some information about where we are when we move from one point to another point. So the the goal here is to to reach with this basic example. I am in the room the front end room and I want

to reach the back end the back end So for this basic example, we consider that the layer three So the destination is already known. So we have 14 here. So on two digit, it's the an IP address of a room. So a room body is the is the same as you as you When you will see three digit, it's a service. I will explain you later what

what it is and four digit, it's the the IP the the equivalent of the host of the building. We will see also an an example of that a bit later. So we are almost ready to start. You can see there is a a shellfish inside of the room. That's the IP table the the routing table that is available inside and it points to the direction. The direction

is the door and the door is the networking interface. So you see the the analogy it will help you to to understand and so we are ready to start. So let's let's move. So I'm just leaving the the room. You can see the So the the door has an information door one. That's the color of the layer two. So that's the the equivalent of the of the

MAC address. It's not you Cilium doesn't use that for routing inside of Kubernetes. So it's just here for the for the sake of of completeness Uh, it's not it's not used. So, when we leave the room, so we set the So, that's the 11. It's a It's the um the room number where I'm starting from. And then I reach the the corridor. So, here you can see

a bit better on the on the left. What is the corridor is the uh the link, the virtual internet um interfaces. You have one end is inside of of the container, the other end is on the host. And uh here it's uh it's just a corridor. That's how I move the all the packet move from inside of the container and reach the host uh on the on

the So, I reach the host. I have another interface, so I am on the on the host. And as soon as I reach that interface, there is a new character you can see. Uh that's the eBPF program. And the eBPF program is is lo- located uh at the output of the interface and is able to process my my packet. So, what it will do, it will have

a look at the layer three information, so the destination, where I'm going to. And it will able to process it. So, eBPF program, for those who don't know, it's um a program that runs inside of the kernel. Yes. And when you do that, so you see a bit better on the left, it just prevents the packet from further processing because uh normally the packet would go to

the netfilter, you can see on the on the left, the netfilter subsystem. It is you go down the stairs, and it is where leaves all the IP tables, NF tables, IPVS. But with eBPF, it stop before is the shortest point at the output of the interface to reach your destination. So, you have a look at the destination 14, and it's straight on my on the left. So,

I can go directly and I have a boost. It's a little bit easier. So, I show you I'm swimming down to the destination, but normally it's more like a it would be really like a jump. You would jump in front of the destination container. Here there is another eBPF program. The eBPF program uh for the from the packet uh routing point of view doesn't do much here.

It's just preparing the uh the connection back when the packet goes back and it just let me enter into So, this is the interface on the host. I travel to the corridor. destination. You see it match. It's 14. And it match uh the the packet. That means I've reached my destination. I enter into the back-end room. And then I reach the port. So, also something important, the

the listening port, that's the layer four information. It has to match with what is from my packet. If it doesn't If it match, uh then it's okay. If it doesn't match, the packet would be discarded. So, the the port uh is uh just uh the listening device uh for the process that is behind it. Okay, so that was for the basic. I hope that that made sense.

Now, we will have a look at a bit more realistic example of how it goes in a in a real uh in a real cluster. So, at the start it's exactly the same. Uh but now you can see the destination is uh not uh the the the pod the pod the the room. Uh I have a three-digit number because it's uh And this is this comes because

uh I want to reach the backend.be. That's what you see on the on the layer seven. That's a name. And inside of the Kubernetes cluster, you have um a DNS resolution. And this component is called the CoreDNS. So, when I want to reach uh any destination and I just have the name, I have first to go to the CoreDNS component to get the resolution. What is What

is the room number associated with uh this name? And then I will go to my So, here the destination of the CoreDNS is not just one pod. It's a service. So, a service is not a a room inside of the of the building. It's It's a an object a Kubernetes object that lives inside of the cluster. And it has an IP address or a number here in

on three digit. The purpose of the service is just to have a stable IP address uh to to serve the the pod it it use using that uh that So, it's just a basic load balancer in front of the pod because the pod, you know, can live and die. And when they are recreated, the the room number will change. The service will not will never change. And

so, that's why we use uh a before. And the service will load balance the traffic to the final pod uh behind. So, let's go to that uh CoreDNS uh room. So, everything is the same as as before. So, now there is a new character. You can see it's a bit uh a bit uh a bigger. It's a blowfish. It's a spiky as well because that's an egress

network uh policy. That means it's going to check where I'm going and it will allow only it will allow or not to go to that destination according to the the the room I'm coming from. So we check my destination. It's always a good idea for the CoreDNS to allow all the traffic because otherwise it will create some issues inside of the of the So we just checking

that if I'm allowed then it does the routing and it does the service load balancing and it give me the room number that is behind that service. So it's the one of the pod of So I'm going So I'm It's like before I'm going to to a room. There is an eBPF program let me enter into the room. And I reach the destination. It's a bit dark

I see it now on this screen but uh it's much it's a destination so I can just enter here. And the listening port is 53 that's the common the well-known port for the for the DNS. It matches and so at this point I receive the the the IP address of the service of my back end. That was I was looking for. Uh so it give it it

give it to me. So normally there is a response that goes back to the front end room. This just a shortcut. I start from the front end room. And with that information I create a new packet. It's a TCP packet now to reach my back end. So you can see the destination that was given by the CoreDNS. It's a 201. So three digit that's a service. And

then I can now reach that So so far is the is the same now. I go out and there is a a network policy. So, each time I go out this room, I have the network policy and it check where I'm going to. As I'm going to the back end, is uh going uh to to allow me, but if I was maybe going somewhere else, then it

will drop the the packet here. And then it does the service translation, give me the the the room number. You can see it start with two. That means it will be in a noticeable in uh my cluster. So, just go up. I'll go a bit quicker. So, now I'm going to move to another uh to another building. So, we use what we what is called a VXLAN

tunnel and that's um an en- encapsulation. It's like putting the information you see on the backpack, it's putting on a bigger backpack and it create some new layer two, layer three and layer four um l- header into that packet. That's the representation of the of the tunnel. That's also So, VXLAN tunnel is an implementation of what we call an overlay. Overlay, it just allow the pod to

communicate whatever the building they are living living on. So, I'm just entering that tunnel. So, the packet you can see that's the inner packet you can see on top doesn't change, but there is some It's not represented here, but there is some outer uh header the outer header are just using some of of the room. So, on the left you see the tunnel is straight, but between

the building is the traditional routing applying. So, in that case it's using layer two and layer three. That's the interface uh of the building, the main interface of the building it's using. So, it's a straight on my game here just for for simplicity, but there could be some some traditional routing between the between the node at this stage. So, at the other end of the tunnel, it's

just decapsulation. So, I get nothing changed on my on my And so, I'm I'm the building number two. this character. Now, it's an incoming uh That means that is checking where I'm So, um I'm only So, I am in front of the of the back end and he's checking if my origin is a is valid. So, if it is the front end, it is and then he

let me enter. So, that's another purpose of having EBPF program here. It's to check the the traffic. So, the destination match. the back end again. Okay, so that was the kind of easier scenario. Now, we will use service mesh to see something a bit more advanced. So, we want the the front end of the of our application, we want it to expose to the world and so,

we want it to make it reachable from anywhere into the world. So, how do you reach the front end? The front end you just have a name front end. dot And no, my fish is not inside of the building anymore, is living anywhere in the big blue. And so, with this name, what he has to is to have a traditional DNS resolution. So, that's the basic DNS.

I give the the DNS give me a an IP. And then I create a new a new packet. So, it it's a it's a shortcut. And normally, you have the response and then from the response you create the TCP Uh and then I have a destination a new destination number. So, you can see four digits. That's the the building somewhere in my Kubernetes. And that's the IP

address of the external IP of the service that is going to receive my packet. We have deployed this service uh load of type load balancer. That means we allow the traffic to come from outside of the And behind that service, we use an ingress gateway. That's going to be the the pod that's going to receive my my So, here there are some traditional routing. You can see

the the routing device. That's the the she seashell pointing to the direction. So, it could be very long and it's a I try to shorten it a bit. Who is very happy? Okay. So, um and so you reach the front end of of a building. This interface uh carry the the virtual internal IP. And you can see I enter to the building, but I'm not inside of

the Vexlan tunnel. I'm I'm on side of it uh because I'm not coming from another building of the of the cluster. I'm coming from from outside. So, what is behind um an interface? There is an eBPF uh And here is able to read the destination and give me the room number of the So, here is exactly the same as what you you know now. So, I just

enter the the And I've reached So, it it's a it's a pod that's a living inside this uh uh this um living inside in inside this room. and that's the ingress gateway listening on the port 443. So, ingress gateway, it could be for example deployed with a gateway API with a Istio. From the packet point of view, it doesn't matter if it's gateway API or the the

older Istio API. It just needs to have a destination. So, my front end, I've configured it to be part of a service mesh. That's what I use That's why I use Istio. And from there is a processing of my packet. He's going to have a look at the layer seven. He's going to read at the at the host name and then he's going to look at your

service mesh configuration. I will skip the details of that, but the packet what it needs it just needs all this information and from here I'm starting the the service mesh. I'm entering the service mesh and a new packet is created with this information. And now I'm using mTLS, the mutual TLS. the the protocol used so it's with with the service mesh and my destination is 101. It's

a service like before. And so I'm going to try to find that destination. The eBPF program is going to give me the pod number, the room number. It's here so everything that it's what you already know. What's different is that this service this application is part of the service mesh and the service mesh uses sidecar in this scenario. And so I have the match for the layer

three and the layer four. But the difference is the the the service mesh uses sidecar so there is some configuration injected inside of that application. That means there is a redirection of the destination. and that's the IP you the seashell you see on the right, it redirect the traffic to not go straight to the application, but go to the side car instead. And the side car is

listening on that 15 006. The side car is just having a look at the layer four and layer seven. It's an Envoy Envoy proxy and is going to do what the service mesh does, observability, tracing, advanced routing. It does all that and then you redirect the traffic inside of the room. So that's the only case where you have a room with another separation. You don't leave the

room. You go on you go on the left. There is a loopback interface that is shared between all the the space inside the same room. They all share the same And then you go through it and that's how you will finally reach the front end through the through the service mesh. I hope you're still with me. Maybe it's a lot of information. Yeah. Now we have a

look at the bus the bus level. That's the service mesh ambient. That's the most way of uh the most recent way of configuring a So we are going to do exactly the same scenario, but with ambient. So from the beginning everything is the same as before because the traffic starts from outside of the cluster. everything the same. Start the routing from outside. Some redirection. Oh, and that's

not good. That's a routing loop with the dramatic effect you see that's CubeCon in 2026, so routing loop is very bad in routing because they that means the packet never network engineering ninja engineering have to to work on that. And when it's fixed, you can finally reach the Okay, so that's exactly the same as before, so I'm I'm going quick on that. So, you enter into the

the cluster. You enter by the the endpoint by the by the ingress gateway. So, any traffic that has the destination of the external IP address for the is going to reach that ingress gateway. And the ingress gateway is going to to process the packet and start the the the So, now we enter into the service mesh, but now the front end is part of service mesh ambient.

So, that means they're going to be some a few changes. The destination, for example, the the the the destination port is different. The layer seven is different. We use the edge bone protocol. So, I've just simplified edge bone mTLS, which is one of of the component of the of the edge bone. And the packet is encapsulated. what is different from the service mesh with side car is

that edge bone is a as is a better performance. mTLS is already established between all the all the point. You already have some connection point, and you use that to reach the destination by being encrypted. But from the packet point of view, there is no changes. There is nothing more ungrateful than the packet. It just need a destination. need the destination for the service. It goes to

one. So you see it's another He's very happy. He's taking some liberty. Okay. So you Do you enter the way Islam tunnel is the same as the same as And you go to the to the other building. Good. So you reach the the destination. That's the the front end. But now that's where the logic changes from the sidecar because now we use mesh ambient. You can see

the destination is different. Uh and there will be a redirection like it was before. When I go here, you can see there is some other component. So just one word about you can see the the seashell on the right. That was does the redirection for the packet to go to the Z tunnel before entering. so there is two new components here. The seashell you see on the

right that does this redirection is using what we call Cistio Istio CNI. The CNI is what comes on top of the Cilium CNI to to give some more advanced routing. set up the the application as part of of the service mesh ambient is going to install the IP table information into the application. That's the that's the seashell. And that's how it goes to the Z tunnel. So

the Istio CNI is a demon set for the you know, so it's just one one room per building is the Istio CNI. And then you reach the Z tunnel. Z tunnel is also a demon set. You have one room inside of the building. And the Z tunnel also has an installation there is a listening socket inside of the application that is being able to process my my

So the packet is being processed by the Z tunnel. So Z tunnel is not Envoy is coded in Rust now and he only process the layer four of the packet. So layer four only what is in in pink and it does what service mesh does but only with the layer four information. So observability tracing and So there is something more to to this because the application is

rich but I have configured a way point. Way point is another component of a service mesh ambient and it's a deployment. It's not a demon set and it leaves also in I've reached the Z tunnel but I have to go to the way point because I want the layer seven also to be to be processed. So when this is configured I have to go that way point.

I will go quick here. And the traffic is the same it's a it's a new edge bone packet all encrypted and you reach the the way Here you you the way point have a look at the layer seven information process all the the observability and everything with the layer seven information. it goes back to the to the Z tunnel. So this happens only when it is configured

to have some layer seven processing otherwise it will stop at the Z tunnel only and So the same process happen I go back to the Z tunnel and here you see it comes from the way point so it doesn't go back and you just finish the the the tunnel here. The packet is unencrypted, it stays into the the front end, and then it goes directly to the

but it's only one one room, no no separation, and then I reach the the I hope that makes sense. It was a bit quick, but that was only the time I I had. So, there could be some more advanced scenario to to see a pod-to-pod communication, for example, from inside of the service mesh ambient. There could be some multi-cluster and everything, but that maybe will be for

a version Thank you. 45 seconds for the question, if you have. It was not on purpose.