Demystifying the Kubernetes Network Stack (From Pod to Pod) - Simone Rodigari, Microsoft

This talk focuses on Kubernetes networking, specifically emphasizing pod-to-pod communication within a cluster. The speaker explains the importance of understanding packet flow to avoid complications during troubleshooting. Key concepts covered include Linux network namespaces, virtual Ethernet connections, and the role of Kubernetes services in enabling load balancing and direct routing. The discussion also contrasts overlay and underlay networking methods for cross-node communication, detailing the networking setup initiated by the kubelet and the container runtime. The session concludes with an overview of debugging tools and methods for tracing packets across services, alongside a demonstration showcasing these principles in action.

Good afternoon, everyone. Thank you for joining today. My name is Simone Rodigari and I work as software engineer at Microsoft. And today we'll be talking about Kubernetes networking. Focusing particularly on pod-to-pod communication within a cluster and looking at packets. can be invisible. We have a set of resources defined in a in a YAML file. We apply those resources. We can see pods are running. Services exist. Services

provide load balancing between backing pods. So, everything seems to work. But what is really happening at the packet level? Clearly, there is a gap if we don't understand the packet path. And this presents a very big risk when something networking related breaks. Because, of course, troubleshooting would be very painful not time efficient. The goal for the session today is to provide a mental model how to trace

packets in a Kubernetes cluster. We will focus in in-cluster communication from pod-to-pod. Today I will cover six different concepts starting from the pod itself. The concept of Linux network namespaces and the importance for pods. The role of veth, virtual Ethernets. I'll move on to the connection between pod and node as well as touching base on the node data path. I will also discuss how pod can communicate

to other pods cross node. Cover the role of Kubernetes services in providing DNAT and load balancing. And also briefly mention how DNS can make our life a little bit easier by mapping names to IP addresses. Finally, I will provide a debug approach that can be useful in a troubleshooting scenario. To start off, we need to provide some ground rules. And generally speaking, those are the expectations that

Kubernetes has in terms of networking. Kubernetes is not responsible for the networking setup in full. Some of the ground rules include pods should have a unique IP address. There should be no network address translation between when pods are communicating to each other. we should operate on a flat layer three so that any pod can communicate to any other pods whether they are they are on the same

node or across different nodes. how is the pod networking configured if Kubernetes is not responsible for it? The process starts the scheduler defines that the pod needs to run on a specific node. The kubelet on the chosen node will use the container runtime to create the sandbox for the pod. The container runtime is responsible for creating the Linux network namespace which will be used by the pod.

And it leverages the pause container to keep this namespace alive. Then, the container runtime will invoke the CNI add which will use the CNI plugin binary to configure virtual Ethernet, the IP address for the pod, and the routes. So that the pod will be able to communicate to the host network including the ability to then communicate to other pods on the same node or outside. So, we

mentioned veth, virtual Ethernet. What is it? Veth is the connection between pod network namespace and host network namespace. Each pod run in its own isolated network and it's connected via a virtual cable which is the veth. Inside the host network namespace we will also find the data path which generally speaking provides routing and forwarding rules. So, at this point, if we look at this diagram we can

see that within the pod network namespace we have an eth0 interface. So, whenever we have an application container that wants to communicate to another pod on this node the packet will travel through the eth0 on the virtual cable, reach the veth on the host side and then the data path will establish the packet needs to be routed on the same node. And given that is the case,

obviously it will be routed on the same node. But what happen if we want to communicate to a pod on another node? We can see the eth0 interface also on the host network namespace. At the moment is not connected to anything. If we want to have pods communicating across different nodes in the cluster there are primarily two approaches. One is overlay and one is underlay. The overlay

uses encapsulation over the node network. It creates an outer packet with the IP of the node as a source and the IP address of the remote node as a destination. The inner packet will be our original packet sent by the pod which will have the source IP address of the pod itself and the destination um our destination pod. Overlay is quite portable. It can run on different

environments whether we are on prem or in the cloud. But of course, it provides a little bit of overhead. The second approach Underlay has the assumption that the pod IP is routable on the network. And sometimes this is referred to as native routing. It provides a better performance. Whether we can leverage overlay or underlay, It depends on the CNI of choice. And some example of implementation for

the overlay are VXLAN and IP-IP. BGP. For this talk, I'm not going to talk I'm not going to focus on implementation. I just want to provide concepts. So, right now we're able to communicate between pods on different nodes as well as on the same node using the IP address of the pod. But, what happened the pod is restarted? We know that in Kubernetes, if a pod is

the IP address could change. So, direct pod-to-pod communication is unreliable. To solve this problem, Kubernetes is providing us with a resource called which gives us a virtual IP called cluster IP and provides load balancing across ready pods. The implementation of the service data is done via kube-proxy or IP tables or NF tables or eBPF-based CNIs. To provide a a comparison between kube-proxy and eBPF I was looking

specifically at IP tables mode for kube-proxy. And the approach is translation via netfilter IP tables allow for DNAT to back end end points and it's also using connection tracking, conntrack. uses a programmable data path. The service lookup is done via BPF maps. And there is a direct internal forwarding without having any rule traversal. eBPF is more performant and it gives us a higher level of visibility. So,

it is the modern approach. However, for this talk I will demonstrate the usage of IP because again, I want to focus on and provide a mental model. The service semantics are the same. It's just a different data path. And again, this depends on the CNI of choice. Some CNIs might provide eBPF, some others don't. Also, worth to mention that kube-proxy uh has different modes, not only IP

tables, also NF tables and IPVS. So, I mentioned a couple of times DNAT. What is DNAT? I also mentioned that services provide So, DNAT simply means rewriting the destination IP while preserving the source IP. I mentioned that given to a service is a virtual IP There is no interface attached. So, we cannot use the the service IP for communication from the pod. So, the pod never talk

to the service directly. So, if we have a scenario where pod B is sending a packet to pod A using service IP we will find that the source IP address will be the IP address given to the pod the origin pod and the destination will correspond to the service IP. So, that this the packet. Within the service data path DNAT will be performed. will then have destination

of one of the ready pods backing the specific service. So, now we solved the problem of chasing a moving target because pods are ephemeral. But, we are still talking about IP We still have to remember IP addresses, which is less than ideal. We can use DNS to map names to service IPs. So, services provide a stable IP. DNS provides stable names. The default solution in Kubernetes is

CoreDNS. CoreDNS is responsible for watching services and then point slices and keeping track of all the changes. And it is deployed in Kubernetes as a traditional application, let's say. So, we have a deployment, we have a and a bunch of other resources. So, if we have a pod that is trying to uh is sending a DNS query in the cluster for example, NS lookup to a service

name pod A SVC, as we can see in this example. This query will be handled by cluster DNS which has a service of name kube-dns. Kube-dns will forward this request, this query to one of the healthy and ready CoreDNS pods which in return will respond with the service IP for the service that we uh send the original query for. So, basically the cluster IP of SVC. So,

right now we have a enough information to at least start tracing a packet through Kubernetes. If we have a scenario where a pod is trying to communicate to another pod across different nodes using a cluster IP service and let's assume that the DNS exchange already already happened. So, we already have the cluster IP. As I mentioned previously, the original packet will have destination cluster IP. So, initially

we can conceptualize that as a service abstraction. Once the packet reaches the data path we will have DNAT performed. At that point the destination IP address of the packet will change into the pod IP for one of the ready pods. Finally, on the data path, there will be routing and forwarding decisions. In this case we're routing to a remote node. The packet will be sent across the

It will reach node two and finally will be uh sent to pod B. So, if you want to debug Kubernetes we have a number of tools that they are Linux native. For example, IP which allows to inspect interfaces and routes. We can check translation based on our for the CNI of choice. We can use IP tables or eBPF tooling for inspecting maps, for example. We can observe

packets using tcpdump or other tools for packet capture. We can check DNS queries and DNS resolution using NS lookup and dig. And also, we can access Linux network namespaces for both the host and the pod using kubectl debug. And this is quite useful because it's allowing us to create an ephemeral container within a running pod so that we can inspect the pod network namespace. We can also

use kubectl debug to create a pod on a given node so that we can access the host network namespace. And this is exactly what I'm going to demonstrate. So, the setup for the demo is intentionally simple. I have two worker nodes on a kind cluster. The CNI is kindnet, which is based on kubenet is again the default. And we're using kube-proxy with IP tables mode, default option.

I will inspect IP addresses, verify that the pod to pod traffic works using the IP address of the destination pod, also using the cluster IP of the I will do some packet tracing and also use DNS. So, I will use the service name as well. So, we'll test all of that. So, I want for you to focus on the orange because the packet tracing will be done

on the host network As we can see, packet sent from the front end pod will travel through the eth0 interface on the pod network namespace, will reach the vet on the host, will travel through the the data path where DNAT is performed and all the routing and forwarding decision and then it will reach the eth0 interface to leave the node, travel through the network and then reach

the other Sorry, one moment. I'm having a technical issue there. Okay. So, first of all, I have a pretty quick demo, obviously. We check that our pods are up and and they are. They have an IP address assigned and that's done via the IP address management plugin I mentioned earlier about the CNI add plug call. And we also have a a back end with a cluster IP

assigned. Now, if we create a debug pod, sorry, a debug container within the front end pod, for the front end pod. We can see that we have the loopback interface and we have an eth0 interface. If we look carefully on the eth0 interface, we will find the IP address 10.244.1.11, which corresponds to the front end pod IP, as expected. check the IP routes, which which are configured

by the CNI. And these allow the pod to communicate to the node. Now, if we try to use the IP address of the destination pod, which is our back end, this works fine. If we use the cluster IP, it also works fine. If we use the service name, same result. So, what's happening? If we inspect the using an NS lookup the DNS query for back end, we

will see that the server is 10.96.0.10 and this corresponds to our kube-dns And the response we receive corresponds to the cluster IP of the back end Now, if we want to inspect what's happening on the host network namespace, we can create a debug pod on the node, the same node where the front end pod is deployed. If we check the interfaces, we will see a different set

of interfaces and this time we will see also a vet There is only one pod deployed on this node and that's why we see only one vet. If we had 10 pods, we would see 10 vets. Now, if we look closer into the eth0 and we inspect the IP address assigned to the eth0 interface, we can see that it's coming from a completely different range, 172.18.0.2 and

this corresponds to the node IP Now, if we do some uh packet capture, I need to remove the this bar, otherwise you're not going to be able to see. Well, we can wait 2 seconds anyway, it's going to skip up. So, if we do some packet capture with TCP dump, I use the flag I for interface any, Q for quiet, and I set as a host the

IP address of our front end pod just to avoid any other traffic. I used -nn to skip any host name resolution so that basically we just see minimal information. Uh I want to show the source and destination for each packet capture and that's where I want to focus on and also what interface those packets are going through. So, if we send another request using the IP address

for the destination pod, we can see the entire TCP exchange and uh if we focus on the first two packets that were captured, we can see that the first one is coming inside the host network namespace via the vet. The source has the IP address of the front end pod and the destination is the back end pod. Then, the second packet will go out from the eth0

interface. The source IP address >> [clears throat] >> and the destination IP address are exactly the same. So, there is no network address translation. Now, let's try to do the same thing using the cluster IP. And this time I will capture only two packets just to to make it easier to visualize. So, this time we can see the source the IP address of the front end pod,

the destination the assigned to the back end service. But the second packet will have the source and the destination will have the IP address of the back end pod. And this is demonstrating DNAT in So, something happened between the packet travel through the vet and reach reach the If we want to understand what happened, we can inspect the IP tables rules and if we focus on the

cluster IP assigned to the back end pod, which is 10.96 uh 168.11, we can find that there is one entry which corresponds to 10.244.2.15, back end pod IP address. There is only one pod. see the D NAT in action, but we don't really see the load balancing in action. So, let's try something else now. Let's try to use the service name. And capture again only two packets.

So, this time the traffic is UDP, is not anymore TCP. And we can find that the source IP again is the same front-end pod IP. The destination this time is 10.96.0.10, is the kube-dns cluster IP. And when the packet leaves the eth0 interface on the node, the pod the destination will be a completely new IP address, which is 10.244.0.3, one of the two CoreDNS pods. The second

one. Now, if we inspect again the IP tables rules, Apologies, you you're not seeing the command, but you will see it in a second. And we filter by kube-svc and we grab uh the chain related to kube-dns, so 10.96.0.10, we will see this time two entries. So, one is 10.244.0.2 and the other one is 0.3. So, this time we can also see not only the D NAT,

but the load balancing. And in fact, IP tables is using statistic mode random with a probability 50% for each of the two. So, if the probability is 50%, it means if we send another in theory, it should it should use the other pod IP address, right? Let's try. Let's capture again two packets and let's send another request. And we can see that this time the other IP

address was used. So, this effectively shows how D NAT and works in Now, you probably noticed that the communication between front-end and back-end is working fine. It's worth noting that also the other way around is working And this not always ideal, right? So, just as a teaser, I'm not going to cover that today, but we probably want to have a little bit more security. And the fact

that Kubernetes by default allow any pod to communicate to any other pod, it's not really sticking production. So, to limit that, we can use network policy resources to provide a sort of a firewall for And this again is enforced in the node's data path using IP tables or eBPF, for So, key takeaways for today is that Kubernetes networking is is built on Linux. namespaces network namespaces, they

with the ability to have a isolated space for the pod. Virtual ethernet provides communication between the pod and the host. Then we have routing. Some CNI use bridging. Services provide a stable identity over ephemeral pods. And finally, if we want to follow the packet and troubleshoot it efficiently, we need to know what is the packet path. Thank you for attending today, and I know it's late, so

you you're probably looking forward to go for dinner. Enjoy. And uh if you have any questions, >> [applause]