Let Your Network Speak! - Nadia Pinaeva, NVIDIA & Joel Takvorian, Red Hat

This talk focuses on network observability in Kubernetes and the integration of observability tools using the Net Observe operator, which utilizes eBPF agents for collecting network traffic data. The speakers, Nadia from Nvidia and Joel from Red Hat, explain how Net Observe can monitor network flows and correlate them with Kubernetes resource information like pods and services. They discuss features such as DNS decoding, tracking TCP latency, and the ability to create custom metrics and alerts based on network flows. A live demo is presented to show the practical application of Net Observe, including the visualization of networking events and debugging network policies. The integration with networking plugins, especially in managing network policies and understanding traffic drops, is also highlighted, demonstrating how Net Observe provides valuable insights into network security and performance.

All right, I think we can get started. Welcome everyone. We are very happy to see each and every one of you at this time. We'll try to keep you entertained and not falling asleep with our lively demos. So today we're going to talk about network observability and how you can even better integrate observability tools with your network and actually let your network tell you what's happening. So

my name is Nadia. I'm a senior software engineer at Nvidia and I'm working on Kubernetes networking. And I am Joel Tak Voyant, software engineer at Red Hat, working on network observability. Um so we'll start by introducing Net Observe. Um it's a an operator uh open source operator for Kubernetes to observe network traffic uh based on eBPF agent. So it uh basically installs some eBPF agents on the

cluster on every node and going to collect information from all the traffic that it sees and send back send that to collectors. It's really open source. For the most part it's uh CNI agnostic. So for the most part I mean that some features sometimes um are CNI dependent like the one that we're going to see today is something when you want to correlate information from network policies.

Um network policies are something implemented in the CNI. So that's why sometimes we have uh CNI dependent features. But for the most part it's a Uh it was started at Red Hat as a network observability operator and more and more we are adopting a more vendor neutral uh stance and uh trying to push that maybe at some point as a CNCF sandbox project but we're not there

yet yet. what are the the features that it has? it collects network flows from packet data. So it reads packet headers in the kernel, get IPs and ports information from source from destination and that's basically the the raw data at the basis of network flows. On top of that, it's going to correlate with Kubernetes information. So Kubernetes resources are that are IP based like pods or services

or nodes are going to be correlated with the IPs and so injected the pod name and name space etc. It's going to be injected in the flows. Other kind of features that it can do is a DNS decoding like capturing DNS errors for instance or name name servers. RTT is a TCP latency, packet drops. So when a packet is dropped by the kernel, it's possible that we

get the drop cause and and inject that also into the into the flows. For enrichment, it's also able to do nothing to to follow nothing get information from cloud availability zones and soon there's going to be the new pull request currently open. So soon we will have a TLS and quick usage tracking. In order to see if your cluster is secure as expected on TLS and things

like that. And with OVN Kubernetes, some features that I just mentioned was network events like when a network policy is allowed or denied, that's going also also to be visible in URI and enrichment for network segmentations and one thing interesting also is the API extension because with all of that data, you're going to see a lot of network flows, but you can also create your own metrics

based on that data and when you create metrics, you also can create alerts. So you can really customize for your own use case. So going to show the demo. Live demo. So, this is NetObserve. There should be something. Yeah. Okay. this is the view where you're going to see all the network flows in your clusters. So, the every line here is a is a flow. So, you

see it's a by source name, destination name, and name space. So, it can be pods, it can be services. There are more information here, the pods, etc. And well, I'm going to remove the filters because that's for something later. And we you can Sorry, you can manage columns because there's We don't show everything on this table, but there's much more information that you can get. You see

there's a long list with DNS information for instance. Here that's For this example, I've enabled DNS feature, the only one that I did for the demo. And I I also introduced a a little DNS typo in in one of my pods so that I can show you how that works. When you click on a row here, you get more more information about the every You get for

instance the protocol, the TCP flags, the even the network interfaces that are being where where it's that was collected from. And packet and bytes counters, things There's a topology view. So, from all that raw data, we can provide more visualization options. So, the topology shows all the the traffic between your different name spaces, and you can change the aggregation level by owner for instance or by resource.

So, owner is like deployments, demon sets, that's the pods owner. And resources is the the the more narrow when you drill down down to the pods and services and individual IPs. And in overview you get some high-level metrics, high-level charts with the same kind of aggregation you can do by name space owner, etc. And you can see here a bunch of uh different DNS errors for instance.

So the more you enable features in the eBPF agent, the more you get information and then here in the in the charts. Um and the network health is something that we've done recently to be able to show alerts or issues in the network traffic that you have. So NetObserv comes out of the box with a set of predefined alerts or or health rules that you can uh

configure. And here for instance, I see that in my application which is called Measure Arena, I see that I have too many NXDOMAIN errors. So if I just go here, so I have a runbook which is a basically a static page that's uh explain the error that it is. And uh inspect network traffic, so I'm moving back to the to the other view in network traffic. But

you see there is a new filter that has been added. By the way, I can really uh type whatever I want here. I'm going to type DNS name for instance is not empty, so I'm going to check all the DNS name that I have. And if I look here, that was the typo that I introduced, so there is something wrong here and I can quickly spot it.

Uh and by the way, you can also notice that there are um combinations of suffixes that are used and that's because of the DNS search in the cluster. When you you maybe know that um uh the address it be ambiguous for DNS and it's going uh the resolver is going to append several suffixes and do searches based on that and that's a way where you can uh

potentially improve performance with your DNS queries uh by by I don't know adding a trailing dot to the URL as that that's kind of things that are explained in the runbook. That's it for my demo. Okay. All right. So, now you have seen there is there are lots of features and we haven't shown like even a half of them yet, but they are they exist separately from

the networking plugin. So, observability just is there. It's looking at what at what's happening, but that's it. It doesn't integrate with what actually implements the networking in any way. So, we were thinking maybe we could change that and get some benefits from that. So, the main pain point we're trying to solve is network policy debuggability. If any of you are familiar with the network policies, that's a

Kubernetes API that ensures network security. Basically, what it does is it says, "I want to isolate these pods in this name space and that is all I want to allow." So, that's a very simple example, but that's basically what it is. So, now pod A in the default name space is isolated, but pod B can still reach it because of this ingress rule. That's TLDR on network

policies. Now, another important part about network policies is that there is almost never just one. It's a name space resource, so all the name space owners want to protect their name spaces usually, so they will create lots of them in every name space. You will very likely end up with hundreds and thousands of the of them. So, it is getting at some point very difficult to understand

which network policy allowed or dropped your traffic. And then it's been a pain point for a long time. Um now, another point to add to that is we have not so long ago introduced an admin network policy or maybe cluster network policy is the latest version of is a cluster scoped uh resource, which is very similar to the namespace network but it is for admins and it

works on the cluster level. So, now it adds an extra persona to that, which is an admin, who could configure cluster-wide policies that also do something to your traffic. But let's say if you're a namespace owner and you run your applications, you may not even be aware that those cluster-level policies exist. But they still may affect you. So, the whole debugging problem is getting even more complicated.

you cannot really do that on a generic level. So, we have been working with the Net Observe um and Oven Kubernetes together. So, Oven Kubernetes is a networking plugin that implements a very high level of what it does is it translates Kubernetes objects into Oven database entries. Oven is an open virtual network, which is another thing that translates these things into OVS if you've heard of that

before. So, open vSwitch flows. uh again, a very high-level view of what's happening is we have a network policy. That's a Kubernetes object. Then, there is Oven Kubernetes that translates this network policy into an ACL. So, ACL is a thing that tells you what to do with the packet if it matches a given condition. So, in this case, it will tell you, "Okay, if the packet is

coming from pod A and its destination is port B. I want to drop that. That shouldn't be allowed. So, this part is like control plane how it's programmed. Now, here when the actual packet comes, what happens is there is a check for a match. So, you check if that condition applies to this packet and then the action is applied to it. Can be deny, allow, pass, but

anyway, that's what happens. Now, without any integration from the network observability point of view, you will see that the packet disappeared. You wouldn't really know why. So, we were thinking, can we make this better and actually provide extra information from the decision-maker here, which is the network plugin, to tell more. So, the idea was to add an extra sample to things like ACLs, for example, here that

would then, every time a packet matches this condition, it would send a sample somewhere with the packet and the ACL that made this packet be dropped or allowed, potentially, to some sample collector. Basically, whoever is interested to know that information. that sounds easy, right? So, we'll need to dig a little bit more into how that actually looks like. There are a couple components involved, but we've covered

that part already. So, on the top we have the Open Kubernetes pod. Lots of components are not depicted there, but you remember the main part. Network policy is translated to the ACL and the sample in the database that eventually goes to the host obvious database, and then there is the obvious kernel module that actually, well, handles the packets. Now, the packet comes, there is a match happening

somewhere Now, I want to see a sample. Before it would just like forward it or drop it or do whatever, I want extra action, which is happening here now. That's like a fairly recent feature. what happens is we send a B sample, we use the netlink multicast and some of the kernel capabilities to basically let whoever wants to know know that something has happened. So, what this

thing will do is if you have some listener or a subscriber for that specific type of event, through this interesting way, you actually will get a packet and the original ACL data that will tell you why some action has happened. So, while you are still here in the oven Kubernetes pod, we ourselves program the network policy to ACL mapping. So, we can map it back, right? Because

here I just get the ACL ID, which in the Kubernetes world doesn't mean anything. So, but with this special look up, I can just, well, map it back or I can figure out what network policy that was originally, so I can provide the actual Kubernetes specific information after Now, that, using the netlink multicast and moving all through all these things on like every packet, potentially not necessarily

every packet, that's configurable, maybe slightly slow. So, Yeah, so where net observe comes in here is that the BPF agent is going to uh inject this hook into the send B sample function, so it gets all the information, all the parameters of that functions, and every time that the function is called, the hook is triggered, and so the BPF agent is able to get that information, aggregate

that into hash maps in the kernel, so there's different ways to do. Could not directly send it to the user space, but that could sort of defeat the problem because one of the interest of having hash maps here is to aggregate data and to minimize the amount of data to copy to the user space. So, that's where that's where it is performance solution. So, we get this

ICL ID from the P sample function and we are going to correlate that because we also have information about the the the high the IP headers and things like that. So, we are able to to know which packet was involved and from that information we are able to re-inject that into the network flows that we have. So, so we do this correlation and with for with a

given flow we are able then to to associate it with this ICL ID. And then that's moved to the user space. And on the user space that's going here to be correlated back to the Kubernetes object like Nadia was saying. And that's the point of integration. So, you can see that Net Observe and the networking plugin actually have a couple of points of integration because they have

to talk to each other to actually provide the right information. But, it works. So, Now, I'm going to show you a less brief pre-recorded demo of using all the upstream components that you can find using Net Observe and Oven Kubernetes together. I also will add a link for the blog post saying exactly which steps to take. It's fairly easy. You will be surprised how easy that is.

Okay, let's see if that works. So, we are going to do a very simple case where I'm going to just create a couple of name spaces, one pod in each, and we going to all the way work with well, one network policy mostly. So, I have two name spaces. I create a server in name space blue and I create another one in namespace red. The general idea

is blue should be able to curl red because red provides some important data. Now, what we start with usually when you deploy something is you want to check if your networking is set up. So, you just ping something and it doesn't work. So, you're thinking maybe I applied some network policies that are dropping that and you don't see anything. So, probably the networking is not set up

correctly. But, then if you go to the net observe, you'll see these flows that Joel actually showed you before. So, hopefully you can read this interface. Now, we have a new field here which is the network events which shows the result of that whole integration we were just talking about and that's the extra information that we need here. So, the namespace blue was reaching to 8888 and

you can see the network events is actually telling you what's happening. It was dropped by admin network policy by the name drop Google DNS in the egress direction. So, even though you didn't create any network policies in your namespace, there was something created by the admin and you we can infer from the name what was the idea. Sometimes you cannot and that means you need to reach

out to your admin of your cluster maybe, but at least it kind of gives you an idea of what's happening. And in our case, we didn't really need to reach that IP. It was just to test the network. So, let's try to just pick another one and see if Okay, so that's better. We can reach something. Now, let's see if the original idea of like pod blue

trying to ping pod red actually also will work. Uh for that, we'll need to find the red IP, I guess. Makes sense. >> Okay, and there is like a web server running there, which just returns the time. Simple thing. all of that works. We are happy, but now we haven't secured anything. We actually haven't applied any network policies ourself, which was the idea to make sure it's

actually somewhat secure. So, we create a network policy in our namespace that will allow the connection we are trying to make here. We just specify on this protocol, this port we want to allow. Everything else should be denied. So, we'll try the connection again. that doesn't work. You may not really know why, or if you're really paying attention, you may know why. But, we'll go to the

Net Observe again and see, because you should trust me that happens all the time. Um so, here is our connection namespace blue to namespace red. Very easy to see and navigate. And it says that it was dropped by the network policies in names in namespace blue. So, the network policy we have just created. So, we need to take a closer look at that again. And you can

see that we actually typed the wrong port. It was 8080 and not 80. So, I allowed not what I actually wanted to So, I need to go and fix the network policy after that. So, and try to ping again. Well, ping is the wrong word, but I'm going to keep using that, I guess. So, it works now, uh which is good, which is what we wanted to

work. We can go and just double-check with the Net Observe again that blue to red is now allowed by the network policy that we created, as expected. Everyone's happy. Now, we want to see if something else is dropped, because we wanted to only allow this connection, so everything else actually should not work, right? That's the point of network policy to secure your name space and drop all

connections that should not be allowed. So, we'll just try the original thing again that we saw working before. Now, it doesn't work anymore because it's hopefully being dropped by our We'll go and double-check again because the result of the connection doesn't always mean it's been doing what you expect it to do. So, yeah, you can see here that blue to 1 1 1 1 1 dropped by

network policies in our name space. So, works as we wanted. as you know in real clusters, two nice name spaces do not always keep leaving just together. Someone else comes up with a new yellow name space. I as a blue name space actually also want to be able to connect to that name space. So, what I usually need to do here is go and just update my

network policy to allow that, too. But, let's see first if the connection will fail or what happens before I do that. So, that's the yellow pod IP. We'll just do exactly the same command and curl it on this yellow name space from the name space blue, and it works. But, I haven't updated my network policy yet. So, something must be wrong. And here we can go and

ask the net observe again what's happening. So, you can see the destination name space is yellow, and it was allowed by the network policy We need to go and take a look at it Let's see how quickly I can type. No typos. So, the network policy here shows that we have only filtered on the port, if you can see. There is no mention of the name space.

So, what this network policy actually does, even if that's not what I wanted, it only feel it allows everything going to that port. So, any namespace, yellow, red, whatever, anything on the internet with the same destination port is going to work. That's not what I wanted. So, I need to go and update the network policy again. I will leave the port because that's what I wanted, but

I will also add a namespace selector that will only allow that thing going to red and yellow namespaces and not everywhere I want. So, that's side on or third update or something to the network policy trying to actually get it right. Um we'll go and try to ping the yellow namespace again. It still works, which is good, even after we updated the And now let's see if

another namespace ever comes up, is that going to be dropped or is it still doesn't work? Any bets? So, let's get the IP of a green pod and just check if our network policy is correct or not. That's kind as expected. So, now you can see when we created a new namespace, it wasn't just allowed because of the port. So, the namespace selector actually works. You can

see that the net observe says that it was dropped by the network policy to namespace green, but blue to yellow was allowed because we updated the network policy to allow yellow and allow red namespaces, but the green wasn't allowed there. So, that's basically the full demo. I hope it was somewhat possible to follow what was happening. Keep in mind we have been just playing with one network

policy this whole time. There are usually kind of hundreds and thousands of those. So, in real life it would be much more difficult to spot a similar problem. So, we have seen that this kind of tooling could help you with this cross persona communication when there are admin network policies and name space policies and you can see if there are any admin network policies interfering with your

traffic easily whether it's dropping or allowing sometimes unexpectedly. You can also get a fast debugging of drop connections. So, when something is dropped you just go to the net observe. It will if there is a a reason for why it's your direct answer. If not, you need to debug further. And sometimes connections are allowed but for the wrong reason which is another problem because network policy is

there to provide security for your But if everything works, no one will double check even though you may be allowing more than you wanted to originally. So, that case we also kind of try to demonstrate a little bit. And then in the future, so all of this was like based on this ACL and kind of traffic control things but more types of samples may and probably will

be added in the future to provide even more networking insight on why any of the networking decisions book were made. It could be related to routing, forwarding like um network class, all kinds of networking based decisions that will directly explain to you why something And yeah. Yeah, and the last >> point is on that observer. Like I said before, there is an flow metrics API is an

API that allows you to create custom metrics. So, that that works with any things any things in the network flows. So, that works also with network policies. You can create metrics um that track the number of the counter of denied policy or allowed policy or whatever and create a lot based on that. Yep. Is that it? All right. So, that's all we wanted to show you today.

Thank you for your attention and if you have any questions, we are happy to take them. We have 5 minutes or so. Just in case someone wants to ask a question, there is a mic. Please just go ahead and So, what's the retention of the network traces and does it require storage? Yeah, so the retention So, NetObserv by itself does not manage the the storage side. Uh

it can connect to it can export the flows to Grafana Loki. That's the main thing where you get all the details of the flows as as logs, structured logs, and then you configure the the retention period like you want using for instance the Loki operator. It also derive all these information as Prometheus metrics. So, again, that's not something managed directly by NetObserv when the storage part. It's

all delegated. Thank you. And last thing is that it can also export the flows to custom exporters like through OpenTelemetry uh logs or metrics to a Kafka produ- as a Kafka producer, then you plug a consumer and you do whatever you want with that. Or also using the IPFIX protocol, which is something dedicated to NetFlows. And so, the And then you you also manage the consuming side

at this at this time. Thank you for the talk. Um I have a question about is it possible to extend or maybe you have a plans um to such feature for network policy for others CNIs like maybe Cilium, Calico, or something else? Um Or is it possible to extend their or you open for pull request or? That yeah, that's a very nice proposition proposal. Yeah, we are

open to pull request of course. So there is no plan on our side to do that but there is no objection that someone takes this the the the lead on this on the such a feature. Like I said before we trying to uh be more vendor vendor neutral and to be more CNI agnostic. Like we are CNI agnostic but from the network policies it's currently just a

VNK. Um so yeah, could work. Now Cilium has also its own solution for that but if you see reasons to do that yes sure. Yeah, it also requires usually integration from both sides. So just doing something in that observable not be enough as we showed before it like proper integration from the plugin too. So but if someone does that that could be done. Yeah, yeah that's that's

right because yeah it's a negotiation between the both And one more question in context of OVS. Uh do you support only OVS kernel support data pass or any others like OVS docker as well? I don't know if we've tested that with OVS docker yet but I mean we will actually probably try to move in that direction. And does it rely on a OVN only or a just

OVS would be enough? Well if you configure all the same things here, right? So it all goes like the the whole sample thing in the starting with the OVN database it has its primitives in the OVS then eventually so if someone else configures the same things it's going to work the same way. Okay, good. Yeah. Thank you. Hi, I'm probably answered but uh do you have maybe

in your own map or do you know if it's possible to easily connect and directly to OpenStack since it's using also OVN and Open vSwitch? We were actually working with the OpenStack team. They were interested in that like at the in Yeah. Red Hat? Yeah, but I mean I don't know exactly what they I know that the OpenStack team at Red Hat has done has used NetObserved

already for several things. I don't personally know where they are with this integration. Yeah, I mean there there was a plan that is possible it's I guess just a problem of who does it when but considering they're using OVN it's also like it's a bit easier because they just need to put the right samples into the OVN database. >> Blocks are >> Exactly. Instead of OVN Kubernetes

like the OpenStack part could do that absolutely and they were actually considering that but we don't know exactly like we can't talk on the priorities or specific plans or timelines but it's totally doable. Okay, thanks. All right, I think we are out of time so thanks again for coming. Thank you. And for your questions.