Rust Vs. Go: Building a Container Network Stack From Scratch - Matt Heon & Shivang K Raghuvanshi

In this talk, Matt Heon, a principal engineer at Red Hat and core maintainer of Podman, discusses the evolution of Podman, a rootless, demonless single-node container engine. He introduces Netavark, a new networking tool developed in Rust, addressing limitations faced with the Container Network Interface (CNI) and enhancing performance for container networking. Matt explains the rationale behind choosing Rust over Go, emphasizing its static memory management, advanced type system, and error handling capabilities. He shares his thoughts on the challenges encountered during this transition, the successful integration of a new conntrack library, and how the Linux Foundation's mentorship program facilitated the development process. The session also highlights the potential for ongoing contributions from newcomers in the open-source community.

All right. So, starting off, I am Matt Heon. I am a principal engineer at Red Hat, and I am a core maintainer of Podman. So, let's start off with what Podman is. Anyone heard of Podman? Show of hands, please. Oh, excellent. Love to hear it. So, we'll rush through this one. Podman is a single node container engine, meaning we have the roughly the same problem space as

Docker. We are deliberately not a Kubernetes runtime. If you want to do that, you want container D or Cryo. Podman is natively rootless, meaning that you can easily run it without root privileges. It's also natively demonless. This is very interesting for embedded use cases. Means that Podman itself goes away when you're in an idle state. The only thing running is your workload. Absolutely no resources other than

that. We provide a Docker compatible command line interface and a Docker compatible API, so you can easily switch over. We also have a Podman specific API, which exposes some of the things you won't see in Docker, things like quadlets and easy ability to manage containers using system D, and things like pods, which we have a similar concept of to Kubernetes. Project originally started back in 2017. We've

been CNCF sandbox since 2024, hoping for incubation at some point in the near future. All right. So, now let's talk about how Podman networking works. Initially, we use CNI. Show of hands, who knows what CNI is? All right. So, CNI is the standard network interface used by Kubernetes. It's a series of plugins with going from very basic stuff, just establishing bridge networking, to highly advanced multi-machine, what

do you call it, switch networks, and yeah. So, it has a lot of capability, and it was also very well established. So, back in 2017, when we were looking to rapidly iterate, just get something out the door, it made a lot of sense for us to be using CNI. Uh in addition, we thought that there would be some big advantages having commonality with Kubernetes in that you

could use Podman as sort of a sidecar debugging thing. You could run a Podman in your cluster. You could connect to the network, see what was going on. That seemed like it would have certain advantages. Unfortunately, no one actually did that. So, kind of a waste. Also, it turns out that CNI wasn't really doing all that we needed. So, as time goes on, Podman has been a

product for a couple of years now, we're feeling the need to write our own plugins because while we are actually only using about 5% of what CNI offers, we don't, as a single node container engine, I have no need for multi multi-node networking. I don't need any of the fancier plugins. And because of that, we are not using the vast majority of CNI's capabilities, and it's actually

only providing about 50% of what we need. A lot of our DNS solutions have to be written internally. We are writing our own plugins. Finally, there is this big UI mismatch. We have a Docker compatible CLI, but we have Kubernetes compatible networking. So, if someone comes in with a Cilium network, what what do I do with that if I want to show that to the user? I

have to figure out how to squeeze that into a command line that was never really meant to display it. It gets awkward. We get a lot of bugs about this. It's not really a good solution. So, in 2021, we get a backchannel from one of the CNI maintainers who says they are considering dropping support for some of the plugins we use, specifically because we are creating extra

workload for them. Now, this is entirely fair. Can't blame them at all. CNI is meant for Kubernetes, and if we are creating workload in awkward places that no one else in the Kubernetes ecosystem is using, it is entirely fair of them to drop us. However, it does put us in a slightly awkward position. Now, they were just considering it at this point. They were trying to sound

us out to see how how bad this would be for us, and that's led us to a bit of a decision. We can either try and fork things, because we have to do something. We have to do something. We cannot just sit here and let our networking stack go away. Forking, however, is suboptimal. We're going to have to continuously rebase our stuff on top of their patches,

and also, as I said earlier, CNI is not really an ideal solution for what we're doing. And because we are only using about 5% of what CNI does, a rewrite actually seems to be feasible. We're at a position where the amount of code we would have to maintain in addition to what we're already doing did not seem that high. It was still a pretty steep step. We

would be roughly doubling our commitment in the network department, but it seemed like it would be reasonable, and it would give a better user experience. So, what we ended up building is something called Netavark. It is a complete container network setup and teardown tool operating on a single node. It handles bridging, MacVLAN, and IPVLAN drivers, and also everything you'd expect to get container networking working in addition

to that. We've got some ioctls in there. We have routing. We have firewall configuration, things of that nature. And relevant to the topic of the talk, it is monolithic and written in Rust. So, Podman is actually monolithic Go. So, this is a big departure for the team. It was released in 2022 alongside Podman 4.0. It was about 6 months after we realized we were going to have

to do something. So, we turned around relatively quickly. And it has a sister project called aardvark-dns, which is probably going to merge into the main project at some point in the near future. But anyways, aardvark handles container DNS. We have some slightly unique needs in there. Uh it's basically a conditional DNS server that responds based on what type of container you are and what networks you're in.

There are other servers that do this, but ours is a lot more lightweight and configurable than those. So, core question. Why did we not go with Go? At this point, we had 4 years of experience in the project writing Go. The entire Podman codebase is basically Go. We have a tiny bit of C for C Go, but yeah. We actually almost wrote Podman in Rust. Back in

'17, a much younger me was looking at it and thinking, you know, it would be nice and neat to use this fancy new language that offers all these additional capabilities. But uh we did not, largely because of ecosystem compatibility. There are so many good libraries in the Go ecosystem that do so many good things for containers. We already had a preexisting library for storing containers, a preexisting

library for speaking to the registry. Rewriting all those in Rust was just too high of a bar, and Rust-Go interop at that point in time seemed pretty poor. So, we ended up doing Podman in Go, and it's okay. It is the most thoroughly okay language that I have ever worked in. I'll just put it that way. Uh I'm sure everyone here has heard of or read several

articles about how Go is not a good programming language. A lot of that starts to resonate when you've worked in it as long as I have. Our maintainers are just They're at a point where they want to try something different. And my management chain, to their credit, very supportive of this. As long as we thought we could deliver on time, nobody was going to raise too many

eyebrows about this. So, why why Rust specifically? As I said before, we already had it in mind. We were thinking about it when we went to write Podman, but there were a couple very specific reasons why we really wanted it. We did not want a garbage collected language. This largely comes down to aardvark-dns. We have a persistent DNS server now. This is the first time that Podman

has done anything in the persistent daemon category, meaning that now, for our embedded use case, we want to seriously worry about how how much resources are is this going to use? Like, how is this going to perform? And that really leads us to say, let's not risk it with garbage collection. Let's just go with a static memory language. In addition to that, we really liked the Rust

type system. Back in '21, when we were deciding this, Go still did not have generics, and at that point, frankly, I did not believe it was ever going to happen. I had read all the blogs, but I was genuinely amazed when it actually did come out. We liked Rust's error handling. I believe this also predated the package or error in what do you call it, Go. So,

we were still doing a lot of error wrapping using custom libraries. It really did not feel good. Meanwhile, in Rust, we have this extremely expressive type system that came together with the error handling in wonderful ways. I think the moment it all clicked for me was when we wrote Netavark's error handling, and we had type conversion between error types, so everything could return this wonderful result type,

and Netavark error for all our stuff. And when we returned this error, all the other errors would just naturally coerced into it, no effort required. Question mark operator makes returning errors instant. Absolutely wonderful. Chef's kiss. We loved it. And the thing is, Rust does have a lot of complexity compared to Go. Go's biggest virtue in my eyes would be the simplicity. It is a wonderful language to

do code review in. It's a very easy language to spin people up on. If I have a new engineer join, I have great confidence that they could be submitting meaningful pull requests in basically no time at all, week or less. Rust is not necessarily the same. So, we knew we were making a trade-off here. At the time, it seemed to be perfectly worth it, and frankly, we

were also tired of Go that we just did it. So, how did it go? Netavark itself has been quite successful. Most of our developers like Rust. The feeling is not unanimous, but the thing that's different between now and 2020 when we started this is LLMs. Uh I have people who tell me that they don't really like working in Rust, but if they have a coding assistant, they

don't actually mind. So, it feels like the distinction is starting to narrow in that sense. The language choice mattered a lot more in the pre-ChatGPT days than it does in the post days. Uh code review, I will say, is considerably more difficult in Rust. Go, it is immediately obvious what code smells are because the error handling pattern is very easy to see from a glance. Rust, not

necessarily the case. It's a lot harder to nail down exactly what a specific block of code is doing if it uses any of the more advanced features. So, it's more tiring and more time-consuming. Again, possibly something here that LLMs can help with, but we're still experimenting in that area. And one thing that might be unique to our specific use case, we have a limited number of available

developers. So, Netavark artifact turned out to be a relatively small problem space. We haven't had that many big feature requests after the initial code was written. And if I'm going to spin a developer up on Rust, if I want them to start working in this completely new language, I want to give them something big enough to really sink their teeth into. We haven't had many of opportunities

to do that, so it's actually been hard to get the team more knowledgeable with Rust. We're starting to do additional projects in it. We've got a new version of conmon, if anyone's heard of that, that's going to be coming out in Rust soon. So, we're starting to move more in that direction, but for now, it's been a little hard. I'll also note that that is not the

official Netavark logo. That is a bit of Google Gemini. So, yeah. All right. Now, as I said, I have a co-presenter who wasn't able to make it. Fortunately, he was able to record something, so we are going to Hi, everyone. My name is Shivang Keragwanshi. I am currently a computer science undergrad, and I was really looking forward to being there with you all in Europe today alongside

Matt. Unfortunately, due to the current geopolitical situation, I wasn't able to travel, but I am incredibly grateful to the KubeCon team and Matt for allowing me to record this segment so I can still share this journey with you. Today, I want to talk about my experience as a Linux Foundation LFX mentee. I had the privilege of working with the Podman container tools organization to help resolve a

rather tricky networking issue. I'm going to walk you through what the bug was, the missing Rust tooling we had to build to fix it, and how guided mentorship makes solving these kinds of problems possible. Now, the issue we were facing involved a mysterious drop in UDP traffic. Normally, when uh an external server sends UDP packet to a container, everything just works fine as intended. But, if those

packets happen to arrive before the container has completely finished starting up and listing on that port and IP, the Linux kernel's connection tracking system, aka conntrack, sees that no one is listening. And the conntrack interprets this as a failed connection attempt. It drops the packet, and unfortunately caches a stale or invalid record of that failure. The result is that the sender gets completely locked out of that

specific IP and port until that stale entry eventually times out. But, like uh you could see how this plays out here. Uh on the right, the external server sends the traffic. In the middle, conntrack intercepts it and sees that there's no active listener and caches the failed failure. Now, sometime later, on the left, our Podman container finally finally spins up and starts listening on that exact IP

and port, but now it's too late. The next time the external server tries to send data, conntrack checks its table and uh sees the cached stale entry, and it just drops the packets. It was a frustrating experience for users because the application was running just perfectly fine, but the network simply refused to let the traffic through. Uh conceptually, the fix for this was quite simple. We just

needed to tell Netavark to flush out those stale conntrack entries the exact moment a container starts listening for traffic. But, when we actually tried to implement uh we hit a major roadblock. Netavark operates in user space and is completely written in Rust. The conntrack subsystem, whereas, lives deep inside the Linux kernel. To communicate between the two, you have to use the netlink protocol. Um when we started

looking into how to send a flush command from uh Rust to the kernel, we realized something surprising. There was no netlink conntrack library in the Rust ecosystem. The existing Rust netlink libraries did not yet support conntrack. So, before we could fix the bug in Podman, we had to build out the missing conntrack netlink library for the Rust community. And uh this was a huge learning curve for

me. To build this, I had to spend a lot of time reading through C kernel code and looking at existing Go And studying Wireshark hex dumps just to understand how these messages are structured. >> [clears throat] >> Every message requires a standard netlink header followed by a netfilter specific header. And uh finally, payload containing nested attributes that actually tell the kernel what to do. Using existing crates

like netlink-packet-core, I was eventually able to piece together the Rust struct and the serialization and deserialization code for the uh conntrack get, delete, and new commands. I submitted these upstream, and thanks to the help and reviews from the maintainers of those crates, we got conntrack support merged into the Rust Now, with the library finally in place, I could fix the bug. I hooked the new library into

Rust Netavark's port forwarding setup and teardown logic. And to make sure we had only deleted the bad entries without dropping active healthy traffic, uh we set up two rules. We matched the container's IP against the reply tuple and the host port against the original tuple. Now, Netavark could send a net a netlink conntrack delete message to the Rust right as the container came online. These uh stale

entries, as you guessed, were flushed out immediately, and the the UDP traffic flowed perfectly. And our integration tests finally turned green. Now, obviously, this was all made possible by the Linux Foundation's LFX mentorship program, uh which pairs newcomers directly with the core maintainers on real open source projects just like this one. My mentors, Matt, Ion, and Paul Holzinger, did not just throw me into the deep end.

We took it step by step. And uh I started by just learning the domain and doing some smaller warm-up pull requests just to get comfortable with the code base. And uh one of the reasons this project succeeded was the weekly syncs. Uh like, having that constant feedback loop and space to ask questions allowed me to break down a terrifying problem into manageable steps. Now, programs like uh

LFX prove that with a welcoming community and patient mentors, newcomers can also contribute to critical infrastructure. I am deeply grateful to the Podman team for trusting me with this, especially to Matt and Paul for their time and guidance. Thank you all so much for listening. I will now hand the presentation over to Matt, who will talk more about the broader ecosystem impact. Thank you. >> [applause] >>

All right. So, I just want to say that Shivang did an incredible job, especially because this is on my team, we got far enough that we root caused it, and we identified that it was a netlink problem, and we identified that it was going to need a complete new library written because we didn't want to just do the tiny bit that we actually needed, just in case

we need more in the future. And at that point, it just sat on the shelf because it was just enough work to not make sense for the amount of annoyance it was. Having the ability to get an LFX mentee in to get this implemented was pretty great. I think it was a great thing for him and for us. Yeah, fixed a very frustrating bug for enough people

that it mattered. Not enough that we could justify putting all our time into it, but fortunately, enough that we could get Shivang on it. And upstream contributions here, the netlink library that uh Shivang wrote is now upstream, so anyone who wants to do netlink in Rust, that is now available. Uh community, we took Shivang and we made him into a Podman contributor. He was uh pretty good

when we got to him, actually, but I think he's a lot more knowledgeable about the project now and about open source work in general and he certainly seems to have found it valuable. He was the one who wrote the capable and confident contributor bit. I don't disagree, but credit to him. And he wrote a blog about it. If you grab the slides from the schedule, you can

actually find a link to that. Got a bit more in depth info from him because he wasn't actually able to be here. Okay. Thank you everyone and I believe we can take questions now. >> Hi. Thank you very much for the talk. I have a question about the Go versus Rust decision. Looking back now it's like 4 years later, Go has generics, Go has maybe better error

handling depending if you like it or not. looking back, would you make a different decision with the knowledge you have right now? And how do you feel about the availability of Rust developers especially in the agentic world where most people don't write code by hand any longer? Sure. So, honestly, when I think about these go back 5 years and do a different thing, I mostly think about

writing Podman in Rust, not the other way around. Uh we're broadly speaking, we're quite satisfied with it. We haven't availability of developers is not perfect, but we are not a solely Rust shop. So, I can get people contributing to the team on Go first and then experimentally have them work on Rust see if they like it. Again, with LLMs, it's not as big a deal to say,

"Hey, I'd like you to spin up and be productive in this new language." I would still like them to achieve some great level of conference so they can contribute to code review. I would say code review continues to be our hardest problem, but yeah. Uh from my perspective, while Go is definitely a lot better than it was, the advantages that we've seen Rust offer in just ease

of coding and the abil- developer satisfaction mainly, I would say, is the big advantage. Yeah. Hi. Thanks for the talk. And as a yeah, primarily Go software engineer who always looks at the greener grass on the Rust side, but never dared to delve over. What are What were the biggest friction points taking like a Go dev team on a Rust journey and what are yeah, things that

might be better in Go, things that like lessons learned at love to hear some insights there. I would say our biggest issue was probably as we got accustomed to Rust, but we're not experts in it. We wrote a lot of bad Rust. The net of our code base is not the best. We're going back and we're progressively refactoring, improving, but what we originally wrote is really sub

optimal and I would say that's less of a concern if you're coming into an active project, if you've got mentors who really know the language. We took a jump into the deep end without really having any of that and I think the results they work, but the code base is not in the best place as a result. Uh specific things to keep in mind, huh. Compiler warnings

are honestly incredible. I would just say uh have someone with you to review your code who really knows the language and is willing to teach you things because the compiler is quite happy to tell you about advanced features. Maybe not necessarily happy to tell you where to use them. Hi. Thank you. Great talk. Uh I still have some containers using the older networking driver, the slirp4netns or

however you pronounce it. Yes. >> Is netavark supposed to completely replace that by now or how does it relate or So, slirp4netns is going away, but it's actually not being replaced by netavark. So, we have a different networking stack for rootless containers which uses netavark above it. So, we have slirp4netns which forwards from the host and then netavark is inside the container network namespace allowing multiple containers

within the same pod or the same network to talk to each other. So, slirp slits sits beneath uh netavark and is being replaced by something called pasta largely because we the pasta maintainer is incredibly responsive to issues and he's delivered something that is very performant and has a lot of features that we really including some that are going to in the future allow us to hopefully make

things a lot better at port forwarding for rootless users. it is going away. We will migrate you without any problem as of Podman 6 which is coming out second week of May. So, look forward to that. Okay. Thank you. Um I'm late and I think I missed something, but I have a two small question. You mentioned that you didn't need a garbage collector in uh why and

now how easy a way did you choose in Rust instead? Uh the issue So, we were okay with garbage collection in Go for Podman because as I said, Podman is a very short-lived process during container run it. Unless you're running an attached session, Podman run goes away as soon as the container is started. So, our memory management issues are ended quickly by the fact that Podman itself

just goes Wha- Uh what was the second half of the question? Sorry. Uh sorry. Uh how easy a way did you choose in Rust Yeah. Uh so, Rust specifically offered us very low memory management for persistent process or low memory usage for persistent processes which is not instant. You have to be careful with it, but it was easier to deliver the level of memory usage we wanted

with Rust and with static allocation than it was with Go and garbage collection. Hi. I have a question. Hi. I have a question about Rust package management. So, going from Go where anything is get repo you can import it to Rust where you have a centralized repository, was there any friction? How How did your team find this transition? I would say that we had an easier time

in Rust, but that is largely because of mistakes we made in Go. So, the Podman project, we have Podman and then we have a series of libraries under it and whenever we do a release, we have a five-tiered vendoring chain where we have to do a release of project A, vendor that into B, release B, etc. etc. In Rust, we have not had as many issues because

we just deliberately decided that we were going to keep things very simple and very flat. I would say that while the points you're mentioning are friction points, we just we kind of knew that we didn't want to do the same degree of complexity we did in Go. We deliberately avoided it and because of that, it hasn't really been an issue.