When DNS Blinks: Scaling and Hardening CoreDNS in Critical Cloud Infra... Yong Tang & John Belamaric

In this session, Yong Tang and John Belamaric provide an in-depth overview of the CoreDNS project, detailing its evolution and features over the past ten years. They highlight CoreDNS's flexible architecture, which is plugin-based and written in Go, making it highly adaptable for service discovery and cloud integration. The speakers discuss recent project updates, including security improvements and various plugins that enhance functionality, such as support for different DNS protocols. They also address DNS-related incidents that have occurred in the past year and outline strategies for building resilient DNS infrastructure, particularly in Kubernetes environments. Key configuration options like node-local DNS cache and multi-socket support are elaborated upon, showcasing how CoreDNS can efficiently handle requests and improve performance across large-scale systems.

Good morning. Thanks everyone for joining this session. My name is Yong Tang. This is Google from This is John Belamaric from Google. Today we are going to discuss about the CoreDNS project. Both [snorts] John and me maintain yourself CoreDNS project. We have been with CoreDNS since at the beginning. It's almost like 10 years. Yes, that's it's really amazing. So in today's session we are going to cover

several areas. First we are going to introduce CoreDNS as a little background. Then we going to talk about the project update. After that we are going to discuss the lessons we learned over the past year, especially some of the DNS related incidents across internet. And then the mitigations and how to make robust, reliable and the flexible in critical cloud infrastructures. To start with, CoreDNS as everyone knows

is a flexible DNS server written in Go. It was developed almost like more than 10 years ago actually. >> [snorts] >> It has a focus on service discovery. One of the biggest biggest advantage of CoreDNS is that it's a plugin based architecture which means it can be easily extended. If there is any new features or functionalities you you want to add and it's not in the existing

plugin, then you can write write a plugin yourself easily as long as you know how to write in Go. The The CoreDNS has been the default DNS server in Kubernetes for 2017. That's almost like 10 years as well. >> Because of the adoption of Kubernetes CoreDNS itself has a seen a wider adoption as well. Uh we have a lot of uh uh uh usage all over the

place. We have lots of uh like uh companies, organizations, institutions. They all use KubeDNS whenever they use Kubernetes. Uh in addition to uh its association with the Kubernetes, KubeDNS also support uh it's also a DNS server, so it does support additional uh protocols like uh DNS, DNS over TLS, uh [snorts] with gRPC, quick, and HTTP 3. Uh finally, KubeDNS has uh also support uh cloud integration. >>

Uh if you have a servers on uh DNS servers on AWS, Azure, or Google Cloud, you actually can use the KubeDNS as a central point of information to connect to different backend to uh to support different backend so that you have DNS with central place to hold all the information. Uh KubeDNS has been growing exclusively for the past 13 years. Uh as of right now, we have

uh more than 400 contributors. We have 14,000 stars. We also have uh growing list of public adopters. The number of public adopters here it is limited because uh the adopted here are just the the ones that's willing to share their uh public, but because [snorts] everyone use KubeDNS uh Kubernetes, so whenever they use KubeDNS most likely they're going to use KubeDNS, so the adopter list is much

bigger than 38. We also have [snorts] uh 29 maintainers. Uh the latest maintainer was added uh actually later uh last year. Uh he made a big contribution to security, so we added him as a new maintainer. That's uh really the best [clears throat] team uh really from uh from Europe I well. Okay. So, since last year Koding has to made a significant progress on several areas. We

made a new release just 2 weeks ago. That's 1.14.2. There are several plugins has been released over over the past year. Nomad plugin. That's a HashiCorp Nomad support. DNS over 8083. That's another [clears throat] one. And finally, we have proxy protocol which allows you to resolve client ID based on load balancer. In addition to that, we also have a features like a connection pooling, multiplex connection, and

API limiting on Koding On the security side, uh embarrassingly, the a few CVEs discovered over the past let's say 5 or 6 months. Uh four has been addressed. The additional four CVEs is in the process of being being fixed. And we expect this one to be released with 1.14.3 which uh I think it will be around a a week or so from now. Uh I'm going to

hand over to John to cover. Yeah. Um thank you. Yeah, so um well, you may have uh you may remember recently there was an incident where a lot of the internet went down. Um and uh that was a a DNS issue in Amazon's uh US East 1 region. Um so, we thought it was an opportune time to talk a little bit about some of the things you

can do to make sure that your DNS infrastructure is robust. I mean, if you were at the keynotes, you saw DNS uh we had we had a DNS failure there that caused some problems. Um so uh it's always uh it's always DNS, as they say. the uh you know, being such a central service where so many uh so many things are layered on top of it, you

know, it it it brings to mind the classic XKCD uh cartoon that I'm sure many of you have seen before of, you know, so much of the internet, so much of what we do uh relies on just a handful of um open source developers. And similarly, DNS being one of those projects uh that that's uh heavily uh heavily relied upon. CoreDNS uh of course wants to do

some things to address that. One is um compared to like traditional DNS servers, it's considerably smaller code base and considerably I mean, it's 10 years now, so it's not not that young, but you know, 10 versus 40 is a difference. Um so it it's uh built in Go, as Young mentioned, which means it's it's a a memory safe um language, uh which helps right there prevent a

lot of CVEs just simply by having that um that that runtime layer. Um but uh but there's a lot you can do in your configuration that will help you build resilience. so a few of the options that you might be interested in sort of taking a look at uh are things like prefetch. So, prefetch is a configuration um that allows you to kind of uh as it

sees records that are frequently used in your in your um in your DNS query stream, uh it will actually uh go out and as they approach their TTL, if they're an upstream record that we cached from somewhere else, it'll go out and and fetch it again and make sure that it's available locally. So then you're eliminating, you know, in the in the actual like hot path of

when somebody needs to request this, you you're eliminating a potential bunch of additional latency because you're keeping that cash fresh. as you for things like lame duck is another configuration option we have that allows you to as you're uh scaling down maybe because your traffic is coming down, we make sure that we serve all the requests and and drain those queues before we, you know, take down

the server. Um but I think probably the more interesting one from a resiliency point of view. So So there are different caching there are different ways you use DNS. One is in a caching architecture, one is in an authoritative server architecture. CoreDNS covers these cases, but the most primary most common use case for for is uh is kind of a caching layer or or a Kubernetes authoritative

server, but but even there it's really kind of a caching layer because the the the source of truth is is the API server. So in that in that sense CoreDNS is is um uh stateless. And so the serve stale option is an option you can configure within the the CoreDNS server that allows it to survive the loss of the upstream server for some period of time. So

if you're primarily forwarding things so in your cluster, say you're using in your CoreDNS in Kubernetes, um requests are going to some outside service, if the the upstream DNS that's providing [snorts] that um you know, fails for some reason, you'll continue to serve the existing records even beyond their uh as as stale records just so that you don't get a a total failure. So in the in

the in the loss of the sort of US East region DNS, you know, this could have helped you uh survive a little bit longer. It We're not going to serve them forever. Uh so, I think that outage was more than an hour, so maybe maybe you still would What's that? Yeah. But um but in any case, at least it can help you uh gives you an hour

to figure out some sort of backup plan to keep your services up and running. Um but one of those backup plans also could be using multiple providers. So, in that caching architecture where CoreDNS sits closest to your clients, um you can configure it to have different upstream DNS servers so that they can uh if one of them fails, it will pull it out of that forwarding pool

and and use the other uh the other DNS server, so you can back with different ones. This is kind of gets back to what Young mentioned briefly like a CoreDNS instance with different sources of truth um either to create a a single uh cache layer for like your services that are running in Google Cloud versus your services that are running in EKS. So, we can we can

read the APIs of those and load them in serve them from a single DNS server, but we also of course offer the the caching layer. So, it's kind of like you can avoid failures in any one of these individual services um by by uh populating the other services with some of those DNS entries. So, that's uh that's some of the the features we offer around that. some

of the other uh both scalability and performance issues. So, if you're running um, uh, typically, like I said, in the in the cluster, um, some of the more recent changes, and I'll talk a little bit more about multi-socket uh, in a minute, but this is a way we uh, can increase the the uh, vertical scalability of of CoreDNS. So, we'll go into that in detail in a

minute, but uh, one of the things I would want to emphasize in in a resilience and features that we we offer is what we call node-local DNS cache. So, this is a a Kubernetes feature. It's built on top of CoreDNS, and what what happens is uh, you can enable this feature, and every one of your nodes will get a sort of specially purpose-built uh, CoreDNS binary. Going

back to the plug-in architecture, you can actually take CoreDNS, and you can disassemble it, you can reassemble it in a new way with different plug-ins. So, this is what the Kubernetes community did when they built node-local DNS. So, CoreDNS itself is a set of default plug-ins, which include things like reading from authoritative zone files. Well, you don't need that when you're just using a local cache on

a on a node. So, they they took out all of those, took out the main routine, so it's not running in the ordinary main routine from CoreDNS, but instead it's its own uh, specialized special-purpose node-local cache for only uh, you know, in the Kubernetes environment. Um, and that you you deploy that daemon set, and it doesn't talk to your API server, it actually just serves as a

cache, but um, it will do two things, or three things, I think, I'd say. One, it allows you to configure stub domains, so what that means is that instead of uh, you when you when you make a request say uh, a service within your organization that's outside of Kubernetes, rather than going through the central CoreDNS and having to forward up to your DNS infrastructure, you'll go directly

from the node to your local corporate DNS server, which can provide a sort of shortcut and not put load on that central CoreDNS. two, it obviously provides local caching. So, the all the DNS requests that would normally go up to the central CoreDNS first go through this node-local service, which is likely to cache it. And then three, it actually changes some of the IP tables rules that

are normally installed basically UDP we part of the problem we have in There's two problems we have in Kubernetes that people see often in in DNS. Um and that's that the the end dots configured for DNS in Kubernetes can create a whole bunch of extra calls that are useless, that go all the way up to the the the central server. And those are all done over UDP

normally, and UDP, if there's one failure in UDP, it's a 5-second timeout. And so, node-local DNS will upgrade the the connection between the node and the central DNS to TCP. And that It's a It's a long story. It's kind of a long walk, but we had a lot of failures early days of Kubernetes where the connection tracking table in the Linux kernel would fill up because UDP

entries aren't removed from connection tracking table until they time out after 5 seconds. Whereas TCP, as soon as the connection's torn down, right, that they they're able to take them out cuz it's a connection protocol. So, um that's another thing that node-local DNS does. So, for instance, at Google, um some of our largest customers were seeing we had a an incident recently where one of our larger

customers, you know, they've got thousands and thousands of nodes in these clusters. They're they're running and you know, occasionally one node will just start failing DNS and nobody really knows why. And well, it turned out it's like you you look at our SLAs for the underlying network and they're like, I don't know, five or six nines or a lot of nines, but when you've got thousands and

thousands of requests, you know, thousands and thousands of nodes doing thousands of requests, you've got millions and millions of things, you're going to see DNS failures. You're going to see failures from that underlying fabric. And so that's what was happening. Occasionally there'd be something going on somewhere in the networking and all of a sudden, you know, you you'd be seeing a bunch of failures. So, took a

long time for the team to debug this and figure out that actually there's nothing wrong. It's just ordinary failures. So, we turned on node-local DNS for this customer and all the failures went away. All right, so this is a important thing in your Kubernetes clusters that um that that I I highly recommend. Jumping a little bit more into that scale question, one of the the other areas

and maybe I'll talk about this, the proportional auto scaling. So, by default, the auto scaling configuration that's recommended with most Kubernetes installations is uh using a number of CoreDNS CoreDNS instances that's proportional to the number of nodes in the cluster and that's another thing I sort of recommend is probably not the way you want to do Um and especially as we go into the um multi-socket discussion,

we'll come back to that in a second. Um Anything else you want to cover on here? >> No, I think I'm finished. >> Okay. so what is the multi-socket? This is this was in 1.12, so it's it's not brand new, but it's uh a pretty interesting um change that we made that had a pretty profound impact on how CoreDNS scales. we had a problem for a long

time where uh at some point you you give it as much TPUs you can, you keep giving it more CPU, and it it just won't scale. It it flatlines out. And so, it took us a while to figure out that um examining the internals, this is kind of what a CoreDNS process looks inside. You've got a socket. We pick things There's a server an internal structure called

a server that that uh starts go routines, and they serve that socket. So, they they're just pulling pulling packets off of that socket. Um and then they dispatch them to the plug-in pipeline, and depending on what plug-ins you have running, right, they may take more or less time uh to execute. But, um we would see where we would reach a peak load, and we would run out

of the ability to pull things off of that socket. What we thought it was was that we couldn't pull things off that socket fast enough. But, through some experimentation, somebody tried out one of our more recent contributors was like, "You know, I think maybe we're just the issue isn't so much that CoreDNS isn't able to scale, it's that the way that the kernel's delivering packets to CoreDNS

isn't scaling. And if we split those packets across multiple sockets, maybe uh we'll see better scaling." And um lo and behold, that worked. Um and so, then uh what that led to was our next step, which was to um use the the SO_REUSEPORT option uh on the socket, which lets you have many sockets open on the same port, and the kernel will just distribute the incoming packets

to that port across those sockets. And then uh if we match that internally inside of um Corden S with with go routines, uh or rather with um we we have enough sockets, then uh we can launch multiple servers, launch multiple go routines against each socket, and we can we can kind of more uh scale more linearly. So, what were the results of this? Well, here's an example

without multi-socket. Um this just does cache, so we take all upstream out of it. It's so we're taking packets in, we're looking up in a cache in memory, and we're returning it, right? So, it's very very narrow functionality set of functionality. And you can see that um we peaked with two CPUs at like 40,000 QPS, um and adding more CPUs did nothing. if you had two sockets,

we could see that we went from being able to serve, uh you know, 40,000 QPS with uh two CPUs uh or with four or six CPUs, and um you know, to doubling that, more than doubling that. And we keep adding sockets, and we see that each time it's basically almost linear growth of performance with the number of sockets that we're running. So, uh with the with the

number of CPUs based on the number of sockets. So, this was clearly the the central bottleneck to vertically scaling Corden S. what that means, I mean, look at this. This is this is three 300 you know, over 300 uh thousand QPS uh from a a single Corden S instance on a single machine. So, um most of your clusters probably don't have that level of uh QPS going

on. So, what does that mean for you in your in your in your clusters. it means one, you still don't want to just run all just one one instance because that's a terrible failure domain, but it means you probably don't need to run I've seen clusters running 80, 90 nodes with you know, with with CoreDNS on them. That's ridiculous, right? We should not need to do that,

but with previous versions of CoreDNS that you had to scale horizontally, um you couldn't just run a couple of big nodes with you know, dedicated to CoreDNS, but now now you can do that. So, the cluster the cluster proportional auto scaling that we did sort of traditionally with CoreDNS in uh in Kubernetes can give way to um basically just a CPU based auto scaling we we as

we hit, you know, the CPU limits on the machine or whatever it may be or whatever we've allocated that pod, we can horizontally scale, but for the most part, we can just have a couple instances of of CoreDNS. So, I I you know, I recommend experimenting. Don't just jump in and do it, but I think this might serve you better from a both a resource utilization point

of view and you know, not having to horizontally scale up and down means um you know, you you may have fewer latency issues with your with your DNS. So, you know, that's a few stories about CoreDNS and how how to make it more resilient, how to make it more performant. Um and I think from there uh you know, Okay, I can I can cover a little. Yes,

uh CoreDNS has been, you know, has been here for almost like a 10 years. Uh it's a mature product. Um it's a pretty big community. Uh but on the other hand, we also want to have contributors, but one thing I want to mention is that it's not like we have a lot of contributors going to contribute like every day, but we have [snorts] lots of contributors that

contribute for like one or two features, and that's one of the interesting aspect of CodeIgniter's community. Many contributors, they contribute big feature, and they say, "Okay, that's the only thing they care." They jump in, they become maintainer, and then, [snorts] you know, you have other ones coming in. So, that's one reason we have like 30 uh [snorts] maintainers. That's the reason for that. Yeah, they'll tend to

they'll tend to like if you're interested in a particular plugin, right? People often contribute a plugin, they become a maintainer on that plugin, and then but they don't necessarily have to maintain the rest of the the project. >> Yeah, but I'm going to give a couple of example. Actually, several plugins are actually contributed by even Google Summer of Code. Intense, okay, just an example. Yeah, and the

some some of guys are going to be very interested when when they pick up the Google Summer of Code, they actually contribute to one big plugin, and then they say, "Okay, yeah, if you already make a significant contri- uh contributions to CodeIgniter, maybe you can be a maintainer." They say, "Sure." And then they start maintaining one specific plugin. That's the model we adopted in in CodeIgniter because,

you know, like uh it's really a open source project, and uh and then, frankly, it's not like a backed by big companies. It's more like a purely from community. Yeah, it's purely we So, Young and I worked together at a company called Infoblox many, many years ago, 10 years ago. And uh and and Infoblox for a while did a lot to back it and contributed a ton

to it, but um I I the I haven't seen the the Infoblox folks that much in the in the community lately, so um hopefully I guess I'll this is being recorded that some people might get mad at me, but but I haven't I haven't seen too many of them. Um so, you know, it has become very much just like an open source it's the XKCD thing like

we're a bunch of unpaid people, this is not our day job to support this, but um we do it because we believe it's important, so we'd love to have additional uh along those lines. Yeah, I like like I said, it'll need to be a let's say 24/7 like everyday to work on that. And then you'll you may only just be interested in one specific problem, specific feature,

and then you make contribution. You could you can be a maintainer and you only maintain a small feature, but that's a core essential piece of coding as in the pretty much a Kubernetes ecosystem. Yeah. So so we have 5 minutes left. So there's Q&A. There's a mic here or we have another one here if anybody has any questions. Yeah, okay. I think it's on. Thank you. Is

this working? Yeah. Yeah. we've recently had some issues in um like classic issues with the multi-tenancy environments where you have one user in um in in a name space in a Kubernetes cluster suddenly starting to send a huge amount of requests to CoreDNS making it fail but for all users in the cluster. So they're not even able to resolve um resolve domains outside the cluster. They're they're

not able to resolve anything or even start the pods sometimes. So that's an issue we had and um we wanted to see if you have like a general recommendations on how to how to deal with this sort of problems. I mean I I think the first thing I would do is the node-local DNS because then your failure domain goes from cluster to node. Like I mean you

>> [clears throat] >> I mean if you've got like depends on what they're doing exactly, but Yeah, except that the um the nodes are not the are not >> as well. users, yes. Yeah. Yeah, it's reduces if it doesn't eliminate the problem. >> Yeah. There are strategies you could take if you've got a misbehaving tenant, you could possibly uh add a webhook or some sort of a

mutating admission policy that takes all of their pods for those namespaces and like maps them to their own instances of CoreDNS that they can crash all they want. Right? So, there's a DNS policy on the pod and so you could redirect them to a different DNS service. That would probably be your It's a multi-tenant service. So, there's nothing you can really do to protect that completely other

than, you know, at that sort of client there. >> The idea would be basically to create sort of dedicated CoreDNS for for each user. >> For that user. Yeah, I think I want to add one more thing. You know, you think about DNS, DNS is massively rescalable. Even though people say that's a simple. So, uh why is massively scalable? Because it goes to internet. Just think how

internet works, how internet DNS works. You're going to realize you actually can apply additional caching layer at every possible scenarios. Possible and that's how internet works cuz if you have a offending, let's say, tenant, then you can certainly have a dedicated DNS server or you can split out, right? Or you can do different ways so that you can uh you can on the one hand, you'll try

to leverage all the CPUs possible, but if your DNS is actually spread out to different workloads, like it's not just a DNS dedicated server, it's actually pretty [snorts] lightweight. So, I think a scale out is one of the best way to handle that, just like a how to scale out internet on DNS, right? Okay. Thank you. What one thing you reminded me of the one thing like

with that, I talked about one giant node with with many CPUs and and and right, you you could potentially run into that issue there. It scales really well, but the you do you can run into an issue on some cloud providers where the number of CPUs [snorts] and then the amount of packets per second they allow you to put through are are mismatched. So, you have to

watch. It's not just CPU. You could hit packet per second limits on the on the machines before you hit the CPU limits. Which I don't know why what you said reminded me, but it kind of made me think of that. Thank you. Yeah. any other questions? We're out of time. So, thank you very much. Enjoy the rest of your day. >> [applause]