Evolution or Revolution: Istio as the Network Platform for Cloud Nat... Mitch Connors & Daniel Grimm

In this session, the speakers, Mitch Conners from Microsoft and Daniel from Red Hat, discuss the evolution of the Istio project, focusing on whether recent updates signify progress or radical changes. They review the history of Istio since its launch in 2017, highlighting key developments like the introduction of the Istiod control plane and the move to a monolithic architecture, which simplified user upgrades. The talk also covers the addition of new proxies, including Ztunnel and Waypoint, to improve performance and user experience, especially in managing large applications and multi-cluster environments. They touch on the ongoing contributions from the community and advancements like Gateway API inference and enhanced ingress configurations that aid newcomers to the service mesh landscape. The discussion emphasizes stability in the project while addressing the demands posed by contemporary needs such as AI integration.

Uh, thank you for coming to the Istio maintainer track talk. Uh, we're going to be discussing whether the change in Istio changes that are happening in the Istio project are evolution or revolution. Is this just a continuing iteration or is it something entirely new? Before we get started, you never know how to write one of these maintainer track talks. So, I want to see if we can

divide the audience into two camps. Uh, one camp is, I'm not really sure what Istio is. I've heard about service mesh a little bit and I want to learn more. And the other camp is, I'm I'm pretty familiar with Istio. I just want to hear what the latest and greatest is. So, who wants to learn what Istio is? How many of those have we got? Oh my

gosh. >> Oh, wow. That's a lot of you. >> I kind of thought it might be like that. And uh, how about the latter camp? Who just wants to hear what the latest and greatest in the project is? Okay, so like 2/3 on the first and 1/3 on the second. >> I'd say almost 50/50, honestly. But, yeah. Yeah, okay. Well, we'll make sure to try to dial

the content on that. Uh, my name is Mitch Conners. Uh, I'm a principal software engineer at Microsoft. Uh, I've also just joined the product managing management team for AKS networking. I'm a maintainer of the Istio project since 2018 or 19. Uh, and I'll have Daniel introduce himself. >> Yeah, my name's Daniel. I'm a software engineer at Red Hat and I've been working on Istio since I joined

Red Hat, actually, in 2019. Yeah, so. It's coming up to be 7 years. Yeah. It's always cool to give a talk with somebody who came up together in the project at about the same time. So, uh, our agenda today. We're going to take some time looking back on where Istio has been, what the original design was for the project back in 2017. Uh, we're going to look

at how health is doing. Are we still seeing active contributions and maintenance? And then, uh, we're going to talk through some of what's new in Istio. And given the feedback we received along the way, we're going to make an effort to make sure that we're kind of giving this at as much of a 101 level as we can. So, let's launch into this. Uh, Istio 0.1 launched

in 2017. Uh, and we still maintain the website today. I was kind of impressed that we do that. You can go back to archive.istio.io and see all the old stuff. >> Maintain is a stretch, but it's there. Oh, yeah. I mean, it's it's horrible looking, but it it exists. Uh and this was sort of the architectural diagram on how we explained what Istio was. And if you're

an Istio user today, this might look a little unfamiliar. Uh you've got resources like mixture mixer and Istio auth that don't really exist in Istio architectures today. Pilots, as you do see that word still floating around, but we have a new name for it. Anybody know? Istiod control plane, yep, both correct. Uh and so those those are sort of equivalent names, but uh we'll talk a little

bit more about that evolution. All the way back in 2017, one of the uh design docs where we talked about project direction, 0.1 had just shipped, and we said, "Okay, we need to talk about where we're going with this thing. What are our objectives?" Uh this was a document written by in collaboration between Google and IBM at the time. Uh we wanted to decouple routing configuration from

proxy implementation. so that's why a lot of people have asked, "Why doesn't Istio just expose the Envoy API directly? Envoy gateway and other projects do that." Well, that's because the goal was always to abstract which particular proxy you were using under the hood. Our goal was not just to allow you to be a control plane for Envoy, although as we'll discuss in a minute, that was actually

what we ended up doing for the first 4 years of the project. Uh the goal has always been that we would expand beyond that. You can even see explicitly we wanted to support different proxy implementations. Uh we called out Nginx as the other proxy implementation. I don't think that's probably going to happen anytime in the future, but uh we have actually expanded to other proxy implementations and

we'll be talking about that as we go. So, let's talk timelines. After the 0.1 launch, uh took us another year to get to 1.0. Uh we expanded from three to six microservices as part of your control plane in 1.0, we expanded from our own private REST API to use CRDs, but there were 37 of them in 1.0, which was pretty darn heavy and overwhelming for a lot

of our users. Which one do I use for what thing? Um but we added a multi-cluster support pretty quickly after that. Uh over the next 2 years, we heard that running a microservices control plane for microservices was challenging. You really needed a service mesh to manage your service mesh control plane, and since it couldn't be turtles all the way down, we went ahead and consolidated all of

those components into what we now call Istiod. That is our monolithic control plane. Uh and we found that that made it much easier for users to upgrade. Uh in particular, they could run different versions of Istiod in parallel within the same cluster, within the same mesh, so that's uh you could progressively roll your workloads from one version to the other with an easy rollback, uh not have

to a blast radius of an entire cluster every time you upgrade things. So, that was our canary based upgrades which followed in 1.8, which was also in 2020. 2022, we finally got to the point where we were ready to announce a second proxy for the Istio project. We'd spent 5 years running just Envoy uh as a sidecar. Basically, every pod gets its own Envoy. It was a

brute force way of capturing traffic and directing things. Uh it was never optimal, but in the early days of a project, you're not worried about optimization. You're worried about proving what you can do. So, sidecars proved that a service mesh was valuable, uh but when we started talking about the need to be run in an optimal manner to re- have good resource utilization, uh as well as

ease those upgrades that we talked about, sidecars weren't working for us. You had to restart every pod to have it upgrade its sidecar, and it wasn't really visible in the Kubernetes API at a high level which ones had the new sidecar, which ones had the old So, we introduced the Ztunnel, a layer 4 proxy that runs per node uh outside of the cluster, and handles just MTLS

and security cuz that's the only thing that every pod needs uh is MTLS and layer 4 security. And then we said if you do need those layer 7 features, you're not out of out of luck, but we're not going to do it in a sidecar. We're going to go ahead and allow you to spin up what we call a waypoint gateway which runs in its own deployment,

can be upgraded independently just like the Z tunnel can. Uh and that'll give you the full L7 functionality that you're used to. Uh the real magic there is that both onboarding and upgrade can now be done without any restarts of any pods. Uh so that was a very exciting announcement. We finally got to our second proxy in 2022. Let's take a brief look. We promised we'd look

at project health. Uh you want to take that over? >> the other angle is people, right? So it's not just features. We've just talked about a bunch of features that were developed over the years. But uh one thing that I really proud of and really like about the project is just the number of contributors we've we've seen. So you can see the massive spike there with the

1.0 release and then for like 2 years we sustained a really high number of monthly unique contributors. It's like above 150. And the new contributor is also interesting. That was it was about 50 per month people coming in and having their first contribution to the project. And it's sort of dwindled down. Like if you if you've been to KubeCon 2019, that was my first KubeCon actually, everybody

was talking about service mesh. It's hard to imagine now. It's not AI related, right? But there were other hypes before and service mesh was kind of in the middle of that back then for 2 years. And you see that it's like the numbers are going down, but we still have like at least 30 new people every month that make their first contribution. And we still have 100

people per month actively working on Istio, which I think is a great achievement and shows that the project is actually in great shape and just stabilizing over time. >> Yeah, actually that well, that deserves some applause actually. Uh one thing we didn't put on this side, we've been able able if you were with us way back in the early days, Istio 1.1 took was it like 11

months to get out the door for a 3-month project? You guys have been on those sorts of projects before, I'm sure. Uh it was really painful us. It was more painful for our users. Since the 1.2 release, we've been able to maintain a consistent uh quarterly cadence for all of our releases with the longest delay, I think, being 2 weeks of any release since then. So, we're

really proud of that record. It's what enables all of our users to stay up to date, get their security patches on time, our features are able to make progress, etc. Uh so, very happy to have all of these contributors on board helping make that happen. Absolutely, yeah. So, yeah, let's get to some of the uh recent news. And absolutely drew this myself. This is not AI generated

at all. Um we have a bunch of things we I I hope we don't have to rush through them too much. But there As I said, we still have lots of contributions. So, there's a lot happening still in Istio to this day. And I'm just going to talk about a few things. Mitch is going to talk about a few things. So, one of the things that have

happened over the past year and I promise this is the only thing about AI uh is Gateway API inference extension support in Istio. So, this has actually merged, I think, summer last year was the initial support. And it's it keeps getting uh improved and we're adding more things as the API evolves. But basically, what it's about is like when you're running LLM workloads, you have some specific

challenges that you have to overcome, right? So, the models are huge. They're like hundreds of gigabytes or sometimes terabytes. So, you basically, you have to have them loaded on the nodes that you're routing your requests to. You need to aggressively cache prompts, like especially if you're working with like coding agents with huge context windows. The tokenization of your input and you're always sending the entire chat history,

right? So, tokenization can take seconds. So, we're talking about second delays when you're hitting a cold cache. So, you don't just have to hit a pod that has the model loaded. You also should hit a pod that has a warm cache. So, those are things that were not really you were not really able to do easily with just Gateway API. So, the folks over at the inference

extensions project came up with an API that helps you achieve those things. the way they did it was they introduced something called an inference pool. I don't know if it's a great abstraction. Technically, it's like it's a load balancer on steroids. And we can talk about, you know, how that's going to evolve in the future, but we support it today. And what it is is basically a

wrapper around the Kubernetes service. And for a Kubernetes service, you can say, I have this other service, which we call endpoint picker, that's going to help my proxy decide where to route the request. And the way it does that is by using the external processing feature of Envoy. So, this is something that happens at the ingress gateway when the request comes into your cluster. Um the Envoy

will actually forward the entire request to the endpoint picker because to figure out which model is needed and what cache you need to hit, you actually need to look at the entire request, not just the headers, you need to look at the body, too. So, we're routing like we're forwarding the entire request to the endpoint picker, that has all the information about where your models are running,

what the cache status is, and that will then return to Envoy a combination of IP and port that it will then use to actually route your request. And that enables a lot of the uh LLM routing use cases that that you need when you're hosting like large language models. Okay, move moving away from that, but staying in the ingress realm, uh we have another thing that is

still cooking. It's a bit early. You might have heard that ingress engine X uh has been archived live on stage here and it's been deprecated a few months ago. And there's a lot of people who I speak to actually regularly at KubeCon who would like to use Istio or a service mesh, but they don't quite know how to get started. There's a lot of operational complexity with

introducing something like that because it's just so deeply woven into your uh fabric of your uh network. And actually organizations are afraid of introducing it. So, one of the easiest ways to get started is to use it for ingress. Just configure your ingress gateway using Istio and start out with that, figure out some of the routing and policy configuration, and then later on you can roll out

all the service mesh features. So, how we're doing that is we started by actually my colleague Ricardo actually pushed that PR at the end end of last year by being able to selectively watch CRDs so you my SDD should only care about gateway API CRDs. And we want to continue on that by a sort of lightweight installation profile where you install an SDD that's focused on ingress

and maybe it doesn't even start all this all the things that are needed for for a real service mesh, right? You don't need a CA in there. You don't need sidecar injection. You don't need a Z tunnel for ambient mesh. You can have something very small and focused and that's easy to operate and reason about. And then when you're ready you can enable the mesh features and

roll out Istio to your cluster completely. I forgot to add the statistic to this slide, but our research shows that over 50% of the cores running Istio in the wild are actually only making use of ingress. So, they're already utilizing this pattern. They're just getting lots of things installed on the cluster that aren't really necessary for them and it's sort of extra baggage. So, we're working together

to make sure that we can give them an efficient install that gives them only the resources they need for that use case and we'll have more about that in the upcoming release. Right. So, let's move on to Z tunnel. And this is about how Z tunnel is actually used outside of the Istio project. And that might might sound strange. So, you've already heard Mitch talk about Z

tunnel a little bit. This is the purpose-built proxy for ambient mesh. It's running on the node and it allows you to do L4 routing and MTLS on your node basically. And it's it has way higher throughput, and lower latency than if you were to use like full-fledged Envoy proxy, simply because the code base is much smaller, it's written in Rust. It's also easier to maintain because it's

a much smaller code base. And the the nice thing about that, it turns out it's pretty pluggable. You can use Ztunnel and plug it into other projects, other architectures that are not just Istio, and get some of the the benefits out of it. And people are doing that, and especially CNI providers have started doing it. So, Calico, for example, they're actually deploying Istio entirely for you. They

have their own operator, the Tigera operator, that will install Istio for you when you ask it to. And it will also set up Ztunnel. And they have a fork of Ztunnel, actually, that makes sure that it works fine with their CNI implementation, because one of the the issues today that we still have with Ztunnel is as the name suggests, it tunnels your traffic, right? All the traffic

that goes to your pod and that leaves your pod is tunneled through Ztunnel on a specific port, and that's 15008. So, to a lot of CNIs that actually So, when you're writing network policy to the CNI, it looks like the only traffic that ever happens is on port 15008. So, it's hard to write policies based on that because no matter if it's 8080 or 443 or 22,

to the CNI, it all looks like the same port. So, you can't be as fine-grained as before. So, that's why they made actually a downstream change to to fix that when you're using it with Calico. The problem is it's not quite applicable to other CNIs. So, we need to figure out how to come to a general-purpose solution for that. With Cilium, they actually went another way. They

don't install the entire Istio control plane. They created something that they call the Cilium agent, and it uses the simplified XDS configuration protocol of Ztunnel. So, Ztunnel is not just smaller, easier to reason about, it's also much easier to configure. It's so easy to configure that they were able to build basically their own control plane for it. And they're deploying that for you, and they have also

made uh a bunch of downstream changes. So, for example, because they don't use Istio, they need something else that provides the certificates for them. And they added Spire integration. So, Spire is a is a CA that supports the Spiffy standard uh like Istio, and uh they actually also added UDS support so that their their control plane actually speaks to Z tunnel on a Unix domain socket on

the same node. Um so, those are all changes that are uh also flowing back into the project. There's PRs open for all of them. And I think those are great examples, you know, how Z tunnel can be employed outside of just the standard Istio setup, and it all benefits the wider ecosystem. And then there's more actually. There's one vendor working on uh capturing L7 metrics in Z

tunnel, using OpenSSL for uh FIPS compliance. Some people are even researching how to do DPU offloading with Z tunnel, where you use basically these smart NICs to perform your TLS encryption for you, and have even faster on-the-wire encryption. And yeah, we're staying with Z tunnel and talking a little bit about multi-cluster. Yeah, so this this next section is about multi-cluster, but since we've got enough people in

the room who wanted to learn more about Istio, let's take a brief primer. Uh Istio is all about making your network programmable the same way that Kubernetes makes your compute programmable. Now, the way that we do that is by capturing all of your traffic. Once and that you can capture it with sidecars, you can capture it in ambient mode with Z tunnel. Once we've done that, we're

going to apply automatic mutual TLS authentication and encryption for every piece of traffic in the mesh. We're going to allow you to write authorization policy for that that uses those same identities. That's what's going to set us apart from CNI mTLS for now, is that CNI mTLS is still going to under the hood need IP address for network policy. We're never looking at IP addresses for policy.

We're looking specifically at the identities that have been cryptographically verified. We're then going to allow you to use the gateway API to do all sorts of magic for traffic controls that we could never cover in a exhaustively, uh but weighted load balancing, round robin load balancing, least con, uh topology aware, the list is pretty much endless and you can go and look on our website to see

how you can control your traffic. And lastly, we generate observability for your traffic in the form of audit logs, uh or access logs rather, uh metrics usually consumed in Prometheus, but any open telemetry compatible provider, as well as automated distributed trace spans for your entire application end to end. And all of that happens with no need to touch any of the code in any of the pods

running in your cluster. So, that's overall kind of what a service mesh is and how Istio thinks about our features, but it turns out for most of our users, one cluster is not the right number of clusters. How many of you are running only one cluster in production? Okay, we got I want to hear about that. That sounds cool. We should talk afterwards. Uh so, for most

of us, our mesh extends well beyond the bounds of one cluster because a cluster is a regional entity. Uh multi-cluster has been around in the Istio project for some time and here we can see two clusters. Uh service B, you'll notice, runs in both clusters. Services A and C are located in only one and we've got different pods. Something to notice actually, we've got overlapping IP's across

those pods. If we were using IP addresses for our multi-cluster routing, this would be a significant problem for us. We would be unable to connect these two networks. If you tried to peer these VPCs, that would fail. However, Istio routes across clusters based on service name. Uh and in particular, this is a big change that came in ambient. Inside car mode with multi-cluster, we sort of kind

of had to have global knowledge. That is, every proxy, which means every pod, needed to know about every other pod in every other cluster on the mesh. Uh for those of you who are been around since we used to say, is it web scale? I don't know if you guys remember that phrase. Uh this is not cube scale. Uh this is an N squared algorithm that's going

to fall over. Usually, we saw customers get up to maybe eight clusters before scalability became a significant concern in this model. Um We had I'll describe very briefly. We had a sort of solution for this with east-west gateways in we called it multi-network Istio. So, your pods are on completely separate networks. The only thing that is accessible between those networks is east-west gateways, but the problem with

that is, remember how I told you about how great that mTLS security was? Uh it was not quite end-to-end mTLS. You had mTLS from the source to the gateway, and then another mTLS from the gateway to the destination. That sort of messes with your ability to say the policy is cryptographically enforced. You have It's not zero trust. It's like a little bit of trust networking. Uh Yeah,

so you had to choose. You could have a super secure multi-cluster network, or you could have a super scalable multi-cluster network. Well, we don't really like making trade-off choices like that. So, in ambient multi-cluster networking, we've had a very strict policy that there is no per pod data that is traded between Ambient will have no idea what's going on at the pod level in other clusters. It

knows where to find the east-west gateway for that cluster. It knows what services and what identities are associated with those. And then we're going to give it an account of instances, roughly, that could be stale so that it knows if it's doing load weighted load balancing, it can evenly distribute traffic. But, it's not going to know about end-to-end. And then when we route to the east-west gateway,

we establish an outer tunnel of mTLS so that we can say, here's where I'm actually wanting to go, what pod I want to go to. Then we establish inside that mTLS connection an inner mTLS connection that is in fact end-to-end from the source pod to the destination pod. So, we've effectively married scalability and security for your multi-cluster deployments. This is has reached beta just in the 129

release, so I would encourage you Usually with the Istio project, beta is where we start to say you can you can put this in production. It's pretty safe. Our standards for GAA are exceedingly high, but we do expect to make the GAA probably by the August All right, let's check time. We have like 3 minutes, so good. So, I'm going to go ahead and break the promise

that we weren't going to talk about AI anymore. Thanks for that one, Daniel. We've had a little bit of a struggle in the Istio project as AI has really changed the needs of a lot of people's networks. Uh we've focused so much. We talked about that history and how stability was so important and upgrades and consistency. Your service mesh is supposed to be boring. But a gRPC

networking is not very boring. MCP, the most commonly used a gRPC protocol, anybody know how old it 15 months. So, if anyone is telling you they have a stable MCP API, uh I don't want to call them liars, but you know, it it's it's a little early to call anything in MCP stable. And likewise, all of the other a gRPC protocols that are emerging today. So, we're

happy to announce our integration with Istio's third proxy. We're partnering with the agent gateway project. Agent gateway is going to be able to terminate mTLS the same way that a waypoint does. And agent gateway exposes its own set of APIs that are all about routing to LLMs, a gRPC, etc. We just gave a demo yesterday or Tuesday about how to uh centralize API key management and do

token-based rate limiting per application using Istio with agent gateway. Uh happy to show that off. It should land on the blog in the next week or two as well. Um but it has all sorts of features and the advantage is the Istio APIs get to stay boring. Uh you're not going to see MCP land there anytime soon, but if you want to use MCP in your Istio

mesh, the agent gateway APIs are completely compatible and they play nice together. This is going to be announced as alpha in the 1.3 release that's upcoming as well. So, keep an eye out for that and let us know if we're getting it right between having quick-moving innovative APIs and boring stable APIs. It's a challenge, but we'd love to get some feedback on that. Last thing we have

sort of a new class of contributor that we've seen arise in the Istio project. They've always been around to some degree, but it's getting more and more important. >> them have been around for a while, yeah. There's a lot of them. And that's our power user contributors. Most of the people who work on Istio traditionally are paid to do so full-time. This is not a volunteer-based project.

I work for Microsoft, you work for Red Hat. We've had contributions from Google and Solo. And I I started listing and now I'm going to leave people out and that sucks. I'm sorry, I shouldn't have done that. Say it one more time. >> companies, lots of great very valuable partners. But what's been very interesting is these are all end users of Istio. They do not get paid

to work on Istio. They get paid to use Istio. And they found something about it that sucks. This is This is not a perfect project. No one's here to tell you that. There's always going to be room for improvement. There's always going to be an edge case that we didn't think of. But these end users were so passionate about their particular use case, they didn't just open

an issue and tell us that this sucks. Although that can be valuable and helpful, we do appreciate those users. They actually went all the way to making design docs, getting them through the approval process, and submitting a pull request for these features. So, really want to thank these particular four from DocuSign, NetApp, AutoTrader, and Inovex. And we'd like to invite you to consider if maybe you, the

next time you run into something that isn't great about Istio, it's a great project to contribute to. We've heard from new new contributors that it's a very welcoming community and we'd be happy to have you join. So, with that right on time. >> Yeah, we have some time for questions, actually. Oh, yeah. And if you want to get feed uh us feedback, of course, you can scan

that that thing as well. Uh we have Mike there. Thanks very much for the whirlwind history tour. Um if I understood right, uh your solution for multi-cluster, it still needs the east-west gateway, but you're you're now doing the double tunnel, so you can still have the mTLS. So, uh there's no way architecturally to avoid that extra two hops without running into scale issues. Is that right? That's

right. The only way to avoid that extra hop is a flat network. Uh and a flat network eventually runs out of private IP addresses. I mean, there's a lot of private IP addresses. There are a lot. So, I mean, if if your needs are for smaller smaller scalability, if you're not looking at pushing that 2 million pod limit from RFC 1819 or whatever it is, um there

are some users who are working on building flat networking uh for multi-cluster. I'm going to say, personally, as a product manager, that's not something I'm going to recommend to my users. Uh if you're not hitting that limit today, there's no guarantee you're not going to be hitting that limit tomorrow. Kubernetes use cases have a way of spiraling out of control and growing into behemoths. If I understood

what you were saying right though, it's not so much the IP addresses, it's more the global knowledge that that is the issue. Or did I miss that? >> Yes. Yeah, so uh when you're within a single cluster or single network, the client side does all the load balancing, and that means that the client needs to know about the available endpoints to load balance to, and thus the

difficulty. Yeah, okay. So, without needing unless the only way to avoid global knowledge is the east-west uh gateway. Okay. I I I won't say that's the only way today. We spent a lot of time iterating and thinking on this. If the feedback is the east-west gateways are particularly problematic, that's another thing we would really like to know about. Uh we did a lot of testing on the

effect of throughput uh of the double encryption, because that does cost in terms of CPU and memory. I don't have the numbers to give to you off the top of my head, but if you go to the Istio blog with the multi-cluster announcement from September, you can see the performance numbers there to understand how what sort of hit you can expect to take there. Awesome. Thanks so

much. Yeah, thank you.