What's New in gRPC - Kevin Nilson & John Feig, Google

This talk focuses on the latest developments in gRPC, detailing its evolution over the past decade and the new features aimed at enhancing its usability and performance in cloud-native architectures. The speaker explains how gRPC facilitates efficient communication between microservices using protocol buffers and HTTP/2, making it suitable for large-scale applications. They introduce new user guides and video content to improve accessibility for developers. The session also highlights exciting upcoming features such as proxyless service mesh capabilities, load balancing enhancements, and integration with AI and machine learning frameworks. The speakers encourage community engagement and emphasize gRPC's growing importance in modern application development.

Welcome everybody. Thank you for coming to hear us talk about what's new in GRPC. I'm John. I'm a manager on the GRPC team. Um we've also got Kevin here who's been around for a little bit longer. And yeah, we're going to talk to you about some new features and and functionality. And uh just a little bit of intro about the GRPC project. Kevin. Thanks, John. Yeah, so

I've done a bunch of these KubeCons over the years and you know, one of the things that's always super exciting to see is uh just the growth of GRPC. And this is one of the things we always like to kick things off. Uh let folks know, you know, GRPC is been around for 10 years, super stable, continues growing um over time and definitely want to thank all

of you for for making that a reality. Right. Uh we appreciate everything that you guys do using the library, but then also, you know, a lot of the contributions. And so if any of you um has contributions you'd like to make, things you want to uh engage more with the team, please reach out to John or I after the the talk. We're always open to you know,

we would love to work more and more with folks, get more people involved um in GRPC and continue this uh great growth that we've we've seen over the years. Um I wanted deep dive just a little bit before we drive into sort of technical content. We got a bunch of usage, right? So uh Maven Central um for Java, 7 million views a week, so that's a really,

really huge number. Uh 304 monthly downloads in Python. And uh 29 million weekly downloads on NPM. And so it's really great to see all this and uh you know, GRPC just continues to be more and more relevant everywhere, becomes a ubiquitous way of doing um you know, RPC calls. And so absolutely if you haven't if you're if you're new to GRPC, check it out for sure. If

you've been using it a long time, many thanks. One of the the feedbacks that we got at KubeCon a few years ago was at the time, this is about 3 years ago, people were kind of seeing GRPC as a little bit harder to to navigate the documentation, harder to get started, harder to understand some of the deeper content. And so we took that feedback from all of

you at KubeCon and we've been making a large investment, added a bunch of new user guides as you can see four new user guides and a shift more towards a lot of video content. We think people kind of like consuming a lot of these short form videos, kind of getting an overview, getting started with things. And you know, giving you a point to understand the the big

picture of things before diving in deep. And so definitely if you haven't checked out the YouTube channel, check that out. It's a great place. We keep doing more and more in that area. Another thing I want to tell you about GRPC Conf is a big conference we have coming up. The last few years we've done it at the Google headquarters for cloud, cloud headquarters in in Silicon

Valley. So it's a great opportunity to come and see that. This year we're looking forward to you know, running an event again in the Bay Area. We'll either do it on the Google Cloud campus or at the Computer History Museum. We thought that'd be kind of a cool venue, so we're we're trying to figure that out. Haven't finalized all the details yet, but we have something coming

sort of you know, in in the fall for that. Would love to see all of you there. It's a great place if if you you know, would love to have you as a speaker or as an attendee. So, definitely check that out. And again this year we'll be doing an event in Bangalore. So, John and I will both be at both of those. Look forward to seeing

all of you there. Hopefully you can make it. the last thing I wanted to show is just sort of, you know, some of the the usages of GRPC. We're over 10 years old now. Things continue to grow. We get a lot of great contributions from folks like Netflix, Data Dog. Been doing a lot of work with Broadcom. We'd love, like I mentioned earlier, you know, if you

want to work with us, please let John or I know. And you know, the usage out there just continues growing. A lot of contributions from LinkedIn recently. Working with Paul on that stuff. And so for sure, you know, it's really exciting to see all the all the usage over the time. So, with that I'll I'll hand it over to John and he'll drive into a bunch of

the the technical stuff. Thanks. All right. Thank you, Kevin. So, what makes GRPC so successful? GRPC has been proven an exceptional fit for cloud-native architectures. It's designed to enable efficient communication between services using protocol buffers to reduce message size and HTTP2 for network efficiency and real-time communication. This makes it ideal for microservices that need to exchange large amounts of data. It's language agnostic nature makes it easy

to deploy on various platforms and infrastructure. It also allows developers like you to build and deploy services in your preferred programming language. This portability is crucial in cloud-native environments where it's common to build microservices in different languages. Last but not least, GRPC enables an architectural platform for microservices enabling teams to break down complex problems into small, well-encapsulated services. This enhances developer velocity in large organizations and allows

smaller teams to work on different components in parallel, accelerating overall development process. In addition, gRPC provides features like load balancing and retries to improve fault tolerance and ensure application reliability in distributed systems. We use gRPC heavily in Google. etcd, containerd, Kubernetes, and many other Google Cloud projects use it for their internal communication. gRPC has become the popular choice for building modern, scalable, and efficient applications in cloud-native

environments. Over the past few years, we've committed to making cloud-native adoption easy, so you can scale your services quickly. Proxyless gRPC service mesh streamlines the deployment process, eliminating the operational overhead associated with managing and maintaining sidecar proxies. This approach not only reduces complexity, but also improves resource efficiency, making it particularly appealing for large-scale cloud-native gRPC comes with many features, which allow you to bring mesh capabilities to

your applications easily. AI and ML have been quite popular for the last few years, and gRPC is revolutionizing the way AI and ML models communicate and operate. Its inherent efficiency and speed, and robust support for real-time data streaming, establish it as a key building block to train large models that we have today and serve inference requests quickly. This efficient data ingestion process, or efficient data ingestion and

processing facilitated by gRPC, significantly accelerates the model development life cycle. Training large-scale models involves handling and processing enormous data sets. gRPC's efficient data serialization powered by protocol buffers, combined with its ability to handle large payloads makes it a robust and reliable solution for streaming vast amounts of data directly into training pipelines. Thanks to its low latency and high throughput, gRPC is efficient at handling inference requests. Prominent

machine learning frameworks such as TensorFlow utilize gRPC to manage and process a vast number of Nvidia's Dynamo Triton framework handles the development handles the deployment and servicing of AI models use gRPC as their transport. ensures that real-time applications receive predictions with minimal delay. We're doing a lot in the AI and ML space. gRPC is going to see you through AI and ML journey. And we're here to

show you how. Next. Okay. Over the years, our developer community has raised a number of challenges to us in building out the cloud cloud There are three main pillars in our approach to providing solutions to those challenges. The first should be no surprise to you if you've been to any talk on gRPC in the last 5 years. Service mesh. In 2019 we decided to set out on

the long journey of creating service mesh with outside car proxies. We did our GA in 2020 and haven't looked back since. New feature requests have just come kept coming in since then and that trend did not stop this year. Just about every new feature I'm going to mention today is related to service mesh in one way or the other. Next, observability. gRPC went all in on open

telemetry a couple of years ago making it easy for gRPC users to collect logs, metrics, and traces. That journey is not complete yet and you'll see a number of improvements we're making in open telemetry space. Finally, modernization. The The moves fast and it'll leave you behind if you don't keep up with it. This is true for every language that we support and for new languages rising in

popularity. Some of our most exciting projects come from the modernization pillar. So, with these three pillars in mind, let's look at some upcoming features. We'll kick things off with a pair of killer's proxyless service mesh features, EXT Proc and EXT Authz. EXT Proc is about modifying incoming requests and outgoing responses on the server, similar to gRPC interceptors. While EXT Authz is about determining whether incoming requests are

authorized. Both of these features use an interesting plugin mechanism. While interceptors need to be compiled directly into your server, both of these features work based on a gRPC call out model where the server calls out to a pre-configured gRPC server in order to fill the the feature's functionality. So, for EXT Proc, the request and response modification could be performed by a separate gRPC server, possibly co-located with

the server, or possibly centralized, rather than compiling in an extension. If you're a platform team, this is a dream come true. The power of gRPC interceptors will now be available to you to enforce across your entire system. All you'll need to do is deploy an EXT Proc server and configure your service mesh to point to it. The same functionality will be available on the client side as

well. That is, the client can be configured to modify call out to an EXT Proc server based on its request and responses. EXT Authz works very similarly. Each incoming request will result in an outgoing request to the application or from the application server to a pre-configured gRPC server, which will answer the question, "Is the sender authorized to make this request?" This new plugin model will offer you

new levels of flexibility in building your This feature is still in the design phase, so be on the lookout for a GRFC laying out all the details later this Next, serverless integrations. Service mesh operators generally don't just run Kubernetes. Whether they like it or not, feature teams in their organization often run workloads on various serverless platforms as well, such as Cloud Run. Unfortunately, there has there historically

hasn't been great interoperability between these two platforms for service mesh. For GRPC proxyless service mesh in particular, we've seen good support for Cloud Run in GKE. In fact, Kelsey Hightower demoed exactly this when proxyless was first introduced. But the other direction, Kubernetes to serverless, hasn't worked out nearly so So, with Cloud Run in mind, we've designed and implemented a handful of features that make this for a

first-class experience. First, we've added XDS host writing host rewriting that enables the use of vanity URLs in XDS targets, something very common on serverless platforms. Next, we've added JWT token-based authentic authorization for outgoing service mesh mediated RPCs. And finally, MTLS off based on spiffy identities. Putting all these features together, the integration between serverless and proxyless service mesh is now truly first class. HTTP connect is a special

HTTP verb that turns an HTTP connection into a tunnel for a TCP connection, allowing various protocols such as SSH, FTP, or GRPC to tunnel over that HTTP connection. This is useful in two ways. Forward proxies to enable egress from a network environment that otherwise disallows egress. Or reverse proxies to enable ingress into a network where ingress is otherwise restricted. Uh an example is use HTTP connect with

a reverse proxy to individually contact servers deployed in the public cloud from an on-prem environment. When those servers are otherwise not accessible from the public internet. gRPC has had support for tunneling with through proxy since 2017. The function The functionality was defined in the very first GRFC. We are now making this functionality much easier to configure. Previously, it was only configured manually in a per-language way. Now,

it is being extended to integrate with xDS. xDS would send configuration based on the destination server server service where HTTP connect is needed. Users of the early versions of this feature already take advantage of it to create a single service mesh across their on-prem and cloud installations. First, with forward proxies, so enabling egress from a network environment. Oh, I'm sorry. This got duplicated in the notes. Um

onto SPIFFE. Next up is support for multiple SPIFFE trust domains. As a brief intro, SPIFFE is a system for assigning cryptographic identities to individual workloads and then verifying those identities. Those workloads could be virtual machines, Kubernetes pods, or anything else. The cryptographic identities can be used to establish mTLS connections between workloads. As you might know, setting up an mTLS manually is a huge hassle. gRPC has supported

configuring SPIFFE-based mTLS via xDS since 2021, but only via CA files, certificate authority files. We're now adding support for SPIFFE trust bundles. This would allow you to take advantage of multiple SPIFFE trust This is useful for a bunch of scenarios. Uh separate root of trust for development, staging, and prod Useful to ensure that less secure environments, staging, cannot accidentally access production. Each product area within your company

might handle its own identity management. In this case, clients should be able to interact with workloads within the same trust environment and across domains. You can expect this to land in C++, Go, and Java this year. Okay, like I mentioned at the beginning, service mesh and observability are two of our strongest pillars. And the intersection of these two spaces is just as hot. We're adding Open Telemetry

metrics for weighted round robin load balancing policy, the pick first load balancing policy, and xDS client, the component gRPC library that connects to xDS control plane. In addition to these 12 metrics, we're adding cross-cutting labels to indicate the xDS locality associated with the metric. This would roughly correspond to a cloud zone. Originally, all the gRPC hotel metrics operated on a per-call basis. Recently introduced Uh we recently

introduced a framework for non-per-call metrics. For example, xDS client is used across multiple calls and even multiple channels. The non-per-call metric framework enriches the observability enable available to users of gRPC Proxyless service mesh. Together, these features will make running a service mesh with gRPC easier and more reliable than ever. Another new gRPC feature is custom back-end metrics for load balancing. This is a metric mechanism in the

gRPC library that allows you to inject your custom metrics at the gRPC server. And these metrics can be used for load We follow open request cost aggregation standard. And you can report your custom metrics The first option is your server attaches the metrics to the trailing metadata when the RPC finishes. Another option is to period is periodically sending metrics out of band. Custom back-end metrics is available

on production now. And if you want to learn more details, check out our documentation at the short link, which takes you to our developer guide with example code in multiple languages. We recently added support of weighted ro- weighted round robin load balancing policy. And it can be used with custom back-end metrics that we just covered in previous slides. It's really simple to configure on your server if

you're just using CSM. Just set custom policy as WRR with the parameters that based on your use cases based on your use cases. And if you prefer to send metrics out of band, set enable OOB load report to Additional parameters are available for fine to fine-tune the behavior of WRR You can still leverage our WRR policy if you're not using CSM. And here we have an example

in Go. You can set load balancing config with the configuration in JSON format when calling dial in your application. Once you configure the LB policy as WRR, the next step is to send metrics from your backends. Here is the formula on how gRPC load balancer selects a backend with metrics you can send to us which includes CPU utilization, QPS, EPS, and error penalty. Below is an example

of OOB reporting. In gRPC Go server code, you can create a server metrics recorder with options that fit your needs like minimal reporting interval then register your recorder and start sending metrics like CPU utilization, at any places you'd like. And that's all you need to do to enable WRR provided by gRPC. More details can be found in the short link at the bottom. Next, we have some

very interesting developments in the AI space. Two protocols enrich AI models with external data and capabilities have absolutely exploded in popularity over the past year. MCP, the model context protocol and A2A, the agent to agent protocol. MCP gives developers the ability to enrich LLMs with abilities of any tool you can export via an MCP server. You can imagine search, file access or any other traditional software The

second protocol, A2A, gives long-lived AI agents the ability to collaborate with one another in a structured non-English language. A2A was built early from early on with gRPC support. And the protocol is fully defined in terms of protobuf. The MCP HTTP transport was based on JSON and designed in a way that makes it difficult to support a typically horizontally horizontally scaled load balanced deployment. Since then, we've gotten

many requests to enable MCP over gRPC with all of its rich features. In August of last year, we made the announcement that we would be working with the authors of MCP to add support for gRPC. We're now in the process of publishing an experimental fork of the MCP Python SDK with gRPC support as a proof of concept. We're contributing our work to Anthropic to build to add

pluggable transport API to their SDKs that will allow us to distribute the gRPC transport as a separate package that plugs into their SDK. Adding something like a new transport to the official MCP specification and SDK will require a lot of community interest and usage. uh oh, sorry. The experimental fork can provide that signal to the community and help determine next steps. gRPC has really established itself as

a key component in the AI space, and I'm looking forward to seeing what AI applications it will enable next. And this in particular is a place where we would love feedback from the community. So, if you have some, if you tried this out, please let us know. And last but not least, we have the newest entry in the family of officially supported gRPC languages, Rust. Rust has

become incredibly popular over the last few years with its safety by default and zero-cost abstractions. And no language is complete without first-class gRPC support. So, we're working to make sure that support is We're building on top of the already very popular Tonic implementation, bringing it to parity with other first-class languages like C++, Java, and Go. And that will mean a full XDS implementation so that you can

use gRPC Rust in your Proxyless service mesh deployments. We've been working directly with the author of Tonic to make sure that we deliver the best experience to both existing users of Tonic and to new users of gRPC Rust. That work resulted in a back in August at GRPC Conf. And expect more to be available soon. Our GRPC Rust team did an in-depth talk on the new library

at GRPC Conf. In case you weren't able to make it, you can check out the video on YouTube. And where's Kevin? All right. I'll have you back on in a minute. Wrap it up. Awesome. Well, thanks so a lot everybody for coming. Um John and I'll be around for any questions. Uh do want to remind you there's a bunch of different content I put up here um

on the slide. Uh if you haven't seen the the GRPC.io, that's where all of our documentation is. We're doing a lot of work there. Um YouTube channel, subscribe. That's where you know, we're trying to do more and more of an investment. Uh we're running meetups in the Bay Area and in Bangalore. And then um occasionally we'll do those also as kind of a global thing where they're

um online and people can can join those. Um we've got a mailing list. This is a great place for you to basically share some of the things you're doing, ask questions of the team, uh keep up to date with any announcements or any security vulnerabilities or new releases and things like that. And so definitely subscribe. And then um on X, you know, on Twitter, um is another

place. So again, thanks to everybody for coming out. If you have any questions, uh there's a mic here. We have a mic here that we can run around. And uh absolutely look forward to seeing you um at GRPC Conf later this year um in the Bay Area or uh in Bangalore. So first question. Yeah, thank you. Thank you. Yeah, I was I was holding you up. Uh

I'm just curious if you guys are tackling uh replication at all cuz that's something that comes up quite a bit. Like how do you how do you implement GRPC to do your own replication is what I see quite often. but just wondering if there's anything on the road maps of that. I'll take that, John. Um it's a good question. Um And I think Yeah, I'm not aware

of any specific things that we're doing related to replication, but for sure, I mean, shoot us a message and let us know. >> Yeah. Kind of, you know, we'd love to hear more about what you're looking for, and then Yeah, we can we'll work that out. Uh well, first of all, thank you for the update. Um I'm wondering about the load balancing capabilities. I'm coming from uh

the observability space, where you have lots of open telemetry and and uh uh or also the the whole Grafana stack, where they um transport vast amounts of data, so the messages are quite big, and especially on Kubernetes, you want to be topology aware to kind of save on cross AZ traffic and stuff. So, what is the capability there? I've I've looked a bit into it, and uh

the resolver uh updates are not happening as quickly as one would expect. And especially if no subchannel goes unhealthy, uh there's no connection rebalancing and stuff. So, are you looking into this at all? Because, as I said, there's lots of money to be saved there. Yeah, we're we're I don't know, John, if you want to take it or me, but we're definitely um make like the load

balancer policies is one of the areas where we spend like a significant amount of time. Mhm. Uh John showed the newer weighted round robin. Um we also, you know, have the ability for you to plug whatever load balancer you want, or you can write your own custom one, which typically we don't recommend. Yeah. Um if we don't have support for what you're looking for, let us know,

and we probably can add it in. Is it topology aware? Yeah, I think that like that specifically is something we should probably chat about. Um you know, it sounds like a an interesting performance improvement that we could probably look into. Yeah. Thanks very much for the update. Um, I was wondering uh with Gateway API and GRPC route uh interesting capabilities, but yeah, it seems like many providers

GKE and Google Cloud don't yet support GRPC route. Uh and I was wondering what's the strategy there? Is that just a roadmap thing or there's a different approach? What's the thinking there? Yeah, so the the Gateway support something we added what maybe a year and a half ago uh GRPC route. Um, actually gave a talk at KubeCon in one of the Europe talks. I don't remember where

it was. But uh for sure, I mean that's something that we're trying to get more and more adoption. It is available. I think just, you know, whatever you can do to help, you know, get awareness of that. but it is, you know, something that we would love to see grow. You know, Gateway being a lot of the future of the way that people configure things in in

Kubernetes and that's, you know, why we have that that support there. Yeah, was there something in particular? Yeah, I mean, you guys are both at Google, so I'm curious why GKE in particular is not yet supporting that. Yeah. Yeah, so I mean, it's just a prioritization thing. Looking for the right maybe not the right, but just looking for someone who's putting in a specific request for it

and saying like, yeah, we want to be this. You know, for those types of things, we're always looking for sort of a keystone customer or partner to work with us, tell us it works, you know, adopt things early. And so if you're interested, let us know and we should be able to work together on that. Okay, great. Thanks very much. So, I do just want to note

we only have about 1 minute left and then Kevin and I will be around if you want to come chat with us after the session's over. Awesome. 1 minute left, maybe we'll end it here. Thanks again everybody for coming and um you know John and I will out in the hall or up here depending on the next talk stops and so thanks so much and good luck

with GRPC. Great. Yeah, thanks