Spin-Up, Test, Tear-Down: How Trivago Runs Developer Preview Environm... Armin Aminian & Jan Wozniak

This talk explores how Trivago manages its developer preview environment at scale using Kubernetes and KEDA. The speakers, Armin and Jan, discuss Trivago's infrastructure, which consists of over 100 microservices deployed across two regions for optimal user experience. They highlight the challenges of ensuring new features work reliably in production, particularly when many developers are working concurrently. By implementing automated deployment of preview environments per pull request, the team allows developers to test new features in isolation without blocking each other. Jan introduces KEDA, explaining its role in handling automatic scaling based on HTTP traffic and ensuring a smooth user experience with minimized cold starts. The talk concludes with practical insights on setting up preview environments and the collaborative effort involved in maintaining their Kubernetes architecture.

Welcome to our presentation. I'm Armin. I'm a sorry at Trivago. And my name is Jan and I'm a good maintainer. So yeah, today we are going to talk about how Trivago runs the developer preview environment at the scale. So yeah, let get us started. So how many of you have used Trivago to like a find a combination? Okay, okay, not bad. I expect more hands from my

colleagues. So it will be affect your salary soon. so probably >> by the way, talking about your colleagues, have you used Trivago to book your accommodation? >> No. Trivago we are a travel meta search engine that compare accommodation prices across multiple OTAs. So OTA could be booking.com, Expedia and our goal is to help the user to find the best deal. And you might have heard hotel Trivago

thousand times in from our TV ads. Trivago infrastructure. So we have more than 100 microservices. We deployed those across two regions to give a better user experience and latency and everything is running on GKE Kubernetes cluster. So when you suppose you Jan opens Trivago or our application, he search for accommodation. What would be happen? So the request some request would be fired against our graph care router

and our graph care router is responsible to send or find out those to our sub graph. And those sub graph are going to you know find the prices, accommodation details from our search and the price infrastructure which are connected to So what is the environment exactly? Let's follow Alex journey. So, Alex, he's a one of our lovely developer. He's working on one of our important services. So,

imaginary service here, so booking service. And he needs to add a new feature to this booking service. And this booking service also has a dependency to both search service and the payment service. So, test pass. Everything looks perfect on Alex machine, and he said, "Yeah, it works on my machine." It's like a famous word of engineer before introducing a big incident. Um but will it work in

production with the real dependency that we have on our staging or on our production? I I think I'm not going to give away a secret, but it didn't. >> It didn't. He merged it. He introduced a SRE pissed. 100k lost. So, what doesn't work? So, in the traditional approach, if you want to ship something on a production into the production, you can't say, "I will test it

on my local. It would work." But the local completely is completely different than our production or staging. Or you can be YOLO. it. I want to ship to production. Or you might ask yourself that you have a staging environment. Why you are not using a staging environment? But if you have a like a scale of Tiago with 200 developers, so they need to wait for a staging

to be free. Or the test would be unreliable because Yiannis can create a PR, and then it affect my change. Or I can blame him. Yeah, you broke my test. Everybody will blame me. Yeah, yeah, I mean, it's our our job, so. uh yeah, Alex open a pull request and a CI uh pipeline start running and he goes for a coffee. So, meanwhile, a magic happen. A

preview environment gets automatically uh deployed for him. So, and he would get a URL like a PR uh the number and booking.internal.trv uh dot cloud. So, what get exactly deployed? Suppose we have uh this uh like three services in our production, so search, booking, uh and the payment with this version. In our preview environment, we are going to only deploy uh the service that uh is changed

in the pull request. And here, the Alex uh get a uh preview, so the booking service. And this booking service is going to uh connect to the payment service and the search service uh as a dependency in the uh our staging So, now Alex can test everything. Uh we have a fully covered uh integration test, so it can find uh issue as uh early as possible. with

the URL, he can share it with the uh with the QA. So, there's no need for QA to say, "Hey, can you deploy it on my machine?" And of course, it's very useful for our front-end engineer because they can uh see the like a like a feature and share it with their team pull it and to see if it's exactly implemented as expected. So, now Alex can

merge uh with confident knowing that change would not break anything in production because we can catch all the issues as early as uh possible. That sounds very expensive, right? Like running a environment for every engineer, for every of their PRs? >> Yeah, yeah, yeah, that would be very expensive, but you know, these days uh 10x engineer, so burning token, more feature, so Yeah. But let's see what

we uh uh uh going to do. So, why why our preview environments are so important for us? so, I mean, here here in at the cube car, you had AI AI AI AI. So, now you need to be 10x engineer. You need to ship more feature every day, 10 PRs per day. So, in So, we need to have give it confidence to our engineer and plus to

this area to not face any incident. So, in Tiago, we have more than 200 engineer working daily and we have just more than 100 PRs active for just our front end or our web application. And more than 100 for our back end services, which we result would be 200 preview environments. So, there are so much preview. So, we need to find a way to like make them

cheap. So, our goal is to give engineer confident to ship the feature. So, our solution is preview environment at the scale. So, give every developer isolated environment to test, change, and ship with confidence. So, no blocking, no conflict, and no drama. It's worth to mention that what was our starting point. So, we had our front end application on GCP Cloud Run and the GCP Cloud Run is

like a serverless a container platform that lets you to deploy containers without any you know, pain of Istio. so, and it also support a scale to zero. And then we had our back end services on GKE. Sorry. There you go. I'm finally useful. But, there was a big difference. For our back end services, we we didn't have any But our front-end, we had. And the good and

bad of the Cloud Run. So, the good things about the Cloud Run is you just pay for what you are using. Just a like a is a request based like a billing. But at the end, it was way more expensive for us in compared to the Kubernetes. And of course, we had to maintain two separate architecture of monitoring, logging, deployment, which was very painful. But was was

very good for our preview environment because we could handle more than 600 front-end preview with just 100 euro per month. So, our strategy was to unify everything on Kubernetes. But how to keep cheap preview So, our goal was to solve preview environment for back-end services first. And then if we can find a way, we can migrate everything to our front-end migrated to the Kubernetes. So, one solution

can unblock both problem. But so, as you might know, Kubernetes doesn't scale to zero. It does 1.36. But let's see what what what we have here. So, if you have more than 600 preview environment, each needs two pods on average, and one CPU, one gigabyte, so Node.js. And it would be more more expensive than our production. And you might said, "Yeah, you can use Chrome base scaling."

But it doesn't work because we have a like a flexible working hour, and like myself, I'm also crazy. I always start at 6:00 a.m. It doesn't work for me. And as I said, so Kuberentes 1.36 support scale to zero, uh, which uh, doesn't work for HTTP based scaling which Jan, uh, is going to talk about uh, that. So now I pass to Jan to talk about interesting

part. Right. Thank you. Uh, just a quick uh, research. Uh, how many of you know KEDA? If you can raise your hand. I think that's a majority of the room. How many of you use it in production? Oh, fewer hands, but still good. Still good. Thank you. Uh, so for those of you who didn't raise a hand, uh, there's a quick summary. Uh, KEDA is a CNCF

graduated project, uh, written in Go and it extends Kubernetes native auto scaling capabilities. Uh, it plugs into the wide range of third-party software. So pretty much anything you can measure, the chances are there's going to be integration already built in KEDA. How do you configure KEDA? Uh, there's a brand new resource. It's standalone API. It's called scaled object or scaled job, depending on the type of the

trigger. And KEDA itself has two modes of operations. Under the hood, for majority of the scaling, one to N and and back, it uses HPA. It delegates everything essentially to HPA. So thanks HPA maintainers. You're doing our job a lot of uh, easier. And then it has the second mode of operation and that is scale from zero to one, activation and deactivation. because that wasn't enough for

Trivago, uh, they also wanted kind of a smooth with reasonable cold starts uh, situation. Uh, they uh, found out that there's also HTTP add-on. So quick introduction to that. Not all scalers are built in and baked into the KEDA source code. Uh, some of the scalers so are called an external scalers. You can plug them through a GRPC interface. One of them is HTTP add-on and that

allows you to scale based on the HTTP traffic. There are a couple of notable differences. One that's very big and important is that in order for it to know about the traffic in real-time, it needs to proxy all of your traffic. So, there's a component called interceptor. And also one that was pain in the ass for the users and everybody complained about that user experience is that

it's not configured through scaled object, the standard API of KEDA, but instead it has a brand new object called HTTP scaled object. For the activation deactivation, there is also additional feature that's kind of a selling point of HTTP add-on. The first request is cached in the interceptor. It's queued. Your application is getting started. Once it's ready, it releases all the requests. It releases the hounds and they

all hammer your application so it respond to the traffic. next I would like to tell you that not only um practical knowledge is what we should know. There is also history and I'm a little bit of a history nerd and I think if in order for you to know a success story, it's important to have a little bit of insight into what has happened during a certain

timeline. So, HTTP add-on, it's a project that started roughly 6 years ago and I split the entire timeline into three eras. The first one I called a pioneering years. Highlights from that era, the project was started by two excellent Microsoft engineers. It was kind of a envisioned as a alternative to KNative serving which was not GA at the time. And it started off pretty strong. It was

popular, successful. Unfortunately, roughly 2 years after the And the they needed to move on and the project was abandoned a little bit. I'm pretty sure that if that was the time when you tried HTTP I don't you probably had a very mixed feeling about it. Luckily, that's not where it ended. The second era, I call that a community revival. Community people picked that up, volunteered. They tried

to maintain it. Towards the end of the era, Kidefy, company that employs me, also started to be interested in it. We created our own fork and tried to also help with the maintenance overhead. And I also think that was roughly the time when Trivago tried the project. So, how was your experience? It was terrible. Completely terrible. Wait, wait, we're we're on the same page, remember? >> No,

no, not not in this topic. Yeah. But I mean, joke aside, it was working for our back end services. So, if you remember, I mentioned that we planned first to migrate our back end service or I mean to use preview for those. it was okay for like a three to 400 object. But after we migrate our front end application, I was hunted down every day by QA

and engineers because everything was not working. And I decided to hey, let me to write a tool for it and get rid of the add-on. Yeah. But after some discussion, we decided not to reinvent the wheel again. Yeah, but that was still very useful because you reached out to maintainers, reached out to the community and there were some good additions from originating in your project. By the

way, I don't think we mentioned the name. It's called Go-Zero. You can still check it out. It's still live on on GitHub. Yeah, yeah. I think it has like two stars, maybe three. I'm I'm one, you're second. >> Second, yeah, yeah. So, yeah. >> But it support GRPC which you didn't. Yeah, and that was that was like one of the features that was missing in the HTTP

add-on. It was just plain vanilla HTTP 1, and for your front ends you needed GRPC. Uh so, thanks to you Bjørgo zero, we kind of backported it. It was the same language, Go, not that hard. Uh but, yeah, that was that was an example of a successful collaboration. So, that's the second era. Uh and the last one, and that's the most recent and also the most promising.

Uh Red Hat also joined uh the group of maintainers of the HTTP add-on. So, right now we have plenty of maintainers, uh stable cadence of releases, a lot of new features uh got backported donated from Keda file so, and implemented by Red Hat. And finally, I can say that aloud with confidence, we're getting rid of the HTTP scaled object. So, if you are using HTTP add-on, and

if you hated HTTP scaled object as much as I have, uh it's going to be gone. So, that's the entire timeline. Uh thank you for bearing with me throughout the history lesson. But, back to practice, back to what's important. Like, what is different between standard scaling and scaling through the HTTP add-on? Uh not all scalers were born equal. Uh some of them they just care about the

metrics. Some of them, it's possible the HTTP add-on might be the only part of that group, uh they also live on the data path. So, every single network request for your auto-scaled application goes through that interceptor. Those are challenges, and also there are some advantages thanks to that. Um major, and I think that's probably obvious, but I'm going to say it still, when the traditional scaler goes

down, it either stops the auto scaling or auto scales to the fallback number of replicas if you configured that. Uh but, your application is probably going to be okay, at least for some foreseeable future. HTTP add-on, if that one goes down, if the important part, the interceptor, if it goes down, traffic doesn't go to your application. So, that's uh that's a scary thought. as mentioned, there are

advantages and disadvantages. One of the major advantages and selling points of the HTTP add-on is cold starts. So, instead if you use Prometheus, scaling based on Prometheus, the first request, uh you have to configure retries, you're hammering your uh network path, you're sending repeatedly request until your application is ready, otherwise your users get 503s, and they have to hit the F5 refresh button. With HTTP add-on, a

little bit more smoother. Still not perfect, but at least they don't have to hit that they'll still have to wait for the cold start, but more pleasant user experience. Uh, unintended benefit of the HTTP add-on is if you run Ingress, and you are not quite ready to migrate towards uh the Gateway API, you can actually get some of the Gateway API features in Ingress, uh such as

routing by header, for example. Uh, you just configure it on the HTTP scalar objects, and you point all the Ingresses towards the interceptor, and interceptor will do the advanced routing for you. And as benefit, you also get auto scaling, which is not uh And last but not least, uh in a typical scalar uh for Prometheus, it doesn't matter to Prometheus whether he's saying, "Hey, there's five requests

per second." or if he's saying, "Hey, there's 50,000 requests per second." It's still a number. He's reporting one single number every 15 seconds. The proxy, again, it has to process all of those requests, so you need to auto scale the scalar. Um yeah, we all need to go a little bit deeper. Uh there's ways how to mitigate the limitations. Um for example, for the fallback when with

standard data you configure fallback, there are additional features on top of how you wire your application, how you send the ingresses to the interceptor. Uh and you can do that dynamically either by inject injecting endpoints to the service or dynamically changing uh the ingress reference. Uh and in that case, whenever HTTPRun goes down, you're not completely down, you're back to the standard fallback mechanism. Enough about theory.

Now, let's pray. >> So, uh yeah, I need to pray. Uh demo gods, our platform, uh engineering, it's our sitting here. Uh so, let's see. I have a demo. Okay. Okay. So, uh here it's our uh front end repo. I create a like a demo. Uh I can I open a pull request and in this pull request I'm doing some CSS and uh like a TypeScript adjustment

just introduce some change. And we are going to see what would be happen for when I open the preview. let's see. Um So, as you can see, yeah, we have a two uh like a services in this preview. So, main user and the story and both are scaled to Uh so, now if I get a watch and then I can open in preview. Sorry, I put I

need to put it into the bash because of uh URL and the security. So, it's in the iframe. Yeah. So, yeah, as you can see the it has start the pod let's see. Finger crossed. It's running. Now, they still So, I think this is a perfect opportunity. We can do another survey. So, how many of you use HPA in HPA? Okay. Yeah. So, now we have Uh

so, as you can see it's like hello KubeCon 2026. yeah, and then it's working. And I also like a after 10 seconds or not used, it's also back to the scale to zero. So, to just for a demo. And as you can see here it's uh uh it's scaled back to uh zero because I didn't have any interaction with it. Yeah. Uh now, let's back to the

slide. Armin, would you mind if I just finish the survey really quick? I was also curious how many of you use VPA in production because usually there's like 1 to 20 ratio for every 20 users of HPA. I think that checks out. Yeah. Anybody using the in-place resize on VPA? Some hands. Nice. I like that. Okay. Now, let's a little bit talk about the like a details

uh and the user flow. So, the user open a pull request and every PR automatically trigger the SCI workflow. And in our GitHub action we are doing the like a same I think all of you are doing like a same. So, we just build a Docker image, push to the registry, and we are using the Helm and Skaffold to render the manifest for a specific preview and

push those into the GitHub branches. And as you know, if you want to deploy any like a manifest on the into the Kubernetes, you need to have Argo CD and Argo CD application. But as a preview environment are dynamic, you need to find a way to make those like a create application for those dynamic environment. Because of it, we are using the Argo CD application set plus

with two like a nice generator pull request and good generator. The pull request generator is watching your repository pull request. And if you see that there is a PR with a specific requirement, I mean labels, it can deploy those manifest into your Kubernetes cluster. You might be curious about the YAML, so I will share it later with you. So. Thank you. I've been waiting for that for

a long time. >> yeah. So, what would be happening inside of the Kubernetes? We have a two URL, PR URL and the commit URL. Don't ask why because it takes more than 20 minutes to explain the issue and those kind of things. But the PR URL is always pointing to the latest and healthy pod, but the commit URL is pointing to the latest pod. And then they

for instance user open the PR URL, the request to the ingress issue ingress and then to the KEDA interceptor and interceptor check that if there is a it send it to the pod. If not, it keep it in the like a queue or cache. And scale it up and then later forward And we have a complete picture. So, when the like a user create PR, CI and

then branch, GitHub branch, Argo CD and then on the Kubernetes into the Kubernetes So, the magic is when the PR open, environment created, when the PR merge, environment deleted automatically and idle environment means it scale to zero. And on our stack we have Argo CD, KEDA, Istio and the Helm. Everything is automated and no manual, no task. Except for the tickets to a yarn. Those are manual.

Yeah, yeah. You still open tickets manually. And what we achieve here? So now we have a parallel development environment, so more than 200 engineer can work simultaneously. Uh it's cost-efficient thanks to KEDA, so we uh scale to zero and we are just pay for what we are using and we have a unified platform, no cloud run anymore. Uh and we have a faster delivery. So happy developer,

happy ops and of course happy uh CFO. Uh yeah, about those uh YAMLs that I mentioned. So in the uh right side uh there is a complete guide to preview environment on Kubernetes. I prepared like a full step-by-step. You can I'm sure you can also use it to have a preview environment on your uh environment and of course there is a killer KEDA tutorial that I prepared.

You can even start it and see how you can have a uh preview environment. And on this side we have like another use case of uh KEDA. We are using it for uh scaling our uh sinks uh based on the Kafka lags uh with KEDA which also save a lot of money uh for us. And yeah, so thank you. uh yeah, any question? We have a uh

microphone here. And yeah, please there is a microphone. Thank you for the presentation. Uh probably probably your lovely developer need some test data to test his new feature. How do you generate this test data in a new environment? I mean most of the preview environment that we have it just stateless. They are not related. We don't need to have any data to generate it for that those

specific environment. uh there is no a stateful like application that we need to uh or we don't have a stateful application to to say. Not as database? We have a database like at the like at those subgraphs or the graph QL they connected to the database, but they are just reading it from the database. They are not writing to the database. So uh a little bit different.

Sorry? If a new feature requires So if a if a new feature requires a new table, how do we solve that? I mean I didn't think about it uh to how to because we don't as I said we don't have this uh uh requirement to but uh one thing that we can do we for instance we can have a like a our data prepared as a like

a backup and then we can create a Kubernetes like a temporary uh in the preview environment and then inject the data to it. Uh but uh as I said for our use cases It's okay. Uh the URL? Okay, sorry. Okay, thank you. Sure. Hey, um how do you surface problems with scheduling? Like obviously the browser is just going to sit there and hang. do you want to

answer that? So in this particular case it's QA engineers facing. So they are trained to be patient. Uh But I mean when you because as as Jan mentioned the interceptors keeping the request in the uh I don't know in the in the it's all It's in a queue. queue and then when see that there is a pod it send it and you need to have a like

a application that is like a a scheduling plus the application be ready with very fast. So for our use case as you saw it was just less than 30 And of course in our CI we have a task to it wait for availability and see even the pod is available then it start on the integration test. >> Ah, so you're kind of lint you're effectively linting the

schedule ability >> before you get to this point. >> But but in general there are different techniques and heuristics that can be applied. Uh one of my favorite is we At At Keda we have other customers and we whenever the bootstrap time is too taxing for the user we recommend them not to scale to zero but what we call a shrink to zero which means you auto

scale with HPA and whenever there is activation deactivation you don't scale to zero you keep the last replica on a standby mode by VPA slashing the request and resources. And then when the first request comes and Keda activates the scaled object you give it a full resources and then HPA from then on. Hello. I think thanks a lot for the presentation. I think it was great. I

have one question around Why do you need GitHub or GitHub actions in the first place? Because now your development cycle depends like relies on GitHub and GitHub runners and GitHub runners are not alive. Sometimes you need to wait for minutes. Isn't it a toil? No I actually valid for our use cases because thanks to our platform team we have a very nice like a self-hosted a GitHub

runner that's always there are like a I think 10 to 20 a standby always. That's just getting the GitHub runner for us is just 10 seconds. Less than 10 seconds. if you check it in the documentation it it doesn't mean that you need to always use a GitHub runner. So or any any CI or >> It's more of a blueprint. >> Blueprint. So you can extend it

to any >> any use case. But why do you need CI in the first place? That's what I don't understand. >> So how you from your preview environment? Yeah. Okay? So in the pull request the code is changed. You need to have a you you need to build the image. But you you are currently deploying a deployment to a preview environment. Can can we assume that's a

namespace? It's a namespace but it's You need to deploy the container image there, right? If you provide direct access to your like the local environments of users, then you wouldn't need GitHub. You wouldn't need actions. Users could directly deploy to these environments. Yeah, absolutely. Like there there's multiple different ways how to like slice an onion or apple or whatever is proper fruit or vegetable to slice. But

this was one architecture decision, a set of architectural decisions because Trivago has their source code also in GitHub. It felt natural to put it all together with as minimal amount of third-party software apart from GitHub. And there is another thing. Our plan was to our preview environment like a whole process from the creating the PR to the ship it to the preview environment or the our staging

should be exactly a replica of our production because in our preview environment if I touch for instance Istio object Mhm. in the whole process I mean building rendering I can also test the infrastructure changes too. So that was the one of the our main goal too. Thank you very much for your time. Thanks for the talk. I'm curious do you run KEDA in production as well with

just different scaling rules? Yeah yeah we are using scaler. like a blog I explain so we have more than 60 Kafka things that they reading from the Kafka and push the data to the database but they need to be always there is the data would come for us not regularly. So then we are using the Kafka to automatically scale down or up those workers.