Banking on Reliability: Cloud Native SRE Practices in Financial Services - Clément Nussbaumer

This talk explores the reliability challenges faced by Post Finance, a major bank in Switzerland, particularly in relation to their Kubernetes infrastructure. The speaker, a systems engineer, discusses the implementation of Site Reliability Engineering (SRE) principles, highlighting the importance of Service Level Objectives (SLOs) which were established to reduce failed requests and improve overall service quality. He details their transition to a stacked etcd topology to enhance API server performance and outlines the insights gained from monitoring tools and techniques, including load balancing and graceful shutdown procedures. The speaker emphasizes the need for continuous end-to-end testing of services to catch issues before they impact users, and he shares a real case study involving unexpected 502 errors, attributing them to mismatched timeout settings between the ingress controller and backend services. Ultimately, he advocates for the use of open-source tools and the need for precise monitoring and testing to ensure system reliability.

Well, welcome everyone. Thanks for joining me to this session on reliability topics in a Swiss bank. I'm glad you're here to this afternoon. And uh let's get started. Let's start with a few words about me and some context about the company I work for. So I'm a Swiss systems engineer. I work for Post Finance, a relatively large system systemic bank in Switzerland. I happen to live on

a farm, which is a pretty fun fact. We are where I try to implement lots of cloud native solutions for my wife and her associates. I play music and I'm a young father and uh for post finance it is a as I said quite big bank in Switzerland where we operate on prem kubernetes clusters vanilla kubernetes clusters I talked about it last year uh we were migrating

from cubadium to talos it's still an ongoing progress and um yeah we we are building most of the solutions around kubernetes by by ourselves as you will see in this talk and yeah we are a bank. For us, a failed request means that someone is at an ATM trying to to to get some money or trying to pay in a shop and it doesn't work. Perhaps there

is retry. Per is some delay. Perhaps they are stuck there for 5 minutes. So, we really cannot afford to have failed requests. That's really quite bad on our customers. So, that's why we have uh implemented lots of solutions around uh site reliability engineering principles. And the first one I want to cover is service level service level objective. So back then we didn't have any SLO service level

objective. We just had some feedback from our users. So the application developers for the finance for example or other applications for the bank or the other developer of the platform and it was like oh well this morning did you did you see it was pretty slow. K9S feel slow. I did a cube cut get it took perhaps two minutes to to to get a response and this

was temporary and we basically ignored it because if you do not have really a a target you just ignore the the bad uh services that you could get. So this was pretty subjective and we ignored those uh this bad quality of service and so someone suggested that we take a look at service level objectives which is something really well defined in the Google's sur book and which

I will not cover in too many details here but the idea is that if you provide a service I really encourage you to define an objective for that service for example for the API server one of one of our objectives is the availability we want that less than.1% of the request request fail on our API servers otherwise it's going to have a bad it's going to be

bad customer experience for the application developers for example we also put uh we also implemented other types of SLOs's but that's the idea you set an objective on one of your services and then you can use a tool like for example slo to help you define all the metrics recording rules um that you need to really implement this objective and monitor it uh in the long run

and so this this tool tool here. SLOs is an open source tool which are linked down below. And the idea is that you have your the total query. So all the requests that arrive in your API server. You also define the error query. So all the ones which did not go according to plan. And then it will generate a nice dashboard like this one where you can

see well when it's green you're happy. When it's not well you know that you have something to fix and you the really interesting part is that it lets you know exactly when you had an issue. And so it also lets you know it it provides some a timeline for correlating for example a cluster maintenance an upgrade of your API server or something like that with um a

broken SLO so with uh your indicator going really in the red and that's when we started to do that that we noticed okay every time that we make a cluster upgrade for example we have really bad latency on the on the on the read read only for example SLO and so that's when we started to reinvest investigate that. So this was the motivation for looking at our

HCD topology. So API server and HCD topology back then three years ago we had something like that. Well this picture is perhaps more appropriate. Every API server was connected to every other HCD uh in the cluster. So while this sounds good the idea is that well if an HCD goes down every API server is going to balance across the two other HCD in a three node cluster.

The ID sounds good, but it also has the implication that every time you perform maintenance on any of the control pane nodes, the other two ones will also be affected. So, we implemented the so-called stacked HCD topology which is documented well on community documentation where it's a bit simpler and it already started to improve the situation with regard to our upgrade. So in that in that topology

all of the all of the API server they just talk with the HCD which is collocated on that node which is also a bit faster because you don't have to also go through the network to the other HCD HCD servers. So this was the first fix that we implemented and we thought okay we're good with this one we should not have any more SLO bridge uh when

we when we perform the next upgrades but it was not enough obviously so we thought okay perhaps we have to migrate or transfer the the leadership of the HCD server away before perform maintenance. So HCD as you might know distributed database we have one leader and the other ones are followers and we thought okay when we perform a maintenance on a node we should likely move the

leadership away from that node if that node is the leader. This will spare us the leadership election process and perhaps we are going to improve the quality of service a little for for users. We tried that. It was slightly better but it was still not the issue. It was not the full solution. And so that's when we actually went back to the graphana dashboards of our control

plane and really started to look what is happening on our API server. So we have three API servers, they all talk to the local HCD. So now it should be good. But somehow when I when I stopped this first API server, the other two which should be ready to take the traffic, they are really slow to process this traffic. And so we noticed actually that one of

the API server was always doing all the work. The other two were just having fun. one of the API servers was like had four to five times more CPU and memory usage. And so it turned out that the the the load was not properly balanced. When we when we investigated a bit further, we thought that we had a misconfigured load balancer. So we have a layer four

load balancer. So just TCP plain TCP load balancer in front. We had run rabbit configured. So we thought perhaps there is an issue here. We tried to change the parameters of the load balancer in front of the API servers. We set it so that it would send the traffic to the least uh loaded cluster. But that still didn't help. We still had one of the API server

doing all the work. The solution was actually that you have uh with HTTP2 clients, they would open a TCP connection and it would stay open forever and then all the requests would go on the same connection over and over. So during a maintenance we have for example the third API server which would be u the first one to be ready for example and then all the cublet

would connect to this API server they would open a TCP connection the connection stays open forever and then all the requests all subsequent requests will only go to that API server. So whatever however you configure your load your your load balancer those requests will land on that API server say and so this one will be having a lot of load and when you take that one out

the other ones the issue is that they do not have any cache fil the caches on the other load balancer are not filled because they didn't have any work to do so one API server has 200 megabytes of of RAM or something like that and the other one had 2 GB because it had filled its caches with all the communities resources and there's a fix In the

API server flags, there is this small go away chance which is a fraction that you can set on your API servers and it will um it will set the fraction of frequest that will receive a go away frame go away. something from the HTTP2 spec which I put on the right hand side and the idea is that you you issue a lot of requests on your API

server and at some point the API server it decides okay it's your turn you're going to receive a go away so I will keep finish I will finish processing your last request but I will not process any new request from you you will have to establish a new TCP connection so you have to reestablish a connection to through the load balancer and this might be on the

same API server or this might not be the case and so as soon as as you start setting this flag you wait a few hours and then all the API servers start to have the same load. The load really balances at that moment and that's when we really solved the issue because at that point if we would taken away one of the API server the other two

API server were ready to do the work because the the load was balanced the cash were filled and nobody noticed that we were doing an upgrade. So that's that's the end solution. All of that's thanks to setting SLOs's because otherwise we would just have continued to ignore it because it wasn't that bad but it wasn't good. So that that is the first takeaway of this sess session

I guess. Let's move on now to open source monitoring tools. So at post finance we develop a lot of of tools. A lot of them are well inhouse but all of those which we can open source we try to open source. Today I want to talk about two monitoring tools. More specifically the one on the right we use a bit less because we in incorporated some of

its features in cubers but hosters basically it was here to prove that it was always DNS because we all know that it's always DNS and cubes uh cubers it is a demon set which does uh continuous health checks in your cluster so it runs on all nodes and it performs a series of checks. Uh if we start from the lower layer, it starts with checking that it

can access the API server through the API server direct IP address. The next check it does it also accesses the API server but through through the DNS address of the API server. So here if the first one so if the IP address of the API server check is green and the DNS one is not green then you know that you have some sort of DNS issues at

hand then it also checks the the service of cubes itself. So through the the yellow check it will make sure that your CNI is working properly and finally the the the kind of end to end test it goes through your ingress uh and back to your service and back to your pot. So that's the really the most important check for us and we we really rely on

that to to make sure that the the traffic is incoming to the to the cluster and next to those four types of checks. So servicing API server DNS IP we have the neighboring neighborhood check and the idea of this check is that cubers is going to query all the other cubner spots to to know the state. So it will record the latency and it will record any

error that happens when it queries those nodes. So so that we then uh can identify if a node is starting to be a bit slow we can really identify and pinpoint and say okay this node has an increased latency perhaps there is something wrong with the hypervisor. This is the type of stuff that happens more often than you would seek and perhaps you have to move the

node away from the hypervisor because it's too loaded. And so um yeah those checks we perform we implemented cuminus with with go and we use the http trace instrumentation which means that we also have all the http trace events available when we when we perform those requests be it for for all the request types that I showed before and so it means that we know precisely how

much time it takes to start a DNS request how much time how much time it took to get the response so that's a DNS don event and so on and so forth so we can really detail the all the all the all the connection stack we really we really know precisely how many milliseconds it took for that and so um also something that we added is that

the when there is an error we also record that as a metric and so if there is a DNS don error for example we know that there is error with DNS which could be that uh there is too much traffic on the switches and network congestion and the UDP packet was dropped or that the DNS server was just misconfigured at that moment And so I said that

we were having uh that we were crying that every cubess was crying every other cubess. That was true and that was an issue which was brought about by the community in the issue number 55 which was I think raised from some by someone from from Zalando if I recall correctly and so uh they didn't directly provide a fix but they started the discussion and they encountered the

bug before we did because they had much larger clusters than we had in terms of nodes and so they had clusters with thousand nodes. So thousand nodes making a th00and requests to every other node it has 1 million requests every time that uh well for the for the frequency that you have configured for typically 5 seconds. So that's a lot of load just for a small monitoring

tool. So we we thought to ourel that we we needed to inst uh after much discussions we implemented something like a hash ring. So the ID um here you have all your nodes here for example let's say we have 15 nodes uh this here is shows the linear order what we do we hash the name of the nodes and then we reorder based on the hash of

those nodes that's that gives us the hashed order and then if we check for example node one which now landed here it's going to check the five next neighbors so if click on node one we see okay it checks the five next neighbors if I go back to the linear order it appears randomly distributed And through this hash ring the the interesting thing about that is the

metrics are going to be stable. If you restart cubnos it's going to pick the next five nodes uh with a deterministic deterministic random order. It's not going to just pick new random nodes uh at random because otherwise it would explode the cardality of your metric server and your observability team will come at you. Lessons learned. So that's what we did. So basically we went from something like

that where every node was quering every other node and well it's n squ so the more nodes you add and the more it grows to something like that where you can define okay just just query five other nodes. So this one will query those this one will query those and in the end everyone is covered. We precisely know when a node has a latency issue and we

do not put that much load on the on the metric or primeus or victor matrix in our case servers. So that was uh something that we that we got to to do because of a community issue that and then we solved it and afterwards we we benefited from the solution that we had implemented. Something else with regards to cubners, we we also implemented an SLO for cubn

and we noticed that when we were restarting the cumn demon set, we were always losing some some HTTP requests to cumulus and it didn't make much sense until we noticed that u we had to configure something which coordinates calls the lame duction shutdown the ID or graceful shutdown the ID when we receive a termination signal from your from your hypervisor or from cube cublet or whatever you

should not directly exit and stop and stop serving traffic because uh the problem is that engineext or the CNI they will get the information that you that the pod is going to be deleted at the same time as you do. So there will be in some occasions some packets that that will be routed from the CNI or from engineext or from get API whatever implementation you use

to your pod when you at the same moment as you receive the the termination signal. So you should wait a little bit before uh before just uh stopping to serve traffic. Typically in our case we wait five seconds and that's that's completely solved the issue. So graceful done was also a learning that we got thanks to the thanks to the thanks to implementing SLO which brings us

to part three of this talk the continuous end to end testing. So we are providing a lot of services on our clusters like namespace self-service provisioning people can order PVC they can order network policies rules automatically and all of that is stuff that can break and that breaks regularly um and we don't want our users to be the end to end test. We don't want them to

open an issue because first of all it takes time to for us to investigate that and they shouldn't have to suffer the the pain of uh encounter encountering a broken service. So our approach that is that we we implemented end to end uh test in Golang uh with something with by reusing the end to end framework from the community 6 group and the idea is that we

have an end to end test in go so go test and then we define test in go which perform actions such as creating a new deployment such as uh creating a PVC mounting it writing data to it making sure making sure that your CSI provider is working etc etc And through doing that we we we are able to know precisely if a change to for example the

if we update the storage provider and we see the end to end failing one hour later we know okay it must be related to that. So while we cannot open source directly our own test because it wouldn't make much sense in the in the in your case what I did is that I implemented uh some open source end to end test which you could fork and implement

for for yourself and the idea it just runs those four tests here basically creating a new board creating a new PVC writing data to it etc etc and so it runs as a con job inside the cluster it runs in my case every 15 minutes and it pushes the the results to an OTLP server in my case it's and Victoria metrics. So then you can build a

nice graphana dashboard like this one here where you have all your tests visualized per environment per cluster. So here in this case we had one cluster which had a failing service. It was something like the namespace creation was not possible for new users. So we just had to investigate it and and fix it. Yeah. So again another meme test everywhere. That's what we like. And this brings

us to the to the fourth part of this session, the 502 mystery. So uh on the link here you can access the a chat room which will be moderated by Sebastian. Thanks Sebastian. U and so you can the idea is that I will present you with the evidence that I had a customer open an issue a 502 they faced 502 errors and if you don't mind I

would like you to to submit your idea in the chat as to why the the cause of this 502 errors could be So this is this is hosted on my home lab at home. So we'll see if it breaks or not. That would be fun as well. So they were facing an issue that they were having they the customers were u encountering five or two errors in

the browser. It was really not often a few requests per million uh requests. So 8 to 10 failures per day on a service with two million requests per day. and it was uh it was impacting a Tomcat based ENOS and so we we started to investigate. So the the 502 errors were uniformly distributed across all engine. So we thought it must be something else. There was no

real real pattern in time in user agent in endpoint or client. All the application ports were healthy uh in the logs. We couldn't notice anything. The application had not been stopped at that moment. the CP CPU CPU limits was fine uh or actually removed. So that was not the issue either. No issue with network policies. We tried to reproduce the bug a lot of time with Ksix.

We really sent crazy amounts of of requests to that service. We never we were never able to reproduce the 502. And so the only correlation we had is that the more requests there were the more uh of those 502 errors we were getting. But it was kind of a constant rate. So if uh some people are there we have some ping some high the English controller. Yeah

that's that was our first ID HTTP2 connection timeout uh idle connection timeout this could be something as well but it was something DNS. So DNS we can we can exclude. It was not DNS. Uh dropped connection from the gateway race connection in connection close. Actually that's the Looks like someone read the abstract of the of the talk. That's good. Um yeah there are some funny replies. That's

good. That's what I expected. Um it's so connection timeout idle connection timeout. It's also the same one. It's not about directly HTTP2 but the issue is the following. So let's go back to the slide. Um you were quick to find it out the so we continued the investigation. Well you might not need this hint here but ignorance engineext well we all know that it's been archived but

still uh when when it was logging the when it logs access logs you will have a label in your logs which contain the ingress name and the name space. So it's quite easy to to search for an ingress and to search for errors. That's what you would think at least. But actually if there is an error you you're not going to contain it the logs are not

going to contain the ingress name or the name space or whatever. It's just going to contain the FQD and the host name of the service of the ingress. Don't ask me why but it it lost us a few hours to find that out. And so once we found that out, we found that out we we saw that uh the error was actually the following one upstream prematurely

close the connection while reading response from the upstream which doesn't make much sense. But after like uh working on that for a bit more we know we knew that uh uh it was really an issue between engineext uh and the back end. So then the Tomcat application at some point enginex was forwarding a request to the Tomcat application but the Tomcat application had closed the connection and

so uh I I built or CL helped me build a visualization of what was happening. So the perhaps before I start the visualization I can explain that engineext had a connection timeout of 60 seconds. EngineX when it opens a connection to a back end for example Tomcat or any other application it expects that backend connection to be open for at least 60 seconds which means uh that

if that other back end is not configured with the same timeout you will you might encounter the race condition that we'll see here Tomcat by default is 20 seconds so if I start the thing again a bit slower perhaps uh we have the first get request to the back end and then on the back inside a timer starts a 20 second timer starts. After those 20 seconds

the the connection will be closed by the back end and if by mis luck or Murphy's laws uh something like a new request arrives from the from the ingress engine X which is going to happen anytime now. If a new request arrives at the same time as the as the Tomcat had sent the the fin packet so had closed the connection uh you will encounter this race

condition. So it's really rare that's why it was hard to reproduce when you knew that we were able to reproduce it with K6 by scripting some some tests where we would um incrementally increase the the time or the add some sleeps between the requests that we were making. But once you know that uh it makes it makes sense that you receive those 502s because it's really that

you have the HTTP request coming in as at the same time as the fin or reset and next to that reset packet from the from the back end. And while this is nice animation, I think this sums it up even better. Yeah. So the fin was there before the five before the request and that's why we were getting those 502 bad gateway errors. And in the end

the fix is really pretty simple. All you have to do is increase the the tomcat uh keep a lifetime out to something which is bigger than ingress engineext. And you might think that it's only impacting engineext but actually it's really not the case because uh you you should check your new gateway implement gateway API implementation. Uh I think has something like 60 seconds as well for or

10 seconds I can't recall but you should check that as well. If those timeouts are mismatched, it will happen as well if you have a lot of traffic and if you care about every request that some of those requests will end up in a 502 or some other error code. But the same error can happen in any any situation. It's not only about ingress engineext and the

combination with which brings us to the the takeaways of this session. So service level objective I wasn't convinced by them at first. I thought that it was not really needed. We had some dashboard it was enough but it was really the driver the motivation for us to transition from something like okay the cluster feels slow today to something like during the last upgrade at 9:45 we really

had an incident and it was correlated with the upgrade of API 7 number three and so without SLO we would still be blind about those or we would keep ignoring those issues then uh when it's possible I really encourage you to open source your tools it really helps the community but it also helps you as as we've seen here in example and this keeps repeating it's someone

opens an issue you start a discussion or they just implement for example a pull request with the helmchart for your application and then you don't have to do it yourself so open sourcing your tool is really something that I can only recommend then uh with regard to test end to end test inside your cluster which test all the services that you provide for us this is really

a gold mine it it's it's it's uh it spares us the effort to to to have or it spares our customers mostly the the pain of storage provider or broken self-service. And finally, uh, every error matters. We could think that five out of 1 million requests which fail or end up on in a 502 is not something really important. But when you really dive deep into the

subject, you you notice that you learn something about it about it. And that's actually what we found out what we found about with the with those uh timeouts which were not aligned. we we can apply that to any other uh situation in the future as well with with regards to our new getway API u implementation that we that we picked. And yeah, that would be it. Uh

here's the again the same chat room. If you have questions, I will take those the questions through the chat room because I've noticed that nobody goes to the microphone when we ask questions. So I'm really happy to take some questions there and otherwise we can find me at the end of the of the talk. Thanks a lot.