Cloud Native Theater | Data on Kubernetes Day: Tuning Elastic on Kube... Luuk Stolk and Artur Gromek

This talk covers the optimization of Elastic Stack performance on Kubernetes within ING's infrastructure. The speakers, Arthur Gromek and Luuk Stok, detail their experience with performance issues related to Elastic deployments and how they identified storage layer bottlenecks. Using tools like the Elastic stress tool and Portworx, they conducted tests to optimize replication settings and volume configurations. They implemented new storage classes and features such as auto journal and no discard options that led to significant performance improvements. The session emphasizes the importance of thorough testing and monitoring in a complex IT landscape, highlighting lessons learned during their long troubleshooting journey.

Good afternoon. Thank you for joining our talk on uh tuning elastic on Kubernetes. Um Yeah, Arthur and I both work for ING. Maybe Arthur, you want to give a >> Yeah. I'll I'll try to say a few words about myself. So, my name is Arthur Gromek. Uh I work as a DevOps engineer for ING KaHubs Poland. I've been in ING since 2013. In 2019, I joined ING

container hosting platform team. And together with Luuk, we've been building modern cloud-native infrastructure for ING Group. Luuk, maybe you're you're the talker. Yep. So, my name is Luuk Stok. I have been working for ING since 1992. So, quite a few years already. In various roles from the application developer to solution architect to indeed uh engineer in the cloud-native space. Um Yeah, we would like to take you

on a a quick journey starting uh with the consumer suffering from uh from bad performing elastic uh stacks on on on Kubernetes until the implementation of fixes that that actually uh got the increased in performance to to a to a reasonable level. But yeah, before we dive into the uh uh into the details of it, um maybe it's good to first first provide some some context. Um

First of all, about ING. Um yeah, as you can tell by uh by the facts and figures, um ING uh is is uh is a large um large bank operating on a on a global level in more than 100 100 countries. Um 30 million customers. Um 60,000 employees. So, yeah, very large. Um And yeah, with that of course that comes a uh quite a large IT landscape

at at scale um where indeed 1/3 of the people are working in the in ING tech among us. Yeah, a a large amount of the IT is running in a private cloud, the ING private cloud, short called IPC. Yeah, that's also where the main Kubernetes platform at ING runs. Um yeah, it started part of back in 2017. So, yeah, we were there from the beginning more or

less. Um and it's called the ING container hosting platform. It's a standardized Kubernetes hosting platform based on OpenShift. Um yeah, we've got two distinct offerings. One is a multi-tenant cluster setup for stateless workloads. And um also a range of dedicated clusters where hosting where we're hosting data services. So, for the for the stateful being in in banking, being a financial institution uh uh things like risk and

compliance are uh of of of of of a high essence. Um so, yeah, we provide a zero trust privileged access uh model. But which basically means that that no person uh can access the cluster with with right access. Um so, that's So, yeah, with that in that way we guarantee, let's say, compliance of of of a platform to our consumers. Yeah, our consumers are basically the application

teams with within ING who want to who want to deploy and run their their applications containerized. We provide it as a namespace as a service offering. So, consumers can can request a namespace on platform through self-service in an IPC cloud portal. And we provide tight integration with with Azure DevOps where with standardized controls consumers can can deploy their applications to uh to our cluster. Um yeah, some

facts and figures about ICHP. We started off as a um um yeah, so so as a small install base with few clusters, few nodes, a few parted name spaces uh back in 2017-18 and it's grown into uh a major platform within ING as you can thousands of teams, thousands of name spaces, nearly nearly 65,000 pods running on it. But also 100% uptime and zero security breaches which

of course is uh uh it's very important especially because we also host components that are part of of critical business change like payment engines uh uh mobile banking, online banking, uh etc. yeah, to dive down a bit a bit further, I mentioned earlier that we have a stateless and stateful uh very variant of our cluster um which I will try to explain a bit in the in

this slide. So, on the on the left-hand side you can see the the instance of for stateless workloads where we provide the Kubernetes platform uh with a name space as a service layer on it so that's those are uh configurations, security controls, APIs, etc. that we provide to make the platform compliant and consumable. And then consumers can request a name space via the portal as I as

I explained. And of course these stateless applications they uh obviously you allowed to use the thermal storage in the cluster, so they need to use let's say external data services for their consistency. that can be on bare metal or on VMs or uh appliances but or also containerized. Um and then that can be uh built on top of our cluster. And the Elastic Stack is is an

example of that. Where we still provide the the the the the base layer of of name space and service. Added to that also the the Portworx uh that we used for as a storage provider on our uh on our clusters. And then uh a specialized team uh is responsible for for developing the the data service. And that data service then becomes consumable for uh for the uh

IPC consumers. Yeah, Arthur, maybe you want to dig into uh Sure. Let me show you what our custom offering looks like at ING. Our platform is fully based on OpenShift and uh official Elastic Search operator. We use Portworx to deliver local persistent volumes to Elastic deployments. And all data ingestion comes directly from Kafka. We operate four large OpenShift clusters. In total, we have 44 worker nodes, each

equipped with 96 CPU and uh 1.5 TB of memory. We host nearly 800 active name spaces and manage 2.7 TB of storage. And roughly half of the storage is already in use. That's the brief introduction to our offering. Look, go ahead. Yeah, so what this picture is trying to show, I don't know if it's really readable for everyone, but it's a bit detailed. It's the different uh

different layers in the in the in the in the in the Elastic Stack with uh to begin with the uh the elastic layer where, um yeah, where sharding and replication is happening. So, that's the first layer of, let's say, some redundancy in in in storage. Um, the the second layer is the the the Portworx layer where we also have redundancy in place in terms of of Portworx

volume replication. Um, and at the bottom is the the the storage layer where we have rate five implemented. Um, yeah, I'm telling this because it's quite relevant for, you know, the in in the context of the the tuning we had to do. Yeah, and this this might seem a bit indeed redundant or over the top in in in a way, but yeah, we had good choices to

put the the storage redundancy in in on different layers. yeah, with that I guess we we can start diving into a bit more into the the the the tuning and the performance issues that we were facing. actually the first symptoms of a slowdown was were detected by Elastic engineers, Elastic team. They noticed that indexing rate performance started to drop, and they also observed more and more pods,

Elastic pods with right time out error. The error message is visible on the on the slide, you can see. However, the symptoms were visible in Elastic search pods, they weren't clearly reflected on the cluster metrics. So, yeah, that was quite tricky to us to to find the the what's going on there. So, but at that moment we decided to stop onboarding of the new customers. Yeah, and

initially, indeed, as Arty was saying, you know, there were symptoms of of bad performance, really bad performance, Um, which were noticed also uh visible for the Elastic team in uh in in in their stack and their layer and in and in their observability. But yeah, initially we we didn't see much. We we had looked at our dashboards and you know, it was busy but nothing uh no

anomalies, nothing major happening as far as we could see. So yeah, that got us a bit into uh Yeah, the the status quo of you know, where where to begin, where where does the problem lie etc. So yeah, we had to start digging a bit bit further. So by then we um we all among among others also got got Portworx involved to start helping us troubleshoot. Um,

yeah, and we also started to leverage other tools like uh file and IOstat and everything to uh to try to to find out where the the bottlenecks could be. And then at some point indeed we we started to see some uh some signs of uh of uh to issues in the storage layer. Uh indeed long IOs, volume latency, those kind of things. So we knew, okay, we

were we are possibly onto something. Yeah, and then the next phase begun begun basically. Uh we had to do a lot of testing. of course for one to you know, to get to the root cause of things uh further but also to start testing different kind of scenarios to see where we need to tweak and tune things and improve things to to make it better. Um, luckily

we were by then able to reproduce the the issue in an isolated test environment which of course to a large extent helps to uh to run all the different test scenarios and to try out different configurations without uh impacting uh impacting our consumers. the Elastic team team developed a uh uh a a testing uh within Elastic stress test tool. Um basically simulating data and ingestion etc. into

elastic to to make it a very representative way of of But at the same time we were also um we were also leveraging fio to do direct testing, synthetic testing on on on Portworx volumes etc. to see to see some effects and and issues. Um Yeah, we went through all kinds of different scenarios. So we tried testing replication one versus replica two. Um different rate setups, rate

five, rate 10 of course, different storage devices, SSDs versus NVMe. Um we were and are still running SSDs, so yeah, we also wanted to see the difference there. And we even started testing with different storage backends. Um we are currently running a software defined bare metal clusters with a local disk, but we also started to do some some synthetic synthetic testing with the virtualized on on some.

But yeah, that the these tests showed differences in in performance as you would expect of but also some unexpected results actually which I will begin to a bit later. Um but you also came to the conclusion that quite a few of these options are just not feasible, at least maybe as a last resort, but definitely not initial initially and not really feasible as as ways to go

because yeah, moving from rate five to rate 10 or re-platforming everything to from bare metal to virtual has such an impact which basically means that you need to build up your entire stack again and that would have a lot of impact and on our consumers and and on migration. So, then we decided to go let's first focus on uh changing the configuration of of our stack to

see if that would help to improve the performance. And that's where Arthur can explain a bit more. Try to explain more. Ah, it's your still your slide. Oh, sorry. Yeah. Yeah, that's that's an important one indeed. Uh the with the elastic stress stress tool that uh the elastic team um started to use, uh we were able to to to establish baseline tests, which of course are very

important to have to have baselines in order to compare different setups. Uh So, basically what uh what we did was um uh deploy one elastic cluster, um start ingesting, and ramp up the CPU usage to 100%, and then slowly start adding additional elastic clusters on on the same OpenShift um and to see what the effect of that was. And of course, we could see that the ingestion

ingestion rate would go down as we expected the performance degradation, but we what we also could see happening was that the um the CPU CPU consumption was starting to drop, which of course was interesting and for us another uh indication that that we need to have to we have to focus on the on the storage layer for for the optimization. I know what you mean. Thanks. So,

uh to find and understand the root cause of this slowdown, we decided to start a deep dive troubleshooting sessions with Portworx engineers. And based on the results of this session, Portworx provided us with a a bunch of optimization optimization recommendation that we should test. And to validate those recommendation, we prepared four new test storage classes, and we benchmark them against the realistic elastic search workload. the first

storage classes we built or we prepared was bare replica one and replica two in order to compare how the replication could affect the write And then we prepared the storage classes with a new features from Portworx. Let me go through these new features that Portworx introduced. So, the first what I would try to explain will be journal device with new IO profiles. journal and auto journal. What

is journal device? It's It's a additional disk that should be at least as fast as a Portworx storage pool. And it's really efficient with handling heavy sync IO workload and by journaling data and metadata. And auto auto journal profile is slightly different than journal because it can monitor incoming IO patterns and can dynamically switch between the different IO profiles between non-profile or journal depending what's better. And

a few words maybe about the no discard and auto first trim options. So, by default volumes are mounted with discard options. That's a good It had the pros and cons. Uh for example, you have if you delete large amount of the small files, then trimming operations will be triggered. And but frequent trimming operation could also affect system performance. To mitigate that, Portworx introduced the So, when you

when you mount it volumes with auto first trim, Portworx decides when to start the trimming operation. So, if can observe the application load. If application load is really high, then trim operation will have low priority. And yeah, that could help with a with a the performance issue also. So, as I mentioned first, we tried to compare replica one versus replica two. And uh uh replica one consistently

delivered better performance and lower uh latency compared to replica two and replica two suffered from cross node replication sync overhead and this test was one of the key finding that allowed us to narrow down the root cause of the issue. And then we compared the auto journal profile with journal profile. And both profiles provided performance boost, but adaptive behavior of auto journal was more effective for elastic

search write operations. That's was the conclusion for our test. And finally fixes. Bringing everything together, we conclude three things. First, uh bottleneck was located in the storage layer, not in the Kubernetes configuration, not in elastic search configuration. what else? We also conclude that not optimized volume options and replication overhead were major contributing factor to the slow down. And the new features from Portworx had positive impact over

write performance. And that's the snippet of the chat we had with elastic engineer and they confirmed that the performance after our changes nearly doubled. So, that was a good message from them. And final fixed implementation. Here's rolled out as a production fix. We updated Portworx to version 3.1.0 and our daily operator to unlock the new features. We also remounted all the volumes with replica one. Uh and

of of course we prepared the new storage classes. What else? We also introduced the auto first run options with no discard. We started to use journal device with auto journal profile to boost the And we also mounted Portworx runtime with RT ops conf high parameter. It's what it does is actually allows Portworx to consume more CPU resources and memory resources during heavy load. Uh but we stayed

with rate five, but yeah, look will explain why we stayed with rate five. Yeah. Yeah. As I mentioned earlier, when we when we tested all the different scenarios with different rate settings, different storage devices, etc., etc., we we of course saw differences that you would expect in in in in in in in performance. but yeah, what so monitoring and testing pitfalls, the um yeah, one of the

things that we uh were struggling with as I'm think sort of hinted on at the beginning already is to to actually troubleshoot the issue, you know, and you know, to to correlate uh the the the the the findings in the elastic stack in the in the elastic test results and correlate them to uh to events or basically non-existing events in uh in uh in the ICMP layer

or in the storage layer. Um yeah, and that that partly is is related to what I mentioned here in the second bullet point. Um yeah, on the granularity of of the metrics, we had a relatively low interval for the metrics and on top of that also all kinds of like aggregations. And yeah, that could that can of course lead to basically vital information missing like certain peaks

or or dips or whatever can I'm I'm not basically not showing up in the in the metrics. Um so yeah, we had to do some tweaking there, but of course on the one hand you don't want uh the what you don't want your monitoring to become that heavy that it basically becomes a let let let let's say noise on the cluster, uh, itself, right? Because, uh, then

it's going to start affecting the performance. So, yeah, it's it's important to find a balance there. yeah, and then with, yeah, let's say the unexpected results, we, um, so, one thing we found, especially with the synthetic testing, is that, uh, with Portworx replica two, uh, it was performing better than replica one, which, yeah, seems sort of counterintuitive. Um, but, um, yeah, that is, I think, partly explained

by some mechanism that, uh, apparently Portworx has in in place when you have got a replica two or higher or higher, that they basically they merge or coalesce, um, the the sync IOs, um, in order to to to increase the performance. So, instead of doing them all sing, uh, um, as a single write, they, uh, basically merge merge them. Um, but we what we also found was

that, um, in the in the in the in the context of Elastic testing, um, a replica one was actually faster than replica two. So, yeah, that that's an interesting finding that we never really got to sort of fully understand. Um, but, yeah, something similar we also saw with the the RAID 10 versus RAID 5. I mean, RAID 10, again, with the fio testing, RAID 10 was, uh,

was significantly faster than RAID 5. But testing it in in the the the complete Elastic stack context, um, uh, that difference was actually not that significant. So, um, yeah, I guess, you know, when you're testing in a broader context, all kind of other variables come into play that can affect the the the the performance in a different way. So, um, um, but, yeah, the actually, the the

last one was the most interesting one, um, where we found that we need also with a synthetic testing the the the virtual platform with with a sound storage was actually faster than bare metal with with local disk. Again, we we didn't we didn't pursue that very much, but we thought it was a very interesting interesting finding. But yeah, we we didn't want to re-platform, so we didn't

look into that further whether it should the whether the performance would also remain higher if if it would be on the fully elastic load etc. So that's why we we sort of discarded this, but yeah, it's a sometimes it's it's not what you to be. Yeah, and I guess that that sort of wraps it up a bit. Yeah, is the journey more important than the goal or

the destination? Um Yeah, we um Of course, yeah, we surely could have done without all the the time and effort and energy that it was needed to to get to a level that we that we wanted to. And as you can tell by the timeline um we've been on and off working on this issue for for a long time, basically nearly 2 years in order to get

it all back up and running again. And to to fully open up the onboarding again for the consumer, so that had a major impact. But But yeah, as always, you know, along the way you also learn a lot. So yeah, in hindsight of course, it's we've learned a lot about the elastic and the Portworx and how how yeah, this the performance tuning works works together. Um yeah,

and that basically wraps it up. Yeah, so yeah, ING's got a booth at booth 799. Artur and I there tomorrow till 11:30. So, yeah, if you have further questions or other interest and yeah, come and find us at the booth. Um, but yeah, that's it. Thank you. >> Thank you for listening.