Building Cloud Native Culture in a Bank: With Open Source as a... Marcy Paramonova & Stéphane Cusin

This talk explores the journey of transforming a private banking institution, Pictet, into a technology-driven organization utilizing Kubernetes and open-source solutions. The speakers, Marcin, a DevOps engineer, and Stefan, a senior DevOps engineer, emphasize the importance of culture and collaboration in successfully adopting Kubernetes at scale across Pictet's operations. They discuss the evolution from traditional enterprise technology stacks to modern infrastructure, highlighting the challenges of gaining trust in an open-source environment and the need for shared practices. The session details how finding common standards and engaging teams through collaborative practices, such as 'genius bar' support sessions and internal tech talks, fostered a sense of community that propelled their Kubernetes platform's growth and adoption, resulting in over 1,900 clusters with zero application changes during migrations.

Okay, hello everyone. That's very nice to see you all. So, we should start our talk. Welcome everyone. 1900 Kubernetes clusters. Three hypervisor platforms. And zero application changes across every migration between them all. Nice to meet you. My name is Marcin. I'm a DevOps engineer and product owner and especially Kubernetes administrator at Pictet. My focus was and is uh the culture around the platform and the ways of

working around it. It's culture, it's people, it's teams and the shared practices that goes with Uh today, uh I will explain to you how we get to those numbers. Thanks. And I'm Stefan, senior DevOps engineer and Kubernetes platform owner. Uh I'm I'm Pictet for 12 years and last 7 year running Kubernetes in production at scale. So, together we are going to tell you one story from two

angles. The platform and the culture that made it possible. Marcin, let's start with where we come from. Sure. We come from Switzerland and we work in a private Swiss bank. Pictet is a multinational private bank that was founded more than 200 years ago and is headquartered in Geneva. It offers wealth management, asset management, and asset servicing to private clients and institution. Private banking is um culturally very

regulated environment, even we can say by its design. And very conser- by culture. So, not exactly the place that you would expect to find a team of engineers shipping a Kubernetes infrastructure like a tech startup, right? So, we have a Pictet tech division. Within this division, we can count 1,500 engineers. And we are focused on building business-critical applications and running them. We are distributed between three tech

hubs, which are in Geneva, in Lisbon, in Luxembourg. Our mission is simple to explain, harder to execute. We are shipping infrastructure to almost all the product teams within that organization. So, where do we sit in all of that with Stefan? Um there is a team who is responsible for internal developer platform. That's what we do. That means Kubernetes at scale, automation by default, and to be honest,

in our context, our product managers, product owners, they are free to choose their hosting environment. They can choose their hosting solution. It can be cloud, it can be on premises, it can be on virtual machines, or Kubernetes clusters. So, I should add that we're not only shipping infrastructure, but we are creating a platform that chooser the users they choose to use. Our story covers uh roughly 15

years, 15 past years, and during this time, technologies has evolved, and we had to make a lots of choices. So, at some point, the question became, how do we modernize within a private bank while keeping with industry standards? And for that, I'll pass it to Stefan so he can introduce you our compass during our journey. So yes, thanks. The answer for us was the open source. Not

because it's free, not because it's trendy. So because open source give us something we desperately need a shared standard. A common language between teams, between vendors and between tools. But pointing in the right direction in a banking environment first required one thing. The trust. So when we started, open source was an obvious choice in a bank environment. Was not an obvious choice. Um there was a lot

of questions about support, about accountability. Instead of trying to convince people with theory, we started running real workloads. Over and over time, something changed. Trust was not something we decide in a meeting. It was something we build by operating the platform day after day. And slowly, open source stopped being seen as a risk. It became the foundation. But trust alone wasn't not enough. We still need a

consistent way to decide which technology to adopt. Not based on hype, based on criteria. So quickly, I will show you the simple framework we have really to choose the technology. So every time we consider a new technology, we ask yourself five questions. Is it actively used? Sure. Is it open or controlled by one vendor? Is there a real ecosystem around it? Can we audit yourself? Do we

have the skills to do that? And it is well documented. So oops, sorry. So this five filter give us a shared decision framework. Not a guarantee, but a discipline. But technology decision alone were never going to be enough. What do you think, Marcin? I think you are completely right. And everything that Stefan said before um is a framework that we have chosen. But great decisions, uh for

example, choosing a technology, they scale. So, that's the complexity. More teams choosing to use the platform, more use cases we discover, more dependencies there are. So, at this point, we have the technology, we have the framework. But what we understood and that platform don't operate in isolation. What we understood is that the platform is a collaboration systems through which a lots of communications are going through. Developers

and product teams depend on a platform team. Platform team depends on application teams. And both need to go forward. And for that, they need shared standards. at this point, we needed to find them understand how do we create them. And we were majorly inspired by open source. So, not only by technology, but also the ways they scale the technology. Kubernetes, Argo CD, major projects like that are

created by a collaborative effort of many contributors, of many um teams, companies, ideas, and interests. So, at this point, our journey of transformation needed also a cultural part. So, let's talk about it. So, as you know, uh open source uh users not only consume the technology, they contribute to it, they create communities around it. uh they exchange about the the technology. So, our interpretation of, uh, one

of those exchanges was our genius bar sessions. At the beginning, we were the first team who created this, and we wanted to step back from a usual supporting Jira tickets and offer something new, fresh to our users, and open support sessions when they can come and we could solve their problem with them. And over time, a lot of people have seen the interest, so we were joined

actually by a lot of infrastructure teams, cybersecurity, network, identity, cloud teams, and today we sit all together and we, share solutions and we help to solve problems collaboratively together. The tech exchange. We, uh, created a lot of possibilities to, uh, knowledge exchange internally or externally. We connect engineers in, uh, Geneva area. We, uh, share our expertise, and we're trying to solve the problems faster Um, also, one

of the, technical exchanges that we really appreciate and one of the thank you that we want to say from the stage, I think it's for our friends in CERN, our neighbors in CERN because they're also in Switzerland. Not only for the technical exchange, uh, with their Kubernetes team, OpenStack team, but, uh, especially for sharing their mindset because it's genuinely very inspiring for me. We're also, uh, within

this technical exchange are encouraged now to give conferences, which was not very obvious for private bank before. And this is very important and it matters because it's one of the ways to give back to community of open source, which is important for us. We also try to create some, uh, events for the recognition of adoption of our platforms. We introduced super user rewards for the users who

actively contributed to improving our platform. Well, their feedback sometimes hard to hear for us and a little bit painful actually helped us to create something more beautiful and something better. So we're grateful for that. we created some rituals that are unofficial in meetups within our organization or after work events. And of course something more structured for our users when they can come and they know the news

of the platform, new features, and especially they can help us shape the road map and see the demos. And sometimes we gather more than 150 people on those meetings, which makes us happy. Learning. This comes hand-in-hand with everything that I said before. I think we have to say that in our job it's very important technical skills. And we're really lucky and we have to admit that that

we have a lot of resources that we can use to upskill technically. Access to platforms of learning, hands-on, certifications, this is very important. What I find more interesting in that as engineers we also can benefit within Pictet to upskill our soft skills, which I think helps globally everybody to work better together. not only there is traditional ways to upskill yourself in learning, but we see something that

is community driven. For example, people create clubs or communities around technology and they maintain it. And one of the examples that I particularly like in our environment is today's tech talk. It's an internal conference that our colleagues do, a small number of them, on their free time and they inspire all the bank every second week. They bring external speakers and they organize it on their own time.

So, if you have any good ideas that you want to pitch me, I'll be happy to hear them during the KubeCon. Don't hesitate. And I really have to say that my colleagues rock. What I want to say about the ownership is that this was a crucial change that really changed the engagements within the engineers. We are not only trusted to execute tasks, but we are trusted with

making real decisions and making real impact. Experimentation-wise, we are always invited to experiment and feeling of this is very great. We even a competition yearly where everybody can submit their ideas and the best one, the one that wins, actually gets implemented. So, a lot of our improvements came from bottom up. And collaboration-wise, we really feel and we really value this opportunity that we have to organize within

the team in a way that is comfortable for us and that we find valuable. As with other teams, we also can create events, rituals, and everything that helped us. And as long as we can explain why it's valuable, well, usually it gets accepted. It sounds small, but it really changed the whole vibe. I can continue with engineers not specialists. It's not about replacing specialists. Of course not.

They are valuable and we'll need them. It's about complementing them in a way that it feels that we value people who can understand the systems end to end and as a whole. As well as open source, um it didn't change It's It didn't change only our technical sta- stack, but it changed the way that we perceive and value human skills. So, what we see is that when

you use open source with open standards, your skills become very transferable. So, you don't relearn something from zero. You build on a shared uh foundation that's shared across the open technologies. And I'll finish with engineering mindset. I think it's the most important thing. And what is important is not what you know today, but your capacity and ability to tame new technologies, to learn them, to execute, and

operate them. And basically speaking, being an engineer is about problem-solving. Being an engineer is to be passionate about that. And being an engineer means to share your passion about problem-solving. So, this is what uh our cultural change uh made for us. And I'll pass it to you, Stefan, so you can show how it looks in production. Yeah. So, thanks, Marcy. So, open source is not just a

software choice for us. It's a commitment. If you only change the tools, but not how people thinks, collaborate, and build, you will hit a wall. A tool don't scale in their own. The culture has to scale with them. What Marcy described now, the genius bar, the knowledge exchange, the craftsmanship mindset, these were not surrounding decorations. That was the platform strategy. Without it, the Kubernetes adoption would have

stayed. And what you are about to see, 1,900 cluster, three different hypervisor, zero application migration. That doesn't happen just by deploying Kubernetes. It's happened when engineering culture and technology evolved together. Let me show you the technical side of that journey. So, back in 2010, we had a very standard enterprise stack. The GTUI application server running on IX Unix and IBM power. Nothing unusual. But even a simple

version upgrade could take years. Sometime we had to run the entire stack in parallel until every single application was fully migrated. So, changing anything means a project, a teams, a schedule. Uh nothing simple. Not just a simple change. So, it's very complicated. So, the question became how do you evolve a system like that? The first source bit was the application layer. We moved to Apache Tomcat running

separately on virtual machines. aligned better. Uh but sorry, but because it aligned better with how we manage risk and isolation between application. And that first decision taught us something important. Community plus engineering skills that was make it scale. And we prove it. 1,700 Apache Tomcat instance running on The team did an incredible job to host But managing all this was not easy. It was always It was

not always predictable. Uh it was made with Puppet, but we have a lot of manual work, a lot of snowflake, more pet than cattle. You can probably guess where we are going next, sure. The Kubernetes. And that when Kubernetes changed everything. Not just for the platform, but how we think infrastructure? It gave us a real abstraction boundary. An application no longer need to know what OS is

running on. It no longer need to know which hypervisor is underneath. It that decoupling that was made every migration possible. On top of that, we added Argo CD as a GitOps engine. Every cluster, every deployment declared in Git. No manual step, no snowflake configuration. And once everything is defined in Git, everything become predictable. So, from there, everything is driven through Git. That's the standard way now. Just

one move commit, and your application change of running target. You move the folder, you move your configuration, then the target Argo will redeploy it on the new on the new So, the team can do it as a self-service. They can move, they can schedule their their change. Or, as an infrastructure team, we can also um do the migration for the for the application teams. And everything is

changed, everything is visible in Git. So, they can review the change, and we can do it together. So, at the end, that's this is a journey from a bank running application Tomcat to a platform with true workload portability across data center. But, the biggest change was not the technical part. We move from a ticketing-based model to a self-service platform. From waiting to deploy, from waiting for operation

teams to a platform engineering. And once you have that, the right model, the right abstraction, things start to scale. And this This what it look like today. We run over 1,700 communities cluster on premises on our private data centers. Many of them are single node. One cluster per application. That's our isolation models. On top of that, we operate around 200 cluster on AWS following your software factory

for the developer platform itself. And beyond that, product team can also provision AWS account directly. Fully self-managed outside the platform we provide. Marcin, do you want to show how we got there? So, now you see the curve of adoption of At the beginning, you can see that it was kind of slow and it's understandable because version one of our platform was actually deployed on dedicated hardware on

moonshots and was more made as a project rather than a product. So, technology was there, but the day two culture mindset was not yet. Actually, the infectious point of adoption came when we started to put as much energy into the community around with users as much as in the platform itself. So, in a few years, we have acquired almost 2,000 clusters and going. So, to close, we

want to share with you three things that we would tell our past selves. please start. Thanks. So, the the the first thing, so open source was never a directive from the management. Uh it was never a dogma. It was never a told us you must use open Uh it was a compass, something that helped us orient in decision. Uh when when Kubernetes appeared, we recognized it immediately.

A strong community, transparent governance, and real production user. The compass pointed there. We followed, sure. And now we are watching OpenStack the same way. Um the key is that a compass doesn't force you in one direction. The It helps you recognize the right opportunity when it arrives. The The The second point, early in the journey, we built a first version of Kubernetes on bare metal. We treated

it like a project. We delivered it. We got a good adoption. But the day two operation was incredibly hard. Uh not because the technology was broken, because we had we hadn't built a culture around it yet. So, nobody knew what to call. Responsibility were blurry. Team was consuming the platform without really contributing to it. So, incident Every incident uh feel like a fire drill. So, it was

complicated. So, we realized the deploying Kubernetes was the easy part. The hard part was building the community uh of practice around it. Uh the shared knowledge, the ritual, the ownership model. Without that culture, no platform survive its own success. And the last one, the most actionable mission we have, you cannot wait for culture to emerge on its own. Uh you have to design for it. If your

team complains about being interrupted all the times, don't blame the other team. Ask yourself, have you defined a structure for people should reach you, how people should reach you. A genius bar session, an on call rotation, a shared documentation space. If you haven't built a structure, you can expect the behavior. something we tried didn't work. We keep it the one that did. But the act of trying,

of deliberately designing how team interact, that's create the culture we have today. Structure first, the culture follow. Thank you. So, of course we know that every organization is different. Every regulated environment as well. It has its own constraints. So, we're not saying that our path is the best or the only one. What we're saying is that you need a compass during your journey. There's something that will

guide you when everything else is uncertain. For us, it was open source. So, now we can ask you, what's yours? And if you want to share your answers with us, please stay connected. We'll be happy to talk about it after the session or during the questions. And if our journey has inspired you, we have a QR code. You can see what we're doing in Pictet Tech. And

we can say, thank you so much for your attention and please ask your questions with the mic. Hey, hello. Can you hear me? Um so, I wanted to ask which infrastructure as code tools you use to deploy all these clusters? You use like a open tool for Terraform? So, we have created an internal tool to internal assembly of kubeadm, a lot of tools Cilium to to build

a Kubernetes cluster. And the for the user it's Argo CD, the platform and Helm and standard way. Mhm. All right. Thanks. Hello. Thank you for the information. Um my question is more about the people side of the change. So, technology is relatively easy, but how did you overcome the resistance you had from the people? Yeah, well, this one I don't know if my No. Um I don't

know if um it can happen over a day. First of all, you have to think about it as a process, a long one, and you have to introduce step by step new things, like new rituals, new meetings. Like it depends on what you want to achieve. You have to go step by step, you have to test it, try it. Without Genius Bar, that's the the only example

that I gave, we have started it in the one place, it was not very good. It was one big screen, so people were feeling shy to share their problems because they were feeling, ah, I'm not uh you know, I don't want to appear someone who doesn't understand. So, when we found like the third try, the rooms with a lot of screens, there when it really came. So,

you have to try things, you you will fail surely, but you don't have to stop. And you take feedback of your users. So, it's a long process. Did it come bottom up or you had strong support from the management? It came absolutely from bottom up. With with a strong support top of management. Thank It's also important. First, congratulations. I think it's really powerful what you guys did

knowing a Swiss bank. Like the culture change is really powerful. My question is how do you guys engage with the you said it's a contribution, so you hear from the the the users and they contribute to your platform as well? How can you elaborate a bit how this kind of Yeah. this kind of interactions, not just service providers, but like bringing out the target space. >> We

We We create internally kept same as Kubernetes. So, to the help hosting enhancement proposal. So, when you have a lot of people ask some new feature, I need this. So, when you say, "Okay, create a pull request and describe what you need really you need." So, then you less enhancement. And then you can explain You can write while it's possible or not. And the other team can

also see which What is in your stack or it's in your backlog. Mhm. Yeah. And did it happen that they also raise PRs to your stack or saying, "Hey, that's what I want. Here's a PR that would allow it." Or is it still in a conversation mode where they give you a requirement to uh to to get that It's more in term of requirement that in implementation.

Thank you so much. Thank you. Hi, thank you for your presentation. transforming a bank using open source software can be really challenging. And I wanted to ask you if there were any challenges or pushbacks in regards to the security aspect of transforming and using all these tools. Well, I think we have included security within those discussions. I cannot say I was here from the all the beginning

of this transformation, but I was here last step of that of the virtualization level. So, in the criteria of decision that Stefan has presented, there is an aspect of a security. And so, we were taking a decision collectively. It was not one person who was saying yes or no. So, we assembled all the people who were needed and all the competence domains in order to not make

an error. So, they were with us in this conversation. And of course, they had questions. But, they were supporting us more than pushing back. Thank you. Thank you. Hi, thank you for presentation. I have question, how you solve the problem of uh software what goes end of life? What what you use? the soft that's totally end of life software. >> Yes. What like when you have some

software what when it's open source, you heavily use it, but it goes end of life. how how you mitigate this problem? How avoid this problem? Yeah, when we use a lot of open source software, yeah, uh projects are um bought by the company. We are using CoreOS, then Red Hat stopped the project, so we go to Flatcar. Uh sometime some um the Kubernetes also the release management,

so end of life of the version also it's very difficult to to continue to use. So, the Compass is the Sense of project. yeah, at the end we use most of all the project are in the Sense of. Okay, thank you. Hello, actually impressive presentation, thanks. So, uh you never mentioned about cost. It's obviously balance between reliability and expenses, so was it win-win or new solution was

incredibly expensive? You have no secret. the cool Well, it's um depending about which technology we're talking about, it's a different conversation for me. As for um when we evaluate solutions to create something, uh Uh, we have a goal that we want to achieve. Of course, if it's a, um um, cost balancing problem, we would be very attentive about, uh, the technology and its cost. But, when we

choose to adopt a technology, we don't say we want a vendor one or we want an open source one. We see it as a large palette and we evaluate it through our framework and we choose the one that fits best. And depending on the context, depending on what we want to achieve, the dialogue is Okay, thanks. Thank you.