Cloud Native Theater | Data on Kubernetes Day: From VMs to Ku... Gabriele Bartolini & Laurent Parodi

This talk explores the evolution of the DBA role within HSBC as they transitioned from managing PostgreSQL databases on virtual machines to implementing a cloud-native architecture using Kubernetes. The speakers, Gabriel and Lauran, discuss the challenges faced during this transformation, including the need for automation, regulatory compliance, and self-service capabilities for application teams. They highlight the importance of community contributions to technologies like Cloud Native PostgreSQL (CNPG), emphasizing its design philosophy and security architecture. The talk details the bank's strategic imperatives, including reducing total cost of ownership and eliminating technical debt, while promoting database management as a core engineering function.

So they said that the DBA role was dying in the cloud native era. They were wrong. At HSBC, the DBA didn't just survive the move to Kubernetes. They led the charge to co-design the internal datab database platform uh for the bank. So welcome to data on Kubernetes. My name is Gabriel and today with Lauran we will talk about the journey that his organization has taken with posgress

kubernetes and cloud nativep. So I'm uh Gabriela I'm a data on kubernetes uh am community ambassador so I'm I'm really happy to be here today at this event and uh I'm also a posgus contributor. I've been using posgus for 25 years and uh I'm also uh a cloud native PG maintainer since you know six years ago I started to with my team here uh to bring uh

the database uh to kubernetes and I work for EDB I'm a VP chief architect uh for Kubernetes >> thank you and I am Lauren Parody I work for HSBC I'm the global product owner for posgress SQL So before we dive into the future of the cloud, I wanted to take a moment uh to look back. Tomorrow marks exactly two years since the posgus community lost prematurely one

of its brightest lights, Simon Riggs. Simon was a visionary and the primary architect behind many of the features that moved posgress from a standalone database to the enterprisegrade solutions that it is today. Over two decades, his vision and his code helped shape the database that we all know and love. But beyond the technology, Simon was a friend, a mentor, and a leader. So the QR code that

you see here points to his last p public speech uh titled the next the next 20 years for posgress in which he laid out the vision for where posgress needs to go next. So thank you Simon. >> Thank you Gabriel. So we will divide the presentation in five parts where we'll so we'll start where we came from the VMs and then walk through the new blueprint we

designed how we modernize the stack why cloud native PG became the engine of the stack and finally where we headed. So it's like this windmill. It was solid, built to last, but it was designed for a different era. That was our VM world. HSBC is one of the world's largest banks operating in 56 countries with 40 millions of clients and thousands of IT engineers. We have been

managing posgress SQL for long time and crafted a robust and realable database infrastructure using VMs. We built complex tooling with shell scripts, open source solutions backed by unseable playbooks and API to hide the complexity of the regular DBA tasks from the users. For example, high availability, backups, patching. That was a safe world. But this safety had a price. It rely on the DBA being part of the

loop. If something went wrong at 2 am, it wasn't an automated self-filling process that fixed it. It was a DBA and it created a bottleneck. We were masters of a craft, but we were managing pet databases and not cattle. The bank modernized and the pressure came from multiple directions at once. Our application teams had new expectations. They no longer had the bone weights to master advanced database

topics, replication failures, point in time recovery, failover scenarios. They didn't want to own that complexity anymore. They wanted to consume databases like a commodity. Developers just needed a connection string as an abstraction layer for the database infrastructure. They wanted to do dev, not ops. At the same time they had work with public cloud every day and they knew that on GCP for example you can spin up

databases within minutes with backups three replicas job is done we were offering ours on premises and it wasn't just about speed with public clouds you have new uh new versions new extensions uh new features our internal database service felt frozen by comparison So we pushed toward more automation APIs once a zero touch model and it helped to some extent but it's it created its own problems. The

automation layer became ownerous to maintain and slow to improve. Every new requirement meant another playbook, another pipeline, another thing to break. We were patching the wrong foundation. And before we could think about a new foundation, we had to be very honest about our constraints. A global bank isn't a startup or even a typical tech company. We can't just say, "Let's go all in on managed cloud databases."

Not because we don't want them, but because we can't always use them. Regulation encors us. In many countries, the regulators leave us no choice. The data must stay on our own systems and we have strict cyber security controls, auditing, patching, SLAs, secrets management and the list goes on. Vundering is simply not something we can accept. Digital sovereignty is not a nice to have. It's a hard requirement.

So any solution has to work with our terms with our controls in our environments that shaped every decision we made. So like the romance watchmen on this painting we didn't want to sorry we didn't wait for trouble to come to them they moved. That's exactly what we did. We stopped guarding the old world and starting building the new one. We were driven by two critical goals. One

to kill the technical depth and a strategic imperative to reduce the total cost of ownership of the database platform. We had five pillars. A sovereign debas strategy with 100% open source, no public cloud dependency and no vunder locking. Agility provisioning must happen within minutes not hours. It must be fully scalable and it must be fully self-service. Self-filling we must be insulated from hardware failures, automated failover, replica

recreation, compliance obviously with backup and restore including point in time recovery. We must have auditing and we must enforce the patching. Last but not least, observability. We were moving from legacy custom monitoring to native promises metrics with deep fleetwide visibility. This new foundation is how we transition from managing pets from to managing cattle. We did not arrive at this architecture overnight. My journey into data and kubernetes

started back in 2019. watching the operator landscape evolve, evaluating zylond doors and crunchy pio and I was always searching for something that didn't feel like legacy tool repackaged in a container. What drew me to cloud native PG in late 2023 was a design philosophy. A genuinely native Kubernetes approach. No external fellow manager, no patron bolted it to a container. It delivers a lower connective connective load for

our team and for our customers without cutting corners on stability. But choosing the technology was only half the work. We had to build the knowledge. Local meetups helped and a hands-on workshop in DB ran in Zurich in early 2025 was a moment our confidence solidified. Seeing maintainers who truly engage with their users and not just ship releases matter to us. That kind of community accountability and CNCF

governance is what made cloud native PG the right choice for our sovereign debas strategy at HSBC. So the direction was clear. Now we had to walk that long road. We made several key decisions to ensure security, compliance, and agility. First, we opted for multi-tenant Kubernetes clusters with an aim to shift to dedicated Kubernetes clusters for posgress SQL later. Namespace based isolation with a strict airbag and this

is important. Our application teams have zero direct namespace access. They cannot touch modify the Kubernetes subjects. All interactions are declarative via APIs and integrated with our in-house entertainment solution. This is how you do self-service without losing control. We are also integrating natively with Ashik Vault for database secrets using the database secrets engine which handles the rotation of the credentials automatically. Disaster recovery is managed through replica clusters

and patching is enforced, not optional. Let me dive into In the VM world, patching was an event, often a stressful one. In our new platform, we've turned it into a controlled life cycle. Our patching strategy is built to minimize the risk. Every week we ingest the latest cloud native PG operands into our private scanned registries. We use image cataloges with each within each name space to give

us absolute control over which version is running where. We follow a phased rollout. We never patch the entire fleet at once. We start with development environments to validate the behavior then move to staging and finally to production. Because cloud native PG handles rolling updates pod by pod, we maintain availability while we upgrade. By keeping the blast radius small, focusing on one cluster at a time with granularity

at the namespace level, we ensure that an operational bug in one area can't take down the entire Typically, we stick to a monthly schedule, but because the process is now fully automated, we have the muscle memory to respond to critical CVS in a matter of hours if needed. We've moved from being afraid of patching to making them a non-event. This satisfy both our security team and our

change management process. So let's go deeper now because if you're placing your organization in the hands of an operator, you need to understand how it thinks. So cloud nativeg or CNPG is not just a tool from ADB my company. Okay. So although EDB is the creator and main contributor, Cloud NetPG is an official CNCS CNCF sandbox project. By moving into the CNCF, we've committed to vendor neutral

government governance and long-term sustainable opensource future for an organization like Laurance. This is a critical de-risking factor and it means that the bank is building on a foundation that follows the same rigorous standards as the kub kubernetes project itself. As maintainers and posgus contributors, we are not just building a tool. We are contributing to the global standard for how posgress uh should run in a cloudnative world.

So let's look at the main principles behind CMPG. We just don't run posgress. We leverage the Kubernetes API directly. We've removed the need for external managers like uh rep manager, patroni and consensus tools like HCD. Instead, we store the status of the cluster directly in Kubernetes and we use a lightweight instance manager as speed one or entry point of the container. The instance manager acts as the

bridge between the cublet and the database handling all probes start up livveness and readiness and reporting health directly to the control plane. This minimalist design isn't just about lean code. It's also about a reduced attack surface. In Laurent's world, fewer moving parts means fewer binaries to scan, fewer ports to open, and a much faster path through security audits. So for time reasons I cannot go deep into

the full technical design of cloud netpg in this session. However I highly encourage you to read more about how we built the operator by scanning this QR code. It's a great resource for understanding our reconciliation logic and the main resources that um make it happen. So standard statefuls. Oh, do you want me to go back? I'll leave it. You can scan the code. One, two, three. Go.

Let's go. Standard stateful sets treat every pod as an identical twin. But a database like posgress doesn't work that way. We use direct volume management to orchestrate pods and persistent volume claims individually. This gives us a role aware life cycle allowing us to manage the primary and standby servers differently. We can apply tailored configurations or patch replicas first followed by a controlled restart or switch over. For

backups, we provide a pluggable interface. This means that we don't have any hard dependency on a specific legacy backup for posgress backup tool for posgress. So for historical reasons we provide a plug-in for BMAN cloud which uh we we we created to enable backup and recovery on an S3 compatible object store. But for large data sets, we also provide support for native uh Kubernetes volume snapshots to

provide uh faster backup and faster recovery operations. Our maintainer Leonardo Czechi who's here is also working directly on volume group snapshots for Kubernetes and is a developer for Kubernetes storage. So finally we addressed the day two compliance reality for a bank. Logs are a security liability if they are not handled correctly. In our design logs are never stored anywhere within the container. We bypass the local f

system entirely to prevent any local tampering or data leakage. Instead, everything is streamed immediately to standard output in JSON format. By using multi-channel streams, we split at the source the standard posgress logs from for example the PG audit events. This allows the security team to route audit uh audit data directly to a temple proof vault while operational logs stay with the DBAs. We've also built the Prometheus

exporter directly into the instance manager. It provides over 50 native metrics out of the box focused purely on the database and allows you to add custom metrics. So like these students, we're still learning, still building even in the dark. So we're not done. So as we look at the road ahead, our goal is to move cloud netpg from the CNCF sandbox into incubation. Technically, we are pushing

the boundaries of what cloud native means We are moving towards distributing posgress extensions like PG vector as image volumes leveraging the new primitives in cloudpg 129 to make extensions as portable as the database We're also evolving our plug-in architecture by officializing the Barman cloud plug-in and removing Barman cloud support from the from the core. So we are making the operator leaner and more pluggable for a bank.

Security is paramount. So we pursue full open open FS open open SSF compliance to guarantee a secure software supply chain. Finally, we're building even more DBA logic into the operator with declarative datab database roles and a comprehensive data control language support uh enabling grant and revoke operations. We want every aspect of the database um of the database global objects to be as declarative and repeatable as a

Kubernetes deployment. So this is the moment we answer the previous question. Um is the database administrator role dead? No. But we were required to evolve. We had to leave our comfort zone and move from being eyesshaped experts to T-shaped then pi shaped engineers. It wasn't enough to just know posgress anymore. We had to build a second equally deep pillar of expertise in Kubernetes along with GitHubs. We

had to stop being consumers of the platforms and start being the architects of it. By seeing the journey through and making it happen, the DBA team has at HSBC has transitioned from a back office support function to a core part of the platform engineering mission. We defined the blueprint. We evaluated the operators. We designed the manage service with our Kubernetes platform team. And that's how you stay

relevant. And that's how you accelerate. A few things we've learned the hard way. In a bank, day two operations are governed by strict guard rails. For example, we cannot patch whenever we want. We must respect specific maintenance windows different for each application. Currently, we use an external scheduleuler to trigger this. Though we are working with a community to bring this onto the cloud native PG road map.

We also live by the principle of list privilege. Even as the owners of the data platform, we do not have cluster and mean access on the Kubernetes clusters. And that's all right. We've had to carefully engineer our back so we can manage image cataloges and controllers without needing the keys to the kingdom. Furthermore, working in air gap environments require a robust strategy. We use customized overlays to

manage the complexity of deploying in isolating networks where every image pool must be internal and verified. It works but it requires discipline and automation. So where are we going now that the infrastructure is automated? Our application teams can finally focus on business deliverables. They don't have to worry about backups or H anymore. Instead, they can use the rich observability we provide to zero in on what actually

matters, per tuning. We're also looking at the next versions of Kubernetes with Posgress SQL 18 to assess how we can better manage posgress SQL extensions thanks to the work that the cloud native PG team has done with a native with a new image volume extensions feature. I'm super excited about it. And as we gather more data from our fleet, the ultimate goal is to move toward AI

assisted tuning. We want the platform to eventually suggest index optimizations or configuration changes based on real time traffic patterns. We've built the foundation and we want to build the brain. Thank you. We're happy to take questions. >> Thank you. >> Hello. Uh are you running this on production? >> Yes. >> And like did you have any instance? What was your experience? >> Yes. We've >> like about

troubleshooting. >> Yeah. We've been running to go SQL instances for 15 years in the bank. No like I am talking about like what was your tr like experience with troubleshooting the database on kubernetes like when one of your postgress post failed for example >> most of the challenges are taken care of by the operator itself. So the traditional world with VMs where we had to deal with

replication issues and so on it's now gone for us. So most of the issues we deal with are related to performance and tuning the resources allocated to the port and so on. We're still having time for many questions. have you question. Well, um my uh question relates to um high availability setup and uh just would like to hear your experience regarding the failovers especially uh when there

is really high load and uh uh really uh large amount of write ahead logs and like what are the meanantimes to fail over and to recover. Yeah, it's uh also for large databases which are generally more than 15 20 terabytes something. >> Yeah. So right rightly so the the amount of um right ahead log is is an important um metric to consider when you tune u the

the cluster configuration. Okay. Clanbg allows you to uh control a a few settings uh that can speed up with Kubernetes the detection of a permanent failure. Okay, because that that's the main main main thing you know to if if the failure is trans transient it might in some cases might be better to wait to to perform a failover okay instead of failing over. So you can also

delay that. So there are there are few uh knobs for the cloudpg allows you to to control. Let's say that the default settings of clonerpg are quite conservative. So we prefer to uh perform a fa take more time to perform a failover. Uh although you know even by default normally it takes 10 15 seconds okay out of the box to perform the failover but in some cases

this is not considered aggressive enough. Okay. Uh for for some customers but what you're saying is that depending on the um amount of uh uh write ahead log the replicas might be lag lagging. Okay. Or uh in case of fail over you could have data loss. So you can also configure synchronous replication to make sure that a transaction is written at least on a on a replica

or there's also a setting in posgress that can be configured also at transaction level where you can ask that a transaction not only can be written on a remote host but also replayed by the posgus uh replica replied replayed replayed so it means that all reapplied by the replica. Okay. So, um that that allows you to slow down the the the right operations but have a better

uh story in terms of recovery time in case of failover. Okay. So, it really depends on you. So um posgus in my opinion is already fast even with synchronous replication which I would never put a database in production without streaming replication. Okay, let's this is the foundation. Okay, but if you use synchronous replication, you you you you are more protected. We also have another feature called failover

quorum that we introduced that for example in case of absence of of um quorum of the synchronous replicas. It does the operator doesn't perform a promotion. Okay, so that you are sure that you're not losing data. Okay, it really depends on uh your RTO goals and RPO goals, but it's generally all tunable. Okay, so that's don't know if if I answered your questions, but if you if

you have more qu because for time reasons uh I'm I'm at the booth EDB booth uh which is over there. If you have more questions, I'll be happy even today or tomorrow to to respond. Okay. And >> thank you so much to our speakers. We are on time. >> Yep.