Operating Postgres Logical Replication at Massive Scale

This talk presents the intricacies of implementing Change Data Capture (CDC) for PostgreSQL at a massive scale, delivered by Tristan Amati, a solutions architect at ClickHouse. He begins by explaining the concept of CDC and its relevance in replicating changes from PostgreSQL to systems like ClickHouse for real-time analytics. The speaker navigates through key PostgreSQL mechanisms such as logical decoding and replication slots, emphasizing the challenges posed by PostgreSQL's write-ahead log as it relates to CDC. He highlights the architecture of ClickPipes, which facilitates seamless data ingestion from PostgreSQL to ClickHouse, and discusses the benefits associated with adopting ClickHouse for analytics, including its performance in processing large datasets. The session concludes with insights into the future improvements and expansions of ClickHouse's CDC capabilities.

Um So let's all put my name as Tristan Amati, I'm a solutions architect at ClickHouse. Um we're going to be talking about operating PostgreSQL CDC at massive scale. Curiosity, uh anyone here familiar with CDC or using it uh in production like any form of CDC? It doesn't have to be PostgreSQL, but from different systems. Okay, cool. Um so for those that aren't familiar with the general topic

that we're talking about, I thought it might help to um start with a quick primer on PostgreSQL change data capture as CDC as we refer to it. Um PostgreSQL CDC is really um represents a logical stream of changes that occur in PostgreSQL uh that external clients can consume. So right, being able to take all of the data that, you know, is change is not just being ingested,

but also changing uh in in one system and and mirroring it, if you will, or copying it over to another uh for different purposes. This stream contains messages such as inserts, updates, deletes, and uh schema DDL commands as well. That external system that's relying on it, that's consuming it, um can consume to continuously maintain a consistent snapshot uh of the PostgreSQL database. Um PostgreSQL CDC powers many

real-world use cases. For example, replicating PostgreSQL databases to other data stores such as ClickHouse uh where I work, ClickHouse, um for offloading um analytics, more like being able to do real-time analytics uh in a system like ClickHouse or pushing out to Kafka for real-time streaming and more. Uh it also supports other use cases like online data migrations, um PostgreSQL version upgrades, and and so on. You get

the idea. I Oh, did I lose it? Oh, no. I want to introduce a couple of um PostgreSQL specific concepts related to PostgreSQL CDC as um they'll be important for this talk as we progress. Um the first is logical decoding. When you hear us me saying logical decoding, I'm really referring to the logical replication of the data. This is a technical mechanism behind PostgreSQL CDC. Uh it

helps transform the PostgreSQL transaction log, which we refer to as a a wall, a a write-ahead log, um into streams of meaningful changes that uh or the external system can understand from one another. These When you hear me say meaningful changes are really the upsert the up inserts, updates, and deletes. Um the next concept is uh a lot replication slot, which is the building block of the

logical decoding um that we talked about previously. Logical replication slot essentially captures the changes happening in PostgreSQL through the write-ahead log, through the wall, and enables streaming of the wall into a logical format. Uh so under the covers, it's basically keeping track of uh already consumed and to be consumed by the client. So being able to understand like where have I left off or like where am

I going next? Um so, you know, basically it's a delta structure that keeps track of the wall uh in an understandable format. The reason this is difficult is because logical decoding is very tightly coupled with the PostgreSQL process. Um consuming changes requires an active database connection uh and PostgreSQL retains that wall um uh until those changes are acknowledged. So if a replication slot isn't actively consumed, the

wall starts to pile up, uh it eats up disk, uh worst case, it can crash PostgreSQL. So it's very fragile in that in that regard. Um in PostgreSQL, logical decoding is complex and non-linear because of the wall, um the write-ahead log as as you as you're aware. Uh it records and the commands and as they occur in PostgreSQL and and they're very interwoven within the system and

and decoding it is very non-trivial. So it's not just sequential like, "Hey, you know, the this record was updated. This record was inserted." It it creates a very meshed uh format in in the write-ahead log. Um so basically, they're not grouped together in a very easy way to be decoded. So it so it's a little bit a little bit more difficult. It's very different than MySQL's uh

sort of bin log in the sense that which is a lot more transaction ordered and um making it very easy to to decode. So we're working around some of the nuance sort of difficulties uh specific to PostgreSQL in this regard. Um so again, yes, very fragile, needs to be handled with care, and we'll talk a little bit about all of the uh you know, the steps that

we took along the way in order to make sure that we're accounting for these PostgreSQL specific uh nuances. So This is an example of a client um being able to consume from PostgreSQL data. This is a Python script where initially the block that you see at the top, this is where uh PeerDB specifically uh executes a command that says, "Hey, start the replication slot that begins decode."

Uh which is obviously a necessary step. Uh and then at the bottom there, you can see the client sending back sending information back. It has to say, "Hey, PostgreSQL, I have read the write-ahead log up until this position." Uh otherwise, PostgreSQL doesn't know that it can flush the write-ahead log on its own without without acknowledgement. So we'll talk a little bit about uh ack and um the

sequence by which uh it takes place in in the entire flow of things. So now that we've covered the key concepts behind PostgreSQL CDC, let's take a deeper look into why ClickHouse uh is uh is a good, you know, pair is very well for this and why it needs PostgreSQL CDC at scale. Uh before I here have heard of familiar with ClickHouse? Well, I know I see

Planet Scale here. I know we've we've worked together with that team before, so thank you guys for joining as well. Um ClickHouse is an open-source oriented distributed database. So when we say this, we're referring to, you know, open-source. It's been around since 2009. Um it went into production in 2012 um supporting some of the largest just analytic platforms in the world. Uh it gained huge momentum and

community around that open-source. So in 2021, ClickHouse Inc. was formed uh where we focused primarily on building out a fully managed SaaS platform for ClickHouse to make it more seamless and easy to use uh you know, both from a getting started perspective and especially from a, you know, cost and um efficiency perspective in terms of like the time and effort spent um around managing ClickHouse. Um it's

a columnar database, so essentially it's column oriented like other databases that you've used where um storing data as columns, you know, storing the back-end data files, every column as its own file on the back end. Uh you gain efficiencies from an aggregation perspective in that regard and reduce disk IO from from how you read the data. Uh and it's distributed. You have the ability to shard the

open-source or use the fully managed ClickHouse cloud in a separated compute and storage and as Um and it gives you a lot of uh flexibility from a horizontal scaling perspective. So um there's a lot of there's a lot of opportunities in how you want to architect that. Um and it's an OLAP database, so you know, it's purpose-built for analytics. Um the type of, you know, aggregations, heavy

group buys, uh being able to do things like joins, real-time visualizations uh against large large amounts of data. We had uh Tesla at our annual user conference about a year ago talking about how they're ingesting, you know, 1 billion records per second into tables that were storing quadrillions of records from a from a metric analytic perspective. So being able to hear use cases like that is is

really great. So one of the most common um source of migrations to ClickHouse is PostgreSQL as a source system. Um as the data size grows in PostgreSQL, um the analytics become more difficult to perform. Um so this is when customers adopt ClickHouse to offload to a system that's appropriate for it. Um to adopt ClickHouse, one of the common architectures includes replicating the transactional data in in PostgreSQL

using CDC and and making it essentially analytics ready. Um this is where, you know, the story kind of picks up and CDC comes into play. Uh and the ref the reference architecture that we're we're presenting here, this sort of like unified data stack, is really um it's widely used across lots of our um customers. We'll talk a little bit more about that as So the whole um

this is where also where ClickPipes comes into play. ClickPipes is meant to be our turnkey engine that allows you to ingest data in real time multiple sources uh CDC like what we're talking about here today, PostgreSQL, uh also MySQL, MongoDB, uh but also from streaming systems like Kafka and Kinesis, uh as well as object storage like uh S3, GCP, um Azure Blob, uh and data lakes coming

soon as well. The key word here is that it's managed. So you're not building the individual components that we're going to be talking about here today, uh and you're not having to maintain these complex data pipelines as well. Our goal is really to abstract that complexity from you and your team so that you can focus on, you know, building out your system and scaling your environment. Um

and we're going to click into a little bit of um you know, how how we achieve that. But um this is an example of uh what that PostgreSQL CDC would look like. Um so you go in and you create essentially in ClickPipes, you're creating a pipe, uh your credentials, uh in minutes, you're able to see the data populating into ClickHouse as a result of that. Under the

covers, it's using concepts that we're going to dig into like logical decoding, the logical replication slots, workflow orchestration, and things like that. But um from a from an user's perspective, it's meant to be an interface that makes it very easy to just like, you know, enter your information, uh designate the information that you want to push over into ClickHouse, and let it do its thing. And I

mentioned that ClickPipes supports many sources, but as you might expect Postgres is one of the largest and most common source systems being fed into ClickPipes to populate into ClickHouse for real-time analytics. Over 500 of our customers, including many enterprises, use Postgres CDC as a source. We move 250 terabytes of Postgres data into ClickHouse on a monthly basis, which corresponds to 300 billion plus individual records. So, from

a database count perspective, like looking at individual pipes that are being creative, there's over 1,000 some customers have more than one database that they're moving over or systems with organizations and department teams within within their organization. And the largest single ClickPipe database exceeds 60 terabytes You can see here on the slide some of the customers using this unified Postgres ClickHouse stack through CDC transfer. And the quote

there that you see at the bottom, Debezium is similar How many Raise your hand if you've heard of Debezium? If you're familiar Yeah. So, Debezium is similar. What we've found from our experience is that many of our customers migrate off of Debezium because most reasons around performance, speed of the actual CDC transfer itself, and reliability. Reliability mainly because of how We talked about it earlier, how is.

So, we'll talk a little bit more about all of the components that we got into for that. Postgres CDC is ClickPipes is It's actually powered by Pure DB. If you've And it it it It's essentially the underlying components of everything that but we've migrated a lot of that functionality into the ClickPipes UI to be able to just make it easier to use. But if you wanted to

use it from an open source perspective, you're more than welcome to do so. Mike's cutting in and out. Let me know if you guys can hear me okay. I'm not sure why. Um ClickHouse acquired Pure DB a year and a half ago. And they've been part of our team and kind of expanded upon and built out the ClickPipes um the ClickPipes system into what it is today.

Okay, now that we understand the scale and why ClickHouse Postgres CDC is important to ClickHouse, let's dig into that architecture itself. So, the first thing we're going to look at is all of the infrastructure components, all the pods that are working together, and then we'll look at the actual state flow of So, looking at the infrastructure, starting out, the user accesses ClickPipes UI and kicks off a

ClickPipe or API if you're using, you know, Postgres sorry, Pure DB open source. But they're in the UI and then they execute a ClickPipe. Once they kick off a pipe, a GRPC request goes to the flow API, what we refer to as our flow API. This essentially takes all of the requests from the UI and passes them to Temporal, which Temporal is a workflow framework. We use

it as our workflow orchestrator in this case. And Temporal is essentially triggering a bunch of different workflows responsible for the data replication and it and it also keeps track of the running states, which we'll get into next. On the top right corner in yellow, you can also see that we have a catalog. It sort of you know, sits on its own. It's just an internal Postgres database

used to store the state as a replication progresses. It's fully pluggable. You can, you know, you can It can be managed or self-hosted. We kind of don't have a lot of opinions in that regard, like a Postgres RDS machine like off on the side, just needs an endpoint to connect to. Eventually, we might bring that into the fold, but that's a piece of the architecture that's sort

of like an appendage sitting off. Okay, so there's two important phases of the CDC. First is copying the existing Postgres data, which you'll hear me refer to as the initial load. And second, the continuously syncing changes as they occur, sort of like the ongoing. So, let's start with the initial load first. You can see here there's there's two components there in green on the bottom. They are

They are the snapshot worker to get a persistent view of the database. We take a snapshot connection on the source database and start reading the initial tables from there. So, if a customer had 100 terabytes in a table, the initial load would be the heavy part, and then the the ongoing CDC would be the the day-to-day updates and changes that they're making from that point forward. The

flow worker is responsible for reading data from Postgres and staging it in an intermediate sort of storage environment. We use object storage. It could be S3, it could be MinIO, GCS, Google Cloud Storage. Um in an optimized format. We use Avro with Zstandard compression, but different you different options are available. So, while this initial load is happening, there's the continuous CDC. There's this concept of what's referred

to as a replication slot, which essentially buffers all of the changes while we do the initial load. So, once that initial load is complete, we start reading the replication slot for all of the changes that have been accumulated, like what I've missed thus far, and the data is synced from the internal stage to ClickHouse. So, essentially, what we're doing here is we're reading from Postgres, writing to

S3, reading from S3, writing to ClickHouse, and it's it's fully asynchronous as well. I'll talk a little bit more about why that's important later. And it's important to highlight We keep talking about the fragility of of Postgres, but it's very important one of the factors that we had to consider was the fact that we always have to keep the size of the replication slot low. Otherwise, the

source database grows and the system gets upset, and it can crash. So, we read the data from Postgres and immediately push it to object storage for acknowledgement so that so that we could just get it staged and and keep that replication slot as low as possible constantly. So, instead of doing a pull, push, acknowledgement because that's enough time for the replication slot to grow, we do a

pull, stage, acknowledge, and then push. So, that that's sort of like slight difference gives us a gives us a a benefit in in maintaining a low replication slot size. This whole initial load that we're talking about is parallelized for performance. After the initial snapshot, we copy copy multiple tables at the same time, and even within a table, we partition into segments and copy multiple segments at a

time as well, sort of like in parallel. And through this whole process, we store all of this state in Temporal. We'll get into the state flow next and look at that orchestration. But each component of this architecture that you're seeing is horizontally scalable and fully stateless as well. It's running in a virtual structure. So, like, you know, ClickPipes UI, I mean, that's that's the UI, but the

flow API is is a pod, you know, Kubernetes, the you know, the worker the snapshot worker is a pod of itself, and and that has benefits in of itself that we'll get into in a moment as well. So, at a high level, snapshot phase optimized for fast snapshots, and then the CDC phase is the stage to as staging the data to S3 for quick acknowledgement so that

we can get that replication slot low. Got it. Now, let's take a look at the state machine. So, at the very top, you can see the setup flow. This This part's pretty simple. It's responsible for preparing the replication job. It's doing some of the initial steps necessary, like creating the replication slot and publication in Postgres. It's setting up the corresponding tables in ClickHouse with the corresponding column

mapping. So, create this table, it's empty, be ready, here are the columns, you know, integers, strings, low cardinality fields, things like that. And it's ensuring that everything is ready for the replication job to begin. That Next is the snapshot flow, which you can see in the middle in orange there. This is essentially handling the initial load. So, fetching the existing data from Postgres tables into ClickHouse. The

snapshot flow essentially has two substates within it. You can see those above it in blue with the with the loop there. So, the snapshot transaction is responsible for establishing and holding a consistent snapshot connection. This ensures that data consistency during the initial load is maintained. Then there's the partition initial load. It's a loop composed of multiple states that synchronize the partitions in parallel, enabling efficient bulk transfer

of data from Postgres to ClickHouse. So, essentially, we chunk the the tables into partitions and in parallel load them into ClickHouse as a as a for loop. So, the advantage here is that if things break, we don't have to restart the entire process. We just continue from where things left off. Once the initial load is complete, the system transitions into the CDC running state. You You can

see that the far right block in red. This state is responsible for continuously consuming changes from Postgres replication slot and syncing them to ClickHouse. So, the CDC running state is is a component of that. You can see those at the very bottom in blue. Those are the two substates. The sync loop continuously is reading the replication slot and pushing to S3. More specifically here, we issue a

command to Postgres to start decoding the logical replication. We talked about how difficult reading that that slot is. It begins doing the decoding and of each transaction, giving us the individual statements, right, update, of what happened on the source data. We encode them into Avro, and then we store them into S3. We just continue doing this for bulk loads. The next sort of component of the CDC

running state is the normalized loop. picking up where it leaves off, reading the S3 Avro files, pushing it into ClickHouse. So, it's click collecting those changes, putting them into that raw landing table that has been created in the setup flow. and then we use sort of like a landing table and then we migrate that into like a like a final table This is really helpful because if,

for example, ClickHouse is down, we don't want to stop reading the replication slot, right? We don't want to lose that connection, otherwise it can it can crash Postgres. Both the sync loop and the normalized loop that we're talking about here operate asynchronously as well. They're decoupled and don't depend on one another, so it helps us, you know, just from a from a maintenance perspective. And all of

these nuances that we're going through and from a like a complexity perspective, they really highlight, you know, why we're able to build an entire, you know, like enterprise grade CDC system from this. It accounts for all of the sort of like nuanced difficulties and changes. I mean, we're talking about customers that have 50, 60 terabytes, you know, at a time. And in ClickHouse this takes like the

the full process of everything we went through it takes under 10 seconds. Like that that's sort of like where we are right now. Pushing to other destinations like other databases from Postgres can take up to 20 minutes for the entire but being able to do that in under 10 seconds is something we're really proud of. We have customers pushing us to sort of meet more like sub

1 second and to end. So, that's something that that's that's sort of like our next And you know, we refer to each of the states here as idempotent, which is, you know, it really makes the system easy to recover, allowing it to safely resume without having to reprocess, restart the whole process from scratch. So. As we started looking at the architecture, we really had to ask ourselves,

like is this too complicated? Like have we overly like complicated the the process? And you know, we we don't think we have. In fact, we think we've built it in a way that's very modular instead and necessary for enterprise grade CDC. Let's look at some of these advantages that we get from building like each component the way that we have. Infrastructure isolation by design, really this is

saying like failures or scaling events in one component don't cascade to to others. So, if the UI fails, the CDC can still continue, right? When we were talking about the flow worker earlier, if the flow worker fails, the snapshot worker can still hold a consistent connection, right? So, like because they're decoupled and because they're set up, you know, as individualized pods, they're they're sort of impervious to

Scale any component independently. So, for example, we were talking about the doing a lot of the work. It has to pull to the data, processes to Avro, pushes to S3. It's sort of like the heaviest part of the of the operation. You can add resources to the flow worker independent of other systems without having to just throw, you know, CPU and RAM to the entire system and

hoping it goes to the right place. You're not just throwing spaghetti on the wall, you're you're specifically isolating a specific component that needs those resources. Each state is independently retryable. You know, we talked a little bit about this, but for example, when you're doing the initial load, let's say you get an out of memory error, you don't have to restart the entire process because we're storing all

of this in that catalog system, the Postgres catalog system that we talked about. So, instead of just instead all you would have to do is just scale up the those components, the flow worker in this case, and pick up where you left off. We're decoupling workflows as needed. I showed you how we separated the sync flow and the normalized flow and talked about how they were decoupled.

The advantage there being that for whatever reason ClickHouse is down, Postgres isn't affected because that process is already running and flushing the slots into S3 as an intermediary. So, it ensures that the system is safer and more reliable in that regard. And we get full visibility. We, as well as our, you know, our customers using the system get full visibility with Temporal. It is so incredibly core

to our infrastructure. We wouldn't we wouldn't be able to live without it, quite frankly. We get visibility into every detail of of every state of the process. So, how long it's taking, you know, what was the error, you know, if we needed diagnose it, we have an easy way and interface to be able to jump in and take a look at exactly what happened, reducing our, you

know, mean time to resolution And it's Kubernetes native architecture, so all of those components that you saw are are pods. It helps from a deployment perspective, it helps from a scalability perspective, orchestration, things like that. So, I wanted to go through a few important decisions that were like very important to us, at least. There's probably more, but you know, we we wanted to make sure that it

was purpose built for Postgres. So, the decisions we were making were were unique to, you know, working with So, for example, one of the optimizations that we do is parallel initial load. We chunk the tables into partitions. These partitions are essentially CTID based partitions. It's logical, it's not they're not physical partitions. In Postgres this is just a pointer of the row and tuple on disk. So, page

number {comma} offset, if you could see from that example. the order of things. We start a snapshot connection on a large table. We partition the table logically on the CTID and then we stream them logically into ClickHouse in parallel. The advantage being, you know, it lets you move terabytes of data in hours instead of days. As an example, I saw a lot of hands raised when I

mentioned Debezium. Debezium doesn't have this concept of parallel snapshotting in in a single table level. They do actually, I should clarify, they do have parallel snapshotting, it's for tables. So, they're they're able to parallelize multiple tables, but not within a table. So, you know, if you had a 10 terabyte that can take days. I don't know, in some cases weeks, depending on the resources you have available,

or it can fail along along the way. In ClickPipes this takes hours. We always ensure that the replication slot is is being consumed. We have mechanisms in the state machine to never give up reading the replication slot. a very important part of working with with with the CDC CDC architecture overall. Because if we give up that that the replication size grows and we crash the OLTP database,

which is unacceptable in in in a production system that we're building. So, being able to maintain that connection is incredibly important. If we if we dove into Temporal and and looked at the state of things, you'll see that we've got connection replication slot connections that are maintained for, you know, days and and that's that's intended for for a pipe. And we have full transparency into how the

system operates. So, not just for us, you know, internally, but also extending these features. These are some of the views that our customers are actually able to monitor, you know, for their system. We let them look at the replication size, replication slot size over time. We allow them to create alerts on the replication slot size in case it's growing, so they want to be notified before it

gets too large or if it's starting to become something they need to focus on. They can see the number of connections. And because we recognize that Postgres is so fragile, we always keep a close eye on these metrics as Another factor that we we took into consideration was really obsessing over the experience that that our customers, our users, have. So, as an example of we have ClickPipes

CDC constantly being created by new customers. And you know, one of the things we wanted to make sure that we did was we wanted to be able to constantly perform validation checks before they, you know, just fire off and execute something. When you hit that execute button, you have this sort of like idea in your mind that like things better work, right? If you run into errors

after that, it it creates a poor, you know, user user experience. let's just say you're creating a new pipe and there could be many failures through the process of just creating it. So, maybe Postgres is not reachable, maybe Postgres is down, maybe you don't have enough permissions to read certain tables, maybe ClickHouse couldn't be, you know, connected to. We don't want the customers to create the pipe

and then have an error afterward. In the UI itself, you can see from this shot that we have here on the right, we perform over 50 validations that are actively being checked as as the ClickPipe is being created, so that we can let you know up front instead of fire off and execute, okay, there's an error and now I have to go dig in and find out

what it is. We validate all of that up front. If if if any one of those checks don't validate, we don't we don't allow you to even create what's referred the mirror. So, we don't allow you to kick it off and and fire it from there. And we provide specific notifications when things do go wrong. If a replication slot size hits the defined threshold and the alert

that we were talking about earlier, we push a notification to different systems that you that you set up. And it includes a doc with an explanation of why it's growing and what you can do about it. This usually helps kind of self-diagnose and you know, from having to create a support ticket. So, you get the notification, that's great, but also giving you some contextually, you know, relevant

information that you can actually do something with is incredibly important as part of that as well. And we provide granular granular configurability. We find that our our customers are very specific about exactly what they want to do, like what column like down to the column level, down to the to the type level. They want specific columns to be replicated because, you know, maybe for reasons of PII.

Um, once the pipe is created, they want to add new tables, they want to remove old tables. They want to specify what the table engine is uh after it's been created. So, and we get it. It's important. So, um, we provide lots of configurability as you can see here from some of these screens um to be able to do that. And and there's more that we could

be doing. So, there's more to come in that regard some other uh steps that we're taking along the way um, you know, areas that we feel we can improve upon, areas that we feel are are gaps. Um, lack of comprehensive support for schema changes. So, we we talked a lot about inserts, updates, deletes. Um, when we talk about things like alter tables or alter columns, uh the

the hard part is that information isn't backfilled. Um, it's not captured by the actual replication slot itself. So, we don't have an easy way of being able to read it. So, that's an area where we feel like we can improve and we're looking we have some ideas we're we're working on um some some things for that. I mentioned it earlier, sub-second replication. So, end-to-end being able to

achieve all of this um, you know, because we push to S3, uh because of our ELT process where essentially you heard me mention that when we populate into ClickHouse, it goes to a landing table, it goes to a final table. Some of these little things like add up to that 10 second uh I don't want to call it lag time, but that 10 second sort of like

end-to-end uh latency. And we're working on kind of getting that further I I As I mentioned earlier, we've got pushing us to get closer to like, you know, single digit or sub one second and um it's something I'm excited to see the team build out as the year continues. replication slot growth remains a common operational concern. It's like one of the biggest like tickets we're constantly getting.

It's It's not on us. It's, you know, like it it's on Postgres, like to be fair. Um, but it's something that we need to um focus on very heavily. Uh and logical decoding is very single-threaded. So, being able to think about how we parallelize some of those efforts um are going to be very important. Um, there are differences in data modeling between Postgres and ClickHouse also. Like,

the syntax is just different. We're both ANSI SQL compliant. So, like, you can you can just, you know, for the most part copy-paste your SQL statements in ClickHouse, but ClickHouse has some nuance differences with regard to things like table engines and settings that can be configured for optimization to get better performance out of it. Those differences in syntax can trip up some people sometimes. So, being able

to uh abstract that a bit is is something that I think uh a lot of people will get will get value from. Okay. And in case you didn't know it, uh ClickHouse now offers its own managed Postgres like right from within ClickHouse. So, you can get that fully unified data stack that combines Postgres uh for transactions and ClickHouse for analytics with no additional complexity. So, not having

you know, hook into a separate Postgres system um as as you've had to before, you can just spin up a Postgres instance right from within uh ClickHouse as well. So, being able to replicate your Postgres data to ClickHouse using that CDC transfer in ClickPipes just becomes even easier. Um, you get a PG ClickHouse as you can see right there in in the very like top center. Uh

you get PG ClickHouse extension which lets you query your ClickHouse data for the OLAP queries from Postgres. So, your application can just use Postgres as a unified query layer for both the transactional and analytic queries without needing to connect to multiple databases, just simplifying, you know, the architecture of your application. Um, most managed Postgres services use network attached storage, but NAS um like uh Amazon EBS, which

requires a network round trip for every disk access um and it can potentially introduce latency. Our managed uh Postgres uses NVMe locally attached storage. Um, so so it's on the same server as your database and it we've done benchmarks. Happy to share, you know, a bit of that. It's going to offer a lot of performance benefits for for teams that are using it. So, not just not

just getting a simplified architecture with, you know, with a OLTP system that you want to unify, but also getting better performance out of it as well. Feel free to sign up. I've got information here, you know, a link that you can jump on on. Um, so if you want to try it out and uh experiment with it, uh you're more than welcome to. Um, that's where you

would go. And um feel free to connect with us as well. Um, we're going to be here all weekend. We've got a booth. Uh if you want to join us on Slack, we've got a community page that has that's the QR code that you see there on the left. Just feel free to zoom in if you want to isolate that one. And we're hosting um dinner and

uh drinks tomorrow for everyone to join. If you want to register for that, um you can click that click that uh QR code on the right. There'll be a couple talks that I think you'll get a huge kick out of. Not from ClickHouse. We have we like to have our our users, our community, our customers on stage um being able to present on our behalf. So, thank

you all. Really appreciate your time. And um yeah. Uh I'll I'll be I'll be sort of like in the crowd here if