Taming Chaos: Deterministic Simulation Testing for Distributed Systems - Unmesh Joshi

This talk covers deterministic simulation testing in the context of distributed systems and its importance for ensuring system reliability and correctness. The speaker explains that distributed systems, like Kafka and Cassandra, require rigorous testing beyond traditional methods such as unit tests and chaos engineering. Deterministic simulation testing is presented as an enhancement to fault-focused testing, where faults are systematically introduced to evaluate client-observed behavior under various conditions. The talk includes examples of quorum-based read and write systems, demonstrating how simulation can help reproduce consistent behaviors even amidst failures. Key concerns such as network delays, thread scheduling, and system clocks contributing to non-determinism are examined, along with how testing frameworks like Jepsen can validate system responses under these conditions. The session emphasizes the need for precise control over execution to achieve reliable testing outcomes.

So, I'll be talking about deterministic simulation testing and um it's in the context of distributed systems. And when I say distributed systems, uh you can map it to any product that you know, uh like Kafka, Cassandra, any cloud database that you use like Aurora or MongoDB or anything else or or Kubernetes or any big data product. So, when I say distributed system, map it to um some

of that. you you must be aware of um different types of testing, right? So, there is unit test, integration test. Um and I don't know has anyone heard of chaos engineering or chaos monkey? Yeah. So, um and then there are products like Jepsen uh or Jepsen test suites which are um which are very prominent in testing practically all distributed databases uh in the world. Um so, that

third test category that I have uh listed there, uh it's it's fault-focused test. Uh it's essentially uh means that you you run a system uh like a Kafka cluster or a Kubernetes um and then you introduce faults. And then you test client observed behavior to see if your system is still uh responding uh to client. So, this deterministic simulation testing uh actually sits uh into and enhances

this third category of uh category of testing. Uh we'll run through uh a very simple quorum-based read and write uh example uh to demonstrate uh how deterministic simulation testing helps uh you in in in testing client observed behavior when you introduce faults. And just to recap when when I say quorum based read and if you have used any distributed data you know that typically your read and

is called successful when so you manage like multiple replicas for for your data. Like if you are using Kafka, any message that's produced it's generally replicated on number of replicas and that that number varies from three or five typically. And you call it a success when the data reaches at least majority of the majority of the replicas, right? And that's essentially called quorum mechanism. So in this

case, we we have a cluster with three servers. And the names of the servers are Athens, Byzantium, and Cyrene. Those are Greek city names. And when this user Alice is writing this data title equal to be it's it's a simple key value. after dawn You can see that it's not getting response immediately. It's it's first returned to the server that it reaches to. Then it's replicated on

other replicas. So these three are the replicas where data is replicated. And even when one request is failing like that that second request that's that's going to the Byzantium server. This coordinator which is Athens it checks if the request is replicated on majority of the servers and then it responds to the client, right? So this is the basic quorum mechanism. And then the read uh works same

as right. Whenever a read request goes to any of the servers, it first checks with all the replicas. And as long as it gets like the response from majority. Now, in this case, you see that the response it gets on is only from Athens and Byzantium. It it will return the response to the user. So, this is Now, whenever we talk of things like quorum and replication

and in general distribution of any sorts, it becomes tricky because there are partial failures. And and when I say partial failures, you saw in the last slide that out of three, one of the replicas was unreachable, right? So, the entire cluster is not down, but part of the system is unreachable. And distributed systems, they are essentially called distributed not just because you distribute requests or distribute data,

but they remain responsive even when there are partial failures. So, part of the system is down exactly when the client is sending you the requests. And that also makes distributed systems extremely hard. Because failures overlap live traffic. As and when your your client is sending you the request, your part of the system is going down. >> [clears throat] >> Now, chaos engineering is is or chaos monkey

as a product which made it popular by Netflix um is one way to test uh these things, right? So, what what chaos uh engineering does or chaos Monkey as a tool does is you can see that there is there is a component um which will introduce random failures. So, while the client is sending you you can see that that there is this Nemesis component which is uh,

killing a particular node. It's killing that siren. you're still getting the the the success back. So, what you will test is you will you will run this Chaos Monkey or other tools in the Chaos engineering uh, category. Um, you run your cluster, you send requests, and then you you introduce random failures. You you bring some of the nodes down or you kill some of the processes. Uh,

and you still expect that uh, you're getting the response back. Now, the focus of this category of tools, which are which are Chaos engineering tools uh, focus is to see if the system remains available and responsive. So, you will you will still check if you're getting uh, let's say if you have a HTTP service, you're getting a response 200 okay. Uh, or your P99 latency is within

acceptable limits. Now, the tricky part particularly with distributed systems uh, and distributed data systems against like like Kafka or MongoDB or or any other uh, database uh, is that it's not enough to see if you get a response. You need to also confirm that you get correct response. So, availability is is not not enough or checking availability is not not enough or just checking latency is is

not enough. >> And that's where your typically the feature and performance test that you run. Um so you run that with entire cluster up or you run that with this kind of chaos engineering tools killing random They are not enough because they don't check for system correctness. Uh and that's where there is this Jepsen uh framework which is again a universal standard almost for testing any distributed

data product. So you pick a name pick a name a database and you will see that there is Jepsen suite available for Uh and if you if you are tracking any of the open source databases uh you will also find typically in their documentation that this particular product has been tested by Jepsen and if there are any um bugs which are found uh and how those are

fixed. So it it's it's part of their documentation. Now what Jepsen does differently uh than the basic chaos uh engineering mechanisms uh is that it not only has this component which introduces failures but it also has a mechanism to track what clients observe. you can see that in in this case you have like two clients Alice and Bob and Alice is writing to this cluster uh where

there is this initial state of the title key is say Before Dawn and now you're writing After Dawn. So you replicate just like before and this Nemesis component which is introducing random failures it it introduces failures. >> And then the second user it's it sends some get requests, let's say. And it gets some response back. It will either get initial state or or later uh depending on

when it executes and we'll we'll see why this is the trickiest part uh to deterministically say which one executes first and which one later. But what this uh Jepsen suite does essentially is it tracks what clients are observing. So, do clients get uh value as as before dawn uh or they get this value as after dawn? And then it runs through some of the consistency checkers uh

that they have to verify if the consistency guaranteed by this particular clustered product is achieved even when there are Now, the interesting uh thing to see here is that uh you must be knowing that there are a lot of formal methods uh and and specification languages like TLA+ in in the context of distributed Uh obviously, all these products which are which are heavily used in production systems

like Kafka. Um they're obviously tested uh very heavily, right? So, there are uh you you might be knowing this term test pyramid. So, there are lots and lots of tests uh written for these products. And that's why they are uh they are uh considered and used very heavily uh in in production systems, but even then, you will see that when there are Jepsen tests run against this

product, you will see there are some or the other bugs found. and that is the that is a very interesting thing to see strong specifications which are which are verified, even when there is a test pyramid uh proven test pyramid and and lot of different kinds of tests written for this open source product. Even then when you run a system and introduce random failures, there is a

possibility that client expected guarantees are not fulfilled. Um and which is which is something to remember particularly in the LLM era when people talk of spec-driven development and treat your code almost as a black box, that systems which are coded for 10-15 years, tested heavily in production, even they get bugs uh in the in in the in the case of random failures that happen in in production

systems. Uh and that is the reason your Jepsen tests or Jepsen kind of tests are always with your system running and introducing random failures and then client verified um behavior. Now, there is one issue with uh uh with Jepsen kind of testing or chaos kind of testing. And typically how this chaos or Jepsen kind of testing is done uh is you run your entire cluster and then

you introduce random failures. And when I say introduce random failures, you can either crash a process, you introduce network lag, you partition certain network paths, you crash a disk, and and stuff like that. The The challenge with randomness is that if you run it once by introducing random failures, and you you find certain bug, and you want to rerun it because you want to debug, uh or

you want to again see why the failure happens. It's extremely difficult to reproduce exactly the same set of events and exactly the same set of responses um when you run it twice or thrice uh again and again. And and that's one of the biggest issues uh with practically any testing mechanism um in distributed systems that you cannot reproduce exactly the same uh set of responses or exactly

the same behavior when you run it multiple times. we'll just see one example of why you don't see exactly the same responses or exactly the same behavior, or you can't guarantee. You might get or you might not get. And we'll [clears throat] we'll again go back to the same example that we have like uh we are writing um this key and value title equal to After Dawn

in a clustered um data product. and we are replicating that. So, uh this Athens is replicating that on on other we crash a particular node. So, in this case, we are crashing this node called Siren. And we get back response from from Byzantium, right? So, we are introducing only one fault that Siren node is is down. And in this case, because we are able to achieve quorum

because you get response from this other server Byzantium, which uh which which uh allows us [clears throat] to uh check for majority and and return the But what if there is a second run in which that response from Byzantium is now delayed. So in that case, it might happen that um client request times out. Right? So we we introduced only one failure. The other delay that happened,

it was probably a small delay um in in network delivery. So this is the reason, and we'll see there are other sources of this non-determinism in any computer system that makes it extremely difficult to to rerun exactly the same thing and expect the same response back. >> So let's look at uh some of the sources of this non-determinism. So as you can imagine, there are there are

four four things. There is system clock. So as you know, any um has a clock chip and and from which your time of the day or um uh is is determined. Then any system essentially runs on multiple threads, and threads are scheduled on on CPU. So there is some thread scheduling happening. Uh then there is network, and there there is disk, right? So essentially CPU, memory, network,

disk, four four physical components. Um whether you're running in cloud, running containers, uh or doing anything, uh these are the four four things that you need to worry about. >> And we can see now uh this one example. Uh, and let's say this client is sending a message to the coordinator. This Athens who was coordinating replication, let's say that we we call that as a And you

check whether a request time times out by essentially setting a timer, right? So, you check um if you get the response back in in a given amount of time. Uh, if not, you you uh time out that request and return uh error to the user. Now, this clock uh when you rely on on system clock, uh the tricky part is that there is no guarantee on how

fast or how slow uh your system clock runs. And that's the reason you typically have services like uh uh network time protocol services which are continuously running uh on any server. They run even on our laptops. Uh, and they sync your local clock to a clock uh o- over the over the network. But, let's say this this clock is slow. And you receive a response back. So,

there is no time out that happens. Uh and you return a success. But, let's say you run it again. And now your clock is advancing faster. It can very well happen that your time out fires before you receive the So, two runs without you doing anything just because how the system behaves, the clock you get you get two different responses, right? In one case you got the

response back, in the other case you got timeout. this is one of the biggest issues, unsolved issues you can see you can say. And if you see source code of any of the popular products, you will see some work around to make sure that this issue doesn't doesn't impact your your timeout And we'll I mean just to demonstrate how how you can check this in the code,

I have two code bases that we can look at. So, one is Apache Cassandra. And Apache Cassandra, are you aware of Apache Cassandra? It's a it's a distributed key-value store. You can see that there is this component called failure detector. And this failure detector essentially checks if uh if other nodes are down or up based on the heartbeat that they get. Now, what you what you can

see here interesting check that that they check So, what you typically the the way you can expect typically how you check if the timeout happened is that you get the latest time, you check the last time you received any response. And if that diff is about timeout, you will you will say that it it timed out, But there is this additional check here saying that max local

pause non nanos. And what that and it it's it's a homework to you to check in Cassandra code base why do they have that kind of a check. But why they added that is let's say if if if before this line executes, there is a garbage collection pause in in your process or let's say this thread is not scheduled on the CPU. Then even when you got

the response back just because you were not executing this part of the code incorrectly mark as as failed. Right? So so you will find this kind of a check um in in other products as well. Cassandra is is a very good example. And a lot of newer products like if you Have you heard of etcd? It's it's used in Kubernetes clusters as a metadata store. So etcd

[clears throat] runs this algorithm called raft. And any distributed system obviously has this concept of a timeout to check if other nodes are alive or not. there is this thing called heartbeat tick and election election tick. And if you see where they are used, again that's a homework to everyone. Yeah, but the reason that they have it is they don't rely on system time to if the

timeout has happened. they configure timeout in terms of logical ticks. And logical ticks is essentially just a number of times a particular method has been executed um uh after we got last last heartbeat. And it is just a plain integer that you increment every time that that method executes. >> So this is an example of how um how you just can't control uh some of these parameters

like like Other source of non-determinism is the way the threads are scheduled. So So let's say there are two two users and one sending the right request and one a read request. Now in first run, let's say the thread executing the right request scheduled first. And what that means is that it could store the value. And then the thread executing the read request that's scheduled later. In

this run, that read will get the value that was stored. But it might very well happen that you fire exactly the same requests in in same sequence. And the thread scheduler schedules your read handler first. Or the thread that's handling read request first. And thread that was writing later or after that. then two exactly same runs like two you fire two exactly the same requests at the

cluster just because the threads are scheduled in a different way, you will get two different answers. The same is true about network and disk. Uh you can see I will I'll just have have same slide showing two runs side by side. So in the first you see that there is a replication that's that's sent to a read replica. Um and then there is a user called Bob

who is reading from the In the first run, because the message was delivered before the client request was handled, the client Bob is or the user Bob is able to get the response. In the second run, the network delivery is delayed, and it can happen for any number of reasons. Maybe the network buffer was choked uh, when when you ran the second time. And in that case,

you see that the user did not get the latest latest, update. And the same happens with disks as well. I mean, if you know uh, the the storage architecture, lot many layers between. So, when you when you write, uh, code like file .write, and you write some data, there are layers from operating system to actual storage device where your data is stored, either a solid state drive

or a mechanical hard drive. And just that write call does not guarantee anything. So, if you if you just write, uh, and your process crashes, there is absolutely no guarantee that, uh, your data is actually available on the disk. You actually need to, uh, invoke, uh, a sync API. Uh, typically operating systems have something called as fsync or file sync, which synchronously, uh, write data to, um,

the storage and wait till your storage device actually confirms that, uh, your data is uh, stored on the on the device. Um, and, uh, by the way, there are a lot of corner cases, uh, even with that fsync, uh, that were found uh, very recently. Yeah, even in very mature products, uh, like Postgres. But, you can see that two different runs uh the first run you you

crashed after that fsync was successful. And in the second run, the process crashed before the fsync was successful. So, in the first run you could recover that data because the data was actually persisted on the physical drive. Um in the second run, you could not because the fsync call failed. So, this this four things uh they make it make it deterministic that even if you run your

program multiple times um firing client requests at it. And when I say program, a cluster and cluster running multiple processes. Um and you introduce failures and you fire client requests, uh you get exactly the same responses every time you run uh run that. So, we need to find a way essentially um to work around these non-determinisms uh which are baked into your system architecture uh or or

hardware components essentially. And to do that um if you see, I mean, we saw that etcd code where the the parameter name was election ticks, uh right? Or or heartbeat ticks. Now now we'll see where that tick term comes from. Um but, if you see our computer subsystems, like there's a uh unit and and and network controller and network uh unit and then there's a disk controller

and disk unit. anytime anything executes, uh what what typically happens is that uh it have executes uh on a clock tick. So, there is a continuous clock signal going to your CPU, and when it receives a clock signal at at that signal, it reads instructions and executes them, right? So, clock cycle um uh based execution. So, I mean, just to demonstrate when when you are we are

executing our replication requests uh or or client requests, you can imagine it uh that the when they are actually executed on the CPU, which are still scheduled on the CPU, you you are receiving this this CPU tick uh when when they uh when they execute on When you do any network operation, um essentially your data is sent to your network interface card. this network interface card uh

or network subsystem uh has a network controller, which will decide when your data is pushed onto network and delivered to a particular process, right? So, it it also happens uh with its internal uh internal logical steps or something guiding its internal logical steps. The same is true about about disks. Um so, do So, so, logically essentially you can you can imagine that each of the physical subsystems

uh in our computer, they have something that that decides when a subsystem takes a next step. Uh in in case of CPU, it's about executing the next instruction. In case of network, it's about uh actually pushing the network data to a process uh or sending it over the network. And in case of disks, again, the same. Either you acknowledge the right to the uh operating system or

you actually uh take a step to write or persist that >> Now, if we need to control when these components make progress, obviously we can't control the hardware, But essentially if we need to remove this non-determinism from from this execution, we we need this control. We need precise control on when a particular subsystem makes progress and when when process executes an instruction, when network message is delivered

or when persistence happens. So, as as we said before, essentially these these four things need to be controllable. We need precise control on on these four things, system clock, process execution, network delivery and and and disk persistence. So, what we can do essentially we we stub from the program that we write um uh and storage. And then if you make sure that all the execution happens in

a single main thread and we don't use any other threads for the critical state update that that we are doing or the main part of our program, then we can control sources of non-determinism uh in in in a precise way and we will see in in code how that happens. But let's say uh we are running a test and that logical tick source that at a hardware

level was happening with a clock tick or a network controller or a disk controller. Let's say if if you can model our process or model our program in such a way that you can you can pass it through your test. So you when you say next tick the next execution happens. when you tell your simulated network that okay, progress now at that time your message are messages

are delivered. when you tell your simulated disk that now you need to persist or acknowledge the persistence when that happens then you can possibly write a program. And when I say program, a program like Kafka, a program like etcd, a program like Cassandra. So one thing to note here is that the the programs that we are talking about they are not toy programs, right? So So exactly

the same Cassandra code or exactly the same etcd code if we code it this way the same code that executes in is the code that that we are testing. So we are not writing any different code for testing some of the mechanisms. We are testing writing exactly the same code that runs in production just writing it in a in a slightly different way. what we can do

is we'll let let's look to see how it actually looks looks in the code. So I will just demonstrate um a small unit test. Uh I mean this is not really a unit unit So what this is is exactly what we saw in the sequence diagram where there is a cluster of three processes, Athens, Each process essentially is is of type quorum KV replica, which is essentially

quorum-based key value store. And then we have multiple clients. In this case, a single client. a client is sending a set and a get request. Uh in the second test, it's a But we are controlling how this message moves from client to each of these processes. And how it's it goes on the disk and how the response is is returned back. So you can see here that

we set up this cluster which is which is the entire cluster running in a single process. And then we progress this cluster through this ticking mechanism. And we can we can check what's what happens inside a ticking mechanism. But first we can see what this quorum KV replica looks like. And it's a simple key value store which handles different client requests. And each client request essentially is

broadcasted to multiple other replicas. >> And essentially when the winner when a server handles a particular internal request uh Sorry, not not this one. Yeah, you can see that it's it's it's storing it in a key value store, which is which is here this this key value >> So, this key value store is um is essentially a replica and and this is the the part of the

framework that um I have coded called tick loom. And what this tick loom does is it incorporates all the building blocks of writing this kind of a tickable uh tickable software uh which which then you can test um um in a in a deterministic way. So, every component in this tickable uh you can see that uh it it extends this tickable interface and and this tick method.

You can see that uh there are different Um we'll we'll look at one of those. this network and you can see that there is a a real network, which is a non-blocking using Java non-blocking IO. And then uh a simulated network. We'll see what happens in in a non-blocking IO network, which is not a in a non-blocking IO network, when you use and which is a real

network, so when you will run processes um using this framework, you will actually configure this as a as a real network. But this is where you to see what kind of messages are um are received or what kind of messages they need they need to be sent. In a simulated network, what you can do is you can introduce different kinds of failures. So, from your production code's

perspective, it doesn't know if it's using a simulated network or a uh a real non-blocking IO network. But what this allows you to do is uh you can look at another test. So, if you if you look at this read test, what we're doing here is introducing a failure. So, we we say that we you partition nodes, these two nodes from that third node. Exactly the scenario

that we talked um in the code. And you can verify what happens when this kind of partition is is introduced. So, this is just a basic basic thing how you how you can set up The interesting thing um um happens when when you can actually keep your cluster running for a particular duration and you track all the responses that your client is and then you pass it

through a verifier. So, we'll we'll see what happens um with those kinds of tests. The front is two. So, you can see here a test called fault injecting simulation runner test. And what I will do is uh I I will just run this and and show what what happens uh when you run this particular test. So, we are running it for 120 ticks. when you run this

kind of a test, you see uh there are there are two kinds of things which are tracked. Uh what are the faults introduced across uh the cluster and the other is what operations were invoked by the client and what is the response that they that they got back. So, this particular history, I mean, if if you explore Jepsen kind of a you will see that when when

Jepsen tests Kafka or Cassandra or or other like products like MongoDB, it essentially verifies this kind of a history, exactly this history. So, you can see here that uh client two first invoked a read. Then there is client one that invoked a write uh with some random key and a random Um and then client two got a response back from its read uh and the value was

this. Um and you will get this entire history for this entire duration this test was And the interesting part, the deterministic part of of this execution is that even if I run it multiple times, the the faults were introduced in a random way, you will get exactly the same history. So, the client observed behavior, multiple clients that were getting the get and set responses, you will get

that in exactly the same way. And what how this helps when you keep this kind of a test running, let's say for for a couple of hours, and at some point you get a a failure, that means your history checker or consistency checker flags that something is is not right, you can run it exactly the same way again and again. and you are guaranteed to get exactly

the same set of responses back. And the same set of failures introduced Question. Is it used for all the kind of all the kind of all the kind of all the Any No, any distributed data system, you can say. Yeah. Not for the domain application domain. Not for the application domain, yeah. I mean, you can use it for um I mean, if you if you are really

worried about uh what uh what values you get back uh and correctness of those values, but typically what what happens is that your application relies on a particular product, like MongoDB, for example. So, it is important to check for the behavior of MongoDB kind of products. Um and what kind of guarantee that they provide uh when failures happen. Is it different frameworks that you That will produce

non-deterministic results and then write to the It will It will produce non-deterministic uh results, yeah. So, if you have a particular set of framework that is tied to a particular Yeah. Yeah. So, the thing is that uh uh a lot of these products uh they are not designed determinism in mind. So, so Kafka or Cassandra, they will use actual threads. Uh they will they won't have a

seam to actually stub out your network or disks. Um there are some products like etcd we saw. It was partially built like the only the timeout uh heartbeat timeout portion of that is isolated uh into these kind of ticks, uh logical ticks. Uh but there are some products like FoundationDB for example uh or TigerBeetle database, they are designed ground up with this kind of deterministic testing in

mind. Thanks. Sort of follow up to that Uh follow up to that question. So, I I don't know we will ever tinker with this kind of things ever in our day-to-day work, but we have a system where um we use Hazelcast. uh we it's a distributed system, and we sub I mean it's not like the same task is done by all the bots. Yeah. There will be

our system is not we do not have like um an external cluster of Hazelcast. Every pod is a Hazelcast cluster, and we just submit work to each pod, and the challenges that we are facing is like like if a pod goes down, we need to be able to retry, but Yeah. fundamentally we have an issue of idempotency. We need to fix that. But in either of those

cases, can I use Jepsen or any sort of testing framework similar to that one to figure out if I'm getting the right results. Yeah. So, so I think Hazelcast is tested with Jepsen. >> that's what I checked now actually. Yeah. Yeah, for Jepsen Our system is like yeah, multiple clusters. Right. So, your system is relying on Jepsen to provide a certain uh sorry Hazelcast. Hazelcast to provide

certain guarantee. Correct. And you can definitely verify with Jepsen if uh Hazelcast is providing those guarantees. No, I am I didn't We are not checking that one. I don't want to check that necessarily. Like I just want to check the resiliency of the my system. Like for example, if a Hazelcast goes down, am I retrying it properly? Yeah. Retrying it with a different cluster. >> Yeah, so

you will need to build a test suit like this um introduce a failure and uh and check uh verify if your system Okay. is behaving properly. So, you need to build it yourself. So, even with Jepson Uh-huh. for example, for each product a test suit needs to be built. Okay. Because you need to precisely configure what kind of client requests are sent and what kind of responses

to expect. Because this checker is application specific, right? What what responses you send and uh sorry, what request you send and what responses you get. You need you need to define that. Is that a considerable effort like It is. It is. It is. Yeah. Once Yeah. So, one of the other things with where I hope a framework like Tick Tock can help is when we read books,

particularly distributed system books, right? Um a lot of problems that are discussed in distributed system books are these kind of um non-deterministic executions like clocks running fast and slow, messages getting lost, messages getting delivered late, uh storage getting lost and stuff like that, right? And there is no easy way to actually write code and see what happens when these kind of events uh occur. Um with with

Tick Tock, I think when you read a particular portion in a book, you can actually write code to see what happens um when the scenario in the book uh is is you when you read a particular scenario in the book. So, I have I think one of the examples I think we are most out of time, but uh I'll just show you one one example. Yeah, so

uh are you aware of the book called designing data intensive applications? It's it's a very popular book, right? And and that book has a a section on quorum based systems and a limitation of quorum based systems and where it explains why basic quorum mechanism like what we saw cannot guarantee consistency condition called linearizability. Right? Now that particular scenario when you read it in the book and and

this is exactly that scenario that you have one writer. Again, there is a cluster of three nodes. I have named those nodes exactly the same And then there are two readers. This writer is writing a value x equal to one and it sends it to Athens. It's applied to Athens, but it's replication requests are delayed. And while this right is ongoing you have two readers reading this

value from this cluster. This Alice reader I mean the request can reach Athens and let's say Byzantium. But Athens has applied this x equal to one. So this Alice is able to read the latest value. But the second user Bob it can reach only Byzantium and Cyrene two out of three. So it still gets a successful response But this Bob gets old value of x. And after

Bob's read request completes the replication the delayed replication requests are delivered to the remaining of the network. Now when you read it in the book that designing data intensive application, obviously you need to imagine right how how this delayed execution happens. Now, what you can do with Tick Tock Boom code this scenario. So, you can you can set up a cluster with three nodes. You can set

up two uh you can introduce a delay of exactly those two messages which are the replication messages. And because this is also integrated with uh Jepsen, what you can do is you can assert on the the execution history that you So, that's that's one of the advantages. Like, even if we are not we are never going to code a system like MongoDB, at least when you read

a particular documentation, you can write code and um evaluate uh what might be happening. And by the way, with LLMs at hand, um writing this test now is very easy because what you can say is you describe a scenario, and because you already have a framework like Tick Tock Boom with with an example test, uh LLM can very quickly generate a particular scenario test for you. Cool.

I think we're almost at the end of the time. So, any any questions? Yeah. The scenarios, what do you have so And I Yeah, yeah. So, predictable It's It's a It's a predictable order because we are stubbing the network and You have to tell the user that you're sitting with a simulated It's not putting it in the It's a similar I mean, it's it's set up in

the same process by stubbing network and disk, but the actual code that you execute is the production code of uh So, each um each object of that quorum KV replica, for example, it won't know if it's running in a in a real cluster or or a single process in a test. But that's the capital to provide. Yeah. No, not Jepsen tickle them. So, in case of that

chaos monkey we are depending on like introducing a random Yes. Here you are controlling Here it's controlled randomness. Yeah. Essentially, you need to take the You need to take this Yeah. Yeah. You need You need to design your code in such a way that uh you you you stub out different things and you introduce the failures. Yeah. With every you just please you just have practice with

I think that's the outcome here. The thing is you have to come up with like No, okay. So, for data systems, the industry practice is to use Jepsen. Uh but Jepsen as of now at least it doesn't have determinism in the sense Jepsen test with random failures once uh no guarantee obviously that you will uh run the next time you run it. Uh and that's the reason

you have like TigerBeetle or or etcd partially. they try to code it in a certain way multiple runs of uh your cluster they they uh produce exactly the same trace. I think it's feedback to identify Since the responsibility to identify the scenarios uh which can go wrong lies with the application owner. So, is there a way or any framework for which can help us to come up

with that? Definitely I don't know Come up Come up with scenarios? Yeah, come up with scenarios because it >> No, I mean it's Yeah, it it is difficult. Uh it is difficult in a way but and that's the reason you will you will see that even seasoned and uh heavily used products uh which are like products like uh etcd for example. Which is used in production for

10 15 years. There you will still find there are there are certain bugs because coming up with these scenarios uh is I think the hardest part. Particularly with distributed systems, there are um there are specification languages like uh and what you do what you what you can do is you actually specify your algorithm up front in TLA+ and um it will that TLA+ tool will test that

algorithm thoroughly through all the all the different variations. but obviously that's not enough because after that TLA+ algorithm is is verified the specification, you still will have your own code written in a different programming language and it can still have bugs. And that's the reason Jepsen tests are are so popular. Yeah, but those scenarios I mean that's the tricky part. I think. Cool. Thank you. >> [music]