Beyond Stateless: Distributed Transactions with Autoscaling and Consistency on K... Jumpei Nishitani

This talk addresses the long-standing challenge of achieving both strong consistency and seamless autoscaling in distributed transaction processing on Kubernetes. Jere Nishani, a software engineer from Hitachi, presents a new approach that reassesses traditional architectures, highlighting the fragility of online transaction systems and their vulnerability to failures. The session explores how the Paxos transaction orchestrator can facilitate distributed transactions while maintaining agile scaling capabilities. By reengineering the transaction management system and employing a generalized state machine, the architecture gains both responsiveness and resilience, enabling efficient transaction handling without compromising on speed. The speaker also demonstrates the effectiveness of reactive programming in enhancing performance metrics and emphasizes the simplicity of a unified API for developers. Ultimately, the talk conveys the possibility of harmonizing the principles of strong consistency and scalability in modern cloud-native applications.

Hello everyone and welcome to CubeCom. Uh let me start with a question. Uh how many of you have ever had a choice between the strong consistency and scalability? How many of you have been told you can have one but not both? For year uh we've told this is a fundamental trade off. We just have to accept. Well, today I'm here to tell you that this is the

first choice. My name is Jere Nishani and today we will go beyond stateless. We will explore the practical way to get the both distributed transactions and seamless autoscaling on Kubernetes. This is a story of how we challenged one of the hardest program in modern architecture. A little about me. Uh I am a software engineer at Hitachi and I focused on the distributed system and transaction processing and

I have worked for years on the challenge of the building reliability large scale system. This experience led me to the work I will present today. And here is a roadmark for our session today. We will proceed in three main steps to discuss a pra practical method for achieve strong consistency and seamless autoscaling on kubernetes. First we will take a deep dive into the online transaction needs and

the painoint structure of its state machine. To solve the problem, we must of course first deeply understanding it. We will build a shared understanding of why traditional system are so fragile and why they are so difficult to scale. And next once that problem structure is clear, we reframe our central question how to overcome the state machine scaling problem. And this question was a starting point for all

of our research and development. And finally uh I will present our answer the transaction processing mechanism of the Paxoscom transaction orchestrator. Here I will explain how transaction monitor integrates with the HPA horizon scaler to allow status business application to the scale efficiently while retaining a sheet property based on the real world experience and example. This session will offer the actionable insight in the system design and deployment.

So let's start with the first item on our agenda. Online transaction needs and the painoint structure of state machine. Online transaction processing is an IT system that interactively update data for things like a payment contract or settlement. Here online sh rem means interactively or synchronous but OLTP has a building challenges. A failure in update control can cause the loss or corruption of the business data especially that

involved money. This can have a major social impact. And the most difficult part is this. Even though they are so important, these updates are very weak against the system failures and network delays. They are important but fragile. This dilemma is our starting point. To understand the problem better, let's establish a common understanding of the typical three tire architecture we often seen in the enterprise system. At the

top we have a presentation layer. This is what user directory interact with the web front end whereby the react or view mobile applications and more recently conversational interface like AI chatbot. This layer demands a relatively fast cycle of the change such as a UI UX impro improvement. In the middle we have application layer which is a business logic. This is where microservices often running on the Kubernetes.

It's where new future and specification change happen. And today even faster innovation is happened right here such as autonomous decision making and personaliz personalized by the air agent and the bottom is the data layer and this is our system source of truth the system of records. Here you might find a traditional relational database or in some case it main frames. So if we want to innovate why

can't we just change these layers? The reason is that many services still giant monoliths. It is because the switching cost to break up the mono is too high and this cost is mainly for the stateful parts in the sor layer like the database and the transaction monitor. So uh we are stuck with a moist because of the high switching cost. This lead us to our main question.

How can we problem in the online transaction? To answer this, we must go back to the basic to the definition from the gym grave the first of the transaction processing. So what is a transaction? Uh for this we look at how the great computer scientist Jim Gray explained it in his paper. He said the transaction concept comes from the contract role. What does this mean? He use

a Christian wedding as a typical example. Now uh after period of the negotiation the minister confirm that both people agree and then declare the contract complete and this idea is very important and everything decided by one final formal action and it cannot be undone. And from this concept of the contract row we can derive the following three important property that the transaction must have. First transaction most

of the legal protocols and second it either happens or it does not is all or bounded by a contract or non or third once a transaction is committed uh it cannot be obligated. When we apply this concept from the contract law to the computer science, we see that the most real world transaction can be seen as a transformation of the system state inside the computer. So next

let's look at the tool we use to perform this state transformation. So how do we implement the logical system state transformation? to answer is a state machine. A state machine simply define the current state and the rule of the transitioning to the next state when event occurs. Let's take a very simple e-commerce order process as an example. An order is first created in the play state and

next when the customer successfully completes a payment the order the state transition to the checkout and finally when the warehouse packs and ships the item the state transitions to shipped. As used in C, business operations are essentially a series of the such a state and using state machine to clearly define the state of the core business concept like order placed checks defining the business world themselves clearly

and robustly. So state machine is a very uh natural and powerful tool for the implementing the transaction logic we deal with. However, the powerful tool has a critical weakness. The state machine approach has two main weakness. First, the single state machine is a weak against a failure. If a failure happened during a state change, the current state can become unknown and this can stop the business process

completely. And second, when we try to fix this by the distributing it, we find another weakness to handle failure to handle fault. We often copy the state machine to the multiple node. But the message need to achieve these copies in sync can also fail. If a message is lost, the state becomes different on the different node. The whole system becomes inconsistent. Now imagine you scale this weak

model K3. You add the more nodes, you are the more replicas, you get more point of failure. At some point the complexity just exploded. It is no longer sh sample failure. You lose understanding of what is happened in your entire system. So what happens in the real world when you lose understanding as this diagram shows a distributed stateful part lead to the combin explosion. We can no

longer logically understand the whole system and the technical complexity is not just a technical problem in online transaction failure affect the user directory. For example, imagine an ATM fails in the middle of this impossible to understanding state. As a result, the machine might fall to retain the customer card. This is not just a bug. It is become a catastrophe. It can become a serious legal and financial

problem for the company. So how do people try to solve this text answer is make or node apply state change in the exact same order and this is what algorithm like a merge pack source raft and the job already do. So in practice using a tool like a rust based ATCD to store state can solve many of the these issues and let's look at the concrete example

of the textbook solution in a financial transaction. If all microser share the state using a system like ATCD then the service themselves can be status. This allows them to the redundant and scale. It sounds like a perfect solution and this architecture look like a sle but it is not just a simple uh online transaction must be fast usually under 1 second but atomic broadcast is naturally slow

and expensive as you add more microservices the cost of the saving state adopt this roll down uses up your limited time budget. This problem directly hurts the micros service scalability. As you add more innovative micros service, the cost of the consensus kept growing. Finally, the response time limit becomes a scaling limit for your micros service. So what should we do? Our approach was to step back and

completely redesign the trans transaction model from the beginning. So what does it mean to the redesigning the from the beginning? It means we must reflect on the transaction concept. Now let's remember the property from the contract law that we introduced. Now let's officially map them to the technical term. We all know the transaction must obey the legal protocols. This is a consistency itself. It either happens or

it does not. This is admission. And once transaction is committed, it cannot be a buged. This is none other than the durability. So you see what we call a seed is not just a technical acronym. It is a transaction of the universal real world concept a contract into the world of the computer science. Our approach started by you know the focusing on the just most fundamental principle

headon. Now how do we build our system on the deep idea? The key is a generalized transaction concept. In the diagram, you can see we change a specific business state machine on the left into a general technical state machine on the right where the states like active partial committed and committed or aborted. Then we take the next important step. We are revisiting a transaction for cloud native

system. We take that general state machine and turn it into a new conceptual component a modern cloud native transaction monitor. As a diagram shows a micros service now talk to this welldefined abstract component. uh this show evolution of our thinking and its core insight. As you can see in the diagram we have abracted the complex comp concept of the transaction into the two pure element asset properties

and state transition. By abstracting the transactions in this way, we lay the fundamental for scalable and resilient distributed system. Our solution is build on the very foundation and it's what I'm going to show you next. So what is the core technology inside the transaction monitor we build? The title say is a transaction processing mechanism of the pexos transaction orchestrator without scaling. We call it the PCT for

short. This is the engine that dering our new model. Let's see its main principle. The main principle is a basic change in thinking. You don't solve the scaling issue by tuning the storage or consensus. You solve them by rearchitecting the transaction management itself. So why was the rearchitecture necessary? In traditional systems, transaction management was often tightly coupled with a large middleware. As a result, the transaction management

function in itself becomes a bottleneck for the entire system scalability. So we decided to completely separate the responsibility of the transaction management and introduce a new lightweight and scalable component dedicated to the single task. And that's exactly what coordinator quum the PCGO is. And inside the new clean component we completely the generalized state machine we designed earlier in its pure form. This is a foundation that Paxos

commit provides. It is a core design philosophy of separating of concern and kubernetes make this idea pretty practical at large So let's see how this new architecture works with a sequence diagram. When annotation uh when annotated business method runs the PCTU library implicitly starts the transaction and send a begin message to the coordinator and next the application runs it business logic shown here as update for example

a withdrawal method. Then as the method is about to finish, the library request a vault from all service involved in the transaction asking are you ready to commit and collect the result. The important point is that at this stage each service is just setting its intention and whether it is ready. And here is the core of our The vote result from the each service collected by the

library. A broad broadcast to the distributed group of the coordinator the coordinate quorum. Then the distributed coordinators uses a vote they received to from a distributed consensus on whether to commit to a board the entire transaction. This means that even if a single coordinator fails the entire system does not stop. This robust distributed consensus inside the coordinator group is the truly identity of the Paxus commit. Once

this final decision is made by the Paxus commit, a commit message is returned to the database and the business message finished successfully. Each service board on the outcome and then the coordinators agree on one final decision. This is a foundation of our architectures mechanism and this design gives us another powerful benefit independent horizon on port to scale and look at the diagram on the left a transaction

volume grow the storage business applications managed by the deployment scale automatically using HPA. But the important part is warned right our quorum type coordinator managed by the stateful set can also be autoscaled independently using HPA. we can scale two different kind of the workload in different way automatically. This is very efficient. We also focus on the improving a transaction response time with a reactive processing. Faster response

time directly improve the user experience. To do this, we use a reactive programming with a spring web flax and separated the request and response thread. With this model, update can run in the parallel. The response time is not blocked by the slow databases update or by the number of the microservices. So what is the result? Even as a business microservices grow our architecture stays fast for two

reason. First reactive processing lets update run in parallel and second the final decision pass does not depend on the micros service topology as a result consensus reached in about 10 milliseconds every time now let's prove the impact of the two main strengths reactive processing and topology independent finality with actual data. This graph compares the response time on the reactive model versus non-reactive model inside of PCD architecture.

First, the non-reactive This assume classic broking model where requests to service are proceeded one by one. As you can see the response time gets a linearly worse as more service added. On the other hand with a reactive model even as the number of the application grow from the two to eight the response time is mostly flat at around 35 millisecond milliseconds. This clearly shows that with reactive

processing the number of the participant services no longer bottleneck for the response time and there is another important piece of the information here. The consensus time in the response time. Notice that this time is always constant and extremely short in the B model. This is the rectory proof that our topology identity independent finality. The Paxus come at the heart of architecture are always stable and fast regardless

of the number of the services. The combination of these two strengths is how architecture achieve both high performance and scalability. Besides the speed, we also focus on the resiliency. Distributed consensus enables a resilient transaction processing. This design show how on the right if one of the coordinators fed on the transaction processing, the surviving measure of the coordinators guaranteed a consistent result. The final this decision is still

made by this majority where a stable response time was about 10 seconds. Now after we push the performance and reliability this far you might ask is many application code going to get complicated to avoid that we offer a single API that is like a jakar eja developers don't need to worry about the complexity of distributed system and you just add one transaction annotation to your business method.

let's go back to the question I asked at the very beginnings strong consistency or scalability. For years we are told we had to choose. Today I hope we have shown you that this by going back to the first branch of the contract law by separating the promise of bashet from the mechanism of the state transition and by using the distributed consensus mary we have pro the strong

consistency does not have to limit the scalability with the right architecture you can have Course the technology and architecture we discussed today at the core of our work. If you are facing these challenges, we'd love to talk to you. Thank you very much for your time and attention.