Crypto Trading: JVM Techniques Behind Bitvavo’s Microsecond Revolution | Oleg Lobanov & Marcos Maia

In this talk, Oleg Lobanov, a staff engineer at Bitwala, discusses the company's evolution from a startup to a more mature organization and the resulting changes in their software architecture. He emphasizes the significant growth of the crypto market in Europe and how Bitwala is positioned as a regulated entity focusing on security and compliance. Lobanov goes on to explain the design difficulties encountered with the original architecture, particularly issues with latency, throughput, and deterministic processing. He describes their transition to a replicated state machine architecture, utilizing Raft consensus for reliability, and the adoption of Simple Binary Encoding (SBE) to enhance data processing efficiency. The session highlights the current achievements of Bitwala, such as processing high transaction volumes with low latency, and the strategic decisions made to ensure scalability and performance.

[music] >> Good morning everyone. And um glad to see you all this morning, early morning on a my presentation. I hope you have you had a cup of coffee before that. Otherwise, I'm going to try to enforce your brain to work a bit and >> [sighs and gasps] >> So, uh >> [cough] >> My name is Oleg Lobanov. I'm a staff engineer in in Bitwala. Um I

joined Bitwala 3 and 1/2 years ago and um my background is primarily trading and e-commerce. Um I have almost two decades of experience working in this in this area. Um yeah, and I was using different languages during and different platforms during my career. Uh the JVM, C, C++, and Golang, all of that. I'm originally from Azerbaijan, but I live 7 years in the Netherlands in Amsterdam. Um

so, let's um today I'm going to talk about the um our journey as as a company as Bitwala, how we transitioned from a more startup-like company to mature from the technical point of view. and how our software architecture um evolved during this transition. Uh it's interesting that um not a lot of people understand how big the the crypto market is in in Europe. So, we are essentially

the second biggest market in the world. And it's keep growing. And what makes the the Bitvavo stands out in among all the competitors? So, it's the regulator regulatory clarity. We are regulated company based in the Netherlands headquartered in Amsterdam. We keep all the data in Europe. So, all European citizens are safe. We work closely with all the regulators. We follow all the rules of the GDPR, MiCA,

um whatever is required. The institutional adoption. So, we provide [clears throat] different type of instruments to to our clients. Uh and the institutional adoption of the crypto trading is growing faster since the introduction of regulations on on this market. expands over the Europe and pay attention on localization and globalization. So, what does it mean? We take every country um individually and try to localize our projects on

this country. Um we built our platform in a way that's it can be easily scaled to scaled globally and support um a massive adoption among the world. >> [clears throat] >> it's interesting that Bitvavo is well-known company in in Benelux. We are essentially number one there. Uh but it's not well-known in in some countries. Um but the interesting fact is that we 50% in euro denominated spot

crypto exchange. So, what it means, we process 50% of the volume from crypto to euro market. Let's say you want to buy crypto with euros. Yeah, we process 50% of this of this traffic. Um so, some more information about about Bitvavo. It's founded in 2018. We have 500 plus employees. As I mentioned, it's not a startup anymore. We call ourselves scale-up. Um we have 2 million verified

retail users on the platform. And we operate in 27 countries in Europe. So, you can go to the to the to the Play Store or App Store, download the app, register there, yeah, make some trading. Um financial information, we have 7 billion assets under custody. We are also custodian, not only not only the trading company. And the yearly exchange volume is more than 100 billion the goal

of our company is to become the exchange number one in Europe. So, we want to be like a benchmark for other competitors. We have technical ability and uh um to to reach to reach this goal. So, if you didn't try Bitvavo before you or you didn't know about about us, please download the application and give some feedback. It's very useful. Um so, before I jump to the

to the technical part, I want to understand how many of you know what exchange is or can can you raise your hand? How many of you traded uh at least once? All right. And how many of you know the what what the order book is and uh Perfect. Thank you. So, let's go back to the uh when Bitvavo started as a startup. It was just a few

engineers uh working on on this product. Um and it's amazing how um they could manage to scale this product and take over the the Benelux market. Um this is the diagram of um uh of our core system, the exchange part of of Bitvavo, how it looked like. So, we had the gateway, which receives orders via web sockets and rest. Then we had matching engines, which basically store

and process um all the incoming orders, let's say the the market orders, limit orders, cancel requests, and so on. Also, the the user balances, the risk checks, it was doing everything. And uh the connectivity between um the API gateway and matching engine was done using the Redis Pub/Sub. So, when the user sends an order, uh it goes to gateway, then it goes to Redis uh Pub/Sub channel,

and um later it reaches matching one of the matching engines. So, we had one matching engine per uh per market, essentially per uh pair. And then matching engine uh stored uh this processed this order, did some risk checks, and stored it in MySQL database. Um then [clears throat] um um If the order matches with the with the counterpart orders in the order book, the trade happens, it

also was written to the database. And then we had division which reads uh from a database um bin log and um produces the the the the events to to the Kafka basically as as an audit log for our regulators and the post-trading systems. So, we have the log the entire log what happens which trades happened and what was the order transitions in the system. it worked. It

helps us It helps us to ramp up the system and because when you are startup, it's the your goal is to go faster on the market acquire user base. Uh but this is we had some system limitations. The one of the limitation of this of this platform of this architecture is non-deterministic processing. So, what does it mean? Um we had a database and highly parallel processing on

the matching So, what it means when it receives an order, it starts a parallel processing pipeline and then you cannot guarantee the mathematically the the order of So, it's really hard from the from the regulations point of view to to uh the mathematical proof that you received order A and then order B and you process order A first before the order B processing. So, the this illustration

shows you two images. On the first image, we have two threads. They process the the some inputs. Do some changes on the numbers and let's say write to the to the outputs list. So, we can see that in the first on the first run, the thread A took event A event one and event four while in the second execution, because of the scheduler on operating system, and

the spread A processed events two, four, and five, and they ended up in totally different order on the output So, the second limitation is the shared resources. Um since we had Redis and database at some point, we reached the physical capacity of the um of those systems. So, we scaled our platform to the most to the instances in AWS, but we we reached the physical limit. And

contention became a problem for us. Um I especially like this one, >> [laughter] >> unpredictable high latency. So, you see the cats and it represents the traders on the platform. So, what Why latency is important? We have few types of users on the platform. It's a retail users, essentially the mobile and web users. We have the professional traders, institutions, can be like banks or um other companies,

and also high frequency traders. So, we call them market makers. Those are the traders who bring the um liquidity on the market. And this cat represents the those market makers. So, what they do and why latency matters for them is >> they have an algorithm which tries to predict the price on on a market. So, they collect data from multiple exchanges, um and they try to predict

what going to be the price for the next milliseconds or 5 milliseconds. And they try they then they make a decision and place an order on a system. They always try to be on top of the book to to match [clears throat] with the the counterpart orders which come to the system. And if the system latency is not predictable, sometimes it's like 1 millisecond, sometimes 5, sometimes

200. Uh they cannot adjust the the the algorithm uh >> for that. And essentially they lose money. And uh the next limitation is the um the throughput and scalability. Uh again, so since we had um >> contention on the system, we had common resources like a database and Redis. Uh the throughput and scalability is limited. So even though we can scale horizontally, but there is a physical

limits because of the contention. And how the perfect system works uh look should look like in our um in our view. So we should have uh high throughput at least 10x uh we had in our legacy system. Uh we want to have low and stable latency. And um high availability and Uh because we want to we don't want to rewrite a system every 3 4 years. We

want to build like the a solid foundation that we can um improve um and scale uh scale later. And more importantly, we want to have 100% determinism. Uh it helps us for from the not only debugging but but also regulations point of view. So we can prove mathematically that um we how the the order processing works uh in on our system. So we started looking in around

and um these evaluating multiple um options. And we ended up using the replicated state machine. What is that? So it's a pattern where you have an a log of input events if you process this log on multiple machines in exact same order, your application should reach exact same state on all those machines. Um and what it gives us, uh so it guaranteed determinism um because if you

process the log again, you can guarantee that if you if you process multiple times your application reaches exact same um as as it reached before. Um lock-free execution since um this pattern requires you to use a single thread to guarantee the ordering of events. You essentially become lock-free and the execution is very fast in in this case because you don't have the the CPU contention um problems.

Um high availability um since the the pattern is called replicated state machine, we don't have a single machine, we have multiple machines. The log is replicated on all those machines. and we have exact same state on not only a single machine, if it goes down, we have we have more in a cluster. And also from the development point of view, it's the another benefit is the the

simplified debugging. What does it mean? So essentially, you can download the log from uh production run debugger on your local machine, start processing the log with with the exact same application, and you can put the breakpoint uh in the app and um validate why the state is different uh than the the one that's um that was desired. for um consensus, we considered like a different implementations, but

we ended up using Raft. Um so, let's look at what uh Raft is and how does it work on on a high level and why we chose uh Raft. So, let's imagine that we have uh five machines, five nodes of the cluster. One node is a leader and all the communication uh in this um in the Raft protocol um goes through through the leader uh leader node.

Let's imagine that the gateway sends a command set X to five, to transition the state of X to five. The leader node receives this uh this message and um distributes it to all the followers. When at least half um of the of the nodes re- acknowledge that they received this uh this event, then the leader commits this event uh to the to the input log. Uh and

distributes this commit command to participants of the cluster. At this stage, the the cluster nodes can um transition can process this event. Before that, it's in the log, but kind of uh waiting for uh acknowledgement. Um so, all the nodes transition the state to X five and then um essentially the client receives the response from from the leader that's the the state has been transitioned. the the

golden rule of uh absolute determinism. No, you have to follow few rules when you designing the the system in that way and when you using the um the state machine in memory, since the data is in memory, we have to be extra careful uh with that. So, we shouldn't have side effects. What does it mean? You shouldn't call APIs or do some rights, uh, Sometimes even reads

are, um, not safe. Why? Let's say that's, uh, the leader sent an API call to another service, and, um, the response was okay. And, um, when the followers processing this event, they also sent, uh, this API call to the service, and because of the network blip, um, that the response was was error. Then you have, um, a different state on, uh, on the nodes, and this is

the problem. Uh, because, um, you cannot rely on on this data anymore, um, and, um, your state is undeterministic. So, you should not, uh, use, um, wall clock time. You should only always use the logical logical time that that the Raft, uh, consensus module provides you. Why does it matter? Uh, the time is not, uh, exact same on all the machines. Um, if you have like five

five nodes, even if you use the, the most precision, uh, algorithm to synchronize the time, it still had the nanoseconds are not synchronized, which means that, uh, you might have o'clock on, um, leader node, and 12, uh, 01 on, uh, one of the followers. Um, and again, if you if your logic is relied on the on the time, and you have some jobs to to schedule, then,

um, yeah, you're going to end up in a different state on, um, on the on among your nodes. Uh, you should never use randomness. It's another problem. never use um, Java util random package. technically you can use that, but you need to be very careful. You should always seed the randomness with uh, the the sequence number of the input event. So, it's still going to be random

numbers for for your system, but it's going to be pseudo pseudo random. So, it's mathematically proved that every time you process the the input log, the you're going to generate exact same random numbers for every tick. And uh, the the next one is deterministic iterations. Uh, so we should be really careful how which data structures do you use and how do you use them. Let's imagine that's

uh, you use the hash map, which is commonly used the data structure among developers. And um, you iterate over you need to iterate over the the the the elements in the map. The iteration over the the hash map is not >> [cough and clears throat] >> is not deterministic in Java. it doesn't guarantee you the the order of processing uh, in a hash map over when you

iterate over >> So, what it means, uh, again, let's imagine you have like five elements in the map. The leader processes in 1 2 4 5, but because of the memory layout on the follower machine, when you iterate over the map, it processes the the backwards, which is the the correct [clears throat] processing from the spec, but potentially your state going to be different because of the

different um, iterations order. oops, sorry. we we chose the the data structure so the pattern for for our system, which uh is the the replicated state machine. We have the the state in the in the in memory. We don't have database anymore. We don't have Redis. Everything is fast. But, uh bottlenecks the next bottlenecks is the communication protocol. Uh initially, we're using JSON for um messaging coding.

Um but, it has a few limitations, I would say. So, the first one it's uh the it's text-heavy protocol. And um it's really uh hard from the computation perspective to process this um this this protocol to serialize and deserialize. It requires Let's imagine that we received the the the the limit order on a system, and we need to find the price and the the the Uh in

in order to do that, we need to traverse the entire message and find uh process all the all the quotes and parentheses. Uh and also since uh since in JVM strings are uh immutable objects, um it generates a lot of a lot of garbage uh and uh as we know, the garbage collector in JVM can be your enemy at some point, especially if you are working on

the low-latency applications. We started looking around. We uh assessed a few alternatives for for um JSON. We tried to optimize the the parsing in the JSON first. Uh but, we ended up using the the SBE. Um so, it's a it's a acronym from simple binary encoding. It's a library which current guarantees you the zero copy and zero allocation. It's schema first. Uh, it's a binary protocol. Um,

instead of using the instead of decoding the the the message into the JVM object, it uses a flyweight pattern. So, what does it mean? It doesn't allocate. Um, so it gives you access to the raw raw data in memory and you can read the the bytes and operate with those bytes directly. Um, the fixed offsets, it's easy for from the CPU perspective to utilize CPU caches. let's

say that's the in SBE you define that your price that's is at offset And your market is at offset 10. Then you can easily offset and read this data instead of trying to guess uh, where this data could be in in a non um, in other formats like like a JSON or protobuf. Um, again, it's cache friendly that's schema first. So, essentially speeds up the the processing

with that we fixed the the encoding and decoding. We ramped up our performa- performance even more. But at some point we started reaching another bottleneck and um it's a paradox. where the when the reliability becomes your your your bottleneck, your enemy. Um, what I mean by that at first we were using TCP over and we're using JSON over TCP. Then we transition from JSON to SBE over

TCP. Um but it still the TCP is very wordy. Uh it's reliable but uh it's you pay for that. There is a cost. Because in [clears throat] TCP when you send a message the the receiver should act that it received this message. It's very safe from in the beginning when you start a project um on the on the high loads on low latency you hit the limit.

can we use something else? So is there something else available um instead of TCP? Can we use UDP? But what is the what is the limitation? Why no one is not adopted globally in uh on the when you develop some services and communication among services. The difference is that UDP doesn't give you uh the reliability guarantees. So it's faster, it's uh the the datagrams uh are much

smaller than the packets in TCP. But it's basically fire and forget principle. So when the server just fires a message, it doesn't know if the recipient received this message or not. Uh but can we make the the UDP actually reliable? Yes, we can. What if we build the the reliability layer on top of the UDP as on on a user level? What if we add the sequence

number to every packet that we sent? In this case, the the recipient can um detect the gaps. Let's say the recipient received one, two, and then four it can basically say to the to the sender, "Hey, please resend me message number three." And instead of ACK-based system, when you ACK on every message that you receive, uh you trans- we transition to the NACK-based algorithm. What does it

mean? Instead of ACK-ing every message, we all we we only shout when the gap is detected. it speeds up the system. It will dramatically reduces the traffic the the the ACK storming on on a on a network. Um another thing we added the the periodic heartbeats. Since there is no physical connectivity in UDP, you just send the packets. You need to know if the client is still

active. So, >> periodically the every client sends a heartbeat message the high watermark. Basically, what does it mean? It sends, "Hey, I processed the up until 1,000, the sequence number 1,000." So, with that the the server counterpart knows that the client [clears throat] is still active, is alive, and it processed the the stream the packets up until yeah. So, we didn't want to reinvent the wheel. Yeah.

>> Where did you get the idea from? I have I have an idea where I got got this from. It's exactly from the same it's from the fixed protocol. They use the same technique over UDP. >> Um let's uh have questions in the end. So, I I put it on the stack. I'm going to answer to this question. Um so, we didn't want to reinvent the wheel.

We started looking on the market. we had some requirements. And we found the the the the Aeron. And in the first our CTO was a bit confused. It didn't seem to improve any latency on the system, but man, it's it's very comfortable. Anyway, it's it's another Aeron. So what is the Aeron? It's the this huge set of libraries which provides you the reliable UDP connectivity and it

also contains the the raft consensus basically covers our requirement for And that's why we we decided to to use Aeron as the as a backbone layer on on top of which we build our business logic and and our applications. this is how the system looks like at the moment. So we have a cluster of multiple nodes. every node has the the ingress log. It's the the log

of input messages that's the the cluster receives. It's the ordered log. Um then every node processes this log and produces the the the output log which is called archive log. So every order transition, every trade is written sequentially to to the output log. And then we have post trading systems. It's any type of service that's that we have in Bitvavo like the market data service, the legacy

services we still use legacy database to sync the data and sync orders and trades there. But we do it not on the hot path. We do it's um um on the post-trading uh phase with its own pace. So, it doesn't affect the the ordering flow. Even if the database at some point becomes slow, uh you still can trade and see the the active trades uh fast. Maybe

the history going to be a bit delayed, but um in the in most of the cases that's uh acceptable. Um so, in this case, the the post-trading systems, they connect to the output archive um log. They read the transition of the orders. They read the the the trades, and um they send either the response to the clients or the market data and um the audit log and

so on and so on. when you build the the replicated state machine, um be careful and you need to build the guardrails, which basically going to help you um on the on on production. So, the first um the first rule is the you should not use um configuration files or environment variables uh or anything like that for feature flags or configuration. Why does it Why does it's

um Why Why is it important? Let's imagine that you have a cluster of again five nodes. You flip the flag on the leader node. It It's still processing an events. And maybe your feature flag uh enables processing in a different way or um maybe it's um some new feature on a system, then the the state on the on this node going to be uh different than the

state on the the other nodes. Even if you do it instantly, you still there's still a chance that one order going to be processed differently on different nodes because of the So, if you want if you really want to do that, there is a way. The the feature flags and configuration should be part of your of your state. So, you should have at you should add the

the admin commands to the cluster. Um and you basically flip the feature flags by sending admin commands. It becomes the part of your ingress log. And when you rip replay the events uh up until event X, the feature flag will be disabled, and then it going to be enabled on all the machines in exact same order. Uh the upgrades um is another challenge that you need to

solve. What is the matter? We are crypto exchange. We operate 24/7. We don't have downtimes. deployment is is a problem in this case. >> So, when you deploy new version on on one of the node, there is still a chance that um you fix some bugs or you added new features to to the code, and um it might state transition and processing of the So, you need

to be extra careful again that your the new version of the code the the the historical events in the exact same way that the previous version processed it. Um again, if you add new features, please cover it with the feature flags, and then flip it using the admin Then you are safe. Um so, no historical rewrites. If something happens, it happens. You should never uh stop processing,

go back, and reprocess the the the input events in a different way because you already produced the the output. It was uh probably consumed by other systems, distributed uh in the whole company, and probably also sent to to the end users, regulation regulators, and so on so on. Um and you cannot simply go back and say, "Hey, now the the event X should be um differently, and

the state should be different." Instead of uh reprocessing the events, you should use compensation. Again, some admin commands that uh help you to transition transition the states, and compensate the the the failed uh state transitions. the the last one is the You need to detect the the divergence uh in a state. Uh ideally, you should have like multiple layers of protection. The first layer is your CI.

Since the the state mach- the replicated state machines gives you an ability to easily debug your system by pro- processing the input on any machine, why not to use this input on the on the CI? What you can do, you can um fire a CI job, which processes the the inputs from production. You can anonymize it. uh in the end, you check the states on on every

uh release. If the state is different, it means that your processing has been changed. And you cannot simply deploy this this version to to the prod. The The second layer is actually cham um checking this state transitions in on production. So, what we do, we periodically uh run the job which um compares the the state uh among all the nodes. If there is any difference, any diversion,

then uh there is an alert and we start an investigation and um we fix uh this divergence. we now have the the um the architecture that scales, that processes events very uh on the architecture layer, but since we are um on uh on JVM and uh we need to follow some some rules. What we did, we defined a few um a few layers. Uh the check if

you the checklist. Um the first layer is the is the memory hygiene. Zero we we don't allocate. We don't have unattended objects uh in the application. Um if you open the code uh of our core application, of our uh low latency pipeline, it doesn't look like a typical um Java application. Uh it's more like the the C or C++. So, we create an objects uh on the

startup. We pre-allocate the buffers. We pre-allocate the the object pools. We um allocate objects on on the runtime. We just reuse the the the allocated um uh memory. So, what it gives us, we don't the the garbage collection runs to be honest, we have some allocation on the system and uh we have maybe one or two garbage collection phases in a day, which is still fine. So,

another thing another rule is you should not use strings or boxed objects because again, uh strings are um are [clears throat] is a problem in in Java because it's immutable. Every time you you work with string, um it allocates uh and then the the garbage should be collected by the garbage collector, which affects your tail latencies. Um use uh bytes arrays instead uh and reuse those byte

arrays or if if the string is static, like the market's uh market name, you can just simply use primitives. Um you can have like a mapping on the gateway side that's BTC EUR has the ID on the on the gateway level, when it receives an event for to create an order, it's going to send ID one as a market to the cluster and then the cluster responses

back that hey, trade happened for user X on um market one. Then it's uh uses the same mapping to transition from ID to string and then return the response to the client. Use always the primitive collections. uh because they're faster, they don't allocate. There are a few open source libraries, the Agrona and the Fast Util. Um concurrency is another um layer of uh and it has like

a checklist in a in our um development pipeline. So, we always should have like a single writer. Why does it matter? Because of the contention. And ideally, you you have like a single thread which produces the data. Um and you always should use the the the wait-free algorithms to distribute this data to like other threads if you have uh if you have them. Otherwise, if you have

locks, and um at some point, the the system scheduler going to be your your enemy and you're going to hit the the contention issues. Uh and the then the last uh layer is the uh operating system and hardware. You you can pin your application to the CPU. So, in this case, the the scheduler of the operating system not going to touch this CPU, not going to schedule

anything uh else. And you go you're going to be going to have a guarantee that all the power power of the of the CPU, all the processing time going to be allocated to your application. So, you can process events uh faster and you can do the the networking operations faster. And uh then the um the next thing is the the the networking part. again, at some point,

even the the the kernel layer of networking becomes a becomes a bottleneck. In this case, um you have to go around. There is the kernel bypass technology, uh the the Solarflare or DPDK. Um what's how does it work? Instead of using the TCP stack from operating system, you directly connect to the physical hardware, the physical network card. You read the buffers and you send the buffers directly

to the hardware, which eliminates the spikes on your tail latency um >> uh >> [sighs] >> Um so I was in the era of AI, I was using AI for for this infographics and uh and to make it cool. >> [gasps] >> Did it help like all this this effort? yes. So, the end-to-end latency is less than a millisecond. So, P90 is uh 500 micros. Um P50

is even even less. Uh but what is it's not just the processing time, it's the end-to-end latency from receiving an order, sending it to our processing pipeline, processing it, and receiving the response from the processing pipeline, and sending it to the client. The internal processing time is just 5 so we don't have spikes because of the GC and we try to optimize the the processing algorithms in

a system. It's not just uh comparing two two numbers, it's actually the the risk processing, the the order book processing, finding the the counterpart orders, match them, and uh generate trades if there is any. And the current end-to-end throughput is uh 100,000 uh requests per second, orders per second, and it's not the the the physical limits of the platform. It's just the the the point where we

are we don't have to uh improve it now because the capacity of the system is way more than the actual production traffic. Uh and we can easily scale it uh multiple times by introducing shards because now all the processing for all the markets that we have on the platform is happening on a single Um so if we just shard it, we can expand the the throughputs a

lot. With that, uh I want to thank you all for listening my presentation. if you have any questions, um we still have uh few minutes left. Please go on. Yes. >> Uh I have a question about uh gap detection. You you showed that if you uh you have one, two, and four, and you miss packet three or message three, for example, and it calls back exactly Yeah,

like system missed the message So how does resending work? Is it like like resend out of order or like another stuff working under the hood? >> So um yes, the question is how the system basically resends the the message the the events number three when the it detects the the gap. The way it works, the the sender has a buffer of recent recently sent events uh recently

sent messages. Um and when the client detects a gap, the sender can go over this uh this buffer, find the missed message, and resend it via the same channel. So when the the the recipient receives this event, enables the the the application layer to access the input log, um and then start processing that. So completely hidden uh on the networking layer. On the application layer, you just

you just the stream of Mhm. of the log. You don't see all this um uh back and forth and resends. So, it's implicit for the client. And yes, so we had a question uh regarding the the FIX. >> Yes, so well, for me it was interesting to notice that exactly this protocol FIX uh for uh work over UDP. And normally they use it TCP IP, but you

can also operate it because exactly this mechanism is in the FIX protocol. >> Um yeah, so the question >> So, I don't know what you what implementation you are using here. >> Yes. >> But I think the idea is the same, and that's not surprising to me because it's >> Yeah, so the question is do is it like a similar to FIX uh and UDP in FIX?

I would say kind of. FIX is is a standard and doesn't define the network communication layer. Uh it can be anything. Usually it's TCP, but uh there is like FIX P protocol, which uh transitions a text-based FIX to binary FIX. It also uses SBE underneath. Uh yes, it also use you can use Aeron for communication in in FIX. And uh to make the UDP reliable as well.

Um I would say it's the Uh it's a new standard and uh it's not widely used. Maybe for market data, but usually for ordering yeah, the the TCP is used in in FIX. Any any other questions? >> Yeah, so I'm wondering how it works network packet level So, the question is if you have a big message and TCP has a limited MTU, so you can send limited

data in a single packet in a single datagram in UDP, how do we solve the the split and how the recipient part understands and reconstructs this this data again. So, we every TCP datagram has a sequence number. On the on a high level, the the network communication protocol, it doesn't know anything about the the messages. It operates with the datagrams and every So, if it detects the

the gaps in datagram sequence number, it requests the the sender to resend the the the datagram that it missed. Then on the application layer, combine the the multiple TCP sorry, UDP datagrams, merge them into single message and then process the this this buffer if needed if your message is bigger than the single datagram. But usually the advice is to have your messages smaller, so they fits into

single datagram. Yeah, it's easier for processing and debugging and all this stuff. >> So, why did you use Java for this and >> Excellent question. Why do we use Java? First, historical reasons. Since we started as a startup, most of the stack was written in in Node.js and the matching engine was written on Golang. So, when I joined the company, that was the state. And then [clears

throat] um we had an option either to use C, C++, Rust, like any compiled language where you have uh access to to the memory and um you control the the objects uh in the or you use Java where you have some tradeoffs, but you can easily tran- easier It's going to be easier transition from Node.js uh develop for Node.js developers to Java than uh from Node.js to

C and C++. We either need to replace the entire team and uh it's uh yeah, it's a it's a more problematic. Or we can have the the the rules and uh guidelines. And uh we can help developers and we can teach them how to write the the code in Java which going to be almost uh the same going to have almost the same performance as as C++.

Maybe like following some rules, uh if you look at our Java application, it doesn't look like the typical typical Spring or Java Java application. We don't use any framework. We use pure uh pure Java and we use Kotlin to be honest. I'm going to answer like why do we we use Kotlin. yeah, so we track the life cycle of all the objects. Um we don't have unattended

objects. We have we preallocate, we use buffers. So, in the end, I think we reach maybe 95-98% of the C, C++ performance. The tail, yes, it's it's still faster, but we we move faster in the development process and we don't need to replace the entire company developers to rebuild the system. And again, like why would we use Scotland? Because it adds some some sugar into the compilation

time as well. For example, you have inline functions in in in Kotlin. So, we can instruct the compiler to inline the function into the bytecode instead of relying on Java on JVM to do it on runtime. You can do it on compile time in in Kotlin. Yeah, so we are running out of time. Uh >> Say it again. Um yeah, the question is do we do we

use like the latest Java Java versions? We use Java 21. the the Valhalla and all those projects, they will not affect us a lot because we the processing is in a single thread. We don't use multiple threads and uh we did some benchmarks. Uh the most um the thing that helped us the most is the ZGC. When we transitioned to to generational ZGC garbage collector, so it's

guarantees you that even when the garbage collector kicks the the the your tail latency not going to be affected by that or like just tiny bit. Um all right. Thank you all for for for listening my presentation. I going to be outside the the the room for for a while. If you have like more um yeah, you're welcome to to ask. I'm happy to answer them. Thank

you all. >> [applause]