Building GPU-Accelerated AI Libraries in Java with TornadoVM, Thanos Stratikopoulos

This talk covers the integration of GPU acceleration into Java applications using TornadoVM and its application to AI inference, particularly with Llama 3 models. The speaker, Thanos, a research fellow at the University of Manchester, introduces the challenges Java developers face compared to languages like C++ that support GPU natively. He explains how TornadoVM allows Java developers to offload computational heavy tasks to the GPU without needing to understand GPU programming. Through demos, he showcases Quarkus applications utilizing the GPU for local inference and demonstrates various optimization strategies like kernel fusion and memory management. The session emphasizes how TornadoVM and GPU Llama 3 projects enable efficient AI model execution on Java frameworks and invites developers to engage with these tools.

Okay. So, we all like magic. To say something capable to do it for you. But for this magic to happen, we need to have a good hardware. And now all the magic that we have happens on GPUs. And why that's why Nvidia and Huang are so rich. Yes. And um this topic is very special for me because I also played a lot of JVMs on GPUs. Actually,

Java on GPUs, not JVMs. So, um what these guys do is really awesome. And what this team actually does with TornadoVM is really awesome. And uh of course, as a good Java conference, we need to have some really tough hardcore stuff. Not only just pure magic. Do me something beautiful and it will make you a nice Tik Tok movie. So, for this thing to happen, we really

need um to know what happens underneath. And uh with TornadoVM, we can access the best way the GPUs directly from Java. Which makes this tool really, really amazing. And those guys, I know them actually from that team for quite a lot of time. They actually were one one of few people who brought us Java to JVM. So, um it's going to be really hardcore with cool demos

with no clouds and no tokens used as far as I know. Yeah, so please won't welcome all way from Manchester, our Greek friend, Thanos. Please welcome. >> Thank you, buddy. Thank you very much for this amazing introduction. I'm so excited to be here. It is my first time in JPrime and my first time in Sofia as well. So, I'm really excited to talk today about two open

source projects that we have created at the University of Manchester and you may be wondering why someone from the University of Manchester come here to talk about the JVM project. Well, the reality is that two projects are thriving. They are open source. They have a community. And they have been also integrated with frameworks like Quarkus and Quarkus LangChain for J and LangChain LangChain for J. And my

goal today is to introduce the projects, introduce the problem at the beginning, and then jump into actual demos where we show Quarkus applications uh running a local inference accelerated on the GPU on this system, a MacBook M4 system. So, starting with an introduction of myself for those who don't know me, I'm Thanos. I'm a research fellow at the University of Manchester. I have been working in the

TornadoVM team for the past 8 years. I've done work at different layers within the TornadoVM uh software technology. So, I have worked on the compiler side where we support for different backends like OpenCL, Metal, CUDA PTX, and SPIR-V. I've done work in cloud deployments with GPU capabilities, especially AWS EC2 instances. And I have worked in many uh as a solutions architect in many applications and use cases.

So, enough about me. Here is uh the social media accounts where you can actually connect with me. So, let's start with the gap and what is currently missing from Java in order to compete with other programming languages that are thriving and dominating in the AI era. The reality is that the missing part is that these programming languages, especially C++, they have native support for GPU acceleration. And

this is something that we don't have in the Java ecosystem, in the JVM natively supported. And this is something that breaks in a sense this kind of promise that was initially inspired behind the JVM system. So, write once and run anywhere. Now, with these accelerators that are coming in our computers, we're not we're not natively to do that. We're not we're forced to use different ways which

we are not used to. We are used to having Java APIs and run code our applications and then launch them everywhere. So, in order to fill to fill this gap, which was our main objective of our team since the very beginning of the project, the goal was to seamlessly enable Java developers to access this hardware and offload and exploit this high performance without the need to know

how to program GPUs. And I will show you more in the next slides. I think it's better if we start with an overview of how currently Java developers can utilize LLMs in order to create AI-infused applications. They can use frameworks like Quarkus, LangChain for J, Quarkus, Spring AI, and the idea is that they're using LangChain for J for orchestration in order to invoke LLMs. And most of

the LLMs are hosted in the cloud. So, there are some REST API calls into SaaS. So, like OpenAI, Anthropic, Google Gemini. We have as well Mistral. And this is great. The response is coming back from the LLM. We can utilize it in in applications, in our AI services. But the reality is that we want also we are very interested in running inference in a local environment. And

the reasons is that we can have uh security constraints, privacy constraints. We may not want to send the prompts or maybe some sensitive data that we have in the cloud or in the providers of the LLMs. So then we we can run the models locally on our systems. And if we want to stay in the Java and without uh migrating to other programming languages, we have some

solutions already that they are running inference on CPUs. Like uh Llama 3 the Java, Jlama, Dema 4, and Qwen as well, Qwen implementation. And we have also the JVM wrappers. So we have the solution where we can actually use ONNX Runtime. We can uh use Ollama, Llama.cpp, vLLM. But then we are responsible of using this JVM wrappers and maintain them. And this can be a problem. Now

in this talk, I'm going to present you what we have built on top of uh TornadoVM. It is an LLM engine that can run inference and offload parts of the the computationally heavy part of the network on GPUs. So this is a GPU Llama 3. And uh I think it is a good time probably to start with the first demo. So we have prepared an agentic demo.

It uses the GPU Llama 3 with Quarkus LangChain 4J. And the idea is that we have uh three agents. They are chained. So it is a chain uh uh pattern where we actually send a write the Java class to print hello world. The first agent takes it. It generates the the code in Java. And it passes the response and the code that we have analyzed to the

code writer, which is a second agent. That what it does it is essentially to take the code and write it in a file, in a class file. And then we have the third agent that we are launching it in order to build and run this class. And the this demo is also in GitHub in our GitHub space in the link that I I have there. So, if

I go back, I want first to describe a little bit my setup. So, I am on a MacBook 4. I have two separate windows. As you see here, the top window where we will run the application and the bottom window where we have some real-time metrics of the system. So, we are interested in seeing the at the bottom window, the bottom right side, as you see here,

where we actually see the what and the usage on the GPU on this laptop. So, if we see here, I have Yeah. I hope it is visible. Is it? Can you see it? No? Okay, let me increase a little bit the the fonts. Now? Excellent. So, we are here. We are going to run this demo. We're using the Java command. We pass the arc file for TornadoVM.

It has all the flags that are required at runtime in order to run with TornadoVM and then compile the code of the network to run on the GPU. And then we have some modules as well where we define the device memory that we want to use on the GPU. We pass the JAR file. And we have in our Quarkus application, we have a main method that actually

accepts as arguments um the prompt. I can do that. It starts the Quarkus application. As we see here, it launches a Llama 3.21 billion model. And here are some parameters as well about the model. It runs on the GPU, as we see here. The first agent has already returned the code for a printing Hello World. It has passed to the second agent, which has actually written the

code into a And finally, the the build and run was successful, and we got the output, which is Hello World. And what we see here at the bottom right side, it is that we had the GPU working for the inference. And this is what I wanted to show you. going into the IntelliJ, I can describe more about the project and this particular demo. So, as we see

here, we have the pom file open. specified that we want to add the dependency for the GPU Llama 3 artifact in Quarkus LangChain for J. So, we are a model provider in a sense. We have integrated with Quarkus LangChain for J. And a similar dependency exists also for So, in this case, in this demo, we have used this dependency. And then what we did it is to

actually go in the application properties under resources to configure the model and model information. So, we say here that we want to enable GPU. >> [snorts] >> We specify the name of the model. And in fact, we say that we want to use as well floating point 16. So, we're using quantization as well. And then we have this Quarkus code. It is the Quarkus application in a

Where we have, if we go to the main method, we're invoi- we're invoking the run method. And then inside the run, we are calling the agents. So, we have three agents, as I described earlier. The first is the code generator agent. The second is the write agent the writer, and then we have the runner. So, we start at the beginning, we take the prompt, and we are

invoking the code generator. We If we go inside the code generator, we'll see the definition. So, here we are putting some uh user message in order to create the context for this agent, what we want this agent to do. And in this case, we say write a complete compilable Java code. The Java code should have should have with this kind of block, like in uh markdown. And

uh yeah, uh facilitate the generation. So, then if we go back, we are we're having the second agent. So, in this agent uh we are using the annotation register AI service to register it, but we are also specifying a tool, the Java coders tool. So, this class contains some methods that we have created, and they can be used as tools from the AI agents. So, in this

case, in the system methods, where we describe the context, we say you are a Java coder writer. You can use uh the write file tool, which is the method that we offer in the Java tools uh class, uh in order to store a Java program into the class And similarly, we can go to the third agent, the Java runner agent. And again, we are passing uh this

tools class, and uh we're saying that you are a Java builder builder porter. Uh you need to build and run the the code the code that you are getting as input, and uh you can use the build and run tool, which again, it is a method that we have defined in the Java coder tools class, and essentially, what it does it is to compile the class file

and run it. And return back the response. So, this is what I wanted to show you in a sense in It is a Quarkus application Quarkus LangChain for J application and it runs locally inference on the GPUs. So, I think with that we can continue. Yes. Go back to the slides. Perfect. So, now we'll move into the next part, which is how we transition from a CPU

to GPU inference in pure modern Java. So, if we have a very abstract view of what a transformer architecture looks like, in a sense it is a sequence of operations and these operations they are mapped into actual mathematic equations. We're not very much interested into the mathematics, but the idea is that you are having a user prompt it is the input. Then you are going through the

embeddings, you embed this kind of prompts, the tokens of the prompt into specific vectors. You are going through this kind of uh equations sequentially one after the other. You produce some data and then you have some iterative. So, this is an iterative process until you will reach the score that is good in order to have a reliable output. So, we're not interested into the mathematics as I

said, but we're interested we have tried to make it easy We have spotted that some of these equations they are using matrix vector multiplication. So, this can be a very compute heavy uh it um task and this can be what we can actually offload from these networks on the GPU. So, this started because we observed the Java implementation of Llama 3.java project. we said, "Okay, which parts

could we use and offload them on the GPU?" And this was what we actually found out. And if we code a matrix vector multiplication in Java, it would look like this. So, it may not be very complex in the eye, but in a sense, we have a lot of iterations, and we're doing a lot of processing. So, it can be computed computationally heavy. And as I said,

what we did, it was to take the llama 3.the Java implementation, profile it, and we realized that 81% of the time was spent actually in matmul operations, and particularly on matrix vector So, then the rational question was, "Okay, how can we make it faster? Is it possible to stay in Java, have a Java implementation, and make it faster?" And here is where Tornado VM stands. So, we

have this technology. We have created it at the University of Manchester. It is an open source project project. It is a production ready. So, we have had the first production deployment last year in the context of the European Space Agency, where they used Tornado VM to accelerate data analytics for data that are coming from the space. So, by using Tornado VM and GPU acceleration, they have managed

to accelerate the pipeline from 30 days to 5 days, which is significant improvement. So, here the idea is to show you a little bit the landscape of Tornado VM. it is at the center. At the left side, we have compatibility with different JDK distributions, like OpenJDK, SubstrateVM, uh Zulu, Temurin, Mandrel, etc. We are working with JDK 21, and we have support for for JDK 25 as well.

On the right side, you see the hardware backends that we support in TornadoVM. It is OpenCL, uh CUDA, PTX. We have uh SPIR-V support, which is a binary format. And recently, we added as well native support for Metal. That means that if you have a macOS, you can also run run with Metal. We have developed some in-house uh development tools, which we have used, especially the majority

of them they have been used for the GPU llama implementation and the debugging. So, we have TornadoVM Pulse, which is a profiler that gives some metrics on running on GPUs. We have Tornado Visualizer, that visualize some bytecodes and helps us to understand which data are sent, how many times, once, every execution, and this kind of behavior of a of an application. We have Tornado Insight, which is

a JetBrains IntelliJ plugin that helps uh beginner programmers to understand supported by TornadoVM and the TornadoVM API and what is not supported. So, that's the idea of explaining uh what is supported. And then we can use as well third-party tools like NVIDIA Insight Systems or Intel VTune in order to do in-depth profiling. We are doing uh frequent releases. So, with every release, we are pushing as well

in Docker Hub uh containers. We are pushing containers that they have um as target NVIDIA GPUs, so people can use the container if they don't want to actually install drivers or the CUDA toolkit and these kind of things. And we have as well a container for Intel graphics. For every distribution that we do, uh we are pushing uh in Maven Central the JAR files of TornadoVM of

the new release. And uh we are compatible with SDK man. So, this means that uh you are not required anymore to build TornadoVM on your systems. So, you can simply download the SDK and play with it. So, it will be very easy. I will show you in a bit. So, what is good as well, it is that um we have presented TornadoVM in several venues, and uh

the videos are are available on YouTube. So, we have different presentations about uh the API, the programming model, uh applications that we have built, like the Bllama 3. So, there is a lot of resources that someone can start with. But, as I said earlier, what we are interested it was matrix matrix vector So, this is a sequential implementation in Java with uh vanilla Java. As we see

here, we are using primitive uh array types, float weight for the weights, the Y. So, it is single threaded, it is simple, and uh produce correct result. It can be slow because the complexity is high, and there's no GPU use. So, what we did at the beginning it was to try to parallelize this code with the TornadoVM API. At first At first we used the loop parallel

API. In a sense, loop parallel API is a part where it exposes an annotation that developers can use in order to give a hint to the compiler that a for loop can run in parallel. in reality, if you see here, the only difference is that we have added this kind of at parallel annotation. If there is a dependency, a data uh between the iterations, then the compiler

will not be able to correctly produce parallel code, and we will fall back into the sequential code. But, if there there no data dependency, then it is a hint that the induction variable, in this case I, can correspond to the thread that we can run on the GPU. So, this way we can parallelize the work items on the GPU, the threads on the GPU, that can process

the same uh computation, but on different data. And this code can still be correct in Java if you are running sequentially in Java with annotation because it is skipped. So, a second uh iteration was if we can further optimize matrix vector Now, as Java developers, this can be a little bit intimidating, in fact, but the idea of this API is that someone who is familiar with GPU

programming models can actually code methods in Java. So, highly optimized kernels as In fact, I have used it in order to port existing hyper performance kernels and use it through TornadoVM in Java. And as we see here, the key concept is that this kernel API, what it does, it is to expose uh an object, as we see here, the first argument, what is called kernel context. And

this kernel context inside, it includes specific primitives that are available in CUDA or OpenCL. So, in a sense, as we see here, we can use the context to access a specific group ID or a local index of the work item. We can actually allocate uh local memory, in this case uh with float type. In a sense, local memory, it is like putting the data in the cache

of the GPU threads. So, we have faster accessing because we don't go to the DDR. So, these are kind of optimizations that are familiar in CUDA, we are not familiar in the Java world, but the interesting part is that we are not we don't need to be familiar with that. So, as long as we have a kernel that works, we can port it and we can run

it and get the performance. And that's the idea behind this API. Now, as I said, if you do want to do a quick installation, you can try to use SDKMAN in Linux and macOS, where we push every time the new releases. You can select a specific backend. So, in this case, the available versions for macOS, which is my system, it is to run with JDK 25 or

JDK 21 and select the Metal or the OpenCL An alternative way it is to go to the website of TornadoVM and go to the link {slash} downloads, where we have actually instructions how someone can download the SDK, export the variable TornadoVM home, and then start running the examples and many things. And the third way is if you are willing to contribute, you can fork or clone the

repository. You can go inside and run the script bin TornadoVM installer, that will install dependencies and will make you ready to start playing with the So, I think now I'm ready to jump into the next demo. The idea here it is that I want to show you how to install TornadoVM with SDKMAN, which versions I'm running. I will run matrix vector row example. So, let's jump back

here in the terminal. So, on the right on the bottom right part, again, you will see some spikes. That means we run on the GPU. So, if we go on the top terminal, I will try to increase a little bit the fonts. So, as we see here, we can and SDK list tornado VM. This will return back the available options that we have installed with SDK. So,

in this case we are we have selected uh 4.0.1 JDK 25 OpenCL. And if I do tornado version, we can validate that. So, we're using this version with OpenCL backend. We can do tornado that does does devices if I write it correctly. So, here we identify the the accelerators that have been identified in this uh laptop. So, it is the M4 GPU. It returns back as an

OpenCL device. And what we can run as well it is the what we see here we invoke the Java the tornado file that has the flags for tornado VM to enable uh the JIT compilation and the runtime. Then we pass the class path to the jar files that are shipped with the tornado VM And then we are launching this example the matrix vector row. And as we

see here it runs. We saw the spike on the bottom right side. And now without doing any specific warm-ups or optimizations, the first time we run we see that uh it is 16 16 times faster with the kernel API compared to Java sequential code. And with the app parallel annotation, it is uh eight times faster. Of course, this is not in isolation. It is now on the

laptop that I'm presenting. So, this is an idea how someone can install the code the tornado VM technology and they can start running some first examples with tornado VM. And I think it's a good idea to go into tornado in the uh GPU llama 3 So, I hope it is visible the code. Yeah, I I think. So, see here what it I am inside the DP Lama

3 inference project. So, it is in GitHub in the behind lab GitHub space. And what I'm interested here is to show Tornado inside. plugin that we have released. It is open source. So, if we go into plugins, we see that we have installed Tornado inside from the JetBrains marketplace. We are using the version 1.5.1. And what it does it is that we see here on the right

side this icon Tornado VM. it throws this panel where we actually see Tornado VM tasks. So, this means that it can detect from the class that I have open which are the tasks that are using the loop parallel API or the kernel API in if we go into the actual structure of the project, we are inside the source code. We are in the Tornado VM package. We

are in the kernels package. And here we have different classes. So, these classes are they contain the kernels that we have built for specific models like Granite, 53, when, etc. But we have as well one class that's called Transformer compute kernels layered. So, this is what is called from every And as we see here, we can a little bit have a look inside. So, we're using for

example for the flash attention we're using the kernel API and we are instantiating and creating some specific IDs that we are consuming later. We're allocating local memory to get more faster data accesses. what is interesting is that someone can use this uh uh IntelliJ plugin in order to do two things. The first one is to to use static analysis. So, if I go inside this kernel and

I do at some point, I don't know, maybe here, it doesn't matter. I do something like throw runtime exception. Then, this is identified as red. And if I go inside here, it will say, "Oh, TornadoVM, throw is not supported inside the kernel. GPUs cannot raise or handle exceptions." So, this gives an idea, a hint to programmers, to Java programmers, what is valid and what is not valid

inside the task. So, this is through the static analysis. And then, we can actually click the play button here in the TornadoVM plugin in order to run dynamic inspection. This what it will do is to take the task that we have used this syntactic sugar, the annotations, or the kernel API, and actually try to compile it to a lower code. So, if we try here the process

hits flat flash attention, it will throw the generated kernel on this console, and it will throw a message that the testing was completed. So, this gives a hint that we are ready to go to deployment, actually. We don't have any bug or we don't trigger something that is not supported. So, this is a good idea of how we can then go, build the the application, and try

Okay, let's continue with the next part. building GPULama 3 with TornadoVM. This is a high overview of the software stack. We remain in Java. So, we have uh, the Java the Java AI applications using uh, Quarkus, Spring AI, their microservices. We can use LangChain for J for orchestration, for LLMs. We are now targeting local inference, so we can use some other implementations like Llama 3.java or Jlama,

which they're doing CPU only inference and they're optimizing it. So, for example, I think Llama 3 is using as well Vector And or we can use GP Llama 3, where we can combine CPU inference with GPU inference through TornadoVM. And as we said, TornadoVM can support different formats. So, here are the backends as well on this list in order to target GPUs. So, we have many different

model families that we are supporting. In fact, uh, the GP Llama 3 project started about a year ago. So, within this year we have added more families and we are also adding more families. So, I think support for Jlama 4 is coming in the next weeks. Uh, the GP backends that someone can work reliably is OpenCL, PDX, CUDA, and Metal on macOS environment. And uh, as I

showed earlier, we have support for quantization types like FP16, Q8, Q4, and we have integrated with Quarkus, LangChain for J, and LangChain for J. So, we are growing as a community. We have welcomed a lot of external contributions, uh, like the addition of new model families like Deep Trust Raw, the ability to run on with a metal backend, uh, quantization support, some CI/CD integrations in order to

do a better testing for the Quarkus LangChain for J integrations. So, everything is running on our regressions tests on the servers in the University of Manchester. Now, moving forward, how we have managed to optimize our LLM engine? as we saw earlier, there is a lot of complexity inside the code, but what we do in essence is to use some concepts from the two worlds. In a sense,

we have the GPU world where Nvidia is doing a great job to optimize kernels. So, we are familiar and we're monitoring this kind of optimizations and we're trying to include them in our And the second world is the AI frameworks world where we have implementations like Ollama, vLLM, where people can optimize things. And we we are taking this kind of optimizations and we're trying to implement them

as well in our engine. I will show you some examples. So, how we have built them? We are taking the GPU optimization concepts. In we can say, I don't know. I mean, we One of them is the kernel fusion strategy. So, the idea is that within one kernel, you can invoke other kernels. And then by launching one kernel, the top level, you are gaining because you are

not anymore required to launch and dispatch five kernels. So, in essence, you're using one instead of five. And this is much faster in because you reduce the dispatch time. And the dispatch time is when you say, "Okay, I want to go into the GPU driver, compile and run this kernel." So, this is one. Using local memory is the other, as I said, in order to reduce the

accesses. And what we do is to take this concept from as I said, Nvidia libraries, and we actually adding support in Tornado VM. And then seamlessly, we are integrating it into the LLM world. have a step back. So, previously, I showed you a little bit the transformer architecture. So, here what we do it is in a sense the mapping between each operation in the uh and the

corresponding task that we have created in the GPU Lama 3 project. And as we see here, we can create a task graph, what we call in the TornadoVM uh terminology. And one task graph can have many tasks So, the tasks in a sense they are the GPU kernels that we are running on the Currently, with the GPU Lama 3 project, I think for Lama 3 1 billion

we are launching more than 9,000 kernels. So, more than 9,000 tasks are executed on And in this case, I think we have marked with a red box the actual kernel that does the fusion, so what I said earlier. So, if we go in this slide, we see that this is fused feed forward with Silu and Swilu activation uh method. And we are using uh float array types,

we are using as well half float array types, so we're using quantization. We are actually invoking one, two, and as we see here I think this is Yeah, with a blue it's the linear projection for a weight one, the linear projection with weight three. Then the Silu activation and Swilu activation, and we actually write the data at the end. So, we store the uh gives significant improvement

because uh one dispatch time as I said earlier. Another optimization that is coming from from TornadoVM side, it is the ability to handle memory, the GPU memory in an automatic way. So, from the Java world, we can define what it is a task graph, as you see here. It is a new object, we create a new task graph. We pass a name into the task And then

we define the data that we want to copy. And in this case, you can see that if we want to pass actually the 7 billion parameters for a model, we can do it once because this is going to be read-only data. So, we don't need to send the data every time we execute. Then we can actually say, "Okay, the input in this case for this task it

we want to be streamed." So, we want in every execution to actually stream new data that are inside the input. Then we actually define the task, which is taking a string name to identify a particular task. Then it is the method reference. And then we have we pass the actual arguments. And we can use more methods that are exposed from the TornadoVM API. For example, in this

case, we can use the persistent device, which is an optimization that we added at the API level. "Take this data and stay on the GPU. I'm going to use them again in the next round. So, no need to take them back and then take them back in." So, this gives some optimization and since it is it can be useful as well for energy efficiency because every time

we move data from the GPU memory back to the to the main memory, we pay some cost. And we can do more things. For example, we can define the output as we see here in the transfer to host method. But then we may want to actually take specific ranges within the output data. So, no need to actually copy all the data back, the output data. We can

selectively say from offset zero, take a bunch of that of elements that fits within the vocabulary size. We have done some performance analysis. I think this is on the previous round of our release. So, in this case we compare, as I said earlier, the Llama 3 implementation that uses a vector API and Panama API. And we're running three different models. Llama 8 billion, when three 8 billion,

and Mistral with 7 billion. And these are the bars, as you see there. And we have some different workloads examples. Uh the chat example, the streaming example, top memory, and agentic. So, in the system that we run, it was in isolation. We did warm up, and the CPU inference run on an Intel 9 CPU. While the GPU inference run on an an Nvidia card, the 5090. And

as we see here, we can achieve up to five times uh more throughput. Higher throughput. So, moving onwards, more optimizations, we have added support for prefill decode. This is an optimization that is coming from the AI frameworks. And particularly, we were inspired by vLLM. So, in this case, we noticed that we were paying a significant cost, which was at the beginning, the tokens were streamed one after

the other on the GPU memory. So, then we were sending low data, and we were paying the overhead of data movements. So, by applying this prefill phase that actually batches the tokens within specific sizes, for example, 32, we managed to receive better performance, a higher performance. So, as we see as you can see in the pull request 102 of our repository, we have managed to achieve 3.55

performance for long prompts. And for short prompts, we have achieved up to uh times faster. And currently, we have an optimized the decode phase, which is the next part that's going to take um place in the next release. Here are some features that we are taking from the GPU side. So, one of them is CUDA graphs. The idea is here that we can actually use this terminology

from the CUDA side and in fact do free replay of the kernels and we can unlock up to two times more tokens to process. in our goal in our road map to add support for MMA tensor core instructions, and this is something that we believe that it's going to yield more performance as well. So, this is part that will be integrated in the Tornado VM side and

then it will be used in the GPU Llama 3 project to run faster on the GPUs on the Nvidia GPUs. And in the middle side we see as well that we are interested in adding more support for SIMD intrinsics that are coming from the PTX world. So, Nvidia is optimizing this and we believe that in this particular case, I think this instruction gives the opportunity between GPU

threads to access data paying the cost. So, without this is the idea of actually optimizing the the data that are inside the local memory. So, in the cache memory in order to avoid the misses on the memory. And we move into the next part, which is the integrations. So, I have already said, I hope you are not tired. We have integrated with LangChain for J and Quarkus

LangChain So, in these links you can actually see information about how you can integrate the GPU Llama 3 implementation in your LangChain for J or Quarkus LangChain for J projects. And the idea is that, as I said, we want to enable GPU acceleration, but in a seamless way, to make it transparent. So, the users, they shouldn't know that what to run on the GPU. And we can

move into the next demo. So, here we have a Quarkus LangChain for J um application. It is from the vanilla it is the weather agent demo. And in this case, we have three agents. The first one accepts a prompt from uh the localhost 8080 regarding uh putting a specific uh city name and asking the weather. Then uh this actually um extracts the city from the side and

passes the city into the next agent, which is the geocoding service. And there, it actually retrieves the specific longitude and latitude for the city. And then actually invokes the final agent for the weather forecast service, which will return back the actual response that the weather in Sofia today is, etc. how here I want to show in this demo how a Quarkus LangChain for J application can change.

So, in this vanilla example, it is uh configured to run to use OpenAI. So, how we can change it to run the inference locally with GPU Llama 3 and accelerate on the GPU. So, first, we update the dependencies in the pom file. Then, we actually change the configurations of the application in order to specify the specific uh model that we want to deploy and some characteristics regarding

the the model. Uh that they are available in this link. And then we can build and run the And here in the bottom link I have actually what we what we will do. What we will show you now. So by moving back to the terminals we can actually go here. I will try to make it bigger. So yeah, as we see here we will start the Quarkus

application. But before going there, maybe it's a good idea to go to the IntelliJ. So let's go to the Quarkus LangChain for J application. So here we are in the Quarkus LangChain for J repository. As you see here, I don't have modified the application. What I have modified it is the pom file. I have added the dependency for using engine. And as well specifying the GPU Llama

3 for deployment. And then the next part that I changed it is in the application properties to actually put this kind of configuration. So I want to run in this case when three with 1.7 billion parameters and use quantization F16. And then I'm going back to the console. I start the Quarkus in the development mode and I pass as JVM arguments the tornado arc file and the

actual device memory size that I want to use. And this will start the service. The engine service. As we see here, Quarkus started. So now we are ready and what we are interested is to see on the bottom right side the GPU spikes. So if I go to the Google Chrome, I can actually try this. Oops. We're waiting to see the results. So, as we see here

in the Quarkus side, the model that has been deployed is Qwen 3 1.7 billions. It is running on the GPU. We see that the GPU has started offloading and accelerating parts from the network. And if we go back to the Google Chrome, we are going to receive the output from Of course, this will run inference locally on my laptop. So, I don't expect that this weather forecast

will be uh because today was was raining. But, as we see here, we got the response back that the weather today is partly cloud, which I'm not sure if it is correct. With maximum temperature of 22, minimum 14, etc. But, what is interesting is that we saw a Quarkus application and how we can modify it to run inference locally on Okay, moving moving forward. So, jumping into

the takeaways. So, what's coming next? We are continuing the work in optimizing the GPU Llama 3 and Tornado VM projects. What we are interested is to continue the batching side, as I said earlier, with the prefill decode optimization. So, the the goal is to continue and increase the batch sizes and the support. And the idea is to also monitor what's coming from OpenJDK projects like Project Leyden,

Valhalla. We have integrated with Project Panama, what is coming from Project Babylon. These are very interesting concepts that we want to be aligned. Um a second optimization, it is the ability to use what we are calling CUDA unified memory. The idea there is that we we want to eliminate the data copies between the actual uh JVM side memory and the GPUs. So, by using CUDA unified memory,

then we are able to access the same address space from the host on the GPU side. And this will give more performance. Then we are interested in uh adding support for uh multiple devices. So, imagine a system a server where you have five GPUs. We want to be able to scale up the LLM to accelerate it on both the resources that are available on the And then

we are we are adding more GPU optimization features um in the projects, as I said earlier. some key takeaways, we have seen today that um the gap between Java and running and accelerating AI inference on GPU is closed. So, with uh two open source projects like TornadoVM and GPU Lama 3, we have managed to enable it and we are optimizing it. This is an iteration So, GPU

optimization techniques are uh working good from the Java side. Um more information is that uh uh they are available on the TornadoVM website both for the TornadoVM project and the GPU Lama 3. they are production ready. They have integrated with frameworks that have been used by multiple people uh and the very large community. So, yeah, with this uh I invite you to actually try it. So, if

you are interested, if you are using Quarkus or LangChain for J, you can actually try the engine, try TornadoVM, and let us know about any feedback. So, here I have the resources about TornadoVM, GPU Lama 3, Docker images, the examples that I showed today. And you can actually follow us in the social media to learn more about what's coming next in the next releases. And finally, I

wanted to say that this work has been funded by several European research projects. So, we have managed mature the technology of TornadoVM through many years. So, yeah, this is just to acknowledge them their sponsorship. Thank you very much. And if you have any questions, please come forward to collect as well some stickers that we have and some goodies. And I'm happy to engage with you and discuss

more. >> [applause]