About this talk
This talk focuses on enhancing control over large language models (LLMs) through Java, including current and future capabilities. The speaker, a Java developer advocate at Oracle, discusses the challenges of achieving predictability and accuracy with LLMs, particularly in terms of how they follow commands, which is heavily influenced by their training data. By exploring the integration of ONNX models with Java applications using the Foreign Function and Memory API (FFM), the speaker demonstrates how Java developers can efficiently interact with these models. The session covers concepts essential for understanding LLMs, such as training, fine-tuning, inference, and how Java can facilitate smoother operations between AI models and databases. Additionally, the speaker introduces Project Babylon, aiming to allow Java code to be transformed into various programming models, promising a more seamless integration of AI technologies in Java development.
Full transcript
Um it's going to be a packed session with information part of it coming from the machine learning world as well from AI machine learning world as well, but I hope you're going to find it interesting from a Java developer's perspective. Um so I'm on I'm working as a Java developer advocate in the Java team at Oracle and today's talk is going to be about how we can
have a better control over LLMs but from Java using Java that we have right now but also Java that will become available in the future using some of the experiments that we're doing in project Babylon. That's the second part of the of the talk and the future talk. So um we are still under this hype I hope. Do you still like LLMs? Can please raise your hand
if you still like them? There's still some like in the in the audience. Well, we like LLMs but what we have been wanting for a long time or at least what I've been wanting for a long time is actually to have more control on how accurate the response is when I am working with an LLM. So this means that I want to have that predictability of what
I'm working on. as a developer I realize that well I have to understand how that is working in order to know how to control it better. And I assume that you're using a coding agent in your day-to-day life. Um yeah, so pretty much you know when accuracy and predictability of an LLM is important is like when uh you're noticing that where there are some commands that are
not exactly followed or not exactly understood and that is coming from how that LLM is performing the reasoning and is based on the training data and of course how that reasoning works. So the talk of today aims to explain a portion of how that would work from the Java's perspective and how we can control that as Java developers. like you, um I see I feel that the
environment of computing has been rapidly changing since 2022, and nowadays some form of a generative AI uh is the essential tool as development tool box. And we need to adapt technically and culturally. So, this is an exciting time, but it's also an anxious time. And it's very hard to filter out all the nonsense that is related to the this new trend because there's still much that we
don't know. Many of the models that are very performant are also closed source. But in the end, all the trends brought the same feelings to us as developers. What we have is actually adaptability and resilience to build a more hopeful time where we control it better. And well, Java has been great at adapting to changes for the past 30 years. And so, as nowadays, Java developers probably
you notice that Java connects models to many information sources such as relational or vector databases, enabling models to respond more accurately without constant retraining. And well, more generally Java orchestrates interaction among the models, the tools, you've seen it with LangChain for Java as probably, and information sources, all to achieve autonomously a goal. But the central software unit is usually the AI model, uh which we call generically
speaking the AI model, but the AI generative model can be machine learning model and so on. oftentimes from the perspective of our application, we see that we're consuming that model through frameworks and APIs. And this is great because we get fast adoption path, we get access to frontier models. It's less operational overhead for us. But to control how good inference works, uh we need an extra inference
control layer between that rest and native runtime. And it would be great if developers we could control that layer. Because that layer decides how requests are being batched, how runtime back end, by runtime back end I mean how the GPU or CPU is being used, how the session or how the engine is configured, and of course how tensors are placed in memory and other stuff. And to
assert better on how the layers work, um I thought to reflect on the journey of an AI machine learning model. So, there are a few stages that go in every model journey. It starts with training, which is the foundational learning phase, and it's very computationally intensive, that's why you hear that it needs a lot of CPU or they actually need GPUs these days. this is the phase
where the model analyzes a massive data set in order to learn patterns and relationships. And this first goal is actually to create a an accurate model. Now, the second one is the AI fine-tuning, which is a shortcut to training, because after you trained and that takes a while, you don't want to go to another phase just because you realize that you need to fine-tune to a specific
business that model. So, AI fine-tuning is a way to do that because you provide a smaller data um to the uh to do a little bit more training, and you save some resources and time to achieve your goal. Now, the last two are inference, which we're pretty much doing very often when we're using a coding agent. Um inference is the execution phase of our model. And you
probably heard it a lot of time. This one is using the pre-trained and fine-tuned model to make fast predictions on new data. Now, each individual prediction is far less computational demanding than the training. That's why we use this one. Uh but, delivering million times of predictions in real time requires highly optimized and scalable infrastructure. Fortunately, yeah, that's one. And of course, yeah, AI serving is the process
where we are deploying and managing that model for inference. So, it can be uh that we're getting it as a REST API or we're getting as a library or managing. And that's why we might also manage, depending on the serving, we manage the infrastructure to handle those requests. Taking a bit closer look with these concepts in mind of the life cycle, uh let's take a closer look
at the AI models. Now, uh conversational generative AI system like ChatGPT, for example, is powered by deep learning model language models, uh which are specifically the GPT family of models. Like other deep learning models, they are ultimately represented by uh learned numerical parameters, often called weights, which can be stored or exchanged in model file formats uh with different uh endings. And probably, if you went to Hugging
Face, depending on the runtime that you're using or you're downloading, you've seen that there are a few extensions there for the format like .pt, .pb, which pretty much come from the PyTorch world. Then, .onnx, specific to the ONNX runtime, GGML, which is the format of the GPTs, and so on. So, these models come in different formats and different runtimes tailored to interact with those. sometimes we get
them as files. But, more specifically, at a high level, the generative AI inference process is a pipeline that repeats over and over for each generated token. And we start with input tokenization, turning text into next token ID. Those IDs then are going through embedding and which is mapping each token into a dense vector space. Then the core work happens in transforming that you see there in a
different color. Because the transformer layers apply attention and feed forward blocks to produce next token logits. That's why they're coming back cuz next token logits can go again to transforming and then sampling transforming. Now the final step is decoding when the token IDs are being converted into text. So with the four focus on the transformer. A transformer is not just one a big operation. It's many repeated
layers. Each of one itself is being a composition of tensor operations. You can think of inference as executing a large computational graph. So you can imagine a graph repeated across many many layers, many tokens and so on. And this is where the performance and portability is becoming hard because this graph is large and is structured. The same patterns repeat but with different weights. Again weights, the constants
that we had it earlier. And well, efficiency depends heavily on how the graph is represented first and then executed. So both representation and execution matter for LLMs and also for machine learning models. And this leads us to ONNX, a way to represent that graph. And the way to represent that graph um is uh with sometimes with the dot ONNX um file format, but also it can be
uh like community has done before by doing uh several uh projects with LangChain for J. So probably many of you Have you used the LangChain for J embeddings so far? So you're not using Can you please raise your hand if you're using LangChain 4J? So, just a few hands. Okay. So, LangChain 4J is a very popular Java project for interacting with AI concepts, and it has the
concept of embeddings as well. So, ONNX is present there, but also there is another project that's been started by community early in January this year uh to produce inference API in Java over ONNX models. It's called Inference 4J. It's pretty young. So, given the adoption of ONNX by the community, I thought it would be great if we're talking today about how us as Java developers can represent
such can actually can work with such a model using the Java of now through the FFM API from Project Panama, and then we're going to talk about the Java of tomorrow where we can actually represent the LLM model, the graph, and then we're going to execute it. In both cases, we're going to execute, so we're going to perform inference, but we can look at how easy it
is to maintain that, how beautiful it is to maintain that as a Java developer. Um and hopefully inspire you to build something more. is a format well it was useful for representing a foreign um model for machine learning or for LLMs, but is also a runtime. However, foreign is the common denominator here from Java's perspective. So, Java knows nothing about ONNX. It doesn't come bundled in the
JDK. It's a separate project by my Microsoft. So, in this case, when there's a foreign runtime and a foreign programming model, we need something to connect to it, right? So, what do we put in between our Java code and that native library? Uh so, the answer to that was foreign function and memory API Java bindings, FFM bindings. More specifically, use this kind of API in terms of
interacting with native libraries because as some of us know, uh the Java objects are allocated on the heap. Heap is memory managed by the garbage collection, which is great for most of the object objects. However, the native memory is memory outside of the Java heap. And that is requiring the C API, of course. Uh and that one is much more difficult to be garbage collected. So, what
this API gives us is giving us a few handles, a few ways to work with native memory elegantly from Java by having bounce checked access to off-heap regions, and we can do that with memory segments. But also, we can free the entire allocation of these memory segments that are controlling the native memory uh in a deterministic way with objects called arenas. there can be an integration pattern
of for, you know, reaching out to ONNX and having my first LLM executed from Java. what I have done for the first experiment, and I'll going to demo soon, I used a tool called Jextract to generate these Java bindings. So, I didn't code necessarily all of these binding myself. I took the easy way out. It's not the most performing ways out in way out, but is a
starter point to get more acquainted with the FFM API. Um so, I used the Jextract um built to generate Java bindings from the ONNX runtime JNI C headers. Um because, well, I have a native library that exposes C library headers. And thanks to Jextract, I can generate Java code that would make my interaction with that library easier. We're just running a few We're just one command. Secondly,
because I still need the native library to load my LLM. I loaded the native library in my code. um, I resolve the C symbols with uh F and downcall handles. Now, what [snorts] each C API symbol is exposed as a Java method backed by a downcall handle in my Java code. And what it means is that Java allocates native compatible memory for C strings, for output parameters,
for opaque native handles, and more. But it can also do lifetime management uh wrapped as memory segments. Next, um, I passed the native memory via memory segments, as I said. Then, yeah, yeah, we read the return all these of uh parameters, but all in Java. And of course, I explicitly destroy the native objects by closing my arena, which is pretty much a Java object. So, this was
pretty much the integration pattern that was done for the demo. It looks simple. Um, it takes a little while to uh concoct it if you're not acquainted with FFM or Jextract. But if you're trying the demo, you're going to see all these resulted in in Great. So, with this in memory mind, I was like, "Okay, let's deploy an execute a Mononyx model." So, I chose uh Llama
model from Hugging Face, the one uh that's quantized, and it's Q4 F16 that ONNX just took it randomly from the llama 3.2 ONNX models on hugging face. And also because I wanted to utilize less memory on my computer, so that's why I went with that one. So I'm using that model to execute it, but what I'm doing is like because we're using an LLM, what I can
do with an LLM, the simplest thing is like to ask it questions and then respond. The LLM responds based on the questions that I asked. Now the fun part of this project was that I have a Java client and the Java client can communicate with the native library with what I had earlier in my integration pattern with the Panama bindings, which I created with Jextract. And then
I have the ONNX JNI runtime that is doing the actual work towards the execution provider, which can be CPU or GPU in my case. So ONNX runtime is just native code. Mean Java, I'm just managing that. Let's go to the uh demo and see how that would look like. Let me see. Yeah. Let me make this bigger. Do me. Or not too big. I have a little
script that uh you know, summarizes everything that I've done manually the first few times so that you would know um until getting to automatic, you run it manually a few times. So what I pretty much did, I got the sources of ONNX JNI. So I got one of release of the ONNX JNI. Now ONNX JNI native library also needs the ONNX runtime, the simple runtime library. So
it needs another native library. And I downloaded that one, too. So I put them on both of them side by side. It's a requirement of the native library for JNI to work with that. So got those two, moved them somewhere in my local machine. I think it's on uh Onyx Runtime JNI 0.11 OS 6 arm64. And then, in order to work with jextract, jextract needs at least
three things. It needs to know where is the native library or detect somehow where is the native library. It needs to know where are the native headers of the library because it's going to generate bindings based on those native headers. And it needs an output directory where it's going to put the generated Java in my case for jextract, I just said, I want to target the package
that's called like this. The output is my source main Java. It's a simple Maven uh project. And of course, run this on the main C header that's called org.openjdk.jextract.C.h, which is in the release of JNI uh that I downloaded from uh the internet. Pretty straightforward with this. Um let me see if I can just make this a little bit bigger. Java 25. Just making sure that I'm
running with the right Yes. Okay, so we're generated. You can see here there are some warnings. Well, there are some structures in C that um the generator skipped over, but doesn't affect the generated sources. Now, if I'm going to my package, I have my foreign package with all my details of the generated function. And next is like how I'm going to use all these bindings on my
on the side of Java, right? So, how I'm going to do it? Well, first of all, I in my main uh class, is that I need an arena because I need to manage this native memory. I need to allocate somehow the memory in that arena. Uh, by the way, C pointer and all these, these are um, constants that are defined in Java, but they're mapping their memory
segments around native memory, around pointers. Which is not very easy for us to work as directly in Java, but it's nicer when you have a wrapper auto-generated that safely does this for you. Then I said, "Okay, I need to work with the model." And that's called via the Java class OGA create model that was generated again by Jackstraw. Uh, the model needs a tokenizer to to process
the tokens, but also to give the generated uh, text. Um, a tokenizer stream, and of course, generator parameters. So, this is just my initialization part. When I am starting my program, what I'm saying, "Hey, get the native library from um, from here, from your from a path. Uh, then, you know, from the ONNX model that I got from Hugging Face, just read the model and pass it
to the native library via my Java code. And we're going to see, how do you do that? Well, first of all, I'm using this prompt method in the same class. And in the prompt matter, what I'm saying, I'm I'm saying like I'm calling the native uh, library with OGA create sequences for whatever is in my uh, in my um, in my model. Do a lot of uh,
tokenizer encode and append token sequences, and for each layer in the model, um, I am executing the OGA generator generate generate next token because we need to go through that sampling transform to sampling transform in the inference loop as well when we're going through the Java side. And in the end when everything is being processed, everything is being scripted to the output. Let's put this to a
test. So, let me make sure that I am Yes, it's enabled native access. The library path I'm using the library from my project there. And let's run it. I hope that I'm Let me see. Yes, it's started. All right. A poem. And that. We can ask it different things, but this is a very old model. I mean, it's a trained with old data, that's what I want
to say. So, it pretty much creates a very nice poem about Java. And every time that I'm asking it, it kind of creates a different poem. And depending on training data, the models become more creative with the poems. Uh so, it's was a fun experiment from uh from my point of view. So, with the Java of now, we can access the native library, which is pretty pretty
cool. I'm depending on that model representation that it's in a file that I need to parse all the time. And this can be a problem that's, you know, can be growing over time. I mean, it's still good from uh Java developer perspective. But what if I can do better and I can represent that model myself in Java? And this is about the Java of the future. So,
first of all, I looked inside what's inside an ONNX model. Now, this is a simplified version of an ONNX model. A machine learning model would look like let's see. An ONNX model that is for the LLM that I used looks like this. Because it's next next next token and so on. And this is just a very simplified Obviously, as a developer, I cannot go and make this
graph manually myself. It would be a little bit challenging, so to speak. But, let's see a bit what is inside it. First of all, each model graph has information about the major data. It's a computational graph, right? So, that.onnx model is a binary file that defines how the neural network architecture is going on. Each node in the graph is a tensor operation and is called matmul, add,
conv, and so on. Now, the data that is moving in between the graph is called tensors and you can represent as tensors. And this is can think about it as a multi-dimensional array. Weights are included as constant And they're the initializers of each node. And of course, for a neural network layer, weights shape the relationship between inputs and outputs. But you probably have heard that there are
also biases. Well, biases are shifting the activations so that the layer is not forced to go through zero. Now, serialization, because you need serialization in between those is provided for ONNX by something called protocol buffers. And of course, um ONNX needs an intermediate representation before reaching the hardware. And that intermediate representation is defined using proto files. You don't necessarily see them, but that's what pretty much the
models are using, those proto files. So, that was pretty much very simplified what it is in that.onnx file that I would get from huggingface. And the next challenge was like, okay, let's see if we can represent this as a graph somehow and have it in Java. So, we need a way to write the graph as a Java program and then have it transformed into that foreign that
the native library understands, so that I don't go through the onyx.json anymore. And that could be achieved with a feature that is coming in project Babylon, which is a separate um a separate project from OpenJDK. It's still under OpenJDK umbrella that aims to transform Java programming code into other programming models. In this case, the other programming model is onyx. So, this is just one example of what
it can be done with Babylon. Specifically, it uses something called code reflection and probably as Java developers have heard about reflection and you're like, "What's this code reflection different from the other reflection that we all know about?" Well, code reflection comes with a few uh different concepts. Code reflection is about treating Java code as a model we can inspect and transform. It pretty much lets us to
model Java code, not just to execute it. Uh then we can build and transform that modeled Java code. Then we can access those models of Java code uh inside methods and lambda expressions. And then lastly, we can interface with non-Java programming models, well, which is crucial when we are targeting other intermediate representation like onyx. um this is pretty much a way to encourage the library authors to
expose new programming models to Java developers. This is about pretty much our aim. how does code reflection look like? Is it easy to work with code reflection because it sounds complicated, right? Well, here's a concrete example. On the left, you can see that the Java source code that's annotated with @Reflect, right? And the point of that annotation is, instead of just running the code, we can capture
a representation of it. And that capture happens at compile time via javac. Then, um in that, we have a small uh fragment of code that receives the code model. And then, you have finally an example of how you can read that code model. So, pretty much, if you want to and see what that reflect does, uh you can execute the small code there, and you're going to
get the representation of compute that compiler sees of the Java code model. Uh now, what I want you to take away here is that in Java, we can reliably extract the structure of the computation, and we can get a model that is explicit about operations, about control flow, and data flow. So, this is how code reflection is getting more powerful. Now, another portion is transformation, because it's
not just reflection on the code. We also need to transform the model. you pretty much see that both transformation using that core of func that evaluates each function. Um it's pretty much Java Java, right? It's no no different. Well, the left resembles the natural Java code that you write, and the right resembles what actually the simplified partially evaluated form that will map cleanly into the map construction.
So, this helps us to like um make this process feel like magic. So, when you're using @Reflect, you might feel magical. You start with normal Java, then you capture the structure, and then you go to to um the transformations. So, yeah. Using code reflection, what I ended up, you know, having, based on what I had earlier with FFM, because I already had the execution via is that
since I have direct access to then maybe I can represent the ONNX model with Java and then reflect and have an intermediate layer that is does does the processing of transformation. And then I'm just sending the model to the native library without loading any of that big ONNX.json file sorry that dot ONNX file that I got from huggingface and just work with this in solving the problem
for me to like load a file parse all the file and then send it to the native. I'm just representing this without any any need to go further on the file and it also gives me a way to manipulate form of the model in the future. So what happened is that what we did in the Java ONNX script that's how we call this library we're lowering the
Java code via the code reflection model. Then the partial evaluation comes with the evaluation of constants and of course enrolling of the constant loops. Then the transformation to ONNX code reflection model then the transformation to ONNX protobuf model because we have to serialize that and then execution using the ONNX API. So how did we did that? Well from ONNX specification and resources like we had before I
had that generation of you know ONNX API but we also needed to generate three main building blocks the operators the Java API that can code basically against ONNX operators. So we had the ONNX operators that were in the graph explicitly done in Java. Now this design was very simple each operator becomes a static method on the ONNX operator class that is being uh the demo. And you
can find almost I think yeah, all the ONNX operators there um annotated with that reflect because we need that reflect on those as well. And uh to speed up things, whenever a new ONNX release is happening, we have a class that's called OpGen. We run on the new release of operators and we automatically generate them in the files. Next. Uh we need a way to give those
proto buffs and to share how serialization Uh so since we need uh those proto buffs, we also need a builder for that. And the ONNX builder is providing the classes and subclasses uh to represent the Java proto buff building blocks. So we don't need to load those from files. It pretty much is uh having representation for graphs, for nodes, attributes, tensors, um etc. once we have the
ONNX model binder, we still need to execute it efficiently and safely from Java. this is the big puzzle. The big puzzle that end up being created is that the ONNX in this case feels like a first-class target from Java, but it has strong typing and clean developer experience regardless of the evolution of the library and of the model's evolution. So it's easier to migrate or to evolve
such a library with this approach. um you probably see some more information here. Well, uh tensor in the case of our library is an ergonomic type that we pass, uh so you're not dealing with raw buffers and shapes everywhere. And of course, the transformer is the one who's transforming the Java code thanks to code reflection and maps it into the operators. An ONNX runtime is pretty much
an evolution of what I generated before. A nicer um nicer entry point so that whenever the foreign API changes and my Java bindings change, I always have like a nice uh wrapper that provides a steady evolution for my client code. And [snorts] it is wrapping all functionality that is foreign for us. We created all this as a proof of concept for code reflection concept in And it's
available also under the code reflection examples in Babylon. So, let's see how this goes into action. So, my second repo is about Babylon. Uh sorry. Is the are the examples from Babylon. So, what you see here, this ONNX file, is under the Babylon uh code repo, Project Babylon code repo, and under OpenJDK open source, then so on. Like I had before in my um example, I still
need the ONNX runtime, a genai dylib, and the ONNX runtime dylib, which is pretty much the um native uh library to execute. But, I don't need the uh dot ONNX file anymore. Um I can uh thanks to um code reflection, and then something called ONNX. And so, I can build the uh that I can build from code reflection And how does my model look like? Well, is
this Llama model. Ignore the red parts. Uh IntelliJ sometimes reports it whenever I'm building again Babylon. It doesn't load it 100%. So, what I do in my Llama model, in my constructor, I'm just loading the token weights uh from the model Q4 ONNX data. So, I just need the weights. And the weights file for even for an LLM is way smaller than the LLM itself, which can
be gigabytes. So, I just need the constants So, when I'm in initializing the model for uh that I'm going to use code reflection on, I'm pretty much uh going to need only the weights for this. I'm initializing And the next portion is the forward function, which is actually going to do uh in help me with tokenizing my text and have the all the communication between sampling and
transformers. So, pretty much forward is my transformer, but represented in Java. and I'm annotating with that reflect. So, pretty much at reflect is helping me to forward all my Java input and my uh actually my text input to my from that my Java client gets to the LLM itself. And all these are operators, again annotated with that reflect. So, everything created and done with Java. Now, the
Llama demo is pretty pretty straightforward, and you can see it's pretty similar to the one before exact except for the fact that now the model is built from memory. Uh sorry, Um I'm creating a session. And pretty much I am going to, you know, read my input and then pass it over. Uh let's the Llama demo as well. Stop. Oh, dependency analysis. This is so annoying. Yes,
I said Java 8. Blah blah. Run. Why are you Just give me a sec. Why are you taking Java 8? Sorry. You're This is the I have another solution. Always rely on command line. Okay, Babylon. Just making sure. The thing is that Babylon recently got, you know, things for 28 March in it. So, I don't want You know, I'm going to depend on probably internet and some
other stuff. Uh but um I really want to work with Babylon JDK, which is requires building on its own. So, it's not the JDK that's available right now. It needs to be built like any other JDK. It's pretty easy with C make. And I will connect to also any coding assistant. They will know how to build the Java the JDK. So, yes. Yes. Yes. Run. Maybe I
should have said skip tests. Uh okay. Oh, it was working. Starting. When you're under the pressure of the Oh. What? Clear. Starting again. I'm going to persist into this until cuz it worked. Come on, yes. You're ready. Okay. All right. Write me a poem about Right. Yeah. Write poem with Java. Java is pretty fast for a Java created LLM model. I wrote it on a local and
yeah. Um so this this was a a pretty fun experiment to work with. And the main advantage that I find as a Java developer when I have my Java the model done in Java is the fact that I can efficientize a bit how I'm working with the methods that are for representing the operations for my graph. So sometimes for machine learning I was able to create another
graph for my for my model. Okay. So that was the demo for this one. But my dream is a as a developer and as Java developer is actually to achieve with that something more than this. It's actually to achieve the composition of inference like having my own agent running with my Java code. And then well, we can go to more orchestration through Agentic and all Now what
I want you to take away from this presentation is that we're having a few perspectives about AI in the Java platform. And what we aim with all the changes that are in Java but especially the ones that I demoed today is that actually to give you the building blocks so that you can experiment yourself with a native runtimes like ONNX or PyTorch if you want to go
there and we can discuss about that in the ask me anything. Um and with these foundational blocks um, you can evolve more. And if you're wondering like, "Hey, what are the foundational blocks?" Besides what I showed today with foreign function and memory API, which is the finalized API since JDK 22, we also have the vector API for implementing model run times on the CPU, for implement then
we have working on Babylon, another small library that's called Heterogeneous Accelerator Toolkit, and HAT for short. And that one is meant to implement model run times on the GPU, so that we don't need any more the library that is modeling that is helping us to um, you know, go through the execution provider, but to be able to do the run times ourselves, which will be even cooler.
And then Valhalla, I pretty pretty much a lot of you have heard about Valhalla is to express numeric types uh, which are flat in memory and that's going to help us a lot on the training of the model side. And of course, for people that are still asking like, "Hey, well, we're working with Python, we want to work still with PyTorch." Then there's the project Detroit for
uh, deploying and executing PyTorch models. That's the end goal of Detroit. So, it seems like there's a tower of Babel that we're building in the Java platform, but this tower of Babel has an end is actually a stable tower of software written in Java, uh, that is accessible to use to write in Java, which would be great and amazing. So, this is the portion of what we're
aiming for the for the future with what has been built so far in the Java platform. Uh, but since we're almost at the end of the session, I want you to focus on two things that I demoed here. First of all, uh, that code reflection is a new powerful API that is for transforming into various programming models. And it starts with a simple annotation called @reflect to
represent that foreign programming model. it allows you uh evolve your Java code into something that you can analyze, rewrite, and so Uh secondly, code reflection transformations. They're like partial evaluation and simplification. Uh those are the building blocks that bridge the gaps between computation models, which are very important for um the LLM side and for the AI side of it. Thirdly, there's a strong synergy between FFM and
Babylon. Because FFM API from Panama enables direct operation of external technologies from Java now. So, you can do this now. You can operate these external technologies from Java. and fourth, well, uh project Babylon and um and Panama together, well, they can unlock new domains for us as Java developers. And I think that it can bring us a beneficial uh a lot of benefits to as we are
Java authors in general of Java code. Um and we can support a wider set of programming models and runtimes. Let me insist a little bit on this one. Uh and just to give you a heads-up, the Duke with a hat is Project Panama. The ancient text is Project Babylon. So, these are the Dukes for each of those. Uh didn't give a explanation for the Dukes. Uh so,
whenever you see those, that's what they are meant for. Now, we can discuss more into the ask me anything section. I There's much more that I couldn't comprise in this session. I mean, it's pretty dense to explain both APIs, but I'm looking forward to discussing more details about how you see these and how we can evolve this in the future. Do not be shy to connect with
me. Do not be shy to connect on the mailing list that are for these associated projects and their evolutions because a lot of people are sharing their what they have built with them. There's a lot of great libraries that the community is building and open sourcing. And of course, you can give feedback on how easy it is to interact with Project Babylon once you tried it. And
yes, how how beautiful you believe it is or how difficult you believe it is. It's still an experiment. We're still building a lot of documentation and examples around it to make it more easy to understand, but definitely it comes with a new um let's say with a somewhat new programming model compared to what we're used to, but um as developers, we still value elegance of the code
that we write, all of us I suppose, and because elegance of code means that it's easy for us to maintain something in the future. Uh so we're still accepting feedback on that. Feedback on me and my presentation can be given there. the slides that are used in this presentation with all the links to the are available there. so you can access that, and if you cannot access
them, please uh just reach out on social media and I'm just going to give them link to you. I'm not hiding anything and all the examples are available. Uh so we have 4 minutes left. If are there any questions? If there are not any questions, I can show another demo that I forgot about it. Questions from the audience? Okay, no question. Everything was clear. Amazing. Which I
I would like to believe that it was clear, but uh >> [laughter] >> Okay, one last trick. No, it is not that big. Oh, there's a I was going to show you one last trick. >> Thanks. Could we use to train another kind of models like to image generator or something like that? >> So, the question was can we use this to train models? The Java ONNX
Script library at the moment is doing only inference. We are looking into training. Um so, we are also looking into doing training and show an example of training as well. But yes, code reflection can be used also to develop the training side uh of these runtimes, for example. So, it for sure I can say that it works without having a point to show it. But because the
training side of that is available also through ONNX, but also I No, I cannot speak for PyTorch because there's a different kind of work there. the training side at least in ONNX, it's very similar to the how the inference is in the library. So, for ONNX, yes, I think that's possible with Babylon to be done. Just with FFM, no. training is about also representation of that graph.
So, yeah. You need to wait to represent the graph because and just with FFM, you're just executing. Oh, but Babylon could help indeed with the training. That's an excellent question. Thank you. Okay, there's no other question. I can run the demo. Hopefully, friends, the Okay, so in the Babylon example repo, we have also class that's called ONNX Lift. It shows this pretty bad. Um but this ONNX
Lift class that I can recommend for you to try um is a way that you're if you're downloading a dot ONNX file from somewhere and you want to get the I don't know the weights and you want to get the reflected model. Uh you can use this ONNX lift class to get the reflected model, meaning the Java code with that reflect annotation on it. and the way
to do that is by In my case Can I make this bigger? Uh so in my case what I have to what I had to give is like where the ONNX model is, just a file with the ONNX model, where I want um class to be given to be created, the one with that reflect. And then a name for that class, in my case is Instant Lama
model. Now given that I was unlucky earlier with my run, I wouldn't want to try my luck with this one, maybe. Maybe this one works. No, it doesn't work. So, yeah. Now I I changed my Maven some Maven configuration very very briefly before the presentation, sorry about that. And yes, um we're always discussing a a bit about how uh the every time that two new Java versions,
the IDEs get confused. And whenever we're using 28, I mean, it's pretty much like there's 26 that was just released and you're giving me 28. But I can tell you that the ONNX uh lift is working and it helps a lot in like just, you know, taking a model from an ONNX file and have generated Java with that reflect annotation. And then what's left for you is
just to do the other instantiation part and have fun with it. So, I would say to give it a try to this uh to this project that we're we're building. Um and of course, if you, you know, encounter any any issues or if you encounter any obstacles, give it a shout out. I am eager to explain to you how it works. Thank you very much. >> [applause]
