Jakarta EE 11 meets AI: Building intelligent microservices with virtual threads and Jakarta Data

This talk explores the integration of AI with Jakarta EE, emphasizing the use of existing building blocks within the Enterprise Java platform. The speaker explains how to leverage Jakarta Data repositories to avoid the complexities often associated with AI development, such as the need for Python or separate vector databases. The presentation covers the implementation of cosine similarity search using byte arrays in JPA entities and highlights the use of virtual threads for improved concurrency without the hassle of thread pool tuning. The speaker illustrates a coding demonstration to showcase the practical application of these concepts using a minimal number of dependencies, ultimately arguing that 'boring tech' can effectively facilitate AI workloads.

[music] >> Good afternoon and welcome to a very boring talk on Jakarta EE and AI because um Enterprise Java is very boring. If uh how many of you know Adam Bien? Everybody must know Adam Bien. So Adam Bien has been preaching about boring platform for more than a decade and that is what we are here to talk about. Boring is good because boring means less problems. So

let's get started. Um who am I? My name is Lukman. I'm a technical writer at um Azul and pretty much part of the Azul family now from Payara. So anybody who knows Payara knows that we've been part of the Java EE and now Jakarta EE ecosystem for a very long time now. So we are used to boring tech and I'm going to talk about very boring tech

and why you should be using uh boring tech. So what's our agenda for today? I'm going to speed up because I know everybody's tired and this probably one of the last talks. So what's our agenda? We're going to look at the claim uh boring stack smart AI composable using the existing building blocks that you have as basic as uh it can be. The stack is going to

be Jakarta data, virtual threads, and then uh Java cosine similarity search. Live build. We're going to run through a a quick look at the demo and then virtual threads, why they matter, the payoff declarative versus programmatic uh runtime model swap, and then some takeaways we can explore after. So what's the claim? The standard narrative about AI is that you need Python to do AI, a vector database,

a separate service, thread pools tuned to your CPU cores, a lot of intimidating stuff. So most people just hear about AI, especially decision makers, and then it's very scary. It's like there's a whole lot of things that need to come in. but I'm here to tell you that it's very boring to start exploring AI on the Enterprise Java platform, Jakarta EE. Now, every one of those assumptions

will be challenged here working with boring tech. So, what's the problem? Now, three problems that you will keep hearing is that, first of all, thread pools, yeah, you need to tune the thread pool, blah, blah, blah, blocking HTTP calls to the LLM, each call ties up the There's a whole lot of things that you hear about that. Now, that is true in some context, but not necessarily

when it comes to adopting AI. Then, also, there is embeddings. You need a vector database to store the data. You need Python everywhere. These are all true to some extent, but not necessarily things that should inhibit you in exploring AI on the Jakarta EE platform using the existing building blocks that are already available on the platform. So, Boring Stack is the smartest choice you can make. Three

things you won't need. First, you don't need a separate vector database, as we'll be seeing very soon. The embeddings live right inside your JPA entities. So, we're going to be using Jakarta data repositories to load the chunks and then do cosine similarity search in Java. A few lines of code and then we should be done with that. Second, again, no Python, no LangChain, just native pure Java,

exactly what you have existing at the moment. You don't need to bring in anything. And then, no thread pool tuning. Virtual threads on Java 21 gives you millions of virtual threads you can just use and discard. Most of the time, you should be good. Now, the core claim is that your existing Jakarta EE knowledge composes naturally with AI workloads. You won't need to learn a new stack,

especially your teams will need to learn a new stack, a new firmware firmware, a new set of APIs to be able to start testing incorporating AI into your existing applications. So, with that established, let's look at what the stack actually means in practice. So, Jakarta data, how many people have used or heard of Jakarta data? Perfect. So, Jakarta data is pretty much if you use Spring Data

JPA, it's the equivalent on the Jakarta EE platform. The good thing is that it's now standardized, so it's part of the Java EE or Jakarta EE standards, and so we know it's going to be there for the foreseeable future. So, Jakarta data is the equivalent of Spring Data JPA for Java EE applications. Now, the only external dependency here is Ollama. So, we'll need to run the model

somewhere, so we'll use Ollama to run existing smaller models on this machine. Here. So, this is an architecture of the the the stack that we've just talked about. So, two layers. The top row is the request flow. The browser sends REST request to the back end, which then routes it to Ollama. The bottom row is the data layer, PostgreSQL with Jakarta data repos for persistence, vector search

for similarity, and virtual threads for parallelism so that you can scale to accommodate your users, especially if you wake up one day your application is on the front page of Reddit or Hacker News or some other internet front page, you would have to immediately find yourself scaling to meet unprovoked and unexpected demand. So, it's very important your application can scale to meet those. Now, the key choice

is the data layer. There is no PG vector extension, no separate vector database. Embeddings are stored as byte arrays in a standard JPA entity right next to your existing data. The vector search class in this application loads all chunks from the JPA entities via the Jakarta data repo, then it computes a cosine similarity against query embeddings in Java. Now, this is very important because most of the

time when you are starting to adopt a new technology, it is good to start with the basic building blocks you already have without any external dependency. If you start with a lot of complexities at a go, you would have to maintain those complexities. And every single third-party library you draw into an application is a possible attack vector that you need to keep in mind. And so, if

you wake up one day and a specific library that you you depend on has been compromised in one way or the other, you find yourself uh wanting because if there is no fix, then it means you are likely exposed to uh exploration of that vulnerability. So, starting to adopt any technology is good to start with basic building blocks that you have so that you can then layer

complexities on it in a much more structured way. So, vector search does the search from the byte array in the database entity fields using cosine similarity. And then it returns the top matches. Now, for a few hundred chunks, this is almost instantaneous or instantaneous depending on the machine you are testing or Now, for a production scale with millions of vectors, you would want a vector database. But

for starting, again, if you are starting to adopt AI into an existing enterprise application, you would want to test with these existing blocks to see where it fits into your application. One of the problems we have seen is that it's AI, so you wake up one day and then your application has to use AI. It doesn't matter. There's no proper business analysis. First of all, what does

AI bring to those applications? What are the complexities we are likely to face? The security risk, the business risk. There's no assessment. You just wake up and everybody and everything and their mother and everybody's AI now. But most of the time what we are doing is that we are adding complexities for its own sake. We are drawing in third-party libraries all over the place, so a single

deployment artifact is 200 MB overnight when it could be just 30 or 40 MB. These are all things that generally shouldn't be in in in an application that you want to have long lived to serve your users. So, those boring stuff here is very important as a building block for you to start testing AI in your application in a much more isolated, much more scoped way, so

you can know whether it works for you or not. Now, that is the theory before we work through the code. A quick sanity check, of course, to see that it actually works. As usual, it works on my machine and then it may not work on production machines. Always happens. Everybody has had it before. So, let me bring this guy here and then let's hope it works here.

So, let's come here. Who is Let me just ask this model. Who is Octavio? How many people know Octavio? Perfect. Octavio is the is the spec lead for Jakarta data. He did an amazing job getting it as part of Jakarta EE 11, so I felt it only honorable that we see if the model actually remembers him based on all that we built. So, let's see what uh

brings to us. Now, what is happening here is that I have the embeddings stored in the database as part of the And this search here is happening in the back end. We are just using Java cosine similarity search to match what is returned from the embeddings and see the top results. So, let's see what the model uh tells us. Now, for those curious, the model I'm using

here is Google's Gemma 4 with 2 billion parameters. That's what I'm using. It's their latest uh Gemma small scale. Uh okay, good. So, we have something. It says "Octavio is a speaker of the session." I just made this up. "Jakarta Data 1.0, a new era of data access in uh data access at the conference." So, pretty much this is one of the sample conference data I did

that we loaded from the database and then made the search on those. Okay, so let's continue. Where's our presentation? So, this is all based on boring tech. Very important. This is all based on very boring tech and we see that it works. So, it works. Now, what's next? Here. This is the full stack Jakarta Data 1.0 Concurrency 3.1. Now, Jakarta Concurrency, which uh 3.1, we shipped as

part of Jakarta EE 11 is a specification that supports virtual threads. So, we need to pass the virtual thread the virtual equals true uh flag and then behind the scenes, instead of using platform threads, it's going to use virtual threads. And we all know virtual threads are, at least for most use cases, the best when it comes to needing uh concurrency. So, we just pass this flag

with a few uh gymnastics at the back end and then we have Jakarta Concurrency defaulting to the virtual threads instead of using platform threads. Platform threads again can be very expensive. And then we have CDI 4.1 JAX-RS or Jakarta RS with HTMX in the front end and then of course the obligatory microprofile config. How many people know microprofile? Perfect, good. So, microprofile is the de facto for

some of these things in application development on the Jakarta EE platform. So, what we don't use is PG vector, there's no database, LangChain for J, there's no third-party library. The only dependency this application has is the Ollama which of course we need to run the model in. So, instead we are using byte arrays in JPA entities as binary large objects and this gives us programmatic control so

we can decide a lot of things. Of course, again it's at the the other side of some cost because then now you have full control and as they say with more power comes more responsibility. So, you can easily shoot yourself in the foot if most of the time you don't know what you're doing. So, there's always a trade-off. I have to admit that there's always a trade-off

but having more control when you are starting to adopt a technology is much better than pulling in a lot of black boxes and magic that you don't fully know how they work. So, again it's a matter of today or tomorrow, whichever you want to do the trade-offs and the controls that you need. So, these are the things that we don't use and this is what you use

more code but zero magic. You know everything and how it works. And so, if something doesn't work, you know exactly where to zoom in on. But if you pull some big library in and something doesn't work, okay, yeah, good luck. And if Claude doesn't know what to do then or you run out of Claude tokens, your your window is closed for the week, you are finished. So,

yeah, you know how it goes. So, live build from empty war to deployment. The main event, seven steps each adding one capability to it. Now, these are the JPA entities. We have a document chunk and we have a conference talk. These two are stored in the database. I can bring my IDE here. And then we can see the pre-computed pre-stored data as I did them. So, this

is the close this guy. Close this guy. Yeah. So, this is the database. These are the document chunks and you can see we have the embeddings computed and stored here as byte arrays in the database. So, at application startup, the application compute the embeddings up front and then stores them along with the entities in Now, this gives us the flexibility of faster performance as against slower startups,

but pretty much gives us everything we need at run time. So, at run time, we don't need to compute anything. Everything is computed up front. And again, the embedding field is a byte array stored as Postgres byte array type because I'm using Postgres. Anybody who's not using Postgres and is using especially Microsoft MS SQL, please let's have a few words after. So, yes, this is a Postgres

database that is using byte array to store the embeddings. The conversion to and from a float array happens through the embedding converter. We're looking at that. It's in the code. A utility with two static methods backed by byte buffer. The JPA entity doesn't know about vectors. The JPA entity simply sees a binary large object and stores it accordingly. It doesn't care whatever we are using it for.

The conversion is a concern of the vector class, not the persistence layer. So, we have this separation where the data layer is concerned with the literal data to be stored in the database and then the middle layer does whatever it wants with those talks. Now, the conference talk is a standard JPA entity along with the document chunks. These are standard JPA entities, and that is why Jakarta

data is able to manage them. Back to our talk here. So, next slide here, Jakarta data repos is pretty simple. I'm sure everybody has seen this. Even if you haven't seen Jakarta data before, you've probably seen from Spring Data JPA, and it's pretty much the same construct here. It's a lot of declarations of your various methods, and then the implementation does the actual implementation for you. These

are all done at runtime. So, we declare the various search endpoints not endpoint, the various search methods, and then the Jakarta data implementation does all this for us. And all this was led by Octavio Santana. So, props to him wherever he is. Same thing, raw JPA. So, raw JPA, what we used to write is you need to write so entity manager.createNamedQuery or createQuery select e blah blah

blah blah Now, Jakarta data just comes in, and we need to just say find by this or look by this. And again, this were all originally inspired by Apache Delta Spike. How many people know Delta Spike? Apache? Perfect. So, these were all inspired by Apache Delta Spike. They used to have I think it's a dead project now, but yes, these were all originally inspired by that as

well. So, now it's a lot more declarative when it comes to searching. Of course, there's always room you can create your own named queries and everything, but Jakarta data is there to ease you from writing for most of the simple simple queries that you would do manually. Now, this is where some kind of magic happens. So, this is the cosine similarity. The vector search class is the

replacement for PG vector, Pinecone, uh Weaviate, or any other vector database. It embeds the query using the Ollama embedding model, loads all chunks from the chunk repo, computes cosine similarity between the query vector and each chunk's stored vector, sorts by score, and returns the top result. So, the question we ask the model about Octavio, this is where some kind of magic happen to find the best result

to return to us. And the result I would say was pretty accurate. So, the cosine similarity method is pure math. There's no third party, there's nothing here. This is pure math. This is pure debuggable code that you can step in in your IDE and see exactly what is happening and do all the swearing that you can do like, "What the" Yeah, we all do all of that.

So, you can step in and do all the swearing and everything. This is pure Java code, no magic, no black box for that. 15 lines returns a score between zero and one, where one means identical. The similarity results wrapped called purse, each chunk with its score for scoring. Now, one trade-off worth noting, this loads all chunks into memory. Like I said, there is always a trade-off. When

you're starting to adopt a technology, especially for existing applications, you want to start with a very small subset feature in your application. And for those cases, this is pretty much okay. But, it's a trade-off worth noting that for this example, we load everything into memory for 50 million and you would want a vector database. So, beyond a certain point, then you need to start exploring the existing

options. But, for starters, generally, you can start with the existing component. Again, boring tech is good. So, for vast majority of internal AI tools, conference assistant, documentation, Q&A systems, especially documentations, this approach is sufficient and avoids an entire infrastructure dependency. Again, anything you pull into your application is one extra door that you would need to guard against. And in today's day and age, exploring vulnerabilities is just

a matter of prompting. People who don't even know how to code can create vulnerability exploration. So, always keep that in mind. Assume the whole world is out to get you as long as you put something out there. And it is much easier and faster these days with the aid of LLMs. So, always keep Now, programmatic retrieval augmented generation, this is step four. The AI service is the

entire rack pipeline in one method. First, vector search finds the 10 most similar chunks, then we format them as context with source labels, and then we build a user message that combines the context and the question. Finally, we call the chat model through chat model factory and return the response text. Now, we embed the query, find similar chunks, build a prompt called LLM, 25 of code. No

magic. Adam Bean is going to be very happy when he sees this because his entire life this is what he's been preaching. So, the declarative alternative is an interface with assistant message at user while back AI service. If you're using LLM LangChain for J, these are some of the components you'd be using. And I spoke about it on Tuesday in my first talk. So, first talk was

looking at that side. This time around we're looking at the boring side. So, logging which chunks were retrieved is a line of just a simple log and all of that. So, changing the prompt format at run time is a string edit which annotation cannot give you. Now, the critical piece that makes this worth 20 extra lines is the chat model factory [snorts] which enables run time switching,

and that's is step five. So, model switching you can also switch models at run time. Config values are injected via config property rather than, of course, imperatively loading it programmatically. Now, the CDI approach is the idiomatic Jakarta EE pattern, type safe, testable, and the container handles conversion and gives you a sensible default that you declare if nothing is found. Now, the chat model factory holds the current

chat model in a volatile field. On startup, in the initialization, we set a default model from the injected configs. The switch model method creates a new Ollama chat model with the requested name. So, dynamically, we can change the model that we are executing. And this sample has two models. So, up here, you can see we have the first one, and then we have Mistral, as well. So,

we can also change dynamically models at runtime. Any questions so far? There'll be pop quiz, so pay attention. I know you've all just had lunch, but there'll be pop quiz, so pay attention. Okay, so virtual thread injection. Now, step six, the data loader generates embeddings at startup. At startup, application is configured to start up and then load the embeddings and do all the gymnastics that we talked

about. So, each chance makes an HTTP call to Ollama's embedding endpoint. Instead of raw JDK executors, we use Jakarta Concurrency C 3.1 3.1, as we said, with uh virtual threads. So, this gives us the chance to be able to scale as and when the need arises, especially if we are dealing with significantly large data. Now, the configuration, as we saw, is just one class. The application creates

a managed executor service backed by virtual threads, registers it in the uh JNDI namespace, and then pretty much makes it available for us to use in the code. And then, of course, we need somehow to expose everything in the back end to the front end. And we are using Jakarta REST in here to expose the data to the front end. And pretty much that is all there

is. This is the full complete pipeline, 18 files, all standard Jakarta EE, all very boring, all designed to make Adam Bien very happy because we are using the platform. And this is pretty much all there is, and you have a rag system that has no third-party dependency. And so this means that you can test retrieval augmented generation, you can bring in AI into your application without drawing

in third-party libraries, and you have full control over things that happen that you don't expect, especially if you wake up tomorrow, for instance, and there is uh CVE declared for LangChain or some other party dependency that your application depends If there is no fix immediately, or you have no support for it, then it's going to be a bloody day for you. But if you have all this,

you are using the platform, you are not exposed to any third-party, and you should be good. the demo, you've already seen that. Let me walk back to this guy again. Maybe we can try a few other things. See, I have 10 minutes here. So, we've tried Octavio. Maybe let me ask a few five questions here. We have built-in questions. This may take some time because this laptop

is pretty old. But generally, on a much faster laptop, you should be able to have this machine, not just laptop. Of course, desktop machines, you should be able to have this to work instantaneously. So, while this work, we can just go back to the code. No, the presentation here. So, platform, why why virtual threads? Of course, the as we said, virtual threads are generally something you should

default to instead of using platform threads because platform threads come with non-trivial costs for non-trivial applications that you should keep in mind. So, if you have virtual threads, why not use virtual threads instead? Of course, nothing is free in this life. So, virtual threads do have some caveats. Synchronized blocks pin the virtual thread to its carrier, defeating the purpose. So, instead of use reentrant lock instead. Of

course, uh concurrency is never straightforward. There's always test and then find out what works for you, but there are some caveats. These are some of them. Thread local is per virtual thread. If you create a million virtual threads with thread local state, you get a million copies. So, treat that carefully. Our code does not use uh thread local, either. Now, virtual threads are suitable for input-output-bound work.

CPU-bound work, like matrix multiplication, should stay on platform threads. So, these are all things that you should know about virtual threads. The fact that they are cheap doesn't mean they are eternally cheap. There's always some kind of cost. So, you want to keep those in mind when adopting it, especially for non-trivial workloads. And you should also know where they work best. They are not suitable for everything,

but they are a good start when it comes to uh concurrency in your enterprise Java So, this is the request path same virtual thread executor you saw earlier, now in the request path rather than the startup. And these are all things that we can explore in the code. I'm looking at my time here. The payoff, reflecting on a programmatic versus declarative. On Tuesday, I did the declarative

uh talk, and then this time around we are looking at the programmatic. Now, there's always, again, a tradeoff. You would need to do an analysis, carry out an analysis of your own situation, and see which one works best for you, side by side. Now, on the left, the declarative approach, an interface with at register AI service, powered by LangChain CDI portable extension, about five lines, very clean.

But the AI service is immutable after construction, so you cannot change the model without rebuilding. Now, on the right, the programmatic approach we walked through earlier, the AI service asks the chat model factory for the current model on every request. Now, the factory holds the model in a volatile field. When switch switch model is called, the factory creates a new Ollama chat model with the requested name

and atomically replaces the reference. So, it it depends how much control, again like we said, you want. There's always a trade-off as with programming. And I was once told by platform architect that the best answer for every programming question is it depends. So, it depends. There is always a trade-off and there is no hard and fast There's no religious you know, attachment that it has to be

No, it's You just have to know your situation, the application you're working on, your objectives, and then you can decide which to pick. Again, both have their own ups and downs. You will just need to decide which to pick. There's always, of course, we can give you recommend general broad recommendations, but in the end, it depends. Now, where this approach breaks is Yeah, of course, millions of

vectors when you have, again, non-trivial data, you have to look for a bigger hammer to squash the problem at hand. CPU only inference. Now, this is a simple laptop GPU or anything, so this works fine. But when you need that power and you need to now decide between CPU and GPU and all the use out there, then you know which way to look. Of course, horizontal volatile

field is per instance, push model choice to microservice config plus config refresh. So, these are some recommendations when you need to meet some of the edge cases in this. But again, it depends on the situation. This is just pointers that there are basic building blocks existing that you have that you can start with when you are trying to adopt AI in your application. Now, knowing the edges

is the difference between a pattern and a fad. And this is pretty much all software in every platform architect that you ever know in your life who is experienced knows where there is fire and where there is water. So, knowing the edges is the difference between a pattern and a fad. And that is pretty much all experience is. The more you've been bitten, the more experience you

know, the more you know which corners not to pass. So, this is the pattern. Now, here are the edges that we've explored. Now, what are the takeaways here? Who benefits? For your team, no Python in no context switching, no dependency, no external dependency. Jakarta Data gives you fewer lines than raw JPA. Raw JPA is very verbose and almost physically painful to use. So, Jakarta Data is in

there to make life easy for you. Now, virtual threads, of course, you don't need to worry about pool sizing for most use cases. Every file is standard Jakarta EE and Java, so it's debuggable. You can stop work in, do all the necessary swearing to, you know, let all the frustrations out and all of that. Pure Java, pure debugging. Now, for operations team, runtime model switching, standard war

deployment, everything is dockerized, same monitoring as everything else, same stack. So, it's pretty much pick, drop in, and then you are good. The only external dependency, again, is Olama. Now, your ops team won't adapt to new patterns, your developers won't switch to Python. It's pretty much the same thing that you already know. Three numbers to remember: zero external databases, one HTTP call, have all the models, and

then an embedding is about 50 lines or 50 chunks by virtual threads and that is for this sample data. Of course, the objective of this is to show you that boring is good. Start with boring tech and then when there is a need to bring in much more firepower, then you can explore third-party libraries. Any question? thank you very much and I will be at the Pyra

booth there or Azure booth actually, sorry. I'll be at the Azure I'll be at the Azure booth should anybody want to follow up on this talk. Thank you very much. >> [music]