About this talk
In this talk, Lukman from Azul discusses the importance of integrating local AI into Jakarta EE applications to avoid dependency on external API providers. He emphasizes the risks involved with cloud-based AI, including vendor lock-in and potential account suspensions. The speaker explains a tech stack that includes CDI, LangChain, Ollama, and PG Vector to run AI services locally. He describes four progressive modes for incorporating AI, which range from initial retrieval augmented generation to full agentic workflows. By leveraging Docker for deployment and existing Java technologies, developers can create robust AI applications that maintain control and reduce costs associated with external AI services.
Full transcript
[music] >> Okay. Hi, everybody. Welcome. Good afternoon. My name is Lukman, and I work with Azul as a technical writer. So, this afternoon we're going to talk about building intelligent applications, or building intelligent monolith applications, supercharging your enterprise Jakarta EE applications with local AI. Now, to set the stage, I'm going to ask two questions. First of all, how many people use cloud here? How many people use
cloud? Okay, great. Amazing. Gemini? How many people use Gemini? Great. How many people use ChatGPT here? Amazing. Great. How many people use Jakarta EE, or have heard of Jakarta EE, or know Jakarta EE? Great. Amazing. Amazing. This is going to be awesome. Amazing. So, let's let's set uh the stage here, and then take a look at those headlines here. Cloud takes down fintech startup as Anthropic suspends
over 60 accounts. The chief technology officer warns, "Never put your eggs in one basket." So, this guy woke up one day, and then 60 of their accounts were banned. Anthropic just banned it. It's automatic. They uh whatever it is found them for whatever reason, and banned them. And so, they woke up one day, and they had lost access to their AI. And that is the stage. That's
what we're seeking to explore in this talk, running your own inference locally, so that when you wake up on a Monday morning, you don't find yourself back to the stone age, or stone or back to the AI stone age. So, let's get started. Now, back to our presentation uh here. presentation local host colon 8080. Good. You've seen me. My name is Lukman. So, I will skip this.
What's our agenda? Why local AI inside your Jakarta EE applications? I think that should be pretty clear and with that we could even close the talk. But why local AI? So that you don't wake up back to the AI stone age because Anthropic or Open AI or Google especially decides that yesterday you logged in from Pakistan and so they are banning your account. They are suspecting something.
You don't want that. The stack. What's going to be our stack? It's going to be CDI. How many people have heard of LangChain? Of course, everybody knows LangChain. So, LangChain for J LangGraph, Kook from JetBrains, J Lama, Ollama, and PG Vector. This is going to be our stack. We're going to look at four progressive modes of building different composable AI patterns in your Jakarta EE applications. I'll
briefly touch on CPU, GPU, same or different hardware. Deployment The deployment is dockerized. Everything is in Docker. How many people use Docker? Nearly everybody. Yes, if you're not using Docker yet, please see me after the talk. We need to have a few words. So, Docker and all of that. So, let's go on. Let's start. local AI? Okay, so it's not sinking. That's fine. So, why local AI?
Why why AI on the Jakarta EE platform? It's quite simple. Like we stated, if you are using an API and it's not just AI API, any API that you have no end-to-end control over, then you are at the mercy of the API provider. So, you could wake up one day and then your access has been terminated. Or you could wake up one day, there was another discussion
on LinkedIn where this developer complains that his key was compromised and Google doesn't have anything to stop a compromised key. So, if your key gets compromised, you make a mistake and somebody gets access to your key and start hitting requests with it, Google doesn't care. Google will just keep as long as the requests are coming in. So, you don't want that. You want an end-to-end control of
your infrastructure so that you can have that better control over your local AI usage. So, now, how do we get started here? No mouse. No space for mouse. Never mind. So, the problem with AI integration, typical AI integration, you need external API calls. Data leaves your network and we all know what that means. API calls, they scale with your usage. So, if you have a really non-trivial
application, of course, the AI will need to understand that and everything is a token and every token is money. Now, the latest currency in town amongst us developers is tokens. So, very soon it's not going to be how many lines of code, it's how many tokens have you used. That's what we're going to be assessed on. I think you will agree with me on Vendor lock-in. Now,
if you are using Anthropic, you are locked into Anthropic's ecosystem. If you are using OpenAI, you are locked into their ecosystem. I think you are using Jakarta EE because you don't want vendor lock-in. So, your AI should also not lock you in to a specific vendor. Network latency on every request, of course. It has to go out somewhere. And then, of course, there's also Python. A few
years ago, when AI started, Java didn't have much AI. Everything had to be in Python, and so that is still a problem in some guys' eyes. Now, what if AI could be local? You have zero data leaves your JVM. Everything is running as a process in your Java virtual machine. No API cost. No vendor dependence is because you are independent. You are running everything on your own.
No round trip with the network. Everything is just there. It works. Pure Java. It runs by your existing component. So, your CDI component, your EJB component, your messaging component, your whatever component, your REST resources, your AI application, your AI usage runs next to this component in the same deployed artifact. I think that's quite compelling. So, you are independent. And feel free to ask me a question at
any point in time. There'll be pop quiz. So, please pay attention. We all just had lunch. So, there'll be pop quiz. Uh the winner will get something from me special. Okay. So, let's go on. So, I think this is very very uh compelling to have your AI application running locally where you are not dependent on all these factors, or you are not susceptible to these factors that
I just mentioned here. Uh bring this guy here. You are not susceptible to them. Now, no Python, no API key, no microservice, no any new paradigm. It's just you using your existing knowledge, everything you know, all the things your developers already know to incorporate AI into your applications. Now, for this demo, you don't need Python. There's nothing against I've nothing against Python, but if your team writes
Java, your AI infrastructure should be in Java. No API key. There's no external service to talk to, authenticate against, or any such thing. Everything is within your internal silo. No separate microservice, the AI runs alongside your wall. Now, what do we run? We run Ollama. How many people have heard of Ollama? How many people use No Ollama? Perfect. Amazing. You're all Ollama handles model inference locally, you
don't need to go out. Now, it's pretty much a simple server to serve, for those who don't know, it's to serve AI models locally. So, you just pull in your Ollama image or container, wire it, and then pull whatever image you want. For this application, we are using Google's Gemma for 2 billion parameter model. That's what we're going to use for this application. Now, you have CDI
that you can How many people know CDI? Jakarta context and dependency injection. So, you have CDI to wire the various components of your application. So, pretty much like I said, you are using your existing component, you are using your existing infrastructure, and CDI is just there to wire the various parts of the AI component into a single cohesive whole in your application. And then we use PG
Vector, a Postgres extension, which is pretty much for vector embeddings in the same database. I'm sure everybody here runs Postgres. If anybody here is running Microsoft MS SQL, please let's have a few words after the talk. But, I'm sure everybody is using a Postgres, so you just add the PG Vector extension, and then you have an embedding database, and pretty much that is all the the the
stack is. Nothing more, nothing less. It's everything you already know composed together into a cohesive whole. So, how is it? Let's take a look at it. There are four moving parts, all connected. The browser talks to the back end through legendary JavaScript. I'm not using much JavaScript in the application, is just small HTMX snippets here and there. Sends requests to the back end. The back end is
running on Pyra Micro Jakarta EE 11 on Java 25 in the Docker container. And that's pretty much all the front end is. Calls the back end. And inside the application CDI manages all the AI beans. CDI is managing their lifetime, their their dependencies, all of that scene that you need to all the plumbing. CDI does most of that for all of that for us. LangChain provides the
RAG pipeline, the retrieval augmented generation. LangChain has amazing declarative RAG support. So, we just slap a few annotations here and there, extend a few interfaces here, and then we have a a RAG pipeline. So, LangChain does that for us. It talks to Llama over HTTP for inference and PG vector in the database for database storage. Now, there's no separate database because you're already using Postgres. So, you
add the extension and then you are good. And this is pretty much the the the stack that we have. So, there's declarative, there's agentic, there's composition, and then there's in-process. And all these are using things, APIs, different projects that you can explore in your application. Now, LangChain plus Ollama. So, LangChain is pretty much the glue for putting the AI into Java properly. It's the Java equivalent of
the LangChain in Python. So, same patterns, document splitting, embedding, retrieval, response generation, and all of that. And then Ollama makes it easy for us to run the models locally. Of course, you need to run a model that your machine can support. If you pull in a 600 billion parameter model, you'll burn your CPU. So, you need to support, and that is what we're doing. That's why we're
using Gemma 2 billion parameter. We use two models, Gemma for chat, and then Nomik Embed Text for embeddings in the application. And all of these just run on the your your existing machine. It runs on this laptop, which is a very simple, very under-whelming laptop. So, it should run on your server just fine. And this is pretty much the infrastructure that application is built with. Now, the
live the fun part that we are all waiting for. Let's walk through exactly how this is built. There are incremental five incremental steps. Each step adds one capability on top of the previous one. So, we build on top of each other uh running uh the application. Now, before then, let's probably take a look at it here because I have it compiled uh pre running pre deployed the
local host uh colon 8080. So, I have it pre deployed here. So, this is it. I'll come back to this in a sec. Back here. First of all, we define the data source because whatever it is, we need to talk to the database, and this is the data source. How many people know this method of defining data source as against the persistence.xml? So, there's a This is
a simple way of defining a data source. Instead of using the persistence.xml, you can define it declaratively as part of your code. Of course, there are pros and cons to this kind of construct. So, you can define it this way, especially if you have one database. You can define it this way. So, we have a data source that will be accessible through JNDI via the name that
we defined there. So, first of all, we define the data source, and this data source is what we're going to use to connect to the Postgres database. Now, again, what I want us to notice here is that we are using everything that you already know. There's no new thing here. These are things data sources are things you've been doing since the beginning of time. And so, it's
the same Jakarta EE data source definition that we are using. And this data source is going to connect to the Postgres database with the PG extension to access our embeddings. Then we have producers, CDI Amazing. CDI has this powerful feature where you can literally produce your own component beans, whatever your application needs. So, in this example, for instance, what we are doing here is annotating a set
of methods with the CDI at producers and then giving them scopes. So, we are producing an embedding model and we are producing the embedding store, the type of embedding store that we are going to use in the application. It's a text embedding store. So, we are producing this component. When do you need to use producers? You use producers, CDI producers, when the bean in question is not
part of your application, but you want CDI to manage it. So, if you look at this code, for instance, the embedding model is not part of our application. It's part of the LangChain library. And we want CDI to control its life cycle for us. One of the ways of doing that, achieving that, is through the producer mechanism. So, in this way, when we do at inject embedding
model, the CDI runtime is going to call this method and then it's going to take whatever is returned here and inject it into whatever field we have declared a dependency on. So, wherever there is an at inject embedding model, CDI is going to call this method, get the contextual instance, and inject it into that field. And so, we are pretty much converting LangChain component into into a
Jakarta EE CDI bean. And so, CDI now is going to manage the full life cycle of these dependencies for us. We don't have to do anything. And as we are saying they are application scoped, these beans are going to exist for the life cycle of the deployed application. It's a single thing. Think of it as a single thing. So, we are going to have a single instance
of all these. And whenever there is a request, it doesn't matter how many times, it doesn't matter where these are being used, CDI is going to make sure there is always a single instance that is used to satisfy request for those. Again, we are introducing AI by using the ex- existing mechanisms and components and constructs existing in our Jakarta EE infrastructure. Now, we have created our beans.
The next thing is JPA. Everybody uses JPA, right? Everybody is using JPA in one way or the other. So, we are using JPA to store both JPA entities and then some part of the injection. It's pretty simple. Again, your existing infrastructure, everything you have. We have the injection flow on the right. You are already probably using something similar to this. When the application starts, CDI would load
some files. What kind of files is CDI going to load? These are JSON files that exist we embed these data, the rag data, depending on how you're doing it, you can embed them up front, convert them to JSON, and then read them. This is one example. So, CDI at startup, we are reading this and then converting it into something and then storing it into the database. And
pretty much, again, the same existing infrastructure using JPA. Remember, the first step was the data source creation. So, we created a data source and this tier is going to use that created data source to collect to the database. Again, there's nothing new here. These are all things you already know about. Then, the AI itself, the AI component itself. Here, we have a conference chat assistant that we
have. Again, you can see we have annotated with with some stuff from LangChain, which is the assistant message with UI Jakarta EE conference, blah blah blah. You already know all of those things. And then, how do we wire all these together? An interface with annotation becomes an AI service. LangChain will do that for us. And then, we wire these together. On the is the wiring. In a
post-construct method, we use AI services {dot} builder to create the implementation. We pass the chat model. Now, if you go into this class, you will see that we are injecting a chat model. We are injecting a content retriever. These are all beings that we are producing in our application, and CDI is going to manage it for us. So, we are taking existing component from the various AI
libraries, converting them into Jakarta EE component for CDI to manage, and we are pretty much creating a loosely coupled application by declaring declaratively declaring dependencies on this component. So, in the @PostConstruct uh life cycle, this method annotated post-construct is pretty much a method that CDI will automatically call when this bean is fully instantiated along with all of its dependencies. When everything is ready, CDI will call this
method, and then this is where we are doing our instantiation. So, this is where we are setting up the conference assistant here in the post-construct method. And again, it's CDI doing all of these things for us. The next step, then we expose a chat endpoint, which is a resource endpoint at Jakarta or Jakarta REST or JAX-RS. Everybody knows this as JAX-RS, but it's now officially Jakarta REST.
We have a JAX-RS resource chat resource that we expose and then the front end calls this endpoint with a question. So, it says, "Call this endpoint ask a question." And then you just ask the question with a mode and everything. And this is pretty much all that we are doing in the application. And this is the full uh pipeline for the first part. Remember, we said there
are four parts of this, the four modes that we are looking at. This is the first part, the data source, database connection, CDI beans. All of these parts are part of the first part, which is the retrieval augmented generation part of the application. Any questions so far? No. No question. But I'll be There'll be a pop quiz, so pay attention, please. Then, the second one is the
agentic We've looked at declarative. The second one is the agentic uh flow. Now, RAG is good for teaching the AI model about your existing whatever documentation, whatever, whatever it is. But then the the new norm is agent. Everybody is using agent here. Who's not using agent here? Please use agent. Whether you have used for it or not, just use agent, please. Just so that you can also
go on LinkedIn and say you are using agent. But yes, the next step is agentic workflow LangGraph uh state. We are using LangChain graph for J state graph. The LLM decides to use RAG or answer directly. Pretty much, we give it autonomy and say, "Based on this question, decide whether you want to use RAG or you want to answer directly in a chat. In a sort of
chatty way to the user. Then the third mode is agent composition. So, we have Cook. Cook is from JetBrains. It's also an agentic framework for the Java platform. Specialist agent as tools orchestrated by planner. Then the fourth one, we have Jlama. How many people have heard of Jlama? Okay, so amazing. Jlama is Java inference without any external dependency. We'll look at that. So, the fourth model is
Jlama. Let's continue. The agentic workflow is pretty much simple. So, start. Then the agent will classify the query whether it should use retrieval augmented generation, which we saw in the earlier part, or directly answer. Then it will generate a response. And then pretty much send it back to the back end. So, the first five steps were straight line. Define the data source, create producers, load and embed
the specification or data that you want to use as part of retrieval augmented generation. We define the AI source, expose the end point. That's fine. But in step two, this is the agentic workflow now. Beyond the rack, now what we are doing is we are introducing an agent that can think. And when I say think, not like human thinking, but agent thinking. So, the agent will decide
what to do. So, the first node is classify as the LLM to categorize the query. And then the second one would be when you categorize the query, make a decision, and then answer based on the decision that you made, whether you generation, or you should just chat with the user based on what you decide. We set it up and then forget it. Now, this is a fundamental
shift from the declarative model that we've seen so far. In mode one, the pipeline always runs the same way. In this mode, the LLM decides the execution path. It's a state machine where the transitions are determined by its classification, and we are using LangGraph to make this easy. You define nodes as functions, edges as routine functions, and the framework handles the execution. And this is pretty much
agentic uh workflow using the same component, the same construct that we've seen so far. Now, how does it work? It's pretty simple. We have a conference agent component. Again, you can see this is a CDI being annotated with @ApplicationScoped. There's always a singleton of these beings. Most of the time, default to singletons unless you have a specific need for any other context, default to singletons. And you
can see we are injecting the various components again in here, chat model, content retriever, and then we have compiled graph here, which is the workflow uh field. Here, what we are doing in the post construct is setting up the schema, and then we are giving it classification. We are saying, "Look at the query, and then based on your pre- known classification, decide what to do." The schema
defines the state that flows through the graph. Four channels, the original query, the classification result, the retrieved context, and the final response. Now, each node reads and writes to this shared state. Now, all of these, I want you to notice, are CDI components, application scope, post construct, compiles, chat model. These are all things that we have injected. And this pretty much creates an agent that we use
in the chat resource to answer a question, depending on which routine we take in the chat resource. Of course, the code is available on GitHub, so you pull it after the conference and at your leisure time. Then we have agent composition. Agent composition compared to mode two had one agent with branching logic, which is look at the query, what do you use? Chat or you're going to
use rack. Now, this mode here composes multiple specialist agent into one orchestrator. Now, Koog is from JetBrains. How many people have heard of Koog? k o o g from JetBrains. It's one of the new frameworks they've introduced about 2 months ago or a month and a half ago for agentic development on the Jakarta on the Java platform, actually. Now, the planner receives the user's question, analyzes it,
and decides which specialist tools to invoke. It can call them in sequence or in parallel. Then it synthesizes the results into a coherent answer. Of course, that's what uh LLMs are good at. There are three specialist agents. We have speech spec search tool, which wraps our existing content to use the existing retriever augmented generation that we saw earlier. It reuses the PG vector rack pipeline from step
one. Then we have a version check tool that has static data about a number of Jakarta EE specifications and their version history and all of that. Now, code example tool delegates to the LLM with a generation system prompt. So, this particular one is a little bit complicated because instead of one agent making a decision, a two-way decision, this one actually analyzes and then spawns a number of
sub agents to do the work. The planner decomposes the question, picks the right specialist, and then synthesizes the result. All of in your existing infrastructure using your existing components, the existing knowledge of your people, and And how does the code look like? Here is our tool definition. On the left, a tool definition. A class that implements tool set from the Co library with methods annotated at tool
and at LLM description. the method body is in, of course, plain old Java, plain old boring Java. This one wraps our existing content retriever. Then on the right, we have the orchestrator wiring, three tools registered in a tool registry. The registry discovers the at tool via reflection, then the AI agent builder receives the registry, a system prompt, which is and then an Ollama executor. The planner agent
is a CDI being initialized once. Again, it's a singleton. Now, these are This is from the Co library from JetBrains. And again, we are reusing existing Jakarta EE construct to put all of these together. So, you have at tool set plus at tool, you have a declarative definition. Then we have the tool registry builder that discovers this pure Java. Again, Co is It has two parts. You
have You can use it in Kotlin. We all know JetBrains is very huge in on on Kotlin. Or you can use it in Java. Here, I'm using it the pure Java part of that framework, and it works just fine. So, this is the code for that. Then the last one, the last step, this is the end game. Modes 1 through 3 all depend on external Ollama process.
The first three that we've seen, the retrieval augmented generation, then we have the single agent, then we have the orchestrator, all depend on the Ollama container running and returning results. Now, this one here is different. Mode 4 removes Ollama from the equation, and then you have Jlama running LLM inference inside a JVM using There's no Ollama, there's there's nothing. It just runs pure LLM inference in Java.
So, on the left side, we have the architecture for modes 1 to 3. The request comes from the client to the Para instance running the front end, then Ollama, the front end, of course, the JAX-RS endpoint passes on to the various back end for the queries. All of those queries hit the Ollama instance in the container. Now, there's a network hop, even if it's localhost, it also
means Ollama must be running. If Ollama crashes, modes 1 to 3 stop working because they rely on Ollama. Ollama is our cloud in the container. Now, on the right, mode 4, the inference engine lives inside the running JVM process. There's no external call to any container outside. No external no network hop. The Panama the Ollama sorry, the Jlama uses the Panama vector API internally to do its
own magic. So, instead of running Ollama and then making a call to it, this time around, we are running locally in the JVM as a Now, how does that look like? It's we have an application scoped Jlama chat model producer, which is a CDI bean, and then we are injecting some configurations, externalizing some configurations that we are injecting. What we are injecting here is the model and
then the location of the various location in the container to keep the models that are downloaded because Jlama go out there, download the model, and then store it somewhere in the container. So, we have this externalized through MicroProfile configuration, and a few instantiations in the Atmos construct method. So, we are using the Jlama chat model builder, and then we are passing the model name that we want
you to go out there and download. Then we are passing a few uh stuff like configurations like the temperature and quantized model at runtime, blah blah blah. And then we are creating a Jlama chat model. So, this particular code here is going to create a model for us that is not running in Ollama, but it's running as a JVM process. And I think this is something that
everybody should explore, especially for smaller models, situations where you can make do with smaller models. The rest of the application works on chain graceful degradation. If the model isn't available, in this case Ollama container is not available or something, Jlama uses the vector API internally for matrix simulation. It's using Java 25, so the code in the uh repo runs Java 25 in the container. So, you don't
need to have Java 25 installed on your machine. This are four models or four modes that you can explore in your application. So, retrieval augmented generation, agentic, single agent, then you have orchestration, then you have pure Ollama running in your application. Now, if I bring up the code here, expand this. Yes. Now, if you go to chat resource, this is where everything is happening. If there is
a question here, let me close these guys here. If there is a question, first of course, you also need to do some kind of validation. So, we also validate up front because you don't want somebody posting full Homer's Iliad as a query to your application. That will crash everything. So, you need to first validate the input. So, we validate the input and make sure that you are
not asking uh uh you're not passing a string beyond a certain number. And then also, we make sure that you are not trying to game the system by saying you are just trying to use system uh prompt injection or something. We We validate all of that. And then, we just render an answer. So, AI chat service takes the question and then a mode. Now, what is the
mode? The mode is pretty Which of the modes do you want? You want declarative? It's pretty easy to see. We can go in here. And then say, do you have Do you want agentic, orchestrated, in process? Or by default, we run declarative. So, if it's agent, we're going to run a single agent. If it's orchestration, we're going to run the orchestrator. And then, in process, you're going
to delegate to J Lama running internally. And then, by default, if you don't pass anything, it defaults to declarative, which is the retrieval augmented generation. Now, these are four patterns of AI that you can explore in your existing applications so that you don't rely on external parties for some part of your AI. Of course, you can't run everything locally. You can't fully be independent. But for most
things, running a local model works fine. How many people have tried Google's a Gemma for models that they released about a month ago? They are very powerful, very capable. I was quite surprised actually by their performance. And so, these are models that are coming out that you can incorporate into your application so that you don't have to wake up one day and then your AI provider has
banned your account or for whatever it is. Also, much more importantly, you have existing applications that for regulatory reasons EU has the AI Act that has some kind of requirement audit tracing and all of those things. Those are all things that you need to factor into your development and AI adoption. So, these patterns are available for you to explore in incorporating local first AI into your existing
enterprise applications without needing to go outside. Of course, you will need to fine-tune a few places. You need to adopt it to suit your specific application, but the whole point of this talk is to let you know that it is possible to do all of those things. First of all, in pure Java. Second, using your existing enterprise application component. You don't need to migrate to spring or
quarkus or anything. Your existing Jakarta EE application should work fine in adopting AI as part of your application. Any question? Do you have any question for me? No question. Are you sure? Okay. So, in the absence of any question, the code is available on GitHub. How do I send the link to? Do I send you a link and then you send it to them? Okay. So, I'll
send her the link and then she can share it with you and then you can take a look at that. I can see my time is running out uh down here. So. Okay, that's fine. So, I will just continue because I like to to continue. So, I will just continue here. Okay. So, now these patterns are available for you to adopt them as part of your AI
adoption into your application. So, you are not tied to a specific vendor. And generally, you should, especially for local development, you you you really should take a look at this. Now, if you go back to our presentation here, one second. Okay. So, here, if you go back to our presentation here. So, Ollama verse Jlama, external process versus in-process, there are a few differences in them. This is
an honest trade-off. Ollama runs a larger model in this example, in the application code, Ollama runs a larger model as an external process. Of course, you will get better answers, but you have a network hop and a separate container to manage. Jlama runs a smaller quantized model, Gemma old model, 2B, 2 billion, inside the JVM. You lose some answer quality, but you gain operational simplicity, zero external
dependencies. So, it's always a trade-off between which one you want to choose. In the end, it's an it depends situation. You would need to benchmark based on your own needs, and then pick one of them. But generally, you should start weaning yourself off wherever possible, weaning yourself off from third-party AI providers. Any question? Nope. So, you start weaning yourself off these providers, and then you don't wake
up with unexpected bill here and there, and all And this it is going to get very interesting if you read the AI landscape, because at the moment, what's is happening is that venture capital is subsidizing a lot of the cost for us. But when the time comes for AI providers to actually charge real prices because venture capital has dried out. I think we are all going to
start feeling it. So, the days when you'll be paying a few cents for a million calls will be over and then a few thousand tokens is going to cost you an arm and a leg. I think we'll all start looking for ways to run locally. So, before we get there, before the venture capital funds run out, I think you should start exploring running your own models as
much as possible locally. And I think Ollama and Jlama are two excellent examples that you should explore because they they have good trade-offs that you can measure locally without incurring any cost. Of course, cost in terms of time is always a constant, but in terms any external provider, these are these are existing components and frameworks that you can adopt. You can start trying. You can start pulling
in into your application and testing. You don't have to do a rewrite. You don't have to cut everything off, but you start somewhere probably with just one component or one aspect of your application. Test, look at the result, and then start finding ways to refine them so that in the end you are you are ready for the inevitable when the time comes for us to all pay
actual prices for AI inferencing, AI API calls, and millions per token, and all of that. When the time comes, it's going to get brutal, and so you should start now. You don't need to rewrite your application. As we've seen, it's the same construct that we know, the same Jakarta EE construct, the same data source, the same CDI mechanisms that you can incorporate into your application to have
AI first locally or local first AI without the cost. Again, as we saw in the beginning, somebody woke up and their access had been cut. And so, I am here to tell you that it is possible to wean yourself off by running your local AI alongside your existing Java EE or Jakarta EE applications without needing to rewrite, without needing to migrate, without needing to do anything other
than testing and finding a suitable balance between which of the existing frameworks and constructs works best for your use case. Yes, please. >> Hi, thanks so much for the information. I was curious, do you have an idea of what the time horizon is for that need to become more localized as a as a developer community? >> The time horizon, I can't give you a specific number, but
if you read the general atmosphere, last year for instance, more than half of all the data centers that were promised for AI were not delivered. And the other half were not completed. They were started, but they were not Recently, we all heard Oracle had to let people go so they can find funds to support AI. So, I I think the writing is on the wall. If you
read the general atmosphere, that a time will come when the venture money runs dry and we have to pay actual cost for AI. And when that time comes, the few cents we pay per million tokens will just not be there. And so, if you have an application that 60 developers, and then suddenly you have to pay actual price for APIs, I think you are going to feel
it. Very much. So, it's easy. You don't have to do a big bang. Start small with the smallest, the least expensive part of your application. Start with that. Start weaning those parts off to local-first AI. Test it, refine it, and by the time we get to that tsunami, you would have been ready. You would You would have identified your own patterns that work for you. And then
you you should be good when the time comes. But it is coming. I can't tell when, but it is coming. That is for a give That's a given. It's Any more question? Okay. So, thank you very much and that will be it. I'll be around. Anybody uh you want to have any more discussions, I'll be around and then we can we can hang out and have a
few more chats. So, yeah. Thank you very much.
