Practical Python AI in Java with GraalPy - Sachin Pikle

This talk presents GraalPy, an open source Python 3 runtime within Java, designed to enhance Java applications by utilizing Python libraries efficiently. The speaker emphasizes the powerful integration of Java and Python through GraalPy, allowing developers to run Python code directly within the Java Virtual Machine (JVM). By embedding Python in Java, it significantly reduces the latency and operational complexity usually associated with microservices architectures. The speaker demonstrates a sentiment analysis application using a popular Python library, Vader, highlighting the performance advantages of GraalPy, particularly when generating native images for Java applications. He discusses the setup and dependencies required for using GraalPy in a Micronaut application, showcasing how it enables seamless cross-language calls while retaining the Java ecosystem's reliability.

I am I work as a product manager for Graal and Micronaught Technologies in Oracle. And let's get started with today's session. Before that, I will start a quick timer so that I stay on time. Okay, so let's a quick show of hands. How many of you use Java? Okay, great. How many of you also use Python? Okay. Both Java and Python. Have you heard of Graal VM

before? Okay. And have you heard of Graal languages before? Graal Pi, Graal JS, Graal Wasm. Okay, so if um You know, it's been a long day. If you remember just one thing from today's talk, that will be Java is your main application backbone. You bring in Python where you need the data AI or any other libraries from the Python ecosystem. And Graal Pi will help you run

the two together, right? So, that's what you should remember from this talk. So, Java is great. All of you use Java here. So, you know, I won't talk much about it. Strong language. Also, AI support in Java is is accelerating. The Spring AI, LangChain, 4J, etc. But whether we like it or not, Python is still ahead, right? Python gets a lot of capabilities before they come to

to Java. Even things like Open AI, for example, when they had their client library or others had the same thing. So, my first wish list is I want to use by those Python capabilities which are not in But I want to use them from my existing Java applications. Second is we all use in Java we use Maven Central. All our libraries are in Maven Central. It's the

richest source of reuse. Um but it's not the only ecosystem. Right? There are in Python for example, Python Package Index or PyPI is a similar ecosystem. It is has a ton of libraries. The second wish list is I want to reuse these other ecosystems like PyPI or even NPM JS for example in my Java applications. How do I go about doing that? The standard thing would be

oh yeah, let me stand up microservice since the common architecture. So now you have a Java microservice and you have a Python microservice. They talk to each other over the network. Exchange a whole lot of data. And both have their own processes to follow. What time? Oh, okay. Sorry. Um and but this approach works. No doubt this approach works. A lot of teams use But it costs

more in terms of latency, data movement, and also in terms of the operations, release coordination, etc. So no doubt the architecture is valid. But is the additional cost of doing this only for a limited capability in Python really justified? Well, that's the other question that we need to look at. So with this I think it's a good time to introduce GraalPy GraalPy. So what really is GraalPy?

GraalPy is an open source um Python 3 runtime which is written in Java using the Graal Truffle framework. We have two modes. There is a standalone mode which you can use as a replacement for CPython. But today's talk is all about the embedded Python which means we will take the GraalPy embedded version and run it inside the JVM. Runs on any OpenJDK 21 plus distribution. But it

runs best on GraalVM because we have the Graal JIT compiler, which can go and optimize the execution of your Python code. And for people who are familiar with GraalVM native image capabilities, you can now convert your entire application as a native image including the Python that's sitting inside. So, what changes when you introduce GraalPy into the equation? How do things change, right? So, instead of having separate

services, now you see I've got a single JVM application and I'm embedding my Python directly Now, the Python is running Python code is actually running in process alongside my Java code. So, it's in the same process. It's not a sidecar pattern that we're using. And because these are in process calls, even though they are cross Java calling Python and Python calling Java, they are it's you have

all the benefits of in process. There are no network overhead, And one important thing to understand here is the execution even though it's Python, the Python code is actually sandboxed. So, it cannot get out and do things unless you explicitly allow things. So, things like creating a process, creating a thread, or even doing IO for example, accessing native code, or even environment is all blocked. There's a

whole list of API uh which is available in the context GraalPy context API where you can go and allow certain capabilities depending on what li- what the library that you're using needs. So, with GraalPy, what happens, right? So, GraalPy lets you use the Python capabilities from your existing Java applications without you having to give up the Java infrastructure. So, all your builds, everything that you do today,

you can reuse the whole thing. Build observability stack, everything that you have remains the same. So, I think I'll do a quick uh demo. Um so, what I have today is a little uh sentiment analysis natural language processing application. It's running on local, so I don't have any I'm not calling a LLM model from some cloud somewhere. And uh the other thing is I'm using the Vader

sentiment analysis Python package. If you are not familiar with it, it's a little thing. Uh I will upload and this is uh I'm uploading an e-commerce product review files uh so that everyone understands that. And then the Java application will read the file, pass it to Python, pass the review text to Python. Python will do the sentiment analysis using the Vader library, and then return the uh

values back to Java, and then we'll display it back to the user on the screen. So, let me jump into uh VS Code. It's um okay. to speed up things, I've actually run a few things ahead of time. So, I think I've kept the But I'll show you the uh the steps here. Um so, to run locally, this is a Micronaut application. For people who are familiar

with Spring Boot, uh Micronaut and Spring Boot are very similar in nature. When you look at the code, you'll realize it's very similar. It's identical. So, uh in this case, um just as Spring Boot has a boot run to quickly um start your app, we have a Micronaut run, which I've done here. The second option that we have is uh you can convert your application into a

native image, so you can package it as a binary instead of packaging as a JAR, which is the second option. And the third option is obviously you can do Maven package, which will give you a fat JAR or an uber JAR, which you can uh then use in the application. And we will go ahead and inspect the JAR, and I'll show you how the whole thing looks.

But right now, I think what I'm done is I've run the MN run command and started uh the Micronaut server. Um and think can go to the So once once you start the micronaut server, it it starts in this case it's running on my local host on port 8080. So I'm just going to the browser. Is this visible or you want to make it bigger? Visible, right?

Okay, perfect. So let me the same thing I said I will go ahead and upload a e-commerce product review file. I have a simple product review for I think a positive review first. This is for the reviewers, "I'm really happy with the coffee grinder. It works like a charm." So positive, right? It's pretty positive. So when I run Vader sentiment, the result is quick and um I

will show you what the library is. So for you, if you're not familiar with Vader sentiment analysis, uh it's available on the GitHub. Um and the way it works is um it will There is a compound There are multiple labels. There is one label called compound. It will check whether it's greater than 0.00 0.05. If yes, it will mark it as positive. If it is less than

minus uh 0.05, it will mark it as negative. If the value is somewhere in between, it will call it as neutral. So in this case, in my case, the value was positive because everything was good. And the output is shown at the bottom, right? So the output of sentiment analysis is shown at the bottom. So you have the compound score, which is 9.966. Uh and this one

has 62% positive words and 37% neutral words, nothing negative. Let's look at the second one quickly. This is um a negative review. And this one says a horrible product, right? So this is I'm not reading the whole thing, but it's basically a horrible uh vacuum cleaner. Um and then if I run the thing, it goes and shows me a negative value and the compound score is negative

0.8. So now let's Now that we've seen this, let's look at the a few things in the code. So let me go and show you what are what's happening behind the scenes. First thing I said there are three modes. One is the MN run, second one is a jar, and third one is the native executable. Just to save time, I've already run all three and I've captured

the startup times. So this is the native image captured the startup And um this is the jar I've captured the startup time. So let me just bring up the Excel. And so the native image is about 204 milliseconds. MN run was about you know, 2,500 and jar is about 2,900. So if you calculate the speed that native image brings to Java applications, you'll see that the startup

times are almost 12 times, at least 12 times faster compared to a jar-based deployment. So that's one of the advantages that you'll get with native image. So now I think let's look at the other pieces of code. So let's start with the pom.xml. This is a Maven application. So let me start with the pom.xml. Let me hide the terminal. So in the pom.xml, the first thing that

I want to show you is the dependency. In this case, I said it's a Micronaut GraalVM application. Micronaut has built special integration with GraalVM. So we have the Micronaut GraalVM module, which will make things a little easier, but you can still use Spring Boot and directly you can call GraalVM. That's fine. But in this case, Micronaut has made it easier. The second thing I want to show

you is the dependency, which is we have a special GraalVM Maven plugin, which will help you pull like in your Java build. So from your perspective it's a Java application, you're using the Java ecosystem, you're using Maven or Gradle for builds. Within that, we have plugins that have been provided. So, this is the GraalPy Maven plugin, and if you look at this library on line 87, I

am calling the Vader sentiment package from Python package index. This is not in Maven Right? So, I'm pulling a Python package inside my Java build using the GraalPy Maven plugin. Similarly, I've got the plugin for uh Gradle as well. And we recommend that you pin the versions. Usually, Python's a bit lenient, but we recommend that you follow traditional Java practices where we like to pin the version

for reproducible builds. Um the next one that I want to show you is uh how how have we I've shown you the pom.xml. Now, if you look at source main resources, I have uh this folder called org.graalvm.python.vfs. Now, VFS is a virtual file system that we embed in the Java environment. So, the standard convention is that we keep your Python code in the VFS folder under resources.

And this is an example of uh a Python code where I'm calling the Vader sentiment library. I'm instantiating the object, and then I've got a function called analyze review JSON, which will take in my file name and the review text. And then it'll call the uh Vader sentiment library on line nine, the polarity scores method, that will tell you whether um it'll check the value whether it

is greater than 0.05, so it'll mark it as positive or negative, and so on, right? So, it'll do the classification. And then it'll return the whole thing as a whole JSON object as a string back to Java, where it's passing the file name and the sentiment. So, this is what I think you can see in the output at the bottom, right? So, I've got the file and

I've got these values in the So, basically, whatever was returned to me by my Python function will first come into Java and then it'll come outside. So, if I have to show you how this works, the next thing to look at is a interface called um sentiment module. Now, sentiment module is a public interface, but if you observe the method the method doesn't follow the standard Java

convention, right? Like it is a It looks like a Python method with underscores in the in the in the method name. And it really is that, right? Because what I'm doing is I'm defining the contract of the API in Java, but the actual implementation is coming from Python. So, what what Micronaut has done to make it easier for for developers is they have given a annotation where

you specify the name of the Python module that you're trying to import or trying to use in the application. So, in this case, I'm using the sentiment app, which is what I'm giving. And the annotation will do three things for us. So, one is it'll go and create the execution context which GraalPy requires. It'll load the Python code and then it'll make this an injectable component which

you can then use in your service classes and your controllers and other framework components. So, let let's then shown you the file. Now, let me move to the controller. So, the next thing is the controller. In the controller, I have got the There's only one endpoint on line 23, which is {slash} review {slash} analyze. So, that's the endpoint which is called when I hit the orange button

on that screen. And then it'll call the GraalPy sentiment service {dot} analyze, right? It's passing the file name and the bytes. So, I'll go into the analyze method. This is my GraalVM sentiment service class. Um I am injecting or Micronaut is injecting the sentiment module for me, very similar to Spring Boot, right? It's the same methodology, right? Everything is injected. You don't need to instantiate. And then

the once once that is done in the analyze method, I am calling the sentiment module.analyzeReviewJson, the same interface that we had defined earlier in Java. So, I'm calling the interface, passing in the file name and the review text, and behind the scenes, Java will pass this to the underlying Python. So, it'll load the Python object, it'll run the Python code, and return the value back to me

as a payload. As a string as a Java string, right? We saw the Python string in the code. Now, it's coming back to us as a Java string. And then I'm using the Micronaut JSON mapper to read the payload and populate a an object um which I will use for further processing, which is called which is basically a record. It's a Java record. A nested Java record.

There is a file name, and then there is a sentiment um record, which is possible which has all those same values that we saw uh here, right? So, it's the same So, this is the I showed you the controller, the service, and the record. Let me now show you the output of um of the the whole thing. When you run the Maven build, I said we create

a JAR. So, let me let me actually show you how that would look like. Let me go back to the terminal. And I think I already have the terminal. So, I have a JAR viewer here. Let me open the JAR viewer and show you what got packaged inside the JAR. So, I'm running uh I'm opening this uh fat JAR, which was the output. Uh is this uh

I don't know if I can make this bigger. Let me try. I can't make it bigger, but I hope you can at least see it. Uh this is the VFS directory that we talked about. So, inside my JAR at the top is all are all the classes. And this directory which I've expanded is the virtual file system. If I expand the SRC, this is the Python uh

file which we had defined in the source code. And then in the virtual environment, this is like a Python virtual environment that you define in uh you see all the uh Vader uh sentiment library and the dependencies which Vader pulled as a part of the build. So, all of that is now packaged as a single JAR file in your application. So, I think uh with this I

will uh stop the demo here, otherwise I will run out of time for the slides, and then I will go back to the slide deck. So, we saw this in action. Now, I think I'll go ahead I will Now, I'll breeze through the slides because I think I've spent a little more time on the demo. And if you have any questions, we can take them offline. How

do you get started with GraalPy? See, this is a very simple uh I showed you a complex demo, but I'll just show you how to get started quickly. So, uh you have a simple class, you import the GraalVM context, polyglot context, you create a new context using the builder, and then you can evaluate any Python code. The same thing applies for JavaScript and Wasm. The structure is

the same, but today's talk is about Python, so I'll focus on Python. This is an example of inlining Python code. The Micronaut example was where we had code in a Python file and we loaded it, right? So, there are two ways to load Python in GraalPy. One is you can inline it either as a single line. This example shows you a multi-line uh Python program. So, I

could have taken the whole Vader sentiment uh function and put it in my Java code. That would have still worked. And Intel If you're If somebody is using IntelliJ, IntelliJ has a feature called language injections, which will help you uh format your language inject injected language also properly. It will give you code completion etc. etc. But we still recommend that you use a separate Python file. This

is how you would go and do that. So, the recommendation is you use inline code for small tasks like POCs etc. and when you have some real applications, you would switch to independent Python files. There are two ways of accessing the Once Once you have the Python file loaded, now Java needs to talk to Python, right? So, there are two ways of doing it. This is an

example of a Python function. There is a library called QR code and it does a simple make function. It takes a string and then gives you an image of the QR code for that, right? So, if I were to convert this using dynamic access, first I would get a handle to the value object on the line And then I will use a dynamic style which is get

member. And in the string I'm passing in the name of the function which is making this case and then I'm executing it with the parameter. This is like reflection So, the danger is if I make a mistake in the type If I make a typo in the function name, it will not be caught at compilation, right? So, it will You'll know it only at runtime. So, we

we only recommend this pattern for quick exploration POCs The second option which we saw in the micronaut example was that you define an with the same method that you want to use in the Python implementation. So, Python code has a make method which takes a string and returns an object. And then I get the value object and then I say on line eight, I say value.as and

then I pass the name of the module which will give me the QR code and then I can call it like a Java interface. So, I can say QR code.make and then your code looks like Java. It is type safe and much more recommended pattern. So, you define your contract in Java and let Python provide the actual implementation in this case. Uh I've shown you this using

the Micronaut Gradle dependency, but basically GraalPy GraalJS, GraalVM are dependencies which are available in Maven Central. You can just include it in your build. Right? So, in this case I'm including the Python package along with the polyglot thing, but the same thing applies for JavaScript or Wasm. Uh how do I import third-party packages into my build? I showed you we have GraalPy plugins. So, you can in

this case I'm importing the OpenAI client uh with a particular version. This is the Maven example of that. So, I'm Gradle and Maven. And then I think we showed you I showed you this as well, which was Maven and Gradle will help you fetch your Python packages as a part of the build. We will bundle Python code with the application, the same picture. You have a choice

of virtual file system embedded inside the JAR, which I showed you. It's also shown on this picture. If you don't like that option, if you have a use case where your code needs to access files from the real file system, then you can put it in the external directory as well. You can choose whether you want to put it in the JAR or you want to keep

it outside. We recommend bundling it in the JAR. Mostly, that's a good use case. You have one file. And you have the choice of JAR and native image. We saw the advantages of native image. One of the advantages, startup time. We saw that. And then recommendation is to lock um dependencies for repeated builds. We saw this as well. We saw the example of Micronaut Graal languages. So,

you keep your framework in charge of the application lifecycle. You define the contract in Java, let Python give the implementation, use your controllers, you know, service classes, etc., dependency injection in Java. And then return Java objects to the rest of the thing. So, it's the same pattern whether you're using Java or Micronaut or Spring Boot or other frameworks, it doesn't matter. So, you can use anything. Micronaut

has special integration and spring is I think planning to to add that as well. A little note on the performance and throughput process call because these are all in process calls, they're naturally going to be faster when using microservices. But GraalPy's strongest advantage I said is the JIT compilation. You're used to Java world, so you know what JIT compilation is. But it'll help you with if you

have pure Python code, we don't we're not dealing with any native extensions like C or Rust for example, then it'll help you with this. So, the best thing would be pure Python code and long running so that the JVM gets time to go and JIT and do the JIT. With native extensions, it's a slightly different because you are dealing with C code or Rust code. And in

that case, the JVM doesn't get a chance to go and do the JIT compilation. So, the performance will be on par with CPython. Otherwise, if it's pure Python, the performance will be better than CPython. In terms of throughput, there are there is a recommended pattern. So, instead of using one context and having multiple threads go at it, then you get into the issues with Python um locking.

So, instead of that, we recommend that you have one context per a worker thread. And then you have a shared engine. There is an engine above the context which will do the caching of And keep the hot code cached, right? So, that can be reused. So, the sweet sweet spot is pure Python code, in process calls, JIT compilation, shared engine, and one context per worker thread. So,

if you have this combination, I think it'll work well. You will get better performance and throughput with your Java plus embedded Python application. What about the Python I said you have access to what if you know the first question was I my wish list is I want to access packages from PyPI, right? So, what our team does is with every um like we have the continuous integration

setup for about 600 Python packages. There is a URL that I've included on the top which is on our website where we publish the packages which we test and we say how much compatibility we've achieved with each package. So, you can go and take a look at it. But measure I I would say a lot of Python AI ML data science packages just work out of the

box. And these are the examples popular ones like NumPy, SciPy, Pandas, Hugging Face, PyTorch and AutoGen are some of the libraries and then you can click on that and there's a whole table that you can see and how much percentage is passing etc. Uh there is one thing to note. I said if um application depends on Python libraries that use native extensions like Rust code or C

code for example, then you need to be aware of one thing which is a lot of Python packages today don't support GraalPy. For example, if you go to PyPI and search uh there is a string which you can put there and search for the implementation. You can select GraalPy and you'll see a list of packages. So, when uh package owners have now started adding binaries for GraalPy.

Uh the key ones here on this slide are Pydantic core if any of you uses that or Jitter which is a faster way of doing JSON processing in uh Java. Uh it So, in addition to library owners who we are working with and getting them to onboard, we also publish our own wheels on this website. So, if like some of the libraries like SciPy etc. you will

see that we support them through our own wheels. So, our team develops binaries and pushes them to uh to this site and we have plans to publish more. Uh but the one thing to note is if it's unavailable, right? If uh things are not available, then GraalPy will attempt to build things from source. So, we'll attempt to build from source, and then you might run into complications

because then that dependency then see might have other dependencies and so on. So, the best thing is uh pure Python code followed by Python packages where wheels are already published either in PyPI or by us. And we have Picnic who's our customer and uh who's one of the like they they don't pay us, right? It's an open-source product. So, but they have a talk at uh they've

done a talk at Devoxx UK recently. I've included links. They are implementing a rule engine. Uh Picnic is an online retailer. Yeah, and uh they have a rule engine where their business analysts can go and define JavaScript rules or Python rules, and they will run it using uh Java uh they're using both GraalJS as well as GraalPython in the same application. Uh closing takeaways, um keep Java

in charge of the application. Bring Python where you really see value, right? So, if you see a library which is not available in Java or uh it brings real value which Python Java doesn't have an equivalent, then use it. Ship the whole thing. Package and ship the whole thing as one single deployment unit. And the important thing is make sure you validate the package compatibility. If there's

too much of dependence on native extensions like Rust and C, then probably don't use it, right? Uh the sweet spot is slightly different. evaluate that, look at the performance trade-offs, and then do it. It's an architectural choice. It's not a compulsion or a universal rule. It's one more toolkit in your you know, like it's one more tool in your toolkit. That's what I would say. This is

the website. You can go to the website. There are two doors. Left-hand side is the embedded door. If you click on it, you'll get into the embedded world. And the right-hand side is uh standalone. And uh sorry, uh you want to take the picture? There's llvm.org/python. So, that's the site. And then I think the last slide is links for all the demos. You can here the slides

will be circulated as well. So, you need to be able to get it. We have a whole list of demos outside. I have given the last demo is mine, but there are more sophisticated demos there. So, thank you very much. And >> [music]