About this talk
In this talk, the speaker Milan discusses the complexities of working with Large Language Models (LLMs) in Java. He begins by analyzing the gap between high-level AI concepts and the underlying mathematical operations necessary for implementation. Milan explores practical applications of AI, including tools like Grammarly, and shares his own experience creating a text editor that uses machine learning to evaluate sentence complexity based on features extracted from language data. He addresses the challenges of model training, including the need for training data and the use of classifiers to determine sentence complexity. Milan also covers the nuances of running models, including ONNX format and the importance of tokenization. Overall, he aims to empower developers to engage more deeply with AI technologies and implement them in their own applications.
Full transcript
[music and singing] >> Okay. So, Milan is always special. Yes. Milan was from the very beginning for many years with us. And everything he does is perfect. And uh I think that today you're going to learn a lot uh exactly about what this LLM is. At least I didn't know what LLM is until yesterday. >> [laughter] >> Hard to believe that. >> Yes. And um now we
are really happy to have Milan. He's always a great friend of our community. And uh having him is something special. And yes, as a great professionals professional, he will tell you really cool talk. So, please enjoy. Well, welcome, Milan. >> Thank you. Thank you. Wow, what an introduction. Oh, uh that's uh Mitja. I always set the bar high. So, you can you have the opportunity to be
disappointed, right? Uh right. Okay, I got to admit it's a bit of a clickbait title. Uh you know, but uh then again, everyone's talking about AI, so it's not that much of a clickbait anyway. Um but um we'll go through a lot of Java code. that will be mostly multidimensional vectors. So, it's not fun. just be warned. Uh and if you want something more entertaining, then now
is the time. Okay, um I've been at this conference so many times. I've represented I if I remember right, like three companies at this got very conference and it's the first time I'm here representing myself. >> Um, I'm currently an independent consultant uh helping companies mostly introduce AI in their workflows whether that has the OC or some other AI development flows. Um, also I've been doing software
architecture for pretty much my entire career. So, that's that's another thing that I consult. Um, all this to say if you feel the need to talk to someone, give me a call. Uh, all of the things that I'll skim through in this talks are in a lot more details in a few blog posts that I've post posted recently. So, if any of the things I say today
kind of you know, brings up some sort of curiosity, you can go and check the blog post for for more details, diagrams, and and things like that. And so, all this started sometime ago when I started looking into the more the whole AI thing uh and models and and you know, the the whole craziness. And I got like picture of two things. One is the heavy math,
all the computational graphs and all the magical things that happens with machine learning and scientists and whatnot. And then the code. And what I was missing was a bridge between the two. It's like, oh how do you go from hello, how are you? What's the weather today? To multiplying matrices and numbers and you know, heavy crazy mathematical operations. So, start digging into this and by by that
I mean I started looking at where is this quote and quote AI applied to. And it goes from all the way from like chat GPT through, you know, all the other chats, through all sorts of different applications, for example, like face recognition and face login, face ID, through Canva, which is now heavily using AI to generate whatever, to, you know, video avatars, or mobile applications that you
can take a photo of something and it's going to count how many of that pieces are there. And then this is the just like a very very few examples. If you start looking into this, it's like a a whole universe out there of of applications for AI. And so the question that kept bothering me is, can I build those? I mean, I can I know pretty much
what an LLM is. I know how to call an API to, you know, open AI or Anthropic or whatever. Like, yeah, that's that's easy peasy. But can I build something more? And do I have to use somebody's services? Do I actually have to use OpenAI or chat or or Gemini or Anthropic or can I do it on my machine? Like, what does it take? And can I
do it with Java because my entire career has been in the Java field. I'm recent years, I'm working with different languages and whatnot, but still Java is the thing that's kind of like made me who I am. So, it's kind of interesting to to see if if that can happen actually in Java. And so, I picked up a thing that's Grammarly. You're familiar with Grammarly? I know
it's not very good with Bulgarian. but if you work with English, it's one of the tools that actually helps a lot with grammar. And they have one of these features that at some point it tells you this sentence is too long or too complex for the reader, so you should consider shortening it. I was like, all right, that's a simple enough feature. I want to implement in
an application of mine. So, how do I go about that? Do you know Can you Can you Can any of you feel confident at this point that you can implement this feature in a text editor? Two. Two people. Okay. Yeah. Good for you. I wasn't. Uh so, I decided to try it to to to start experimenting and see how that goes. Everything you will see uh the
code is on GitHub, obviously. Uh at the end there's going to be a slide with QR code, so if you want to wait there and uh you can get it from there, but for the impatient just thought I'm going to put a slide with it here. So, here is the demo app that I created and it's a simple text editor. You paste some text and you click
this button and it highlights some text. It highlights it in yellow if it is reasonably long sentences that you should consider and highlights it in red if it's definitely very long and complex sentence and you should definitely not use that. So, how does that work? the application, as you can imagine, it's not very complex. It's a simple uh Java Swing application uh we have fixed two constants
in there. We say we have something that is the score of the complexity of a sentence and basically say if the score is between 0 and 1, so if it's above 0.45, paint it yellow. If it's above 0.4 65, paint it red. That's basically the the whole logic in here. But, how do we get the score? So, we get the score from this predict difficulty method. And
now, this is where the things become interesting. So, you pass it a sentence. I'm not obviously not showing you the whole splitting into sentences thing, which is trivial. Uh so, you get a sentence from that text, then you pass it to this method, then you get a then you get a score. How do you get a score? Like, what is the logic what is the algorithm that
tells you the complexity of this sentence is between uh between zero and one is like point 3578. Right? There's no algorithm that that does this. So, you have to go and do some probability calculations. So, we need a training data. Now, any machine learning experts in the room? Data scientists? Okay. If you are in the room, for some reason not revealing yourself, for the next 10 minutes
you need to do Stick your thumbs in your ears, and then go like this. Right? And stay like this for 10 minutes, otherwise your head will hurt. Okay. So, now this is me pretending to be a data scientist. and don't do this at home. Okay? So, but basically, if you're not Google, and you're not in a position to ask millions of people to play stupid games and
rate sentences, so you can collect all the data, and then build some, you know, enormous base of information of what is considered long or complex sentences, have to improvise. And the way you improvise is you get some sentences from somewhere. I got a database from hugging face that has like two over 2 million sentences in And you kind of simulate what, you know, the big guys would
do. Uh, so I extract features from these sentences. And what is a feature? Feature is a mathematical thing, a number that represents something about a sentence. For example, it's the number of words or the number of characters or the average length of the word or whatever. If you're a mathematician, you kind of know what makes sense. For me, I just decided to go with these three things.
Right? And so once you figure out what is your feature, a number that represents something about that sentence, you can do things like classifiers. And a classifier is a simple thing. It basically gets those features about the sentence and produces Uh, and and that's it. And so now you can have several or as many as you wish, I have three, classifiers. This is average word length classifier.
This is char count classifier. And there is a word count classifier. So basically you're saying like, "If this has so many characters, consider it long or whatever." Kind of simulate how people would think about Now, and that that that after you simulate how people think, you got to Now, how you simulate people? In Java, we do this with threads. Right? So you can create a number of
threads. I have 10. You can go with whatever you want or whatever your CPU is able to handle. And then you create those threads and basically through all the sentences and say rate them, score them, you know. Um, and how a virtual person behaves, you can specify that. Here's a P1 that, you know, 60% of the time pays attention to the character counts, and 30% pays attention
to work count, whatever. You can define what whatever you want, and you know, you have some artificial people with artificial brains um scoring And you can run this. Uh, basically. It's going to take some time. Depends on how many sentences you have, depends on how fast your machine uh, and whatnot. And you can see here 10 threads going simultaneously, uh, calculating the scores for the, uh, sentences.
And what you end up with is, uh, in my case, two files. Actually, there's one file that is being later on split into two files. One is the training data, and the other one is the test data. Now, you may be wondering what's in those files. So, let's let's open one of them. Those are CSV files, uh, which basically contain the sentence and all the scores that
our artificial people gave to that particular sentence. something's wrong with this, or I'm pressing the wrong thing. Ah. Uh. Never mind. Ah, okay, good. Um, uh, and uh, all the the ratings that the people gave, and the the last column is the median. So, we have 10 people rating, uh, a sentence. You get all the the the uh data that they gave, and the last one is
the median of all those. So, this now you have a data. Now you have information how complex a sentence Then what you can do is train a model. How do you train a model? Well, you need a mathematical function that can calculate how complex a sentence is. Again, you need a friend mathematician who can tell you this. Um my mathematician friend said use Which is basically get
a number, multiply it by the number of chars, uh and then get another number and multiply it by the number of words, and then add those and add another number to In more science data science terms, this is the function. It's the sum of weights mul- multiplied by features plus bias. If you want to use the the wording of the machine learning world. So, now with this
in mind, we can actually train a So, to train a model, you basically you go and load the uh data files that we already have with the scores. So, that's what we do here. And uh we load the train sentences and the test sentences and then we have this internal representation of a model, which is basically a class that holds our weights, two of them, uh and
the bias. Right? And so, then we just go and say train. And once you the training is over, uh uh no, so sorry. First, uh we need to kind of extract and normalize so we can the the number of words, the number of characters, all those things from the sentences, and then we can say to train, and then we print the result, what weight we calculated for
each. And so, fine once we do this, we get our test features, we do the same, and then iterate over the test sequence how accurate our model is. And once we're done with this, and we are happy with our model, we are saving it. Now, it's fancy way to say Java serialization. And we'll come to that in in in in in a bit. I kind of skimmed
through the train model to the train method. So, let's look at how complex that train thing is. And something's really wrong with my pointer, but it's okay. So, basically, what we do is we start with filling this with numbers, these with some numbers. We set the bias to zero. And then we go through a number of epochs, and those are the iterations, how many iterations we want
to go over the data. And so, we go I think it's 200,000 in this example. And then we go in each iteration, in each epoch, we go through all of the examples in the training data, and ask the model to predict a score for that sentence. We'll predict something. Obviously, it's going to be far away from what we have from the our humans. Right? And so, then
we calculate the difference between what the model gave us and what the people rated, and then we create things like um what is called gradients that shows how far away are we, and then we adjust the weights and adjust [snorts] the bias, and we go to the next epoch. And we do this over and over and over again. And once we're done, it produces something. It calculates
those weights, right? In our case, what you will see it's produced the the what you will see here as trained model parameters is the weight zero represents the that we multiply by chart chart is 1.1151, and the other one is 1.1035, and and then we have minus 15 53 whatever for the bias. That's our model. That's the those XYZ that we have to store somewhere. And then
every time we want to calculate a sentence, how complex a sentence is, we just multiply using that formula. Uh and we tested this, and it shows 96% accuracy, where accuracy is defined we are 15 per less than 15% off. that's where machine learning folks get headache cuz this is like yeah, not not not not anywhere near acceptable. But for our use case, it's more than enough. And
so we save this into a file so we can use it later. We can use it with different applications. We have our model saved into a file, and then in another application we can load it and use it. And the act of using it is called inference. And so, this is where the our predict uh uh, calculate score from the demo. come uh, predict difficulty, sorry. Uh,
predict difficulty method that I skipped earlier. And if you look at how it is all implemented, it's super complicated. It basically repeats the logic that we had in the model. Um, so basically do the does the calculation, gets the model, gets the weights and the bias from the model, and then multi- gets the features of the sentence and multiplies them, and then you get the score. That's
the entire of uh, of this application. if you've been doing Java or any programming language for that matter long enough, and especially if you've been doing architecture, your head probably hurts at this point. It's like, this is nonsense. Like we store some numbers somewhere, then we load them, then you need to know what the algorithm was so you can use the same algorithm in a different applications.
You know, we talk coupling, decoupling, all those things. Forget about it. You know, we just go wild. Save numbers in serialized Java files, and you know, believe it or not, this is how the whole AI happened. This thing is hugging face. As of today, as of when I take took this screenshot, which was a couple of days ago, it has close to 3 million models publicly available.
I wouldn't be surprised if 2 million of those are serialized Python objects. Uh, I don't know. I don't have the stats, but I wouldn't be surprised. That's how the whole thing happened. People just build those things, generate those numbers, those weights, put them in files, throw them up there, and then do a read me that basically says, this is the algorithm, right? And presumably you use Python,
so you just call the algorithm in Python, load those weights, and voila! You have an AI working. becomes a real mess when different vendors start trying to figure out how we deal with this. Like she's can't you know, just can't store serialized Python data and call it AI model. So, different formats of data storage start to popping up. And when you go to hugging face and try
to look for models, you will see a whole jungle of, you know, different files and you have no clue what to use. Or at least I didn't. So, a friendly walk through the jungle of um um model files. They're largely in two categories. Uh weights only formats is what we just did with our Java AI. It's basically a a format where you just save the numbers, the
weights, right? And you don't know what the algorithm is, except they may tell you this is the the weights for this what they call it in AI in the machine learning world architecture. So, if they say if they say if they say this is Transformers, presumably you know what the mathematical algorithm for Transformers is, so presumably you know how to use those numbers. If you don't, good
luck. Uh right? But all those formats, they're basically weights only. They only have these numbers in them, and that's it. Now, with time people start realizing this is not always optimal. So, there are new formats that also contain not just the weights, but the computation graph. So, you now not only know what the the weights are, but also have the algorithm of calculating them. And one of
the probably most popular one is the PB extension, which is from TensorFlow, and it stores the computational graph as a protobuf file. They also have a version that can store different things in a folder. with a specific structure, so you can load this and of course use it with uh with Python. TF Lite is another one. It's basically to uh optimized for mobile and web. and this
one is particularly popular because that's the format that um a llama cpp uses. Uh and on on top of that a llama uses it, so most of the large language models that you use with a llama are distributed in in in this uh format. And relatively new kid on the block is uh ONNX format, and this is an open neural network exchange uh that is one of
the very few that actually is cross is platform independent. So, you um with different languages and and whatnot. And you'll see in the next few demos how I'm using it uh in Java. One thing that I found extremely helpful when I start working with this is this app. It's called Netron. You you can load it from the internet, like you visit the URL, the application loads, and
everything that happens after that happens in your browser. It doesn't send data to to the the server. It's all client-side. And what it does is you can give it a model, a a file that you've downloaded, and it will print how that computational graph looks like. So, here you look at a uh and you can see all the nodes in in that in that model. If you
click on the node, it's going to give you an information what that node does, uh what the computation is, what's the input, what's the output, and whatnot. It's it's a very um very interesting one. For for but before we dive into that, I said mostly self-contained. And the reason the reason I say mostly is because even those formats, they don't have everything you may need. So, one
of the most common missing things is a tokenizer. Basically, LLMs particularly works in this way that when you send text, it converts it to tokens. Tokens come from a vocabulary. So, when it sees a word and gets a token out of it, it doesn't send that word as a as a parameter to the model, but it goes to the vocabulary, figure out what's the ID of that
word, and then sends that ID. Right? And so, to to prepare the data for an LLM, you have to do this tokenization. And for that, you need a token a tokenizer that tells you which what's the IDs of different tokens. And that information is typically not found in this model, so you need to know which model uses which tokenizer. Right? So, yeah. Um there's also different files
often attached to this that are like information about how to process and whatnot or configuration. Uh so, yeah. So, that just that you know uh what you're up to if you start digging into this. So, I have a model. Presumably, you have all the information to run it how do you run it? Before you can run that, you need to understand how those models work. Or what
is the Please. Yes. What is the the anatomy of that model? So, there's different ways to slice and dice this, but I find for me personally to understand that is to divide into like four different stages. The first is ingestion, and this is where you have to and and remember, we're talking about working with the model directly, not with somebody's API or service or something. Somebody's API
and service is what does all these things for you right behind the scenes. So, the first thing is ingestion. You get some data, and that data most likely, 99.9% of the time, is not in the format that the model expects. So, you have to do something with this data to prepare it in the shape that the model expects. You can't just throw numbers at it and and
expect it to calculate something. You have to do something. You have to prepare it to be in the form that the model expects. And what model does internally, typically as a first base, is a projection. It gets that data and converts it to some other data that it makes sense for the calculation upcoming. So, for example, if you talk about LLMs, you send it a vector of
token IDs. So, you get the words, split it into tokens, and what you send to the model is 1 3 755 or whatever, like the the IDs of the tokens. And that is useless for the LLM. What it does is it converts each of these token IDs to a vector of features that represent what that token what is the semantic context of that token? Those are things
that it learns during training. But that's the projection phase. It kind of it gets your data and converts it to something else. And then it's the processing phase. This is the large mathematical computation that does all the magic, which honestly I have no clue how it works. But it does the job. And at the end, that processor gives you some numbers, some tensors that have some semantic
meaning. And typically that's very large amount of numbers and typically what not what you want. And so what you have at the end is so-called hat or hats. A model can have multiple hats attached or or and hats is also swappable. You can have a processor and then attach different hats to it and get different results. So for example, an LLM typically will produce next token, but
you can swap the hat with another hat and instead of producing next token, it could be used for classification to tell you whether that text is about something or not. Right? And you don't have to replace the entire model. All you have to do is replace the hat. So, that's so much of a theory. So if you look at the diagram of this model particularly, um so
you click on the input node, what you will see is the uh input box here, what the model accepts. And it's a I know it's small and in next few slides I'm going to zoom in, so you're going to have a better view of this. And it's not important for this one, but basically this tells you that the input is a vector is a um tensor, four-dimensional
tensor, uh um that has parameters like 1 3 and each dimension is like 1 3 and whatever dimensions. So that's what the model expects. This is how you have to put the numbers together to pass it to the model so it can calculate something. And then, this is the the the next few mathematical operations is the adjustment phase. Like how it it it it converts those numbers
to things that make sense to it. And then, the entire thing after that is the processing body. Like tons of tons of math happening in there. And this is very small model, by the way. Like literally very small. So, that that whole thing is the body. Uh the the the processor. And then, if you look at the very very bottom of this particular model, it actually has
three hats. And you'll see them right here. And it produces three different And if you look at the outputs, you'll see it has nine outputs. And And those hats produce nine things. So, that's basically what we're talking about. And we'll be going to We'll be using this model in a second, so you're going to have a chance to look at it closely. So, you now understand how
models work, conceptually. How do we run them? How that whole thing inference work? So, obviously you need the hardware. Uh but then, how you talk to this hardware? Because there's different types of hardware. You can have Nvidia, you can have Apple M something like I do, or IBM chips, or whatever. There's those tons of things, and you need something that understands how to work with those things.
And those are Some of those are mentioned here. Um so Uh uh sorry. Uh this is the the inference runtime on top of those chips. Right? That That is the thing that understands what comes from you in terms of numbers, right? And knows how to put it on the hardware to do the calculation. Right? And so, for example, what we're going to be using today is ONNX
Runtime. ONNX Runtime understands ONNX formats. It can load model in ONNX format and knows how to put them on different architectures. So, in this particular case, I'm running on Apple M4. It knows it's Apple. It knows how to prepare this. It knows how to use my GPU. If I were running on Nvidia hardware, it would do the same thing with Nvidia uh and and whatnot. this is
the overall of how that thing works. Now, let's look at some examples of code to see how uh it is to do this with pure Java. No Python involved. The first thing is a facial recognition. How many of you have done any facial recognition or facial detection? Yeah, I figured that much. Uh so, it's not Surprisingly, it's not that hard, but you know, it's uh it requires
some digging. So, here is a uh the Buffalo L model, one of the very famous models that many image processing programs use to um do facial recognition. Uh actually, that's facial detection cuz they're two things. Facial detection is the like this tells you there is a face. Now, facial facial recognition is on top of another model kind of on top of that that tells you this face
and this face are the same face. Uh right? That's So, so it's not recognition, it's detection. It basically tells you where the faces on the pic- the faces are on the picture. So, this works with uh uh the expect- And this is the model you were buil- you've been looking at uh for the in the previous slides. Um so, uh how we use it is we instantiate
the Onyx runtime environment. and then create a session. Session is how we talk to that runtime. Uh and then we load the model. I have written a small cache that loads the model from hugging face if it's already loaded, loads it from my disk. And then creates a session with that model. That's basically telling how to send this data to the uh to the model. the code
after that is uh preparation and execution. The model expects that the image is in RGB format with no transparencies. So, y- you could accept images from all kinds of different people. You don't know what they are, so you have to prepare them to be in the format that the model expects. So, this is the first thing you got to do. Convert them to RGB. If there is
no alpha channel, remove the alpha channel, blah blah blah. Now, the next thing you need to do is resize and pad. This particular model expects images to be 640 by 640. Now, chances are you're never going to get exac- you know, you're not going to ask your users to exactly give you 640 by 640 images, right? So, what you need to do is resize them before processing
to be no longer in uh 640 and then pad them. So, put black pixels in the in the spots where they don't fit. And this is exactly what resizing pad does. >> [snorts] >> And then, you go through the image and get all the pixels and the three values for each pixels for the red and the green and the blue. And and this is your float of
pixels values. And then, do some mathematical things and then you prepare them finally for the model. you convert that array of numbers into the format that the model expects. And the input here is a tensor with a shape of 1 3 input height and input width. Now, that basically says for every height and width, so every point in the X and Y, the three is the values
of red, green, and blue. Now, what is the one? If it's when you start looking at this, you're like, that makes no sense. I'm just working with one image. Like, why I add the dimension? Well, it turns out in if you do a production system, chances are you're not going to just run one image. You may have tons of images to run. And you can actually, and
you should probably, weights and and predictions for multiple images at the same time because computationally, it's much better and much faster to send 10 or 100 or whatever your hardware can handle images that can be that can go through the computational graph at once than do it sequentially. So, that first one dimension is basically telling you which image that those pixels belong to. Um right? And so,
that's pretty much it. Then you create a tensor. Tensor is a thing that ONNX expects from that input. the the the input, that's exactly what it expects. Now, as I said, it gives you uh nine outputs. Now, the [snorts] reason it gives you nine outputs, these nine outputs are actually three times three. Every single head of of this uh goes with different So, basically, how that model
works is it creates a frame of so many pixels, and then it's kind of So, if that's your image, it creates a box, and it slides that box over your image like visually cuz all it does is calculating numbers, right? And it basically tries to match things that it knows. >> Uh right? But, it doesn't know how big is the face on on the on the on
the photo, right? It could be very tiny one. It could be very big one. So, there are three different sizes of of uh areas that it slides, and that does the three heads. And so, basically, it tells you, "I'm going to go uh the first one is uh 1,200 uh because this is a I'm not going to do the math in my head. Uh right? But, it's
basically like the small rectan- go the bigger rectangle and the and the bigger rectangle, and those are the three. And for each of those heads that goes, it produces three The first is the probability of there is a face at this position, right? And calculate some probability. The second is the probable the the box. So, it gets you like at this point, there's probably a a box.
The second is the size of the box. And the last one is some features about where the eyes are, where the nose is, where whatever. This is why you get nine results out of this model. And this is what the model produces every single time. So, if you just care about the faces, you're just going to grab pieces of that information, but you can grab a lot
more. So, this is the the three uh the three heads, basically. And if you look at the code how we get this, Uh this is exactly what we do. Uh Uh we get the output tensors from the result after we run the model. And then we go through the three different shapes that are because we get the result separately for each of the hats, right? And so
we iterate over the uh the results and we get only those tensors that we are interested in, convert them to whatever we need to convert. This is the math. And remember we scaled the image to 640 by 640. So we now need to do the math. When it tells us it's like three pixels here, we need to calculate this to where it actually is in our actual
image, in actual size image. So there's a lot of math going on here uh to to do this. and so finally, you store that as a detection. So you basically say at this point at this coordinates X and Y, there's probably a face and the height and the width are like this and the confidence that there is a face is like that. That that that's how it
works >> And if you don't believe that it actually works, here is the proof. You can click on run and it's going to open an application and then you can just load an image from a file. That's going to be a very famous person. and it's going to detect the face. Okay. So that's the uh the first demo not going to have enough time to go through
all of this. Um so I'm just going to do uh I'm going to skip GP GPT-2. Uh so GPT-2 uh is is a fun model to work with. Uh and the in the only reason I put it here is actually is because it's fun, because it's the the the grandfather of the LLMs. And it has it shows you how to how to do it Actually, I'm going
to have to do this because if then the other model will not will make no sense. So, if you look at the GPT uh, model, you will see that it's very big. what happened here? Please do. Uh uh uh Yeah, that's exactly where I Okay, so, um you will see it in a hole in a second, but it it has two inputs. One is batch size, again,
because you can put multiple inputs at the same time, and then is the sequence length, like whatever how many tokens uh you have. It also has the attention mask, and that's also thing that most LLMs have. It basically uh tells you of those tokens are actual tokens, because if you have multiple inputs, you can have one one input that is like 10 tokens, and the other input
is 20 tokens, and but the the sizes would be whatever by 20, so the first token the last 10 are non-words, right? That's why you need uh an attention mask to tell the first one, "Don't consider this." Um all right? And so, and then it puts logits. And logits are basically for every And you can see here now the shape. The shape is the batch size, or
how many, um um the the sequence length, and 50,257. Now, 50,257 is the vocabulary size of GPT-2. That's how many tokens it knows. Everything that you put into the chat get converted to one of those 50,000 257 tokens. That's That's it, right? And so, what it gives you is the probability for each of these 50,257 tokens of being the next token in the sequence that you provided.
That's how the model That's everything the model does. Nothing more. Right? And um it also gives you these things, key value stuff, which I explain, but um hopefully um it will work uh for Well, we'll know for the next demo. So, this how the model looks if you look at the diagram. And when I start scrolling, you will see that some blocks are repeated. you can get
the pattern. Like, things are happening and then the same thing will happen in the same blocks and the same blocks. Those are so-called layers. And all the LLM models, most of the LLM models, at least the uh the the ones we work with are the designed this way. They have layers, so-called attention layers. So, they um vectors on every layer, and then they can recompute and then
pay attention to what they computed earlier, and yeah, whatever, complicated stuff. Uh but what is important is that if you click at the output, you will see that it provide it gives you logits, like we explained what it is, but also give you all these key value stuff here. And when we scroll enough down, you will see that they're in exactly the number of the layers. And
the reason is every layer produces those keys and values. So, in human terms, this is basically an information of like the keys are what other tokens are relevant to this token. The values are how other tokens are relevant to this tokens. And we'll see in a second how that is and why. So, unfortunately, I cannot speed up recorded videos. I'm just going to have to wait for
it to scroll. Uh but basically, it's the same uh the same thing. Uh you you create the environment uh and then you you load the tokenizer this time. This is what I said, it's a separate file. And from the tokenizer, you get the tokens. There is one special in GPT-2 that is end of text. Uh and that token tells you that's that that we uh have finished
uh our conversation. So, the next thing you do, you load the model from the cache, and you ask the people to provide a um uh their their prompt, you use the tokenizer to convert their text into tokens. Uh and then you collect those uh so you can respond back to them. Uh you also need to create the attention mask. In this case, it's easy because it's just
one chat. Uh and then we go through the what's called the generation loop. So, until we reach the max tokens, we basically tell the model to predict the next token. And we do this with session run and inputs. So, the first thing we send is the tokens that we converted from the user input. Now, when it when we get the output where are you? work. Here you
go. When you get the you get the uh the logits, right, for the last word. You calculate the You get the the one with the highest probability. That's, by the way, not how our model works. There's different algorithms for picking what what you want to get from as a result, but we get the one with highest rated. And then, we put it back input. And then we
send it back to the Right? You will see here. And we'll do this until it tells us is the last word. Right? And this is pretty much how all LLMs work. Um and so, if you run this, it will prompt you for a text. And you can say, "Hello." Something. Uh I hope you can see on the bottom of the screen. And who want to guess what
the response would be? Funny, isn't it? That's not what I expected. Um turns out, after a lot of research, was not trained to end the conversation. It just doesn't know how. It keeps generating tokens. Right? And so, what you see here is our loop reached the max token count. That's why it stopped. Right? Otherwise, it would just keep generating tokens in a Uh even though it has
end of text, it it it does it It does have it as a as a token in the vocabulary, it just doesn't know how to use it. And that's why, by the way, if you go on Hugging Face and start looking for LLM uh models, look for the ones that have instruct in the name. Cuz it turns out people train models to generate next tokens, but not
to be conversational. I said like training a model to know how to have a conversation and when to stop apparently a different thing. uh that's the the fun part about uh about GPT. do I have a hard stop or can I go two more minutes? All right, cool. So, I'm uh just going to skip the next uh small two is basically a better uh an LLM that
you can use on mobile devices and uh uh and in a web application, and it's interesting, but we don't have time for it. So, I'm just going to skip to a different type of model. And this is a text-to-speech model. Uh and the reason I wanted to show I want to show you this is because it's not one model, it's four models. Because we are very often
we are like, "Oh, I'm just going to use a model." But in reality, many times we are using multiple models. So, in this particular demo, to generate to uh to generate uh to synthesize uh voice from text, you have to use four models. This is The first one is the text encoder. The second one is a duration predictor. The third one is the vector estimator. And uh
the the final one is the one that converts this into uh a voice. So, >> Hello. Welcome to this demo. It's not often that an AI model has the opportunity to speak at a conference. Thank you very much for these 12 precious seconds of your attention. >> Yeah, it's a recording, but believe me, if you run it, it's going to you're going to get the same result.
Uh so, I'm not going to have the time to go and explain how all this works, but basically what you have in this demo is uh four models that work together to produce that. The first model letters sounds or or rather how they call amplitude whatever the the the the right term for this is. Then then another model gets that and estimates how long that video that
that the the final audio would be. Now, having these two, they go to the third model, which calculates all the the timbers, the the the speed, and and all kinds of things. And to the third model, there's also a different configuration where you can provide some different numbers representing different voices. And so that that it applies like voices and does all the mixtures and and whatnot. And
it and it has the semantic meaning of like this noise after this noise makes sense. But this after this doesn't make sense. And basically the model works in a way image generation works. The that third called diffusion I believe. It is a noise reduction. You start by generating an audio that's pure noise. And then you go iteration after iteration after iteration and you remove the noise. Right?
Uh and then you kind of clean that audio uh up until you get something that sounds like what you just heard. A lot of image generation models work the very same way. They you say I want an image like this, they generate noise and they start clearing the noise. The reason they can do that is because that's how they've been trained. They've been trained by getting images
or voice or audio and converting it to noise. So they know the path from real to noise and and and inference go the other direction. All right. It's very interesting how you work with these models but unfortunately we don't have to go to go through all of this. So promise I'm going to give you the URL and the QR code is down there. So I hope I
didn't bore you to death. If you want to get in contact for any of this or you want to see it out there, I can I will happily show you. And thank you very much for staying to the one before the last presentation at this conference. I hope you're going to enjoy the closing keynote the party after that. Thank you very
