About this talk
This talk explores the integration of a custom machine learning model within banking applications to categorize financial transactions. The speaker, Bakar Sabat from IBM Ambro Bank, delves into the intricacies of neural networks and the essential parameters needed for training them. He explains how machine learning models can be created and fine-tuned, particularly focusing on challenges such as overfitting and underfitting. Additionally, the session highlights various neural network architectures, including convolutional and recurrent neural networks, and how they can be applied to different data types. The speaker also discusses the significance of loss functions, optimization techniques like gradient descent and Adam optimizer, and demonstrates the model implementation using TensorFlow and Kotlin.
Full transcript
Good morning everyone. Can you hear me? Is everything fine? Nice. So we are all familiar with AI right? I mean you can't miss it. It is everywhere. Everyone is talking about AI. we are also using it is quite useful and some of us have already integrated AI into their applications. However, I believe that the majority of us we still consider AI as a blackbox. We have some
prompts we get some output. We don't care that much about how it works and how does it look like. Well, today we are going to open this black box and we are going to look inside it. We try to create our own machine learning model, train it, uh see how can overcome certain challenges and deploy it in our Android Auto or uh iOS application and use it.
My name is Bakar Sabat. I am working at IBM Ambro bank. And since I'm working at the bank, we are going to create a machine learning model to categorize our banking transaction. We have lots of banking transactions and our engine which is powered by AI another buzzword. It will categorize them to different categories. So we have a good overview and insight of our expenses from our transactions.
We need two things. One, the description of transaction or the title and the amount. And based on these two parameters we will return a list of category with a probability score. So here this uh jumbo rotordam for 50 will be categorized for 95% as groceries. Let's create our machine learning model a custom one. Here things will get a bit disappointing. Why? Because it is not that easy.
You need to have some background knowledge. You need to know some parameters that you can change it while you're creating your machine learning model, while you are um training it and while you're testing it. So in the upcoming slides, I will try to make it as simple as possible and we don't have a lot of time to go into the details to give you an overview and
also uh introduce certain terminologies that we are going to use. Now let's open the black box and with an AI or machine learning model the core of everything is of course u neural network whether it is a generative AI or AI the core of it is neural network. This is a very simple curl uh neural network. It has one input and one output and it has one
uh hidden layer which the whole magic happens. This hidden layer has two neurons and if you look inside the neuron if you will see that there is an activation function. It is nothing else than just a very simple mathematical function. Uh ReLU is the one of them. It's the most used one because it's very cheap in term of GPU and CPU. uh it's a very simple one
and you also have another ones like this fancy relu I would say or a handy one which actually compresses the values large values between zero and one and this way you will get some kind of percentages and a handy one another handy one is a softmax and this is we are also going to use so this one is used for um classification and categorization um If our
the output of our neural network has some scores that doesn't say a lot if we apply this uh softmax it will convert them to probability list. Now we know hey the probability is 90% that the item on uh second or the index one has been chosen. So now we know what's inside neuron. What else do we have? We have connections between each neuron and these connections they
have weights and biases. With weight you have to multiply the incoming uh number and with biases you have to add it. By the way neural network is all mathematics. So if someone tells you hey I waste my whole life studying mathematics and never use it should maybe they should think twice. So let's take an example here. the input uh node has seven it will be multiplied by
two which is the weight and minus one which will minus one will be added so the outcome will be then 13 and when we say hey I'm going to train my machine learning model it means that we have to adjust these values these weights and biases so that output of our neuron will be close to what we have been expecting how are we going to uh train
For that we need a algorithm with one of the algorithms back propagation. So we start with initializing over weights with random number and biases with zero and then during the forward pass we have our trainings data. We will pass it through our network and something will happen and the actual value will come and then we will see what was the actual value and what was the expected
value. So there must be some kind of error. We calculate the error using the loss function and using this loss function we can go back and update our uh using back propagation update our weights and biases. A certain point we think okay the output of our neural network is good enough. So we stop training and then uh we continue we we're going to use it. Otherwise you
will keep doing this iteration as long as you're uh you're not satisfied. Let's create a very simple neural network to show this process. Uh so here we are going to create a neural network a very simple one to predict the uh bacterial growth curve based on our observation our data set we have three data points. So during the forward pass yeah actually this this must be some
kind of an curve going between these numbers uh these dots. So during our forward pass we will pass this data to our neural network and the output will be something well a bit far from what we are expecting. Then we use a loss function. There are many loss functions. The one that we are using a lot is uh sum of a square uh errors. We calculate it
and then we adjust the values and every time when we adjust we see that the loss function uh output will be decreased and decreased and decreased. So for smaller neural networks you can do this steps you can adjust it manually but if you're dealing with a larger neural uh neural networks this is not possible. We have to come up with a better algorithm. What we are going
to do now is to plot the output of the loss function for each step. So obviously the first one the loss function will be very high right because it's the first step. But if we adjust the values, it goes down and down. And at certain point, you passes the expected value, it goes up. So there must be some kind of graph or curve between these lines. And
the goal is to find weights and biases where this loss function is minimum as far as possible. there is another terminology that we have to remember and that is a gradient descent. With gradient descent is an algorithm to find that minimum. How does it work? Imagine you are standing on a top of a mountain. It is very foggy and your goal is to get to the valley.
You look around and you try to find the steepest downhill slope and you take one direction to that one step to that direction. So every time you are calculating the steepness of the slope, what do we use? You all remember derivative function, right? Or tangent line. So every time you are calculating the derivative function and your goal is to get to the value. So we know you
don't have to remember how it works but gradient descent is used. So you have to know that uh we can use this parameter later. Another thing that we have to remember is the learning rate. At the beginning our steps are very big. Why? Because we are far from the valley. But as soon as we get close to the minimum we also have to adjust our steps. And
that one that we can do is with learning rate. We use learning rate to adjust the next step. And this again another thing that we can fine-tune during our uh training of the model. There are problems with gradient descent subtle point or local minimum. Uh gradient descent when it's in the local minimum it says well I've done my job. Good luck. So maybe there is a global
minimum instead of local one. Then we can use ADM optimizer which pushes us from the local minimum and goes to the global minimum. Be mindful that Adam optimizer works nice but it is very expensive in term of GPU or CPU. So you can only use it once you you think grid indent is not enough. Now we have a little bit knowledge of terminology. Let's go to what
type of neural network can we use. There are many types. uh the ones that we are usually using is dense neural network. This is used for classification etc etc. But there are other things that we go through. The one that we are using for pattern recognition and um large images or voices videos as a convolutional neural network and it works as follow. Again we are not going
deep into the detail. Imagine we are willing to recognize or categorize this uh image. What one way what we can do is to create a matrix of bits and byes and feed that one to our dense neural network and it will be categorized but this is not really doable right for smaller images doable but for a larger images it's not doable and it's also very sensitive. So
instead of looking to the entire image, what we can do is try to find certain patterns in the image like in a set of car you try to find a wheel, the window, the door and that is used you so you create a filter or a pattern and then you create a convolution or a or a patch and then you slide pixel by pixel over the entire
image and this way you can recognize patterns. On the lower level, you will be able to recognize vertical lines, horizontal lines. When you add layers on top of that, you will see patterns. And if you go on a higher level, you will be able to see objects such as person, car, etc., etc. So convolutional neural network if you're dealing with images you we have to use for
sequential input or actually for a large text where you have to uh predict the next output based on the history just like chat GPT we are using RNN chat GPT doesn't use RNN they are using transformers with self intention so we are not going to that direction but RNN is used if you need to have some kind of um history or you want to predict something new
one next token based on the previous tokens. Okay, enough about those uh neural networks. Now we have a problem here. Neural networks only works with text with uh sorry with numbers this uh mathematics and if our input is text one of them how can we represent text? There are several ways. One of them and simplest one is um one hot encoding with the local vocabulary. So basically
what we here do is from your training data set you get all the text you tokenize it create words and each token will get an index and when you encode in the incoming text you use that index to replace the word. One problem is that there is no relationship between the words. If I say neural network is cool or neural network is awesome, they have different they
have they don't even have meaning. If you want a relationship between the words then you can use word embedding with word embedding is an matrix of multi-dimensional like hundred of dimensions where words with the same simil with the same meaning or similar to each other they are on the same dimension approximately and you can calculate the similarity rate score and you can replace words this all actually
chat GPT also works and there are by the way there are many open source libraries that you can use for different languages from Google or OpenAI. Okay, now let's see what kind of challenge do we see. You have created your neural network, you train it and the output is really not good. So those dots or crosses they are the our expected values and that line is the
output of our neural network. Obviously this is not neural good one. It is underfitted. You have to increase the complexity of your neural network and see how it works. But the biggest problem is overfitting. This is where your neural your function goes through all data points. This is like students who are remembering everything by heart and when they go to examination and the questions are slightly different
they got they struggle. This is exactly with other overfitting and that is the biggest problem in creating neural networks. How can we? Oh yeah. Obviously what we want is a balanced one which follows the training of our data. It doesn't go through all the data. So this is a balanced one that you're looking for. If you're dealing with overfitting, there are a few things that you can
do. One of them is L2 and L1 regulation. If you remember from the gradient descent from the mountain to the valley, there is a loss function. So L2 and L1 will punish those loss function. So it never gets to the valley actually. So this way you don't have a perfect neural network. Drop out function is another one. Obviously your uh neural network you have a lot of
neurons. So for each epoch for each round of training certain u uh neurons will be inactive. So it will get some probability. The most important one is data augmentation. If you are training your model with the images of cats or dogs, instead of using only the images, you can also add color filter, rotation, flipping. So your neural network is familiar with those variation as well. Enough about
thing. Let's go to the code. I'm using TensorFlow. You can use PyTorch. Uh I'm using TensorFlow because I love it and because you can convert TensorFlow to TensorFlow light and you can embed it with Android or iOS application. There will be another workshop maybe they're using the same thing as so you you already know we are having 16 categories just made up it's a proof of concept
we are not using this in IBM bro this is my own side project 16 for we are creating generating data again it's data is generated uh also the amount r per category is different for grocery is lower and for travel is very high because obviously what we want is that amount is as important maybe not on the same level as important as the description. So if you
say Amazon you bought something Amazon for €30 it will be categorized as entertainment but if you buy something from Amazon for bigger amount let's say 500 it should be categorized with the highest probability of course as electronics. Usually I do this part interactively but now we don't have time. It's only 30 minutes and this is the output of our interaction. I would say for the input layer
we need two neurons. One to represent our text and one to amount. For a text I'm using one hot encoding. If you want to use word embedding fine for the numeric one for the amount I'm using m mix m uh minm max uh scale so you can normalize the amounts as well for the output one well the output will be 16 uh neurons each of them representing
one um category remember that the output will be some random scores that doesn't say anything so we have to apply the soft max function. So then you will get the probability list for the neuro uh hidden layer. I started with one hidden layer end up with two. I don't know why because I was trying it was not I was not satisfied. So I made a bit complex
uh activation function for all inter hidden layer is of course relu the cheapest one. Uh again I was pl I was trying to overcome the overfitting. So there's a dropout function, a dropout value and L2 regulation and of course uh yeah for the optimizer maybe gradient descent was enough but I I end up with Adam optimizer. Let's take a look at the code. So with again I'm
using TensorFlow and I'm using uh Cotlin. Why? Because I'm an Android developer so I love Cotlin. However, if you are familiar with Python, it's better to use Python because in Python you have many better libraries that you don't have to deal with low-level things that you will see here. So, you initialize your text, your input, you initialize your amount input and for the hidden layer, well, you
define things that we have seen in the previous slide. And here you see that uh the weight has been initialized with random biases whether zero and we do the mathematic like weight times input plus biases. Uh if we go down so you also apply your regulation and you also apply your dropout nothing special for the output layer. Yeah we get some random number so you apply softmax
and you will get a Okay, during the training you have to decide number of rounds that you want to train the epoch. You get your data, you initialize it, you give it to the runner and it will be trained. You will get some useful information that you can plot here. What we see uh is two things. One the training accuracy and the other one is the validation
accuracy. It works as follow. Your training data set will be divided into smaller part and a big part. The big part will be will be used to train your model and the smaller part will be used to validate it and for each epoch they will shift these uh parts. What we here see it's not from my model but from another one. What we here see is till
epoch 10 everything goes fine you're going up but after that my validation accuracy goes down. This indicates that my model is being trained with the data that I'm providing. But if I provide unseen data, it is not performing well. So this is an sign of Okay. Once you have created your model, you trained it, you can convert it to TensorFlow light. You can use any back end
or you can uh upload to Firebase if you have Firebase. If you don't have Firebase, you can just embed it with your Android or iOS application. and your client site in Android in this case you load it you load your uh uh and for prediction remember that you have to encode with the same lo same vocabulary actually that you have created to train the model otherwise you
will get strange results and for the amount you have to standardize it or normalize it using minmax the same minmax as we used to train And then you can predict. Well, this is also very simple. The input you have to define it with a which with a tensorflow uh the output and then the output will be array of u probabilities. You can do some fun stuff with
that. Zip it together with your categories. So you have a great map of probability of uh categorizations and uh probability. Let's uh take a look at the code. Yeah, not that the code but thing. So, so this is I have created an app here. We have uh Amazon for €50. It is already you can see is it visible? Yeah, maybe I can make it a bit larger.
So, yeah, it has been uh uhoh. Yeah, it is categorized as uh uh entertainment or recreation. But if I have like two 300 I say predict now the probability is getting a bit higher that it is electronics and the entertainment is going down and if I go even more higher like 500 then is of course uh or 98% 99% it is uh uh electronics you can also
use how many minutes do I have in jumbo and you can add add something else here like the address blah blah it doesn't matter and for €50 it will be groceries for 100% because jumbo was only for groceries that was the demo and I don't think we have time for Q&A but if you want if you have questions I will be here at the aonom so we
can talk about it and this is my LinkedIn for recruiters I'm not looking only if you want if you have questions if you want to have contact with me. So thank you
