Jago de Vreede: Lessons Learned Using Machine Learning in Java

In this talk, the speaker shares his journey of creating a LEGO brick sorter using machine learning and Java. He discusses the challenges faced in selecting machine learning frameworks, comparing options like Python and Java, and ultimately choosing to implement a sorting machine in Java. The session covers various topics such as data collection, model creation, and training techniques, highlighting the differences in performance between machine learning in Java and Python. The speaker explains the importance of preprocessing images, using data augmentation, and selecting appropriate models to improve accuracy. Throughout the presentation, he emphasizes the significance of a well-constructed dataset and the nuances of fine-tuning machine learning models to achieve optimal results. The talk concludes with reflections on transferring models between Python and Java and a demonstration of using the trained model in practice.

[Music] ladies and Gentlemen please welcome our next speaker Yago D Reed presenting the topic Lessons Learned using machine learning Java good morning yeah good morning good morning everyone uh welcome to the second session of today my name Isa I work for a consultancy company called open value in the Netherlands but this talk has nothing to to do with what I do in my day-to-day job this talk

has everything to do with my son having way too much Lego he uh nine by now uh but this talk of this idea started around three years ago and he has a really big box like this like that by that uh with all Lego sets uh disassembled or mostly disassembled in that box but he really likes to build those sets so a repeating task for me became

figuring out Lego bricks in that big bin uh and trying to sort them so I wanted to build a LEGO brick sorter machine uh and the first part for that is well recognizing which bricks are there so I thought machine learning would be a fun way to do that one to learn machine learning and to to uh well eventually have a sorting machine but still building but

this part is is is quite done yet so 3 years ago I stood here at a a road Road a junction I could go two ways I could either go in the easy way the python way or go into the Java way I'm a Java developer myself who here develops with Java anyway as this is not the Java conference um more than half that's nice probably filled

out the non Java developers um but as a Java developer I like Java and well I thought it could be done well the road is on fire and there on the construction but there is a python on the right road so I of course took this path uh three years ago I didn't know anything about machine learning so I was bombarded with a lot of terminology like

loss function gradient descent hyperparameters ebx whatever so what I'm going to do in the next 40 minutes or so is take you through the Journey that I took uh in the last a few years and uh tried to get you up to speed with machine learning and and how to use them so if you want to do machine learning in in Python it's quite easy there are

a lot of Frameworks in Python um there's kiras spy torch tensor flows those those are the the most popular ones but if you're going to look at Java there aren't that many well if you filter out all the the Frameworks that haven't been released in the last year uh and are free there are some paid Java uh machine learning Frameworks but well there are not many of

them so if you Googled three years ago for machine learning in Java you came up with uh deep learning for J that was the top result and that's actually what I started with but then I became looking a bit further in into what kind of Frameworks there are more uh and I came across Alki but Alki is a more mathematical uh validation framework so you can Implement

your model and you can validate that yourself um I'm not really a model builder yet and I don't want to build models I just want to use it I see it as a tool not a mathematical exercise so this was was not for me but then I came across the uh deep yava Library It's actually an abstraction layer on top of uh more popular Frameworks like tsor

flow and pytorch so let's compare those because I was started out with deep learing forj uh and maybe this new framework is a little bit better so I did some benchmarks and I run these benchmarks on a AWS machine with a nice GP unit um and as you can see there is a rest in peace sign for the um deep uh uh for for J Library uh

because that was killed by the lenux subsystem so that that didn't even came to result but the ggl library was really a lot faster than the other one so it was time to change uh Frameworks for me but what about Java versus python then because if you want to do machine learning you should do python then right that's what everybody say well I did some benchmarks there

as well and it turns out that Java is a little bit faster than python not by much but a little bit faster uh as you can see the U the py torch in for python um that's that's this one um is a full second slower than the Java one so that's not that much um but well well that's that's the training part but what about using them

then again it's not by much but Java is a little bit faster here as well so what gives why is python not that much faster or why why they so comparable well it's actually because you're not using python for machine learning no you're using python as an API towards a C++ engine so learning you're using C++ and the same engine can be used in Java as well

so we just have a different API and this API is an abstraction for other apis as well so uh the same API can be used for pytorch tensor flow mxet um and some others so that's that's also nice but well we're essentially using the same engine so that's why they are comparable so let's start out with a simple exercise then probably already seen this right this is

how to confuse machine learning cookies versus dogs well you could of course start with jet gbt you could ask jet gbt and we'll get an a reasonable answer but if you switch around two of those images in that thing you'll get the same answer because jet GPT is hard V to give this answer to this image so it's not actually doing image recognition here so let's do

some human learning first because we want to know how humans learn and we'll see if we can map that on machine learning then I'm going to show a picture of a dog or a muffin in the center of the screen in a moment and raise a fist if you think it's a muffin and raise an open hand if you think it's a dark it's quite easy what

was that all nice hands it was just little doggy so good for you next one everyone got a muffin that's nice so that was this muffin next one little bit faster are we still with us see some doubting hands but all are dogs good it was this dog what was that is it a mop with a fist or is you thing still think it's a dog with

a hand uh I 5050 split that's usually the reaction I get uh it was this little doggy the problem that were what you were having was that you were trained on the dog versus muffin and now suddenly a completely different dog came up and a completely different I don't know what uh but I saw some people were doubting is this or that and that's exact same thing

that a machine learning model will also do it was very confident in the first few pictures that it was a dog or a a muffin but with this picture you were doubting and that was the probability that you were right so you probably guessed and this is also what machine learning will do if it doesn't recognize the image it will just guess and say with a certain

probability that it is that thing probably going to be a low probability if it was strange for this dark or muffin so let's do this for machine learning then well going to take this meme as an example so what we first need to do is split out those uh dogs and muffins into two categories you have to fix that as well because if there is only one

image that is wrong in your validation set then that's going to be an issue so this is going to be our validation set this is the images that we're going to validate our against so we're not going to show these images to the model uh when it's learning just use it as a validation so we need a lot of images to train this model actually we're going

to use around well maybe a thousand images per category to trade this model so a good way to get images from the Internet is to use Google and use the uh image search well there's API for that in Python it's only a few lines so the devil on my shoulder used python here because this is a lot more code in Java uh but I downloaded 2,000 of

the dark images and 2,000 muffin images great I thought then I was done this is the data set and maybe you can spot some problems here there are some duplicates so that's a bit of annoying because we don't want the machine learning model to be really fixed on a specific image so we want as many different images as possible so we need to clear clear that out

a way to do that is using a image hash I think everybody knows what a file hash is you can use md5 or sha for a file hash but the image hash is a thing like a f has but it doesn't really care about resolution or sizes or compression or whatever so you can actually compare different images against each other and if they are quite similar you

can actually calculate how much similar those images are so it's really good to remove all those duplicates so again we can use Python for this um I won't bore you with the code here because that's quite quite long um but I removed everything that is similar than 1% so this was my original data set or at least part of it because there were over a th images

here I'm going to run that only the this was left so I got 86 images per category left so that's not really a good start so apparently there's a lot of copy pasting going around on the internet um little bit of resizing and that kind of stuff and using it on different sites so Google isn't really that good for data data sets but luckily there's a different

side I think you're all familiar with flicker um you can download images there as well and they have a nice assortment of muffins and dogs is here as well so we can use that again with a python API and uh when we run this we have a nice data set uh I downloaded around 250 images from flicker because they were quite big and well I don't want

to strain that API so we got now around 300 images per category maybe a little bit more bit less but we want to have around a thousand images so how can we do that well quite easily actually because the um you can use data argumentation so you can take in uh one images so the this image is the original image and then you can rotate the image

uh maybe flip it around and those new six images are completely different for the machine learning model to to use so how can you use that well again with python here uh um but there are some Java implementation as well but you can use the uh image data generator from kiras and you can say Okay rotate by this amount and do a random flip and that kind

of stuff so that's nice um and with it I got around, 1500 images per category so that's nice it's good way to start so now let's start building this uh in Java well first we're going to start new project new pom um and we're going to import a a bill of materials from DGL a bill of materials is a pom file which has all the dependencies already

in there and you can use that uh well you don't have to specify specific versions later on so we're going to use a um a data set a model zoo and a model zoo of mxnet mxnet is a engine that we're going to use just like tensorflow uh this was the engine that was implemented the first but nowadays byor is also a good uh good option to

train this what we have then we're going to create a class learning and then we're going to load our validation set well we're going to load the uh uh the data set with a repository path and we need to do some transformation because our machine learning model needs to take in uh inputs as a specific size so we're going to resize our images to uh 224x 224

that will be our input size for our model and we're going to say a add transform to tensor because images are in red green and blue and red green and blue are values between 1 and 255 uh machine learning models don't like that they really like floating Point numbers between zero and one and the Tom tensor actually converts that uh in 255 to a0 to one so

that's that's quite nice and then we're going to say we want to say uh we want a sampling size a sampling size of eight or uh that's a a batch size to it so let's zoom into that so batches are the number of items that are processed within a iteration of a training so a complete training if you go uh through all your data that's called an

Epoch and that Epoch is cut into iterations and within that iteration you will send in a number of samples and that's your bat size and the lower your bat size is the more uh diverse your model can become so there are some studies around here so lowering your bed size will improve your model most likely but it will also increase your time so if you look at

the times if you can see uh with uh only a bed size of one so feeding the model one images at the time it's quite slow um with two images it's well it's doing twice as fast and it's ramping down nicely so for me with this small set of two things a bed size of eight really works well for me but if you have a uh a

large thing that that you want to train for example if you want to train on a thousand Lego bricks you probably go for a 64 B size or maybe higher so with that we have a data set loader and if we run this we get a nice progress bar great we doing something but now we need to train a model and which model because because there are

a lot of pre-trained models already out there uh and predefined models these models are tested against a image set of a thousand images um and you can see the results here on how accurate they are and how fast they are so I took a model that was reasonably fast and reasonably accurate because I want to run this on a Raspberry Pi or a other low power device

later on so I need to have something that's quite fast and also something that is also available in uh in our framework uh and I took for this the reset 50 model it's quite famous for image recognition it's quite good so let's use that so we need to get that model so we need to create a instance of a uh a model and we're going to say

we going to put a block onto that model and that block is a reset model resnet version 4 and here we're going to say again that 224 by 224 and our shape has a three there and that three is for the red green and blue great uh and the number of layers are 50 because it's a reset 50 model there is also a reset 125 model of

152 model U so you can change that around and we need to set the output size to two dogs and muffins then we're going to use that model uh so we're going to use a tri block for this and we're going to add a uh a trainer because we wanted to train that model so we are going to uh set a default training configuration with a loss

function uh we'll get to that in a bit and we need to evaluate on how good that model is and we're going to evaluate on the accuracy of this model so that's about it for the for the trainer so now we want to add some metrics because we want to see how good this model is while we're training it so we can actually track the progress during

this uh this training run and we need to set of initialize this train again again on the same shape so red green and blue 224 uh uh and we call the easy train. fit and we'll say we're going to train for one eoch so if we run this then with the magic of of video editing we skipped ahead a lot of time and we see two numbers

here a accuracy and the loss function while training we have an accuracy of uh 7 73% and a loss of 7 uh 9% so what is that well the accuracy is just the number of images that it guessed correctly so it just good or bad right or wrong so if you have an accuracy of one that means you have that the machine learning model has guessed all

the images correctly but the loss function is actually the distance to uh to per to Perfection that includes the certainty that the model has so for loss function it you want to have a zero U for the best result because zero means that there was no deviation between the expected result and the result that it predicted so loss functions you need to get as low as possible

to zero uh preferably but you're probably never going to be at zero probably at 0.1 or02 for example uh but your accuracy wants to have the most F so 100% so if we run this a little bit further we have now a fully trained model and a model had an 81% accuracy with a loss function of .5 great but our validation has a accuracy of 56% so

it's 50/50 not that great but if we run this again we have results so what gives well there's actually a randomizer in machine learning so if you don't want to have randomized results which which you probably don't you need to set a random seat and well I set it to 42 because why not and that means that every time you run this model you get the same

results so that's quite nice if you want to change a little parameter and see if that parameter actually improved your model because if you don't do this you change a parameter and well you never know if that parameter did the Improvement or just the different random seat so we trained the model now but we want to use the model later on so we need to save that

to dis well it's actually quite easy you just call the uh save function but the model itself does not contain the labels so the dark or muffin those are labels and we need to store those labels in a separate F so they're not integrated into the model itself so we just write that into a file and on every line you get a a nice output easy so

if we run this we get our model saved and our net text file uh which contains our labels so that's nice we're done or at least according to the internet because this is where most tutorials end but we don't want to just save the model we also want to use that model so here comes a lot of copy pasting because we want to do the same thing

as learning but this time as doing again we need to create the model why you might ask we're just loading the model right but actually we're not loading the entire model we're just loading the parameters of that model so the model as it is defined here is the uh how does the model look like and the parameters f is just well the The Filling of that uh

of the scaffolding well we're just going to call model. load and well we have our model um and then we need to do some U translation here as well with the training we did translation at the loading of the data set now we do need to do the same things again with the uh classification and if you want to use this we can just call the model.

new predictor because we want to create a prediction load in an image and just call uh the get prediction and we're going to Output the prediction of this uh this muffin here and if we run this you will guess see that it um well guess that it was a muffin with a probability 74% it's nice reasonable and got the correct so if we run this over our

entire data set we can create something called a confusion Matrix on the left side you can see what is actually supposed to be uh classifying and it to did what it did so it misclassified three muffins as a dog and it misclassified one dog as a muffin so let's make things better well if we create more ation or more uh EPO um there you can actually see

that when we train a little bit further you can see that the uh actual results uh well improves and then it later on dies off a bit so actually it's better to stop while you're ahead so uh at some point the model is most optimized and everything you do to that model becomes SP so it's nice to stop when you're at that point so in Python there

is something called a early stopping configuration um but unfortunately that was not yet available when I was uh playing around with this there was a open issue at the uh DGL Library uh luckily it contained some code to get this working so U I had a big mouth at devox last year uh and I said I should create a poot request and I eventually did um and

this actually got merged uh in December last year so if you want to use this now you can so that's that's quite nice uh and how does this look like so within the training configuration uh we previously had just a evaluator but now we need to add a training listener that's going to save the model every on every Epoch because um when we stop that model is

by definition worse than the model uh that was before that so we need to save every iteration or every Epoch of that model so we can actually load up the best uh model that we trained so that's the first part and then we can say a add training ler for the early stopping configuration we can say that has a patience of at least two so there must

be at least two Epoch that are worse than the best one uh and we can say a maximum time that it needs to train or at least a minimum of of three EO for example and when we run this then we get a nice list of uh adult models and you can actually pick the model that gave you the results that that's quite nice so why is

the model getting worse then so it's a little bit like this this this game there's something called learning rate in machine learning and it's like this game that where you hit have to hit the green bar at the exact right moment right everybody got that in in in a mobile game for example it's just like machine learning only thing is machine learning only sees the thing when

it's done it only sees that green well this is the most optimal part but the machine learning model does not know how green that green is it just knows it's green but could it be better or worse and actually when you want to train that model it's not little iterations you want to get to that green part as fast as possible so what you can actually going

to do is with machine learning skip over certain parts and if you take the same steps every time you're going to skip over the best part so what you you want to actually do is reduce your learning rate so first take big steps towards the green and then slow down when you do more EO this is a learning rate for the U exponential Optimizer for example but

there are other optimizers uh out there as well um lot of mathematical things go in there but well the atom Optimizer is the most used uh out there and well it actually gave me the best results as well so with that I reran the application and at Epoch 3 I believe I came up with this uh model which only got one Incorrect and that's the best I

came out with this but it was nice good enough good enough me so there's something else that you can do uh and that's called transfer learning well you can take for for example that ret 50 model and you can also use that as a pre-trained model so a model has input at the top and then it goes through all those layers and at the end it will

decide if it is a muffin or dog but what you can actually do is take a pre-trained model on different things and lock the first uh number of layers and in those first layers there are a lot of basic operations like Edge detection and that kind of stuff you don't know what's in there because it's machine learning but you can estimate that the first layers is just

that and with looking that and leaving out the last two layers you can actually train that those last layers very fast and well you should add up with same results but I couldn't get it any further than 50% so it was just a guesstimation but it is something you can use uh if you want and if you want to use that well you can actually load up

a pre-trained pre-train model if you want um with something like the criteria API of hibernate I don't know if you already seen that but um you can just say okay I have a builder I want to do some image classification I want to have a image as a input and a classification as output and it needs to be a resonet model a group ID and and some

filters like the number of layers and the version and we you have loaded up this model you can actually use it uh as before but you can also change that model you can uh well take around that blocks and you have a full fine grain control over each block and each layer that you want to do so I won't bore you with this but this yeah you

can use this if you want but there is also something for Python and Java because a lot of data experts or or machine learning experts still use Python to create their models but we're using the same engine so what we can actually do is use those together so someone in Python can create a model and Export that as a open model or a saved model it's just

a few lines and when you save this model you can actually load this model up in Java so that's also quite nice it's the almost the same criteria API that we've seen before uh only this point we pointed to a path on our local file system and of course we need to add the uh tensorflow engine as a dependency because well we without not having that before

and we just call the model and we need to do a uh transformation towards this model so we need to do the same transformation that we did in Python in Java as well uh luckily there are not many Transformations done here so we just need to resize to 150 by 150 and we just going call the uh predictor API again and when we run this well if

you not um have set up tensor flow before it will actually set up tensor flow before for you or you can point it up towards a installation of tensor flow if you want to use this version uh well with some red lines at the end it came out with a daisy and some other uh flowers so that's nice it actually worked but what about the Lego then

well I want to use this for Lego not for dogs and muffins so what I did I built a conveyor belt with a camera on top h and a Lego brick will pass um before the camera and there are three uh buckets at the end of the uh conveyor belt One Straight Ahead for the things that it did not detect or were the wrong bricks and you

can actually sort out two different Lego sets uh if you want so it will point to one one box or the other box if it's uh but if you want don't have there so what does this look like a brick will pass the camera and will end up in a bucket luckily so I need some images for uh Lego bricks there aren't any Lego brick data sets

out there unfortunately so I wanted to generate the images for that as you can see looks quite nice right can work with that so there's a rebrickable database out there and that rebrickable database has all the Lego sets in there and all the bricks uh which are used in that specific sets so that's what I use for the set extraction and there are luckily a lot of

AutoCat files out there for every Lego brick or every piece of Lego and you can render that with the application called LD view okay um but that can export to a format for pay and pay is a uh rate tracing application um that's run on the command line so that's quite nice and it's also text based so that's quite nice and then we can generate our images

so what does it look like well first I generated a a lot of images in all kind of different directions and then I need to find out where the edge was uh put Square on that and well uh crop that out but here I came across question to zoom in or not to zoom because I could corop out that specific brick so the machine learning model was

fed the most brick as possible but I could also use the distance from the brick and try to simulate the actual camera and feed it in the entire image so the machine learning model has more information about how big the brick is who thinks zooming in would be better who thinks the other one that's was my thought as well unfortunately it's not the case because if you

zoom in you will give the model way more information the model is trained to predict things and it's well it doesn't matter how far away from the camera that thing is it's just a thing and th giving it more information on how big the thing is by the actual size of it it will actually give worse results because the image size is only 224x 224 that's not

much pixels to go on and if you have a small Lego brick or a small Lego piece it's quite small to begin with and when you convert that to 202 224 pixels would end up with a image which contains the exual brick of two or three pixels doesn't really know what to do with it so zooming in gave me accuracy of 80% while the real size gave

me 65% so zooming in it will be well we're going to create a new model for this uh and actually going to the the learning class that we used before and just change the output size from two to 100 for example and run it so I created a lot of uh Lego bricks um but uh I came to the conclusion that there wasn't any any progress anymore

so what I did here was uh for 10 pieces I created a a graph on how far this uh on which Epoch the best results would came in because training for 100 bricks cost me around 9 hours for a so 10 bricks is lot more easy to work with so scale down because it will scale up quite nicely so if you got the results for 10 bricks

you probably get the same results for 100 bricks only it's going to take 10 times time times longer so as you can see quite nice results but I took some pictures of my phone with uh with some real Lego bricks and I plotted that into the the graph it were only three bricks but stayed around 33% so what what gives well again a confusion Matrix here will

help and what you can actually see that it was just predicting one specific brick so we got 33% right it would switch between the 301 and uh and 302 but it was still just predict one break so that that wasn't really good but that was on the epoch 17 moving a bit further on Epoch 47 it would actually started to yeah will recognize different bricks or at

least try to mix and match different things but it would never ever predict the uh 3004 brick so a little bit of panic uh came to mind so looking at the generated images and the real images you can POS probably see which images are generated and which one are the real images right the top ones are the generated ones and especially the smaller bricks have jacked AES

for example because the resolution wasn't that great to start with so I need to improve this so I created a new validation set and that validation set was really taken from that specific camera on that specific height and well I just took some uh some video and extracted those frames and I baselined my previous model so I got uh five uh exual bricks out uh for this

so the well the percentage came up to 20% so that's no no uh uh that was that was what expected so I need to increase the resolution because those jacked AES were just artifacts of being zoomed in too much so that increased my 25% it's a little bit better but not great so giving it given the model color information I thought maybe if I reduce that color

I would eliminate three channels so three times as less as input so maybe the model would train three times as fast unfortunately no because in those first layers the color information was already stripped but it it reduces noise for the model so if we want to use this gray scale image we need to actually change two things of course we need to change the input shape from

three to one and we need to set the uh the flag to load up the images to grayscale otherwise it will just load grayscale images into a RGB uh image and then it would crash so benchmarking color versus gray it's a little bit faster to do Grays scale but not by much because well in is already lost so giving it color information doesn't really help but it

did improve the accuracy of that that model so so 28% getting better not there yet so somebody came to me and said well maybe you need to lower your contrast because contrast will give you uh lowering your contrast will actually remove those Shadows from those real images so removing those uh well removing or lessening the contrast of those real images got me to 31% it's better then

again removing that contrast from those from the training images to get them at the same level 1% more not by much but well take anything by now and then we could also increase our training data set by using something called horizontal horizontal and vertical flipping like the data argumentation that we did before with the dunk we can actually do this every time and that's because when we

load the data we first had the add transform and and uh for the resize and two tenser but actually this is done every time the loaded so this actually a pipeline so the pipeline is executed every time so if your image is already this size you don't need to resize it anymore so you can save up up time there but we can also do a random flip

and horizontal and a random flip top to bottom so every time that image is loaded it's randomly flipped or not so this gave me free data orentation so that's nice um little bit better but not that much uh then I would wanted to incorporate real images to my data set because at until this point the data set was all generated images so I reduced the uh the

total number to to 750 and well of course the it decreased a little bit but then I added the real images and then it shut up to 70 of 97% so that's quite nice so actually got a mod that did some things so running this through the uh well real real uh uh example we see a lot of good uh classifications and if you filter out all

the non-correct classifications you can see that the probability is all quite low so if we filter out all the low probabilities and take a average between yeah well we have multiple frames to to work with and we take an average of three or six frames for example we have a good result so we got something working so the key takeways creating the data set is really the

hardest part of everything machine learning is not that hard Java can be used for machine learning it's a little bit faster as well so that's quite nice python still has the bigger e ecosystem so well for probably for the uh data experts you probably want to use that uh but models in Python can be used Java so that's about it if want to play around with the

code it's on get up and I know there's a typo in The Benchmark but hey it's it's a rapo name uh and you can find these slides on the uh on the link or at the QR code so are there any questions I think so how much time have you spent on this well uh it was three years ago that I was started uh but it was

mostly on and off uh well mostly off actually because I have four kids uh and I did this in the spare time that I had so if I would squeeze up all that time in this three years I would probably end up with maybe a month or so in the evening or in the weekends it's not that much actually but well spread out across multiple times lot

of contact switches uh when I actually found out that the new framework was a lot faster than the old framework well I had a mental dip so not touched the code for a month or six or so so it's uh well not that much time but it's mostly creating that uh data set all right other questions why Maven why Gradle why an it's just the thing that

I use in my day-to-day job because well I work at a uh corporate for corporate corporation uh so just what I was used to and well not complex anything more than well just use what I was used to apart from seeing slightly faster what other Pros do you see of having Java versus uh python in machine learning well what I've actually seen in the uh real life

that there was some models strained uh with B and what they did was create a rest end point on top of that python application that would take in an image and spit out the classifications for uh with a rest end point uh and that was actually being called by a Java application so if you would use that model directly in Java you wouldn't have the overhead of

the uh well the rest the stuff so that will be a a quite good use case for it and that's it we still have time for several questions from the audience uh raise your hand if you want to ask something and if not I'm here all day and uh thank you [Applause]