Hannes Preishuber: Step into Machine Learning .NET

This talk introduces the audience to ml.net, a machine learning framework developed by Microsoft that enables the integration of AI capabilities into existing .NET applications. The speaker discusses the importance of data science in preparing and transforming data for machine learning, emphasizing that the quality of output relies heavily on input data. Various concepts, including deep learning and large language models, are explored, alongside practical demonstrations of using ml.net for tasks like classification and object detection. The speaker shares insights on alternative machine learning models and tools compatible with .NET, including custom solutions on local devices rather than relying on cloud solutions. Throughout the session, the speaker illustrates real-world applications and the potential of using machine learning for tasks such as image classification and predictions, encouraging developers to leverage these technologies in their projects.

[Music] ladies and Gentlemen please welcome our next speaker H pryhuber presenting the topic step into machine learning.net do you hear me now you hear me so it's pronounced hanis yes please repeat hanis thank you very much it's not hnis okay so uh you C you can ask me questions about Austria I'm from Austria you can ask me questions about the old west bu how to fix it

I have four pieces of them down from Seven you can ask me things about PPD the company I found it um 90 30 years ago uh and we do training in the German speaking Market also Germany Switzerland you can ask me about ml.net machine learning net for Microsoft framework this is the to topic of this session so my goal is 40 minutes is very short my goal

is to give you an idea what you can do in your existing net code to use AI capabilities so we have to do some Basics what you're talking about stolen slide so it's not for me and um take care first thing if you're thinking about machine learning you have to think data data science uh to prepare data to get data to to transform data this is a

really a science that's the reason there are science in that and if you have data you get out in any case what you do which framework you use and so on um about machine learning is artificial intelligence this is from my guess what open AI is doing at the moment and and uh within machine learning is a deep learning which is something you can read books about

that's sick like that where you build your models so the ugly side of mching is what they thinking about or what is our future and they have really really bad examples here on the slides uh what you can do perhaps somebody know the pick video uh the the movie where it is what Minority Report who have not seen Report this is a question of age so uh

in minority Reports say needed mankind to predict crime uh we can predict crime and this is not a joke uh indeed with data from computers uh we can divide in good and bad citizens we can uh track people moving around in systems and fake people moving around in systems this is really reality now so if you go to China they have this technology working for you the

most murder modern thing is large language models the open me eye story and uh there are several issues uh which you have seen or go to through the Press which such that and the last point is that's our future Skynet will take over and I will show you how you can do that and participate in Skynet and be the slave of sky and survive in this system

um my part is I'm not a big fan of cloud I think this is my personal belief in I think the real excitement with AI is to have it on small devices and this small devices are able to do things without any user interaction so rasbery so my talk is about how you can use AI machine learning.net without targeting any API from oh uh Asia or open

API and don't get the bill at the end of the months from them and if you have watched what the market is doing at the moment you see that the Nvidia stock goes up and up and up C they sell graphic cards with Cuda uh command set um because the market seems to have a need for that kind of solutions the question is can I do such

things can I do things like open ey is doing and the answer is yes you can somebody knows these guys this is from a French company uh they developed mistal this is the whole team two years within two years they develop mistal and uh this have the feature jet feature set of uh openi 3.5 R about you can download it and use it so we are in

a field of technology which we can participate use or create I have the same for Microsoft because this talk is about Microsoft mlet this is the team from mlet a few years ago working hard on things we learn in a few minutes large language model the yeah the topic which is in every media and so on uh this is my last slide and this will be my

last demo to show can I use a large language model within my existing net application the answer is yes and uh Microsoft have done pi3 which the funny name tiny large language model that's the reason for the picture bide and uh I will show you how this not my last slide uh how you implement that so before a little bit uh how to do it in detail

so I have a um picture here on the side and some notes Here on the slide okay ml.net is a library for Microsoft they say it's open source it's vendor driven and it targets all relevant operating systems the usual story which Microsoft Is Telling today what you're doing with machine learning you solve problems you cannot and um ml.net is um a little bit complicated to describe so

the easiest way is if you know Entity framework think about the Entity framework AI Entity framework is targeting different kind of databases with different kind of schemers so you need a context you need model classes and so on and uh this is the nearly the same in it's called ml context not DB context and you create a context you create a pipeline with the pipeline is the

most complicated thing in this game because you have to ensure that every data that you put in your pipe or in your in your code is fitting in your model size of data uh labeling of data and so on this is not so easy is is a data science job and uh you configure the pipeline to transform your data in the format you need or your model

needs and then you uh trans uh train your uh pipeline so you get your you take your data make a training and create a model and after the training you evaluate with usually a a smaller data set and say okay let's tried is the expectation of the output of a model that what my um evaluation data set uh says and if this okay go back to the

graphic then you have the opportunity to save your model and if it's not okay you have to redesign your data retrain your your model and save a new model and with a save model you can go and and uh predict anything you want to predict good or bad citizenship I have one slide this is in D the last slide where there is code in it so you

can uh get a feeling about how it is built so we see a model context we see that we have to load some data we see uh the pipeline in step three where transformation is done from features features is a typical description of the data and labels which is the expected output of the data and then I can say Okay train your model in step four and

uh then you can uh in step five predict with some okay that's the point so hopefully everybody of you have seen Visual Studio I'm not sure I have not seen any Visual Studio in uh any session so I create a new project and uh this command line and the goal of this demo is uh here is right to show what Microsoft is oh my God what pass

I have and then call it delete me delete leave me so now yeah funny part I forgot to uh switch the language oficial studio is somebody here who can speak German oh thank you for you two guys I make this demo so uh Windows uh so what I have to do uh Microsoft have a new model builder and uh I find this here so you have to

install this part without you have not this part and with this model P I can do several things jumman sorry about that uh we have uh tabular data tabular data is um in normally yeah some some Excel sheet or something like that in different formats and then you have four different um scenarios which you can use for instance classification yeah we have this is the first one

classification and you can say with this uh sentence we can believe that this guy is angry that's a task uh we will do some machine um um I miss the English word uh Mach scene uh good good question and the last last one is you see the UI is looking a bit a little bit fancy is uh also text classification based on uh context the text not

on table text in the first case so we we are going to doing some image classification and uh I do the smallest and fastest thing I can do so this takes time this a problem and I have a data set I choose this folder and you see here uh triangles circles and on and I have labeled this data in folder so the idea is comes from what

the idea comes whiteboard looks fancy so if you go into whiteboard and draw a triangle so looks really ugly okay doesn't work that's always the best demo case ever why it doesn't work because I have no internet connection I didn't check the internet so my expectation is that this triangle would be a nice triangle it is not next try so this is a feature of whiteboard I

hate whiteboard because it uh needs a internet connection so it's not uh suitable for Airline plane flight or so so and my idea was okay let's try to rebuild that offline with ml.net and for that I have to uh create data I've done that you see the data here and I have written a BL application for creating the data to draw with my pen on my Windows

machine and uh I have puted it in three directories and now I can start the training uh there is the button here now what I'm doing now this is called autom ML and the tool you see at the moment is the model builder it's are the Microsoft names for that and you can do that with table data or whatever data you have to figure out what kind

of uh uh machine learning task do you have what Mass algorithm fits best in this thing this your part then you go here to this tool and say okay this is the best one put the data in train it and uh at the end you get a ml.net model takes not so long one minute so and I have uh talked about the quality of the model and

one is unbelievable believable but true not a good quality one is the best but if you have one then you have a problem in your data so I have a problem I don't know which problem it's not time to fix that and now I can do following can take a circle this wonderful Circle and um predict if this a circle so 99% looks quite good I take

a different image which is not from my training data because this is not fair so we have that set a full triangle text while and you see dry egg is three angle so it's Works quite Qui and um if you want to use that uh model which exist here ml.net you have uh just do the following thing nson uh uh that's uh Microsoft use the first code

Snips code snippet paste it to your project uh or click the second option so and now I have really so it takes uh two seconds uh I have uh two projects in my solution my visual studio solution and the second one is uh able to use this I can I can run it Al so I have to set the start project start Pro and so this the

picture it's a PNG and now I have a command line tool where I use this model and you can use it in so you see uh same it's uh the expectation is it's a triangle okay so we have seen it's not so complicated to do that special task but uh if I take a look at this you have a different um task you have you you must

figure out how big is your triangle and where it is placed not so good and um this is a little bit more work uh for that you can do the training in Visual Studio you can do that but I have not uh uh reached the results that it works it takes long time and at the end I I get the mess model and um there are always

1,000 possibility to do something in the a field there there are so much tools there and this is from Microsoft this is custom Vision AI it's wonderful it's wonderful cheap and fast it works excellent okay so I uploaded my images here to this Tool uh I do a training this green button and all done uh we can see here and Microsoft say uh this is a quite

good model Evol ation of the model and uh I can do a prediction and I've also loaded a image here this one so you see I have three figures in this image and this model have the ability to figure out where are the uh figures and uh make the dimensions or detected Dimensions where it is and you see also there is here a fi with 27% more

than figures here on mine this typically for models models are not never perfect so they are always uh have some site yeah results and it's on you to say what is your your your number where you say okay this is for me okay so um fi so you say 30% is at least the number you need and everything under 30% % but this is you you have

to figure out that in your uh data set environment and code so now a h where this here is that's a point I want to show you can export this model um and you see different kind of models this is the the model ecosystem I will show you in a minute the the hosting platform for models and Microsoft have created Onyx and say it's it's a open-

source format so you can exchange models and uh we have also tensorflow which is from Google which is quite famous and uh other so I can download the Onyx model and use it in my project I have done that with a uvp application with Windows ml which is not ml.net and this works it works really good this minutes okay um what I want to do I want

to download it as tensor flow model I have that somewhere T of flow model here it is now I downloaded this model I I'm sure it's this model or another as a type tens of flow and this is the model round about 60 mix and if I double click this model I can take a look at the error message click okay and uh look in this model

this is a tool uh which is U developed from a Microsoft guy in free time Neutron and you can take a look how is this model built yeah you say there's a a placeholder and this has a type of placeholder and if I go down somewhere is the model ready or the output is ready and this is called Model outputs this is important if you want to

code this model and have no idea how the model is built and this is the point I do here so I show you a little bit the code so first step model context Second Step load the data yeah some images here load the data Next Step create the pipeline uh this is important part resize the images extract the pixels and if you remember 20 seconds ago placeholder

was the input of the model that is the reason why it's called placeholder placeholder is a silly name for that and then it's a load uh tensorflow model because I have a tensorflow model and no Onyx model I will show you Onyx Model A little bit later and here I have a input output colum model outputs you have seen that model and then I can uh train

the model uh create prediction engine predict blah blah blah uh the labels are separated in a text file I can show the labels so this depends on the model if the oops if the labels are in the model or not the reason is um AI always uses numbers internal and never strings so it have to transform it internal also language models to uh tokens and this is

the reason I have this label table and here I have a object detection blah blah blah and uh can run this codes is also command line code and say and check hello world load the model hit the breakpoint click and say CH okay Circle it's called here CH you see it's a CH and it's not from my training data it's a different CH okay we want uh

use cases you can use in your daily business next one oh I should not close Vis Studio next one in Germany and Austria people like to have bathrooms without toilets is it here also the same so a flat costs more money if it'd have a separate toilet so and and the point is if you selling 50 minutes if you're selling uh flats and uh and say okay

toilet or not toilet this can be wrong so somebody must look at the picture and must say do I have a toilet or no toilet and this can I I do and this is the clle demo clle is uh toilet in Austria and what I have done and this is really the case oh we are the picture pictures H where are the pictures I forgotten the pictures

I've I've downloaded pictures or images from the internet from bathrooms with or without toilets uh I have here somewhere there I forgotten where the pass is usually open all this stuff but I closed something state closure CL images mid only okay so looks like that so that's are my training data they are not normalized uh uh different size different format 50 mid and 50 without this is

extra go here so so you see you can see my bathrooms so that's a training data I have really done that so I trained this I show you before how I train that and now I have a a bathroom without I can also take a house or a car or whatever it works and uh what I've done same thing with autom ML model builder uh created the

second project and uh when I run this it says as we have seen before bathroom or no bathroom means no toilet so use case can you really use that within minutes uh next use handwriting and there is a there is a uh large or huge data set uh quite old in the field of AI uh it's called amist and um it's um the target is or the

goal is to detect handwriting and you see here uh zeros in different kind of meta and they're all the same size and all the same look like this is done by some uh data scientists and for this mnist there are also existing models so you're not going to build your own model so you can say okay take a model and um there are different platforms for that

uh and you can use this amnest model within your Visual Studio I dropped this demo because of time uh and go to the next existing model one of the famous existing model is Yolo so let's go bre there so YOLO here we here I have tiny YOLO version two in in Onyx format so I go to somewhere and grab this model download it and put in my

project on um YOLO is have a Target to detect different kinds of objects I think seven or eight cars peoples uh chairs not sure uh dogs cats and something like that and um I have this downloaded and I have here code and if you have uh or you remember my previous steers you see that looks like a little bit I have done before yeah something is doing

a model ml context and then loading data create a pipeline transfer the things because there is called image input and output and so on and so on and now I have uh also a use case for a practical use case um and the question is I have to AI ah you can cannot see that is this a car who believe that this is a car we will

take a look at the opinion of AI okay so I have this picture with a car here and I have different pictures here for instant uh uh okay for in oh or that so these are is a demo application from Microsoft so it's not me and um da the pictures from you have seen pipeline model started and now we predict what is on picture you load the

model you get the scores it gets quite fast and you see uh different things uh you remember the last picture dog score person dog person person so on this picture are two dogs and three persons we take a look at and prove that in a few seconds the interesting question is it set the car and uh AI is not 51% a person is on the picture and

the bonus track of this uh demo is H where is the pass I have opened it I've closed it okay I have to take a look in the code code course the pass is quite complicated where is the pass pass pass pass cor yes it was so so this the part of the pictures and and this code produces output pictures and creating the bounding box so this

is produced from my code here which is stolen from Microsoft course this is a demo project so you see the car is marked and the person is marked and you see also uh the percentage where this model believes that pictures and people on this so and what what is the use case in the reality so you can grab a picture of a webcam cuz this is really

fast grab a picture of a webcam and and checking what person or dogs are passing by your whatever China okay so you have seen um YOLO model using and now we are going to the last interesting thing large language I say there is a a platform for Distributing models and tools and whatever this is called hugging phase so you go to hugging phase searching for a case

and a model and download a model and free is open source and Microsoft uh released a few weeks ago pi three and from PI 3 there are different variants which uh the the difference is from the training data and and and the size of that so this is the smallest one Pi 3 mini 4K instruct whatever that means so this is the small OPI and I can

ask this different things as you please at 1 + one it's quite easy two yeah or six so it's a little bit J here okay so it's it can code it can answer different questions uh the data set is two gigs so it cannot everything so it knows not everything the training data which Microsoft uses is quite small but contains a lot of and the goal is

okay can I use it that in my visual studio my.net project and yes you six okay here it is it's not much um this code is um not stolen but it's there is a project for pi 2 on Microsoft side and I use this code and adapt it for pi 3 it doesn't works if you use it one to one and I have to guess some parameters

which is which which you see will see it's quite funny at the end of the day so uh here is my my model and and this is uh two gigs of data and uh back to ml net I create a model uh here the the the the steps are a little bit different because you have no images so you have a text and text need to be

tokenized it's very important to create uh numbers from tokens or from from text numbers which are called tokens and uh then I encode the input to tokens and um how LM works it's a little bit complicated I pass it and then I say blah blah blah blah uh tokenizer create stream it's not so important and then I have a loop because you have uh to create tokens

and tokens and tokens and tokens this is how open I I also works so every new word is uh created from the past and uh yeah that's the part and you will see that the the results are different to the web chat because I do not know how the parameters works there's no documentation at the moment solve 3x 6 so I have no idea why here is

5X but it do something and uh you yes I have limited the result set to I'm not sure 400 or 300 characters so you do not see the result at the moment ah it's here 0.5 okay it's wrong no it's minus orus 6 it's wrong um okay I have no idea says it's it's quite funny but it's it's not a a thing of ml.net it's a thing

of uh the parameters I do not know I have guess the parameters how to use it so what what is the use case of that so I don't want to chat with my net code or application the use case is for instance back to my single chip solution to run such a thing on a single chip and to switch on or off the lights you can communicate

with your machine you need a function call I'm seeing a API call and can switch on off your lights this is the future I think and this is the end I yes questions thank you Hanes so we are opening Q&A session and there's the first question you can see in the screen from slido uh you see the question no you see not the question I have to

repeat it ah here's a question yeah thank you so first time I hear sorry about that uh yes you have seen that I was very very fast on the training step oh I have still the project open delete me uh model model model training here it is so here's a setting so you have three choices at the moment for GPU support there's something changing for Microsoft side

and at this point there's no longer Cuda needed and you can do it in Asia so Microsoft want to sell you Asia please buy Asia now next question so advice for training data ooh yeah hire a data engineer it's a truth it's not my experience it's so hard for me to understand uh how must the data look like and and I have invested so much time into

to understand how the algorithm is working and for instance uh image detection it's so so complicated or or or large language models with with the vector Cen similarity you have to understand that to to make data which the systems can can use for good models and if you have not time for that hire a engineer thank you now we are out of questions but maybe someone wants

to use this opportunity and ask just straight away if no then we'll have hanis and ask anything uh Corner uh so thank you very much some presents thank you