About this talk
This workshop focuses on building AI applications using pre-trained AI models. The speaker, a technical content creator and front-end developer named Jesse, discusses the integration of AI into apps and emphasizes the importance of understanding how AI works, its various models, and tools such as ChatGPT and generative AI. Participants are encouraged to explore questions like whether they truly need AI in their applications and how to choose between pre-trained and custom models. Jesse shares experiences, mistakes, and best practices in AI development, particularly around data preparation and model fine-tuning. By the end, attendees should have a foundational understanding of AI's role in app design and development.
Full transcript
Okay. Enough. Enough here. Okay. I think we're on, right? Can everybody hear me? All right. All right. All right. Welcome to this workshop, everybody. Thank you all for being here and then I'm so sorry there are not enough chairs here but um we'll make it fun. Okay, are you with me? All right, so how welcome to this workshop on how to build an AI app with a
pre-trained AI. Okay, who's excited to be here even though you had to wait, but we're here now. That's what counts. Thank you all for being here and for making time. And so the goal for this workshop is to understand how AI how you can integrate that into your app with a pre-trained AI model. So I think this is safe to say but how many of you are
using AI on a daily basis? Yeah. How many of you use chat GBT on daily How many of you prefer cloud AI? Oh, I think cloud AI is cool though. We got to give it some props. So, this is an interactive workshop. Okay. And I know there are people standing in the back. And there's actually more reason just to come forward. Okay guys, just to come forward
and to meet other people because that's why you're here at the conference. Am I right? So, just shout out. What is AI? Is it scary? Is it ill evolutionary? What is it, >> sir? >> It's uh scary. Definitely. >> Definitely scary. He says >> it's scary. Like it's out of the proportion. >> Yeah, it can be very scary if you don't know what you're doing, right? If
you don't know what AI is all about, right? What else, sir? Do you know anything? >> It's a hype. It's a hype. Yeah, it could be a hype. Yeah, if you're not careful. Could be a hype. We have a mic, by the way. Maybe you could come around. He gave me the mic, but um >> he's gone. >> He's gone. Okay, that's okay. So, AI is many
things, right? So, just so you know, in this app, we won't build a full AI app, but you will leave with the fundamental knowledge on how to build an AI app because I'm going to teach you what it takes actually to build with a pre pre-trained AI. And um I'm also going to show you the mistakes I made. So, how many of you learn more when you
make mistakes? Yes, that's right. Because even when things go right, how many of you code are coding the whole day and suddenly everything works and you had no bugs. Wait, what happened? Like this can't be true. How many of you had that feeling sometimes? This can't be true. It's too good to be true. Yeah, that's right. So we you you won't build an AI app in this
workshop but you can check out my AI apps on GitHub and of course everything is public okay so you can sneak peek and do whatever I I did and comment as well because I will also learn only from that as well. So this is my profile. Okay, it's Jice and um very short. My name is Jesse, but this just a nickname that I have. Okay, foundational knowledge.
Obviously, we have to go through a few milestones here, right? How many of you are developers here by the way? I got my favorite crowd in the house. Sh. So, what is AI? How does AI work? What AI models, tools, and APIs are there? Where does AI fit in inside an app? And of course, the ins and outs of AI development, aka all my fuckups, mistakes, and
learnings that I had building AI apps. Okay. So by the end of this workshop, you'll have a clear understanding of how AI works and how it's different from traditional apps, where it fits in, and actually how it changes app design. And you're also going to be familiar with some AI tools and models and APIs available right now, and also what it takes to design a functional app.
And moreover, what I want you to walk away with is that the questions that you have about AI and building that into your app, what are the things that you have to think about? How many of you would agree that development is actually just a way of thinking? How many of you would actually agree it's a way of life, right? Otherwise, we wouldn't be here. So, but
the goal of this workshop is to have fun. Okay? Okay? Because when you have fun, you get to learn more and you remember more. Okay? So, when I say something, just go all out. Okay? Nothing's too crazy here. Oh, but wait, let me introduce myself before we go further. My name is Yesie. In Dutch, we say yes. Jessie Day. I'm a technical content creator. I'm also a
front-end developer and I also build AI apps. I have a platform forum called the developer business school and I am interested and focus on building apps that make impact and money. Okay, I believe those things can go together. Are you guys ready? Hey, let's go. So, what is AI in a nutshell? Okay, artificial intelligence. the ability of machines to learn reasons, make decisions without being explicitly told
what to do, right? AI agents. How many of you have heard now of AI agents? Yeah. How many of you are building one now? Cool. Cool. All right. So, of course, AI versus classical programming. You know when you have like for example when we look at logic execution predictability adaptability and error handling you know very fixed like classical programming you could think of the static chatbot know
very predictable must be manually updated takes a lot of work and it's very coded explicitly AI comes into the game and now suddenly it learns it evolves okay it improves over time But error handling, it can still make mistakes. So what types of AI models are out there? Rulebased AI, which a bit of a mix in a hybrid. Machine learning approves with more data. The bigger data
set gets, it learns more, it gets better. Deep learning whereby the AI thinks like a human being because of the neural network behind it that looks like a human brain. And of course there's generative AI. How many of you use Midjourney? Oh, not that many people. Wait, where how many of front-end developers are here in the room? Oh, okay. Okay. Okay. So, okay. I just wanted to
know because I really love generative AI for AI art and stuff, but maybe that's just me. Anyways, what kind of AIs are there? Well, you got the chat boss, the virtual assistants, AI powered search, Google of course, vision, AI, face recognition recognizing from an image, how to, you know, your handwriting and how can that be turned into a font, which I maybe will tell later about all
the [ __ ] I made with that app for those of you who are interested. Speech AI, of course, I don't know anyone knows the app speech. You can put it on a book and then it's very useful for people that are dyslexic because they can just hear u what's in the book. And predictive AI, you know, Netflix, right? When it says when you just only watch one Korean
movie and suddenly your whole feed is full with Korean shows, right? Anyone of you experienced that as well? Oh, come on. Don't lie. Hands in the air. Okay. All right. All right. Trading bots. Okay, so a few takeaways when it comes to AI. It doesn't follow strict rules, okay? It learns patterns with more data that you have. And that said, it depends on your data, how good
your AI is and how you train it. Bad input, bad output, right? AI isn't perfect. It makes predictions, but those are not absolute decisions. And it improves with data, but we have to be careful how to design that data as How many of you have ever built an AI H. Okay. Cool. Cool. So, how does AI work? Well, there are many AIs obviously, but the most common
one, Chad GPT, which you guys just told me many of you use that here on a daily basis, right? So what is Chad GPT? A chatbot that is a generative pre-trained transformer, hence the Optimus Prime, right? So he answers your questions that are based on written prompts. Now it's final about founded by OpenAI as you know and OpenAI says our mission is to ensure that AI benefits
all of humanity. was released in November 22, had 1 million users in only five days. And I believe AI has been here already for a very long time, but when CHBT came up, it suddenly became more known to the public. How many of you would agree? And how many of you are sometimes a little bit, you know, astonished by the speed and growth that AI is taking
over now? Yeah. Right. So how does Chad GPT work in a nutshell? So for example, you say give me a recipe for shrimp tacos and it gives you this recipe and it knows exactly how to write the recipe for how many people and what the the the grams, the ounces, etc. So this AI has so-called advanced language models and CGP has GPD3 now four actually and codec.
So what is a language model? Well, it's just a model that understands human language and generates code. But codeex is really more specific um on code, understanding code as well and generating that. How many of you are using GitHub Copilot? Yeah. Okay. So, how does it work under the hood? Well, obviously a lot of pattern recognition and learning and it generates responses. How does it do that?
Well, the user input that comes in which are sentences, it tokenizes that. So, it cuts the sentences into pieces and then for each word like for example cat in a tree, it will look for a cat. It will look what the tree does and maybe a solution how to get your cat out of the tree, right? Because it looks through the data through that. Why CHGBT makes
mistake? Well, because sometimes it doesn't have the context of things. How many of you sometimes ask CHBT something and you're like that's not right. Right. So, how CHGPT gets better over time? Well, there's something called fine-tuning where it's actually more data is given collected by chat GPT and reinforcement learning. So that means, hey, do you like my response? How many of you have seen sometimes response one
and response two lately? And then they ask for the versions, right? That's how it reinforces the learning on how he does things. Human feedback loops and of course also memory and context windows. Now, I want you to look at your neighbor. And this guy's over. Look at your neighbor. Say hi. Smile a little bit. Don't be creepy. Okay? Say hi. And discuss together for a few minutes.
If you had to build an AI app today, what would be the name? What would be the problem that you would solve? And what kind of AI model would you use? Okay. And go. interest. You don't hear my music? >> No. >> Again. Yes. the mic. >> Yes. Yes. [Music] >> All right. Start running up. All right. Start rounding up. Who has a great idea? Sir, there's
unfortunately there's no mic. So, I have to come to you. >> Spotlight. >> That's okay. >> it's a workshop. It's okay. We're all friends here, right? >> We We said that we should do like an app that you can put the ingredients that you have at home and it gives you recipes. >> I love that. So, he says this app, this AI app will look at your
will know what you have left over in your fridge and then we'll come up with something tasty. >> Yeah. >> Yeah. The best cook, right? Food from scraps. Okay. So that's actually a really good What problem is it solving? >> I am not a creative cooker. So you can help me. >> So good food, no waste, sustainability, all of that. And maybe, you know, having great meals
at home. So >> agree. Awesome. What's your name? >> David. >> David. Give it up for David, everybody. >> All right. All right. All right. So that's definitely a thing, you know. Um, so many people say, "I want to build an AI app." Okay, what problem are you Yeah, it does this. So, we're going to talk about that in a moment. Okay. But I love this problem
that you're solving, David. So, because the question is actually when you build an AI app, is should you have AI? Is David actually able to build this app also without AI or is actually AI a little bit smarter? You know, maybe saying, "Hey, if you buy that creme fresh, then you can make three more meals out of the ingredients that you have at home." How many of
you would agree? Right? So, few questions actually. Where should you have it? Well, for example, if it automates a complex task, also if it enhances the user experience. How many of you sometimes talk to the chatbot of an airline and you get like so annoyed, right? But what if it was an AI or pretended at least a person and it would be way nicer. It would understand
why you're frustrated. It would, you know, know how much delay you have had up until the time or who you had to talk to, right? If the app relies on pattern recognition, if it learns, it has to learn to know somebody better. You know, if it's an AI coach that teaches people that have depression or something, So, if it asks personation and adaptability, but very often you
don't need AI. And maybe that's a bit weird from somebody that gives a workshop about AI to say, "No, brah, you don't need AI. What's that?" You know, but that's exactly why invest time in building something that maybe you could have built simpler. I mean, that's our job as a developer to think about that, right? So, even though I'm totally pro AI, I will always ask the
question, but do you really need it? Should you have AI? Well, if it's a simple rule-based system, maybe not if it doesn't improve the experience or if it makes the app more expensive without a clear benefit. clear benefit or if the uncertainty of how AI will react creates more problems than you know um making the impact it should be making. So to AI or not to AI,
how many of you think those were legitimate questions or does anybody have another thing that should be added to this list? The first question you always want to ask, and I asked David this as well, does AI solve a real problem? And would AI perform the task better than just simple code? And does AI make the user experience smoother or smarter? And can the app handle AI's
uncertainty and mistakes? Especially if you're talking to somebody that's depressive or something. Obviously, there needs to be a legal ethical way of how the AI deals with this person. Now, all of these questions and probably more depending on the state, depending on the nature of your app, if most answers to these questions are yes, you should consider putting AI in your app. But if most of the
answers to these kind of questions are actually no, then a rule-based approach might be So, but you went through all the questions, you discussed with your developer colleagues and all that and you decide to use AI. Then comes another big question. Which one am I going to use? What am I going to use? Pre-trained versus custom AI. How many of you have ever heard of pre-trained AI?
Okay, great. How many of you know any custom AI or want to build one? Ah maybe somebody says I want to hear more about it in this workshop than I know what decision to make. So let's talk about sorry I have to laugh because I'm giving away my age I think with this but that's Let's talk about Pikachu. How many of you know Pikachu? Hey So, pretend
we're going to label him our pre-trained AI model. Okay? And what do we know that Pikachu can do? Okay. I'm not going to do it, but you guys know, right? You guys know like the electric zapping thing and the, you know, the pika pika. This is Pichu. Okay, this is the baby version of Pikachu. Maybe not as famous as Pikachu, but you know, he's here. And here's
our custom AI model from the start. When you ask yourself, should I use pre-trained or should I use custom Well, so let's say you want to use Pikachu, what are the pros and cons actually? Well, Pikachu already knows how to fight, right? So, it already knows electric zapping. So, if we give it a So, it's very fast to implement. We already know it works and it's very
cost- effective. it doesn't take us so long to train him into becoming a good fighter, right? So, let's say we think, okay, we want you to teach, we want you to learn jiu-jitsu. So, we're giving him lots of videos about jiu-jitsu from the Olympics and all of that. And this is where Pikachu now knows how to do the electric zapping and Great fighter, right? But without fine-tuning,
which is fine-tuning when we give him the jiujitsu data set, we're fine-tuning the core skill of fighting of Pikachu, that's actually he only knows how to fight. You cannot teach Pikachu how to swim. Okay, for how many of you are now thinking, "Oh, I get it. I know where this is going." So here he does, you know, the zapping thing and we're going to give it the
data set of jiu-jitsu, a lots of information about that. So he can also do jiu-jitsu. But here we have Pichu. And when you have Pichu, the pro the the the benefits of it is that you can teach Pichu anything. Okay? You can teach it the electric zapping, you can teach it the jiu-jitsu, but guess what? It can also clean your clothes. It can clean the dojo and
it can do your admin and it can you know what I mean? So you can teach it anything. So it can be be as good as Pich Pikachu at some point. But it can even better which is Raichu. How many of you have ever seen Raichu? I see so many faces like like guys I never thought I would explain AI through Pokemon. Okay, but here we are.
So anyway, the cons when you use a custom AI model, a Pichu, then it's going to cost a lot of time because you need to train it. Okay? And that requires of course a higher cost. So what's an efficient way? Why would I if I pick this dude, I can eventually get here, but what if my AI app actually only needs him? That's where you have to
start thinking So this is just an overview of both uh models, AI models. Oh, I went too fast. By the way, guys, I wrote an article about this and you can upload that later as well if you really want to get into it and read into it. I explain really technical what this is all about. So we already talked about this a little bit, but here again,
you know, do you have a unique data? Do you need full control? Is the time to market uh important? So when you push it out to the market, is the cost a major factor? And do I need AI to evolve over time? Well, Pikachu or Pichu, now you know, If most answers are yes, then do the pre-trained. And if most answers are h not really, then do
custom AI. Okay, here comes a moment again where you look at your neighbor and I want you to ask oh there he's there's also a neighbor there are many ways okay and discuss with your neighbor what do you think now that you know a little bit what AI is and what it's done and we're going to go deeper into it okay we're going to dive technically deep
into it what do you think is the hardest part about building an AI app okay Go. It's like really Easy, This is why we went to Starbucks. Did you find the >> Oh, >> that's okay. Anybody this? >> All right, let's round it off. And again, who would like to share what you think is the hardest part about building an AI app? >> Well, they're they're having
a really good conversation probably. >> What's up, guys? Let's keep it central. So what do you think is the hardest part >> Uh for me I think organ like prepare for feeding the model. >> The data for feeding the model. Excellent. What's your name? >> Art. >> Artm everybody. That's a good that's good. We're going to talk about that. The data that you give the AI right
because when you give it bad data if you tell an AI 2 plus two is 10. when somebody ask you 2 plus two is 10 and it gets reinforced what's the truth at that point right so exactly the data what else what else is a hard part about building an AI app guys sir >> the how unpredictable the LLM are >> how unpredictable the LLMs are so
am I training the AI the right way for the right input and how can I measure that >> right for example Tony Robbins has an AI I coach. So, I'm talking to it. But is that really what to Tony would say? [ __ ] How do we know? How do we know? Right. So, excellent guys. Let's talk about AI and code because you guys want to know how AI
works in an app. Is my mic still on? So, this is an AI workflow, right? This is what happens under the hood. There's an input, it goes to the AI and it the AI processes it, right? And that could be anything. That could be anything. And so it gives an output back. Now, this could be anything. It could be text, images, voice, or numbers. The input goes
here and the AI which has already been pre-trained and fine-tuned will do something with the data and then it will have an output which could be a refined text a refined image whether that is AI art or whether that is maybe a summary um from your LinkedIn profile or for your resume or something. So let's look at a real app. Okay. And if you guys are on
my GitHub, by the way, you can check it out at repositories. Uh it's here, the UIUX assistant. Okay, you can check that out. And if you don't have any access, come to me after the workshop and I'll give you access to it. Um so let's open this one up. And it's already here. Oh, wait. Wrong app. It's not the one. is the next one I'm going to
talk about. Okay. All right. So, very simple. Don't look at the front end. I was more busy with the AI uh and the model. So, what does it do? This UI UX assistant can look at an image and can decide whether the contrast is high or low for people that are visually impaired, right? That don't see well. So I gave it a very strict rule, very binary,
high or low contrast according to web accessibility standards, by the way. So let's say I go here, I go to my desktop and let's put this one in mobile 2 and so analyze So now it's analyzing in the back end and it's calculating whether this is high or low contrast. And here, oh, it's low. And I'm going to show you the data set behind it. But what
it does, I have data set of the whole app. I have it of components because I have to also tell the AI exactly what the relationship is. Like what the hell are you looking at? I have to teach it, right? So other than that, I can also give it another file. Maybe this one. Oh. And this is a high contrast one. So same same thing. Okay. Same
thing that happens. Now I also added very simply just an API call to chat GPT to open AAI to give some feedback on hey when I have a high contrast give me tips on how to make the high contrast better or when I have a low contrast give me tips on making the low contrast better. That is just two two words, right? But I could make it
more specific by giving the RGB of the components and of the the the mobile app and give that to Chad GPT and then he knows what he's looking at and what he can give advice about. But for simplicity sake, I haven't gone into Okay. So, let's get back here. Wait, where? Oh, what's this? Let me turn on the presenter mode. All right. That was the AI power
UI UIX design assistant. Okay. Okay, so the first workflow what you see on the front end is where the user uploads an image, the AI analyzes the image and then the AI gives feedback. But guess what? This AI is already pre-trained under the hood. And like this gentleman said, the hardest part about an AI is also what data you're going to give it, right? What data are
you going to give it? So once you have pre-trained it, you save it and that's what you can uh put into the front end to use. How many of you sometimes use chat GPT and then it is able to do this one two three and then one week later it's suddenly able to do one two three four until 20 and you're like what the what did this
guy had for protein shake you know like what happened? Well, that is because they train that under the hood, they save it, they fine-tune it, and then they push it out in one go. That's why there's a big gap uh between those skills. Okay. So again, what we did actually was the input was the design, it goes through the process, and then it outputs a high or
low contrast plus the feedback. Now, I use a pre-trained AI called mobile net, which is able to analyze images. So it knows what it sees. It can um detect the hex colors, dark, light, etc. And it knows what what So that is Pikachu and we with you know I fine-tuned it actually with the data set of Figma because it could already analyze images. It could already analyze
colors but I just gave it more app designs um to fine-tune it so it can do also jiu-jitsu. Now when you work one thing you need to know I'm going to tell about this later when we have time. When you work with an AI model or with mobile net, an image, a v a visual AI, you always have to worry about pre-processing. So what does that mean?
That means the AI can only look at an image when an image is delivered in a certain type of way. It is resized. You have to flatten it out. You have to I'm going to show you the code. Okay, I'm going to show But fine-tuning, that's the hard part. Okay, that's the hard part. you have the data set, so jiu-jitsu videos. And in this case, it's very
binary because we actually wanted the AI to just look at the app design and be like, hey, high contrast, low contrast. So, it's very binary, right? But depending on what your AI has to do, you have to decide what the data is. 2 + 2 is 10. No. Is it Tony or is it too is it too generic? Can anybody say that? Do you guys hear what
I'm saying? So, here is very binary. But it's just extra knowledge. Okay? It's not an extra skill. This AI cannot go swimming. It can only recognize images and analyze images. So people get that sometimes twisted. Unfortunately, this is not Pikachu, but this should be Pikachu studying and going all over the data. And what it actually does, it looks at the data and then it learns and then
it looks it takes one data set and it says oh this app design this component this image that I have good or bad now good or bad looks at the JSON label oh wait I had it wrong oh wait I had it right and it keeps track of that okay when you see the code I'm going to show you it will become more clear but I just
wanted to share is that the data set how you make it depends on the solution that your AI I is giving out your app is giving out. Let's say I want to have uh an AI that detects Pikachu, right? First of all, it needs to detect Pikachu in itself individually. So, this is a good Pikachu standing crawling, but doesn't look like a nice Pikachu, right? Not how
Pikachu wants to look. So, this is bad data. But then it knows like when it sees this like, no, that's not a Pikachu, right? So, it needs to know what Pikachu is individually, just like my components in the mobile app. It needs to know where's the navbar, where's the the smaller buttons, what the contrast is. Now, here we have Pikachu. So, hey, this is a good grip
picture. Here is a good data, but here he's not here. So, now the AI knows, hey, wait a minute. I'm looking for this guy, you know, the yellow squirrel or whatever he is, you know? So very simply said, when you think about your AI workflow, input, process, output, what pre-trained AI model are you going to choose? And what is specialized data you can fine-tune that AI with?
And what is that data set about? You know, good or bad data, very broad, very Tony Robbins, whatsoever. And when you train it, when you fine-tune it, they call that training. But actually, don't get that twisted because actually when you're only talking about training, you train Pichu, but you fine-tune Pikachu. How many of you do understand? Yeah. So that's different and that's because of that iterative nature
that we say training and that's an epoch. Okay. So, so very quick summary what I've done for building this AI assistant before I'm going to drag you into the code. Okay, so I collected and cleaned up a data set. I collected a lot of Figma uh components and then I pre-processed all of those images so my data set is clean. And then I loaded the pre-trained AI
model which in this case is mobile net. And then I fine-tuned mobile net. I said basically to mobile net, hey the contrast, this is the ratio, this is the calculation. Do that with every um thing that you see, every image that you see and train yourself on it. Fine-tune yourself on it. And so it ran multiple epochs and tweaked hyperparameters. Well, I can't show you guys this
because first of all, the data is huge and I'm only making a decision here. High contrast, low contrast. If I would start fine-tuning now, we're still here um tonight late at night. Okay, maybe tomorrow morning. So, I'm not going to do that. But, um, you can check out the code yourself and you can try that out. And of course, you tested it. You tested. Okay. So, here.
Let's see where this is. Can you guys see this? All right. How many of you are learning something? Okay. Awesome. So, how many of you are working with Nex.js by the way? Oh, not that many people. Okay, but this should be okay. So, here you have the design assistance. So, this is just the front end. Okay, this is just the front end where you see the AI
workshop app and where it uploads things. Now when I click upon it, when I choose the image and I do handle analyze, I send it here. I send it to the back end to the API. Okay. And what I'm going to do is I'm going to pre-process the image that was just given here in the pre-process what happens is actually so this is the workflow, okay, from
the app. This is not the training, but I'm going to explain it to you. I'm going to explain the workflow. So here you first load the mo model from mobile net and then you say to mobile net hey all of those models that you have by the way and all the images that you know let's freeze those. Why? Because Pikachu for example it knows how to fight
with the electric zapping. But when you teach it jiu-jitsu you don't want it to forget its original skills. Right? That's why that's why you have to freeze the layers. That's how they call that. you have to pre-process no so with every AI model like mobile net and other AI models they all have different requirements so those they call that tensor so you have to like um that
image you have to prepare it like a cook you have to cook the meal so the AI eats it if the AI is vegan you have to make it vegan otherwise the AI is not able to process the image that's how you have to think about it okay and then here indeed you see it predicting making the predictions and what I have done is like hey listen
you need to predict it yourself you need to fine-tune you need to now apply what you've learned in your fine tune training you need to apply that now again okay and so there is where I gave it um when it's a high contrast or low now you guys who wants to know about the master data set yes so I I was thinking, you know, I want to
have this app and I want it to be high contrast, low contrast, and maybe if it's if it can do other things like um uh you know, if the letters are big enough, if it's small enough, if the And then I thought, wait, how much data do I need for that? Wait, how do I teach AI the correlations between an app design? [ __ ] I need the mobile
design. I need all the small components. And hell no, I was not going to screenshot every component or every app design on dribble that I saw that I liked. And that said, by the way, Dribble has really nice designs. Nobody would put their design on dribble and you know be super bad for people to see. Like don't put bad designs, right? You only put your nice designs
on on on dribble. So actually the qualification was what is my good data and what is my bad data? Well, then you have Chad GPT of course and a lot of minimalistic UIUX is not visible for people that are, you know, visually impaired. So, what did I do? I went to Figma and how many of you work with Figma? Most of you, right? So you can actually
with the API you can call up the UI kit and you can pull out all of the images from the UI kit and then you just have to organize it and with the name and the PNG that they have. So that's what I did. So in order to prepare this master data set so here you have the master but here is raw. So I pulled from Figma
components and I pulled also frames so I could uh show the relationship between those also from material UI. And then here in the in the app itself here before I can even fine-tune the AI. I have to prepare. I have to you have to cook like a month. That's how it feels. Okay. So here I have the cleaners. I have the fetchers. I have the readers and
of course I have the contrast checker that says hey check this to see if it's high high contrast or low contrast here. So this is all raw but here in the master they said this is where you wanted to be because this is what you're teaching your AI. So I had a I had to divide this up because I needed to to show the relationships but also
this is the thing with every button. This for example it had a label. Okay. So I had to like make all of those labels so I could give a certain properties. Okay. oh you can't see it but here here is a contrast ratio. Okay. because I wanted to show the the contrast. So that is where it took that it checks it in puts it in the contrast
checker and that's where it comes out for a high or low contrast uh calculation. So am I forgetting anything? Here is where the model gets trained. Uh yeah. So you load it up from mobile net and then you freeze the you freeze the layers here. So it doesn't, you know, it doesn't forget what to do when it starts learning jiu-jitsu. And then And then again, you have
to pre-process. You have to pre-process the whole data set as well because there's two flows, right? One the you from the user and one from when you're training the the AI. It always needs to be pre-processed. Okay. So that's it guys. Any questions about this so far? Freezing. So freezing means um freezing means that the the so mobile net already knows how to analyze images. >> So
it already has a data set of its own. So Pikachu already knows how to fight with the electricity. If you fine-tune it, if you give it an extra data set, if you don't freeze these layers, then Pikachu is only going to be able to do jiu-jitsu. It forgets about the electric zapping. mobile net. If you don't do it, you have to train it again to analyze all
of the images. Does that any Okay, awesome. So, let's continue here. Oh, turn to your neighbor. Promise. Last time. Last time. Okay. And oh, by the way, you can pull out your phone if you want to have a road map on how to build an AI app with a pre-trained model to go really technical deeper into it. It's an article I wrote for Front Mania and also
yeah you subscribe you're subscribed to my newsletter if you do that. Okay. So how many of you would pick a pre-trained AI model over custom AI model? Pre-trained AI model. Yeah. And how many of you think damn actually I want to make a custom AI model because I'm working with personalized data. I'm I'm having a very special specific idea. An AI like that doesn't exist. But then
you think hyp is already coming becoming so big. Why wouldn't I have that in my app? you know users already know how chubby talks or how you know chip already has a huge skill of understanding human language how are you going to top that right so those are questions um you have to think so discuss with your neighbor what do you think now that you heard everything
what are the most common mistakes when building an AI or when building with AI Hi. What are the most common mistakes? Okay, so I'm going to give you some time. Just going to walk around. because if you're talking about the good stuff, you also have to talk about the bad stuff, you know? >> There's some spots left here to sit. So, if you want to come to
the front, you're more than welcome to sit here. >> So guys, tell me what do you think are with an AI? >> Sir, >> using the wrong data set, choosing the wrong AI. >> Choosing the wrong data set and choosing the wrong AI. That's correct. That's correct. And also like for example in my UIUX assistant I'm thinking now Figma is a huge source right of components where
the hell am I gonna find other sources right um dribble yeah other so I have to like really dig in deep in order to to to to make right data right and choosing the wrong AI model. Anybody else what do you think is a common mistake when building with an AI? Do you have an idea sir? I think AI models sometimes are like outdated data that >>
Exactly. AI models are outdated or AI models don't seem to do what you think it should be doing. Okay. Oh, for example, how many of you know hugging face or heard of it? Yeah. So, hugging face has a lot of AI models that are pre-trained that are trained by somebody. They fine-tuned it or they made their custom AI and then they put it on hugging face so
you can use that in your app. Obviously, they want you to use their AI model. But is the AI model really as good as they say it is? Well, it's only one way to find out, right? That's why we're developers. The exciting part, but also sometimes, well, we all know. So, let me share with you the five mistakes that I made when building an AI app. And
actually today I was almost thinking because I actually was working on an AI app and um I'm going to explain the idea to you guys in a minute and uh in the end it didn't give me the result that I wanted. Big surprise but you know at these workshops at these conferences we're always talking about this went great and I did this and I achieved that. But
actually when you talk to your colleague and they made a [ __ ] and they share about it, how much do you learn, right? How much faster would you learn? So this is why here I'm sharing the five mistakes I made. The first time with the UI design assistant, my goal was too vague, right? I wanted to build the ultimate web accessibility That'd be great. But I think also
if I came up with that idea then Figma would have already built that right not balancing the good data and the bad data. Some of my bad data wasn't you know was still very high contrasty was still very visible. So how would I know that? And also I don't have any visual impairment. Um so it's also hard for me to test it myself and be my own
avatar. skipping systematic organization. Well, here also for the workshop but also I had the story right you know like data set cleaner fetcher because it told the story loading the data mo the the model fine-tuning it but if you don't have that you put everything in one code disastrous I can tell you not defining the training labels the right way how many of you how many of
you saw which which data structure the label was did anybody see JSON. Yeah, Jason. Exactly sir. So the label is very important. The label makes 2 plus two is 10. Correct. Wrong. Right. So the labeling is very important on how you architect your data and under estimating the data prep time. You know I thought oh yeah AI can handle that. We have AI now. But the prep,
you know. So when you are building an AI app, you know, you are not his master, okay? You are its slave. But that's okay. That's what we're here for. Oh, okay. Do I still have time to share this? I have. So I was working on this yesterday and one of my goal always is with a workshop to make it interactive, right? To do something physical or or
um something else. And what I wanted to do actually is this is an AI app and you I would give all of you a paper with a grid on it and you would write your handwriting on it or in Canva or something and then so it would be small letters big letters upload and generate the font. So it would AI would look at this. It would recognize
it as small letters from A to Z and then capital A to Z and then it would turn it into a font and then would give it back to you. Well, as you can see, the app works. It successfully generates a font. I just don't think, you know, my mom could understand me if I would use this, but it's a nice idea. So, we still have time.
So, I'm going to dive into the code and show you guys because this is a was very hard and this is, you know, this is the this is this this was really cool to do actually. Um, let me show you what this is about. All right. All right. By the way, how many of you work with Python? Yeah, Python's pretty cool. This is actually one of my
first times to work with Python. So, we have the app here. All good. You know, it uploads the image and all that. And um then we we send it um to upload, right? So, here we send it to the back end. We got the main PI here, the route. So let me just explain the routes because this is like the is it big enough like this? I
don't think people in the back can see upload the image and again so you first upload the image and again you process the image before you work with it you know the handwriting I had to extract the letters. So here actually I'm using a tr OCR which is a model from Microsoft that I could get from hugging face and I that I put in there but apparently
it doesn't work as well as I thought it would or it wasn't able to distinguish the letters because you know it has certain spacing and so that's harder than actually um I anticipated. So after you extracted the letters that's when you have to start recognizing it. Okay. So what does it do? It extract the letters and then it knows it has to compare that to a normal
alphabet. So it says, "Hey, I think this letter looks like B." Yes. No. And the funny thing is you can see it here, right? You can see it here. Oh, you can see it here that it's it wants to it wants to categorize it. So this one is correct, but this one not so much. Right? This one not so much. It will look like CD DG but
it's actually only letters A E F G and why it has so many characters in general I don't know either but that's why we will not we'll uh learn so then since TR OCR already knew how to recognize handwriting I wanted to fine-tune the AI model with your own handwriting that was my goal I thought it was night for the workshop it was not going to be
a big data set it would be fine-tuned. So that's what happens here when it's being fine-tuned on the user's handwriting. Okay. And then when you have done that, when you have fine tuned that you have to like put it back to SVG or PNG before you generate it into a font, which is not AI by the way, but it's a framework called font forge. And that's it
actually how this app uh worked and how this came about. questions? >> So, >> couldn't you do this without >> Oh, now we get the mic. That's awesome. >> couldn't you do this without AI? I mean, you said your input is a grid. So, you know A is on one. So, basically, you already know the alphabet. >> yeah. >> So something like why do you need AI
for this for ask Microsoft to ask hey this is this is my handwriting this is a a but you already know that >> that's a good question >> because if you just have a a grid on paper you scan it you know the grid so it's fixed positions >> so this is a a lowerase A this is a capital A. So now you can say okay but
then basically now you you already know what what your alphabet is. >> Well my original idea was actually that's a great question by the way. My original idea was that to turn it into a font which you also don't need AI for but actually to generate your font >> you know so bold metallic change that up a little bit. So it was just the diversity that came
with it and to make AI art of it with your handwriting. That was actually my goal. So yeah, see the goal uh went too vague, small, that's what happens. Yeah, that's a great question. Yeah, awesome guys. So yeah, this is it. This was the workshop. Uh thank you guys all for coming and uh if you have any questions, let me know. [Applause]
