Room 103 Saturday Mar. 07 - SCaLE 23x

In this talk, Jack Pierce presents his journey in developing emotional AI robots, specifically focusing on a beginner-friendly project he embarked upon. As a high school senior aspiring to study mechanical engineering, he draws inspiration from the Marvel Cinematic Universe, particularly Iron Man, to illustrate the potential of technology to create emotionally responsive robots. The speaker outlines the three essential components needed to build a robot: input devices, processors, and output systems, while emphasizing cost-effectiveness and accessibility. He introduces Marvin, his prototype humanoid robot, designed to communicate with users and express emotions through simple movements and programmed responses. Throughout the presentation, Jack shares his learning process, the challenges he faced, and the tools he utilized, such as Python and Arduino, to bring his AI concept to life. He concludes by encouraging others to explore robotics and AI, highlighting the importance of ethical considerations in technology development.

Uh, check check. >> I did it. That is the speaker. Yeah, that's usually the case for basically any talks that I' and all talks I've ever been to. Like there are questions at the That's just how powerpoints are in a way. And you have like a full hour until the next one. So if you want to like have some of that time afterwards to do questions, you

can be like, "Hey, like until the next person is over." So you have 45 minutes. So if you don't think yours will extend full time, then you can fill that up with questions. But if you think you're going to take up full time, then you don't have to worry about the next >> Yeah, there That's what I thought. All I know is That's what I'm going to

put on my school for my volunteer hours. I will look out. I just thought I'd show kind of shadow. I won't know. One minute. We'll start in a couple minutes just making sure everyone is settled and um any late arrivals Yeah, just in case people are like What do you want? >> What do you want? >> Test one. Test two. Test test testing test check one two

check one two hello >> awesome thank Test, test, test. >> I like All right, if we can bring our attention back to Jack Pierce with Emotional AI Robots. >> Hi guys. Whoa. [applause] Dang, there's a lot of people here today. That's crazy. Uh, well, first off, thank you so much for being here. I really appreciate it. Um, as the wonderful presenter over here said, my name is

Jack Pierce. My presentation is on emotional AI robots. Simplified, cost-effective, and mostly for beginners because me myself, I don't have all too much experience with this. So, I want to see what you can do with the resources available and the time that you're able to have. So, before I, you know, start throwing facts at you, start throwing a bunch of things, um, let me just introduce myself

a little bit. So, for the third time, my name is Jack Pierce and I am a senior at Roondo Union High School, surprisingly in Roondo Beach. Um, so not too far to get to Pasadena. Love Pasadena over here. And it is my first time at the scale convention. I'm super grateful to be here. [applause] Dang, all of you guys are so accepting. This is awesome. So, um

I am planning to go into college for mechanical engineering. I am not committed to a college just yet, but hopefully that'll change soon now that decisions are coming out. But, um there are two main things that you should know about me before I begin this conversation. Number one, I'm a superhero nerd through and through. This is like one of my main focuses in my life, what I

spent a lot of years delving into. And more specifically, the reason I got into the super superhero genre in the first place was through the Marvel Cinematic Universe. And let me begin with a little bit of a story. My dad, who is in the audience today, right here, um he when I was about 7 years old, I believe, um invited me to go see my first PG-13

movie. This is bonkers when you're a kid. Like there's a few there are a few things that you remember for the rest of your life and one of them is seeing your first PG-13 movie and seeing your first rated R movie and it's just a magical experience. You feel like an adult for a day. So the movie that he invited me to see was Ant-Man. Just came

out. I knew nothing about Marvel at all. So naturally I asked, "What the heck is Ant-Man? It sounds super goofy to a seven-year-old. Never heard of this in my life." and he said, "Oh, it's one of the Marvel movies. It's been running on for a long time. There's a whole universe and everything going over there. Awesome movies. Just bunch of cheesy action you're able to get into."

And I asked, "Well, can you tell me a little bit about Marvel?" And that night he showed me Iron Man for the first time. And that completely changed the course of my life on the opposite direction. Initially, I wanted to go into the arts. Um, my dream school was going into cow arts. I was really into drawing and you know when you're a six-year-old you have really

high expectations for yourself. Cal Arts is a very difficult school. I doubt I would have made it in anyway. But ever since I saw that movie, I have gone from the arts to the sciences overnight pretty much and I began becoming obsessed with all through the Marvel Cinematic Universe, one character that I have deeply followed the entire time was Iron Man. And that um brings me to

my presentation today. So, the Marvel Cinematic Universe, specifically with Iron Man, um differentiated differentiated itself from many other superhero movies or superhero um based on one thing and it also has in common with Spider-Man in the Marvel Cinematic Universe. Um, things are self-made, but specifically focusing on Iron Man, he is able to build a suit in a cave with a box of scraps. And after a bit

of improvement, of course, and a bit more technology and buttloads of money that I, as a broke student, do not have, be able to build systems and robots and technology that are able to stand among monsters and gods. And that is just a magical thing to me that science to its full potential can basically make you super. And that doesn't always stick in just the Marvel Cinematic

Universe. Just look today. There are many different systems that we would have never even thought would be possible 5 years ago. We're advancing on and on every single day with the new technologies we get and even closer to this fictional universe of um the sciences in their full potential. So going back to when I saw Iron Man and that basically changed the tra trajectory of where I

want to go for school and for my career. I asked around to see what would be the best path to possibly go down and of course the answer was unanimous and that was mechanical engineering and specifically in the field of mechatronics which if you do not know that is basically simplified robots it's um mechanical engineering mixed with a bit of electrical engineering if I am simplifying this

wrong I would be glad to be corrected after this but um this is what I knew now that I want to do. So I kind of changed my entire style of learning to be able to fit that path. I started taking all the honors and AP science and math courses that I possibly can, accelerating myself in those classes, taking all the engineering classes that I possibly could.

And not only was it such a fun time to take all those classes and see how much you traveled in basically a year, but you see yourself coming closer and closer to that dream, that broke Iron Man dream that you had in the first place that got you this far. And now that I'm going to college, it's just coming all together in being able to have an

opportunity in the future to finally make that change that I wanted to. Now of course there's a significant thing that I've been leaving out in terms of technology arguably what makes half of Iron Man so powerful that is Jarvis and that is artificial intelligence. This is something I would have never thought when I was six years old. But artificial intelligence, as we all know, has been rapidly

evolving for decades, but more than ever in the past probably 5 years, 6 years. And it has become such a widely accessible thing and such a powerful thing that you could search up on your computer, ask it anything in the history of the world or the history of the internet and it'll be able to figure it out for you in probably less than 30 seconds. And now

the most impressive thing is it could create stuff. It could create images. It could vibe code. Vibe coding is super fun. Tried it out just recently and able to create media. It's just so impressive and so powerful. And the thing that impressed me the most is this is the missing piece to that Iron Man dream, that broke Iron Man dream. I am able to have my own

Jarvis to control my own robots, to control my own arm, to get me a pencil off my desk because I'm too lazy to pick it up myself. It's just wonderful. And now, even though AI of course is evolving every single day and the sciences are evolving every single day, I thought that now is the best time to be able to put what I learned into practice to

maybe start building towards this dream with all the preparation that I've done. So, let me introduce to you, I love my acronyms, my mostly activated relying on voice intelligent network, also known as Marvin. Really fun name. This is him right here if you want to take a look after the presentation. He is a little fragile. I had to disconnect his head to bring him on over here.

But this is him. Doesn't he look adorable? He has eyebrows, eyes, a nose, a mouth, and everything. Sort of like a humanoid robot with a little bit of emotion. Looping back to the title. That is a main thing that I will be going through with this talk. Not only do I want to make robotics sentient through artificial intelligence, but to be able to react as well to

gap between uncanny robot in computer talk to humanish. I doubt we'll be able to get there for a little while, but as close to it as I possibly can to be able to feel for it, feel emotions for it. So, of course, why some of you might be here and some of you might not be here for it, but might be curious. How can you make this?

You know, I'm just a beginner. I haven't been on the robotics team. I've been really busy. Don't have too much time to look into most of this stuff. But how can someone be able to have the skills and learn how to make this? Well, you came to the right place. Thank you so much for being here. And for all the people watching at home, thank you for

being here too. Forgot to thank you earlier. Great. So, of course, every great project starts with a little bit of criteria. You need to have that road map of where you want to go in the first place. And mainly for robots from through my research, um there is always three main components that you need. That is an input which is what you give the robot, a processor

which is what takes the information of what you give to the robot and deciphers it and then an output where it actually does the thing. So for me with this specific robot showing emotion with artificial intelligence, I wanted to be able to talk to it just like any other human or any other friend. So, my main input criteria was to have it understand user speech through speech

to text. Um, for the processor, if you haven't guessed already, you process the command that was said with artificial intelligence. And then finally, it brings it full circle and takes that processing command over to the output where it says it to you and be able to react to it by, I don't know, if it's a robotic arm, just bend down a little bit when it's sad. Maybe

I act surprised with its eyebrows going up if it's a humanoid robot. And finally, the most important thing in this process for me, yeah, to be low cost. I don't have money. I'm saving up for college. And college has been more expensive than ever. So, I needed to be cost effective, understand human speech, be able to be artificially intelligent, and have an emotional output. Great. That's my

road map. Now I need some starting tools for this road map. You can't dig a hole without a shovel. So the most important thing that I always think for my projects is you always have to evaluate the things you already have. And this ties in with the cost effective nature of it. And with me by evaluating this in the first place, it saved me over $100. That

is over half the budget I was planning on this robot. So it just completely was helpful. So first off, best friend of all time, YouTube. Wonderful. Please never take YouTube for granted. It is such a wonderful source and you should always make sure to use it in all your projects. It's wonderful. Great things out there and the internet as a whole. Don't take it for granted. Number

two in Arduino kit from my cousin from Christmas two years ago. Thank you so much, Nash, if you are watching. Basically, a few years ago, while I was starting to get into robotics, my cousin gifted me a kit full of electronic components, an Arduino, a breadboard, some wires, and I tinkered with it a little bit, but then threw it in the back of the closet. And they

found it again. And guess what? It gave me something to control my robotics with. Saved me a ton of money. Things like that are very expensive. So, the third thing that I also had was an old robotic arm Arduino kit from Adafruit. It was pretty much in shambles when I found it. I built it when I was eight, starting to go through my broke iron man phase.

Um, building these kits online. It broke because why would you trust someone so young and unexperienced to build a complex robotic arm and threw it in the back of the closet? But I found it again and it gave me a strength thing to go off of. And finally, some old wires in an Arduino Uno from my eighth grade BB-8 project that miserably failed. And I was so

embarrassed of it that I shoved it in the back of my closet. But by bringing all these things out, I had a lot to start with. Probably over $100 of materials that I could just start programming with. It's wonderful. It's and it gave me some great tools to start my journey off with. So of course for your input processor and output you need things to start with.

So um I processed the information and interpreted it with Python a very very powerful um program. I used um PyCharm. It's wonderful. It's um free to use. I heavily recommend it. And for my output, I used Arduino, which is mainly used for robotics. Super simple to learn, but you could do such complex things with it. And me myself, I had a little bit of experience with it.

So great, another tool to go off of it However, I also had a little bit of a problem. I don't know Python. And this is more than half the project. More than half of my project is the input and the processing of artificial intelligence with Python. Yikes. So, I had a starting point to go off of, I guess, on turn a negative into a positive. So, I

knew where I needed to start because it' take the most time to learn it. So, here's my list. Learn Python. I need to find a tutorial, spend a month learning the language, maybe make a test app to sort of Okay. way, way too much for time that I don't have and money that I also don't have. So, let's try this again. After I thought a little bit,

I had a better solution. Find a YouTube tutorial that's 15 minutes long and gives you everything you need. Boom. Wonderful. Problem solving at its finest. And this video is from CS Coach. You saved me so much time, money, and effort. I prioritize efficiency and this is the most efficient path that I could be able to find. So, thank you CS coach if for some reason you're watching.

So, that simplified my list from 11 things down to three things. Follow a tutorial, grab an API key, which basically is just a license to use artificial intelligence in your program, and debug. Debugging took an awful long amount of time, probably a week with a little help from the internet. But in about that week, I had Jarvis on my computer. This is something I wouldn't be able

to think of um 10 years ago. Something a lot of people wouldn't be able to think of 10 years ago that you could have it so accessible and almost for free on your laptop with the exception of the API key. And it's just such an inspiring thing to see your dream actually come true with enough time and dedication and effort and things being able to come readily

accessible. It only makes you think 5 10 years. What could you possibly do next? You can make an Iron Man suit in your garage with some leftover scraps. Maybe a free program that could give you Jarvis in one click instead of a week of um practice and debugging. It's just a wonderful thing to think about, some food for thought, and gave me the motivation to keep going.

So, if you do want to recreate something like this, heavily recommend this video. I used a little bit of stuff for debugging from other videos. So, go down the list, see which works best for you. So, great. Objective number one done, the hard part in my opinion. So, now we move on to objective objective number two, the Arduino system. And with this, I had three main goals.

My first goal was to try and find a way to receive Python commands into the Arduino. Oh, I got to be honest, I did not think this was possible. I thought that this was the end of the project. Like how can one thing all the way over here in a completely different language communicate with something all the way language and be able to talk to each other

on the regular quickly and efficiently and cost effectively. So I thought either I'd have to drop a ton of money with this or I'd have to give up. And of course in projects you never give unless it costs thousands of dollars to find a way to process the tube. Then in that case maybe find a different solution. So luckily as I said at the very beginning YouTube's

your best friend. So once again I went surfing on the web and I found in about 5 minutes how to do that. Great. I didn't credit a specific video here because I used a bunch of different videos um because every system's a little bit different. Every computer's a little different and every debugging process is a little bit different. So, in about 3 hours, I created a connection

through the comp ports. Wonderful. Eureka. Cool. Step one done. Step two, you have to actually interpret what is sent. Yeah, these two completely different things are able to communicate, but you also have to remember that they are in completely different languages. We need something in common between the two communicate with each other and to be able to make an output. So I had to do a little

bit of rethinking for this one. So initially my system through this Jarvis that I have this Marvin was able to understand the h understand the human through talking able to put an artificial intelligence and to be able to respond with something. If you ask it for a joke it's able to respond with a sort of decent joke. It's it's still working on it. It's still learning a

little bit, but I need a way for it to maybe tell the Arduino to laugh or to make a happy face or something to react to what was being said or what the user said. So then I thought, hey, why don't we just process everything twice? Might take a little bit more credits. However, this completely solves the problem. So first you talk to the robot. The robot

process it processes it with artificial says an output like a joke, and then it plugs both conversations back into artificial intelligence and comes out with the tone of what's going on. So in this case, it would be happy. So you get your joke, you get happy, it sends the joke out to a speaker to you and sends the word happy over to Arduino. So now you don't

have to hardcode a bunch of responses and everything's able to adapt in real time. A common language between the two. Wonderful. Problem solved. Now we are at the fun part. Step three, you have to make some responses in the first place. This is where you get creative. So you can figure out what kind of tones you want it to react to. If you want to act confused,

maybe you have one eyebrow going up, one down, maybe eyes looking in different directions. If you want to be angry, you could narrow the eyes, make it look terrifying. If you want to make it happy, you could do so many emotions, you could do so little emotions. It's up to you. However, you'd also have to adjust it in Python to um narrow down the list a little

bit because of course um the OpenAI that I was working off of can give multiple tones and multiple responses to every single message. So you have to kind of narrow it down to choose between this list, happy, angry, sad, or nothing. And then you hardcode in Arduino to react to those specific Cool. Awesome. So now that we have that done, we could go on to the prototypes.

Mark one, what I like to call hands because it is the robotic arm that I said in the beginning was the one I found in the back of my closet for free. Saved me a bunch of time, money, effort, and 3D printing. At this time, I didn't have a 3D printer. So, I just ripped all the wires out, plugged it back in, put in some new code.

Eureka, you have something else now. You now have an arm that can be artificially sentient, which is kind of terrifying, but also super cool at the same time. When it's sad, it's able to bend down a little bit, look at you all sad with its two screw eyes up there. Um, when it's angry, it could look down at you menacingly. When it's happy, it could shake around

or laugh a little bit. It just is amazing to see how much emotion that it can convey with just an arm. No face, no blinking eyes, just a thing going back, forth, side, side. So, of course, with this project, um, there are many different variations and every variation has a pro and a con. In this case, the pro is I saved a lot of money for being

an old project. Wonderful. This is one of my top criteria. It is very easy to maneuver because it is just a side up down robot with some pre-planned code already. And surprisingly, a lot of emotion could already be shown. It's [snorts] a robotic arm. You wouldn't really think that, but surprisingly, I created a very strange connection to it. Like, it was my best friend or something. It's

It's weird, but it's awesome at the same time. However, there was also some big cons to it. Number one was time. There are bigger ambitions that I had for this project. As you see on the far left with some sketches I want to add a little face onto it. However, this is also from when I was seven. I don't have the blueprints. I don't have the dimensions.

I don't have the files. And it would take a lot of effort to try all of those things or measure them myself. So, I wouldn't be able to add a face onto it or modify it in any way. Um, number two, it's faceless. Although it did express being this super cute it also seemed a little bit limited and would only be enhanced if it did have a

face. And number three, it's flimsy. It was built by 7-year-old Jack and he put together some things that couldn't be undone. So, it was super flimsy as a seven-year-old made it. So, after a little bit of research online, I mainly enhanced the good stuff and focused on the face element a little more. And I came to Mark 2 Sauron, which as you see is just a giant

eye. And there are some great things that came with it. Number one, it is a phenomenal phenomenal design by Aquacat on YouTube. I heavily recommend it. He displayed this in the context of cosplay and animatronics. And just that design made it perfect for this project. And it was a super big design, so it was easy to work with. And also it was an eye. That is exactly

what I was looking for. This one eye had probably just as much emotion as the first arm with less components. However, there are some notable flaws with it as well. Although Aquacat did an amazing job. It is a large design and that's not the that's not the designer's fault at all. This is completely my fault. But in replicating it a few times, it would take up a

lot of filament because it is so large. If you mess up something, then it takes up a lot of time and money that could be spent continuing the project. So overall, that was one big tick down for me. It took up a lot of filament, as I said before, and it's only able to move left, right, and blink. I want a robot that can look upish, leftish,

rightish, and be able to convey more of that emotion. So I needed to contain what I already had an eye, something that shows emotion, something facelike, kind of small because that was a big um tick down from the last design. So then, as you see right here with our wonderful design, we have Mark III Marvin. So, I've grown really attached to this design. I do really like

it a lot. Isn't Isn't he just adorable with those creepy eyes and the self-made eyebrows? Well, there are some, of course, pros and cons with this as well as all the other designs. So, number one, the eye design. Why does it look so clean? It's because I didn't make it. This was by Morgan Manley on YouTube and phenomenal design. Couldn't have even thought of anything better myself.

super compact, super awesome. And because of it, it's able to look in all directions. And since it's compact enough, it's able to be put on a full face. Whoa, it looks more humanoid now than all the other two designs. That's exactly what I wanted. And it's because of that it's able to have a wide range of possible emotions that I could give to be able to add

that emotional aspect that I really wanted at the beginning of this project that could um that connection between humans and technology a little bit more. And one thing that I totally underestimated eyebrows. I was not familiar with how expressive eyebrows are until this project. You completely take them for granted. If you cover up the rest of someone's face, you could easily tell what they're thinking or saying

just based off their eyebrows. And it's just a wonderful thing to see on humans coming to this project. So, I created a little design hookup thing kind of like some antlers right there in order to create those eyebrows. Now, of course, everything always has its cons. To me, it's the best design I've done so far and my final design, the one I went with. However, there are

also some flaws that I'd like to mention. And that is how it kind of has a chunky face design. That's my fault. I'm not a designer by any means. I haven't used Fusion 360 in a very long time, and this is the best I could come up with with the time I had in my busy schedule. But it fit and it worked and it kind of looked

like a face. So, it's the best I was able to do. Uh, number two, lots of wires. As you see in the bottom left hand corner, I kind of created like a spaghetti salad braid of hair over there because I don't have the skills to learn how to consolidate that a little bit more. So, I will elaborate more on that a little bit later. And number three,

something I'll also elaborate on is it took a while to calibrate and debug. Yes, it's such a complex design, but it's the very complex design. It gives you exactly what you want, but at the cost of trying to figure out everything in the meantime. So, be careful with this project as well with calibrating the eyes and debugging your code a little bit because it is a lot.

It is very complex, but it is really worth it if you nail it in the end like any project would. So, now that we finished the project, um, let me give some Oh, I actually have a surprise for you that I created from yesterday. a little demonstration of this guy. So, I asked it a few questions and here's some responses that he gave. >> Marvin, tell me

a joke. >> Why don't scientists trust Adams? >> Because they make up everything. >> That was a pretty funny one. Marvin, can you show me your sad face? >> I'm glad you liked it. Here's my sad face. Crying face. >> That was a sad face. Marvin, can you show me an angry face now? >> I'm frustrated that I can't show faces. Why can't I just express my

feelings like everyone else? >> I know, right? So, as you can see, it's a little rudimentary. Not super um advanced, I would say. And also, I apologize for the um hard to hear audio from him. I am not a filmographer. I totally was able to not catch most of the audio, but I got subtitles for you guys. But anyways, as you can see on there, he's able

to show a bunch of emotions if you ask him to um not give super complex responses, but ones just enough to if you're talking to a good friend or something. And very dynamic. you're able to um talk to him and get a response in a minimal amount of time. So, elaborating off of this, you're able to just do basic conversation with him. So, I asked him another

question. Hey, Marvin, can you tell me a little bit about the Southern California Linux Expo? >> The Southern California Linux Expo scale is an annual event that focuses on Linux and open source software. [snorts] It features presentations, workshops, and opportunities for networking among tech enthusiasts, developers, and industry leaders. It's a great platform for learning about the latest in open source technologies [snorts] and community initiatives. >> I

agree, Marvin. I'm really excited. Marvin, I get to go there this weekend. Isn't that cool? >> That sounds awesome. have a great time at the expo. >> So, as you can see, he's able to respond and react in real time to almost any question that you throw at it. And although it's not super advanced or complex, for a beginner like me who has minimal experience in these

robotics, it just is awesome to see it all come together for the first time. Almost like a dream coming true, having a little Jarvis in a little robot that you could have on your tabletop as well. So now that I finished this project, here are some statistics for you. So I spent about 3 months total on this project, which if I didn't have YouTube would spend much

more. YouTube is your best friend. I will always advocate for that. And some software that I used was PyCharm, Arduino, Open Lab, and Bamboo Lab, which for the most part are all free except the API key on OpenAI. for Fusion 360. I left that out as an exception because I used my students license for it, which is super helpful. Saved me a ton of money and let

me be able to have it for free and use this incredible program. So, if you have the opportunity for a student's license, please use it. It's really helpful. And some hardware that I used was my Arduino Uno that I said at the beginning I had a surplus of from my closet. A bamboo A1 printer. Thank you, Mom. I got that for Christmas and a breadboard and collection

of wires. So, not that much hardware used for this guy. So, it made it very cost effective. Um, made it very efficient as well to work with and complex from not super complex items, which is exactly what I was looking for for a beginner like me. Now, continuing off of that, there are some things that in this project really set me back a while. And first one

that I want to mention is take care in hooking up those servos, especially with a complex eye design such as this. One degree off will set you back an hour. And it definitely did for me. I had to recalibrate everything multiple times because I wasn't able to hook it up the right way the first time because it was a little bit of a small design. It made

it a little bit difficult to go in there with my large fingers and move everything around. Which leads me to my second thing and that is make an easier way to troubleshoot. Like if you could just add a latch onto it or an easier way to disconnect or reconnect things, it improves the process so much. You are able to fix things in a snap pretty much instead

of taking hours like I did and just makes it a more enjoyable experience when things do go wrong because that's always a part of the process. Additionally, uh you should probably not cram your wires like I did. There were a multitude of situations of wires becoming undone and it throwing off my robot and I didn't know what was wrong and I had to retrace the spaghetti head

of wires for about half an hour. So that's a big tip when you're a beginner doing robotic stuff. Take care of your wires or find an easier way to connect things together. Additionally, just as a project thing in general, not just this robot right here, this wonderful guy, uh, keep testing, but also know when you're done. There's a certain place in a project where you always plateau.

You always get to the best it'll be, and if you add any more, it'll take things away or just not improve anything. It may become worse if you work on it long enough, and you might become insane if you work on it long enough. and overly obsessed, which are things that hinder progress. So, please just look at you looking at the robot. Just know when you're done,

but of course, keep testing and keep being curious with things you can improve if things are wrong. Don't settle for less than what you bargain for. Next, learn to design. I am not a designer. As you see here, this is a triangle with a smaller triangle on it. Not very advanced and you don't need to be very advanced in design to create something like this. However, I

would recommend taking the time to learn how to design. And also, just in general, take the learn um take the time to learn how to do Python and Arduino as well. I did skip over some Python stuff and looking through it and debugging it did help me learn a little bit. However, it is such a fascinating and complex language. I completely advocate for everyone if they have

the time to do it, just learn it. It's wonderful and you could create insane things with it beyond your wildest dreams. Same to Arduino. It's just amazing to see you plug in a few wires, type in a few commands, and boom, you got a robot come to life. So learn those as well as learning how to design a little bit at least the fundamentals. use all and

any available materials, experience, people, people also count to be able to create a project like this or just any project in general. Elaborating off the previous one, it is a wonderful thing to be able to have things in the first place. It saves you money, saves you time, saves you effort, and just makes everything so much better and so much satisfying. excuse me, um, with taking this

platform, talking about artificial intelligence and having it in this robot, and I am an active user of it, I use it to help learn concepts in school and um, maybe generate ideas for a certain thing, maybe even in the debugging process. I would like to concede that the current state of artificial intelligence is a bit flawed. So I pulled up a few quotes and I did a

little bit of research and the first one I want to point out is that the next wave of generative AI systems will consume vastly more power than expected. This is from Samman, the OpenAI CEO. And if you don't know, OpenAI is kind of the frontr runner of this whole artificial intelligence movement. And this quote was from two years ago. And he was most definitely right. It consumes

vastly more power than expected. And elaborating off of that with the next quote, it's overlooking significant environment undisclosed environmental costs. There's a lot of costs that we don't know about and probably won't know about for a while that should be more open. We need to know what we are gaining and losing at the same time. And additionally looking at these statistics at the bottom, the stuff that

we are able to get, it consumes a lot of electricity in the nation, up to 10% from Oxford. And this is just a terrifying thing to be aware of. So we use in our systems, we also have to look into ourselves a little bit, the ethics of how we're using AI. Are we using it to progress ourselves forward with maybe finding more sustainable alternatives to data centers

or are we using it to go backward and just expanding without thinking about the costs? Um maybe using to sacrifice knowledge by cheating on stuff in school, substituting real art with artificial intelligence art. It's just we should use artificial intelligence as a tool, as a smart friend, a friend that knows everything that can sometimes get things wrong. That is also a human too and we shouldn't rely

on it. We should use it as a beautiful resource in order to reach our full potential and artificial intelligence's full potential. Like Marvin is just the start here of a simplified beginner system. But if we use artificial intelligence to progress ourselves a little bit further, this can turn into an Iron Man suit in the span of 5 years. So please use it ethically and responsibly when you're

using it in projects like this. Thank you so much for listening to my All right, we will although this uh time is up on our schedule, however, if anyone has any questions, we can open this up to a Q&A. But if you need to go now for uh the hour until 1 p.m. for our next talk, say for something like lunch, uh there is that hour, but

for those of you who do want to stay, um you're more than welcome to. >> That was awesome. how you brought up the climate change and the ethics that's reason I see this fact that it's the environment. So >> Okay. I'll be back. Thank you so much. You're on your Just because um just in case anything It wakes up. >> Hey lately. Express. This is the part.

I just need to figure out how to check it out. >> Thank you guys so much. All of this is so I would say Yeah, I'll probably speaker all I would like it to be a little bit more. So, I'd like express. One thing that was my original because it will be able to do more with Thank you. Russ. I do not know. I don't know. everything

else. That's why I don't think Wow. Well, this mic is working at the very least. testing. There we go. Okay. I'm looking at your bag at the table. battery. I'm not sure how much Hey, sorry. We got started. I hope you don't mind. [laughter] >> Uh, no. Whatever you prefer. >> Hello. All right. There we go. >> Introduce yourself real quick. >> Hi, I'm Connor Karos. >>

Yeah. All right. Let's go. >> Great. Sure. Let's take out the back. This top switcher. down maybe. All right. Good afternoon everyone. Welcome to Scale X 2020 uh number 23. Good afternoon to you too. All right. Um, so today we've got Connor Kelly Deracos on developing open source for is >> All right. Hello. >> Did something get bumped? Sorry about this. Something happened. >> I tried it.

Yeah. problem resolved. Okay, this is developing an open source LED controller for first robotics competition. So what is F FRC? F FRC is the uh first robotics competition. It's a international high school robotics competition. uh students design, build, and compete with a robot. For our purposes, it's a place a need for a new discrete LED controller LEDs make your robot look cool, which is like half the

fun. And uh allow your for feedback to the driver. So if the driver does something with the robot, they know what the robot's doing if it's doing it, if it's working. Um we need something that requires minimal communications uh bandwidth and minimal code running on the robo, which is the main processor for the robot. I'm sure you can understand why we'd want minimal code. If we're taking

up processing time from important tasks for LEDs, that's not great. Um, and preferably it would be open source and affordable with reconfigurable firmware if you want your own patterns. Who knows? So, prior LED solutions were controlling off of the digital IO port, um, which takes up a lot of processing time and isn't great, and you have to wire a DIO, which is annoying. Um then there was

the candle which is communicates over CAN and that was not cheap $60. Um closed source and people didn't really love the firmware on that. Um then there was a connector X which is newer. Uh $50 closed source big and it can't use cans. You have to wire that separately. Okay. Why a PCB? Why not just an Arduino? So there's a couple ways you can do things with

Arduinos. You could just stick some wires in it and some breaker boards, but those are prone to pulling out. It's really big and you have to solder the It's just not great. You can also put it on a PF board, which is like a a prototyping board. You can break it apart, you can cut it. Um, but that's even though it's quick to build, it's still not

the easiest to solder and it's still pretty big. A lot of your space is taken up by things you don't need taking up space. Um, then you can just mount an Arduino on a printed circuit board. Um, this is space inefficient because you don't need all the parts on the Arduino. And generally it looks something like that. Lots of space. You just don't need taken up. So

we need to choose a microprocessor. A microprocessor is just a processor that doesn't run an operating system. Um, and we need it to control LED strip. They're addressable LEDs. You can't just make them do things. You have to send them information. And there's a couple options available for this. Um, down there you have like the STM32, the AT Mega 328P, HTM 32U4, and the uh Raspberry Pi.

I forget the name, but all of these are options available to use as a microcontroller. Um, so USB, this is a important consideration when you're choosing a microcontroller because you can't just stick USB into a microprocessor that doesn't have it. So you need to choose a microprocessor. So it has a number of You need a microcontroller in there that natively supports it. My favorite example would be

a U series AVR, an AT mega 324. Um, and you need extra hardware to support this. There's also some like limitations on where you can put your USB traces, what you can do with it because it's kind of sensitive. um and signal integrity. Yeah, worry. Um so I abandoned that idea in prototype. So I landed on 18 mega 328p which is the the core of the Arduino

Uno. There's infinite code out there for it so that's easy. Um you can program it through Arduino Studio which is nice. Um it has a throughhole package option. This was the biggest consideration. Through hole is easier to replace. You don't need to surface mount solder which is annoying. it's better for open source because it's easier to work with. Um, and you don't need any external IC's for

the microp processor itself. You don't need every all the bits of it are inside the microprocessor. Um, the biggest drawbacks are no USB support. Arduino Unos have a second microprocessor to do that. It's not particularly fast, nor is it powerful. 8 bit data bus, 16 bit address bus. It's not there's not much on there for code. Um, and it needs a external crystal oscillator to go full

speed which can be can take up space. And I placed 1.1 and 10 microfarad decoupling capacitors nearby it just to filter out different types of interference. Um, so this is the port I landed on instead of USB. It's the AVR ISP port. It's kind of rare. Um, but it's AVR architecture specific. It's smaller. You don't need any extra hardware, but you need like really annoying drivers. The

drivers suck. And it's really slow. Um, it fails like half the time you try and program with it, but it's like nice and small, so that's cool. Um, differences from Uno. You can see my beautiful drawing of what you don't need on the Uno is in red, and the only things you need are not in red. Um, so you really don't need most of the stuff on

the UNO. [snorts] You don't need USB. You don't need all the headers and yeah, you you lose a lot of the bulkiness of it. And the minimal setup is outlined in documentation. It's pretty much a crystal, something to reset it and the um filtering caps for the crystal. Um, so connectors, another important consideration, especially for FRC. You have a ton of moving parts in FRC. If you

have a bad connector and something falls out in the middle of a match, you can't just fix it. It's a match. Um, so the initial PCB had through holes that were of the wrong diameter. So that was bad. And you don't really want through holes. It requires you to solder. You can't really replace them if they break or anything. Um, so I went with WGO. WGO are

pretty popular nowadays. They're little wire nuts and they have their own throughhole PCB mounted version. So I chose those. Um, they're a little big. like a quarter of the board is them, but they're quite strong. Nothing's going to pull out of them. Other options would be those. They're called wide mullers, and they're really common after, but things fall out of them. And then there's screw terminals, but

you have to screw wires in, which is just slow. Okay. CAN. I've been using this acronym a lot. Uh, it's kind of a rare communication protocol. It's called control controller automotive network, I'm pretty sure. Um it's uses a differential pair signaling uh method. So if you look down here, you can see what a differential pair is. Any interference is going to because they're tw the two wires

are twisted together. Any interference is going to affect both wires equally and then you just invert your low. You invert one of your two wires, which is you pre you invert it before and you invert it after. And your inversion and then averaging removes the uh interference. So that's that makes it extremely robust. It's why it's it's in your car. So, the reason it's in your car

is because it's so robust when it comes to interference. Um, it's the most common in FRC and devices are daisy chain. So, you need two ports. You devices have CAN in, CAN out, and any signal goes to everything on the CAN Um, so CAN is not super common. Generally, people buying CAN chips are car manufacturers. So [snorts] the most common offering for the specification we use in

F FRC, the 2.0B is the 2515, which is just microchip products 2515. It's just a chip. Um it's pretty old and you need a CAN controller connected to it. And then the faster, newer protocol is the requires a different chip called the 2517. Um but that's only available in surface mount. So again, bad for open sourcing if it's hard to make for other people. Um, so I

landed on the 2515 and the 2561. The numbers don't matter. I just landed on those. Cheap compatible. Um, F FRC has its own idea of what you should be doing with the CAN bus. So they take the extended ID. Every CAN packet has an ID and then the data. They take the the ID and they break it up into chunks down here. Most of these chunks don't

matter. Um, the device number is unique for every single one, no matter what the first bunch of bits and so you presumably you could just filter by device number and you know the main processor was sending you data by just filtering by that. You know, you could just look for your device number and you know unfortunately they decided to not do that. They put a very important

packet called heartbeat um and they gave it an ID of zero and literally everything has an ID of zero that's important. So you if you try and filter by that you're going to get a bunch of junk. So it also requires just tangential requires you to shut down. So another capability our thing needs. Um so the library I use has masks and filters. This is a good

representation of it. A mask is going to tell you what bits matter. A filter is going to tell you what value those bits should have. And if it doesn't meet one of those two requirements, it discards it, which keeps it fast. You aren't filling up registers with things you don't care about. Um so again mass designate which bit matter specify which values of the bit matter. Um

take some testing because can is weird. Sometimes there's just weird stuff. Sometimes your masks don't work. Um [snorts] and then you also have to make it shut down which required some messing around with the mass and filters because that's over the heartbeat and that has an ID of zero and we really don't want anything that isn't important in our registers. So, um, power supply, uh, F FRC

uses motorcycle batteries. That is a motorcycle battery. Um, 12vt everything. Uh, there is a module that converts it down to 5 volts, but that requires extra wiring and it takes up space that really isn't necessary. Um, and then it would be nice if on our if on my thing there was reverse polarity protection because like what if there's a short, what if we mess something up? Best

if there if there's protection for Um quick explanation of capacitors and inductors. I'm sure some of you already know this. Um capacitors pass changes of voltage and block constant voltages. Inductors pass constant voltages and block changes. Um which matters because we need to filter our 12vt in. So capacitors we have capacitors between 12 volt and grounds to pass any changes and then inductor place in series to

make sure no changes get through. Um because we want a stable power source. Uh inductors use magnetic fields. Not great for electromagnetic interference because it's a magnetic field. And I made a mistake. You're supposed to use a choke which has a metal core and not just any inductor. And I used a generic inductor. That's not great. Anyway, you can see the power uh power supply filtering there.

Pretty simple. [snorts] So the power supply IC was there's plenty of options here because a lot of people need to take 12 volts and turn it into 5 volts. Preferably, I would prefer if I had something that was small and gave me 5 volts, one amp. So, I came upon this. Um, it needs some tolerance. Uh, F FRC uses motorcycle batteries, which means your power supply will

droop if you are um if you have high load current, especially when you're moving the motors. It's it can get all the way down to 7 volts without browning out. So, it needs to be able to handle that. Um, and it's frequently over 13 volts at full charge. So you you can't have something that only wants 12 volts. Um so this one seemed like a good idea.

Small, high efficiency, it could take 7 to 36 volts and it can have a common ground. 5 volt and 12vt can have a common ground which makes it just easier to design. I don't need to isolate my two grounds. Um and then I connected it through a shock key. A shock diode is just a diode that doesn't have great it doesn't have a ton of forward voltage

drop. It uh so good for protection. Um, the issue is that it likes to get hot, like really hot, and it has a plastic case, so you can't like put a fan on it. So, it doesn't seem to like melt the board, and it doesn't seem to melt itself, but like it gets really hot, and that's not it's not good. Um, and it could be harmful. The

solutions aren't great because it it has a plastic case. You aren't going to like put a heat sink on a plastic case. Um, so layers. This is another important design consideration when you're designing something with a PCB. You have to decide how many layers you're going to have pretty early on. Um so you have layers. You have uh copper is your conductor. So you have copper and

then plastic and then copper and then plastic. Um [snorts] and you can generally the two options are two layer four layer. Um four layer is more expensive. It just costs more to make. Um but two layer is harder to route because you only have two layers to route on. So if two things cross or three things cross, you have to figure that out without like you can't

just put it on a layer below it. Um and then also you can have a separate uh power ground plane that's really good for signal integrity because it's just bigger. You don't have uh interference is just reduced um and the the loop inductance is reduced as well which is very nice. Um and you can separate your data lines. It's nice. Um so this is the big advantage

ground play. Um, one whole layer can be ground uh on four layer. Generally, you're not going to do that on two layer. Um, you'll see why in a second. Uh, and it improves your signal integrity and it takes full advantage of your copper pore. You get a amount of copper and whatever copper you don't use is just you paid for it, but you don't get it. Um,

and in the first iteration, it was just a trace. It just ran throughout the board. It wasn't great. Uh, that was pretty bad for EMI. Um, in the vision, I made the second layer of the design mostly ground plane, which was bad. That was a bad idea. Um, gaps in the ground plane like are really bad. They cause a ton of loop inductance, which means it acts

like an well, it has inductance. It's active. It resists changes in current, which is horrible for, um, horrible for signals, uh, especially high-speed signals, which I ran right over one, which means even worse. And then really big gaps like the one right there. Like everything that's not blue is a gap. Like the blue the the black parts are gaps. That really big long one at the bottom

is acting like a massive antenna which is super just horrible idea. Horrible idea. Um powering the LED strip. Uh pretty simple. It's just it just wants 5 volts. It the data sheets do call for a th00and microfarad capacitor between power and ground. You're right. But that's pretty simple. You just put it on there. Um, signal integrity isn't bad. Digital is pretty good uh for that. You know,

only two spots it can be. So, you need a full 5 volts of interference to really do anything. Um, and you get a bunch of libraries for it, so nothing to worry about there. Um, crystal oscillators, pretty common component. Um, they just generate a sine wave with some feedback. Um, too necessary. Um, this is what I routed the traces for this over the ground plane. That's why

it's so bad. It's 16 mehz. It changes. Yeah, it's not and I just need to keep them away from 12 volt. Any big interference on these can really mess up the micro uh Extra features I did. Um, you can so for terminating CAN, you need to bridge the two CAN high and CAN low with a 120 ohm resistor. So, you can do that by just soldering in

a header and popping a you call I forget what you call it little shorting thing on there. Um, and you can solder uh you can you have an I squared C port. So, if you want to put something else on there if you want to have more things on your robot, you can do that. Um, and there's a solderable jumper to standby on the 2561 to ground.

The biggest issue on the first run through of PCBs was that I forgot to connect standby, so it just didn't work. Um, so now now you can just short it to ground if you really want. Um, there's a schematic. Uh, not super informative. You can see the chips I'm using and you can see the um over there on the left you can see the power the filtering

for every chip but generally you're not going to derive much from that. There's the full layout of the second of the revision. Um pretty dense. There's a bit of wasted space but that's mostly because of the size of the connectors. Um [snorts] nothing terribly interesting. Of course you can see the ground plane gaps which isn't good. Um, there's the first PCB. It's really space inefficient. I didn't

compress anything down. I just let it it it's pretty sprawling. Um, and you can see that I didn't solder a ton on there. This is like minimal setup. Um, to get it working. Uh, unfortunately, it didn't work, so I went with my second revision. Um, there's the unassembled and there's the mostly assembled. I left out the extras because we didn't need them. Uh, pretty good. Um, you

can see that the WGO take up most of the board, which isn't great. And there's a pen for scale. It's reduced in size by a couple millimeters. [snorts] Um, shortcomings. LED strips turn out to really like current. So, I should have gone with a 2 amp converter, not a 1 amp one. Uh, that was really bad for the amount of LEDs you could use. Um, the board

is overall just really big compared to in particular the Canled, the other the more expensive alternative and that's because of the Legos. If I didn't have the Legos on there, that'd be nice. Um, mounting holes would be useful. You need to mount things in F FRC. You can't just let it dangle. So, no mounting holes is kind of annoying. You have to like tape it down. And

then the ground plane gaps. I should have gotten rid of those and just use a four-layer board. Um, pricing was fine. So PCB manufacturing was $320. JLCPCB gives you like a ton of discounts. You don't really need to pay that much for those. But the most expensive components with WOOS that was $25. They were half the entire price. That was really bad. Like connectors aren't even that

important. And I let them inflate to that big of a Yeah, that was bad. Um, not horrible. It's priced similarly to the connector X, which is our cheapest competitor or competitor uh commercial option. and um yeah, check your component prices before you design the board. upgrades the the 2515 and 2551 are old. It's like really old. They're like 2000s old. Um and you can combine them into

one chip. It's called like the 25651, which is just smaller. And then you can also um the new the alternative to the new uh central processor for F FRC. Well, not the alternative, the new essential processor for FC just uses natively the faster CAN protocol. So, you need to support that to get full speed. So, that has to be swapped out with these newer chips. Um, so

that's in like two years. I'll have to if I want to use this board, I have to swap those. connectors are huge. I should I should have replaced them with something more common. Um, and I could move from through hole to service mount. Through hole is really big. It takes up a lot of space. Most of the chip is just empty. Um, so that's kind of pointless.

I probably should have gone with surface mount, but again, it's harder to solder, harder, a little bit harder to design for sometimes. And the 328P isn't super powerful. If you want to do your if you want your LEDs to do something weird, if you want them to have something cool on them, the 328P isn't necessarily going to be capable of doing that. Um, so I could upgrade

to something stronger to use more complex LED patterns. Uh conclusion, all files are available on GitHub um on that link. Um project was overall a success. It controlled LEDs, which is all we really needed it to do. Um it can control like 70 or something, which is probably more than enough for any robot. Uh unless you like drape them on it. Uh we plan to use it

on our competition robot this year as team 244. And kind of going to get on my soap box here. F FRC like teams really like to just buy like they will buy anything. If you put it on sale, they will buy it and it's all closed source. So all of the software for FRC is free and open source, but all the hardware is closed source and expensive.

FRC could really use some electronics design instead of just wiring. Most of FRC electronics you just wire things together, which isn't great for high schoolers because that's not applicable. there's plenty of people who can wire. Um, and hopefully this inspires other teams to design their own custom circuits to fulfill problems uh instead of just buying expensive alternatives. All right, thank you. >> [applause] >> All right, Q&A.

We have >> All right, so stick around for a Q&A. Make sure to just stay in your seat and raise your hand. And we will be passing out the microphone so that Connor here can answer your question. But uh we do still have plenty of time in this block So let's just uh bring up the first person. >> Yeah, great talk. Thank you. Quick question. Why did

you need um external chip to do the CAN interface? Couldn't you do like a software bit banging or something like that on your AT mega because you don't need a very fast communication? I'm guessing you know to set your LEDs. >> Do are you say are you asking like why didn't why do I need chips to interface with CAN? >> Right. Right. Couldn't you use some of

the IO pins on your ATmega and do some bit banging software solution to emulate the CAN? >> You theoretically could. Generally, the speeds are too high. It's 1 MHz and the problem is you have to you're getting so many messages you have to store them somewhere. So that would pretty quickly fill up any storage on like the 8 mega 328P doesn't have that much storage for that.

Um, and also it's a differential pair. So you need something to do the differential pair thing or else you lose the main advantage of CAN. Um you need something to invert and then to average the two voltages. Uh which is relatively difficult to design. I suppose I could have designed uh something to do that like in like with like diodes but it would be generally bulky and

it would be kind of annoying to build for other teams. Generally you want an external chip for can. >> Yeah. Any other questions >> over here? >> 24. >> Are you still >> Oh, yeah. I'm I'm a junior. I'm in 24. I'm Yeah. >> Okay. Once again, you're still with 2404. You said you're a junior. Um, surface mount soldering could save you a lot. Um, I it

it seems complicated. uh there's organizations that do uh that that sell some small fairly cheap like learn to solder kits that's that that are specific for uh learning to solder surface mount. Uh bonus 2404 is local, right? like near Pasadena or >> Yeah, we're in Pasadena. >> Okay. Uh don't know if you're aware, there's another conference called Layer One that's held down the street at the Hilton.

There's always a hardware hacking village. There are always people there that are happy to teach you how to solder surface mount. If you want to move forward with the design that if if you got the time, it's usually over Memorial Day weekend. Um it's an opportunity to learn and and and maybe rev the design if you feel like it. Um, but the uh the question I had

is cons did you consider instead of using the expensive real estate gobling uh wagos on the board uh why not do a pigtail with maybe a cheaper connector uh just molexes? >> I did consider it. Um I wanted So wiggos are like the new thing in FRC, >> right? They're they're putting them everywhere. And I was a little worried about using something a little weaker because this

is probably because it's discreet. The whole point is you can put it somewhere else. This is probably going somewhere like we're going to have LEDs on the arm. This is going near the arm. I was worried about it pulling out. Of course, in hindsight, that was a dumb idea. I should have just gone with a molex. Um definitely a design regret is using wiggos. Price and size

were just not worth it. But they are good connectors in every other regard. It's just Yeah. All right. Thank you for the advice, sir. >> Questions? No question is stupid if that's what you're thinking in your head right now. Got any final comments, Connor? >> No. Thank you for coming. >> All right. Thank you for coming. uh return at two o'clock for our next There you go.

Very well done. >> Very good. Are you just test test did that work? >> Okay. Oh, wait. >> Does this >> Just give me like a testing one, two. >> Testing one, two, three. Talk a little bit louder. >> Testing one, two, three. All right. 25 minutes. I don't know what I feel like also. I'm not sure. I don't know why. That's what I was thinking. I'll

just like migrate over here and So, it's pronounced sometimes pronunciation is bad and I kind of sucking. We're gonna try this >> is it? Another All right, good afternoon everyone. Welcome back to Scale X23 in room 103 with scales um the next generation. Today we have uh Koshan Dwaldi on creating an operating system from scratch. be sure to answer to uh ask any questions and we will

have a time at the end where you'll all send your seats and then we will pass around a microphone to answer questions. And if you don't have um and if uh if you're worried about like missing anything any part of the uh presentation, we do have a live stream right up here. So on YouTube by like tomorrow or the next couple of days, we will have that

full uh live stream of this room including this presentation. So you can take actual screenshots. >> All right. Uh passing it on. >> Hello, my name is Koshan Deli. It's my talk on creating operating system from scratch. Um first I'm going to start with a resource that's really useful. It's the stuff of wiki. They just like tell you a lot of stuff that you would normally need

to get from like annoying manuals that have also like harder implementations. But yeah. Um so what is a kernel? uh the main purpose is to manage access to hardware and then user space programs which is like what if you make like a C program and compile it that's what it would be on Linux a user space program um they interact with the kernel through system calls just

one of my first goals with problem with this project was a file system but first I did I need to do like basic initialization for x86 so yeah um most important thing maybe for initialization is the boot loader the main purpose is to jump to um 32-bit code um and you go from 16 bit code. You can create your own bootloadader, but it adds a lot of

complexity, so I didn't want to do it. You have to make like BIOS calls, which the documentation is kind of annoying for that. And it like 16- bit assembly. Also, you get like memory map devices from the bootloader, which is nice. Um, they're kind of annoying to get otherwise. Um, another thing I did really really early on was just printing to the screen. You just write directly

to memory location and then it like makes that character show up on the screen. It's like the VJ text buffer that's really old. Um, it's like also color and like the second bite and you like go two bytes over and then the next character. Um, initially I didn't have debugging which probably a mistake. It's pretty simple with QMU. You just connect to like the GDB server and

then it kind of works. Um, for one your space executable right now you can also implement in the kernel but it's a lot more annoying and hard to do early on. Um yeah the next important thing um was paging you. It's the main method that's used for memory manage management on x86 in like modern times. Um you the address space is split into pages um which are

4,96 bytes and pages are managed with a page directory which contains page tables which contains pages um and each the page directory has24 page tables and the page tables have 124 um pages each. So and all in all you get like 4 gigabytes of memory. Um like the first page directory and the first page table will map like zero to like the first two megabytes. the advantage

pagings you can swap page you can swap between page directories. So um then it would mean that you can have other things mapped in the memory. So that's really nice for user base processes because you want them to be at different addresses so they don't they can't like overwrite their each other's data. Um stuff like that. I also um changed my like I initially had a bit

map that marked if a page was free or available um for page allocation that's pretty slow and you also run out of space really easily because you can't have enough space for the entire address space. Um so I changed to a binary tree. Um each node in the tree contains a list of three pages. Um and then the value of the node is how many contiguous pages

are in that list. and then you just pop a value from the list and then that's the page. Yeah. Um it's a lot faster. Um it's more difficult to initialize because you can't allocate memory normally for it from like the normal allocator because it would use it. So I just like went back to a bit map allocator for some of it. Um another thing you need early

on is interrupts. It's done with a table of basically pointers to functions. And there's also hardware interrupts for devices like the keyboard um like the mouse stuff like that just like a timer. Um interrupts are necessary or necessary um kind of for devices because um if you can't just like pull from them that would be really slow. It uses all the CPU time. Yeah. Um, also another

thing they're used for fault handlers because before this you if you don't have fault handlers the CPU would just reset whenever there's an issue if you like access invalid memory stuff like that. So this will make it like print something and then like pause CPU. Yeah. Next thing I did was for the file systems was ATIPO mode since like parallel IO. I don't know why it's kind

of a weird acronym. Um, but you just read from the IO bus and like you give the disc a like on the disc and then read it. Um, here's like old hard drives. I had a picture of old hard drive. Um, it took me a while to implement because there's a lot of weird timings for things and you have to wait for like weird bits to be

set. Um, but yeah, I did have a bug with it. It's like one of the first bugs that took me a while early on. It was pretty simple. I just didn't cast. I casted before doing the bit shift. So everything would become zero in like the like sec on the left side the um second row and beyond everything become zero. Um you can't read at sectors higher

than like 256. Then um I checked all the other code and I forgot about this. That was kind of why it took me so long. But yeah um I then started doing file systems. I chose EX2 um just because it was simple to make on Linux and like there's a lot of good information about it. You can also like make your own if you want, but again

it's more complexity and you have to like allocate the structures yourself which you can get like a program to do that for you on Linux with X2. Um there's like a picture of the how like data is stored for files. Um there's direct blocks which are stored with the file and then there's another block which contains more like pointers kind of to blocks and yeah so you

can get a lot of file space really easily with that. Also made like a API for file file system abstraction. Um it's pretty sim similar to the Linux kernel. Um I like took some of the names but you just like store function pointers to the um like operations that you want to have for a file system. So ext um function and then you would like and then

you have a super block which represents the mountain file system and the like VFS or virtual file system codes which I called it which is what Linux calls it cuz I took the name from them. Um uh we'll call that and then yeah it eventually like each superb block has its own like file system type and then they're like statically initialize and you can get the compiler

you can get the compiler to optimize the like pointer d reference when you like do the like function pointer calls because like it's technically like one memory access slower but I haven't done that yet. It's kind of annoying. Um I did initially didn't didn't account for a lot of things in that API like reading from directories and like making directories I didn't have. Um so I initially

had like this function on the left which would return all the entries in a directory as like a heat pointer to more heat pointers which is pretty bad um with a lot of like wasted memory. Um, I then changed it to just like increment the offset that I can offset in I guess str stored separately every time. Um, and it only return one. This is really nice

because like when you're looking at a path name, you you want to stop after a certain point. You don't want to just allocate space for all of them, all the entries. So yeah, also put a cache on directory entries. Um, I didn't really need it for performance, but it was nice because I can also store like device files and stuff that's supposed to be there permanently and

not get removed from the cache. Um, it permanently stores my points and devices. It's an LRU cache. Um, I didn't really like implement one before. So, I kind of want to do that as well. Um, there's an issue with it. Um, it contains some of the entries for a directory, but all of them are created in the file system. I like spent some time realizing this. Uh

you it's kind of difficult to read from a from the cache and read all the entries in a directory at the same time. So I just fall back to the file system if you're trying to read from a directory. So the cach is only used if you're like open the sys call. Um like that'll use it but like not read dirt. Um yeah. Next thing I wanted

to do was user space. Um which is kind of important for a kernel because it's like the point kind of. Um, so I initially had this I just like memc copied a function into like some user space address. Uh, this is pretty bad because you don't know the size of the function in C. So you can run a space if the function is too large. You also

can't have two functions because you don't know where to like put the next one. Um, and it's kind of hard to call them because all the addresses are kernel addresses. Um, I this did let me test like my Cisco call handler and like the jumping to user mode code, but I I then made executables. Um, I use the L file format because it's really easy to compile

to on Linux. Like everything uses it. Um, you don't need to do like anything with sections if you're familiar with ELF. Like sections are kind of annoying. So there's just program headers which give you like there's like two of them and like the ones that I got uh like the ones that GC gave me. Um, and then it's just like an address and like if it needs

to be read only or not. And they're really simple. Um, yeah, like I did with system calls. Um, you just like usually code just like runs an interrupt software interrupt and it'll jump to the kernel code for the Cisco handler and then it'll like read the state of the registers and do whatever. It's pretty sim similar to Linux. I could have used the Cisco instruction because I'm

on x86. I decided not to because it'd be more complicated and it's not really that much of a performance increase for like what I need. I also want to make make auler after this because I want to run mult multiple things at once because now I have like a good kind of user space implementation. Umuler is just called when a timer interrupt occurs and I just use

a circular link this Q of tasks. I don't really need like priority or anything because that's pretty simple for what I have right now with no tasks that need a priority. makes the kernel like more interactive. You can like if you move the mouse which I'll get to later. Um you can like see it better if you increase the speed of the timer but then there's more

race conditions. So yeah um initially I implemented context switch like this. Context switch is just like going from one the like context one task to another one. So like um like loading a new address base stuff like that. I just had it like um if it wasn't user mode just like call the jump to user mode code and then if it wasn't just go to the kernel.

Um this is kind of annoying because the jump user mode code like isn't it'll like change the reg register state which you don't really want. So I just had to do all all of it in the kernel. Um the way it works is if you save the registers and then that saves a stack pointer and then when you go to the load registers function it'll build a

new stack pointer and then wherever the stack pointer was saved it'll return to. So yeah, um yeah, and then like swap tasks just swap to the next task. Also, it's leaping processes. Um I it's kind of similar data structure to the Linux kernel. I think I didn't really like Yeah. Um [snorts] but you just decrement the head of the queue, the value, the number of seconds it

has to sleep for. And then the next one's just based on that. So you so you don't have to go through the list and decrement every value. It's a lot simpler. Um, and then once the head reaches zero, you remove it from the list and put it back on the scheduleuler queue and then it works. Um, yeah, also want to do x8664, which is like the 64-bit

version of x86. Um, it's pretty similar. You could reuse some of the like drivers for like the discs and like keyboard and stuff. You have to change like paging and interrupts and a lot of the assembly you have to change unfortunately because they have like like 64-bit registers instead. I kind of want to do this because I wanted to work with multiple architectures. I want to support

ARM at some point and it was like a nice kind of simpler way of getting into having different architectures. You have to change the build system a little bit. Also like one of the first times I merged a large amount of code. So that's kind of cool. Um I did have a pretty it was really it was really simple bug. Um but it took me a while.

I didn't realize in the strct for like the task um that I forgot the packed attribute which will make it just um all like contiguous in memory instead of having there's be padding added by the compiler's like 60 bytes or something. So the compiler like wants to optimize it but that makes the CPU not understand it. So yeah. Um, next thing I wanted to do was like

window management and graphics. So first thing you need to do is just get like a frame buffer from the bootloader. Um, those it's pretty simple like grub. You could get a pointer to the frame buffer. There are alternatives to this. You can switch to 16- bit mode. Um, but that's pretty difficult because you have a 16- bit address space and it's a it's a lot harder to

manage. I also don't think you can do it in 64-bit mode. Um, I'd have really checked that, but it's more difficult. Um, you can also like write a GPU driver, but that's awful. Um, [snorts] uh, next thing I did was a window manager. I needed a lot of interprocess communication for that. So, the two main things I did were Unix domain sockets, which are kind of like

normal sockets, but they're like on the local machine between processes, and shared memory. Also, I needed like user based threads, but those are pretty simple. Um, I also just want to improve in general like the user face API before doing this because it was kind of bad initially. Um, I just I put like I know why it says block device. I should change that. But um I

put like devices in the file system like Linux. It's kind of like slashdev if you're familiar with that on Linux. Um, and I just made like keyboard and mouse device files. They're important for the window manager of course. Also need shared memory. Um, for the window manager, you want to like map the contents of the client's window, the client to the window manager window into the like

server's address base. So that's kind of important. Um, you share memory for that. Uh, you just like have the same physical address for some region of memory and then the virtual address like and have it map it to a different virtual address in the clients in both processes. Um, yeah. So, Newix domain sockets, they're pretty much just like TCP sockets if you're familiar with them and see.

It's like the same API. I guess if I wanted to do TCP sockets, I could maybe a little bit easier. Um, but yeah, uh I wasn't really familiar with the API, so I didn't have like I I still don't have like connect or accept like those um because I don't really need them. I only have like one sock that you really need to connect to. So I

had a temporary fix for that which was just sending a pointer um to the socket or sending like the socket number to the um to the clients from the server that the client will then use for the connection. Um yeah I didn't have accept connect. Um the server like is always zero because it's the first socket created. I should I'm eventually going to try to fix this,

but right now I should have like this because it's a little bit simpler. It's also a bug with sockets. It wasn't really a bug. But um I looked I had in the client code I sent a like request to render with by sending like the render code. It's like a magic number basically to the client and then or to the server sorry. And then the problem with

that was I was immediately sending requests to get a key afterwards which wouldn't work. The sockets like have an internal buffer which I think was like last and first out at the time but I need to change the issue with that was that it would like request a key when it shouldn't have. Um the cuz sockets normally like send will not block. I I thought send didn't

block but it does. Um but yeah I the way I fix I just like made the client receive uh like acknowledgement thing. Uh yeah I also had user user space threads which I didn't have before. They're pretty simple. You just like copy the down space the socket implementation blocks. I said um when you're reading from a client you need to like make a new thread in the

server. Uh I could also made like asynchronous sockets but I don't I'm not too familiar with the asynchronous API for C for like a lot of sys calls so I didn't do that. Um I also have window like compositing that was the next thing I kind of did. Um it's just like the process of like making the windows store in like a stack like draw like a

stack. Um right now I just have them like as uh like it's like make the first window draw on top of each other like stuff like that. Um I'm like storing the windows in a stack. Uh the window on the top of stack is drawn the next one then the one after that. It the problem I had that initially um that's pretty inefficient because you draw every

window every time you want to rerender. I later made it so that like you only rerender like part of the window where you need it to. So I have like a video of that. I don't know if it's going to work. Okay. I don't I don't think it worked. I don't know how I I'm just gonna not Okay. Um, also there's I have I made a terminal

process cuz just so I can like run stuff from the window manager. Um, it was like the first actual kind of program I made for the window manager. Um I made like the pipes call which is like what real terminals use to like copy standard in um of the like processes that the terminal spawns into the like like of the shell into the terminals like window. Um

I also had to make like child processes inherit file descriptors the like file descriptors created by the pipe would not be inherited and they print like the normal like kernel debug standard out. So yeah. Um, some other things I did, I made like a unit test framework. I like kind of based it off what like Linux does for getting the data back to like my the host

computer from the emulator. I made like in memory block device for testing. I like ran it. I kind of tried to run it on real hardware. Um, it didn't work. I think the like grub like memory map thing I'm not doing it correctly and like I need to like zero some like paging structures. But yeah. Uh thank uh I'm going to go to a question and answer.

Uh yeah, >> thanks for a great talk. Um is [clears throat] your code online? >> No, I unfortunately forgot. Yeah, sorry. >> Are are you going to share it though? >> I might. Yeah, I we'll try. >> Follow-up question. Um, can you talk a little bit about the multitasking? Are you using test state segments? >> are you using task state segments? DSS is for >> um I

use like just I don't use like hardware task switching. I'm using software because I heard it was really bad. But yeah, >> thanks. >> I So how do you different why why would you make your own OS? Like did you look at some core like Red Hat, Debian, and then you said I can do something quicker based on my hardware? >> I just kind of wanted to

to learn. I'm not like gonna I'm not like planning to make this like a >> improve on or >> I mean I'm going to improve on it, but I'm not really trying to like make this something for people to use normally. >> Challenge learning. >> Yeah. Yeah. x86 hardware is not always fun to learn to deal with. Um Linux obviously and other you know BSD great having

open source operating systems out there to refer to uh when you want to you know see how they figured it out. Can you talk about what kind of books documentation other oss and stuff you looked at? Obviously Linux but what >> So there's like the west of wiki which I think I talked about at the start um which is a really great resource. it like it summarizes

a lot of the like Intel manuals and stuff that like puts like tables that are really nice um that it's annoying to find information for it sometimes. Also, just like looking at the Linux source code or like googling how does Linux do this can be really useful. Yeah, >> I have a dumb question. Is that assembler or assembly? Most of like strruct and asterisk asterisk asterisk. >>

Um a lot of it was C. Like the strruct stuff was C. Yeah, I did have some >> If there's an asterisk, is that multiply or is it code? That's >> Those are those are pointers. >> Any more questions? Any more questions? >> Okay. I could bug you all day because I'm fascinated. Um, >> okay. I got to keep I got to keep you busy with questions.

Um, can you tell us a little bit about your background and how you got into C programming and wanting to make your own kernel and >> and also sorry follow up also maybe you can talk about are you done with this or are you you want to build a shell and a whole tool chain and all that. I'm definitely going to like continue with it. But I

guess I kind of started I made like a memory allocator. Um just like trying to learn C. Uh and then I eventually I want to make an operating system to like do more more of the memory management stuff. But I also just kind of like doing it. So yeah. >> Oh yeah. You don't have to apologize for >> You don't owe anyone an explanation. >> Nice work.

>> Thank you. other questions? >> Did you use any Python to program this? Um, it's kind of hard to get Python running on a like lower level thing. >> Any other questions? Uh, just a note that uh we are live streaming this. it it will be on our Southern California Linux Expo YouTube channel tomorrow or the second day. Give us a couple months to actually give them

the um to clean it up, but thanks. This was Yeah, really great. Do I just am wondering from because you know you're obviously just getting not I mean not just getting started but you're kind of young in the in the the like scale of a lot of people that are [laughter] at this event. Um do you how do you see other folks like your age group kind

of getting involved in this type of work and and what do you think would help them? Because I think the fact that you wanted to do this and decided to do it because it interests you is amazing and awesome and um but it'd be great to have like a lot of people doing that. What I mean what what do you think would inspire other folks like your

age and also from like what what of this did you get at school or did that help anything from like formal education help this or is this all self kind of selftaught? >> It was all I guess selftaught. There's there's nothing really at school. I'm not really sure I guess what I kind of just like found that I like doing it and I kept doing it. I

didn't even What took you so long? Where'd it go? Did you lose your >> Okay, let's go back. interface. Hello. Hello. Yeah. We'll be starting in a couple more minutes. Just giving time for people to come in and settle. All right. If want All right, everyone. If I could have your attention, please. Welcome back again to the next generation. This is our last talk of the day

at 3:00 p.m. Today we have um a Yosh Tanden on AI in Costco shipping processes. And um this will be a live streamed. So uh tomorrow on YouTube this will be posted and in a few months it will be cut down to uh specific talks such as this one. So if you feel like you might have missed something, maybe you've gone to the bathroom, gotten a drink

of water, come uh have to leave early, then it's completely okay. We have that live stream for you. Again, uh we have Yosh Tanden on AI Costco Hi everyone. Hi. This thing work? Yeah. Um, yeah. Thank you for presenting me. Uh, and thank you all for being here. I'm going to be talking about AI in Costco shipping process. Uh, don't mind me having fun with this laser

pointer throughout the entire presentation. Yeah. So, just a quick intro about myself. I go to Polytechnic School in Pasadena, so pretty local here. Um, got my friends in the front. Uh, not far. I hope some of you guys came out here to watch me. Um, hope you didn't have to travel too far. But yeah, thank you all for being here. And yeah, over the summer between my

sophomore and junior year, I'm currently a junior, I got the opportunity to work at Costco. And that was a great opportunity cuz I was able to intern for their logistics program. And in this process, I kind of learned the ins and outs of like what the shipping process is like, how a product gets from a farm to the stores where you can buy them and use them

in your home. And I kind of just helped out by building an AI and that's what I'm going to kind of present today. So before I get into the key details of my presentation, I wanted to familiarize you guys with some of the first things I learned while working at Costco about the shipping industry and how it kind of works. So, in the top left, you'll see

what a distribution center looks like. Basically, a big factory or a retail house kind of place that decides what goes into what truck and where it goes where. Um, if apples are needed in a store, then apples will make their way to the store. Um, I made this really helpful uh image in the top right on how this shipping chain kind of works. So, you start with

the suppliers, then you go to the distribution centers, which are also DC's, then you go to stores. Now that might mean nothing to you guys. So let me go into it a little bit more. Think of suppliers as someone who makes the product. Um in this case if a farmer John, he wants to make apples and he wants to sell them in Costco stores. He has his

shipments of apples which he makes. He's the supplier and the producer. Then a truck comes and picks it up from his farm and takes them to a distribution center. A distribution center makes sure that his apples are ready to be sold in a Costco store. They're packaged into let's say sets of 10 and then everyone's all the checks are good. Distribution centers or DCs kind of decide

where the apples are needed. Let's say the Costco store in Pasadena needed more apples. The distribution center would assign a value that basically says, okay, these apples are from Farmer John's farm. They're going to go to the Pasadena center and the distribution center kind of sends them to the stores via trucking. Now, there's two arrows between these. the suppliers to the DCs and the DCs to the

stores. So you might be wondering how do they get from point A to point B to point C. Um in this case there's the fleet, there's the third party carriers and then there's the Costco fleet between the suppliers and the distribution centers. Uh third party carriers are used and thirdparty carriers could be individual truckers that own their own trucks. They sign on for jobs or they can

be companies. Have you guys ever been driving on the freeway? You've seen like maybe a Swift truck or a night truck. Those are independent carriers. They take things from the suppliers to the distribution centers. Now, you may have also seen a Costco truck. And you might be thinking, what's that for? Those are going directly to and from the store. So, from the store to the DCs and

basically any unlabeled truck you see on the freeway is going from a supplier to a distribution center, something like that. They're all independent carriers. And this is the same for any given store, Target, Costco, uh Walmart, any major retailer. uses a similar process. The way this kind of works, uh, before was for like the last 20 years, there's been something kind of like a Google calendar. Um,

it's called the appointment management system. It's a portal used by retailers to schedule appointments. It's like these apples need to make it to here. So, we're going to use this trucker, and then this trucker, then this trucker. And it's kind of a messy system cuz, as you may imagine, creating appointments, especially for something as large as Costco, can be pretty pretty crazy. So yeah, this businessto business

business portal for scheduling is used by logistics companies to schedule delivery appointments at DC's and then they run into a big complex jumble of delivery problems. Say a trucker was running 1 hour late in uh their travels that can cause a Costco store to be without a certain product for over a day if not uh like a week. It's actually a really crazy system because a single

delay, a single bump on the road can cause a major delay. And obviously, what we don't want is uh Costco's website being quote unquote too hard to use and people being left without apples for a week. Another thing we ran into here is uh I mentioned one of the major problems, Costco's website. People have a lot of trouble with it. Um, we have 10 people in Seattle,

Washington, which is where Costco's headquarters are located that are currently working as um as tech support kind of for this major like website and possibility. They also have their regular jobs. So, they're doing tech support on top of what they actually do for a career. This is a major problem because as I'm going to show you in a little bit, these guys get over 10,000 submitted tickets

of issues every single day. After a quick analyzation, I could tell you 70% of them are problems that can be solved just like that. Pretty easy. They're like, "How do I change my password? How do I change my username? How do I schedule an appointment?" The most basic questions ever, but they still have trouble and they submit tickets. So what I did was I helped design an

AI to kind of fix this problem and kind of give multilingual support because you have to remember Costco's in 10 different countries at the moment. So not everyone speaks the same language and not everyone uh not everyone uh can understand. So it's kind of better if we have an AI bot to get rid of 70% of these issues to make it better for the tech support and

lighten their load. So yeah, here was my gap analysis for this project. So the challenge and the tech constraints, the ch sorry the challenge and tech constraints. We needed a 24/7 multilingual AI robot that could be integrated into Costco's Oracle interface and their Oracle based appointment system and a portal tech stack. Um we also needed a local AI requirement. So the AI provider must be local. If

that's complete jargon to you, it basically means it needs to select from a certain amount of text like a certain database. It can't take any external information and put it into the website. Otherwise, it would be pretty messy because you can't have if you open an AI on a Costco website and it starts talking about anything other than logistics, that's obviously a pretty big problem. Uh, for

this project, I worked with a number of groups. So I worked with the developers, the solution architect and the 10 people on the support staff to kind of design a solution that uses AI to make this problem better or solve this problem for truckers and basically make everything move smoother in the Costco shipping industry. So I'm about to show you the ticketing system. It's kind of a

mess. So I picked about seven tickets right here that were submitted whenever I took this screenshot. There's over 10,000 every single day that are submitted and a bunch of them are simple. How do I book multiple poss so on and this these problems can easily be solved with an AI? You can see on the left there's a help docs page. So the way the Costco system worked

was there was 40 pages of help docs. So if you're stuck with a problem uh then you're going to read through 40 pages of help docs that'll 100% give you the solution. But the problem is nobody wants to do that. Why would I go to a help doc if I could just call up a human and ask, "Oh, hey, uh, can you give me this the solution

to this problem without um me having to read through 40 pages and delay?" Well, the problem is the human solution took weeks to get back through 10,000 tickets. Like, if you're number 40,000 in a queue, you think you probably think that would be slower than a help doc. So, the third solution would be asking an AI tool to help, and that's what I kind of created. So,

here's what the Costco website looks like. You log in and you try to book an appointment. Pretty standard, but what's different now is I've created an AI chatbot that's in the bottom right that kind of helps out with any problem. It greets you with a welcome message. You can close it out if you really don't want it, if you know how to do it, but it'll solve

basically any problem that you ask it. And this already reduced the amount of tickets that we've gotten by about 50% in the last 2 months. We're currently live in Australia and it's a really great experience to see the results. This AI, you can ask it basically any question. Can you help me schedule an appointment? Anything logistics related, it's got your answer and it gives you an ordered

list. So, basically what you'd want the human to do and the human response, but it kind of does it for you. I can't give you the full list, but here are the first nine steps on how to schedule an appointment with Costco to drop off some stuff. And yeah, you ask it a question logistics related. It basically takes it from a database, reads through the entire database,

finds the chunk, takes it, converts it into a text that's for a user, and takes it back, puts it back, and sends it to the to the user, gives them the instructions successfully. Now, if you ask it uh a little bit of a different question, for example, what do you think about US politics? Obviously, on behalf of Costco, we can't be really saying these answers. Um, so

we kind of had to severely limit it down to make sure that could only talk about logistics related issues and problems in the shipping industry. And the way we did that was just limiting it down um, giving it instructions. And I'll get into that a little bit later, but just know that if you're shipping with Costco, you're not going to run into problems like these cuz we've

kind of shut down that whole aspect of asking it anything random. If you put, you might be asking, why didn't we put just chat in the bottom right or Gemini? just import it one to one. It's for this reason exactly cuz we can't have it talking about something random on behalf of Costco. If we want it to be talking about something like a shipment, we can't have

it be talking about LeBron James. So yeah, uh it shuts you down and if you ask it a question it doesn't know, it'll also give you that. So this is another problem I ran into. I don't want to scare off my users by if they ask like a correct logistics problem that it just doesn't know the answer to. I probably wouldn't want to scare off my users

by saying, "Oh, try again. Try again. I can't give you that." It kind of gives you a harsh response. Um, so I had to come up with an alternative solution, which I'll present in a little bit. Then there's multilingual support for all 10 of these countries. So after I made the AI chatbot, I had to do it 10 more times for each one of these countries. So

here is what the Japan website looks like. Um, this is also going live in a couple of months. All right. It gives you the welcome message in English, then in a Japanese uh language, depending on like the dialect on the region you're from in Japan, where the Costco store is located, where you're trying to ship to. Then if you respond in Japanese, it'll give you the instructions

of Japan um of the Japan Costco and what to do, and it'll answer your question basically perfectly like it would in English. And you might be wondering, it's not a direct translation. We had actual translators who are from these regions and speak the correct dialect of each language for each Costco store. So, make sure there's no confusion and no problems that come up in this. And I'll

explain this language thing and how we do it in a little bit. Again, if you try to ask it a naughty question, quote unquote, um it'll probably tell you no. It'll say okay. And it'll try to redirect you back on the track of shipping. Do you know, maybe I can help you with an appointment instead. All right. Now I'm going be going into a little bit of

the tech stack tech stack. So for this project we use the Oracle ecosystem. So we use something called Oracle Apex which is Oracle application express. Um it's for rapid web application development built on the robust Oracle database and relational database. So all of our or uh all of our organized information is put into an Oracle database. It has an LLM and AI core. So we used uh

mostly open AI and then towards the end of my time we started integrating Gemini as an alternative solution but both core at their core do the same thing. Take the data that we've given it and process it into an English language that can be given back to a user that kind of helps with the tech support. It speeds up solutions and it stops problems right at their

core. So for the UI and model training we had dedicated UI to train the model allowing for iterative learning and refinement. basically an instructions box. If you've ever chatted with ChachiPT or Gemini, you know that if you give it like an instruction, it'll most likely follow it. That's kind of what we're doing here. Instead of coding into a behavior, we've kind of made it so that Costco,

whenever they want to change it, can change the behavior of this chatbot on the back end with the UI. So, here's what Apex looks like. Here you can see we've got like 10 files of every single thing that this skin can do. We get the error page, the chat log, the AI agent, the behavior, the supporting objects, the uh we can run the application from Apex. So,

I just wanted to give you a little bit of a background on what Apex looks like. I'm going to give you guys a moment to read some of this stuff. All right, hopefully you were able to kind of skim through that and see like the different features we have. Um, I can't exactly show you all the code, but I can show you the files um and what

we have here. So yeah, these are the broad categories of stuff that we've coded in. Then on this, as soon as you try to exit, it kind of reminds you what you have for each nation. So the US assistant, the Japan assistant, and so on. Um, when I took the screenshot, they were all using Open AI. The one that's live in Australia right now is using Open

AI. Um, but we're also experimenting with Gemini bases at the moment, so we could change that up for the future. And then, um, yeah, it also has the possibility to change a rag source. Right now they're all assigned to one and each country has their own rag source. You might be asking what is a rag source? A rag source is a retrieval augmented uh generation source. So

basically all the information that this AI has that we input from Costco comes from the rag source. So yeah, for this bot there were three main parts of actually creating it. First was the system prompt which is variable. The system prompt is what I was kind of telling you. If you give an AI an instruction, it'll kind of follow that instruction. That's what the system prompt was.

The welcome message was also a variable. It changes per nation based on what Costco in that nation kind of wants their welcome message to be. And then the rag source is data that's taken from Costco's user guide. I'm going to explain this in a couple seconds, but it's taken from their user guide so that it can parse through this text, grab one chunk that they need to

answer the question, and then send it back to the user in a formatted way. So, here's what the rag source kind of looks like. Um, just the code end. Um, and we have the knowledge base. We input it to the knowledge base. You can put a description of this rag source. So, like if you ever want to, if Costco ever wants to switch between rag sources, they

can do that for the future. and it kind of gives you the function body and then some more of the LLM configuration. It talks it shows like how we're selecting everything as OpenAI. Um taking the import that we already have and continuing it on. We used 4.0 GBT 4.0 for uh my experimenting personally. Here is what the backend UI looks like. If I'm a developer working from

Costco, uh we're working on this. I was surprised actually during my time at Costco, not all of my time was spent actually coding and programming. A lot of it was spent prompting which I learned can be just as important as the coding base. So essentially here's what we have. We have the region uh and we have the authentication key and then we have the welcome message. Okay,

welcome message changes. This is the Australia one. You could read this. Then we have the general instructions. So I'm going to read the first one. I'm going to show you my first draft and second draft of general instructions and you'll see how I changed it later in this presentation. But as you can see, I started with you're a helpful assistant specialized in the transportation domain. You'll provide

an answer to a user's question based on below FAQ information and shipment information provided in rag and only tell what the user's asking. So this is part of the solution we had to limit what this chatbot says. Anything that's not in this rag source, we can't take from the existing chatbt uh knowledge base, we can't take it from there because we don't want to show that to

a user. So, anything that's not in this instruction box um won't be shown. It's also how we got it to change languages. So if we tell it to respond in the language that you're asked in. So say a user is uh typing in Japanese. Um an instruction over here. First thing it goes through is it reads these instructions. Reads all of this understands a task. The next

thing it does is okay the last instruction basically says all right the language you're asked a question in you're going to reply in that language as well. So it's going to do that, switch to Japanese, take the information available, use the dialect translator that we have and respond in Japanese after recognizing the language. So here's a little bit of the history that I had while while presenting.

Um, first of all, I tried this was during my experimenting for um uh for like non logistics related things and then the Japanese the top two I did not non-logistics related things and then I did actual questions. So here's kind of what our chat log looks like from the back end. We can track every single question that was asked to this bot at any point of time

since it came online and it launched. So yeah, we can see anything that was basically asked here. Here is my first draft of my general instructions for this bot. You could take a moment. This took me a couple weeks to perfect and make. And then I'll show you my second draft in a second. So yeah, please take a couple moments to >> So is in the general

instruction where you create a limitation so that the questions that you showed about US politics or non so it's something in the backend code as well that I cannot show you but this calls upon that code essentially. So this instruction it gives it to this okay you're not to talk about US politics you're not to talk about anything that's non-legistics related. So yes in a in a

way yes it basically you can give it an instruction only stick to what's the information involved um I could take future questions at the end of this presentation by the way. So yeah but thank you for the question. So yeah if you give it something that you don't know this is one of the problems I brought up at the beginning of this presentation. What happens if it's

a logistics related question, but it doesn't know the answer? You don't want to give it a uh you don't want to give the user like a question or a response that says, "Oh, let's stay on track, please." You want to give it something that can actually you can give the user something that can actually help them. Cuz if you're saying, "Oh, I want to know Sydney Depot

hours." That's a very logistics related question, but um it's not giving a solution. So I had to design something that has the GPT model detect whether it's a logistics related question or not. I put this in the second draft of my instructions which I'll show you which is the next slide. Um essentially what it does is it sees okay there's two paths. Is it a logistics related

question or not? Okay. Yes, it's a logistics related question. You go down that path. Then you see is it do I have the answer or not? It goes through the entire database reads through parses through for the answer. If it cannot find an answer, then it returns the Costco support email. So basically submitting a ticket. And in this way, something I kind of found out later, in

this way, by keeping tech support alive, I'm keeping a human in the loop, which is another major important thing for AI. We can't let it do everything. A human needs to stay in the loop. Here is my final full prompting. This is what's currently live in Australia. So you can read this. It's pretty cool. All right. Uh, I spent about two weeks on that. So, I hope

you guys can like that. If you give it too much, then it does too much. If you give it too little, it doesn't do enough. Now, the problem is you need to find the exact sweet spot of text to give it. So, it functions properly while also giving you a response that's not too wordy and something that a user will get tired of reading. If you give

it, if you're a user and you want to know how to schedule an appointment, would you rather read a list by list like step one, step two, step three, or some AI jargon that says, "Oh, yes, this is a perfect question. I can definitely help you schedule an appointment." Probably the first one cuz you're in a rush. Remember how I said like an hour delay of like

a problem can lead to a week without like products in a Costco which is probably terrible for the user. This way an AI has the sweet spot. It can answer all of the questions that it needs you to answer and it can stop delays right at their core. And yeah, B before I move on from this um you could probably see some of the changes I made

that help with uh navigating what are logistics questions and what are not logistics logistics questions. All right, just a quick analysis. I started with the persona and rag reliance. So your helpful assistant basically the roleplaying of what it is and what it's supposed to be doing its main major function. Second, I did the full prompting. Remember how I told you about the different steps moving left, moving

right on two different paths. And then third was the multilingual feature. So essentially I worked with a actual translator and many of them from each of these regions whether it be French Canada or rural Japan. Um any place that has a Costco store, we had a translator work with me so I can implement their dialect directly into the chatbot depending on where they are. All right, the

next section of my presentation will be learning from this model. what does this model tell us? So on the left you could see my graphs. So this is during the beta phase. Um I hopped on every single nation's um chatbot and started testing it out. And yeah, you can also get a better idea of some of the nations on here as well. And then you could see

some of my testing in what every single question is. And then we wanted to make sure we had this feature before we actually implement it cuz we're implementing it slowly. First Australia, next I think is Japan and Korea and Taiwan. So before we move to those three, we need to have a full analysis of what Australians are asking and what features we can also uh put into

this chatbot. Additionally, I think I explained this a little bit earlier, but I'm going to get into this detail. The way the rag source works for each one of these nations is a bit different. Remember, if I'm in the Costco in the United States receiving a product, uh, a procedure can be completely different in a region like Japan or Korea or Taiwan. So, the way this works

is Costco has 10 20 to 30 page user guides and those are also really complicated cuz like I said, nobody wants to read through those. So, we put all that information into each chatbot. So, Japan has its own rules on what it wants in its Costco stores, what hours it's open, so on. I take that, I translate it into all languages, and I put that in the

rag source so that when the AI is reading down into the database, it can find it, select it, and send it back. And I think that's pretty awesome because each country has their own set of rules. Therefore, each uh chatbot for each country can be specific to those set of rules. Therefore, everything stays up to date. everything stays pretty streamlined. And then another feature that we had,

the GPT 4.0 model, we're working on putting this with Gemini, but currently the GPT 4.0 is able to give us a summary. Have you ever asked ChatGpt or Gemini to like give you a summary of a reading? I personally can tell you I've done that for my history class. But it's a pretty cool thing. If you just ask it for a summary, it can do that again

and again and again. give you a full summary on what these uh the questions are, what the most asked question was, the recurring questions, and then information beyond scheduling appointments. It literally gives you a recommendation on what you should put in next. Okay, so we fixed this solution. All right, the next most asked question is xxx y. I'm sending these users to the support team a lot

because I don't have the information to really answer this question. And because of that, I think you guys should give me the access to this information so I can help answer users. So it basically gives recommendations for itself. Okay, I need this to function better and therefore it's continuously self-improving. But at the same time, I wanted to keep a human in the loop. There's a reason why

we don't give AI every single piece of information because then it'll take advantage with a human in the loop. I've been talking about this throughout the presentation, but keeping a human in the loop is pretty important to keep this AI in check and keep it accurate. With the chat summary, it recommends the user to ask Costco for that information. Uh, I could have designed a system or

I would have designed a system that has the AI go to Costco itself, but that way I'd be eliminating the human. We need to keep a human in the loop even if it's just 1% to make sure that this AI is capable and it stays under the control and stays within its task line. So here are some more challenges I faced throughout this project. First of all,

training the model was a bit of a task because I had to switch from 10 countries to 10 countries to 10 countries asking about a hundred different questions to each chatbot in uh every single country. So about a thousand questions per day and that was pretty crazy just to train this thing and get it up to shape and test it out. The second thing was hallucinations. You

remember those questions I asked about depot hours? Well, I was pretty lucky that they didn't that it showed and it that it referred me to the uh support team because before for a whole month when you asked it about Depot hours, it would just make anything random up. So hallucinations are a big problem with AI. I was able to control it down by editing those instructions I

told you about. I was able to control it down by editing those instructions, making them specific, finding the sweet spot sweet spot. But I ran into a ton of hallucinations and I had to test each and every single possible question. At one point, I even went through the entire ticketing system to ask every single one of those 10,000 questions to this thing and ask it, make sure

it can solve them, first of all, and second of all, make sure it doesn't hallucinate. Then finally, there's building constraints. So again, like I said, this kind of relates with the human in the loop. I could make it fully automated, self-improving. No human has to even lay a finger on this thing. But again, I was asked not to do this, and I completely understand why because we

need to keep a human in the loop for this entire project. Otherwise, this thing just spirals out of control basically. And it can uh as per experience in prior pro Costco projects, it has a possibility of just making hallucinations up and making up stuff that it doesn't know because it gets overconfident in itself. Finally, there was the workflow I had to manage. I had to work with

10 different translators from 10 different countries in 10 different time zones. I had to work with three different United States teams just to build this whole chatbot which is currently live in Australia as I've been saying. But it was a pretty strenuous project to schedule, but uh it definitely improved my scheduling skills, helped me uh manage my workload a ton, and it was a lot of fun.

Then I want to talk about uh some I'm going to transition right now away from my project and some other things I learned were happening in the AI field of And this is also for shipping in general. So yeah, this is the part of my presentation I'll be talking about uh shipping in general and how AI is being used in the shipping uh industry. So first of

all, there's shipment creation. AI is being used to read people's emails currently and create shipments in our supply chain system. So it's creating the shipments uh basically before distribution centers would have to call different stores and say, "Hey, how many apples do you guys have? We have a fresh shipment. We're deciding where to send them." Now AI can track all of this stuff. First of all from

cameras and stores checking stocks and second of all um AI in the distribution centers communicating with each other saying okay we need to send these apples to that store because they're completely out of stock. Then there's shipment negotiation. So like I told you there are independent carriers which are pretty interesting. They take stuff from the producers like the farms to the distribution centers. A lot of these

independent truckers own their own trucks and negotiate their own pay. And this can be a strenuous process because you need 10 people on the phones at all times just communicating. Okay, I'll pay you this much. No, that much. Communicating and kind of just negotiating. And you'd probably think you wouldn't want to negotiate with an AI. That's why our AI solution to this uses a real human voice.

I was completely fooled the first time my boss showed me this. I thought I was speaking to an actual real human. It's again a GBT base that negotiates with truckers over their pay for certain shipments. And I think that's a pretty cool development. Then there's shipment procurement. AI is used to reach out to several trucking companies that have best fit solutions for certain products. So let's say

a company specialized specializes in moving computers and there's a shipment of computers that needs to be picked up. AI will communicate with these companies saying, "Okay, you guys have a lot of experience shipping computers. we need a computer a com shipment of computers shipped to this distribution center or this store. It'll communicate with these companies to work out a solution and again negotiate prices, make sure everything

goes smoothly and pick the correct trucking companies for the correct jobs. Then there's tracking and tracing. So AI is used to track like a shipment and you by using like latitude and longitude basically just uh how your mail is tracked nowadays can be even AI and then shipments are also tracked. It could check traffic. It can check any delays that it might run into, any accidents on

the freeway, any time delays, and therefore it gives an accurate timing of when a product might arrive at a store. Then there's exception management. So AI is used for exception handling like if a shipment is running late or will be missing delivery. Let's say apples are delayed for a week. We can use company B's apples to substitute for that week and see what goes on. AI comes

up with solutions to delays, which is extremely helpful in the shipping industry because like I said, any delay can be pretty catastrophic. Then there's pickup and delivery appointment scheduling. We already saw some of this with my chatbot, but now they're taking a step, they're taking it a step further. AI helps find the best time window in this Google calendar kind of system for when uh apples will

be needed based on stocks, based on selling prices, and based on selling rates. So basically how fast products are flying off the shelves. They basically calculate how much time they need new products by and therefore they kind of help calculate the appointments and the scheduling. Then there's phone call management. Um like we saw mine was just the beginning of tech support. They're still yet to integrate this

but the next project would be working on like a actual kind of like the negotiator similar type thing. Someone on the phone that helps with online tech support uh vocally. Then there's document management. AI keeps track of tens of thousands of documents in the shipping industry. And it's pretty crazy because these documents um they contain information that can be accessed basically at any time by just asking

AI to retrieve a document and that helps out with the shipping industry completely. Then finally, there's invoice payment. So AI is used to process invoice for me uh invoices for multiple companies across the United States in shipping. And that's pretty good because then the truckers get paid in a more efficient manner. It's a much faster way than manually processing uh payments and stuff like that. And AI

is a quick calculator without spending that much Now, just wrapping up here, I'm going to talk about my experience with this project. I came in this summer to Costco thinking I would be coding. One of the major things I learned was I was completely wrong. That was about 30% of my time. Another 30% of my time went into communicating with my team and others. And then the

40% went into prompting skills. Prompting skills are much more important or if not equally as important than the coding skills in an AI process. You can code an AI however much you want, but as long as you're not prompting it correctly, it will not work and you can't push it out to a consumer. You need to be able to correctly prompt an AI before pushing it out,

before having consumers interact with this thing, before testing it. even you need to make sure that it's completely safe to use and it fulfills its purpose which is why prompting is so important here just a summary of some things I actually did I built a sandbox for its intelligence the system level constraints I worked on userdefined constraints technical parameters so on I worked with validation and revalidation

of response and I worked with AI hallucinations which was really annoying I created two exit ramps so contact support or move on and then I talked about chain of thought and I took I kept a human in the loop. So yeah, that's the end of my presentation everyone. I really wanted to thank you guys for coming out. If you guys have any questions, please feel free to

ask me. Um yeah, I had so much fun working at Costco this summer. I'm really grateful to be able to share this experience with all of you. Thank you. >> All right. Thank you. >> If anyone would like to stick around and ask questions, make sure to stay in your seats. And we already have one. So, uh, just stick with your seats and I will pass you

the microphone. Just be prepared. Yeah. >> Hello. Yeah, there we go. First of all, great job. >> And uh so I know uh you you mentioned you rolled it out in Australia first and once it's uh in a broader rollout stage, there are a lot of opportunities for bad actors to come into the play. So how are you uh mitigating prompt injection in the these sort of

situations or how are you thinking about something like that? >> So first of all, it's all very secure. Uh remember we using we're using the Oracle database system and we've basically excluded it. So something like this can't happen. Like I said the prompting was just the beginning of like what it just links back to the code. The code is basically saying you can't answer anything that isn't

along these lines. So there isn't really a problem of prompt injection in something like this. And once you roll out, we also stress tested this and my team and I completely um tried common prompt injections. We tried everything. Everything's totally secure. Before a company like Costco can roll this out, they have hundreds of testers. So yeah, we've already tried everything. Um based on the code, it's highly

specific, highly specialized to make sure a problem like this cannot occur. Cool. So uh the the way I understood your process for creating this you are you you you showed your >> Apex >> right? So, well, you showed your instructions, right? That the prompt that created the whole thing. Is that generating code or is it just uh instructing uh an LLM chatbot to to what its parameters

are? >> Okay, so first of all, it's not generating code. The code is written uh separately. I can't really show that because that's pretty sensitive information, but um the code is written separately. It does two things with the instructions. First of all, it sort of does what you say um what you said the second part. It kind of instructs the LLM. The second thing it does is

it calls upon the code. It calls upon the code to stay silent in a situation where it doesn't want to be asking something. It calls upon the code to answer a question and parse through the text. It's basically like a chain reaction. Think of it like that. >> Okay? >> Like a linear system that includes an AI language processing model. >> But but it is an AI

bot. it when when you deploy it can you somehow freeze it like stop it from learning you know so >> oh >> yeah um that's all done on the back end there is a setting that keeps it monotone so um basically uh I was I was displaying version 1.0 there will be a 2.0 there will be a 2 uh 3.0 Now continue versions will start experimenting with

learning more and more as AI technology advances but at the current stage at the user at the user stage there isn't really or there is a way to kind of just freeze it in the back end of the code make sure you stick to your basics and the most basic version of this is kind of what's given to users at the moment as time goes on um

there'll be more than just what I presented there will be learning and having interactive conversations with a user and that kind of just helps out. I hope that answered your question. >> That did. Thank you. >> Hi. Uh two-part question. The first is did you play with the temperature of the model at all? So if two users ask the same question, will they get roughly the same

response under the same circumstances? >> I worked I this my uh developer team was very uh strict with this, but I was able to modify the temperature on the GPT 4.0 uh area. And um I had like different responses at one point, but kind of like that we were able to freeze it to a certain amount and streamline it based on what the AI is able to

do and not able to do. So once we froze it, the temperature stayed the same and you kind of get a consistent amount. So each and every based on how you worded the question kind of is how kind of a response you'll get. So I'm not saying that if you start talking in slang it'll start talking in slang. I'm saying that like based on what you specifically

want and it tracks every single word, it'll give you a response. So yeah, you could change the temperature on that, but I'd say the temperature is pretty adaptive at the moment. It'll only continue to get more and more adaptive as time goes >> Okay. And the second part kind of a question based off the prompt injection one. Is the initial system prompt given to the model each

time it responds or could a user inter Okay. So the user can't interact with it so long that it loses the context for that initial prompt. >> That that's just to keep everything safe and strict and yeah continue for it to be like a user user oriented system. >> I how do you connect or what is the channel between the AI to the rug system? >> Uh

the can you repeat yourself? The channel from the AI to the running >> to the rag. Oh, the rag system. Okay. So, I'm gonna go back here and show you So, here we go. Where is it? Here. So, the AI is basically connected through the Oracle database. This system, the rag system and the instructions go into the Oracle database and from the Oracle database, the AI is

assigned there and it can pull objects uh and use them for uh answering any question that a user may have. So it's connected through the Oracle Apex database. >> Okay. And did you have a system to save like if somebody ask a question and there is no answer like to teach to train the model for the next customer that call? >> This is what I just showed.

Um let me show you remember this thing. This is the chat analysis. So remember I made the analysis of asking AI to summarize something. Um basically what the chat analysis does it it looks at the most an or asked question and it tells the human which is the developer and the checker it says okay um can you please ask Costco for this information I want to answer

this question but I don't know how so it would help users a lot if I had this information so yeah this is how it improves upon itself I remember I said I can't keep it in an infinite loop I need to keep a human in the loop make sure the human is the stopping point yes no uh the ultimate decider will human but it gives itself recommendations.

So yes, it does learn from each and every single chat. >> Hi. Uh you said you were working on this over the summer, but like how long were you working on like this project specifically? And then also how long do you think it'll be before it goes out to like all 10 of the countries? Okay. So, on your first question, you mean like how long did I

>> were you working on like this >> this presentation? This presentation took me a couple weeks. I had to regain my access to my Costco credentials, which was a little bit of a hassle, but my boss was fine with it. Um, and then uh as per going live, I think this is already live in Australia. We're still uh experimenting with the Gemini based and they're thinking they're

going to want to roll out Gemini in uh Korea, Taiwan, and Japan. So, um it'll be another 2 3 months until the Gemini is probably ready. >> So, right now, um this chatbot is kind of an assistant for people working through the Costco system, right? Do you think there's a possibility in the near future that this could become an AI agent? Like if I were to ask

this bot to say book me an appointment at distribution distribution center XYZ. Do you think that's a possibility in the near future? >> Yeah. Not only that, that's like probably the next project in the next couple years that yes, that is the direct um successor to this project will be something that automatically does it for you. But currently probably we want to keep the human in the

loop until uh it's a little bit more trustable and it flows a little bit better with Costco. um the human wants to be the one making the appointment because the AI can just assume a bunch of things. But in the future, it'll definitely be AI asking, "Oh, I need your information for that. That that if you could just input that, I'll make the appointment." >> All right.

Yeah, that sounds awesome. >> I think you had a question back there. curious why you want to switch like Gemini what is the >> well you probably heard that Gemini is a bit more direct um where chatbt is a bit more filler um that's just something that they want to experiment with so the two licensed models are probably the two biggest models in the world right now

GPT and Gemini those are the two licensed by Costco at the moment um it's not like up to me whether I choose oh if it were up to me I'd probably choose Claude but um that's not something I'm in control of. And the board kind of wanted GBT and they wanted Gemini. And they want to try out Gemini because they're building partnerships with Google and see what

they can do in the future rather than like an open AI base. >> Does anyone have anything else to say? Uh, I think that wraps it up for you. Uh, excellent work. This was really impressive to see. Thank you. testing. Testing. One, two, one, two. Hello, this is John Luke ET giving a what am I talking about? Aaron, can you remind me? Aaron, what what should I

talk about? Okay, today is your your your annual ghostbusting lesson. So, all you have to do in order if you have any ghosts is you're going to call Ghostbusters 212555388. Got it? Okay, good. I know these guys might seem like lunatics with particle accelerators on their backs, but but don't worry, it's fine. Just make sure they don't cross the beams, bro. Okay, that's it. Thank you. >>

Any questions? Everybody. Anybody? Can I have the mic? >> Any questions? Anybody? Question. Question. Ask