About this talk
This talk covers the development of a gesture-controlled robot that can be managed over the internet. The speaker, Vladimmer, explains the architecture of the system, which consists of three main components: the robot hardware, the web server, and the web application. He introduces WebAI, a concept that utilizes machine learning and AI capabilities directly in web browsers, allowing for quick gesture recognition without the need for extensive hardware or waiting for server responses. The speaker also discusses the implementation of TensorFlowJS and a custom gesture recognition model using hand detection techniques to control the robot through predetermined gestures. He concludes by highlighting the simplicity and accessibility of WebAI tools, demonstrating the robot's functionality through a live demo.
Full transcript
Uh hello everybody. So I am Vladimmer and I have passion about robotics. So some time ago I decided to develop this robot and now this robot can do a few comments and can be controlled via the internet. And then later I decided to control this robot using my gestures. But at first let's take a look at the architecture of this system. So it has three main components.
the robot side the hardware then the web server and the most interesting part here is the web application the user side and uh now let's see what actually we can use on this uh web application there are webi there are tensorflowjs library some media pipe models uh AI models and our own gesture recognition model and now actually what is webi uh webi is the idea of bringing
AI capabilities like machine learning, object detection and run them directly into the web browser. U and it gives us a few benefits like AI can run luckily into the web browser. We don't need to send data to the web server and wait for response and we don't need for powerful hardware or app installation. And now we know this but how we can use this for gesture recognition
and it's time for the architecture of the system of recognition system. Uh here you can see that we use a web camera from a laptop or from a phone. Then we use existing uh hand recognition model. Then we need to use our own gesture recognition model. And then based on this predictions of these gestures, we can send comments to the robot and use the web sockets for
it. For example, but now let's see uh let's briefly take a look at this uh hand detection model. uh roughly said it has as output an array of 21 key points. You can see the scheme points of the scheme and these key points we use for inputs for then uh we can use uh these tools like TensorFlowJS and create this model. For example, I created this model
using TensorFlowJS. Then I collected data for each gesture. Uh I use six gestures. Then I train this model. And for all of this I use Z browser. Chrome doesn't matter. And then uh finally I integrated this model into my web application and use this for controlling this robot. And now let's see on these uh three demonstrations video. You can see there are my robot. There are two
robots and there is a current laptop with this web application. I use a web camera from it. Uh you can see some gestures here. Uh it can do six comments and uh it reacts pretty fast on it. Uh and now what I'd like to say in conclusion that uh webi it's not so difficult. It's not so uh strange we can say so and with AI tools we
can create our own custom models and uses have I done in this current project uh thank you all for attention. Uh there are links to my blog to my LinkedIn page and you can reach me after this presentation. I will show you a live demo with this robot and thank you all.
