Daniel Paulus - Integrating Azure Kubernetes Service and Azure OpenAI To Build Intelligent Apps

In this talk, Daniel Palace discusses the integration of AI into intelligent applications using Azure Kubernetes Service (AKS). He highlights the core principles of AI, including privacy, security, and accountability, emphasizing responsible AI practices. The session covers how legacy applications can be modernized with AI capabilities, particularly through the addition of chatbots while maintaining existing functionalities. Daniel demonstrates the deployment process using CI/CD and the advantages of a service mesh for improved observability and traffic management. He showcases a practical example of upgrading an old webshop to include AI features without major disruptions, ultimately illustrating the potential of AI to enhance user experience in existing applications.

All right, welcome everyone. I'm going to take a quick selfie. Let's see if I can get some of the crowd. Awesome. Thank you. Um, so my name is Daniel Palace. I'm the head of the Azure solutions of the factory. It's a cloud uh advisory company and I'm also a Microsoft Azure MVP. So instead of all the AI talks that have been here today on this stage, we're

now going to talk about cloud for a moment. I am going to include um AI in a solution. So it's kind of still a bit of AI. Um so what we going to talk about? I'm going to have a chat with you all about what are intelligent apps, how you can add AI uh to your app and run that on AKS, Azure Kubernetes service. And if we

have time left, I want to show you what you can do with AI and AKS. So what are intelligent applications, right? Um we all know that by now we are in the new era of IT in the new era of computing where everything is AI lots of things are happening there a lot of news everything is going really really fast so for me it started out with

GitHub co-pilot um and then Microsoft announced that they were partnering with OpenAI Okay. And after that everything got super speedy with co-pilots like everywhere um with the security copilot which I think is actually helpful for people to use um and then everywhere copilot co-pilot c-pilot right you hear it everywhere but I now speak with with lots of customers who say we want to do something with this

AI in our applications and they just want to get started, don't think about all the consequences and stuff. So we take a step back and we say okay let's talk about the six core principles uh that have to do with AI like we have to think about the private and and the security and one of the things that Azure is doing with their open eye implementation is

taking this a step further in making sure that you are responsibly working with Azure OpenAI so your data is within your boundaries for example But we also have to think about inclusiveness, accountability, like do we let the AI make the decision or do we check what thinks and check if what it's saying is actually okay. We have to think about the transparency. Do we log the actions

of the AI or not or how are we doing this? um fairness and I want to talk more about reliability and safety, but there's also ways that you can build this in your application. I always advise people to instead of using curl commands, use SDKs because they make your life a bit easier. And we want to deploy with resilience. So we use this or we do this

by deploying to AKS and most of the time all these companies have this old application that they already have and they want to bolt on AI. They want to extend their application with a chatbot um and they want to do that as fast as possible without touching the old application. So in the demo that I have, we're actually going to do this. We're going to change or

actually almost not change an old application and we're going to make sure that we make it smarter. Um, we're going to do that behind a feature flag because we want to pre-est it before we launch it to production and we want to check how the how the performance is actually holding up. Um, as we expect lots of new visitors to our application, we also going to think

about scaling and we're going to do everything with CI/CD. So, I'm going to do this by making use of a service mesh. You might or might not have heard of a service mesh, but this gives us a lot of extra visibility in sense of the service discovery. We want to make sure that um I can find the other applications that we use and I want to uh

do that by using their name uh instead of their FQDN, right? Uh I want to make it myself easy. We use a uh a service mesh because we want to do traffic um management. We want to make sure that if we launch a new version of our we can we can split the traffic and make sure that this new application is actually working the way we intended

to do. Out of the box, we get security by mutual TLS. Uh, and that's that's what a service mesh brings to table. It helps with the um resiliency and it definitely helps with observability and I'm going to show you that in a nice fancy dashboard uh which hopefully shows us some nice graphs. Um, we're going to do this by utilizing the STTO add-on for Azure Kubernetes service

as if we have to use the open source version or the enterprise version, we have to make sure that we we control STTO and STTO is a beast. um lots of movement parts, lots of different parts, um side cars, control planes, data planes. It's it's way easier if we can make sure that the version that we use is actually tested by Microsoft. They handle the scaling of

the control plane. They make sure that that works. Um there's an internal and an external ingress setup. So we can route external traffic but also the internal traffic from our own cluster towards our Uh and it works with Azure monitor uh so it works with Azure Prometheus and Grafana the hosted versions uh which makes our lives way easier and it's also officially uh Azure supported. So, what

is this application? We've built a shop. It's really old. It's been there for a while, but we get lots of questions in from people that that are calling us. So, we want to extend this uh and we want to make sure that we make it a modern shop. So, we we make uh the shop into containers and we host it on We add on a database. Of

course, we use the cloud version of this. And we want to get some basic information about our app. So, we use the um Azure monitor uh with Promedius. And because we want to we love graphs, we going to use uh Kafana. Like I just said, we're going to use for our inbound traffic. So it routes the traffic to the shop. And of course, if we want to

do something with AI, we use Azure OpenAI. Then we build our chatbot and we use the semantic kernel from Microsoft to interact with We then route our traffic, but we wanted to use a feature flag and they now have a new feature that's called app configuration which now supports uh feature flags and we want to make sure that we can scale out and scaling out is something

we can do with Kada and in order to prove that you have Azure load testing. let's show you some some things. this is our fancy application. It's not working. Change. Oh, start mirroring. Now it does. Um this is our fancy app. Uh, like I said, it's it's a web shop. You can buy .NET mugs or t-shirts or whatever you want. and if we put something in our

basket, you'll see that it actually works. So, it's a fully functional web shop, but it's not a smart web shop. We can't interact with customers. Uh, we can't ask a question. we have to call their support team. it's running on um AKS and we have set up um STO so we get fancy dashboards. This is all happening out of the box if you use the if you

use the plin. but we've made a new version of the of the website uh where we included the chatbots. So, what we're going to do is we're going to go to Visual Studio and update Oh, and we're going to update the the store to version 1.1. And we're going to deploy this You see here that the web web is configured. I'm going to watch for the update

to happen. If that's 22, then we're good. As you can see, we we're now uh have version 1.1 in our AKS cluster. It's running. And if I now go to the website, nothing changed. I expected the chatbot or or a a chat window, but it's not there yet. What we have to do is we we we are now in Azure we have this um app configuration. Let

me zoom in a bit. where you now see a feature manager and you see that chat is disabled right now. So if we enable chat we have updated the feature flag to be enabled and if we go back to our website we now magically have a chat functionality. So if we ask our chat um can you you tell me more about the products you are selling. Let's

see what it gets back with. And it tells us that they have a ri they they they have a a variety of net themed merchandise and apparel. So as you can see by not doing actually a lot of work we have now changed a not smart application into a smart application. And as you can see in our dashboard, all the load is fine. The traffic is fine.

There's no no errors happening. We're all going in the right direction. So, as you can see, we now can put AI in everything, everywhere, and all at once by simply adapting legacy applications and advancing them by the use of AI. This is something simple as a chatbot, but you can imagine all the other things that you can do. And one of the cool things that that's happening

in the Microsoft space right now is that they launch a co-pilot for everything. So if you're not experienced with Kubernetes, there's a co-pilot to exist to to assist you with your AKS infrastructure. They can all they can also help out with copilot for Azure and copilot for your YAML files. Actually there's if you let me use the if we go to the Kubernetes service right now we

can have a look at everything that's running inside of our cluster. This is where the shop runs. This is the web specification and I can Oh, it's not uh change. Here it is. Let's go back. So, these are all the name spaces uh that we have inside of our cluster. The e is running. You can see all the containers here. And if we go to the definition

of our web container, this is the actual YAML specification of our pot. It's it's a lot of YAML. Um, you have to know what you're looking at, but I don't know what I'm looking at. So, I ask C-pilot, help me understand, and I can I'm sorry, but I can't exist with that. So the co-pilot doesn't actually work very well. uh explain uh how much containers are there

running. Let me see if this were No. No, it's broken. I thinking let me see if this works. It did found a problem. Weird because it doesn't tell me what. Okay. Well, um, and as you can see, there's also here is a co-pilot. So there there are multiple steps where you can see what Microsoft is doing to increase the of of usage and um help you to

interact with Kubernetes and the containers and Does this work? Yes, this works. All right. So in this talk, I've hopefully shown you what Kubernetes brings to the table for you, right? You can easily run your containers in a uh Kubernetes version that is opinionated but easy to adapt, easy to use, and hopefully make you more productive um when you use it. The cool part is that there's

this whole ecosystem of tools and applications around just AKS which you can utilize to make your uh more advanced and and easier to adapt by using Dapper or Kada draft or all the other tools that are available in the AKS ecosystem. I'd like to thank you for your attention. Uh, I rushed through my demo way too fast, but um, if you're looking for to to do cool

things with the cloud, scan this QR code because just like everyone, we're always looking for new cloud engineers. Um this this repo is available at my GitHub profile pus TM for trademark I hope to speak you later today. [Applause]