Tutorial: Building Intelligent Apps with RAG on Kube... Cedric C, Natale V, Christopher N & Legare K

This workshop focuses on building a Retrieval-Augmented Generation (RAG) system using various open-source technologies. The speaker outlines the integration of Kubeflow and OpenShift to manage data pipelines effectively, while introducing tools like Kafka and KNative for event-driven architecture. Participants learn how to convert unstructured data, such as PDFs, into structured formats through processes that employ vector databases and embedding techniques. The speaker emphasizes the importance of data in AI applications and demonstrates practical steps for setting up a system that allows users to query their data sources efficiently. Key concepts include prompt engineering, fine-tuning of models, and utilizing Jupyter Hub for hands-on experimentation.

All right. Well, thank you guys so much for coming. Uh, happy coupon day two. This is, um, this is actually my first coupon, so this is, you know, quite an experience. Anyone else's first coupon? >> Woo! Whoa. Okay. Very cool. Um, glad to have you guys because this workshop I think is really fundamental, especially right now with AI, but being able to build your own rag system

using a lot of these open source projects like coupeflow on top of Kubernetes. We're also going to look at serving our own model about using Kafka and K native. And so you're going to see how all of these projects work together to build a system that can allow you to do question and answering over what typically is, you know, locked away data. locked away data in PDFs,

locked away data in Service Now tickets, all of these different sources where we could just, you know, we wish we could ask a question of, you know, why is maybe my VPN dropping or something like that? Our users want these uh these questions answered. And so what we're going to do is combine the components of a rag system step by step so you can see how all

these projects work together. And then you guys are all going to have the experience to get hands-on with um uh you know data pipelines being able to use something like uh vector database to store that data and then Jupyter notebook to actually retrieve this. So that is what RAG is and I'm really honored to be here with some of my besties right here uh Lree Carrison, Nataly

Vinto, Christopher Nuland who have put so much work into this workshop and if you have questions uh we'll talk about it in a second but uh there's a AI assistant first if you have something that you're you're wondering about and you want more info or just raise your hand and we'll have be happy to come to you answer any questions you have. uh this is a learning

uh safe space exper uh experience. So you know it's going to be a good time. Now just if you're interested in the slides for example feel free to scan this. It's also on the schedule online but we'll give a little bit of a background about how rag works step by step if you've never seen it before or used rag um in your own uh organization. And then

we're going to set up these kufo pipelines to import data from an API. We're also going to figure out how to do this in an event- driven way. So, we're going to have different S3 buckets where you're going to upload a PDF. We're going to turn that PDF to structured data and then we're going to use um uh Kafka and and K native in order to uh

create a pipeline that'll import this data to our database. So, uh the slides are online as well if you're interested, but let's go ahead and and start. So, why is rag needed? um you know no matter what you're using I I like to give this example whether it's a proprietary model like claude or something that's open source that you can run yourself the data is the key

differentiator between if your application is going to be useful or not so whether it's technical docs whether it's meeting minutes financial documents service now tickets whatever it might be and you know these are just examples of what you can use in a rag database but being able to ask questions about this data is what's going to make your application unique, right? Whether it's for rag where you're

questioning and answering that data, maybe it's PDFs or fine-tuning a model that could be maybe onetenth of the size of a proprietary model so it could run faster but give you the same accuracy of results. All of this is based on the source data that you're using with an LLM. So that's why it's really important how we structure this data into a database and uh doing effective

document storage but also retrieval which is what rag is at the end of the day. So no matter what model you use it's all about the data behind the AI application. Now a lot of us have done uh customization of LMS already. If you ask a model to give you back JSON formatted responses that's an example of prompt engineering. Rag goes one step further and allows you

to when the user has a question pull in relevant data sources. So the top three most relevant by semantic similarity uh data sources to answer a question and the LLM will do the job of processing, summarizing and responding to that question for you. And the last part is fine-tuning. Let's say that you want a model to always understand insurance billing codes or to understand medical terminology. That's

when you would do fine-tuning. And some of the data science pipelines that we're going to look at today can also be used for collecting and extracting data to do that as well. So when we look at the architecture for today's workshop, uh this is all on top of open shift which is a distribution of Kubernetes. It allows us to essentially do all of the arbback that we're

going to have today for the 500 workshop seats that we uh provisioned. Um, so if you have a friend and you want them to do the workshop, please, we have plenty of space. And, um, essentially we're going to break this up into different steps, but this is all cloudnative, all running in containers. First part is getting that data ready to be used with an LLM. Remember, LLMs

can only work with text um, and other types of formats, but mainly text uh, so markdown or JSON. So, we'll have to uh, take our data, get it ready for, um, an AI retrieval. We'll also need to serve a local model uh in order to uh be able to do this all in a sovereign and on premise way if we need to have data privacy or governance

and compliance um uh guardrails. Uh we're going to have the user ask a question. That question will then go and be retrieved uh or or extra data will be retrieved from a database that's added to the original question. So you got the system prompt right here. you've got the question from the user and the context here that will be sent to that LLM that's running and we

get our answer back and hopefully user's happy. So, uh that's an example of exactly what we'll be building today. Now, to get this model in, we have to uh get this data in, we have to set up different pipelines that we could automate so that when we have new data, uh we can automatically add that to a vector database to be retrieved, right? So, how do we

do that? Well, let's take a look at the data and AI application life cycle because when it comes to actually uh getting a model developed and getting that data ready, we're going to be using open source projects today like Jupyter Hub uh and Coupeflow in order to one experiment uh and run tests and and um be able to prepare that data, but then two do that in

a repeatable fashion. Um the other steps such as validation, registry and deployment we'll also take a brief look at. Excuse me. Um so coupeflow uh anyone been using coupeflow or heard of it before? Okay. Awesome. Awesome. It's just a way to automate these AI tasks. So as an example here at the bottom we have processing of uh raw data, right? Extracting the key features we need from

that data to train a model. Maybe that model is ready to go. will validate it and put it in a S3 bucket. And that's an example of what you can do with CubeFlow. Now, when you actually take a look at this yourself, you're going to notice there are pipelines and pipelines once you have the actual uh data like where we want the S3 bucket to be then

we can run that and so you have the execution and all the artifacts as well once a pipeline has been ran. Now, for this part, I'm giving this to my good friend Natali. >> Yeah, thank you. Thank you. Um, so we wanted to show a enterprise use case. Um so for enterprise how do you fulfill your data science pipeline with events and for events we're showing a

use case where you have an object storage this in this lab you will work with minio it's a popular open source project for object storage and then we will generate events from the minio think about you're uploading a PDF or file on minio then you trigger an event that event will be catched and sent into a cafka topic and then we're connecting into a um um a

component of K native which which is an open source implementation on top of Kubernetes for implementing serverless right and there's a component called K native eventing which is able to uh catch and manage and uh and work with events various serious uh flows of flows of events in this case we're working with Kafka uh events and so you will have a K native eventing trigger you have

this CFKA broker create created by the K native component and Cafka source and that will uh so sending a message into the CFKA topic will trigger uh the serverless application the K native application the so-called scale to zero know the or the function that will launch a cubeflow pipeline so we're showing you how you can fully automate your data science pipeline for rag uh with with an

event like Kafka how many of you are using Kafka here today so yeah many of you so I hope you'll find this example interesting and you can be inspired by this. Uh without further ado, we can go in the next slides because then we'll uh let you work with the uh handsome lab. Uh we don't have much time for that. So we we need to be fast.

We'll be fast fast. Rapido. All right. Um so for uh getting that unstructured data into a structured format, we'll use Dockling. Feel free to check this out. quite a popular project um also under the Linux Foundation to take PDFs into structured data um and allows us to do this all locally as well. So when we're serving a model as we talked about before these pipelines are great

for automating data science tasks but when we actually have that model ready to go oh excuse me we'll use something like Kerve in order to do the deployment of the model. So in an example here where we have an AI engineer where when they're accessing an API gateway in order to make requests like a post request to a v1/comp completions endpoint um we have underneath the uh

that that API we've got the model serving engine which is kerve and the model serving runtime which is something like vlm to pull from our resources. Now the user is going to come in with their question. To do that, you guys will be using um JupyterHub environment to run Python code in individual cells. What's really neat about this is we can do this all in a container

using open shift AI. So if we have a specific version of transformers that we want to use, we can just attach that to the container and we don't have those pesky errors at all. Now the vector database, uh this is a good introduction for my friend. >> Hey guys. So just talk about vector databases for a second. So every piece of text that we have is going

to be converted into a number that represents its semantic meaning and the words that are grouped near each other will be semantically similar. Right? And so this is going to help us organize information by meaning which will help AI to pull statistically relevant information. Here you can see an example of what those vector embeddings look like. Um and then the way it works is we're taking text

and then we're converting it into numbers. And this is just going to help our searching be much much faster, right? So that we can search millions of documents in seconds. Awesome. So they're not really our typical relational data storage. It's going to be more about these vectors that capture meaning and then um being able to query and fetch the nearest vectors for context. Amazing. I'm going to

pass it back over to Cedric. >> Yay. Okay. Awesome. And when you have that embedded data now in the context window of a of a prompt, all we have to do because we've essentially taken up and split a document into embeddings as LRE mentioned uh where we query the closest or most similar context to the user's question. We then enhance the context window of the prompt and

send that to an LLM to produce those grounded answers so that we know why the model gave the answer that it did because of these three data sources from say for example a PDF. Um when we actually send this to the LLM that context window we're using something like lang chain in order to abstract the complexities of one connecting to a database like uh MILV for this

example in the workshop and then also connecting to models. So using the open AI completions API. So lane chain kind of puts this all together, makes it quite easy and it's a popular developer tool. But this is the architecture for today and you guys hopefully all will be the happy emoji in the right hand corner. So let's try it out today. Um this is the link for

the workshop. So um the you know we're going to be walking through this, right? >> Yes. Yep. >> Cool. >> All right. Uh sound good. All right. go ahead and get connected. Uh my name is Christopher Nuland. I'll be um kind of walking you through this lab. Um I was kind of thinking about this. You know, we have a bit of a limited time here. Um so

I actually want to make some suggestions real quick. Um when you bring up the lab, there'll be an introduction section, section one, which we'll talk about here in a moment. Uh there there are then two units that you can go through. Um, so here's my suggestion based off of the hands that I showed earlier when we talk about Kafka. If you are here and you would consider

yourself more of a Kafka power user and you're more interested in that enterprise kind of configuration, my recommendation would be go straight to uh unit 3. That's where we're actually going to have the Kafka integration and that will be more about the event driven data science pipelines. If you're here more just out of curiosity and you're looking for that 101 kind of primer into this, I highly

suggest that you start with unit two. For a special few of you who uh you know, you're you're a unicorn developer, you you're 10xer, um go for it. See if you can get both done. Uh if you do, raise your hand. I'd love to know because uh we want to know how long this takes. Typically what I find is that the first part takes about 45 minutes

and then the second part takes about 35 to 40. So don't be stressed if you are going through it, you don't finish it all. Um this is one of those kind of workshops where there's a little bit for everyone here. Uh if you do have any questions, there is an AI assistant that we have integrated into the lab. Uh think of that as your um L1. Uh

we will first have you try to talk with the AI assistant. If it is going crazy and hallucinating on you are completely unhelpful we understand. Uh do raise your hand. We have a few people in the audience who are going to be helping today. And then I am your L3 engineer. I am the guy who wrote the code that you can complain to at the end. Uh

and I will come over and kick the tires if you are having any issues. So, uh, do let us know. I, uh, once everyone gets in, I'm going to wait a little bit, uh, just to give you some time to get situated with the workshop. But there are a couple of sections that, um, we do have a question. >> Uh, you can put in any email, >>

username and password, username, password. >> No, no, there's a the login to the cluster. You will be guided on how which uh, user you have to use. So the first login is just uh to enter this tutorial >> and assign a random user >> and we don't use your email for anything. Uh it's not even saved. >> Yeah, there are 20 Kubernetes big Kubernetes cluster and you

are a user in a multi-tenant Kubernetes in this open shift instance and you will be assigned to one of this cluster. Then you have to log into the cluster and in the instruction you will have the section okay user one password AI ops. The password is is the same for everyone. So don't steal the your friend P seats please. But you you mean user one, two, three,

four, five and so on. So you will see in the instruction at some point >> and your user is embedded into your specific workshop documentation. So you will see your specific user ID. Uh I will go through and do a couple of examples on the screen a little bit later. Uh there's a couple of sections that um just to reduce the time of this workshop we removed

but I will be showing on the screen just to add some clarity in a couple of uh key critical areas of the workshop and do let us know if you have any questions. Um like I said if you are a power user and you kind of want to know what's going on under the hood feel free to raise your hand. you know, we can show you some

specific pod logs or um any outputs that you may be curious with. And just a reminder, like I said, if you are here to learn how Kafka integrates into these types of data science pipelines, I highly suggest going to the third unit and skipping the second one. That's where you're going to get the most um use out of today's workshop. If you are just looking for a

101 primer and you want to just see what these types of pipelines look like, uh, then the second unit is going to be where you want to start. Yes, just a quick heads up on the lab itself. Um, it's all web based. You don't have to install anything. There's a web terminal, there's a cluster. Uh, the only thing you need a browser and if you have VPN

uh proxy, try to turn them off. they might block the web sockets. We're using uh intensively web sockets for the web terminal. Other than that uh it's all web experience. If you have any issue, just raise your hand and we're here to help. >> Oh, so fast. Come on. Give me one minute. >> Which one? Uh, yes. Uh, just a second. >> Can I >> My computer

is your computer? >> Yeah. So, there was a request that I kind of walk through some of this. Um, I'll just put remember what the password is. Yeah. So the the password I had up there a moment I just go here. Uh this will give you a showroom that generates a user. Um I'm user 17. You may also be user 17. We have a number of clusters

that have been created for this. in here it will specify how to log in. Um, let me go over this section real quick. uh your login is going to be I think in the next page but uh this is a hypothetical uh insurance company called Parasel that's wanting to try to get more knowledge base from their service now and their critical uh documentation into their uh AI

ops especially their AI ops chat systems to help reduce load on their ticketing system. Uh so in this case there's uh two sections I mentioned earlier. We have the API to rag which is pulling down from a hypothetical uh service nowesque type of ticket system. And then the second one is uh an event driven where certain key domain knowledge from different PDF and other documents are contained

in a file store that can then be made available to the So I know there was some questions about logging in. Uh so let me go ahead and just get to that section here. Um as I said it's all here in the documentation. This is kind of building up the um the frame here and you can kind of skip some of this architecture since we already kind

of went over it. There is a section here on the overview. This is probably what you're wanting the most uh here on section 2.1 and this is where uh I know we had a couple of questions on what is the login. Um that this is specific to me. As I said, you will have your own user ID that is here. uh and make sure that the you're

inputting the password for AI ops. So if we come back over here to the other screen which has our uh terminal uh in my case I will put user 17 and AI ops here and that will get me into the open shift AI console here. Uh we do have both a um open shift AI console. Uh we also have the open shift console. This is where you

can access uh your different workloads uh your pods information, your um network information, your routes um all the things that are common to Kubernetes and Open Shift uh bigger. And then we also have a terminal here where you'll be looking at things like the logs from your particular pods. So you using either OC or or cube cuddle and uh this is also where we'll interact with the

Argo CD instance that deploys the application that you'll be using Uh Chris, we we haven't mentioned I'm here. Hello. Hello. Hello. Hey. Down. >> Very. Oh yeah. >> We haven't mentioned that the AI assistant has also an Kubernetes MCP server. So think about you're working and you're starting some pod. It doesn't work. You can ask the AI assistant to troubleshoot for you. Uh, and we're using uh

a model dedicated for this lab for the lab flow and a model dedicated for the AI assistant. So, um, feel free to try it out and let us know how it goes. So one of the key parts of this lab is the Jupyter lab. We use both the Jupyter lab to interact with our model, but we also use it as a way to um interact with our

git repository. Uh, so I believe this server is running in the eastern side of the US. So you might see, yeah, the Jupiter might take a couple seconds to come up. It should be pretty quickly. Uh, just to call out here, there are all the lab materials are found in the lab content section. And in this case, the one that I'll be kind of going through here

and there is this uh this 3.1, excuse me, the 3.2. Yep. 3.1 and two. Uh so this is how we interact with this particular model. Oh yeah, we switched this. And here's the pipeline for interacting with >> PDF a little bit better. Yeah, sorry. It keeps >> Yeah, it keeps resetting. So in section 3.1 which I see we actually have mislabeled um so just to call that

out um when you are looking at the lab content uh 3.1 in the Jupyter notebook is going to be the 2.1 that's because we we switched these to put the the more simple one first but I see that hasn't been updated in the um the GitHub. So that is still correct in the lab material. But on three 3.1 you'll download the pipeline here and that's what you're

going to be using to upload uh to the coupube flow. >> Oh for which one? Which section are you in? I'm >> Are you doing section three or two? >> You're on three. Yeah. Then you'll use two 2.1 in the lab content. Yep. Which should be correctly labeled in the material. It's just it it confused me when it came up because um the the repo hadn't been

updated. If you're doing the Kafka section, you will be logging into the Mino um make sure that you're using your um bucket that's been assigned to you. So in my case I'm user 17 and in that particular lab uh I'm going to be using this user 17 PDF inbox and if you're looking for the example PDF, it is mentioned in the documentation, but we do have um

I have it up here on the screen. It's in the scripts folder of the um the home directory here. So I just uploaded the example PDF. F and that then triggered uh this pipeline run within coupube flow. So we have it labeled here as an S3 trigger. So Mino um then triggered that Kafka event which was then consumed and then ultimately triggered this data science pipeline. So

this is a very very common pattern that we see um as we start looking at data ingestion especially uh when we start talking about how we interact our data with uh these large language models. Uh so this is a very common pattern that we see within AI ops and naturally Kubernetes being at the center of uh most organizations data and applications. This makes it an ideal place

to have these types of key integrations. And I'll show like I said I'll show a couple things here in a moment. Oh, we have a question over here >> question. Do you know? >> Yes. Uh the lab will be available today. Um so you can feel free to work on this uh this on today. Yeah, let me check the right time. It will be uh turn off.

>> and we might be able to request >> all day. One day one day you have the time to finish the lab the whole day until midnight maybe more. Um and the tutorial is always available. This the g repo of the instruction are on GitHub pages. Um the script from Chris are in his repo. So everything is on GitHub. What you don't have uh you know from

tomorrow is the clusters available uh preconfigured but this tutorial is on GitHub the GitHubs we use for deploy everything it's on GitHub so if you want to try yourself with your cluster you can either do that um it's uh it's all open yeah so we see that this pipeline has now finished it's done all the data ingestion and the embedding uh so it goes through a process

of using dockling. Um so Dockling then creates the embeddings uh as uh my colleague Lree was talking about in her presentation. Those embeddings then get applied to a rag database that then makes that data available uh to your end users through the LLM. Just to check, do we have any other people waiting on questions? It's kind of hard for me to see from the stage. Just double

checking. Okay, good. over here. Mhm. >> Cedric, do you remember the password of the tutori? just because I've had a couple questions on it. Uh, and that side panel that you have, you have three tabs. You have the CubeFlow, Open Shift AI console, and then you have the Open Shift Kubernetes-based. So if you're um uploading your uh Argo application, that's all done through the Open Shift console,

the one in the middle. Once you kill it I don't know which instance I'm on. I'm on TQ. What's that? Yeah, we can. Yeah, I'm bringing up the admin panel. I'm having trouble finding the admin password here. I thought it usually shows some details, but >> you got to go here. >> And the admin is this one. So exciting. Thank you. >> Thank you so much. >>

Thank you. Yeah. One second. I'm I'm not flogging Make sure we don't have anybody. I don't know. I can't get up. >> Oh, it's just >> Oh, okay. That makes sense. So this is our Kafka view that shows um within in the open shift console that shows our broker. So we have the one broker here that um is assigned to our rag pipeline workshop. This is the

broker that all of you are interacting with. Um so this is actually managing the events for um all the users within this particular Kubernetes cluster. And then here we have the actual trigger that's um linked up with the Mino instance. So anytime that a uh a new source is added. So we see here the source definition from the filter um it's connected as a web hook into

S3 and it's anytime an object is created and um put into that particular bucket. Uh so this is the Kafka trigger that happens when you add in that example PDF into the um uh into the S3 bucket. This is what the Kafka broker looks like. Uh you have access to some of these logs yourself. Just to give you an idea um we have quite a few users

here who are triggering it actually right now. So I see that a lot of you are actually on this section. Uh so we have the message body here on the status and then we see we actually have some payloads as well um from the requests from Mo. So the broker is very active right now. And then we also have a bridge. Yeah. And the bridge logs are

available as well if you want to take a look. Is there anyone who's waiting on assistance? Sure that we're not forgetting anybody. Has anyone finished section two? Uh the first section section two. Okay. Absolutely. >> Sure. >> Okay. I'm told you had a a feeling man. >> Yeah. Sure. Go ahead and reupload it. >> Yeah. Okay. >> The name is very important. Yep. If you don't get

the name uh so there might be a couple of issues here on naming the name especially of the Kafka pipeline is very critical because that's how the broker for Kafka triggers the data science pipeline. Uh so if you put in your own name it won't trigger then correctly. And if you get a pod error when uploading your data science pipeline just raise your hand. Um, I don't

think I think I've only seen one so far, but I think we ran into a resource issue real quick with one gentleman over here. So, if you're if you have a pipeline that's failing, just raise your hand and we'll we'll come and assist with that. Uh for all the assistants in the audience, if you do see a failure, um just delete the pipeline and re-upload. >> Turn

it off and turn it back. >> Yeah, just delete it and upload it. >> It's the fastest way to fix it without having to go through all the Are we on time? >> Yeah. Oh, yeah. I think there's people who do both. >> Yeah. Yeah. It's healthy. So, it's running. I've been watching the logs. Chris, can you can you show the also the Melvos database? >> Because

there was a question about where's the database? Uh we have a name space called MILV and this is where the vector DB is. Uh I'm trying to remember if we have multiple instances. Let me remind myself >> There's one Melvos for everyone. Okay. Yep. And then we also have the one mo instance there as well in its own name It's a multi-tenant in this way. Uh but

this can be implemented also with strict multi-tenency. So in one namespace you have one menu, one cafka, one mil everything is fully isolated. It's your architectural choice for this lab was was okay in this way or for product production and real world uh use case strict multi-tenency will be also uh We can go and show the feedback. Hey folks, how's it going? Woo. I hope you guys

are having a great time so far. Not to rush you, we still have 15 minutes left and the cluster will be on all day. But if you would be so kind to give us internet karma, we have this schedule thing up here that you could either scan or I think in the workshop uh it's also there. And if you had a good time and uh would like

to support us coming back next year hopefully. Please feel free to scan it. Um drop us some internet karma. It'd be really cool. And um yeah, enjoy the rest of the workshop. You still have 15 20 minutes. Still have any questions? Uh happy to help out. And yeah, from me and Christopher and Natalyia and the we really appreciate it. So, cheers and enjoy the rest of CubeCon.

>> Oh, wait. You're right. One second. One second. >> This one's totally on me. I um I used the wrong QR code. Give me one second. Oh my god. Hey folks, sorry about that. If you scanned it, you got another copy of the slides for free, which is awesome. But now, if you would love to give us internet karma, we would really appreciate it. So, thank you

so much. And yeah, cheers. Enjoy the workshop. Thanks, bro. Is there a link to the showroom tutorial repo somewhere in the instruction >> in the workshop in the front page? All right. So, >> yeah, >> we can bring that. Oh, well, we'll leave >> Go down. Oh, here. >> Yes. Like here. And uh you can open an issue to us and say um in new issue >>

in lab 3.2 this is the error I get. Would be nice to suppress the warning. >> Yes. As a user experience. >> I think you have to be logged in >> if memory serves me correctly. People see red because mistake and >> we're not all data scient We don't know the hell they're doing. >> Thank you. Thank you. >> Thank you. Thank you. If you please send

us your feedback. Uh it was really important. But uh thanks for taking also to contribute to the workshop. It's the power of open source. Can I find Um, let me bring it up on the screen. Hey folks, um if you want to contribute to this lab, you have a link in the homepage uh on the of the tutorial uh if you go in the first homepage uh

navigation. Yeah, if you go down down down down down, there's the link to the contributing here. If you find any typo, if you think any want to suggest any improvement, you open an issue and we'll uh get sorted out uh pretty pretty much soon. And if you do actually a pull request for the content when we merge it, the GitHub action will uh deploy on GitHub pages

uh uh automatically. So this tutorial will be available in this link uh always, but we also have a GitHubs repo in case you are interested. Okay, how can I deploy this on my clusters? We we also have this GitHubs repo. It's not linked here, but maybe we can uh >> we'll put the information in there. >> Yeah, we'll put that we'll put it in the in this

repo and uh you'll find it out. >> Uh click navigation on your sidebar. Just click the navigation. >> Navigation. Yeah, click navigation. we expect All right, we have about five minutes left. Just let everybody know. Uh we will keep the cluster up for the day.