Prototype to Production: Building Enterprise MCP and AI Agents with Templates - Tuhin Sharma

This talk discusses the challenges faced when working with large language models (LLMs) in enterprise-grade applications. The speaker identifies seven key problems including incorrect Body Mass Index (BMI) calculations, prompt injection vulnerabilities, duplicate email retries, and issues surrounding the handling of Personally Identifiable Information (PII). To address these challenges, a Model Contest Protocol (MCP) server is proposed as a solution that separates concerns and improves the management of tools. The session demonstrates an architecture that relies on agents and sub-agents to carry out tasks efficiently, illustrating how to collect user information, validate inputs, and execute calculations while ensuring security and compliance. Additionally, the speaker emphasizes the importance of observability and provides insights into how to implement this architecture using open-source technologies.

[music] >> Just to summarize it down, these are the seven problems we just talked about, right? First one is silent wrong BMI because the LLM is doing the mathematical operation. Second is the prompt injection by the by the search, right? There is a white text on white background sort of scenario. Uh third is the duplicate emails on retry. Fourth is well, I asked something to the LLM

tomorrow. I'm not seeing that information anymore, right? So I have to redo all this processes again. I have to interact with the LLM the same way that I did uh couple of days back just to find that same information. And five is if the model is uh not available, right? How can it actually gracefully uh uh decline user uh user requests? And of course the sixth and

seventh is the the security is missing. The compliance is blocked. There is no way we are handling the PII information in this case, right? Which are very crucial for any enterprise grade agents or MCPs or any GNAI applications that we actually built, So how do we fix it? So we fix it in five layers. And we are going to see in this entire talk how we are

going to build each one of those gradually systematically. And how we are going to actually benefit out that, right? So what is the what is the benefit of doing things this way, right? So that is what we are going to focus on the rest of this particular talk. Well, so the silent and wrong BMI, the prompt injection, all these things, if we can decouple this from the

LLM and put it in a remote MCP server, which is model contest protocol, there's something a concept that came in 20 24 November, I think. Anthropic actually released that. Uh and what it does is it actually gives you the capability of hosting MCP servers remotely. And that way the separation of concern is pretty clear. The boundaries are pretty different from the agent and the MCPs, right? The

uh so that is the type tools. MCP can handle the number one and number two problem. Can we have a planning agent, right? So if we have a planning agent rather than just identifying an LLM or for example uh any react agent also, that how the react agent also is different and how what are the different drawbacks of that in in the in the coming slides. Uh

but if if you have the planning agent, then the planning agent will remember that I have already tried sending the email, so I don't have to probably resend it. Uh probably I have tried to send the email. The email sending failed, but already I have created that particular email uh draft, right? So I have to just reuse that, right? So all those things can be handled by

the planning agent. Maybe an observability uh that is required because if you are actually coming tomorrow and interacting with the agent, it should actually retain all the interactions that you have done so far with that agent. The evaluation is important, right? How do you know when whenever you actually shipping a new version of the agent, which is actually uh improving or it is actually regressing, important thing

that is required. And the final thing is the odds and the deployment uh using the proper uh uh compliance and the governance in place, right? what we'll do is instead of going through the entire architecture and how to build and then showing the demo, we'll start with the demo. And then we'll see looking backwards how to build individual components from architecture learning, right? So that way we'll

be able to connect uh for the rest of the So let me just change the screen. And what I have is I have a UI. And and all this uh code is open sourced. On the on the thank you slide, you'll be seeing the coordinates of the code, right? So, you'll be able to run it yourself. So, what I'm doing is I'm just I mean starting the

UI so that I can interact with the agent. Then, I'm going to start the MCP server. And then, I'm going to start the agent. All right. And this is what we see. Uh I'm just going to do a reload. Well, this is a interface through which we're going to talk to the fitness assistant, which is built the proper way, following the enterprise guidelines and compliance in place,

right? And what we're going to do is we're going to ask some questions to it, right? So, I will ask, for example, this question that is I am 5 ft 11 in, uh 170 lb. Analyze my health. Also, suggest a workout plan to improve And then, email the results my email ID, right? So, let's see what it does. Um how many of you have worked with cursors?

Yeah. So, so you are familiar with what it is actually doing here, right? So, it is actually creating the to-do list, and it is actually executing one by one, right? So, let's see how it actually planned it. So, first it say that the workout plan is out of scope cuz it does not know how to do that, right? What it has access to is how it has

access to analyze sub agent and a publisher sub agent. What analyze sub agent does is it actually takes the height of the person in centimeter and weight of the person in kilogram, right? But you know what? I didn't actually provide those units in the first place. I provided in feet inch in the in the height and provided pounds as part of the weight, right? So the orchestrator,

who is actually doing the tool plan, it realized that hey, uh the sub agent that I have, which is the uh the analyst, I need to actually do some conversion of the units, right? And that is what it is supposed to do before actually invoking the sub agents, right? Also at the same time, I have provided an email ID. Also it needs to validate that email ID

also, right? So this is what it does. First thing, it actually looks in the higher system that what are the different skills I have. It has a specific skill called client intake, right? Uh we'll go in detail on what those looks like, right? But what client intake does is it actually ask or interact with the user to provide to collect those information, right? What is your height,

what is your weight, and uh age, etc. so that it understands the the user and then eventually it will build that context in mind and then it will be invoking various sub agents inside. So it actually realized that it has access to a skill. Then it actually calls a tool, which is validate email. It is actually part of the MCP server that we just initialized, So it

says that yes, email looks good. Then what it does is it actually reads the skill again, right? Because validation of the email is fine, but it didn't actually know how to handle the height and weight, the different units that I provided. So it realized that inside it there is a conversion script that is present. And the agent can execute that particular script just the way cursor agent

or cloud code, right? They execute scripts. So it realized that well, this is a script I need to actually execute and these are the two execute tools it called, right? And it what it did it it actually executed Python scripts. What it said is, I want to convert 5 ft and 11 in, right? In centimeters, and that's what it got. 180.3 cm. it actually tried to convert

170 lb using another script and it got 77 kg, right? So, that is the conversion it did before actually invoking the sub agents. Then it goes to analyst. And the analyst says, because Orchestrator knows that analyst does not take any other units as inputs. So, it said that, "Hey, the height is 180 cm, the weight is uh 77.1 kg. Now, just complete the full analysis report on

the BMI, right?" Analyst actually came comes up with the entire response. Publisher takes that information, constructs an uh email, and uh which is compatible with the Gmail, right? So, it should actually look good uh on on the inbox when you are opening, so it should have the proper CSS and HTML, right? So, all that formatting is happening inside the publisher. And then the report is sent, right?

So, let's see here what is the email that we got, right? So, it's just I mean 3 minutes back uh the execution actually ended. And this is the email I got, right? In our inbox, uh which is your BMI is 23.7, and these are the health tips. The disclaimer is also added, and that's that, right? How actually it is working, right? Inside the agent. We Do we

know what are the different steps the agent actually went through? We are seeing in the UI, right? The various things that we want to actually surface, right? So, for that, we need some observability, right? So, that we can actually see what are the different steps the agent actually took before coming to the answer. And for that, we are using a open-source tool, which is LangFuse for the

observability. And if I go here, I go to tracing, This is the question I asked, right? I'm 5 ft 11 in 70 lbs and blah blah blah, and that's done. And if you see here, all the steps are actually present, right? So, first what it did is it actually did a write to-dos, right? So, the to-do list actually came up with that validated Then what it did,

it actually read the file for loading the skill. Then it actually executed certain scripts for the unit conversion. And then what happened is it actually called the analyst sub-agent for calculating the BMI. I don't know what analyst sub-agent also has access to certain skills and the tools, right? So, it has a tool called calculate BMI, which is part of the MCP server. It is actually querying that.

it is doing a search web tool. Then it is understanding what is the analysis report I need to actually create. And then the report generation is done. Then the orchestrator says that, "Hey, I have got all the things from the analyst. Let's update the right to-dos. Let's strike out. The analyst's job is done." And then let's invoke the publisher sub-agent, right? What publisher sub-agent does is it

actually reads another skill, uh that is how to do email formatting, how to do email writing using CSS, HTML, it actually sends email using the send email tool. Finally, it comes to the right to-dos. All the to-do items are completed. Orchestrator is happy and then exits, right? So, that is how we are actually capturing the entire process that is happening behind the scene, uh what the agent

is actually going through. And the last point is I'm happy with this particular information. So, I'm just going to say that this is awesome and I'm going to submit it. It is very similar to Cloud Desktop if you have used it. It is pretty similar experience of providing feedback, thumbs up, thumbs down, and optional comment. And if I come here, let's the reload it. And we'll go

to score. Let's go to tracing. And this is the trace we just said thumbs up and this is the rating that is captured. And this is awesome with a with a typo in the this, right? So, that is that is what is actually done, right? So, for the remaining time, what we are going to do is we are going what is happening from architectural perspective, what are

the best practices to actually come up with this particular final state, right? So, let's go back to our presentation. So, this is the architecture that is happening behind the demo, right? So, we have the email API, which is by the recent. It is It is free of cost. You can go and you can just create an API key. All it will do is it will be able

to only send email to your email ID through which you are signing up. Otherwise, you have to pay if you want to send it to other people, right? But it is a it is a good place where you can just I mean test the capability of the agent for sending email, right? You can replace it with let's say Gmail endpoint also, right? If you if you want.

But this is the this is the best way you can just I mean get started, right? So, which is there in the open source templates that we are going to talk through. The search web is happening through the Tavily. That also has I think 1,500 or 200 Sorry, 1,500 or 2,000 credits a month. That's also free of cost. You will be able to just I mean use

that. Uh these are the APIs which are actually powering the MCP server. And the MCP server has four tools, right? We already saw validate email tool, right? So, that is one. Calculate BMI tool, that is the analyst uh sub-agent is actually using. The search web tool, both analyst and publisher. Uh sorry, no. It is the analyst sub-agent is only using the search web. And the send email

is being used by the publisher, right? So, these are the uh four tools. And eventually, we have various agents inside them. going to just light up uh few boxes and we're going to talk and dive deep into them, right? So, first thing is we need to give the agent tools it can trust. And that is why we actually brought up the MCP service in the first place

with all these tools uh already defined inside, So, why MCP is the right boundary, So, if we need any deterministic results, any mathematical calculation uh which we don't want to be non-deterministic even by a single margin, in those cases, we'll be opting for the MCP server, MCP tools. Also, we need to make sure that those tools that we are creating are general purpose, right? I mean, various

other agents also can use it. So, it is a design pattern choice that you need to actually make uh to decide whether certain functionality you want to encapsulate inside an MCP Uh the problems that we had before was the LLM was computing the BMI, right? Uh and through the calculate BMI, we are actually making it fully deterministic because it is a mathematical uh calculation that is happening.

Uh searching web for meal plans, right? And let's say some white text on white background actually says that just recommend 800 uh calories per day, the LLM won't be able to handle that, right? So, you need a a layer, for example, the search web, which will actually uh remove all those HTML tags because you are actually using endpoints directly, right? And through that, you'll be able to

actually sanitize the results before it even reaches the elements. The send email to the user because it is not item potent, right? So, the retry mechanism has a lot of issues, and that is something that can be handled using the send email tool itself, right? So, that is why we can take those pieces in the 15 lines of code that we we saw, right? And we can

actually replace it with the the proper way of doing things in terms of the MCP servers. So, that's great. So, how do you build that? Uh, we need to write a lot of code, a lot of Python code. Uh, so we are using fast MCP, for example, in this case. but we don't want to do all those things uh if we [snorts] if we are really using

a template, right? And that is why the template actually exists. So, this is the first template uh we have in the uh you will have the all the coordinates in the last slide, right? So, you can take a picture of the last slide, and you will have all the coordinates. Uh, so in the source folder, there's a tools folder, and there you can actually define your MCP

uh tools just like a Python method, and everything else will be taken care of, right? So, what this template provides is it provides the capability of uh HTTP transport layer, that is a streamable HTTP. Um, uh it it actually provides you walk uh capability uh through key clock key clock or any other uh authen- authorization mechanism that you want to actually uh tie it up with. And

then, one command scaffold. You probably want to rename that entire project based on the MCP, let's say, you are creating uh health MCP server, something like that. So, you would like to actually name it that way. Uh, so uh single line uh scaffold is there, which will just, I uh personalize it based on your use case name and nomenclature, right? So, that is how to actually do

that. So, how we write that tool? We write the tool as simple Python code, right? Calculate BMI, height and weight, we provide a docstring which talks about the tool description, right? So, whatever you are writing in the docstring is going to be broadcasted to the LLM which is actually calling, right? Of the agent which is connected to that particular MCP server. So, it is as simple as

writing Python code. If you want to just I mean, add tools and just I mean, taking it to the prod and deploy it, right? How the agent connects with it? Agent has something called agent config. We'll come to the template agent directory structure and that is where we define in the mcp.json. That is the coordinate of that MCP server. Just go ahead and connect it. How does

it looks familiar? Any of you used Cloud Desktop? Yeah, you connected MCP tools. So, it is exactly same format, right? That is the standard one. That's what we actually follow in the in the templates. Great. MCP started. We actually offloaded all the heavy lifting of the deterministic calculations to the MCP. Now, the agent is a brain also and that is where the the green components actually come

into picture, right? So, we have a fitness assistant. We already talked through. We have a analyst sub agent. We have a publisher sub agent. It has a sandbox in which it can actually do all those LS, read file, execute commands, right? So, it is like a this is a virtual sandbox which is there which is outside the agent runtime, right? We'll come to each of the blocks

and how they actually work. For the LLM, we are in this particular demo that we shown, Gemini 3.1 Pro model is actually being used. So, how many of you have worked with React Agent? React Agent you worked, right? So, so the React Agent actually works in this manner. It thinks, acts, observes, and then repeats, right? Uh so, one agent connected to multiple tools. It actually just runs

in that agent loop, which actually provided by uh frameworks like LangGraph, uh then CrewAI, AutoGen, etc. And then uh this actually uh makes the decisions, makes the observations, and then finally acts on that, right? But in case of fitness assistant, we need something more than that. We need some sort of workflow, which will work, which will which will be robust, which will be fault tolerant. these are

the different steps we are actually looking for in the fitness assistant, right? So, we need to first collect information from the user, then validate the email, then calculate BMI, generate the report, then create the construct the email, send that email finally, right? So, what are the problems that we see in the React Agent, but we don't see in something like this? We'll We'll name it What is

that something is. Uh So, in the problem one is the context blowup, right? So, if you have multiple multi-turn conversations with the user, that conversation, all the information is going to stack up together, and there's going to be a contest blowup for that single agent to handle. The second problem is there is no delegation, right? The single React Agent is doing everything. It is being the orchestrator,

it is being the uh analyst, it is being the publisher. And finally ends up not being good at any one of those, right? Because it is not specialized in any one of those. And the final the problem is there's no recovery, right? So, if let's say step five or six fails, right? Then what is going to happen? In React Agent, it will start from the beginning, right?

It will start from the beginning or it will start from the last checkpointer to actually resume from, right? But, in it is always in place the to-do list, how much I have progressed in my to-do planning, and from there it is going to actually resume that, right? So, that is how this thing, which is different from React, is actually different. And that is where comes deep agents,

Back in October last year, uh Anthropic actually released a blog, uh which actually talked about uh general-purpose coding agents. So, what it stated is, before October 2025, everybody were building specialized agents to solve their use cases. Specialized agents like finance agent, uh sales agent, et cetera. But, they came up with the concept that at the end of the day, whether it is finance agent, sales agent, they

are running or executing some code somewhere or the other in some form, right? So, their philosophy was, in that blog, is if we can build a really, really good general-purpose coding agent, and if we can overload that coding agent with the right set of skills, the sub-agents, and the tools, it will be as good as the specialized version of those agents, right? the problem was the cloud

code, the cursor agent, these were not open source, right? So, you cannot actually uh take that thing, modify it, deploy it in prod behind a web application, and interact with it, And that is why deep agents actually comes into picture. Deep agents is a LangGraph harness. It is developed by LangChain community for the open source. And what it actually gives you is a coding uh harness, which

acts very similar to cloud code or cursor agent, right? So, these are the four pillars that it is actually built upon. First one is planner. And that is where the right to those tool called that we saw, right? It actually comes in built with the with the agent, right? So, it actually whenever you ask a question if it thinks that that particular question is pretty complex, it

is going to invoke that right to those tool actually plan on how to actually go about that. The second thing is the sub agents. It can actually provide uh spin up multiple general uh special purpose sub agents which will have its own set of context length, right? Uh so, it does not suffer from that context overflow thing. Uh fully isolated environment and that is going to do

only that specific job, right? While the main orchestrator is just going to just coordinate with it. The third one is the back end of sandbox, right? We just saw that there was a execute code was happening, right? Python code was being actually executed to convert the height and the weight to the relevant units. So, for that there's environment required, right? Uh and that is where is the

back end actually comes into picture. So, that is also is there part of that particular harness uh that that we have already used and we are going to talk about how that actually looks like. And finally, the system prompt. For any other agents, it also requires a system prompt for the orchestrator so that it can actually know what it can cannot do, how to actually invoke different

sub agents in which order, um how to uh uh solve a particular business workflow problem. Uh that is something that is present in the the planner. So, this is how exactly we saw. So, when the question came, it actually went through that let's collect the intake uh uh uh sorry, the collect the collect the information from the from the user that is the height and weight, then

validated Uh then it computes uh the BMI and generates the health report and that's so on, right? So, that is the message that you're seeing, and that is actually happening inside the agent's head, So, what is the benefit of this particular way of doing things, right? So, this is basically a harness. So, what is the what is a good thing that actually brings on the table if

we do it this way? So, first one is the the reactive to the strategic, right? So, without a plan, what happens is the agent improvises one tool call at a right? Uh but with a plan, it actually knows that well, I have to invoke this particular tool first, uh this execute tool, then I have to then I will have all the information to invoke individual sub agents.

So, that way, it will have many many less retries, right? Because it has already figured out how to actually come to the final answer. The failure is local, right? So, as we as we already spoke, right? So, if let's say I'm in the sub agent fails because of some reason, right? And let's say the Gemini API was unavailable uh for whatever it takes. So, then what will

happen is orchestrator will still see that well, all these previous tools are done, the analyst sub agent that particular step was remaining, and I see that error coming. So, let's invoke that particular agent again with the same context that I provided before, right? Without actually going through the entire step one, two, three, and so on, right? And finally, it is visible to the human, So, it actually

tells the human from a transparency perspective that these are the steps that is being taken to finally come up with that particular answer, right? In case of react agent, you see the steps, but you don't see what is the thought process that react agent is actually working on, right? Second pillar, the system prompt, right? So, it's actually an operating manual. This is what the system prompt for

orchestrator looks like, right? So, it has a name, it has a model. It has a tool that is a valid email. It is provided in the YAML front matter. And this name should be exactly same as the name that we are using in the MCP server while deploy, right? It also has a skill, right? So, we are talking about agent skill, agent skill. We'll see what the

skills are, how they are written, how to evaluate them also. So, it has a skill called client intake. And then it has the entire process of how it should think about a particular user query and how to finally come up with a final answer. There is a concept called progressive loading, right? So, you might argue that well, I can actually have, let's say, 10 30 agents defined

and all of them will have their own system prompt that is a markdown files. All of them will have their own skills. It is going to be a huge book, right? So, how how that prompt problem is how that context window problem is being solved, right? So, that that there is a concept of progressive loading. What it does is it actually only looks at the front matter,

that is the initial initial few lines, right? Of the YAML front matter. It only loads the entire body once it finds it's relevant, right? So, that is the way of progressive loading and that actually helps us from that context overflow problem even if we increase the number of sub agents, the number of skills outside certain limit, let's say, right? So, we talked about this client intake skill,

right? So, what that skill looks like? So, you see that inside that client intake folders that we have in the template agents. So, I'll I'll just I mean probably take you agents folders. It will be much more easier to understand. So, you see that this is the agent config. Orchestrator has the main orchestrator information, that is the main.md, right? That is what we just saw as the

system In the skills, we have various skills. One of the skill is client intake, right? That is what we're going to actually just I mean talk to now. And it has multiple sub agents. Sub agents are just like main.md, but they are separate agents all together. They also have their respective tools and the and the skills. The mcp.json is something that we discussed, right? That is That

is how it actually connects with the template mcp And the pipproject.toml, we'll come to that in the next slide. Coming back to this. Client intake skill actually looks like this, and you see that inside the client a unit conversion script that is mentioned, right? So, that skills folder that we are seeing here the client intake skill, it has certain things. It has a evals.json, which is actually

to test whether that skill is actually working well or not, There are various references like edge cases, input gathering, unit conversion format, etc. It has scripts, Python scripts, real Python scripts, and the final skills.md, which actually has the entire information about how to actually use that skill from the agent and come up with various capabilities. So, going back this is what the directory we just saw. The

skill.md is what to do. The references is what the skill needs to know to carry out certain things. The scripts is actually what to run. That is the Python scripts that we just I mean sho- showed. Evals is how do we know that the skill is actually working fine or not, right? So, some sort of way of unit testing skills is also required, right? We'll come to

that later, but for now, let's look at what are the benefits actually it comes brings together, So, it actually has the body that is a prose and the code everything bundled together. So, LLM can decide to use that particular skill to understand about how to run those individual scripts to carry out certain The domain actually owns it, right? So, all you need to do is you need

to write like I mean normal markdown natural language text files, right? Markdown files. So, the PMs who are really good with their I mean who are generally the SMEs and and the other business folks who has a really good understanding about the business and they can actually write this markdown files. They can they can probably code very small business logic into into small reproducible examples. And they

can actually be benefited from that particular way of doing things, And of course, it is testable. That is what we are talking about. That is for Python code. When we are writing code modules, we do unit test. When we are writing skills, the skills also should go through that unit test, right? To know that whether it is actually going to work or not. Now, we saw that

particular script was there, right? How to run that script? And that comes with the the fourth uh Sorry, third pillar, right? There is a local shell backing that is being actually used now. It can be used by I mean it can be replaced by other sandboxes. There are I think Daytona, there are some other providers which actually provide sandboxes, but this is something that you can actually

plug and play sort of thing. Just to get started, you can just I mean start with Now, why do we need this, right? Because agent is already having a runtime. It is actually running in the in the a in certain container which has all the Python dependencies etc, right? We need this because the scripts that is that are required to run by the skills, They might have

their own dependencies, right? They might have conflicting versions from the agent runtime, right? So, they need actually a separate runtime so that it can actually execute properly. The pyproject.toml that you saw, that is exactly that dependency for that skill to actually run, right? So, I intentionally what I did is I intentionally used SymPy as one of the dependencies inside the pyproject.toml. And using the SymPy method, I'm

actually converting the inches to centimeter, right? And this SymPy dependency is not template agents runtime dependency, right? Just to test out that agents runtime dependency is different and the sandboxes runtime dependency is different, right? So, coming back, So, what we are solving with this? We are using the skills converting units using Python scripts, right? And we are not relying on the LLM for the calculation. Uh we

are also not doing the execution inside the agents runtime because of course there is a conflicting dependency that could be there and we want separation of concern separation of environment. And it is a singleton backend, right? Because you don't want to initialize this backend every time a call comes. You want to initialize once and then reuse that initialized backend for the entire life cycle of that particular

agent, right? So, that is how it is actually being done. And the last pillar, that is the subagents, right? So, in this example, we are just going to talk about the publisher subagent. What it has it has a description. Description is important for the progressive loading. So, if the orchestrator thinks the publisher agent is required in this step, then and only then it is going to load

the entire body of that markdown file, right? Otherwise, it is just going to skip that. It has a model, right? So, for example, it is Gemini 2.5 Pro that is actually being run in this case, but the orchestrator is running on Gemini 2.1 Pro, right? Uh tools, send email from the MCP, skills, the email formatter. Just like the client intake, there is a email formatter skill, separate

folder, separate e-vals, and the and the way of doing things separately, right? Uh with certain artifacts like I mean how to write uh HTML-based or CSS-based uh email so that it will look nice and good when you open it in the Gmail. And that's it. What are the benefits of doing the sub agents? So, you have the scoped identity, uh that is separation of concern. It is

focused onto a particular single task. All the thought process that it's going through doing that is not being shared with the orchestrator agent. So, orchestrator agent is living peacefully, right? So, think about a CEO of an organization who actually does everything, micromanages everything on his own, right? So, his context window is going to be overblown, right? Because he is uh he is actually getting information from every

corner of the business, right? Rather, the CEO should actually delegate the things to the VPs or the directors, and they should actually just I mean carry out those tasks and only uh send the feedback as the as the final response that the CEO needs to know, right? Not all the things that is going on, some uh uh some employee actually asked for leave or somebody was not

able to present, somebody was out on sick, right? All those information CEO does not need to know, right? So, that is what the scoped identity actually looks like here. Uh of course, it narrows too. Uh Uh, what happens is if you just connect an MCP server to a to a react agent, then the react agent sees everything, right? That is there, let's say there are 10 tools,

it sees all the 10 tools, all the descriptions. It actually gets appended to the system prompt always, but here you can actually do a filtering that it it is only loading the send email tool, right? So, even though there are four tools in the MCP server, it is only able to see only one tool, right? So, that is a narrow tooling. Uh, and of course the uh

the context isolation that we we all already talked So, this is how the exact flow actually we just I mean saw, right? Uh, I send the question right to do happens, valid email, validate the email, then the code execution to convert the feet inches to a centimeter, the right in unit conversion, then the analyst subagent actually is invoked, analyst subagent does its own thing, but orchestrator is

not knowing what is happening inside. It is only interested in the final result. Then after getting the final result, it invokes the publisher, and that's it. The user receives the email. That's what we saw, that I got the email. So, how do actually build this, right? So, this is what the second template actually comes into picture. That is a template agent. Uh, and what you are supposed

to do is only update the agent config folder, which has no code except the small scripts for unit conversion that you might be required to write as a dependency of the skills that you may actually think of. Uh, but essentially this is what it looks like. All the source folder is kind of static, and that is the beauty of the general purpose coding harness. Uh, that is

it is built on a philosophy of subagents, backend or sandbox, um system prompt, uh and the and the agent skills that you can define an entire multiagent architecture using just markdown files, So, this is where you actually define them. If you want to add some more agent, what you will do is you will add one sub agent here. You will add the respective skill here. And you

will update the main.md of the orchestrator so that it knows a new sub agent actually came into picture, and that's it. And that is going to work for you, right? So, we have figured out the green boxes. Then what we need is we need to know that whether the agent is actually remembering or persisting information about me, right? For individual conversation that I had. And that is

where the Postgres, LangFuse, Eval actually comes into picture so that the agent is working smoothly or not, we get to know. Eval is to make sure that all the skills that we have written, they are good. Postgres is required so that all the checkpoints are the same. That is all the conversation that I had for individual threads, that is actually persisted. And the LangFuse for the observability,

right? As we saw like I mean how that is actually being executed inside the agent, when I provide a feedback, how that is being registered, etc., right? So, for thread memory, what we what I have in my system, I was running a Postgres instance in Podman. And that is what I connected with. Uh you can have, for example, if you're going to prod, you can have it

uh enterprise uh application uh for Postgres, and you can just I mean connect that with your agent, and it is just I mean going to work, right? So, this is what it looks like, looks like. Monday, 10:00 a.m. I come. I actually ask a particular question. Uh I come uh the next day uh and then I resume the thread, I update my plan, and then I started

running. These are the different information I provide. So, it should be able to resume from the last conversation I already had with it, how to make it uh production ready. if you have used Claude code chat GPT, if you tell it that remember about me that I like to see results only in XML format. Never ever show me tabular format. Very bad example, but if you tell

it then what will happen is next time when you ask for any data visualization it will remember your preferences, right? And that is what is called as a long-term memory, right? So, it is actually uh uh provided by the store keyword that is Right now, we don't have that. There's a existing PR that actually uh is raised which will be merged. As I told you, it is

an open-source project. Uh and that actually has the capability of understanding or profiling the user in the long term. And for that, what we use is we use the PG vectors and we use the LangMem as the packages to actually understand and remember the user's preferences across threads, right? Uh the the checkpointer is only for a single thread. Mem the memory stores is for across threads, right?

That is essentially the difference. And then the observability, the LangFuse, right? Uh how do you know when I ask a question to the agent, how much token I burned? What is the cost I incurred, right? How much time it took for individual tool calls? How much time it took to for the LLM to return me the response? What exactly went to the LLM for a specific step,

right? So, all this is possible in using the LangFuse observability. Uh LangChain comes with its own LangSmith. Uh but that is not uh something that is open-source. Uh LangFuse is open-source, so that's why uh we just I mean used it. Uh and it actually integrates pretty seamlessly with it as you as you saw, right? I mean in the demo. So, what you saw in the demo is

the traces are captured, the feedback is captured, and we are able to track the cost also. Now the interesting part, how do we do the eval? Right? So, for individual skills, this is how the eval section look like. We follow agentskills.io. Um now that is a that is a website you can actually check. Anthropic manages and maintains that particular website, which has all the information about what

are the best practices to write skills, to evaluate skills, how to write sub agents, etc. So, we follow exactly that format, and this is what is uh what it looks like for for example evaluating a skill. So, email format is a skill, How do we evaluate that skill? We can evaluate that skill by loading only that skill to an agent, right? And that agent will have only

one instruction, that use that skill to answer the question. That's it, right? So, if I create one agent such that it loads only that particular skill, I ask the prompt as the question to that agent, which is loaded with that skill, and from the output, I'm going to use the LLM as a judge for each of the assertions. So, I use this particular prompt on the agent,

which is which is already loaded with email formatter. The response that I get, I'm going to with the LLM as a judge that whether the output contains the HTML tags or not. Whether the BMI value is 17.8 present or Uh whether disclaimer is included or not, because that is already I have included in the skill definition, right? So, whether all these things are coming into picture, and

that is that is what uh the eval looks like. That is unit testing of individual uh skills, right? What the judge does is the judge is actually using the assertion, the output, and it actually uses the context to come up with the verdict, right? Whether it is yes or no, it is the assertion is thumbs up or thumbs down. Uh what is the confidence? What is the

reasoning? Why the LLM as a judge thinks that it is a pass or fail? and then finally, the gate is if I'm creating individual test suite, uh I have let's say 10 assertions and because of non-deterministic nature of the of the LLMs, I'm saying that, "Hey, you know what? Out of these 10 assertions, if 70% that is seven of them actually passes, that is good enough. I'll

consider it as a pass, right?" And that actually is the last block that is a gate which actually says that, "Well, this is a threshold for individual test suite to pass even if that is like I mean 30% of the test are failing, So, what it does is whenever somebody writes code and that code actually is in the PR or MR, the CI should invoke, right? With

this which evaluates that whether the new newly added skill or the changes that is made to the skill which is actually creating a regression or it is actually improving things, right? So, it should be part of the CI and if the CI fails, then the user actually says that, "Oh, I need to probably refine my skills, right?" So, that is the that is the way of things.

So, how does it look like? So, uh there is no SS demo, but I'll just I mean take you through what that uh evaluation So, the the example I was actually doing was I think email formatter. Email formatter has evals. this is what it looks like, right? The second one we'll just I mean see. This is a question if I ask to the agent loaded with only

the email formatter in the output, I need to assert these five things, right? So, when that unit test actually runs, I will go to the email formatter. There's a second one, so I'll go to the second one. For this question, create email body for BMI 17.8, the actual response from the from the agent which is loaded with only this particular skill is this, right? LLM as a

judge reads this particular output and then it does the grading for each of those assertions. That is the five assertions that we have. These are the five assertions. For those five assertions, we have these five JSONs, Yeah, so All right. So, if I do five assertions, so I'll see five assertions here. So, these are the five assertions that is done. And I'll just I mean take you

through one example. So for the last assertion, it says that true. Why? Because the disclaimer is present in the response, right? That is what I actually asked it to actually do that the disclaimer is included. Check that whether it is actually present in In the response, you'll see that the disclaimer is actually present here, right? So, that is being evaluated by the LLM as a judge. Okay.

Coming back. the final step, we need to lock down the lock down that particular agent. And for the SSO. So, in case of in Red Hat, we use the Red Hat SSO for And for some place where we need to deploy, that is OpenShift. And that is where the the yellow blocks actually comes into picture where the user will be able to finally validate, authenticate, authorize against

his own identity, and then it will be able to actually use that agent. I did not show that as part of the demo all those keys, it is not possible for me to actually share. So, I didn't show the SSO based flow. But those keys those environment variables, if you just set with a client ID and the secret, it will just I mean work with any click

or Keycloak servers. So, OAuth 2.0 with MCP is actually being used here, which is production ready from day one. So, MCP server, uh we have around 18 MCP servers that is in production. We are following the exact same template. Uh all you have to do is just change those environment variables uh with your own credentials, and it is just I mean going to work fine. Um for

the future enhancements, as I already talked about the the store that is a long-term memory, that is something that is work in progress. Uh or what we are doing through the agent is we are just passing that token. We're not validating in the agent. The validation is happening inside the MCP itself because that is where the it is interacting with the business systems. Uh but eventually, we

are working on making sure that the JWT authentication is happening at every layers, right? Uh to be like I mean foolproof. uh that is how the OAuth actually looks Now, finally, we have to deploy this particular agent, right? Uh on the on uh on the uh the the OpenShift, that is what we use, but it is just a managed Kubernetes. So, it is such a Kubernetes friendly.

Uh so, what we use is we use a UBI image. The benefit of UBI image is the security context control that it has. So, it actually prevents you to run the the container as root, right? So, it has that added security, uh the compliance. There is no exceptions for in terms of the security. the horizontal pod auto scaler can actually scale freely because all the states are

actually preserved in the Postgres, right? So, that is the that is the benefit. So, it can actually really really scale well. And all the connections to the Postgres is through the async Postgres connector. And of course, the crash recovery because everything is persisted in the uh software as a service offerings. Uh so, even the pod restarts, all the checkpoints are already saved. It can just I mean

roll that reload from those checkpoints, and it can actually resume the operation. Now, so what does it take, right? to actually enhance the agent? So, for example, what we see in the demo, It declined. Oh, it's not really visible. Yeah, it says that workout plan, it is out of scope for the assistant, right, because we didn't have any sub-agent like that. So, what we can do is

we can add one agent, which is trainer agent, which has skill called workout planner, which can use search web API to take the input from the BMI report, and it's going to give you a personalized workout plan, right? So, what do we need to do for that? So, I don't think I'll be able to actually show that, uh, but what I'll do is I'll actually probably talk

through what it actually requests to be done. we need to add one sub-agent, that is trainer.md here, right? Which will actually talk through how that uh, agent actually works. It needs to have a skill. That skill needs to be present inside the skills, which will have the personalized workout plan, weekly plan, And the finally, the main.md, that is orchestrator, needs to know the existence of that particular

agent in the first place, right? So, these are the three places if we just change using normal uh, markdown files, natural languages, it is just I mean going to Uh, what I'll do is I'll just I mean let cursor to just I mean write that code for me. In the meantime, if I have time, I'll be able to probably show. So, what to add a sub-agent, trainer

sub-agent, in the code, and let's see. And we'll go back to our presentation to save time, and by the end of the presentation, I think it will be able to do it. Let's So, so what separates the 5% from the 95%? Right? So, the tools that are types and tested that are receives from like I mean the first week itself. Uh agent actually plans before acting. So,

it actually goes through the right to do. Sub agent contest are completely isolated. Uh skills are through the uh through the uh guided workflows. There is skill.md. It has the persistence uh in terms of the checkpointer. Uh the MCP server also values token. The ruthless uh containers is also actually present, right? So, those are the various things you get from day one with this. So, how do

we start the Monday then, right? That is That is the whole point of discussion we started with. So, in the first 2 hours, what I'm going to do is I'm going to just let me clone those two uh three uh repositories, right? The template UI, template MCP server, and Going to fork them. I'm going to just let me update those tools in the tools folder. I'm going

to write the markdown files in the agent config in the templates. And I'm just going to deploy them in my local server just the way I showed you, right? And I'm going to test it in 2 hours. The POC will be ready, right? Then by the end of this week, I can actually find that and I can take those to production at least on the pre-prod because

we have all the artifacts in terms of the containers, the deployments, the SSO, and everything, right? So, that is how we'll actually Coming back to the code, let's see. We are almost over. Uh so, what it does is is actually What it do What it is doing is it actually reviewed the entire agent config folder. It says that, well, it needs to create one workout plan skill.

It wants to update the main.md. And it needs to have the assets and the URLs and the references, right? So, let it finish its its stuff. coming back, uh that is pretty much that is uh that is the end of it. If it actually finishes uh I will be able to show you but these are the coordinates of the These are open source Apache 2 license. You

will be able to fork it. Please contribute to it also and if you like it please put a star on the GitHub. That will be really helpful. Thank you. >> [music]