AI-First Repositories: Architecting Your Codebase for AI Collaboration - Prithveesh Goel

This talk is presented by Prithwish, a developer with experience at Atlassian and Expedia, where he has witnessed the transformative impact of AI, particularly large language models (LLMs), on software development. He emphasizes the need for building trust in AI outputs, addressing challenges such as increasing pull request sizes and review times due to LLM usage. The speaker delves into concepts like prompt engineering, context engineering, and establishing AI-first design principles to guide developers in creating effective AI interactions. He discusses the importance of structuring code repositories, aiming for specific prompts, and maintaining a single source of truth to enhance AI performance. Additionally, the speaker highlights the significance of continuous feedback loops and analytics to improve repository instructions and AI integrations, ultimately aiming for a more efficient coding process and better collaboration between developers and AI.

Welcome guys. Like I am Prithwish. I'm doing a again a my intro. I'm Prithwish. I'm based out of Delhi. Um I have worked in couple of organizations. Uh my previous organization was Atlassian. This this this session before this like Uh I was working with Atlassian for almost 4 years. Then I moved to Expedia. And now it has been 2 years for me in So um in Expedia

like I kind of landed in the time frame where AI was picking up and that was a very good opportunity for me to see see the code and everything in a very good fresh lens. And that's where a lot of of these concepts came in. So I'll go I'll start this with one of the quote which recently our CEO made. So she said in this current times

our travelers want trust. They don't want something which is possible which is coming through our LLMs. So they don't want an overwhelming answer that okay like you know they LLM gave me a 10-pager trip. But none of it is actually worth something which I'll pick. That's where like what we want is also as developers trust. So that whatever like I think every one of us would have

all gone through this journey. They like you would have started prompting. Initially like when you were prompting like you saw that okay LLM is producing something and that was the happy moment. Then like the it started coming into doubts. Like okay it's producing but is it good? Like is it actually usable? And then we would get we were into this state of frustration that okay why there

is so much rework? Why it's like you know not matching what I expected to be. So this this has been like on I I I believe everyone's journey. Like correct me if people didn't face this. So, this is what like I'm showing you the matrix from like this is literally I think if you guys would have heard about Dora, like that's a developer matrix and this is

what they published. So, what they are seeing is there is 98% increase in the PR submitted. Like there is average PR size has gone really up thanks to LLM. Then the time to review has majorly impacted. Like that's almost plus 91%. There is another report which has recently came out. This has even gone up to one almost 150%. The error rates have increased and but yeah, still

we see that okay, individual task rate completion is also increased. So, are we actually seeing a momentum? Our managers don't agree. We are not seeing any momentum. So, this has been like majorly the term like you know, the way the terms have changed. Like we got prompt engineering, then we started doing context engineering. People said and now everyone is talking about okay, what's the AI first design

principles. And that's where like I'm touching upon and I believe many other speakers are also touching upon. But again, there is no one fit all solution. So, let's see like how many of you can relate to this. So, just for example, uh if this prompt is given like add an API endpoint for users. How well this prompt is versus the right side of the prompt where add

a user query the endpoint what needs to be followed like happy or GraphQL. And need to write a test like the right side is a better prompt obviously. And but are we telling our juniors or our engineers that you should be giving prompting like that way or we are just giving them task that you need to work on a user query or a user endpoint. So, we

are doing only first part of it. Like where we are telling that okay, you need to do this, but we are not actually training to do a effective prompt engineering where you know exactly what prompt you need to give. Where you come in a kind of a How should I say? Like kind of a handshake what LLM understands and what I should be prompting that LLM should

be doing. So, we are not in that state. So, let's understand like on very briefly. I think you would have seen this, but I'm still I want to connect the dots. Like how LLM works is obviously we have our query, we have some certain system instructions. There is some knowledge which is already sitting in our probably code base or somewhere. Then we have our tools, MCP tools,

or whatever we have connected and there is prior chart summary. This is something which it's getting as a part of context. Everything come you know, builds up as context. Then there is this training knowledge which we know like that's the LLM the versions which are increasing. Those are the prior training knowledge which has LLM has. All of these things combined goes to LLM and it produces an

output. So, we cannot change anything on the training knowledge, but we need to play around with the context which we are providing to LLM. That's something which is completely in our control. And that's where I'll be more focusing on. So, what is this AI first repository? It's basically a code base that makes AI output more predictive. So, as I think that's there is this video which is

circulating everywhere with multiple influencers doing it. Which like they have given the same prompt to LLM and it's generating different outputs every time. But that is something which we cannot do in our code. Like if you would have written a code in your repository, I don't think so you would have written in 10 different ways. You would have written in probably one or two or three maximum

ways, but not like a big variation in your out in your code like every time you give a prompt. So that's where we need to reduce that. structure your repos like you are onboarding a new engineer who is who has a perfect memory but zero institutional knowledge. So I'll go to it like what do I mean? First I'll talk about rules. Because usually when these rules are

talked about later, no one gives emphasis on them. So one very important thing is which is the very which is the basic architecture principle is like there should be a single source of truth. Like do not duplicate your information. Either it's an instruction, anything it is, do not repeat it. Okay. The second thing is like anything you are designing for AI, it has to be layered. It

cannot sit monolithically in one file. People say agents.md, copilot.in hyphen instruction, don't put everything there. It will not play well. It has to be layered. You have your initial files but you need to break it down into multiple layers. go over specificity like you need to be very specific. I'll give you examples for almost all these things. You need to be very specific with their So write

a clean code is not specific. Clean might be different for GPT or Claude or even within Claude also it the same model itself. Okay, so do not give big kind of instructions. So second is like tool agnostic like this is something which is important because I don't think so two developers um system might have the same tools because everyone has kind innovating. Everyone is creating their own

tools, but at least but these instructions or whatever we are doing into this example like that will go inside the repository. So they have to be compatible with different tools with different developers are working with. So don't be very specific with tools, but you need to be still understandable to tools. So I'll again come on to that later. Another thing is like a lot of people have

been saying okay, these are just instruction files. Like how many how many of of you have heard this like these are just instruction files? Is it that useful like these are simple plain English? like have you guys heard in at least from your peers or someone like this is I have heard it a lot of times because I have been talking around this concept so at many

different places. Thankfully some people have accepted it. So a lot of people have said like these are just plain instruction files. They don't hold a lot of value, but they hold a lot of value if done rightly. These this will actually change everything the way AI produces your Okay, so treat treat them as first citizens like treat them as code because in the end there'll be a

time which is not very far. You'll be only doing this. And all the code work will be done by AI. So this will be your first go-to thing which you need to do. Okay. So again then lastly as it's a code as we are seeing like as we are seeing if it's a code thing then it needs to be living like it needs to be it needs

to evolve. So everyone would have seen this agents.md, cloud.md, copilotinstructions.md like I think this has been talked about multiple times in this in this console itself. But even besides that, I think this is well aware thing like I didn't think so this these things need interaction. But this these things definitely need some taming down like you cannot have very large instruction Okay, if you go large, do

you think every instruction like that's a that's a thing you can probably go back and do also. So I think every repository would have already been having these files. Do you think every instruction applies to every query or prompt you give to your agents? No. I don't think so. Yeah, exactly. Like these these are not actually applicable everywhere. But you should have something which is which can

help understand the query, then it these files can navigate the AI to the right place. >> [snorts] >> That's where the first one of the principle which I will mention that make it hierarchical. Like don't over blow these files, but have in hierarchy. Like go to this place, then you'll be navigated to another one, then you'll be navigated to another one. But don't create a large tree

also. these files are still important, but you need to keep them very specific like in some of our cases some of our repository like these are just 60 lines. Like not like something uh like some of the code bases which are pretty huge, they have 200 400, but I still won't be comfortable with that long files. So how do we break things? So we are breaking things

into six parts like agents which we saw, then path specific rules, then skills, agents, MCP servers and hooks. Hooks are pretty underrated, but they they do a lot. Uh so how does a Claude AI first repository would look like? Like it's basically you have your rules, you have your skills, agents, uh commands, settings, MCP. All these things are sitting here. But again, I still feel a lot

of people would have already seen this. And many of you are already aware that okay, these things exist. But are you guys using it? Like, how much you're using it? Using it? Good enough. Nice. And Sorry? Okay. So, yeah. Uh so, if you're using it, great. Uh we'll talk about more like how I'll share like what I what we are using it and how we are using

it. But these definitely play a vital role. Like, if people are not using it, start using But how you break it down, it's something which you have to do it very architecturally. Like, what type of instructions you're putting it. Do not put instructions with which are applicable to every file. Like, put the more specific instructions which are applied to specific paths. Uh in case of your skills,

you need to be really careful. Like, I think everyone knows in skills, there are triggers. Like, you need to use those triggers very effectively. So, if you have skills which are doing a good job in background, you need to have proper triggers which uh basically tells it tells you tells like basically basically kind of connects with the query or prompt you have given. So, that agents are

able to pick up those skills based on the triggers. So, all these things you need So, this is the Claude setup. People might say like, how does it look with uh uh Cursor or VS Code? So, every single uh IDE supports these things. Like, it's uh it's not something which is only specific to Claude or Cursor. Like, every single tool is kind of supporting all these things.

But some might have in a different way. There is another thing like which people might say that okay, no, we are using multiple tools. Like how do we make them compatible with all the tools? So, that's also feasible. It's not It's not very difficult. Thanks to Claude also to some extent. Like Claude is one of the convention which has been taken up and adapted by many different

tools. So, VS Code acknowledges it. Cursor acknowledges it with some exceptions. So, for example, uh the rules in uh Cursor they are not automatically picked up by Claude rules. So, that's where there is one of the gap. Similarly, prompts are not picked up through commands of Claude. You don't need to duplicate it. You can symlink them. So, you can create a hook in your repository. Anytime someone

takes a pull, immediately after the pull, a symlink command can run, which is a very simple lightweight command. And that what that will do is that will put your commands which are there in your Claude folder, that will put in a compatible state for Cursor and VS Code. And this is only when you are using multiple IDs in your company. So, if you If your company's pivoted

to one, then I won't recommend to do all these things. But if you are using, then definitely you need to, you know, pass it on to every single ID because of one thing. Um I'll tell you about that later. I thought the next slide is about that one thing. But yeah. Uh so, I'll tell you why that one thing, why we need to make it compatible with

every editor. But before that, we'll see about this part where uh the specificity which I was talking about like write a clean code. No, don't do it. Uh but rather say like TypeScript uh error like no use of any. Or basically like whenever you do something like your skills to the Suppose there's a coder skill in your um in your repository, that coder skill must have this

command like you can run yarn lint after every commit so that you are, you know, solving those linting errors. Similarly, like, you know, follow our pattern. Okay, this is the very common And I'll also focus on this like follow which is by default any LLM is doing. It's actually indexes your code. It sees what patterns you have already been working on. again, how many of you agree

that not everything written in our repositories is that perfect enough? Yeah. I mean, like, I'll I I won't even consider my code written 2 years back is perfect enough that I want LLMs to follow that again. So, LLM is already doing that. It's kind of it scans your code, it sees the patterns, and it try try to replicate in combination of the knowledge it has been trained

on. So, you need to tell LLM what does a pattern means here. Okay, so this is So, there are two types of information with LLM. One is the information which it is kind of um inferring from the code, and the second part of the information is how you want LLMs to work for the code which is it's going to produce. So, uh similarly, like, uh write rules

the way you write code. Like, basically, I don't think so that's a practice which anyone follows to dump all the code in one file. So, similarly, you can't dump all the rules also in one file. So, you have to break them down. You have to create that hierarchy. If time permits, I'll show you my skill space. I almost have like I'm almost work with 65 skills um

and like some agents. And I'll show you how those are connected, like, what's the ecosystem look like. All right. Yep, please. What do you think of all those like gloves, super powers, get get up? Those are good. Like those are good. Rails loop, GSD, super powers, like these are good. They are basically tuned to work in a spec driven or in a systematic way. But again, they

will not know like what exactly is your patterns, what exactly the standards or guidelines which needs to be followed in a repository. But they are just doing a systematic work. But they they don't have patterns in them. So it's a kind of a process which they are following. I'll answer this in the in the coming slides. Okay. So yeah, this is the ground rule I thought this

slide is about. So this is the ground rule which needs to be remembered, which needs to be followed like anything. So if a developer needs to remember or pass on a skill agent, then it's not an AI first approach. That's where I was telling your rules should be [snorts] applicable on certain parts. Your skills must have effective So that whenever you tell suppose there is a there

is a trigger to write unit test and you have created a skill which is like more tuned towards writing unit test, then you should have exact triggers in your skill so that every time as a developer someone says that okay, you need to write you need to write a test, then it picks up that skill by default. So the problem is again, if every developer was so

effective or so vigilant towards working like their styles were very organized, then I don't think so this whole setup would have even come. This setup has come because everyone is working in a different way and we need to enforce our agents to follow these. So, enforcement is really really important. So, nothing beats this rule. Uh so, now coming back to this then like I'm I'm showing you

this example in a kind of very high-level example, but this was really the case which we would have we had seen in our repository. So, we are using GraphQL repository and we had to implement a new new feature where we told AI to write that code and it actually wrote it like a JSON fetch rather than seeing around like picking up that no need to write a

query. And even that code was written in the component. It created a function and use a fake Um and that's nowhere how we work in our code base. So, we we we do it in separate services and that also has a proper way of doing it. So, all that knowledge is something either you leave it up to AI to understand or you tell AI that this is

a specific pattern you need to follow. come like putting it everything together, like this is how an AI first repository would look like like instructions, rules, prompts, context, workflows, and analytics. So, analytics is also underrated. And analytics is something which you were pointing out like how to measure things. So, we have started like we are also not 100% there, but we have started doing it. So, I'll

also show you after the presentation again my setup. So, what we have started doing is like again, we talked about hooks like we briefly saw hooks. We are using those hooks to capture what skills got used, uh what how many bash commands got run, like all the all the infrastructure which actually a particular LLM chart had, that is getting captured. And we have started using it because

we want to map that with the outcomes or deliverables that okay, these things happen. We will put them in like in in some dashboards and we'll see that okay, the teams who are having more output or more skills usage, we are trying to map them okay, are they are their PRs better? With this work like which is like if we connect these two things, if we get

a pattern that okay, people using XYZ skills have less review comments on their PR. So that becomes a standard and that becomes a maturity signal for that skill to be circulated to other spaces as well. Like suppose if we do it in one team then we can easily spread it out with the numbers that okay, using this skill we have seen an improvement. So these are some

of the measures which we have started using it and these inputs kind of goes back to the system. Just for example, if we are if we are expecting if we created a skill and if we expect that skill could have should have been picked up by itself by the agent based on the triggers and instruction and based on this analytics we we can see that okay, that's

not getting picked up. So there's definitely some gap. So we need to do this analytics that will be again very useful. Uh so that's a very basic question which people ask like because they don't treat these instructions as their for citizens. They feel that okay, it's a one-time thing. We'll write it and you know, this will this will do wonders forever. So that is not the case.

Like these instructions needs to be improved. They will definitely go stale. But you have different ways to do that. Like it's not something that you need to do it manually for everything. So, uh some of the things some of the ideas which you can try out. Like for example, uh it depends on the repository what's the frequency of uh the commits and merge happening in your repositories.

So, suppose if you have a very frequent uh merges happening in a particular repository, so what you can do is like you can have a CICD pipeline. If you're using GitHub, if you're also enrolled with Copilot, or even if you're enrolled with other like I think Cloud also supports that. Cursor also supports uh integration. If you have any of those, what you can do is you can

create an agent running on your pipelines which actually sees what PRs got merged in a specific time period. You can keep that threshold based on your frequency of merges happening in your PRs. And then tell that uh Copilot or whatever it is to create a PR improving the instructions and agents. But then you do the manual review. Don't let everything be done by AI. Then be that

human in the loop in that case because those instructions are something which you are actually giving as tools to your other engineers which probably new new to your team who have just come out of college. You are giving those instructions to them via agents because agents will consume those instructions and your uh developers will use your agents to produce codes. So, that's the cycle you need to

complete. You cannot go away with that cycle. Okay. So, have those workflows. Have those CICD pipelines. If you are not able to do it, just like, you know, take your repository once in a week someone on a rotation basis can tell AI to, you know, just get the history and create a PR. So, even if you are not able to do it as a CICD pipeline, do

a ritual in your team. Tell your team members to do it once in a week or once in 2 3 days. But, this is this is there's no bargain to this. If you don't do this, then over time you'll, you know, like you'll see more worse results like because instructions are saying something else, your code is saying something else, they won't match, and you won't get good

results. A lot of people say after this uh like yeah, this is how the cycle might look like. I don't know why it's showing so small, but yeah. This is how our cycle might look like like you do the PR reviews, you do the scan, extract, propose, and then it goes back for a human review. Okay. So, you should do this. Like this has no bargain to

uh it. Wait. A lot of people say like, you know, uh is it additional work to us? And I think this is a graph uh you might be able to relate. So, right now the implementation cost or implementation effort has gone really down. Like it's cheap to produce code. Okay. And what uh and what is increasing is like how you design your code because your design discussions

have increased because as soon as you design, you give it to AI, it'll be able to produce. Or in some of the teams like in people are not even spending much time on design, they are just giving some prompts, and that's producing code. But, that shouldn't be the case. Ideally, you should go into the design phase, like a longer design phase, I would say, especially when you're

working with AI because earlier I like people who have experienced especially before this AI even while doing the code, you used to evolve your design. Like you used to do the discovery, you used to find out that, okay, you know, this is something which is not matching with the initial discovery, we change the design. But, AI won't do that. If you give it something to AI AI

will come with a solution, no matter how. And for that, no matter how what AI solution will come out, that will either create a new endpoint in a repository where it was not supposed to or like you know, whatever it is like AI will somehow I I think everyone of us know like AI is quite enthusiastic to solve something. Uh irrespective of rightness or wrongness. So yeah,

this is what the graph looks like, but if you are in a loop cycle your efforts will go down. And we need to come down that dotted line eventually. So if you have that cycle of reviews getting into uh like you are you are having a CICD pipeline, your instructions will improve over time. That would mean that your effort of reviewing will come down and not because

the senior engineers are lying on agents, but the code which was produced by AI was itself good enough to uh like not having any many review comments. So you have to left shift your reviews at the time of coding itself. Like the code itself shouldn't produce uh any gap like shouldn't have any gaps. So that's the main intent like right now there is a lot of focus

like you know, implement like do the code fast, but there is I don't [snorts] know how many of you your companies have that focus, but the review focus is still something which people are saying okay, a human in the loop will solve that. And that's where uh the the humans who are in that loop are getting stressed out. They are becoming the bottleneck and I am one

of that bottleneck to a lot of engineers. And even being here in this conference is also like just just uh increasing my backlog of reviews. So don't push out things towards the time like when a review need when a review has to be done, you need to pull up them left shift everything that the code generated is not bad enough. Like that's really good. And all those

things will happen through those instructions. I think you have No, so these graphs are based on some of the reports from DORA Faros dot AI. So, these are not I got this is like this is something which is happening. And even the initial numbers which I even they say the same like your PR review time has gone up which eventually increases your deployment time, your operational time,

but the the individual people who are producing and completing the code has decreased a lot. So, all those numbers also talk about the same stuff. Yep. Yes. I'll agree and disagree both eventually you are you are creating a behavioral change for Because you are giving those instructions which are kind of enforce instructions to our agents that they have to obey them. For people, the the behavioral change

is that they need to shift from that mindset that they need to focus more on versus doing context engineering and agent management. So, the more you will do better agent management or context in engineering, code will be better produced. So, just for instance, yeah, there is definitely a bit of behavioral change in order to create those instructions. So, just for example, like I think again I again

something which everyone would have seen, if you tell AI to do something and AI didn't do it in that way, then you give an additional problem. No, no, no, you should have done it in this way. You do that, right? Like everyone of us, even if it is not a code, even if it is a normal chat, we do that. That's the exact part you need to

add to your rules. Why did it it do wrong? That's the retrospective you have to do. Because if you'll do it once, then that'll actually help you that it won't repeat those mistakes again. No. No, sorry. Do you have any experience like they got worship? They got to prepare uh they probably got And I'm not saying without human interaction right now. But soon. I was talking about

without human interaction. So, the agent like hope for that code rapid and there is check marks for doing in the project. So, they not working so much of the AI ball. Right? Like you said, you are working like you are here. So, such tools to take it and figures like code rapid, code that or check marks security. See, all of these solutions are helping, but in the

end, they don't know your business logic. They don't know your business decisions. No no such tools like even what he mentioned, they are very good tools which you can use for development, but none of them will understand if this code block is here, why is it here? If a particular condition is there, what was the rational behind it? And AI have actually started introducing those type of

bugs. Because AI sees it that okay, this is what the code looks like, it could have been done in this way, but there was some rational somewhere that code was needs to be there and that got broke. And we have seen we have started seeing it in our company also. How are you trouble shooting that if you stuck? And you have that place. If something is not

working How are you trouble shooting? Why did it How do you go back to the instructions and they tell you, "Oh, because of this instruction this way it went wrong." I don't This is Yeah, so this is that granularity is not possible. It's It's again a very lagging metric where you will understand your outputs are getting better. But again, for all these things your testing, your observability,

these type of things is something which every company is right now focusing on because AI is producing and you cannot believe on what it's producing. So you need to have observability or testing kind of So Yeah. Pretty much the same question as the one before about the code review. Um and you said that the AI doesn't know how to do this one. Right? Right. Isn't that a

spec, right? The AI can just do that. Back to make it this way we thought. Which spec? Like the one you are using using using superpowers and all? That spec? No, not the skill, the spec. Yeah, yeah. The specs with Okay. tasks and what the code is supposed to Yep. And also the tests are also at code level. Tests. But still uh the spec which you are

generating is something which you are generating for the new work which you have to do. So it creates for that. But that spec itself doesn't know what uh what the rationals of writing those code pieces or those files in the first place itself. Is that right that you are getting those? Yep. Um that eventually you are coming back to the same thing. Uh you are you will

pick up the rules or decisions. You'll put them somewhere. Yeah, either you'll put them in spec or you'll put them in some instructions. So you're actually talking about the same thing. Because if you talk about over a time period of 5 years, there would have been so many decisions which I have made in And like having everything in long specs versus you take out the decision out

of four or five business rules and you put that as one liner because it's almost like test cases, right? Like when you have multiple combinations, like when you are using feature flags. If you have 10 feature flags, you cannot write a test for every combination. I think like the changes that you're making ideally you want to make them small enough Yes, absolutely. Then especially going with the

order and stuff you wanted MRs and your the mix is not be so small that they are are very easily reviewable rather than thousands of thousands of lines of code. You're actually answering the question which you asked. Usually specs are very vague. You would like to extract one or two, you know, outcomes out of it, do's and don'ts out of it and only capture that rather than

capturing the whole spec. Clear functions that this addresses only point Exactly. So, where will you put those spec outcomes? I mean, in the in the code in the code itself it can mention that this solves task three and four of the spec and just refer or you know refer to that and then it has as narrow context as it needs. But that's what we are talking about.

But the review bots can also Yeah, so that's that's the feedback cycle we need to create. Like so that review bot captures that. Like if there was So, what what I also do in some of my repositories, I scan not only my comments, I scan comments from other people also and those are the right type of instructions or rules which can which should go into the these

files. And eventually what I would what I am measuring it which I covered if I if you're constantly seeing a decline in review comments and our bugs or our rework effort, that's a clear signal that we are doing it right. But if you are seeing more higher review commands or if you are seeing less review commands but higher bugs, it's not a good signal. So all these

things is something which you would need to measure. Um so again like these are some of the benefits which you'll eventually get uh where you are seeing uh reviews getting shorter and that's that's that'll be the very first signal which you'll get. Your new engineers ramp up fast and that ramp up fast is something not you know ramping up not from a perspective that it's a Kotlin

repository or a Java repository. It's more like understanding that repository. Okay. Then again like the context which disappears in your chat like for ex- I think everyone knows how chat works. There is a context it keeps getting rotted and then it keeps building up new context. But the instructions which you would have given probably no no you should be doing it this way and then after seven

eight chats like that context is forgotten that will not go away. At least your important one stays in the context. again like uh all those things like where you which we talked about like your rules are enforced uh like the the ground rule which we talked about. Um and like yeah again all all those things like keep it short and those things like which you need to

So uh how we have been leveling up in our company and like how we are being seeing the different levels of maturity with the AI. So we started with AI as uh um AI as a collaborator where we were working with AI like you know we were working on one file two file we were telling AI can you write this code can you implement this?" That was

the small scale we started off. Anything I'm 100% sure that's that has been everyone's journey. Now AI is started becoming a contributor where actually we are seeing PRs where there's a label GitHub Copilot or it's written assisted We are already started seeing this and that's where like AI has become a contributor already. The last is AI as an operator and that's where the question which you asked

that if you if AI is I think not many companies have mono repos. If AI is working across different repos and that is the case like if you do a task you are not doing it only in single repo. You need to do it in UI BFF or your graphical layer. So how does AI, you know, get to know the end-to-end picture? And that's where knowledge graphs,

rags which there were so many sessions about you would have heard about them and that's where the role the their role comes into the play. In fact, that that gets me to another question which people have asked me. Um I'll come to this metric again. So a lot of people asked me this question like how is this thing different from rags and knowledge graph? aren't they same

like you know, knowledge graph can do all those things as well? Now tell me this. What does knowledge graph has? Knowledge graph is um a USB to LLM LLM has something but you are providing a USB which has information of a wider in a form which it can query in a much faster way. But that is still historical. That will not tell LLM that you should be

behaving or doing things in this way. You're not changing your behavior via knowledge graphs. You're just providing faster information, more knowledge, a larger scope with some effective ways of rags and knowledge graph. So, they are not competing with each each other. They are actually complementing them. Like they are actually helped because after doing AI for the positive things, then then we have started creating knowledge graphs over

our repositories. Uh so, right now like in the domain I have I work with I work with the checkout domain. In checkout domain like we have around 60 repositories and which is very difficult for anyone to remember what's happening in each repository. But like we have created knowledge graphs sitting over that and the instructions and knowledge graphs go hand in hand very well. Like once you have

your knowledge graph, then the question which you ask, you can actually write that instruction in your code that okay, this part is basically being powered by this repo or something. And if you saw the six steps we talked about, one of the step is MCP server. So, we are also trying to embed MCP server as in if someone doesn't have an MCP server, then that MCP server

which we are providing as a part of settings into the user system. And how that is helping is that MCP server is nothing but it connects to the So, our instructions evolved after we also embedded knowledge graph in our space. That will be outside the That will be definitely outside the repo. And even those knowledge graphs, how they are running is like we are we are running

an incremental build. It It takes all the all the 60 repositories. It runs on the incremental change. So, the first run was very long for that knowledge graph, but the incremental changes are like based on the frequency. So, right now it's running every 24 hours. And 24 hours, like I don't think all developers are actually doing a PR every day. So, it's not that much of a

change, uh which is which knowledge graph needs to do. So, it's kind of a 30-minute job every day 6:00 a.m. and that knowledge graph is updated. So, So, it should be a journey, right? You have to start with something you companies start with. Then it just start but the automated daily. Yes, it has to be a journey. Because the knowledge like you can skip also. Like, you

know, you can you can say that, "Okay, no, today I need to have a knowledge graph because there is so much context spread across so many repos. I cannot work without that." That's okay. But, again, that will still tell your agents what is available. That will not tell your agent how to produce. You need to tell your agent how to What if I put it to water

than the outside my reference my workspace? That has been a question with now which I've been debating in my company as That breaks the ground rule that agents should pick up the instructions by themselves rather than people telling that, "Okay, you need to follow this." If you are putting it outside and if that repo is not there in your workspace, you might be working in a workspace.

If that file is not there or if someone else's work only with that repository, that file will not be accounted for. So, how will you make sure if an agent is working in a repository, then it needs to follow those instructions? >> And Claude is a very good example. Like, Claude runs on a deposit on a folder. Like, if you have your file outside, unless and until

you don't tell that this file is sitting here, Claude will never read it. I think you could use the something like a public CLI or Claude CLI, which has got access for all your folder structure. And then it gets specifically instructed CLA to actually open search between these multiple folio requests and set the context right there with the prompt level. So, you'll do that every time you

start a chat? That is the only way to kind of I mean, if the role is draft scenario. No, there some of these things will help them. Like tell your like create your skills in that way that they have that knowledge available that they need to do these kind of things. And another thing is like that's that's a very heavy operation. You are actually I don't know

how token efficient uh you know, that will be because you're every time telling to see all the repositories. And again, as I said, I have if I'm working with 60 repository space, I'm not saying I'm working every time with 60 repositories. I'll be working with probably four five repositories in for a feature. But I would not want to tell AI every time. And then when the compacting

happens, then again, like you know, this this information got lost. Like and I should a knowledge graph here be called from an inception. Can can I say the code I'm going to report to a knowledge graph exists at another level? It can exist, but but like how will your MCP is like I'm just telling you like we have built it over over an MCP because like that's

knowledge graph is deployed somewhere in our CP. And like we are using it via an MCP server, which is easy to connect for everyone, easy to connect for LLMs. It's a good tool. MCP obviously is a good tool to you know, bridge all these tools together. So, I'm not saying this is the only solution. You can figure out your own solution, but we need to understand the

purpose which each thing is to giving us. Like what knowledge graph gives us, what these instructions are giving us. I think those are two different things. That's what I'm trying to say. So, actually public CLA also ask talk to the daily folks. But I that would be the exception to all these restrictions with this people happy on the the CLA it was such all over the uh

world as so it will be that directly access the company for the company. There is a part >> [clears throat] >> The efficiency of sailor at the factory that's like And sorry, one thing we are forgetting, this needs to apply to everyone, not something to your local So I am What skills package like I did see this this sort of picture what they did what they do

is like What skills can you get up and then you can just solve it. This is a very good example which you have given us. How many of those general skills are perfectly working for everyone? Without knowing that specificity of your uh domain. I mean like those skills are definitely not working for me. I won't say they are wrong or they are completely not working, but I

am I am working with all my skills in a three-layered architecture where first layer is what those general skills are giving, extending that to my domain knowledge to my domain whatever it is, and then something which is very specific to me. So my skills are in a three level. And I'll show you that. I'm just keeping a time check. I'll take 5 minutes to also show you

that. So yeah, and these are like coming to these signals like these are some of the signals which we want to track what what you are tracking. That's percentage of PRs um and PR merges, not the raised ones. Uh reviews, days like the turnaround time, and uh like we are trying to we are trying to build a tool to which you can which can scan the repos

and see that these AI first repos whatever we are talking about, how effective those instructions are. So we are trying to create something like a tool which just go to that repo try to do something, uh you know kind of a dummy in work and see that those instructions actually stop doing it a wrong thing or not. So this is still a pilot. I won't say we

are successful, but yeah that's something which we plan to make it more effective. we talked about this So yes, so kind of summing it up. So we have three layers like we have vectors which can recall things, which can help you recall stuff. Then we have graphs which give you relationships. Okay, I think this this is something which we have been talking about and AI first repository

is something which gives you guardrails. I am not saying any of these is not important. Everything is equally important. But I still feel AI first repository is something which will give you more foundational uh even if you don't have knowledge graphs or rags, if you're working only in a single repository, but if you are able or tame down LLMs to your specific needs, if it is able

to produce an output in one go based on our requirements, I think that's the that's your North Star. Probably start here agents file. I I think but everyone would have already done that. Uh if if this is there then probably go and try instructions. Like go ahead do Uh do skills. Like start with instructions and skills. Even like uh you know as I was saying like my

setup uh I literally I have to like when I'm coding like I Uh, I have a crew skill or in fact, like I let me go there only. Is it visible? Yeah, it's too small. Okay. So, I have like, you know, all these skills. Um, like arch skill, argue skill, you know, very funny skill. So, uh, this argue skill, I tell argue that, you know, this is

the topic. I want to argue on this. So, there are two agents which will spin up in background, one against, one for. They'll fight. And they the both of their agents' job is to, you know, prove their point. And then whoever has a more factual answer, they they win. So, yeah, I I used it a lot. Uh, then like, you know, um, even I have skills which

are sitting over MCP servers. And I'll tell you why. And I'll uh, that particular answer which someone asked, like the skills which are available outside, which are like through marketplace and all. I will provide what stuff you can distribute your skills like for your own like if you have a repo, you can create a plugin of skills over the cloud and then they will get installed and

they use them for specific needs. That's right. But I'm uh, that's what I'm trying to answer that it might work for you. It might not work the same way for someone else unless you tell them that, okay, this is a skill. If you are from the same domain, I am also giving you the domain knowledge, but tune it to your needs. So, just for example, taking this

Atlassian skill, it's running over at Atlassian MCP server, but it understands what is my name, what is my space, what is the conference space I'm, what are the Jira projects I'm working on. So, every time I have to go and tell it, okay, create a Jira for me, I don't need to tell the project. I don't need to tell where to create a page. So, all these

things are something I'm I'm reducing my overload. Some Some people might say like how much of overload it is, but when your agents are working autonomously, so I'll give you an example. If I sometimes like I tell I'll show you a skill called crew, which basically works with coder, tester, reviewer, and a PR skill. I don't tell them where to create a Jira then. And that is

something getting automated now. Because the Atlassian skill knows that okay, where to create that Jira for this particular PR. Ideally, a PR Jira would have been created earlier, but in some of the last moment thing also happens. So, like all these skills are there. If I show you a particular graph, so for example, I'll search crew. So, this is a crew, which is kind of working with

uh uh a tester, a Splunk agent, a DataDog agent. Uh so, it's working with multiple agents. And every time I give a prompt, I tell it that I need investigation on this issue. I don't need to tell that okay, you have to go and check DataDog, you have to go and check Splunk, or you have to go and check the history of any particular like what PRs

got merged or what happened. So, it just internally knows everything. And that's where those instructions and those hierarchy, all those things are coming useful. And another thing which really plays well, which I forgot to mention, all my skills have a file called learn. Okay? And this instruction is there with every skill that every time you when you are executing, you see something new, go and capture that

in the learn file. So, every time my skills are getting because they are learning when they are getting used. So, it's kind of feedback loop which is constantly happening. And these learn files are getting better and better. There's nothing in this learn file which says which might be which useful for anyone, but it's the concept which I'm telling that have your skill capture that. And this is

something which I'm saying that's my knowledge, you know, which is specific to me. So, for example, I have a skill which is running over Slack MCP server, but every time it sees that, okay, uh I made a query, it did a it it used the MCP server to fetch something. after fetching that, it saw that, okay, these are the new channels which this guy is probably into.

So, it'll capture that. It'll capture the summary that this channel is used for this. So, next time I don't have to tell that, okay, you know, uh like if I tell the project, it understands from that map, okay, he's talking about this project. This project has this channel and this channel I can go and check. Can you try custom typing that out even to his agent? That

I mean again, the pilot doesn't ask you to make a list. It's a mix. Like initial So, it's a build up. Like I I truly I'll tell you, it's a build up where from the time when I have started with a very small thing. Like I never thought that I have 65 field in the end or whatever. It's increasing only. But every time I felt the need

that, okay, this is something I don't want to tell it again, I created a skill out of it. There was a time where I felt that, okay, all these skills are going in a very different direction. Then I created a skill to create a skill uh because that skill to create a skill now and have all these jargons that you need to have learned, you need to

have eval's, you need to, you know, do that recurring cycle of saving things. You need to be a, you know, folder hierarchical structure. So, all those things is something now I don't do it. Like I it's a very one-liner which I'll give and it'll be able to do things. Yeah, then you you mentioned that your own skill killing tell that how to write a skill. Yeah, so

I just tell like Skiller, that's the name of that skill, like I need this skill based on this conversation. Can you create it? So, it does that. Who write the lord of the garden? The agent. Agent itself? Yes, because the skill has that instruction that you have to capture that. And next time it knows that it has to go to the Google That's also there in the

skill. Yeah, that's kind of the boilerplate of a skill. Which the Skiller agent Skiller skill knows that, okay, I need to create it in this boilerplate. You want to go to the Skiller? Looks like I used to like I didn't need find what's the waiting for. uh I think topic the life cycle of a tool chain. So, like Yes, so some of the things which I just

told you uh while we were talking that like I'm using hooks to uh do couple of things. One thing is to create these dashboard like it's measuring it like which skill got used, how many bash commands got used. So, these type of things. Another >> I I really I'm I'm really fond like, you I work too much with feedback loops. So, uh in the last hook which

runs in the chat where it produces a response there is a another skill which I've told to spin up called Sage. What it does it it captures any learning points from that conversation. So, in the end of any conversation, I just tell Sage, like, you know, anything new. It gives me, okay, these are the learning points we should capture. And that Sage has that knowledge that, okay,

where it need to capture. So, it will capture in the respective skills where it needs to go. So, it's it's a very ecosystem or yeah, ecosystem kind of a thing. So, this is how this is how my workflows look like, like a crew. Um I'll just tell it, okay, crew, I need to code this. It will spin up the coding agent. Uh it will tell it to

test a skill that, okay, you need to write test. Then the tester will give you the give it to review. So, literally then I have plugged it in with my Slack. So, I am messaging myself usually on Slack, like a self-message that I need to do this. And this system is scanning my Slack. It picks up from there and like it has created PRs for me, like,

trust me. Like, I would literally go and I check I reviewed the PR and it was like two PRs was almost in a state like no Because that happened over time. My review itself has three layered my No, not review, like, I will lecture that. So, my coder has three layers. One is a common instruction, second is a Kotlin or a TypeScript base instruction, and third is

a layer which is repo specific. So, when it works on a repo, it identifies this is the language, this is the repo, then the coder skills knows that what hierarchy to follow. Which tools? I have created my own. Which tool? It's nowhere. It's learning on a local host. So, doesn't your learn document become too big to manage? Yes, it does. And what I do is uh after

regular basis, like, why I have kept it in learn and why I have not told it to directly bake into skill because I don't want that learn to capture anything. So what I do at the end like I have again to solve this what you said I have another skill which basically helps me picking up so that skill is basically knows that okay these are the learning

opportunities which is scale which is futuristic not like you know today our bug count was this this is not a learning. So that skill understands that it gives me that these are the learnings which we should capture and scale and then go start. So yeah I my time is out. Uh so lastly just This is my LinkedIn contact. This is a QR code. Please scan if anyone's

interested in Expedia. Uh also like share the feedback. Uh Yeah probably we can take questions outside because my time is up. But yeah like scan this QR code. We have got some merch stickers from Expedia. I have my colleagues. Uh either they or me can help you guys with these stickers and all. So do catch us out after this. Thank you. Thanks a lot. >> [music]