AI Builders as Creative Product Shapers - Mithra Kosur Venuraju

In this talk, the speaker, Mitra from Atlassian, explores the integration of artificial intelligence into product engineering processes. They discuss the evolution of the software development lifecycle (SDLC) from traditional waterfall models to agile frameworks, and now towards an AI-enhanced model that emphasizes rapid ideation and prototype testing in real customer scenarios. The speaker emphasizes the importance of framing opportunities and problem statements accurately, leveraging AI to enhance throughput and streamline workflows. They present case studies from Jira where AI is utilized for ticket triaging, root cause analysis, and improving customer support response times. The discussion also covers the evolving roles of product engineers in incorporating AI effectively and ensuring quality through automated evaluations and guardrails for safe development.

A little bit about me, thanks for mentioning it. I'm with Atlassian. Um my name is Mitra. I work in a product called Jira. How many of you know Jira? Okay, cool. Quite a few. How many of you have used Jira so far? Okay, nice. So, if you use Jira, if you have ever created, viewed, deleted uh work item, story, task, any of those? That's my team, my

org, who does that. So, a little bit about uh that of what we do. And um today we going to for the next uh few minutes we're going to talk about uh product engineering and how we can bring AI native work in product engineering to not just look at output, but how we can bring maximum outcome and have high throughput. If we have to take a step

back, look at the old SDLC model, right? Say 15 years back. I don't know how many of you were working 15 years back, but uh long back it used to be like uh product managers taking months to write PRDs, having very detailed requirements, sending it to development team, they work on high-level requirements, high-level design, low-level design, build something for many months, in some cases even years. And

then sending it to the testing team, test it, PM would come and test it, and then the entire rollout. Then few years back, we short-circuited it, we said, "Okay, that waterfall model isn't working anymore. Let's move to Agile." Uh we had PMs, designers, engineering partner up, collaborate with each other. It was a fairly faster on our own time. But cut to it now, we are in a

completely different state and era, where it's not yours, it's not months. In some cases, especially with greenfield projects, it is even days that we are talking about where the ideation is in front of real customers. So, that's what the new uh era is. And even in this era, like uh the core aspects of SDLC, which is coding, then looking at the outcomes, shipping it to the right

type of customers, learning, and creating that loop, that doesn't change. The thing that changes is how we can expedite these learnings, how we can use AI effectively, and how we can come up with the uh X amount of throughput compared to what we had earlier. So, that's what it changes. Why now? Why are we talking about it now? Why not a decade back? Why not 6 months

back or even like uh a year back? So, in the world of AI era, things again have been changing incredibly fast. So, when we uh looked at AI initially few years back, it was primarily chatbot. We were looking at that as, "Hey, I can ask anything. It's amazing. I can see anything." Then it evolved into, "Okay, this is amazing in rewriting documents. I don't have to write.

I just have to give prompts and it's doing really, really well." But on the SDLC side, the challenges that we had earlier was that it wasn't scaling because the staleness of it, it was not multi-threaded, and the context was very, very minimal. Whereas now things have changed quite a bit. On top of it, the talent. I think almost everyone is thinking, exploring, or trying out new ways

of working. So, that's another differentiator that has changed. on the AI side, we have evolved from looking at it as just a plain chatbot to almost like an orchestrator of agents and work where we can spin up things and it does multiple uh items in parallel and comes back to us. And most importantly, now most of our customers, they expect AI in the flow as well. Of

course, they are also bored with AI slop, putting it in everywhere, so that's where uh being very, very intentional about how we bring in AI and how we embed AI in the right workflows is incredibly important. All right. Now, if we look at the roles of uh the team, so we have product managers, we have designers, we have engineers, and even within engineering, we had different archetypes

like uh a deep engineer, a tech lead, a product engineer, all of those. So, those roles are becoming very blurry at this point because AI has made things open for anyone. Say, if we look at PM, it's very common now for a PM to go and say that, "Yeah, I have an idea. I'm going to try it out. I'm going to build a prototype and take it

in front of customers." And the same thing with uh design. Oh, I can create a like engineering can easily create a quick prototype using Replet or Figma and things like that. Uh same thing with designer. Even in our cases, the designers are coming up with prototypes and like, "Hey, I can build this something really quick." Of course, there are guardrails. It has to go to production, whether

it's production quality or not, that's a different thing altogether, but at least the prototype and uh building things fast, that is happening. And when it comes to engineering, product engineer, what is what is the definition of product engineer, right? It What it actually means is thinking about customers, understanding customers really well, and looking at the problems to solve, and coming up with creative ways to solve the

problem. It like say if someone is asking for X and Y, then challenging why it is needed uh slice it down and and deploy it and learn from it. Whereas the AI builders, this is a term that we use in Atlassian quite a bit these days. It's nothing but engineers who use AI effectively. So, they're not just the adopters, but pioneers, even pioneers to certain extent. So,

when they both come together, and when I say they both, they're not two different people in general. A lot of times it's the same person who thinks about the problem, who knows the customer, who's creative in coming up with ideas, but also leveraging the AI tools effectively, then that's amazing because they have lot of tools in hand and the throughput is very, very high. How do we

like we talked about throughput, velocity, and all of those, How do we do that? Like what are the ways to do it? I think the first one is opportunity framing, coming up with the right problem statement. It's easier said than done. A lot of times if we see in the software development life cycle, what takes times? Not coding. Coding is just one part of it. Identifying the

right problem, is that the right problem? Or oh, that looks good, but no, no, no, after few months now I'm changing it. But it's okay to change and evolve, but statement, having the right question of why I'm doing what I'm doing, and framing it so that it improves the customer experience in a big way. I think that's the first step of it. We don't need only product

engineers uh sorry, product managers to do it. Product engineers take a big role in that as well. And the second one is Okay, I have come up with this really big problem. Maybe it takes weeks to do it or even months to do it. How do I cut it down into thin slices so that it still is meaningful for the customer, but I can try it out.

I can prototype it. I can take it in front of customers. It can be through experimentation or it can very well be sometimes with the just the prototyping with quantitative feed uh qualitative feedback with customers as well. And uh get feedback faster. So, ship fast. um whenever we slice the problems, a lot of times people don't look at it effectively. They look at it as okay, I'm

going to cut it. Cut it in a vertical way. So, it does few things on one layer, but behind the scenes it doesn't like it's not holistic. It doesn't solve the problem meaningfully. So, recommend looking at it as a workflow. Like does that help with that particular workflow? Is the customer going to get something out of that workflow? That's super critical. And whenever we want to bring

human in the loop, and in many cases we want to bring human in the loop when we leverage AI, it's a thin balance of when we do Cuz you don't want to bring in in every single layer because that becomes too much for the user. Um they they won't be able to get through their workflow fast enough, but at the same time say if it's the shopping

experience you have right from picking the right product to finishing the checkout, you don't want to bring in the problem with the first workflow towards the tail end of it because they have spent quite a bit of time towards it. So, identifying where to bring in human into the workflow and bringing in value over there. That's incredibly important as well. Guardrails. this is a very interesting topic.

So, a lot of companies they say that hey, we want to be creative. We want to be innovative. We want to ship things super However, not every company can do that. Does anyone have thoughts on why some companies are incredibly successful where some are not? There are a few reasons. One, the blast radius of failure is very large. It's and it's hard to find. Or um we

don't know if something will fail and when things fail, it takes a very very long time to recover. And most importantly, we don't we don't have the right type of guardrails and which means the rollback takes a very very long time. So, when that happens, folks are scared to do that and they don't have the safe space to do things fast, to recover fast, and the failures

are very very costly at that point in time. So, that's why this is incredibly important to have the right set of safety nets because without the safety nets, especially when we bring in the agents, the chances of us failing is very high. Not just high, we might not even know that something has failed and that's even worse, right? Later on, you don't want a customer to come

and complain or like an audit few weeks later to come and say that oh, that workflow didn't work as expected. This is wrong. That time the cost to recover is incredibly high. So, having the right type of guardrails, like having the safe defaults, especially when we bring in agents, super important. And then the rollbacks. Same uh when I say rollback, it's not just the feature or capability.

Say if you're introducing AI in the mix. Great, it works. If if that doesn't work, if I have to remove, what does it look like for the rest of the workflow? Does it work? Or are there any other dependencies for that particular agent that's going to fail along with that agent? So, is there a cascading dependency that needs to be thought through? Unless it is thought about

and tested during the building process, it is really hard to recover. So, that's why the rollback and the testing of that becomes incredibly important. The last one is the validation portion of it. We touched a little bit on like why humans are important in the mix because the checks and the validations really critical in this one. Hang on. How do we build those loops, right? Like how

how do we build these patterns? And both applicable for us as developers, as engineers, but along with the systems as when we look at it, the learning resource. We'll talk more about how to pick a problem and go about it, but even if we have introduced AI and agents, I would say look at it every like every week to see how things are going, share wins, but

most importantly the failures as well. in a world of AI, it's not like one-time done and dusted. It works, but are we consistently looking at it? Because one, the models and the capabilities are changing consistently. So, what you did few weeks back, months back, there's definitely a better option at this point that you could explore. And then like there's always opportunity even within that to improve. So,

I would say redefine and look at the prompts and libraries, look for consistent improvement. And also the quality of uh the pipelines, right? So, not just uh not just uh looking for qualitative feedback, but how can you bring in the automated quantitative feedback? Having the right evals in place, how to know that this is working as expected. Is it working most of the time? Is it working

70% not 30%? Is that good enough for me uh for my org? I think all of those are important and having the right automated evaluations is uh really important because anytime that you want to introduce a new change, if you don't have it, then you are relying on um the qualitative feedback to come through, which which takes long, or you have to rely on manual testing to

go through. So, very important to have automated evaluations uh for you to be confident about that you want to roll I'll take a use case uh that we have in as I said, uh I work in Jira, one of the critical part of Jira. So, 70% of usage uh Jira happens in that space. Which means there's Yes, there are issues. Uh customers complain. So, we have issues

coming in. Uh we have bugs, we have suggestions, and we have incidents, all of those. The The pool is large. So, the first step is uh it comes to our queue. Our support engineers, they used to take a look at it. They used to do first level of triaging, and then used to send it to the development teams. What we did first is, okay, when it comes

to the queue, what can be done to help improve? In the intentions side, can we stream the pipeline a bit? Can we try to understand what the customers are saying? Is that an existing issue or is it a suggestion that can be put in a different bucket and how to leverage AI to all summarize the tickets and bucket into it. So, that's the first step that we

try to optimize. Once we know that this is an issue, then triaging it and sending it to the right team, that again is a big thing because for a complex product that has like say 20, 30 different capabilities in a page and multiple teams working on it. If you don't do that, then the bounce rate of the ticket is much higher. What it also does is if

the bounce rate is high, days gets lost because it goes to a different team. It's like a hot potato giving to another team hoping that it's their team who's going to fix it. So, that's the second area where we leverage AI to say that, "Okay, this is a problem and how do I know that this would be the right team or the capability where the problem is?"

and sending it to the right team. So, we have mapped uh the capabilities to the teams and then using the AI agents, we were able to pinpoint the issue and send it to the right team. Uh thereby we reduce the bounce rates. The third one is root cause analysis. Say, we know that, "Okay, this is the area where the problem is." Say for example, comment isn't working

as expected. Okay, if we found where it is, but in the code, what could be the problem? Is it in the experience layer, something to do with the permissions or is it with the database layer? Where the problem is. So, we have created a AI agent which helps with the root cause And it suggests the draft fixes as well. I say still draft because we are not

confident that that would be the right fix and if not done right, it has a potential to become a big issue. So, still does a draft fixes and then of course this is where the human comes in again in the loop to look at the fix and decides how to go about it. okay, I fixed it adding the right type of tests. So feature tests and few

other that we want to look for and most the response back to the customers. Like earlier we used to have like a comms team especially if it's a big changes comms team who is going to write those responses and send it back. I think that has much much easier with AI coming into the picture. Because of all of that which like our average time for us to

resolve the tickets used to be 12 and half days. We were able to reduce it to a day and half for most of the cases. And it's 8% faster which is amazing now we have engineers who can do a different type of work rather than doing those laborious job of it sending it to different team all other stuff. And then just in the few weeks of us

piloting we were able to save close to 170 hours of engineering time for us. So that's just one example of where we have used and how it helped. There's another example. I think lot of you might have seen software development life cycle has been evolving over a period of time. Same thing we what we have done is we have identified pockets where we can leverage AI effectively

and how it can come in and help. I think we talked about coding and feature development especially for smaller features straightforward ones and green field projects we were able to leverage AI very very well. So the time it takes for us to do the first commit to taking it in front of customers have reduced a lot. Um and the areas where we also have seen a lot

of efficiency gains is the repetitive tasks that happens, right? Be it end health tickets, vulnerabilities, accessibilities, those areas where one, it's able to bucketize everything and say that these are the things that need to look at and also come up with fixes as well, which has been super helpful. I think I say code review on quality because uh we also have a huge contribution model to the

teams, which means the contributing teams need to adhere to the right level of standards. If not done right, that has its problems. So, the code reviews need to be incredibly tight and we need to ensure quality along with it. I think that is a very good area where AI can come in and help and also suggest uh improvements if their PR isn't of the right quality there.

And then feature flag. Uh all our changes we put it behind feature flag the blast radius and the time to recover is not is not uh low if we don't have feature flags because deployments take time. What it also does is it creates a lot of feature flags in the system, especially after a time, there's hundreds of stale feature flags and this is a very easy way

for us to leverage AI to say, "Okay, I want to see all the stale feature flags say in 3 months. If no one has leveraged it, can you go ahead and clean it?" Of course, again, we leverage uh human in the mix here to make sure that we are not cleaning something that is uh used uh by us. And that in turn reduces the technical debt, I'm

sure. Oh. Sorry. Okay, what it in turn does is uh it has uh helped reduce the coding and development time, improved the review on the quality cycle, especially for the contributing teams and the teams whom uh manage the code base and help with the testing and the deployment pipelines. So we don't look at AI as someone who is just coming and taking away our work, but most

of the team members look at it as a very helpful teammate who is coming and doing all these tedious work for them while they are looking at the what capabilities should I build, what impact should I create for the customer. So that reduces the friction of should I use AI, should I not use AI, is it going to be right or not. So definitely have seen some

cultural shifts as well in the team and the org. Um, how do we go from here? We talked about few use cases, what has been done and stuff like that. What are the takeaways? How do you start? I would say just take one use case, the thing that you feel um uh like okay, I don't like doing this or this is something that I want to start

with. Just pick one and find reimagine how that can be done using AI and start prototyping it. Just start slow. So I would say have the right type of artifacts, figure out which one you want to start uh uh have regular rituals. Like something that I personally do is I take at least few minutes a day, like even if it is 10 minutes a day, I want

to keep up to it because that compounds. Otherwise, it's very easy for us to get into our day-to-day work and say that I'll get to it later. So I would say take time to learn, um use these rituals. Of course, in teams we have weekly learning reviews, we have ship reviews, and then what we talked about rollback earlier, we try to do rollback drills because you don't

want it to be too tied up with the existing ones and But it doesn't work, we need to recover from it. don't look at just novelty saying like, "Hey, that's the next new shiny thing that I'm going to try on." Look for outcomes because that's the most important one. Without outcomes, it might look shiny, but your adoption is going to be incredibly low. And then, don't do

a lot of POCs as I said, find a thin slice and get things out, try it out. And have a rollback plan as well. Um So, yeah, the end-to-end journey, I would say, pick a problem, slice it very, very thin, have a very clear rollout option, learn, and scale. Um if you want to uh try this in your team or an org, I would say you uh

can talk to your manager, see what job is very laborious, where it can help, or have someone like say someone else who worked on or working on some of the AI capabilities. So, you will be wondering of is it AI? Is Should I build an agent or should I build skills which could tie to the agents? All of those questions are very, very valid questions. Talk to

someone who can who has built start learning and come up with the problem statement which will also help you define what the solution should be and take it from there. So, as we said, start with one-page brief today, tomorrow. Uh don't push it too far if you haven't started. Uh come up with the thin slice, but have the right scaffolding, especially if you're integrating with your existing

uh workflows because without that, uh it is the failures are going to be hard. And then, create the learning ritual. If your team doesn't have one, just go do it. I'm sure your team is going to or your manager and your leadership is going to be very, very happy with it there has to be some catalyst who is going to kick-start it and need to get it

done. Um I don't know how much time we have for questions. Two minutes. Okay. Yeah. No, I just prompted it so that you can think of like any areas you want to reimagine. What blockers do you all uh restricts you from doing it? And then, what could be the starting point? Thanks, everyone. >> [music]