Beyond AI Pilots Operationalising AI in Software Delivery

This talk focuses on operationalizing artificial intelligence in software delivery, highlighting the transition from AI pilots to sustainable practices within organizations. The speakers, Mike Kincl from Acceptor and Alex Lucashv from the Forte Group, emphasize the importance of process adoption over merely introducing AI tools. They share insights from their collaboration and detail a structured approach which involves breaking down problems, using AI tools for automation, and viewing AI agents as partners in the product development life cycle. Key results include significant time savings in prototype creation, improved quality of user stories, and the ability to build connectors rapidly, showcasing the transformative impact of AI on efficiency and project delivery.

All right, we're going to be moving on to our next um presentation today. So, this is from rethinking the role to operationalizing reality here. uh Mike uh Mike Kincl um SCP of engineering at Acceptor and Alex uh Lucashv uh chief AI officer at the Forte Group joining us now talking about beyond AI pilots operationalizing AI in software delivery. Please join us on stage. Warm welcome. All right.

our beautiful faces. So um hey everybody um I think a lot of folks here have played with AI tools and you know deploy deploy I pilots and uh I think the question whether or not AI in delivery whether it's working or not that's not a question anymore. I think a lot of us are interested in where and how I can see all those 10x 5x efficiencies that

are being promised by this new thing that can get us much more productive. um we're making individual contributors faster but if we're zooming out at the team level we're still dealing with the same things um there are QA cues the releases are not getting there and I think one of the key challenges that at least I'm observing um is that organization went about this tool introduction slightly

different way tool first adoption versus process adoption organizations they just buying the tools you give them tools and you're setting up you're setting them up in a system that may not be really ready for those tools to be there just yet. So is this pattern about to change? And I think this is a interesting graph. Most of the orgs that I see right now, they're kind of

in the middle. They're still playing playing with prompt libraries. They're still talking about garments frameworks. But there's a convergence that's happening right now is that technology is already there. There's aic flows. You can actually go from a requirement to delivery with a click of a button. Organizations are catching up. They're looking at the way the processes are structured. They're looking at the way they deliver software first

and applying tools later. And I think this is the biggest shift that I think Mike is going to talk about is that if you do 180 flip, think about what you're doing and why you're doing it and then enabling folks with the tools, I think you're going to get to that 10x, 5x, and whatever it is you're looking for. Um, we've been working with bike for a

while now, and I think the approach that he's going to kind of walk you guys through is databased, facts based, and experience-based. So, I think I'm going to let you make around with that. >> Cool. Thanks, Alex. So, I'm not going to bore you with theory today. I think I'm just going to sort of drown you in practicalities in terms of how we've been working with 40

group to deliver towards an agentic product development life cycle and all the steps that we've gone through as we've as we've gone through that journey. But just quickly 30 seconds on who we are. So acceptor we empower business users within financial institutions to build automated processes that deliver trusted data. We focus on the capital market space and essentially what we offer is a data automation platform powered

by AI that allows business users to automate manual processes. It allows them to connect to any downstream or upstream data source, extract any file, any format, any type of structure and basically provide an automation of that manual journey and then we provide a web interface over on top of that which allows you as a business user to operate what you've built. So if there's any exceptions come

in that you need to deal with, you can handle that. On top of that, we provide a number of products. You can see reconciliations, post trade tax processing that we take to market to sell. And then there's a number of point solutions that we also used built from our platform that help solve other customer problems. So that's just a quick insight into us just so you get

a bit of context. So on our journey, the way we approached this really was just to think about it like any other problem. break it down, start small, look for where you can have a lot of value, high impact really quickly with not a lot of effort. So, we started on the um on the augment phase basically, which most people are probably already in or gone through,

which is let's just get the AI tools out there, get people using them, people using GitHub Copilot for things like reviewing pull requests for example, um getting our UX team to start play playing with prompts for prototyping. Then we moved on to the automate stage. So, it's how do we start to do this more repeatably across that product development life cycle. So, looking at things like test

script generation, which I'm sure most people in the room are already looking at. How can we get AI to help us to build out our test scripts um on a more automated fashion? Um how can we start to use AI to identify exceptions across our logs and start to fix those or propose fixes to those for us. And then what we've done is then we started to

move on to this how do we start to think about agents as a partner. So agents driving our product development life cycle starting to build our software but with the control and governance of having humans in the loop. And that's something that we've been working on with 40 um over the past number of months. So just some practical measures. So some stuff that we've been measuring and

some of the outcomes that we've actually achieved. So again these are point solutions at this point in time. So this is us going out to the different functions, different teams within etc and saying what can you achieve with AI. So in the design space we're using AI a lot to build out our prototypes and we've heard other talks about that today. So our our UX um team

is basically using AI to build out all of its prototypes and now we're seeing 83% time saving in building those prototypes out which is great right but actually the benefit we get from that is we can have way more variations of the types of solutions that we want to build and we can validate those to make sure we land on the right one to help solve our

customer problems. The second piece is around our requirements. So, we've been using point solutions, things like AI prompts to help to build out our features and user stories. And we've seen a 3x improvement in and our user story creation. So, we've really reduced the time it takes to create those artifacts. We've also had really good quality artifacts have been pumped out of that process. So, you can

see that our rework rate in terms of reworking those requirements has dropped from 30% to under 10%. So, some real practical gains there. on the build side until recently we've been mostly focused on adoption because engineers are really tricky people to get to adopt things right they always think they can do better so we've spent a lot of time on that actually since we put I put

these slides together we've probably got about 85% adoption now across um our engineering team and product teams which is really good um on the test side as I mentioned we're doing automated testing in these point solutions at the moment so we're seeing a 75% uh reduction in the time it takes to create end to trained user journey test which is fantastic. When we first started it was

our own prompt. Now we've moved to standardized tooling because there's lots of out there and we're seeing some real gains from that. The one I really like is the one at the end. So we built out a fully autonomous production agent which runs in across all of our customer SAS instances. So 170 plus of those. It looks across all of our logs at the exceptions that get

that get raised. It basically investigates those exceptions. It looks at our codebase. It comes up with a a diagnosis and a proposed fix. It then roots that fix to the right tribe that can pick that up and then we have humans in the loop who can look at those and determine whether it's something that we want to apply to remove that exception. So that's super cool and

that's something that we've sort of had in place for about three months now. So these are our point solutions. Um it hasn't all been rosy. Okay. So you can imagine the typical types of things we've run into. So we've got things like our model quality. So obviously we've had to play around with different models to fight the right fit to get those types of benefits from this

and everything's evolving really quickly. So some of the models now are super smart. So we've always got to keep on top of that. When we built out our production agent, which I talked about a second ago, that was a real pain to start with. We had lots of hallucinations. We had to run it for a long time in shadow mode in our production environments to really understand

where we were going to get any benefits from it. But as we started learned how to make sure the context was right and it got access to more data and the models advanced, we we started to see real benefits out of that. And as I mentioned before, the adoption part has actually been really difficult for us. But the way we approached that was to sort of build

a champions program. So we had champions around the business who were really eager to drive this forward and then they went out to work with a bunch of accelerators across the business to start to implement practical use cases to show value and that really helped to ramp that adoption up. One of the really interesting challenges that wasn't on here actually was one that we encountered a couple

of days ago. I think somebody early in one of the conversations talked about the context you need to give when you're working with the models. We were trying to get AI to build out a process for us using our platform like a real user would. And what we found is when we tried to give it a bit too much context, it did a terrible job. And actually

what we've realized is if we just step back and give it the bare minimum now with these new models, it can do a fantastic job. So just by giving it a link to here's our codebase, here's our documentation, here's one good working example, within a day we had it building an end to-end process in our platform like our business user would. So I think there's a lot

of cool things that are going to come from these models. So that's great. That was all the point solutions, but what do we do with those, right? They're point solutions, right? Across all these different areas of engineering and product. We've got to start to bring these together. So we've put together a vision basically that we've been working through and building with 40 around our AI native PDLC

and it's it has three um ground truths okay and it's based around this central agent ecosystem. So how do again how do we get agents driving the product development life cycle with humans in the loop for that governance layer. So the first ground truth is just simple whatever we build in terms of this ecosystems got to allow us to move lightning fast. That is what we want.

We want to get the most value to customers with the highest level of robustness as quickly as possible. The second one is control and governance. You've always got to have that. So governance on the agents that you build out, but also governance across the artifacts because you got good data in, you get good outputs coming out the other side. So that's really critical. And the third part

is called central agent ecosystem. But it's really the user experience because as this AI native PDLC sort of evolves, you've got to have humans in the loop and they've got to feel comfortable interacting in that agent ecosystem. Okay, if you're going to get adoption for it. So that's the third ground truth that we're holding ourselves to. So in terms of some practicalities, what are we actually doing?

Okay. So, we are utilizing things like anthropics um agent system so we can build out sub agents. We're building out skills. We're building out prompts along with 40 across all the different areas of our product development life cycle. Okay. So, you can see on there I've got things like product ownership, dev, QA, DevOps. I've put those on there because they're just easier to talk about. There's actually

more granular prompts and skills that go inside those to make this a reality. But by using things like anthrop anthropics agent architecture, we can parallelize a lot of this work. So we have sub agents working across those different areas of that life cycle. So we can move even faster. And that's something that we're finding really powerful at the moment. So what does this mean for roles? So

right now kind of what we've seen in terms of the way our roles are transforming is that if you look at things like a product owner so where before they were creating all the user stories all of the features obviously now they're shifting right so they're orchestrating they're guiding they're editing so they're providing input into the agent ecosystem and the agents basically building out all of our

requirements or features in our user stories using the standard templates that we've put in place um and then they're doing any adjustments and that goes the same across all of our rules but But as this a agency ecosystem is now evolving and maturing, these roles are starting to combine over time. So what we're going to start seeing is we're going to start seeing less roles but more

specialist skills across those and that's going to basically mean that our teams are going to shrink over time. Okay? And we're going to probably have more of those looking at specialists of requirements that we want to build out. So how are we actually proving this agent ecosystem out this AI PDLC? So what we're doing is we're we've been working through um we did an initial concept where

we scaled out an initial um MVP quite quickly. That was our first pass at this where we were building out a connector and a connector for us is just an upstream system that we want to connect into. So some small isolated service piece of code that we can build that isn't too complex. um and we sort of built that against an existing connector to sort of prove

out what it was going to build so we had something to compare it against and it did a really good job. So from that point what we did is we then went to enhance that product development life cycle. We set ourselves a six a six week challenge which were already halfway through which is we want to be able to build a connector or multiple connectors in under

a day. Okay. Whereas before it might have taken us multiple weeks to achieve that. So we're already seeing that we're making some real progress towards that. We're pretty confident we're going to hit that over the next couple of weeks. So that's super exciting. So what does this actually mean in terms of bottom line? Well, as this agent ecosystem evolves, I think what we're seeing is actually what

would have taken weeks or months to deliver potentially large features or um user stories is now getting converted into hours. And we're proving that through our connector use case I mentioned a second ago. We're already seeing really massive benefits in terms of the quality that we're getting out of this process. Um but actually we see that improving further. So our production agent um our ability to be

able to build out end to-end test cases really really quickly. That's just improving our confidence across all of those particular elements of that product development life cycle. Plus the agents building out now our artifacts. we get really consistent requirements and things coming out of that too whereas before they were they were quite patchy as they're being developed across the different teams. And the third piece is really

staying competitive. So actually we don't see ourselves having to grow our teams linearly as we scale as a business. Now we're going to think about what does a reorg look like for us and we actually see ourselves streamlining. Doesn't mean we're not going to grow. Doesn't mean we're not going to add more people but actually we're going to be able to get more done with less. cool.

So, that's that's it for today. So, um obviously working with 40 group today. So, we're going to spend some time down at the booth downstairs. So, if people have got any questions, they can they can come and ask and Alex and I will be happy to have a conversation and hopefully you can teach us a few things as well. Love to share some knowledge. Thanks.