About this talk
This talk discusses the implementation and practical applications of AI within the context of software development at Current Agency. The speakers highlight the challenges organizations face when adopting AI technologies, emphasizing that while AI can be beneficial, improper implementation leads to wasted resources and diminished trust among teams. They introduce 'Tessa', an AI tool designed to streamline the generation of Jira tickets from project requirements, significantly improving efficiency for project managers and developers. The session showcases how Tessa supports the development process by automating mundane tasks while ensuring human oversight remains integral to maintain quality. The speakers also discuss the importance of evaluating AI candidates based on specific criteria, focusing on high-volume, repetitive tasks that benefit from structured patterns.
Full transcript
All right, folks. It's uh it's 3 o'clock and uh we're going to go ahead and get started here. Thanks for uh coming to the session today. We're super excited about it. Um we are going to be talking today uh about AI. Now, I know you're shocked by this. Um I hope that's what you came for, but we're actually going to not just talk about it. We're going
to show you examples of real things that we've built and that we're actively using in production. Uh my name is Mark Stroshshire and uh work at uh Current Agency and super excited to be here. Um I am located about 15 miles northeast of Charlotte, North Carolina. >> And I'm Mever. I also work from Current and uh I'm based just outside of Athens, Georgia. And I'm really excited
to be So what we want to talk to you guys today a little bit about is just sort of some of the things that we've noticed and how this has shaped our approach and philosophy to what we do with AI. And so what we've noticed um is that companies are are racing to implement AI, buying up licenses, launching pilots, implementing every next thing that kind of comes
along. And for teams who are looking to use AI, this can actually be a really great thing. But if we get into real return on investment, really heavy implementations, what we're noticing is a wasted spend, uh sometimes frustrated teams, increasing skepticism across the market, um and not a ton of showable output. like there's some cool stuff being done, but what's being used in production and what we
think and what we found from our approach that the technology really works fine. We think maybe the approach is a little off and that's what we want to talk a little bit about today. And so the the hype trap is real. Uh it comes from every direction, vendors, press, company mandates, whatever that may be. So you're going to hear push AI, push AI, push AI. And this
comes with a cost and it's not just wasted money. It can also be organizational trust or client trust. So if you have AI fail to produce something or it's draining a budget, then your teams, your clients really stop believing in it and don't want to work with it. And so not all the time do we see this, but we we'll hear, you know, they bought the tool
before they really defined that problem. What are we trying to solve for? How are we recognizing what patterns we need uh to use AI for? And so we think that AI can really be a precision instrument. We don't think it's a silver bullet. And that's what we're going to talk a little bit about today. So I'm going to hand it to TH. >> All right. So um
one of the things that we have a point of view on is looking at AI and approaching it from this sweet spot there. There we believe there's a sweet spot in the middle between all of that hype that Mike was just talking about and and however you define it, the AI fear, the AI doom, all of that kind of stuff. Because I I'm I don't know what's
going to happen in the future. I'm not predicting that stuff. But what I do know is I can evaluate tools today, build things with them, uh show success, show improvements, iterate on those and um and I can say, "Hey, this worked for us and move forward from there." So there is a sweet spot and it it isn't really necessarily compromised. Um but you know, in five years,
u it's hard to imagine what what really would uh AI would be like, but um but anyway, we're excited to show off today what we've built. Uh the product that we're showing off is Tessa. We'll talk some more about it. Um but we think we built it in a reasonable and practical manner. So digging in a little bit, um everything we show today is actually in production.
We have teams actually using this. These are real tools, real projects. We've got real results to show for those. Um so we will we will have demos. I promise we'll get to those. We're excited about it. Um, and we'll cover how we architect and evaluate and ship all of those products uh, uh, that are actually used in enterprise AI tools. Um, and again, Tesa is our primary
use case. We're super excited about this tool. Uh, it's used internally. Mike, it was your idea to for Tessa to start with and uh, and I I think the whole the whole company has really benefited from it. So, we've been it's been exciting to be a part of it. All right. So, these are some terms that uh you may have heard and these are terms that are
worth knowing. Uh they're not the only terms, but there's some things that we'll talk about today. So, just want to kind of throw some of these out there. You can you can obviously uh look them up and read more about them, but um but anything from uh APIs that's still applicable. We've worked with APIs for years all the way through uh MCPs which uh allow us to
connect tools together. Uh but one of the things I'll call out on this list is is work slop or you hear the term AI slop and that's kind of uh that's kind of the the the enemy of so to speak here. That's what we're trying to avoid. We want to make sure that things uh through us having human intervention that our products uh still have our touch
and our creativity involved so that it's not just AI generated and uh is great to work with. All right. So, um we're going to basically show you some of what we built. Get ready for that. But we need to talk about uh a little bit more about why AI projects fail. And there's no need to linger. So we'll just kind of jump right into it. So most
of the AI failures that you see, they they come from automated the wrong things. Uh it's really about judgment. It's about looking at um you know what should stay in the hands of us as humans, right? What should we still be involved with and working with and what what should be automated? Uh what what's the mundane? What's the things that just frankly that in each role that
the person in that role really does not care about doing uh as much? Maybe it's a thing that doesn't make them happy. They're not passionate about it. Um so we want to let AI handle those um those things that are predictable and they're repeatable. Uh let humans focus on things that elevate allow them to elevate and have more expertise and use their expertise. So um all the
tools that we're going to show today use these principles. uh and that's really the main thought line through this uh through this talk today. So the same data that's used to identify a good AI candidate uh which is a lot of the volume and the patterns and the defined output. It's exactly what we would use to measure ROI after you build it. So you can measure ROI
using um AI and uh I think that's that's amazing. We want to know like what can we save and efficiency. Um and then there's still going to be the normal intrinsic values and things that go along with improving the quality of our work and our experience in our jobs and in our roles and things like that. So if it's low volume, if it's if it's a lot
of variance, if it requires a lot of uh judgment, then that's probably something you don't want to do with AI uh as of today, right? So so don't worry about building that, but you can build the things around it. Um, so before we have that next AI conversation, and I know it happens because I've already heard this week someone say, you know, I'm building something. I need
to build something at work and I've just been told I need to build an AI tool, but they they didn't have any kind of guidance on what that would be. So, they've got to come come up with a AI tool and uh I don't think that's the right way to approach it. That's not how we approach it. We approach it as our teams talking about these things
and what's needed uh out in the open. So, um, but make sure that when you're having that conversation with, uh, with a client, with, you know, your leadership, with your teams, make sure to go over those those types of questions like, is this a good candidate? And and and what's the basis for that? >> So, we put that into practice uh with our our product we're going
to talked about today called Tessa. Um, and really this is how we we just sort of thought about our philosophy and said, well, what's a what's a repeatable pattern that we have within our work that we wanted to apply this philosophy to and then see something built that could have a real measurable ROI. And so we're going to talk a little bit about what Tessra is, sort
of the case study and what the return on investment was as well as give some demos, but also talk about like how does this apply to Drupal and what does this mean for our Drupal work as we uh use it in production as Shrop said. So let's talk through that case study and really think about Tessra. What is it? What does it do? And again, I told
you this is about how do we use this in Drupal? And so we think of things with AI as not everything really has to be about AI in Drupal, but it can also be about how AI can support our Drupal work. And so for Tesra, we we built this system to help our lead devs and help our producers on our projects including our Drupal projects. So what
is Tesra? It's we thought about this as every project, every problem that we run into, it's a measure of requirements that we gather from our require uh our clients and talk to them about okay, what's a new feature? What's a new thing that you want to build? What is this that we want to do? Well, that needs to translate from requirements into tickets that become actionable user
stories. Those user stories and tickets need to follow certain patterns and templates to make sure they have acceptance criteria. Make sure all the information is there for a dev to work on. And then all of that information gets distilled down to those tickets, handed to the dev, and the devs can then focus on that, understand it, and say, "Okay, how do we build this?" Right? That that
happens every single time we do a feature. And so if you think about that, that's taking all this information that we have in our heads from a requirement standpoint and distilling it down into just organized buckets and drawers. So, we're like, you know what? We could do is brain dump to something and have it reorganize that into these buckets for us and create those tickets because why
should we have our producers, our project managers, and our lead devs spending hours of time refining tickets when they could instead be focused on building those relationships with clients or working on the work that we're doing within Drupal or with any project that we're working on and do the cool stuff that everyone here does, right? So, that's Tesser. That's where Tesser comes in. And so we really
thought about our philosophy. How do we structure this in a way to make it repeatable? Make it something that is a good test case that we want to make sure, hey, this is something we can sort of reimplement as a framework when it comes to any of these problems, not just test era. So we thought about this. Well, we want to figure out what is our agent
layer. How do we swap out the different LLMs that might power it? Because we want to be able to test different ones. We want to see as the ups and downs and updates of these different systems that are constantly competing with each other are needing to be able to be swappable um as that landscape evolves. We want to make sure that our output is always going to
be the same with Jira since that is our um agile framework system that we use for ticketing. Um and then how do we orchestrate all this with just you know taking in that brain dump make sure that all of it is getting put together distilled and then outputed to Jira. And so that that really presented us with a challenge of we got to find the right tools
to make sure that we could do this and approach this in the right way. And so again we kept thinking about what is our philosophy here and this is the real highlight here. We want to make sure that we can let AI um in this case Tesra produce the output and and do that mundane. It repeats that process but we keep the human in the loop to
make sure that what it's outputting is not work. If I put in a bunch of details and brain dumps, I want to make sure the tickets that are created are the actual user stories that we care about, not something it made up or hallucinated. So again, that's why this testing process was really important. That's why our philosophy of let's try this, make things composable, make things swappable,
all of these things, version what we're doing from a prompt standpoint. We'll talk about that in just a minute. This these were the important parts of making this an actual producible thing. It wasn't just dive in, make the thing. All right, guys. We're ready to go. this is going in production. We wanted to make sure that we had test cases. We knew features that were coming out.
We could test this against different projects as we did this and and evolve the product to make it into what it is today that we actually used. >> And and who really raise your hands. It's it's okay if you do enjoy it because that's fine. We celebrate that. But who enjoys looking at the blinking cursor and writing a new epic and a story or so we've got
somebody and it's okay. It's okay because there's everybody's different and we all enjoy different things, but a lot of people really struggle with that blank page syndrome and that's that's really what Tessa solves for us. We we can help people get past that, but um but thank you for uh being being honest on that. >> So what was the return on investment for this one once we
shipped it? And it's really sort of three main core things. And we we kind of realized this is what we want to f these three things are the things we want to focus on for any AI product that we kind of put out in production for our team. The first was speed. We can kick off sprints faster, get into the work that we're doing faster because we're
not spending that time thinking about and distilling out all of the thought processes to go, okay, let me think about each of these requirements. What does that mean? What are the things? What are the user stories? The acceptance criteria. We can let the thing that has been trained to do that do that faster for us. And that also means we can keep project consistency. If you know
how many folks here have ticket best practices determined or you know explicitly drawn out for your companies, those are things that you might have to make sure you're stepping through and following for every one of your projects to have consistency. Well, in this case, if all of our tickets are coming from the same system and it knows our ticket best practices, then you're going to have that
consistency across all your projects. That allows context switching and make things a little bit easier for your devs to go from project to project. your QA members know what to expect when they're looking at tickets. It's going to be the the template or the style that they're expecting and the information they want there. And then ultimately, and I think the biggest piece here that we really were
driving at, the thing that really foundationally drove us to do this is decreasing that time for tech leads and project managers to have to write tickets. We really want them to focus on the things that they're good at. We want our devs to do the amazing things that devs do. We want our project managers to focus on building that relationship with their clients rather than sitting there
and just going, I'm going to slog through ticket creation for the next three hours. And so that's where Tessa came in. We're going to show this off in just a second. And I promise we have demos, but uh we're looking through uh oh. All right. We're looking through a couple of different versions that we've done. And as I said, we've iterated on this. This is not something
we just made and shipped. We we went through we versioned our prompts. We tested the context. We tested the rules. We tested different LLMs. And so we have two full versions that we've been working with. One of them that's been in use for a long time. The other one is one of our newer uh ones that's really the same architecture, the same context. We just changed it
to a different agent layer and a different interface to see what we could do and get some, you know, some new tooling and things like that to continue to evolve the product. But let's let's demo it. Let's see it. I we keep promising all these things and I want to make sure that we actually can show it off. >> We got we got to show this stuff.
So, all right. So, we're going to jump in here and uh we're going to just talk through as the demo runs. These are demo videos and I'll try what I'll try to do here is zoom in a little bit on the video. So, I'll do my best on that. But, we're going to show you how these are built. This first uh the first one is built using
a a tech um product called Elix. And it's really cool because it's built for enterprises to allow you to prevent it gives you governance. It allows you to prevent excfiltration of data that you don't want to leave, you know, the corporation and things like that. U and it gives you uh you know some some great guardrails, but it also lets you connect up to systems that you
have inhouse, whether it's, you know, Elastian products and other systems, uh Google Workspaces and and let you use that as a context layer uh to give contacts. And that's actually you you'll see this in a little bit, but that's part of it is we we actually partnered with our producing our producer team to to help build out all of the um content that drives this is how
we make tickets. These are the best practices for what a great Jira epic and story and task and subtask looks like which is really critical because if we can't define that as humans, if we can't even do that ourselves, then how can we expect to automate that or have a machine do it? >> Yep. So, we'll show off some of the the context and rules that we
have in Elvex. And one of the things that was important to us when we first started this, like I said, is that that composability or swappability of different uh agentic layers. And so, as we start showing off some of this, um this is Elx. It should look fairly familiar to anyone who's used like Claude or Chat GBT. Um they have a very similar interface. But what we
did is we gave it a a set of rules uh to go ahead and define what it is. tell it just like you would in a project in claude or chatbt uh how you know what are you what do you do how what are the things that you should follow um and then we can we have some further uh configuration down here and this is pretty extensive
uh to >> we let's be honest I mean we worked hard on this and and anybody that tells you like if you watch uh a YouTube video that says you know hey in 10 minutes you'll make $10,000 a day kind of thing I mean I maybe that happens to somebody I don't know but Everything we built takes time and it t it takes effort. So, >> but
here's where we have some of the different uh pieces where we can connect up different data sources. We we connected a a Google Drive where we have some documents that were um our data source to to power this whole thing. And and so we're going to show here in just a second. Um there's a few different files that as Shrop said we worked with our producers and
project managers to make sure that we were following what are some of the best practices that we wanted to document and give context to the system and the agent. What are some dos and don'ts that we want? Like what are things that we don't want in tickets? Uh one big thing that we had to iterate through is some of the early phases of Tesra is making sure
like hey don't hallucinate what the technical details are. This is purely about user stories. How do we think about the features being interacted with, not how do we architect them? We want to leave that to the devs. Um, but finally, there's some other pieces in here too and other configurations uh where we can actually hook up different LLM. So, we tried uh hooking up with OpenAI, hooking
up with uh Claude Opus and a number of different agent layers just to be able to test which one did we have the best results in. All right, we'll jump to the next one I think this is the just the same one. Okay. >> All right. So, we want to show off test in action a little bit. And so, again, this is Elvex. And you can see
here what we're going to be doing is is kind of brain dumping a potential feature. >> And this is it's basically an events feature for Drupal 11. So imagine an existing Drupal site and a client comes to you and says we need an events listing and here's our you know you worked out the requirements and you've got that that high level that's what uh we just provided
to uh to Elvix here to generate this and and just keep in mind we believe in using other tools to help build these things. So this prompt is actually built from another uh tool you know you can use other large language models to help build prompts. Um and so this is uh yeah plugged in. So this is going into an existing Drupal site >> and in addition
to that like this is just a basic prompt but we've tested things like providing documents. So like a client provides us a requirements doc that you know details a bunch of things out we we can give it extra context of course but we can provide that document as well. Uh we can provide screenshots things like that. Anything to give it context because truly this is about a
brain dump. This is think of it as you've been working with a client. They tell you there's a feature that they want. you've got all of this information. Well, that's just dump away over to the to the agent here. And what it's going to do is it's going to look through this and we've got in our context uh a decision logic tree where it it it was
kind of parsing through, okay, think about this information. Is this a bug? No, then it's not a bug ticket. Is this a feature? Okay, then let's start thinking about this as an epic. What are the different interactions that we have detailed out in our notes? Each one of those are going to be a user story. So how do we define that? And so all that ticket logic
is what kind of powers this whole system to say, okay, let me think about these subtasks. Let me think about these stories. Let me think about these tasks, these bugs, whatever that may be. And what it's doing is it's following our ticket templates and best practices to say, okay, if we have a user story, then it has to have a story with it. It has to have
acceptance criteria with it. We need to know how to test it. We need to know how to work with that. Um, and so that's what uh Elix provided for us. >> Yeah. and and I think because of the fact that we work with so many different clients and organizations the the ability to make sure the tickets are consistent it would is that that's I mean Mike that's
critical we can't we can't not do that so so that that was really important um so we will jump to our next demo here and this this demo is exciting this is getting into the V2 piece um for it and this is Um yeah, you want to this is we're going to show so we're going to show the configuration in V2. Uh we use Velvix for that
that first demo. This one is built in uh in cloud. So anybody anybody use cloud anthropic cloud with the app. >> awesome. >> Yeah, that's great. I mean we we we really are happy with a lot of what uh that tool is doing for us and so we we wanted to build it and see and be able to contrast the differences. So, one of the things we
noted with with Claude is it had a a really great Atlassian connector um that we'll show here. Um and you can kind of see that in the instructions piece. There's a couple of connectors there where we show there's an Atlassian connector and a Google Drive connector. Um so, in in the same case here, we have memory turned on and some different configurations and features, but again, we
have our instructions to the project here and to create sort of this micro agent um to create Tesro V2. And in this case, we connected it directly through Atlassian's MCP that Claude has access to so that we can build those tickets directly in Jira as opposed to having to draft them, take them, and then having folks copy and paste them over from Elvex. Um, but in this
case, we did uh make a a a big change, I would say, from a context standpoint. Um, that if you want to play it again, we I think we can sh show it here for just a second. um we gave it a bunch of context files and you can see that here in the bottom these are all MD files so markdown files and something that we've noted
and noticed in our research and and approaching this philosophy with creating these different products is that markdown is really well consumed by AI and so we we're testing with V2 how well does it consume the context as opposed to being a Google doc we've made these markdown files and the benefit of that too as I said we want to be able to version everything that we're doing.
So we can keep these markdown files in a repository and as we make changes to them, we will go ahead and make commits to those changes and we can actually document how is our product getting better or how is our product evolving based on these changes. >> And what's really cool is you can actually let's say that we committed that repo up to like GitHub or another
repository like that. we can actually just point for context just point directly to GitHub and have it pull in that context from there you know versus Google Docs and and I think the plain text seems to help in our testing it just makes it simpler it's not the large language model is not trying to process some type of rich text with a lot of markup in it
and so here is tester v2 in claude and one of the big things that we wanted to change here is the ability to have more interaction more of that human in the loop that is part of our philosophy to continue to engage engage with whoever's working with this and say, "Okay, I I'm hearing you. I'm hearing your brain dump, but let me ask some questions that might
be a clarifying question. How do I think about the the these features to make sure that I'm thinking about this in the right way? Are these user stories the correct user stories that you would think should be here? So, you've already brain dumped to me. I'm validating that. And then let me go ahead and I'm going to work on it." And the beauty of this is it's
going to go through. It's going to do the same thing. It's going to draft it up. in this case is markdown because that's what the Alastian MCP is expecting. And then it's going to actually go through and and ask you, hey, how does this look? I want to keep you in that loop. We want to make sure that we're helping automate this uh stuff to make sure
that we're we're taking out that mundane, but we've got to have that human gate. We've got to make sure that we're avoiding that work slop, so to speak, and make sure that again, this is going to be an actual product, an actual output that we care about. And that's what the this confirmation uh screen is really all about. And it's created like an epic and like a
bunch of stories and tasks already. And and this is the exciting part everybody. I told you there'd be excitement here. You look excited. Thank you. It is now generating the the Jura tickets. So this is really this is really cool. Uh we're excited about it because uh no one's I tell you who's really excited is our producers who aren't copy pasting from a large model into Jira.
So this is thanks to being connected straight to to Jira. And once it's gone through everything, it'll confirm it with you. It'll tell you what you might still need and what it's going to be going and updating. It does take a second for it to run through, but that's just I think limitations right now on the MCP with it lasting that we've noticed. Um, but again, this
is let's think back to our philosophy of what we've been talking about. We wanted to automate the mundane. We wanted to be able to take in and make sure humans are in the loop to be able to validate that process and then have an actual output. And that's what Tes V2's done for us here. And so our last demo here is showing you just what is that
output uh that we wanted to do because we've already automated the mundane through being able to brain dump to an agent. Uh we've added the human in the loop from our philosophy from being able to do some of those quality gates and reviewing and approving those tickets before they get created. And then our actual tangible output here are real tickets. Tickets that can actually be used and
actioned upon by our devs, by our project managers. We can validate these with clients if we're doing that. It looks through all the different pieces. It creates those stories, those tasks, those subtasks, and we have our acceptance criteria and our user >> And that was an epic. And then this is the story that we're we're going through. And then within the story, you can see the uh
the tasks or subtasks right here. And so click into that. And again, you see, you know, the acceptance criteria, the template being observed. Um, and we're we're still iterating on um, you know, the the actually getting the testing instructions to plug in. Um, but we already had that working in uh, V1. It was just getting it now into into V2. But, but that's exciting. I feel like
that deserves a round of applause. I don't know. What do y'all think? Does that feel like I'm forcing the crowd to applause? But I I don't know that we're excited about it. Um, I hope you are, too. This is really what we believe is um is kind of the exciting part of our future and being in that uh that central really uh place of of calm where
you can say listen we can use tools today and move things forward and it's exciting. It's a great time to great time to be alive for that. >> So Sean's going to talk a little bit about like where do we see this going? What does this mean for our Drupal work and how do we how do we how do we evolve it further? I >> I am
so excited about this next part. Um this is this is actually a part that we haven't built yet, but I'm really excited um to to have the team work on this next. Uh this was this is a concept of how you can take this further. So we're already you you've seen everything we're doing. We're uh we're taking inputs the new feature request in uh and it will
go into u uh some type of input. Uh, and I'll explain the NAND connection here in just a second, but uh, but we take that new feature request in and Tesla processes it and then gives us Jira tickets, right? That's that's kind of the high level. But what we're really excited about is the idea that why why go why stop there? Why not go ahead and connect
um, Tessa up to the actual systems you're building off of so that it has context of of an existing site. So if we're building to an existing site, uh Drupal as an example, uh has JSON API, you can scrape the website, you can turn on GraphQL potentially if you install the modules. There's other ways to get data out. So why not say hey we want to build
this event piece but there is uh but here's extra context to take in mind uh of content models fields entity relationships that exist and allow the uh the machine basically to give us things like oh you've already got a location field you could reuse that location field on the new u on the new piece. So, so what we what we're showing here is that um with Tessera
V1, we would need with this is with the Elvix setup, we would need something like NAN, which is like a visual workflow tool to build. Anybody used NAD in um in the group here? So, a couple hands. Cool. It's a it's a neat tool. It's it is open source um and they have a cloud uh situation where you can pay for uh for NAD, but we would
need NAD or something like that. So we can handle the fetching of the data from Drupal. Pull that in, process it, trim it down a little bit, and then feed it to Tessa. Um, and imagine imagine a a red X over Nadin. In V2 world, we think I mean we've been talking about this this week. We think you wouldn't even need Nad with uh with uh Claude
because Claude, we believe we could wire up MCP straight into Drupal and and do that. and we can scrape um using tools. Uh there's a bunch of web scrapers now. They're available. Um so we can do that. So we won't even need that middle section. It greatly simplifies uh the process. But um but yeah, we're we're excited about that. And that's that's the that's a great thing
with Drupal. But beyond Drupal, there's uh you could do the same really with WordPress, Contentful, any any CMS and and really any system that has an API that you need to give context >> especially with structured data. >> Absolutely. And so today, we promised we would leave you guys with something today. And and so now that you've seen this in action, let's really talk about how do
you build something yourself with [snorts] this? How do you take this philosophy and turn it into a framework that anyone here could use? And so that's really what we want to give you today and and talk about is just before you build anything, run it through this framework. Think about this criteria. What makes a strong candidate versus a poor candidate of what you can use with AI?
Uh, and again, this is where I was talking about the AI hype trap, right? Like everyone wants to go, let's do everything in AI. Let's go for it. That's exciting. And there's some cool stuff being done out there. I don't want to uh stop the hype necessarily, so to speak. But I want us to be able to take a step back and say, how do we do
this right and get a real return on investment so that people can actually use AI in in a way that we found to be very valuable based on what you guys just saw with Tesa. And so our strong candidates are really again thinking about what are things that happen all the time. Those high volume repetitive tasks. Uh what are things that have clear patterns? There are things
that have a lot of structured data or structured context with them. They have very similar inputs, very similar ideas. And you might go, "Hey, wait, hold on. You just said we're we're brain dumping requirements into this agent and creating Jira tickets. That's not similar outputs." But if you think about it and take a step back, it it really is. It's it's the same idea of just there's
a feature let's distill this and we can think about that as a similar input from a it's the similar process of how we take uh feature requirements and turn them into tickets. So that's the similarity here that we identified for Tessa. And then ultimately what's your defined output that you really want to have uh created? What is something that we really are going to value the output
of the AI agent to make sure that we're getting that return on investment? And then again, we want to keep that as low judgment as possible. What are things that we're not letting AI make the decisions on and let it more take on the task of doing? And so if we flip that script and we think about it or flip that coin I should say and think
about what are poor candidates, it's really the opposite of all those things. If we have really uh oneoffs or things like that, those are low volume. We don't want that to be there. Things that have a lot of variance on them. There's not a whole lot of similarities between the structures or the things that we're doing. There's not really an output to it there. Like why are
we using AI? What are we actually outputting to it? Or things that take a lot of judgment. things that everyone here at our tables and and listening, we're the ones who have to make those decisions. That should not fall on AI. >> And that's something with uh Tesser I think that's important. We didn't really state it, but we hope it was implied. So, we'll we'll state it
anyway, I guess. But, you know, once these tickets are defined in Jera, that's that's not like all there is. I mean, you still need to have folks take a pass at that and make sure like does it make sense? Is it right? Like so that's still room for human in the loop which is really important but it but the amount of time saved think about all the
other things that that that you know a producer uh a strategist a designer an engineer could be doing while while it's building all these things. So uh you noticed even a few times it it stopped and it asked questions in the demos and we were you know we follow up. We're working to to make that even more seamless. We love that it it's it's asking to be
nudged along like agents seem to do, but at the same time, um I think we all want to get to a place where we can let agents just kind of do their thing, finish the process, and then come back and check in with us when truly it's it's thinks it's completed. >> I said it thinks, you know, name your agents. I'm I'm a proponent of naming agents.
But um but yeah, it's it's a lot of fun. And and I'll I'll leave I'll leave this framework with this final thought. And this is I think what gave birth to a lot of this thinking and our philosophy, but it was was a couple years ago that I was talking with Sharp and we were talking about how do we want to use AI? How do we be
cautious about it, but yet still be at the forefront of it? How do we make sure that we're not falling behind the curve? And and Sharp said something that really stuck with me, which is automate the the mundane so that we can focus on the fun. And I think that's if there's anything that you guys take away from that today is automate the mundane. That's what we're
thinking about. >> Also have fun. >> Yeah. And have fun with it like at the end of the day. And it was it was fun to make test, but it just it was one of those things that we wanted to make sure our devs could have fun doing the things that they are really good at doing. So we wanted to automate the mundane to take that off
of their plate. >> Yeah. and and also just I I think all of that plus we still leave room for technical the technical uh prescription of of the tasks like we do think AI can give a technical uh prescription of uh of these tasks that are generated but at the same time we've got really smart folks who uh who are current who can actually solve the problems
and say no this is how we solve it but the framework is built out in the tickets and the epics and all >> right well so ch how do how does everyone here approach So, uh, yeah, I love this slide. This is, uh, you know, I mean, you don't want to learn the hard way. You don't want to, uh, spend, uh, time kind of focusing on the
wrong things. Um, but yeah, you you first of all, it these things have always been true, right? Like you always want to be able to measure uh, impact. You always want to have data. If we're not I mean, we've said that for how many years in the Drupal community. If you don't start with data and have and collect analytics or you know whatever forms of data you
can to see if you're doing things well uh or or maybe not so well then then you're then it's kind of a problem. So the same thing applies here uh because here's the thing that a lot of folks don't talk about there is a cost to running all these agents and systems and all that. It's you may not see it because maybe you don't pay for it.
maybe your organization, the the agency you work for, what pays for it, but it is costing. Someone's having to pay that bill with their credit card and and it can it can add up. So, you want to make sure that you're making the most out of the dollars and all that spent. Um, and then, you know, maintain like it's code. I mean, that gets back to Mike
talking about committing uh any of the files and things to repositories. And the thing that's really um I feel like I've I've been mentioning this to a number of people I've talked to this week at Drupal Con, but I it's I think it's so important and that's where we're at at current is actually empowering the entire company to just build and be able to allow engineering and
tech folks to step back and support the building. Like we do not need to gatekeep anything. let let people build stuff and then we're there to support get stuff in repos, make sure it's scalable, it's maintainable uh because some of these things that are built might be throwaways and some of these things like Tessa are going to become critical to the operation and you just you may
improve it and all that and iterate but you you're not going to want to get rid of it. Um and then yeah, the last thing really I'm kind of ignoring the the things not to do because it's obvious, right? But the the keeping things modular is is is uh is so important because like Mike was saying earlier uh there there will be new models, new models are
rolling out all the time. There will be new models to try. There will be new ways to add context. There will be things that are faster, cheaper, all of that stuff um that that comes along. So you just each piece of your um all of the orchestration layers, you want to make sure that you can swap out any of those pieces at any time as as easy
as possible. So that's again that's an opportunity for engineering uh with your teams to be thinking about and making sure that that's planned for. But again empowering everybody to go build >> So thank you everyone for listening to us ramble and we really appreciate y'all showing up. But hopefully >> it's great. >> I thank you. That's it. [applause] >> So feel free to scan the QR code,
leave us feedback, but does anyone have any questions? Yeah, let's open it up. Oh, wow. >> How long did it take to get here? >> Uh to get here. Um I would say we've been iterating on Tesra for um about a year a year. >> But to be honest, like the first version of Tesra took us maybe two days to put together and then we just we
kept we would have a few producers come in, use it, test it, give us feedback. We spend a day or two to make some improvements and rinse and repeat just like you would any other project where there's like a sprint cycle or an agile process. That's kind of how we approached it over this past year. >> Yes. >> Um is this coupled to Jira or are there
other systems that we >> test? Uh is it coupled to Jira? Uh yes, Test V2 is uh coupled directly to our Atlassian Jira instance. Um now there are other MCPs out there. So it really just depends on what available connectors are out there. I think that highlights some of the reasons why we want to keep things uh swappable or you know composable where we can you know
plug and play those features as needed. But in this case ours is just directly uh tied to our Jira. >> But if we if we drop Jira and we decided to go to line or to test what >> yeah any other tool you smile >> Drupal >> Drupal. All right. if you want if you want to build a ticket system in Drupal. Controversial statement, but we'll go
with [laughter] that. Um, but you know, you as long if it has MCP or APIs that you know that can wire up, then yeah, you could replace it with that. It's the same it's more the principle and concepts that we're focused on and that's really what we want to talk about today. But that's that's a great cover. >> But no, uh, so the limitation is on the
it's on the tool side, MCP side. >> Correct. Got it. >> Yeah, the MCP just makes it a lot easier. your MCP is not the only way to to solve the connections. You know, Google got A2A and some other tools to help. They're different, but they help uh connect uh this disparit systems, but the but MCP is uh is I think Anthropic may have been behind that
protocol if I remember right. And so they they really fully support MCP. >> What other questions? >> Thought I saw a hand over here somewhere. Andy, >> what what's the security layer in between like like a is there like an OOTH connection or something to make the connection to? Um >> Oh, it is OOTH. It is O. Absolutely. Yeah. So, so specifically for Jer and it's different
for different systems, but for Jira specifically, yeah, it's going to require OOTH and when you connect that MCP, it requires you to like pick which instance. So if there's multiple instances and it's tied to the account, I don't know if you noticed, but like we were talking about that today to mention that. >> So yeah, one of the cool things about that because it's through OOTH, it
it recognizes what user is the one using it. So if is say shop is the one creating tickets in Tessa, all the tickets that get created by Tesa are created through shop. >> And so if I'm doing >> it's my accountability, right? So what's great about that though is it's you know if we have different PMs across projects who are creating tickets then you know those PMs
get the like this is the issuer this is the person who created these tickets for this project and it it's it's really helpful from an organization standpoint. >> Yeah it's a good question. >> Yes it is part of the effort to avoid the AI work slop effects that can occur. Um did you have to like I guess dog food or get input from the PMS on how
these to be actually constructed. Were they the ones that were creating the prompts for you to generate these tickets? Like what engagement did you have with people that are going to be the the consumers of the of the work slop product? >> That's a really good question. So if anyone didn't hear the question was [snorts] how did we work with producers to make sure that um we
were avoiding work slop essentially with this uh product. Um it was a really collaborative process. We started with um a producer and myself worked together to um kind of create a a basis of rules and guidelines to figure out like what were our ticket dos and don'ts, what were things that we wanted to do from a uh ticket templating standpoint, like what was the minimum criteria we
wanted to make sure was in any given type of ticket. And then uh we worked with a couple of different groups to figure out what was the domain vocabulary that might be used because especially in requirements there can be a lot of jargon that's used especially in tech. I'm sure none of you guys know that right. Um so we wanted to make sure that the system had
complete context of whatever it is that we were talking about. Um so we created a domain vocabulary and then as we created tessera the final piece was creating that sort of decision logic tree that was uh purely dev that wasn't with producers and that was just again kind of thinking about the the the human mind as far as how we think about requirements distilling into tickets and
trying to replicate that through text. Um and then from there it was that collaborative process of introducing Tesa to producers and saying and honestly our first version was really cool. It did some great tickets, but there was definitely some hallucination and speed bumps and things like that that we had to to work through. And that's we created a again we treat it just like a project. Uh
we created a backlog of like here's the things that we're noticing from Tessra that need to be fixed for us to use this in production. Uh I think we spent about a month kind of running through that and at that point we felt comfortable enough to use this with an actual production project. Um and and we got minimal feedback as far as like hallucinations or anything like
that that came up. So, it's not to be understated like the number of cycles you went through with with producers to test and refine and then it was nice because you were able to have a few different producers. At least I can think of three people right now who were instrumental in like giving that feedback. >> And and that was that was a huge part of it.
So again, it's you're when you involve people on the team with building things, then they they kind of will help you cheerlead for it and and vouch for it. But you know we did have you know we did have um I remember you know having to remind engineers that like when it kind of didn't do great at first you know that hey remember your software that you
write it will get better. I mean if you keep continue to work on it and doing the right things it will get better and it did. Yeah. yeah thanks. Any other questions we can answer? >> Are you planning on releasing any of >> That is a good question. Um, >> we haven't really put a whole lot of thought into it. >> Not really. Yeah, I think it
would be really cool for us to do that and it's something that we've certainly considered and thought about as like a hey, this would be really cool in the future to be able to release this as a product. But really the the point of this exercise for us was to validate or confirm our approach to using AI in in in our workforce and how how we think
about using AI in uh in a way that really makes the tooling and things for our different employees across the whole company better. Um, and so that was tester's focus to for sure. And if it's something that we can share out in the future, I I >> I think the most open to it, >> but I think the most valuable part, and that's something that um I
kind of put on my my other hat here, [laughter] you know, for current is is we are highly interested in being able to help organizations get to this point without spending the near the amount of dollars and help do it because the real value mark is the process and how we got there. There's already people in the room right now that I know looked at this one.
I can go build that this afternoon. But you can. Yes. But what you can't do is you can't go through all the process that Mike did to go through all the testing and all. And now that we've been through that, we can really uh structure that into a clean quick solution, but it it does still take effort. And and I think that's important. So, you know, it's
it's always easy to look at a product and go, I could recreate it. Sometimes you're right, but sometimes it's there's part there's nuances, right? Not just the >> How did you train your humans to put and gather the proper requirements? >> That's good. >> We didn't and and and that was that was >> some of some of some of the stuff going is still kind of >>
but that was intentional. We we knew that it would be a multitude of people and users with different levels of detailed thought that would go into this. So a lot of the testing that we started with was as simple as like we would prompt it say, "Hey, I'm building an event system. Give me give me tickets." >> That's a rough prompt. >> It's terrible, right? There's there's
no there's no details in there. And honestly, it did halfway decently like at doing that. And that's another reason why we wanted with Tes V2 to have some of that quality gating where it asks questions and say, "Well, okay, well, what kind of event system are you you building?" Like ask questions and things like that to help fill that out because we know not everyone who's going
to use it is a prompt engineer. And that we don't want everyone who uses it to be a prompt engineer to be able to use this as something that's going to be useful for their job. So, at the end of the day, again, we almost intentionally did not. Um and and that's we know that everyone's going to have different technical detail and different level of detail of
putting in those requirements. And I think the one thing that we have instructed everyone on is hey it's a brain health. Put as much information in there like air on the side of too much detail over not enough detail and we'll go from there. >> Yeah. Context window is pretty big system. So you can at least for this type of thing. So which that just means how
much can you add in that first prompt to get things started. >> Yes. Yes. >> Follow up to that. So, do you have a form for your humans to put everything into that will have that back and forth? >> Um, it's for tester v1 it was Elvex that was the system and again has like a chat GBT interface. For tester v2, it's cloud. So, it's it's just
the cloud interface. Um, we have talked about putting and we've done tests before with other AI products that we've worked with of doing like a a React front end in front of it to to have that and simulate that and conversations of potentially doing something like that in the future. >> Yeah. And because it's modular and and all those pieces can shift out and swap. I mean,
yeah, you can almost uh do anything you imagine with that. You know, how it's built out. It's just the concept the connectors between everything is going to be very similar. >> Yeah. How was because you guys mentioned like ROI and stuff earlier. I was wondering how how that what what what was that impacted I mean did you did you let go of anybody because this like >>
I we have been waiting for this question. I feel like I set you up for this. This is great. You want to take >> I I'll start but you you fill in because we we actually talked about this earlier. We thought it might come up. Um, it's it's it's exciting to say in a world where recently even I was at a Trader Joe's and a cashier said,
you know, I think they're told to to talk it up with the customers. You know, it's friendly and it's cool. I like Trader Joe's. But they they actually asked me, "What do you think about AI?" And I'm like, "What do you mean?" Well, I'm hearing it's people are going to lose jobs. And I'm like, "Well, I I guess it depends on how you view it, but I
don't see it that way all the time." And so it's on people's minds. It's a real thing. But um but what what we our experience is more is uh no not not not to eliminate jobs. the purpose is to actually allow people to have time to work on more important human things that they need to be working on a building and I think that's what we need
in this world and that's more of that balance right uh I can't say what other companies organizations are going to do they'll do what they do um but for us you know that's >> the rough numbers are we've been trying to aim for two hours to 15 minutes um now whether we've gotten super close to that just dependent on the project there have been certain areas where
we've turned multiple hours into just, you know, 15 minutes to half an hour of ticket writing, which has been super awesome. But our goal for Tesa to to Strat's point was not to reduce headcount. It was to improve the time that people could spend on something. So again, that they could focus on other >> It's also their quality of life. Yes. Is in their role like >>
So they were receptive to this obviously >> Yeah. For the most part. And and I mean we've had, you know, for the folks who have been skeptical of AI are still skeptical of this. And that's that's a lot of where this conversation needs to to come from and why we have this philosophy to begin with. >> We will win them over. [laughter] I'm confident. I >> think
you can sell anybody. >> Yeah. Yeah. It's like it the product and what it does can sell itself because I think engineers are very pragmatic and they'll they'll go, "All right, all right. This does save time." And and if you're a great coder, you're still going to do great code, right? You're still going to be cool stuff. I heard a cool quote last week around like if
if my developers aren't using AI, I I should be concerned because it would be the same thing as me going to an engineering drafter and them saying, I'm not going to use a CAD product. I'm going to draw this with paper and pencil. >> And that's that's that was the goal here. We don't it's we want to change this from, hey, you just sit there and type
out tickets in Jira to use this automated system because it's going to be that much faster for you and that much more structured output. Awesome. Well, I know we're at time, a little over time, but thanks everybody. We're happy to take questions up here. >> Appreciate it. Thank you.
