About this talk
This talk discusses the speaker's approach to measuring AI adoption within their organization and the lessons learned from the process. The speaker highlights the misconception that the quantity of tokens used correlates with productivity, introducing the concept of 'token maxing'. They share insights gathered from various research sources, emphasizing that successful AI integration relies more on employee engagement and cultural adoption than on technological investment. The speaker presents a maturity model designed to assess and improve AI adoption among teams, detailing its components, including surveys and dashboards that track AI usage. Throughout the presentation, the emphasis remains on the importance of fostering a supportive environment for employees to effectively use AI tools, rather than solely focusing on quantitative metrics.
Full transcript
Uh Can you hear me? Perfect. Thank you. So, thank you for coming. Uh great conference. I hope you enjoyed other speeches. And um thank you for coming to my speech. I will be speaking today about um our way, um our approach. How we um how we measure AI adoption across the people and some of the lessons we learned and uh the platform we built. Uh and I'll
talk about this. So, a couple words about me. So, uh generally last 10 years what I'm doing this is managing a company uh for we have almost 150 people. So, we have about uh 40 to 50 projects. And uh what we do, we actually build products for the clients. And we have this variety of different technologies, different teams, uh and uh and the clients. I also do
some couple other things, but uh my primary stuff this is uh a great company Ralabs. Uh so, I have a couple questions uh to you. Uh I want to ask raise your hand uh if you know um who is from your team or your colleagues you using the most tokens. Just raise your hand if you know um okay, perfect. And the next question is so, there is
a statement uh and maybe you heard already on this conference that a person who use more tokens, this is a person who has a better productivity or this is a person who has a better AI adoption. So, raise your hand who don't agree with this statement. Okay, much better. So, there is uh uh there is a new word. Um I think it's kind of new word. Uh
so, it's called token maxing. this is when the management pushing people to use more and more tokens. And in our opinion, like more tokens it doesn't mean um you have a better work. And uh this kind of stuff is in our opinion, it's it's not a good as goal or KPI. It's good as some temporarily signal, but uh you cannot use amount of spent tokens as some
kind of like metric to tell if your team is good on AI or not. So, this is sort of spoiler what platform we built for our self. And I will tell more about this in the next slides, but this stuff is open sourced. So, if you want to measure team across your um Sorry, measure AI across your team. Feel free to contact with me and I will
share this with you as well. So, how everything started in in our team. So, we have a something about 35 different clients. And clients last year they started and asking how you guys using AI. How safe you you using AI? And this is kind easy to answer when you have a team of five people, maybe 10 people, but when you have more people it's hard to give
some metrics. It's hard to answer about the adoption, about safety, about efficiency. Right? So, that's a problem. So, our manager was shocked we didn't have the answer. So, we decided somehow to take control over that. So, without control of course we can guess. So, my first question was that if you know somebody from your team who use the most AI. So, this is kind of like guess,
but when you have a team more than 10 people in the most cases you you don't know like who use more, who use less AI. So, you need to have some some some system to control this. So, in our opinion so there is three stages of using AI. across the clients what we see that most most of the companies they are actually on the adoption phase. They
trying to pretend they are on the impact stage and they try to measure some impact, uh uh reality that most of the companies they are on the adoption stage. our solution and my presentation is only about adoption, is not about the impact, uh because for the impact there is actually a lot of tools like Linear, GetX, and anything else. You can connect with your Jira, GitHub, and
you can get some metrics and other things. Um and before doing we of course did this huge research, so we read like uh papers like uh Daze, MIT, uh LinearB, and even more. Uh the Google, the Microsoft, and um like some of them are way to to theory, some of them is like two-way enterprise level, some of them require connection with lots of your tools, and we
uh can't do this on our end, so we just decided to build something own. Um but uh this uh benchmarks, this reports we read, they actually give us some good uh insights, and I want to show you a couple couple of them. So, this is like like giving you context why we decided to build that tool. So, the first is that uh this research on the big
company Deloitte, they told that most of the company companies they invest uh like more than 90 percentage on the tools uh like Cursor, Copilot, like tokens, and anything else, and they do not invest more on the people side. Another one is from the uh McKinsey, so they told that if you have your AI champion, like the person who is uh who love to use AI, uh most
likely this person will find um on your business uh how to use AI efficiently. So, this person will become your champion. So, as a company, you must know who is your champions. Uh there is another one from the uh BCG company, so uh they told that um Uh and this is by the way the the opposite to this finding that most of the companies invest on the
uh on the on the on the tools. So, this rule is about that business they must invest more to people and less on tools. Only that will give you like some transformation. Another found another thing that we found that that's called that's this is this is bring your own AI. This is another statement is shadow AI. So, uh 80% of people of employees they use some models,
some tools that they didn't like agree or inform the company. So, kind of like Chinese models, like tools, and other things which might create some issues security issues for the for your And the last thing what we found that only like very small percentage of companies they treat themselves as mature in terms of AI. So, there is a big room to become more and more mature. what
we get from this insights that the human side is more important than infrastructure side. And our like solution this maturity model was fully focused on the this adoption stage and on the human side. I was thinking if from all of my slides you need to get one message, remember one message, that probably would be this slide. That that's like the the Pareto principle. So, that your 80%
of your AI potential impact they will come from the people, not from the tools. So, that's where your kind of like time as a managers, as team leads should go. So, we built set of tools for ourselves I will tell more about this specifically maturity model. So, the model itself it consists of of the six like parts. The first it was a survey. That's what the people
ask. Another one that was a maturity level, like specific level for the people how they use AI. And there is other components. I will just go one by one quickly through of them. And this is since this is a technical conference, I just decided to add the like simple architecture diagram what what we have under the hood of this application. So, nothing complex. So, I would not
stop on this slide. So, we use like Superbase, some AI tools, some Code Rabbit AI for the code reviews. But nothing extremely extremely complex. Yeah. So, the first component of our maturity model that was the reports. So, we come up with three reports. The third one, this is the client report. So, we ask clients some questions. The second one, that's like non-technical report. So, we ask like
sales guys, like project managers how they use AI and other things. And the first one, that's engineering survey. So, we ask developers, quality assurance, DevOps And that give us answer why people behave like that. So, we go simple and we just use Google Form to collect the answers. Second component was the like maturity model. So, based on the answers, we were able to tell on which level
the specific person is. And the level one for us, that was more a cultural level. That's more about cultural adoption. So, it's important to have people to be people um as much as comfortable with new tools when the person is on the initial level. Another component that was the way how we structure the results. So, we able to tell the what is the AI adoption score for
specific person. We can tell what is the AI adoption score for specific team and for the whole um company or organization. Uh then we get some uh dimensions how we measure. So, for the people, so we decided we stop on the five dimensions. So, we can tell that um spec- the let's say a senior engineer, he has a usage on three from five, and we can uh
take this signal to do something with that. We have another dimensions on the uh project level, and also on the organization level. So, uh for the organizational insights, we we come up with this uh kind of like dashboards. So, we can see, I don't know if that's working, but you can see that organization is on level three, so we have 3.2 from five. We have some uh
response rate. We have some maturity across, so we have like most of the people they are on the level three. We have some across by the functions, by the team. And uh of course, the most important, we have this chart I was showing you before. So, we can see that uh we have uh we have probably the lowest score on the impact, and the lowest score on
the strategy, so that's uh for us action items to work on that. And also we have on the team level, so we can take specific team, whether it's a design team or specific project, we can see a level for this team. We can see this chart of the team. We can see across the whole This is all of the fake uh projects, of course, and fake people,
but you can see the specific person of the team and their level, so you can act on this somehow. And also for the specific person, I have like four cases just to show you This is also random uh people, so uh let's say this is a senior engineer. You can see on the top uh his name is Matteo. So, this is his uh personal chart based on
the answers this person give us. And you can see that this uh Matteo, he's more like low culture um guy, I think so. And uh there's a question how to improve this um like his culture in impact low, but uh vice versa, this this person has a good usage and a good skills. So he is a strong individual, but he has a low culture. So our action
item of this is to bring this person to another group of people who love to use AI and um uh so he can work maybe on the smaller groups and share some his knowledge. Yeah, another um another example this is her name is Nadia. So he is more like cultural or visionary person, but in the same she is lacking some skills. Uh and usage. So in her
case it's needed to give her maybe some tools, maybe give her some knowledge, maybe give her some like peer connection with with some more experienced AI person. So that's might be an action item. And the third example this is Uh so this is like the person with the name Liam. So this person has a very high usage, but low impact. So this is a good like signal
maybe to take a look what exactly person is doing, maybe speak with the product manager, maybe assign more more AI related task to this person because this person is very capable, but this person is not giving enough what this person is capable for. Uh the fifth component that was like survey format. So you can see here something like similar maybe to scrum like cycle. So we just
put um So we run this full survey. It's taking about 20 minutes. We run the survey every couple months from 3 to 6 months depends on the team and we put some short surveys into the every every sprint. So that's how we collect the fresh answers. Uh talking about the scoring, I will not stop here, but we are not doing something like simple average. So we have
lots of things. So if person give us like wrong answer on the one question, we correlate with other questions and we got like proper um proper result. Uh there's a couple couple insights how to make effective AI adoption on in your company. So, first what we are pushing that like culture first approach. So, make your people happy with AI. Uh the next is more practice less theory.
So, we decided to stop any AI theoretical webinars we had. So, we are not doing like webinars. We are doing only practice through workshops, through small projects, something like that. Another one is making visible. So, people who use AI who build some workflows. The previous speaker was showing any 10 workflows. the people who build some maybe any 10 workflows on others. We're making that it's very very
visible inside the company. We send some kudos to people. happy to share more and more knowledge. We put some some of the ceremonies into scrum. We ask on the scrum like product managers they asking questions related to AI on the scrum. So, it's forced people to use more AI. And that's summary of my like of course presentation. Like we're measuring adoption but not the impact yet. And
there is some observations across we we measured this on our company like 150 people and few other smaller companies. Some of the insights we have. So, the first is so So, many people they actually use AI but they use mostly AI for simple tasks and not deep workflow changes. So, that's why the token consumption metric in our opinion is not is not a good metric because you
don't know what context, what information people put into this token. So, when you see like 1 million tokens spent, it doesn't tell you anything. Maybe people maybe person just put the huge context into that. summary we had that the managers they often believe that enough support but when you ask people like employees like engineers if they have enough support so they telling another side so they feel
that there is not enough support. So another insight we have that excess of tools is is not a problem right now so people has enough school enough tools and current currently the currently the the focus should be on the like confidence habits on the people and So what else I have? Yep so so real adoption come from the AI champions and practice There is interesting about about
the policies so we decided that we will not create any AI policy of course we have something like high level which models we recommend and other things but we don't we don't have a deep AI policy into that so people can be flexible what they use and yeah and the last one the eighth one that actually people using AI is much more than leadership things that usually
the managers they underestimate how employees engineers That's kind of all. I'm happy to answer any questions if you have. If you again that's open sourced so if you want to survey your team so you can ask me on the LinkedIn. Happy to share this with Happy to answer any questions if you >> [applause]
