AI in the Trenches Real World Wins Without Breaking Things

This talk features Ye Darcen, the CTO of Trenol, who discusses the integration of large language models (LLMs) and automation into their e-commerce platform to enhance delivery speed by 30% while maintaining production integrity. The speaker outlines a one-year journey of adopting AI, describing the strategies that led to both successes and challenges encountered along the way. Darcen highlights the importance of proper metrics to track progress and defines the need for a company-wide shift in mindset towards AI utilization. He emphasizes collaboration between technical and non-technical teams to create a holistic approach to AI implementation that benefits the entire organization, ultimately aiming to evolve into a 10x company rather than solely focusing on individual engineer productivity. The session concludes with insights on how to foster an innovative culture that embraces AI at all levels of the business.

Joining us for our community showcase for our final session of the day, Ye Darcen at Trenol, CTO, director of engineering um for core uh e-commerce platform at Trenol, one of the region's largest e-commerce companies with a 2,000 person engineering team. He's going to walk through how they used LLMs and automation to raise delivery speed by 30% without breaking production. And importantly, he's going to tell us what's

worked and what didn't work. So, please warm welcome for yeet. >> Thank you. Thank you. Hi. Did you see the tweet today? I think Amazon the junior and M developers are not supposed to send code to production due to high incidence in due to genet of course and the senior should approve it. So this happened today even creativity with it. Today I would like to talk about

the what we did in my company to see how we can apply AI to our production or to our environment. So we we brought some things by the way. Thank you CT for inviting me and thank you Hana for forcing me to say that this should be the last session so it should better be good. So I will try my best. Okay. So this is me by

the way I am head of AI and core platforms which is like sellers products and oiling systems and I work in tenure. Uh I co-ound and exit two startups develop and prime. It's also in e-commerce and I try to be a good person father and also a husband partner hopefully I don't know about dating about me but I hope and today I think we will go over

like in four eggs like in a good story like in the act one we will we will how we look at the opportunity in AI and in the second one we will try to see how the journey went through how we feel it by the way it's like a one year journey and in the third one I will show show you about the results that we we

were able to see and in the fourth one I just we give you some like tips and tricks so you can apply to your company hopefully I think you have seen this today in the morning uh I think we we mention about it like 9 5% I think it maybe 9 10 maybe right now but around 90% of all like what companies did to get something from

AI fails right they got zero return this is a big number by the like the five or 10% can get some value from the AI from generative AI or anything that they can put in their system and they can transform how they work we are not in the top at this 10% we we try to be as a company but the thing is like let me show

you what we we are by the way anyone knows no perfect so you have a better shock effect because the first come from Turkey 10 million variation more than that right We are biggest econom player in region. We operate in Turkey of course in Europe. If anyone from Romania I think they will know us also in other countries. We operate in Gulf. We have 50 million customers

300,000 sellers around a million products SQS like a millions of orders per day. So you get the idea how big it is. And also uh in tech side we are doing like we were doing like 800 pro changes per day. So it's like per day and also we have around at 200 2,000 sorry people in tech. So it's a big operation that we do and also we

have some bottleneck. I mean we are one of I think for me a very fast company in terms of delivering value but I mean we we saw that we have some in the lead time it was getting slower. Uh we had some rising cost because complex is high right and also productivity was not that it used to be years before. It's not solved with AI but we

had these issues. We also have some quite issues like we have 100 major and minor instance per month which is I think very high which was very high and we don't have enough documentation and test back then and one day I think it was in end of 2024 or maybe started we we saw cursor right the cursor there was something called cursor that you can use you

can just generate code a mobile can go and deploy something in production in back end on front end side a front end can code in back end a back end can code in front end anything. So people try to get used to this cur thing and we were just happy but we were I mean we are going to get g operation air operation in the company in

the uh tambian company right so of course it don't need the chaos so I will just give you how we went over we said chaos but how we try to manage it we control it so and after that after this year I I can see that I mean that this system this resistant result that we went to it was what we lived with right so in the

thing like airline if you fly I mean it's first time you fly you get terrified right something is flying like tin ch above you and terrifying it's act one or the first ph in the second phase you go into cockpit with the instructor and you learn to fly and the third phase you have like a plane you have some passenger maybe five or 10 you go from

point A to point B and the last phase you become an airline company like big shave or something. So that's how we went through in the journey in in in first as we talk here also there was a resistance right still it says like the the uh junior sorry so I will talk about the first phen what we did was as we start with g there was

some kind of call for we identify champion the champion like someone who's good at tech who can just fix things or find things or just be able to develop or be able to just use AI and show that we can get results within days, not weeks, not months maybe. And we start on a pilot project so that we can see that the benefit from uh how we

get benefit from the AI or generative AI. And we also have to have a like a a safety net for people because this was me by the way not the first one to have a safety net so that we will have some kind of resistance and then we should we can have a like a safe place that they can just try and uh find find out new

things. Yeah, as I said the result was real because people talk about like this AI will like the junior developers will no longer be there or we will promise or the securityities are damn too high quality will suffer all true today as I said Amazon said will not deploy because they have quality issues maybe and they're right or we have security risk I will show you in

a few slides yes we have by the way the risks are high but when we look at this thing we think about like how we can't make phrases perfect I guess if someone gives us a chainsaw we say nah we don't need a chainsaw we just use the crosscut the manual one because you will get weak no you still can use chainsaw but you will need different

skills and different muscles to do that so we are still think about how we can use this chainsaw something very powerful but something very maybe dangerous but you should have different skills and like master site. So we still get the issue like this. So it was fun but then the bad guys the security theor the developers I mean we don't tr them I know and they found

us you know we have a g operation so they said we are exposing our code to outer space we are just having some kind of a problem here we are just giving our data to outside even if we are in e-commerce I mean it's not like fintech but still we have I mean they were very afraid and they're right by the way it's not about like you

can just open everything that you have. So there should system it's not like we should go around them we should involve them. So it was our first fear that we think we can go around security to use a system. Instead we what we should have done is we should include them to our system to way of working so that they can understand what we are dealing with.

By the I this is from appro they right that the I mean this is for the fortune50 companies I guess the risk of the risk security are 10 times higher right now because of the uh AI usage that we think it is always correct and secure and everything because it is not think like this it's like giving a baby a toddler a permanent marker right the creativity

is enormous but it can the baby can drive on your car on the wall on your data and everything. So you don't want that, right? So that should be a system that you should prevent from our developers, engineers to I mean break something also in terms of development, we had some problems because we think that like 90% of the developers think that we are faster, right? actually

we are not it's like 20% so than expected one because AI I mean AI is AI creat illusion of productivity we create more code we create more I don't artifact we deploy more we we create more PRs but what we have what we also face is we have to teach the AI or the LMS or anything to understand our code base we have to debug it we

have to talk with it it's time consuming by the Right. So we have to I mean uh we have to make sure that we are just measuring everything correctly as Peter Ber said decades ago I guess we have to have like a measurement system so that we can manage it. I know that from engineer like GitHub, Google, Dropbox Mo they have like different like metrics to check.

I mean we have our metrics we we were obsessed with lead time incident um PR number etc and also we were we already had that system so it was easy for us that we are we were using for metrics but they have their own metrics to track while doing this change the mind change so I and make sure that you have like a metric that you are

obsessed with so that you can make sure it's going on on track and after we do Right, we go on to phase two which means we start execution. We choose a team customer experience team which works with agents and boards and chat boss. Uh so we obsess with the matrix and we do hands on management. I can say m not I can say interested let's say not

going to every detail but seeing that every metric is going well or if something goes not of the track we make sure that it's going on the track so in the pilot phase we we have MCP servers way back in March or I think last year we have internal community we have cars but in a control like drives let's say and also we choose a suitable domain

which was customer experience as I said earlier so that we can make sure everything going on track and we got those numbers I mean right now this is not the number for every team in my company but we had like 40% of increase in velocity 96% of completion business love outcome because we were able to deliver more test more do more AB test anything which means the

business love outcome right they understand we are doing with something but they understand that there is like a more value here and we had zero instance way back then and also we had these are business results these not tech results this is the business result this customer love numbers the MPS the first response from the BS or chatbot success these are numbers that we can ask or

we can just make sure that a business and tech comes together and agree that we can create video from regenerative airs. So this is me that we also publish to our company so that we can see we are going somewhere right and then it makes it force us go to phase three. So we scale the process because we know that a team like a group of 40

people can deliver a nice video to their business and then everyone is happy. So we are like 2,000 tech employee company right? So we can just scale it. We we like smooth on boarding. We worked as a 40 point engine like panted does so that we can just onboard every team to together and we we had some monitoring IoT platform so that we I mean the cost

is high or the security risks are high. So we should we make sure that all can like one track and we make sure in this scaling process that AI touched everywhere of every stage like the code code coding development testing designing security like enriching the Jira task for example we make sure that they I mean we can use in every aspect of our life cycle. This site

I love very much because our mobile team we have around 150 engineers there they said they had an issue about the button is about the regression right because we are doing big releases as I said 150 developers changing code every day and we are doing a mobile release every week every Monday and this it takes too much time to do regression test to do it takes some

time because we don't have enough tests think that we have operate we are operating in 35 different countries countries we are working with I don't know maybe tons of payment systems billions of products billions of users so we work on with the team and they create their own platform for mobile only and they were able to create test automatically they can initially like the stories and they

can write screenshot test etc so they they were able to get their own benefit from the platform and then we had our second issue we had the production incident again due to PR PL because we don't check the P we forget human in the group human in the process and the AI was a scape goat the developer said I think the AI forget to or I think

the AI put extra semicolons somewhere and then we had some issue the the post was I think it was not that good and we we we talk that AI is a powerful tool but it's the scape cost AI will always be conf confident as it is writing a good code or a bad code or in that I mean AI is it's confident what on when when it

does something so it can houseate right so we should make a better platforms processes all the time so we make sure that we don't fail again so it was a bad bad incident and right now we in phase four as a company we are not in good fa we haven't finished our phase four we didn't start I guess but what we do right now is we try

to make it a must for whole company we we are trying to have like a solid platforms for people to use and also we are working on optimizations so this is our AI platform I'm just I'm showing you to the influence you will get so we have like you can use the new latest news or trainings from here you can have AI agents workflows by day we

use nan if you don't use any please do so and it's very powerful to for us at least uh or we can use like our custom uh work for workflows also. So we are give this platform to like clients our developers right also these are agent platform right now I think we have around 20% of all the issues have like one or two point in in store

point in our jira and AI can code all of them so in Jira if you open your board and just uh take one or two point of like a stock point story to like a a different column then I code it test it and opening PR volume for you. We are not like I think Spotify does that. I mean they can just give everything to cloud and

it goes automatically. We not in that part but I think we will be someday but we are I think we have done 4,000 Jira issues by AI in last a few months. So that's why we are trying to go. So we give those like agent platform to our uh engineers also they can have like mutation test pers or like I don't know coding panel or test panel

anything they can do as an agentic way. So that's what we're trying to give them so that they can use it in their daily life. Also these are like MCP ser these are open source to our company. I mean they can contribute or anyone anyone can contribute or use we get a lot of garbage MCP service but that's fine let them use that let them try that

so they can find the best one for their like business right then we are going I mean think about take what we do we have the coding we have like jira we have like I don't if you are using agile or anything you can like story points velocity incidents every metric you do in tech I mean you can measure them right so this is perfect so that

you can optimize it our fail right now I I say it's fair our tech team they had the all buying right they can say okay we are using this one to improve this metric but I mean we don't make like a whole company idea of like changing our mind set to using AI in every aspect of our business if tech does that only it will not get

value from whole company so that's we are facing right now. I mean our CEO or like top management believes in AI but I mean everyone from HR to growth from finance to anyone should think about how they can use it. I mean some companies do that do this best but as a company of like 5,000 employee it's not that easy. We are just trying but just just

an example. I was in China in Hanzo in Ioba headquarter in the visitor center everyone talking about AI right from HR to like the coffee anyone in the like coffee shops or everywhere even if in the mass bathroom there's like small ads like when you do your business there's a small ads and even if there there are like some news about AI like the Iobas last model

current it's a very powerful model by the way or like news from other companies so the mindset from top to bottom was all about AI. If you have that or if you can have that, I think you get the benefit from AI because not only tech we get we get benefit to you but also the whole company should believe in that one. So for us what have

been changed we have better lead time we have like 700 not only because of AI but the mindset change have like we have more business figure with the same uh developers our instance decreased and as I said one or two stories are coded by AI and hopefully have more happy development team we also gained from the business side we have better generations AI solutions like the HI

feeding mechanism us are more like like faster from our uh systems translations are much faster like they are like sound of our company right not like normal Google translate stuff it's like like the push messages emails are sound like us are done much more easier but today we talk about 10x engineer I think it was a very powerful uh presentation from uh I think from makisia they

said that we are always think about 10x engineer right as developers but if you can convert 10x engineer mindset to 10x teams to 10x company then I think we will get them full benefit it's not about changing making me as a developer a 10x 10x engineer it's about making the company a 10x company more productive faster I don't know more optimized whatever you name it but I

this is the this is the most important thing I mean and the answer to do that I think is in the and the culture also I mean you cannot secure the individual billions by hiring more individuals billion people but you can have a the best platform so that you can have this brilliant platform to everyone so you can benefit from all right that's what we believe and

today I saw that by the way at one year ago 10th of March 2025 s of entropic said that all code may be by AI just money will go this day and I think it is that we are coming to like coming to that right now also carpet said like I'm just coding in English so this is this is not like future this is happening now right

so we are not just carpenters we are just more architects and every conversation in this room maybe or uh in in world have to prepare for this shift so I will let me take time oh so I am just giving What we did as I said you can spark in some place in your organization you pilot it make sure the business and take aligns with it then

you scale it to company or part of the company and then you optimize it that's what we did and in my last two slides I will just uh I ask you to think you know deja right in the matrix you remember the neo sees a cat and it goes and it goes again it's like feel the thing again George K I think said coined it the strange

feeling that you have never seen this before even though you have done million times it's like waking up next to your loved one and seeing hello again and going in love again right so I I would like you to think how would you do your daily work if you were doing it for the first time not about coding think about product management, think about designing, think about

how you get the I don't know right requirements, think about how you do your everything in your job and document it and then we can make it like faster, better or I don't know uh more optimized way and this one I think as it was in his first right I think this one will uh decide on where you are in this picture you will you use AI

as a test typewriter so It can write code. It's good by the way. But will you be like one of the transformers who will who will reimagine how the development will work based on the same rules like I don't know like extreme programming values but in a different way. So this is I think the motivation that you should have if you would like to change a company.

I mean we we were able to start this spark for a 2000 tech company and I think everyone can do. So I think this is my last slide. So this is the last session last slide of today I think. So I hope no one I I I never seen someone step so I think it's okay. So yeah that's me. Thank you. questions. No questions. >> Say don't

leave just yet because I'm sure we will probably have some uh questions for you on Slido that'll be coming up. So let's see if we can get those up. There you go. >> How did you confidently attribute any change in the matrix to your AI initiative? >> I mean we had a lot to try. I mean we can just as a big as a big company we

have two 250 domain teams so we can try anything in different parts. So we can just do AB test. So when we do an AB test in one team and do another in team B, we can see how it affects their performance or their metrics like it's is better a lead time the instance start to increase or I don't know how it affect the their metrics that

we look like then we can be confident that doing those AB test or just trying we can see that if it is like helpful to us or not because for every company it is different we don't have that much document for example and you don't have that that much document and a huge code base millions of millions of lines. It was not that easy to do something

but then we have to find our own solutions to make sure that it like how we do how we right. So that was how we try to do that. Okay. >> Getting ready for that. Thank you so much for uh closing out our day.