How Tesco Uses GenAI to Understand a 40‑Year‑Old Warehouse Management System - Arshad Hameed

In this talk, the speaker from Tesco discusses the modernization of their 40-year-old Warehouse Management System (WMS) through the application of generative AI and knowledge graphs. The speaker highlights the challenges associated with understanding legacy COBOL code due to its complexity and lack of documentation. By utilizing abstract syntax trees and creating a knowledge graph, the team aims to improve code explanation and comprehension, significantly reducing the time required for technical documentation. Key technologies discussed include Java for knowledge graph generation, Python for knowledge extraction, and Azure OpenAI for language model integrations. The speaker emphasizes the importance of human validation in the process to enhance accuracy and avoid hallucinations in AI-generated explanations. Ultimately, the approach aims to facilitate better understanding and operational efficiency within Tesco's technology landscape.

We are from number one retailer company in UK, that is Tesco. But one thing that we hold special in Tesco is about uh uh planet and community. And it's not just uh exaggeration. If you see, it's there right in our uh core purpose, uh serving customers and communities and customers, communities, and planet a little better, right? So, we have a related metric here. Um the other uh

very fascinating uh or very impactful metric that I would uh say about Tesco is uh Tesco actually feeds almost 30% of food I mean food uh food to UK. I mean, feeds UK for 30%, right? And there are many um employees. And not so important, 66 billion company. I'm okay with it because my manager is not here, so. Okay. Don't get Don't get worried. This is not

how the rest of the session will be. But I want you to take a look at it because this kind of captures what our complexities are about the subject that I'm going to talk about. >> [snorts] >> There are very I mean, there's a lot of if if else conditions, uh maybe undesirable acronyms, etc., which makes it very difficult to understand this code. As As per maybe

some COBOL expert, if uh someone is sitting here, may understand it better than the others, but still it it becomes difficult. But don't get me wrong, it's not a bad code. This is a production-ready code which has been serving Tesco for about 40 years and has played a very important uh role in keeping Tesco at that number one scale, right? So, I'm a good I'm a movie

buff. So, let me give you a quick teaser before I start, right? what if I had something like this which can explain that code? I take a word out of it and see and understand what it is, right? And what if it does better? Right, it gives a very structured uh explanation, but then it can also give you a very in-depth technical explanation about the same term,

right? I didn't switch the I switch. Okay, give me a second. This doesn't switch. So, this is what I was talking about. So, if you had a place where you can go and search and get the result and even get a more output out of it in terms of what it deeply means, right? That'll be great, isn't it? So, let's go back. Um to this. So, as

I told you, that code was not bad. It has served Tesco for a lot of years, but at the same time, old things has to be renewed and we we have to move for move further, right? hence, we have a WMS warehouse management system which we are intending modernize. And there is a team who has who's been put together to actually look at it. They have a

lot of issues. One is a 40-year-old system. If you ask a business to change, they'll be like, "No. Don't ever touch it, right?" So, and it has been working beautifully. And this by the way, I had a recent experience of looking at other world class WMS against what we have and it didn't even come close in terms of functionality and operational flexibility etc. Right? Operational performance performance

as well. But then because of many reasons we still need to change. And make an equal in one or even better one. Right? So this team is facing so many issues and one of the primary one is actually understanding what the system is. It is not It is not easy as I showed you the code that is only one glimpse of what what all complexities are there

in that. So this brings me to the topic that I have today. I knew this is not going to work. Understanding the past uh using generic code for of a 40-year-old WMS. I am Arshad Hamid. I work as part of Tesco Technology as SD3. Okay. So what did we do? And especially in terms of understanding what why is a different beast will not get into it today.

But what is happening in the system? Right? Where we we we went ahead and checked what could be the real source of truth to actually look at where do we get the real truth from? Uh who who could we ask? Right? I'm hoping no one in this room is expecting an updated document for an 80s software. If So that was ruled out. So what we what we

came across and what we finally decided is source code. Right? There is a system that is running in production. It is running very efficiently. Everyone likes it and it is it is doing its job. So we can ask the production system itself, right? What is the the code within that, right? So, that's where we started at. So, the first attempt here was to learn what is the

code doing and using generative AI make it explain itself, right? This. Now, how did we go about doing it, We started with the COBOL code. Used an abstract syntax tree of the COBOL code. Generated abstract syntax tree from the COBOL code and created a knowledge graph, right? Graphs has been good. I mean, has been famous and has been has has a special place in the technical industry

even before GenAI has come in, right? But, GenAI is bringing a new renewed energy into the graph because of many reasons why the graph can connect itself, right? So, the graph in this case is the way we we are collecting all the information in a way that we can query it, we can we can see how it happens, etc. And that how it's done is you split

the code into smaller smaller pieces and have a relationship between the code that is that is containing all the smaller pieces and also have a relationship between where it where it is has in having dependency like a call to another part of the code, etc. So, even once you capture such things, which is what is important to know what, right? You have a very good knowledge graph

which I'll show you later which even before going to AI starts to talk, Now, using this uh knowledge graph we first started doing two things. One is entity extraction. The so so many acronyms that was there LLM cannot explain it. Not even someone outside the system can explain it. People who have worked within the system only can explain it, right? So, first thing was extract those entities

and get it explained, right? Um there are many things to it. Um we This is one of the places where the human bottleneck is there, right? And lot of time will be taken here in terms of actually correcting the data and doing things, etc. Hence, we have lot of we have done a lot of work within that to actually optimize how much human interve- intervention has to

be taken and how can we make it efficient. So, if you focus on that, then once the entity extraction is done and explanation is coming out of LLM, someone verifies it and corrects it, right? As a as a human. The other problem that we saw is COBOL code. I think most of the languages will have it, but COBOL code specifically had this problem of how to figure

out its flows, right? Especially when you are trying to explain one small bit of that code, right? What How does it work around it, right? It has some things like COBOL when you when you see them uh you you call somewhere, you have some statements which you where you can call another part of the code, it'll execute and come back. But so, there are similarly other other

call in the COBOL which could go there and just continue the execution from there, right? So, it becomes a little more a little more difficult. So, what we did as another thing was use GenAI again to create an execution for a program, right? A simple technical execution flow using Mermaid script uh but not but generated by AI again. Right? Now, these two had a very high impact

on the amount of hallucination the system had uh when trying to explain the knowledge generation. So, you are cutting that possibility of uh hallucination. in both the cases uh we had knowledge graph being used and we we were able to uh you easily scale the structure and the complexity and the and the length of the code, etc. easily using knowledge graph and do all of this. Similarly,

in the knowledge graph knowledge generation as well, actually explaining the code now, right? uh graph was very useful because we now have the relationship. We now have uh different structures of the program. So, we can start from the bottom. So, we did a depth-first search, went to the bottom, and picked up all those units of code which has no dependency or least amount of dependency, started explaining

them, and then came up a level, right? Each level uh as it goes, you you will have more dependency then you by by then you would have explained that dependency, and then uh the dependency gets explained even I mean the uh upper levels of code gets explained uh better with those dependency explained. Right? So, knowledge graph again, very crucial. The way graph is structured, you can you

can actually do uh do that very well within the graph. So, but uh it is not again which is not very easy then again NLM again not non-deterministic. We have done it uh year we did this year year back. So, the NLMs are weren't that good as today as Then we had to do some more things within the knowledge generation, which is um two important things were

many other things were done, but uh important things were like context matching and a review agent. There were cases where even if the if when you take a particular note to explain in the graph, there was so many dependency that the context is too big, right? So, we had to do some batching around the context. Uh give the context in batches and get the uh explanation better

one after the other, right? And similarly, another one, review agent was another thing that we intentionally introduced because we knew uh once the uh LLM starts to explain, there'll be some things that will be missed. So, you have a pair like you generate, the other agent checks what is missed, and then come back and change the um explanation again, right? Now, that generated an explanation document, right?

Which is usable by users and that's also validated by SMEs. Uh the work is lesser because you would have uh done the entities uh validation already, and mostly explanation is good by now, and users can read it. But then, users reading a big markdown wouldn't be helpful as well for a long time, right? Hence, uh two things that we did was uh before getting to chatbot, one

other thing was diagram would be useful again, Um getting a flowchart, seeing what is the decisions that is being made, and things like that. So, we used um uh similar to execution flow, we used the document that is generated to generate a functional flow diagram, right? With the with uh all the functional decisions and things like that using GenAI. And that was uh that gives you a

pictorial representation of what it is, right? And then, uh again, reading is not good, but if you want to understand, if you ask a question and someone explains you, that's a better, right? So, we put the knowledge knowledge uh documents that we got under a a um and made it to use use it like a conversational database, right? Conversational knowledge base. um made the first cut of

whatever we had to do. And uh the XCode was explained. Now, let me take you through a few parts of the solution so that it will to bring you closer to the solution, right? This is the knowledge graph graph I was talking about. So, there's a few programs together. And now, just by looking at it, you start to see some views. You start to form some views,

right? This is a bigger program, lot of complexity. This is This [snorts] seems to be more structured, etc., right? So, the the the high high context batch, I mean, context window being highest and the context batching useful, uh that is somewhere here. So, some nodes which is too big to explain. The other one. Entity extraction, right? So, we we explain as I told you, we extracted this

and got this out and for a person to for a human to review it. And this is I think a review document. I'm not a expert of this part, but yeah. So, if you see this, that are the even in the even in what is um extracted, there are small bits like W WSPARM. That's not the actual variable names that you saw. So, this is part of

improvement or efficiency improvement of that process itself. So, when you start to extract non-duplicated [clears throat] items like a particular unigram from the code and which is repeated across many variables which is across multiple programs, you start to explain lesser, define lesser but use it in many places. So, that was a big improvement as well, And there were a few others which I don't have time to

go through now, but let's go to the next one, right? Now, how does an explanation of this look like, right? This is an explanation of a UI uh which is explained into different uh sections like purpose, screen layout, and um components of the screen, user journeys explained, dependency, etc., right? So, and uh similarly, a flowchart looks like this, right? So, you have things like the important decisions,

depot automation supported or not, this has to be good. So, like the chatbot that I showed you earlier, I uh used a particular variable name, but you could you you could ask a more sensible, more functional-related question like this as well. What are the rules for determining apportionment, right? In a WMS, warehouse management system, right? So, it gives you starts to give you a very uh refined,

clear sets of uh rules that is within the uh code uh as explanation. Last thing that I want to show you as a demo is an interesting bit which is related to chatbot, right? So, chatbot uh works on drag, you would know many uh regular chatbots work work on drag. it still has problems based on context. How do you manage context? So, such a domain [snorts] like

this, if you take something like if you ask a question like what is warehouse management Or what is the anything in involving a warehouse management system as a word and then do something around it. Almost in if there is a you would know I I don't know if you know internally it's a similarity search. Almost every document or every program in this case will come out in

the similarity document and it's a huge one and it never gives you the right answer. So, when when we looked around at how do we solve it and things like that, we found a open source solution called LightTag. Right? Which is which is very interesting. Again, it it hinges on graph. So, it has two mechanisms. One is it creates a graph to figure out what are all

the connections in this document across. What words it forms a set of words. If you if I show you something right in between here, right? Where is that? That's it is not coming out correctly, but that's the word warehouse management And similarly anything big others that I take, it will be something which is common. Right? COBOL program. Is a is a bigger block there. It has too

many relationship and it is one of those things. So, how LightTag works is it creates this graph and it has a similarity search as well. So, it does a high-level search across first to get a context of the relationship of those words that is there within this and then does a low-level search as well to get particular documents. Combines these two things to get more relevant answers

for you. So, then all those questions which is which has only one document to come out or even thousand documents to come out will be better answered with all of Uh I mean, with this mechanism in place, Oh, getting back to our PPT. Let's go here. So, what is the technology that we have used to do all of this, right? we first built a component for the

knowledge graph generator, which is predominantly in Java. The reason being COBOL's AST parser, which is an existing library, which is called Prolib, is in Java. So, it was easier to build this thing in Java. Then, the central component, which is the knowledge generation, entity extraction, execution, flow generation, etc. done as a Python module using Neo4j and Vector DB as database, And um uh the central LLM that

we use is Azure Open AI. As I told you, it was an uh year-old work. So, at that point, it was GPT 4.1, 03, and 04 mini that we used. uh I uh the entity extracted entities after collected it has to be ingested back. So, another Python module to uh ingest the document. And we used uh for those interactions of entity extraction and things like that, we

use Azure Azure storage account. And then, the chatbot, which is by uh powered by Light Rag and Streamlit in the UI. Apart from all these functional things, we also did non-functional, which is uh famously being called in this uh summit many places as capabilities, right? So, for observability, uh we used uh LangFuse. My monitoring and observability, we we also used Deep Eval. So, again, this is another

thing about uh human evaluation. So, human will take a lot of time in evaluation we if we stress them a lot. Again, a lot of uh uh things related to uh stress um stress or the too much of workload as well will come into the uh uh the uh the evaluation that they do, right? So, we wanted to bring in a little more evaluation which we can

do using LLM itself. Uh Deep Eval is a framework that gives you metrics around evaluation, and we can use it. Right? And then, obviously, because this was an uh early work, we did a lot of experimentation tracking with a lot of co- documentation and also LangFuse. So, coming to success metric, right? So, the person who uh was uh important I I missed this part early. For the

people who are actually doing the modernization, one of the main problem is, again, uh the knowledge, right? But, uh the SMEs are also scarce because it is a production system, and people working on the production system cannot do two things at a time, and they're keeping it at a time. So, SMEs are the ones who will get affected the most because if someone wants more knowledge. So,

SMEs is SMEs is the one first uh benefiting out of it because anyone who as an SME who wants to document that COBOL program, who takes about 15 plus days off for a big program, there are like 10K uh 10K to 30K lines of code in one single file, and there is thousands of them as well, uh the person doing that documentation, for them, it came down

from 15 days to 1 day, including all those activities of the entity extraction and all those things that you have to verify and do. And similarly, about the knowledge coverage, right? Near perfect coverage at the paragraph level. So, paragraph is like a function for those people who don't know COBOL, but it's not exactly the same. It is That's the closest that I can come to when it

come comes to an new language. So, at that level when you're explaining, you get almost perfect coverage and everything is explained. And once you start going up, there is a drop like 80 to 85% of coverage, especially when it's a very big program, like anything big than bigger than 10K to 15K lines of code. But then, don't take it 15% is reduced. 80 to 85% is from

zero or from 10%, right? So, that's a big win in terms of understanding the system. Now, there are things that can be explained in numbers. That is very important, right? SMEs were bottlenecks. They were not bottlenecks anymore. Engineers can actually trace the logic of whatever they see on the screen even if they don't understand COBOL like me. Product managers can see a bit of why, not every

why, but then you will see what is the relation between the things and start to make sense of why can can help in why. And then leaders are the other managers can actually start to look at even planning the system based on this, the modernization itself, What needs to go first, what needs to go next, etc. So, the most important part of the session learnings. So, I

I told you about efficient usage of human already a few few times. So, we we have we have focused on how to reduce that intervention, but keep it very efficient and impactful, right? So, that is very important many people would have been telling even whole conference saying non-deterministic. Right? LLM can be non-deterministic. It can give you full crap sometime. Uh but if you have something someone to

actually review it, it will be good. And that's more relevant even in the newer uh models because it gets more and more difficult to find out the flaws. Even if it was fully fully wrong, you could easily get it. But where where where when there is a small part of it which is wrong, you would be in in trouble. Right? Similarly, the control flow hallucination which I

told you again, right before. It helps to actually get the flow. It's not just explaining the variables and things like that. How things work together is very important. And retrieving a document relevancy of chatbot. This is the light track. I told you how helped agents recover. This is one thing that I missed which is basically uh when you're creating something like a mermaid script to create a

diagram, LLM almost never creates the first time as a um bug-free or a syntax-free script. Right? So, there will be some error in it. So, we we put in a recovery agent in between to actually get the mermaid script, check it against um library for syntax, give back the syntax error back to LLM, fix it. Within a few loops, usually it's all good. You will get a

renderable diagram. Then, final one, limitation of a single document which is what I told you because of a big document. Even now, it is almost pretty impossible for an LLM to do a big I mean, justice to a big document for explaining a 10,000 lines of code uh code which is very difficult, and that's not the way to go. We found it after a few tries, but

the better way would be like splitting into a set of documents which are connected together. I'd overview, and from there split it to different sections that you want, etc., based on capabilities, and then go from there. So, what we learned out of this, the most important takeaways, LLM doesn't do everything for you, right? There has to be expertise that has to be applied. You have to take

it do your sequential and careful designing of your systems, even of the LLM explanation to actually get the get a good output out of LLM. whatever we were doing until now was a good start. It was when we created a knowledge base, we we understood what the end of it, we understood we have just we have not at the end of something, we have just begun something,

right? So, it starts it's like opening a door, right? And or like cleaning a window which is dirty or you cannot see it outside, where you start to see a lot of opportunities. Specifically with respect to modernization, if you see, if you if you look at it from a core at the core, there is a knowledge base that is present for you to work with. Then you

you start to see many things that you can do, like conversational knowledge bases we have already done, workflow explanation, flowchart, and on top of it if you want to take integration specific information out of it, or do a dry run simulation of the code, you can do it, right? If you go beyond that and go into the process of things, you have a very good knowledge base

now to use for your diagram design design generation, LLD generation, even test generation, code generation, everything. And beyond that, even risk analysis and planning, right? I've been talking about modernization till now, but if you consider a knowledge base like this for even a normal software engineering team, right? Which has its code explained as a knowledge base in right at the center, the similar set of possibilities in

even that, like impact analysis. change change design assistance, bug fix bug fix assistant, and even support assistant where you go in depth to the code and check what is wrong, etc. Can now be done uh using this knowledge, right? So, this kind of ends where I want to do. Uh it'll be uh good to see any of you picking this up and do improve our software engineering

much better. You can also uh do it yourself or you can do it with us in Tesco technology, where we are building. So, as I told you, 30% of UK's food is given by Tesco. So, we build uh technology that impacts people every day, right? Colleagues and our customers, right? And uh we have um uh offices predominantly in the UK and Europe, which works out with the

time very well. If you do if you if you didn't realize according to different country entries that are available for you to work, right? Okay. And uh the most interesting one for me working end to end, right? I have myself built uh order fulfillment full system from scratch. This is another opportunity to build something from scratch, and you have opportunities to actually work across. Uh I've also

worked on DevOps, I've worked on GenAI now, and more to come, right? So, if you're interested, take a pick. uh you can work with us, do [snorts] interesting stuff. I don't think we have time for Q&A, but I will be available outside if anyone wants to talk to me. If anyone wants to connect with me after, you can also connect. >> [music]