Can LLMs generate Enterprise Quality Code

This session, presented by Tom Howlet and Prasanjet Sakar from Sonar, explores the potential of large language models (LLMs) in generating enterprise-quality code. They address concerns regarding the reliability, maintainability, and security of code produced by these AI systems. The speakers emphasize that while LLMs can create functional code, they often produce security flaws and bugs due to the mixed quality of training data. Sonar has developed a framework for evaluating LLM output and created a leaderboard to assess different models. The discussion highlights the importance of providing contextual information during code generation and emphasizes the need for robust code review processes. They also introduce the concept of the Agentic Centric Development Cycle (AC/DC), aiming to improve the quality of AI-generated code and minimize development issues.

All right, next up um we have Tom Howlet and Prasanjet Sakar from Sonar. They're going to tackle a question every engineering leader here is probably thinking about right now. Can LLMs actually generate enterprise quality code? Not just faster code, but reliable, maintainable, and securable code. So, welcome Tom and Presency. I'll turn it over to you. >> Thank you. Cheers. Okay, so we're going to jump from leadership

to relatively technical. I guess it's going to be a bit of a shift. Um Oh. Ah, clicker. Perfect. Okay, so um so I'm Tom Howlet um in product management at Sonar. Um you probably know us from Sona Cube maybe. Um >> and myself is Pranji Circle. I'm the technical product man product marketing manager at Sonar. >> Cool. And today we're going to talk about creating enterprise quality

code with LLMs. Right. So, um, as I'm sure you're aware, AI is writing an explosive volumes of code that's generating explosive volumes of excitement, um, of fear as I talk to a lot of the CTO's here, and a lot of questions, a lot of really open questions. Um, so the big question we're going to try answer today is, is AI code ready for the enterprise? Um I

think um this it's um we know that we know that the code is very capable that it's functional that it it kind of uh we know it can do some incredible things but is it secure? Is it reliable? Is it maintainable? Is it all the things we need in the main in the enterprise to be able to be used effectively? And can we gain from those productivity

improvements that we see? So, let's start with an LLM out of the box. We'll move on to kind of what else we'll need later. Um, but the first thing we wanted to know at Sonar was out of the box, is the code good enough? Um, and so the way the way most people look at this is benchmarks, right? We've all seen the LLM benchmarks. Every time a

new LLM comes out, people look at the benchmarks. Oh, excitement. We're another.3% up on uh on some functional criteria. Does that help us in the enterprise? I'm not sure. Um so what the enterprise don't tell us is again um how maintainable will that code be in the future, right? And and again is it secure, is it reliable, etc. So um but yeah, what can we do about

press injection? Um yeah so you know to tackle that challenge we have uh created a kind of a benchmark and we have created uh a kind of a framework that actually analyze the code which is being generated by the LLM as well. So we don't don't uh go by their uh the numbers that they are publishing. So what we have done is we have taken all the

top leaders uh in the LLM and we ran that through our own benchmark or our own um you know evaluation framework. Um to do that what we have done is as you can see that we have taken up the top models. Uh we have gone it through the uh you know open- source 4,444 Java programming assignments and then we run it through the results and the results

were evaluated by the Sonar Cube Enterprise. Um so and you will see that you know there are lot many challenges that we found out um but the inherent inherent question is that why this problem is why the problems are there. So the problems why the codes are not so maintainable or the or the codes are not bug free the code is not perfect is because of what

the code is being trained on. So there are mixed amount of code the same the LM is getting trained some of the good code some of the bad code. So, so there's a mix of code that is being trained on right. So, this mix of quality. Um, so that's one of the other one of the major challenge. The second problem is the codes also are having the

security flaws. So, the code is the LM is you know learning from that as well. Um, and obviously there are hidden bugs, right? Um, so those are the major things that we found out uh from our evolution framework. Um and obviously the nature of the uh the LRM itself because they are not deterministic they are probabilistic right so it runs on the probability factor um so uh

that's the first factor the second factor is we those code uh which is being generated right now from your prompt does not have the context uh what uh what your quality standard looks like right so uh so that's one of the other missing points and it is hard to diagnose hard to explainable Well, because the amount of code that is being generated for uh a single assignments

is enormous and I'm going to show you the number. Um so for that we created the sonar uh LLM leaderboard. So this LLM leaderboard is open for everybody. It's sonar.com/leerboard. Um you can go and see this leaderboard. Um and we have evaluated across 50 plus models so far. Um there are open widths model there are uh foundation model as well as you can see that rightly all

the lead you know LLM leaders are right there um you know opus 4.5 4.6 Gemini you know GPD4.5.4 four pro as well. So we have evaluated all of them. Um and the result is astonishing. Um so we have top five models by the pass rate and this is this is not the number that I'm publishing. This is the number which uh the companies are publishing. But if

you look at the number um in here and you'll see that you know obviously their pass rate there are 10 plus models which have been uh which have crossed the 80% uh limit. Um but if you look at the issues density across all of them I'm showing you only five of them in here because I don't have space. Um but if you look at the issue density

um there are across like 20 in between between like 15 to 20 not too bad but you know this is again the number of issues per thousand lines of code that you are generating right but thousand lines of code you are getting 20 to 15 to 20 different issues. Um and you know uh we can see that opus 4.5 is the highest one I mean the highest

rating right now with a relatively low uh issue rate. Um so next one is uh the complexity. So there are two different thing in the complexity side. One is the cyclomic complexity and one is the cognitive complexity. Um cyclometric complexity is where you're going to see that how many ifs and else if and you know for loop and all the other control statements that is there that

are there. The more number of control statements means the more problem that you have in terms of analyzing the code uh running the test running the what kind of test you are going to run that's one of the complexity the second of the complexity is the psych cognitive complexity which is made by sonar um and that is like how complex the code is which is being generated

uh by the L&M itself by a human being right if you if you want to read that code if you want to analyze that code if you want to fix any bug how about that uh you uh complexity. So if I show you the number here, this is the number uh that is being generated from our analysis. So we found out that GPD5.2 high is writing around

1 million line of code for only those 4,444 Java assignments. 1 million line of code, right? Um and you can see the the comp kind of complexity. So the more uh models the more newer LLM models that uh that are coming up uh they are more verbose. So uh the other day we have analyzed and uh updated our data board showing that uh GPD 5.4 pro or

GPD 5.4 in you know uh medium and high and extra x high uh reasoning effort they are generating 1.2 million line of code for those 4.4 uh K Java assignments. That's just about 4.4 4K Java assignments. There are different code uh languages, there are different frameworks as well, right? So you can see the number of verbosity and then the newer models are actually just not creating the

verbose code but they are also generating enormous number of bugs. So code maintainability, code smell, um you know different type of security vulnerabilities. So there are lot many vulnerabilities that we are seeing um you know that there these models are generating. So newer models does not mean that they are giving you the perfect code that you are looking for. So what we are saying is that you

need to look at these numbers and see that what exactly you are getting on. Um so you do you write the code but you need to verify this as well. Um so here is the point that the number of vulnerabilities that we have seen across GPD40 uh and you know J uh let's say 4.5 thinking 4.5 uh six thinking for for from the Opus is around uh

you know 150 120 150 so per million lines of code we have 150 bugs 150 security vulnerabilities that those are block or security vulnerabilities right um this is really really a a great eye openener for everyone. We when we started evaluating this uh you know from our perspective we you know we were blown away looking at this number but these are the facts I mean we just

ran through this uh stuff um and at the same time you have the bugs per model as well. So these are critical bugs that uh that are you know what we have seen is that the models which are coming up in the in the new newer side they are creating more fascinated bugs. So not those human not those bugs that you have ever seen. Uh there are

new control statement there are new bugs that we have seen they are getting introduced in the newer models. Um so yeah so you have seen this uh problem that uh that is there. So LMS are not perfect. Um you know they are generating code they are doing all of the good stuff but you need to verify as well. Um so these are the problems but uh Tom

uh remind me what exactly are we doing to to fix this problem? Well, we could do what Sony Cube always does, which is catch all these issues in the quality gate and then ask the developer to go back and fix them, but that's not going to prove very popular and it's certainly going to slow down all of the the the productivity improvements we've seen. So, so we're

thinking about how do we how do we improve that? Well, if you have a new developer come into your organization, you don't just sit them down in front of a computer, give them access to all your repos, ask them to code away, well, hopefully you don't, and then let them commit the code, right? You first need to get help them understand okay what are we trying to

achieve? What are the abstractions in the code at the moment? What is what is what is the how how is it represented? How is it architected? What are the standards you have in your organization for writing code? Yet when we do these benchmarks, we're basically just sitting them down with no context. So what we've been kind of working on is making sure that the agent has access

to all of these things it needs to write the enterprise level code in the first place. Um and we're we're kind of calling that uh AC/DC which is seems slightly catchy um the agentic centric development cycle. Um now there's still a human centric one that wraps around this but what we want to get from AC/DC is that the agent is able to generate that good quality code

in the first place without having to go back and the developer tell it to fix things later. So the Arogentic ecosystem as it currently stands, um the first thing is actually giving it the context it needs. Now I'm sure you've all kind of heard of context windows. If you throw tons of stuff in your context window, it doesn't necessarily help the agent at all. The agent gets

confused. It doesn't necessarily interpret it pretty well. So what we're focusing on is for whatever task the agent is working on to give it the right context, give it the context it needs to do that well. The two areas we're focusing on at the moment is one kind of coding guidelines. So if it's about to make a change to a data access component, what are the guidelines?

What are the rules that we have in our organization around creating data access components that it should follow? And we look at that, we look at past issues when people have written data access components, bring those back and and tell the agent, okay, avoid breaking these rules before you start. And that in itself has a great impact. The other thing we do is give it architectural context.

So um if you haven't been on Sony cube in the last few days, you may not have seen our new architecture feature. So now within Sony cube, you can define um your desired architecture. You can set out your models. You can say every controller can only access a server which can access the data layer etc. You can break that right down. You can get as fine grain

as you like and you can create a model and Sony cube will then tell you if you're creating code which breaks your model effectively. if you've got architectural sprawl. So, we're already giving that to the agent. Um, and that makes a real difference. Um, it doesn't just make a difference in the quality of the code and the number of issues it creates, but actually it works a

lot faster because it's not having to dig through all of the code to try to understand the architectural the desired architecture in the code. It can just get it straight from an MCP, get it from the agent. Okay, this is what the architecture should look like in the area I'm in at the moment. these are the specific bits. So we give it the specific architectural information it

needs to carry out that task and that that not only kind of gives us better quality but also helps the agent run with less tokens much faster because it's just working less hard to try to discover this information itself from the codebase and I think there's a lot more we'll do around around context. So once we've generated generated helped the agent generate the code, we then want

to go through what our junior developer would have done with the code review process. Now our goal is to get the code as as clean as it possibly can be before it's presented back to the human. So we've been developing a new service which does a really allows the agent to do a very deep analysis. Um so this isn't just like a llinter that you might have

in an IDE. This is doing full kind of a full analysis that you would normally only get in the CI in an analyzer. But we're finding ways to do this using kind of previous analy context from previous analysis to do that in one or two seconds. So the agent can just do it every time it writes a file. It can go and get an analysis, make sure

it's not created any issues. And obviously we all know the earlier you you identify and fix issues, the faster your development process goes. So we're kind of moving the kind of the sort of agile CI process to an even faster cycle within the agent iteration iteration loops. Um and yeah, we're seeing some good results from that. So when the agent finally comes back with a whole load

of code and it really is a whole load of code now, right, we're not just looking talking about code co-pilot completing things. We're talking about Claude having completed a task and written maybe a thousand lines of code. Quite feasible, maybe more. it does still need to be reviewed. The agent isn't accountable for the quality of the code. It's not going to it's not going to get you're

not going to fire it if it makes a mistake. Um so somebody needs to be accountable. Somebody needs to understand what's been written. And I spoke to a few CTOs yesterday and uh and one thing that keeps coming up is a fear of um developers rubber stamping a change a code and not being able to go through it. And it's really hard, right? you're I think the

state the the stats kind of show if you've got a a change over 400 lines of code the chances that you really kind of review it properly are very slim right you you tend to do the same amount of time reviewing a thousand lines of code as you do 400 lines of code you just don't do it as well right so there's a lot of code review

tools coming onto the market and we're building our own as well which basically break down this change into smaller parts that can be understood and can be kind of reviewed separately and given that context as well. So the reviewer needs the same context, right? This is following this pattern. This is doing this and then you can really verify it. So I think the code review side of

it is we haven't got the tools at the moment for these types of changes. And I think there's a real rush to develop that which we're doing it. So once you finally kind of get through and you've committed the code, you've probably still got a whole bunch of issues on your on your codebase. Luckily, agents are now getting really good at fixing those issues. So we're building

remediation agents that can go through your issue backlog and start breaking them down. But then you can also give that task to agents as well to do that works really well. So hopefully we're we're kind of starting to come together in this agentic centric life cycle which I think we're just starting to see now what development is going to look like moving forward in the enterprise but

hopefully this has given us a bit of a taste. So actions what can we take away? So first thing really understanding the limitations of your LLM. Now obviously we can look at the benchmark. You can get into more details. You didn't really see it there, but you can actually see the types of issues that each LLM is creating. And each one is different. I'm always shocked. It's

a different set of problems. It's a new set of things. They they kind of they fix some issues and then something else emerges. The complexity increases. Something else emerges. They're always different. Um you then think about what context are you going to give your agents, right? And there is your kind of fixed fixed context that will be on your agents MD. You might have skills and things

like that that you're using which it can which can standardize the way it works in a good way. But also think about dynamic context as well. What context can you provide for a specific task that it's going because you don't want to fill your context window up with the whole company knowledge base. It's not going to help. Um measure the impact of the changes you make. I

you hear a lot of stories about people having these huge clawed MDs and it just making their results worse, right? I think you can you can this this stuff is not deterministic. It's not easy to see what the results will be. When there's an upgrade in model, the impacts can change again. So you need to have some kind of way of evaluating the the quality based on

the process you currently have within your enterprise, based on the tools you're giving the Um and then provide the agent with ability to verify the and fix issues itself. Don't leave it to the developer. I guess this is obvious, right? Everyone kind of realizes we can't have developer fixing the problems the agents create. Um an efficient review process is going to be critical. I'm sure everyone's thinking

about it already. We haven't got time to review in the same way. And and finally, start thinking about the backlog of technical debt you had. we now have capabilities to start eating through that in a more economical way whereas in the past fixing old issues on repositories was was generally nobody had time for it. So yeah, so that's it. Um come and see us at the booth

if you want to talk about this more. Um the things we've just shown with the context augmentation and the agent analysis is available in beta now. So I can I can set you up with that today if you want to try it. Um we'd love to hear what you think. love to hear what you're doing around this. Um, it's obviously very new and emergent. There's so much

hype around people in their bedrooms kind of building applications in minutes, but not around what's happening in the enterprise. I don't think people are talking about it enough. So, we love to talk about this. Cool. We got questions. >> Yeah. Um, yeah, I I'll take the first question. Um, um, is it Sorry. Yeah. Oh, okay. that um is it relevant to uh compare the number of line

of code that has not been reliable measure in software development in the past well um that's just one number um so the number of line of codes shows the verbosity means the number of line of code that you're increasing it means that as Tom mentioned that the review cycle gets more and more complex and there are more and more chances that you're going to get uh bugs

security vulnerabilities uh you know code smell and all the other problems right if you look at the older models or the openweight models those those used to write for example we have uh last year we have evaluated a code a model called open coder um that has generated like I think 50,000 line of code but you know the 50,000 line of code and the million line of

code you can imagine the the kind of uh verbosity that we are generating and the chances of getting those bugs so yes it is relevant u but it is not just the only number that we should rely on uh we should rely on the other metrics as Yeah, I think I think the the lines of code is a fair measure of your maintainability cost to be frank,

right? You you need to if you're reading through those lines of code, it's a reasonable thing about maintainability cost. It doesn't say much about functionality. >> Um what would be an average human sparse rate in your leaderboard? Well, um that's so so so this is this is not something that we are measuring on. uh but if you look at this is coming from the from the models

itself or the companies who are making making the foundation models um you know what we have seen so far uh last 6 months all of the models that are coming up including the open word model for an example let's say from Z.AI AI uh couple of uh couple of weeks back we got a model called GLM5 uh that's open model and even that open word model is

uh uh you know crossing the 80% boundary. So looking at that we could say that since last 10 10 to 12 models out of the 50 models have crossed the average of let's say 80% out of the 50 models that we have uh you know evaluated. Is code maintainability still an issue if LM can consistently regenerate working a working product from specifications? I mean, I think this

is super interesting, right? If you've got the unit tests and you've got all of the other checks in place, yes, you can probably regenerate refactor your code more easily, but you need to have like I mean just a just a prompt saying what to build isn't good enough, right? We need the specific requirements. Those are probably encoded in tests in some ways. Um, so I think this

is this is a super interesting question that will start to emerge. We might not think about the maintenance of the actual code, but we still got to think about the specification and how complete that is. I mean, whenever I've done rewrites, there's always new issues emerge, right? So I'm sure that would be the same unless you have a perfect specification which nobody has. So >> and even

even for that matter uh LLMs by nature is probabilistic. So today you generate something and then tomorrow you just run the same prompt. You may not get the same code. You may not get the same exact d of line of code and the verbosity and the personality for that code as well. Um okay next question. Um why do you want only benchmark Java? Well you know that

was a pretty good start for us. Um we started we are right now working on the other programming languages as well. Um and it is a open data set. Uh we picked up an open-source uh Java programming assignments. Um and it was only it was you know readily available. So we picked up Java as to start with um and but you know we are Okay next one.

Um >> I think >> we kind of answered that. >> Oh uh oh it's gone. Hold on. Okay. >> How this differ from just having all of this code base and pointing claw at it. Example architecture diagram and desired architecture. So the I think I I mean there's it's worth experimenting with giving an architectural diagram. I think what we're finding is giving it the specific context for

the job that it's going to do is more effective than giving it something general. But that's not to say that you shouldn't give Claude that good, you know, an architectural diagram as well, right? It could it could still be useful, but that could be huge, right, for your organization. So I mean um we're absolutely not saying this is the only useful context and there's still so much

discovery still to do on context. So yeah I would try different things and test the test the results. That's the key bit. Um are you testing the LLMs directly? The best results of coding are when the model runs in the harness. Um we um you know we have our own evaluation framework that we have developed. So we are using LMS with um you know with our uh

with our coding standard. Um and also at the same time we are um you know we are using the APIs directly from from the from the models uh or the companies that they are building it on. Um so we are not exactly using claude or uh codeex or gemini or you know uh goose or any other stuff but yeah we are using our own framework to run

it through um to check uh you know what exactly are there that's the best way to evaluate a uh you know framework or I would say the kind of a model to I think the next question for you >> the confidential bit I thought you would do that you're in marketing >> um no you are okay to take them uh you're okay to take a picture of

that Um there are uh yeah so the so all the products that we have uh shared these are all either on public either public beta um so yeah this is this is fine for you guys okay um probably last question do you see LLM producing vulnerability free code in the near future um that's very hard to say that's very very hard to say because what we have

seen the trend is on the upper side not on the lower side so the more and more newest models comes newer newer models are coming in they are actually creating more and more security vulnerabilities code smell bugs rather than you know compressing them. So some somewhere they are learning from their previous mistakes which is not generating the obvious bugs. So they are suppressing them they're training that

data on um on that same data but they are creating some more fascinated bugs fascinated security vulnerabilities. So I so the trend is actually opposite. Uh I wouldn't I I don't know whether it will be uh fair to say that you know they'll be uh a completely bug free or vulnerability free uh LLMs. >> I think even if they do you're going to have no guarantee. So

you're going to have a way you need a way to verify it anyway, right? They I'm sure they're going to get better at the the issues they're currently creating. We still need to verify. >> Cool. Cool. >> Thank you very much.