AI-Powered Cloud Native Modernization: From Real Challe... Savitha R, Daniel O, Kenneth K & Duncan D

This panel discussion addresses the complexities of AI-powered cloud native modernization, focusing on practical experiences from experts in application modernization and cloud services. The speakers emphasize the balance between the cost of maintaining legacy applications versus the potential for innovation when embracing modernization. They highlight the pitfalls of superficial modernization approaches, such as lift-and-shift strategies, that fail to unlock valuable enterprise data. The conversation delves into the effects of technical debt on software delivery, the importance of security in cloud environments, and the challenges posed by AI-generated code in terms of quality and maintainability. The panelists explore solutions to these issues, emphasizing a structured, iterative approach to modernization and the need for ongoing mentorship to develop the next generation of software engineers.

Okay, welcome everybody. Oh, that is loud. Uh to this panel discussion about uh AI powered cloud native uh modernization. Um we're going to do first a brief introduction of uh the persons on this panel. I'll start uh with my friend there on the right, Daniel. >> Thanks, Duncan. Hey, everybody. Uh my name is Daniel from IBM MWL and that's it. I'm from Boston by the way. >>

Hi everyone. I'm Ken Kilty from Microsoft. I work on the Azure Kubernetes service team uh with a focus on migration and modernization. >> Hello everyone. My name is Savvitraan. I am I'm a senior software engineer at Red Hat and I focus on application modernization and I'm also a maintainer of the conveyor project which is a CNCF sandbox project. And my name is Duncan Doyle. I am the

director of product at Solo and one of the products that I manage is the CNCF governed uh K gateway uh ingress gateway on Kubernetes. Um so it's a panel discussion as you guys can see there's a microphone up there. So obviously we've prepared some some questions to set the stage and to get things going. If you have got a question for our panel feel free to come

up to the mic. Please come up to the mic and don't shout because this is being recorded as Daniel reminded me and we'll get the show uh get the show started. So to oh >> so what shout so to get things starting so we're going to do a bit of discussion about the challenges that we have with application modernization and every I think everybody have seen certain

aspects of it but we got here some expert that have been dealing with these kind of uh of of problems in the real world in their uh open source projects in their enterprise uh environments. So to start first the the cost of inaction. So usually if you look at applications and legacy applications the cost of inaction is basically the balance between having to spend your effort in

maintenance and compared to what you can do in innovation. And usually it's not so much the cost of the operational expense of your application that becomes problematic but really the lack of innovation that you can do because of the time and the effort you have to spend in maintenance mode. So my question for Kennedy is uh can you talk a bit about your experience with that balance

between the cost of inaction and the uh the innovation and particular about things like the um the modernization theater where people start adopting technology but maybe not utilizing it to reap the benefits. >> Sure thing. Um what we have traditionally seen is uh sort of a cow path to the internet was virtualization taking applications that are running on prem as is putting them into um virtual machines

within Azure. Uh over time we've seen customers move to trying to containerize these workloads but more or less leaving them alone. We we call this just um uh lift tinker and shift doing just enough work to kind of get them into the cloud running in a container as best they can. And the problem with that is that um there's still a lot of data and things locked

away in those applications that's really not accessible. Um there's a desire to get it done quickly uh as cheaply as possible um into the into the cloud, but it makes it a little bit difficult down the road to actually sort of leverage a lot of the data you have spread across your enterprise. Um the other thing that can come up is something that I think is is

very near and dear to the community here in Europe. You know, events like the CRA that are really just around the corner. I believe that some of that starts this year. um are going to require a huge push in doing a lot of this kind of work rather quickly. Um and you know kick choosing to kick the can down the road with sort of keeping things the

way they are um has an unfortunate cost when events like that come up. I'm old enough to remember Y2K. My first job out of college was making corrections in cobalt coat and Y2K. I had no idea this was what it would do for a living. Uh and sort of seeing this come up again, I I I can see that we're um you know in a moment now

where we need to think more seriously about doing this intentionally. Thank you. Anybody else that wants to comment on that? So, I'm not as old. Well, maybe I'm as old. I remember Y2K, but it wasn't my first job. My first job was the transition from I'm from the Netherlands, as you can probably tell from my accent. The transition from the Gilder to the Euro. So, we had

sort of a similar problem with with legacy code that needs to be updated store procedures. Remember those ones? Exactly. Um, so then go to the next topic. So once we have identified like the cost of inaction and we realize that we need to modernize the applications to reap the benefits of of innovation being able to innovate as a as a company. We also have to recognize when

that needs to be done right so at some point you will reach that breaking point that tipping point where your systems have accumulated a lot of technical depth where it's no longer maintainable. And technical depth is accumulated over time. You might have it intentionally in your applications because you just want to cut corners because you want to move fast. But at some point you have this breaking

point where you say the cost of the application maintenance becomes too hard. Software delivery and developer productivity really starts crashing down. So Savvita, do you have some experience around the breaking point identification and the problem with legacy applications when it comes to things like for example security or compliance or aspects like that? >> Of course. Yeah. So anytime you touch a legacy application, it takes more than

a week to figure out what are the risk and what are the consequences of a change I'm going to make and throw in uh security. Um the legacy applications rely on uh perimeter security but then if you want to be on Kubernetes environment then you'll have to go like opt for like zero um zero trust in that means the security should start uh with encryption and then

um from them go go on every layer and so on. So these are like when someone wants to move from one stack to the the latest stack lift and shift would not work in this case because like it's already missing like the principles that would secure the applications and then comes the compliance. Uh for compliance for example like there are several organizations, governmental organizations and then there

are like um uh guidances that says like okay you need to have this uh cryptographic um algorithm in your code to be like compliant with the XYZ. Um the legacy applications will have dependencies that won't support these new cryptographic algorithms and so on. So that's also a problem and the main problem that I think is that when a developer spends more time maintaining the code rather than

building on top of it that feels like a major impact and that feels like a breaking point. Um but tech dipt is everywhere right tech dip is like universal it's just not isolated to one company what we do in conveyor is conveyor is a open source community based project and we believe in that that you solve once and not every single time and kudos to my product

manager who told me about that you solve once not everywhere. Um what what I mean by this is that we have rule sets shared rule sets repositories of uh migration scenarios or like going from one technology to another technology that can be widely reused um by others. These are vetted these are like solidified uh uh like migrated scenarios and everything. So any organization can adopt from technology

A to technology B. >> Anybody else? I'll add a little color to that. Um, you know, the other thing that you have to keep in mind is with um the AI modernization tools, there's a quick gut check of like I'm just going to chat with an AI and I can modernize all my applications out of the gate. I mean, we've learned some pretty hard lessons for that.

You know, at Microsoft, uh, we had a a tool for Kubernetes called Draft that helped create deployment artifacts for legacy applications using templates, right? We thought we could think of all the different scenarios an application could have. we could spit out the Kubernetes manifest and obviously the the vast array of applications out in the wild that kind of failed rather miserably and didn't work for us. We've

pivoted now to something called containerization assist another open source project using an LLM. But even there we thought, wow, we can certainly enhance what we build. AI is going to, you know, create the answer to move all this stuff. And it eventually started creating um a lot of outputs we never in a million years expected. Um it was really quite fun when one week it decided to

start suggesting um Amazon base images for our containers to deploy onto AKS. That was a great week to see those tests. Uh and we introduced something in you we look we looked to the Kubernetes community. We brought in OPA Rego and created a policy framework around our deployment engine to let you control the registry that the you know you're effectively taking in a probabilistic model and making

it sort of deterministic. So, um, it's definitely something to keep in mind with these tools that there's a gut to just go out there and do it. I can just oneshot all this. It's absolutely the farthest thing from the case. So, continuing on that, we saw and it was recently in the Dutch news for uh worldwide news, but I read it on a on a Dutch uh

IT uh website. There's a lot of uh um uh discussion around uh AI slop nowadays, right? There's a lot of uh people that are maybe new to it and that now can contribute to their open source projects but it generates a lot of AI slop a lot of uh lowquality uh problematic to maintain code and we've seen open source projects like for example curl no longer uh

uh admitting uh pull requests with AI slop. So it's sort of bound to this concept what we discussed earlier called YOLO. you only look once. Um and uh or others might call it vibe coding. So now that we see a lot of people adopting agentic frameworks and LLMs, um obviously there's a lot of opportunity for application modernization and legacy code modernization, but I assume that the YOLO

approach although it has a nice ring to it, is not the way to go forward to migrate large legacy I'm an old guy Java based codebases. So can you talk a bit about that? What would be a proper modernization flow for agentic use cases for migration? >> Sure thing. Uh what we have found that's worked is um effectively there's a lot of terms out there things like

spec flow. Um in our modernization tools what we do is we build a plan for the what we're going to do. Um we track the progress u and create a summary in three stages using a git branch when we create it. Um it creates full traceability of what actually occurred. Um it's all markdown so you have a chance to review the plan before you edit um and

change it as you will add or remove things. Um can task are also extremely helpful. We call them task in our framework. Uh the idea is if you're going to do something repeatable I'm going to take a Postgress library that used on-prem basic authentication to a database. This is now running someplace in where there's some form of modern authentication. That's a very repeatable task. There's libraries that

need to be changed. The you know the secrets no longer there. using sort of the pod identity and whatever framework you're running on. Um and so identifying those kinds of tasks that are common, we've created libraries of that in our tools based on code we've looked at and it's been very successful. Um you're effectively trying to make something that's not deterministic deterministic to a degree. Um but

that in conjunction with other things like we talked about like OPA and Rego together um is a very good winning formula that we've seen on large scale migrations. >> Yeah. Uh just one thing it's a very interesting question from the agent framework automatically generate bunch of the pull request at the same time. So that exactly same question we had like another panel on hour ago in the

tech develop experience and a bunch of the people are actually really concerned about the people actually keep using AI tool and then generate source code and then create the pull requesting autonomously automatically every single day and then the senior like a developer architect they actually super big burden to review every single detail and they don't actually verify application and functionality and security comprised at all. So, so

I don't mind actually you going to use AI tool or assist tool but you have to be careful and uh you maybe uh make sure everything is okay. Maybe sometimes it's a manual process but that is you're going to change your like some culture and process how to generate code how to uh verify and how to push and coming rather than just uh allow everything you do

to AI bot to do that. Yeah. But then again that creates another problem as well, right? Because AI generated code and I think there's probably a lot of people in the audience that are start to use like AI and LLM to actually create code and I've seen myself that I can be 25% maybe 50% more productive but it also means that I create a lot of code

a lot faster. So aren't we then like moving that bottleneck from the creation of code to the reviewing of code? and especially since reviewing code requires >> more senior people >> and have an understanding. So c can somebody comment on that on on how we sort of get rid of that bottleneck or try to mit mitigate that? So yeah so unfortunately at CNCF standpoint we don't have

any relevant open source project on the CNCF landscape yet but so but other way maybe you can maybe you can try another AI B or LM to review help you out uh to like a review the code and then check it on every single nonfunctional capability security performance and compliance whatever they call that but I would more like suggest the the more junior developer they whenever they

use the AI tool to try to understand the your business requirement first and then like make a call okay we're going to use maybe simple application logic we're going to use AI tool in case I don't have any like a front app experience at all but I pretty much spend time like a couple years back application java net python doesn't matter and then you're going to maybe

quickly use to create some like not even fancy like a web application using AI tool like a cursor or copilot or whatever you want and then but more mission critical application like a back end based on your knowledge you could maybe ask them some kind of like a syntax or some example but you probably do by yourself to create your business logic because you fully understand the

business requirement rather than AI. So if I ask if you ask some question to AI generated code maybe they mislead or based on some AI model or some kind of training level. So maybe it's a kind of more like a practices rather than one simple solution at this moment. Yeah, that's my answer for actually that question. >> What's working for me is like u going back to

basics like look at the requirements why are we building what what are we building? It's the same case that applies to organization also like you have to provide the good context as to like um we are moving from one framework to another well lift and shift if an application is just using like a session state and you just lift and shift without even thinking about like a

shared session or something like that then it's problematic that's cannot it cannot scale >> or if an application that takes like two minutes to start um then you cannot even use Kubernetes autoscaler feature because it's going to take two minutes and if you're in retail or banking, you're going to lose a lot of millions of dollars, billions of dollars. So for me, what's working on working or

like personally I would say like reviewing looking at the specs or like making sure that what and why what am I changing why am I changing and what's the end goal? I think it's applicable for a new green field as well as like brown field. >> So specifications are important, business requirements are important. But now I'm going to go back to Ken's Y2K cobalt system that was

poorly documented, poorly tested, and we know like AIS are good at understanding code bases, but they have a limited context window. So, they might be a lot worse at actually understanding full systems. So, how do we cope with that problem with undocumented, badly tested systems that we now have to move hopefully using AI to be faster and more agile to this new world? >> Well, there's a

there's a couple elements here. Um, one thing that we've been struggling with at Microsoft is um, finding a way to do this with some of the, uh, openw weight models and smaller models. Um, we have found those extremely good at point in time changes, small code space changes. Um, but the lack of the ability to reason over a relatively large codebase is extremely challenging. Like as a

data point, our P50 um, for Java projects that are being migrated is around 100 to 120,000 lines of code u for projects that kind of enter into GitHub copilot. Um those are fairly substantially large code bases. Um at this point we are recommending um some models you know that um you know our commercial models like cloud sonnet you know for those kinds of reasons and that's a

huge challenge that we are we are looking in and hopefully can work with the community and find ways to break it down into smaller pieces. That's number one not everybody has access to those things and it's it's hard to sort of compete with the kind of data center hardware that some of this has my employer included. um you know in a case where you might need to

do this completely within your own tenant or environment and so that's kind of a big challenge. U the other thing that is important to remember um it has to be sort of discrete broken task. These things do not work well u with taking a full-blown one-shot approach even with a plan. Um we really have done a lot of work and emphasize that breaking it into constituent task.

something as simple as I talked about um you know using an assessment to identify common problems and then having common tasks for each of those such as refactoring you know Postgress or moving from an on-prem queue that doesn't exist to where you're going to go are pretty important um you know and that's that's definitely a challenge that I we're hoping to leverage the community for going forward

>> um I want to add more to what Ken said um the challenges are real and one other challenge that I constantly see is that uh some of The libraries are like very custom to a company uh large enterprise they have their own infrastructure own libraries they reuse across like multiple applications and LLMs are not trained on those right so LLMs are generally trained or the data

that's available outside and they are not available they're not trained on the custom libraries and whatnot so the performance is really bad respective of whatever LLM that you throw at them what helps is like having some kind of custom rules or like some kind of guidance that provides extra context to um the LLM which then can solve the um refactoring or generate the right code which can

be reviewed by a human again because again uh these things are in production these things are in enterprise that matters a lot and these are some of the workflows that we have integrated into conveyor and conveyor AI like how um Ken was saying that there is this git workflow where like it can be like used for um um review or like reused um from the for the

next set of migrations and you look back and you fetch whatever that was solved and then reuse the information that's extra context but that's useful context not u just like random ones. So um one of the things that we are also trying with conveyor is a concept called solution server where um uh migrator or like successful solutions get stored and it will be like reused whenever like

similar problems come uh come again and hints are generated that's extra context is passed to the LLM and so the migration is a little bit more successful and not like randomly generated um bunch of iterating and you're making improvements over and over with new knowledge with new >> feeding the AI with new knowledge and and updated knowledge about that migration. >> Yeah, you you just got that.

I just had that in mind. It's like analyze um fix uh if review iterate go back. It's like a loop >> the aentic loop. There we go. So, I've been asking a lot of questions here. I hope it's interesting so far. We also have a microphone up here and there quite a bunch of people here and I can't imagine that there's nobody with a question. So if

you've got a question, don't be shy. Test. Test. Okay, so it's working. So um I really like what you what you guys said, but I wanted to connect a little bit with some food for thoughts. Um because this is what I'm experiencing personally. I'm trying to do a lot of mentoring for junior people especially juniors people that are coming from the university and what I am finding

out that there are a lot of people that nowadays and I really hate when they do that um copilot told me copilot told me to do this and I wanted to connect with what you were saying that says that in the open source community the maintainers are a bit sorry for paraphrasing but fed up of the um multiple PRs that AI is generating and the senior people

that need to review and so and so forth. So I'm asking myself and I'm asking you as a food for thought. Um who is going to replace these senior people? Because if the junior they are just going to use AI eventually the senior people they're going in retirement. So who is going to do the actual review of that code that goes open source? What is your vision

and your idea there? Thank you. Sorry if >> that really great question. So I don't have a solid answer but I have a I have share my some experience. So so there are many um like a evolving advanced agentic some pattern kind of thing right now doesn't matter Java or any program languages and then based on that agentic capability that will like increase some kind of like

a job performance like a agentic s sur agentic like a senior developer kind of stuff. So which means the the the fundamental level agents they going to do pretty much repeative and like a fundamental job like a just generate code and then and then the more advanced like a supervisor or whatever you call that advanced agent they probably more primarily focus on like a code review or

try to understand the your holistic your business requirement and current business domain. So but not yet because So you require bunch of the like a reasoning platform and models and also relevant application but without that you you don't have any kind of stuff and then you cannot do that and then currently human being actually do that but I'm not saying so all senior people will be replaced

with agentic like some senior architect or some other stuff but even if we have the agentics senior like a developer we still some human being orchestrate and design that kind of workflow multi- agents and for specific like a code review or approval kind of thing. Even if you complain the human in the loop inside the deck workflow and then you're going to last person as a senior

to make a call to approve and push it into the like a production specifically like a mission critical application not like a simple web application. >> No. Okay. Sorry, >> I'm going to approach that from a different angle. Um, I'm going to approach it from people's perspective, community perspective. So, taking a leaf out of open source projects that we contribute here, the awesome open sources. I think

that we should invest in mentoring leadership and grow the junior engineers into the senior engineers. I don't think the job's going to go away. It's just going to evolve. So, we all got to evolve with it. And then like the principles are already there and we are doing it. There is this ladder of growth. So all all we have to do is like provide those guidelines and

grow those junior people into senior people so that the cycle continues. >> That's there's also a big responsibility for educational institutions to make sure that we keep educating engineers because what I found with using AI and I I call it Peter Steinberger calls it agentic engineering which I like a lot more than VIP coding. What I've realized is that I can now apply 30 years. Yeah, I'm

an old man. Uh, of software engineering experience to do all this new stuff, but it requires all that expertise and the knowledge that I've acquired over all these years. So, I think it's our responsibility to make sure that people still acquire that knowledge because that's the only way, at least currently in the state of AI, to actually guide these agents and LLMs to the solutions that we

want them to build. Sorry, I interrupted. >> Yeah, I'll add a quick comment. I couldn't agree more. Uh there's a paper you might enjoy um written by Mark Rinovich and Scott Hanselman that if you've seen I think really needs to be more widely communicated advocating a big change to how computer science and things are taught. Um I'm old enough to remember when co when autocomplete came by

the senior engineers said you people are lazy you're never going to learn the SDKs if you do that. I mean at this point in this stage it's autocomplete on steroids, right? There isn't really any intelligence or reasoning in there in the way we would think you and I having a conversation, right? Um the other thing that's important that and I deal a lot with what you suggested.

We have a lot of junior engineers who come up with those amazing solutions and I'm that you know there's a couple techniques that work really well. Uh somebody kind of alluded to this. Uh I personally enjoy putting two AIs against one another and having them argue my case and present me some findings on to why something is or isn't correct. I do the same thing with engineers.

You know that tell me why that's a good idea and show me. Um, and a lot of times, you know, the things that'll come out are quite fascinating, but you'll find a lot of edge cases that way as well. It's just a new way to think about using this tool like the next autocomplete. Um, and once it becomes part of the education process, you know, it it'll

work itself out. But it's the same thing again when autocomplete and idees and we moved away from text editors, Vim and such. Um it's a very similar kind of of exercise but very valid question and thank you for the question >> and thank you for your answers. >> I think we've got time for we got another three and a half minutes so we got time for one

other question and then we can close it off. Obviously we'll be available after this session as well for for chat. So when you um mention the breaking point between when you have to spend all your resources on maintaining often the case is you have maintenance and feature development and that leaves actually a very low margin to kind of do the refactoring because you're like okay we have

to deliver those features and and so the the tension is between yes I have a pay tax to maintain plus I want to build capabilities of business value or business features. How do you see AI filling that reduces the margin actually of refactoring? So you said it's good for simple repeatable tasks. Okay. How how far can you go and and what kind of formula do you apply

to say okay this is an undertake we can do for a larger initiative of modernization. >> Sure. In the context of modernization um we initially focused almost completely on the IDE experience in Visual Studio Code and quickly realized that there's a human in front of that that doesn't scale terribly well. Um where we were putting a lot of investment we see this in the community with things

like home GBT and other projects that are out there. Um, a lot of this is going to have to happen at, you know, asynchronously at scale. Please run the assessment across this hundred repos, right? Um, I call it the grunt work. A lot of that grunt work will happen. It's almost like if you had a human analyst that did a process left to right and had four

steps. Just removing the grunt work looking for dependencies. I need to build an sbomb for this uh this new regulation that's going to be coming up. Um, by the time it gets to the developer, there's there's a concrete of steps that they need to take. So, we're looking at it as a way to remove that. There's no way AI will be able to do the soup to

nuts. I'm not a big believer in that software engineering is going to disappear. None of us are going to have jobs in that sort of thing. Um, but it's going to have to get something where we almost kind of get out of the context. It can't just be a chat in a in a window and this is going to have, you know, we have to become comfortable

with doing this at scale with, you know, working with agents and coding who are going to run a bit asynchronously, but come back to us with an action plan that that we get in there. Um and so we're putting we're putting a lot of our investment in you know those kind of scale motions with coding agents and a CLI things we know customers can automate you know

with uh their their CI/CD of choice. One of the things that I also want to add is that um when you have hundreds of application you have identified then you can like Ken said that you can automate the assessment one once you have done the analysis and automate auto automate the assessment then conveyor supports uh story points to every issues. So like you can create migration plans

and then you can say like in this particular week how many applications or how much I can tackle in addition to adding new features right. So like you have both the things and then you plan accordingly and then you give that to conveyor AI or like any other tool like the um assistant yeah any other tool that you have and that at your disposal and then you

can do the migration parallelly like splitting your time. >> Thank you very much. Okay. And I think that was all the time what we have like 30 seconds and in those 30 seconds I would like to thank uh Daniel Ken Svita uh for being here. Thank you for asking uh questions. I hope you've enjoyed the session and enjoy the rest of the conference. Thank you. >> Thank

you Duncan. Thank you everyone.