Dr. Andreas Kotulla, @AnwaltJun – AI-Generated Code: Legal Risks and How to Reduce Them #FOSSBack

This talk addresses the legal risks associated with AI-generated code, presented by Andreas Coutula and Chan Yo Yun. They begin by discussing the increasing adoption of AI in coding practices, highlighting a shift in awareness around the potential legal implications of using AI-generated code. The speakers share examples demonstrating how AI often copies existing code rather than creating original solutions, leading to copyright infringement issues. They elaborate on the challenges of distinguishing AI-generated code from human-written code and stress the importance of compliance and legal considerations in software development. The discussion concludes with reflections on the future of software architecture and copyright in the age of AI.

Hi. >> Hi. Yeah. Welcome and good afternoon to our talk about the AI generated code and the legal risks. I'm Andreas Coutula um founder and CEO of Bitsy and today I am here with Chan Yo Yun. >> Yeah, Andreas I think we have to really speed up. We have so much to talk about. So let's just delve in. >> Okay. To start, I would like a short

survey. So if you could please all grab your mobile phones and have a look at this code. I would like to see how many people of you are using AI code generation already or who writes the code in your company. So is that still humans classical approach? Is it just AI or do you have a mixture uh like a dream team humans and AI >> or you

don't write code at all? I'm just curious to see um what >> humans and AI >> humans and an II. So we did the same um the first of these talks two years ago at the Bitcom opensource uh event in Airwood and there the survey it was still a little bit different. So it was not that much AI as we can see right now. >> That was

the AI one. I lied. >> Thank you. So then we would like to have a view on the risks. So what do you do with the generated code? Do you just take it as it is or is it somehow checked if it is legally sound? So there's just yes or no or you don't know. >> You're not obliged to give an honest answer here. I'm a lawyer.

So check with >> And also here we see a big change from two years ago. So many people now are aware that there are some risks with open source and we see that it is being checked. But today we're going to give you some great examples and we want to talk about how all of that works. When we look at AI generated code two years ago when

I had the first talk about that um it just all started and people saw that the AI generation it's not just generating code. So it's a reuse. It's a copy paste. So very often there are parts of the original code in whatever is being produced. >> Wait a minute. I didn't expect that. You mean it's not generating, it's just copying in big parts. So and we'll have

some very nice examples for that. Today the world has changed. So two years ago it just started developers started using AI. Today it's everywhere. If we look at the code or if we from Bitcom every year we do a or every second year we do a audit or a survey with all the German companies and we did that last year and about 80% of the companies said

yes we are using AI to generate code the code that you get from the companies today said to be around 30% is AI generated and on the other hand about half of the engineers they're very concerned about the code that is there because they don't know about the risks and if there are any and that's what we're going to show you today how you can handle those

>> and just a couple of days before we met here I asked you are we really up to date do we still have the problem I thought maybe technology must have must have improved and you said no it's what did you say >> it increased a lot so it's um you had the concern that AI is not important anymore >> but it's getting more and more >>

question is are we using AI code obvious viously. Yes. But the second question is, is it still copying? And obviously it does. >> It does. And we have we added some nice examples in there. >> Yeah. We we'll talk about the charted rewrite today. The one that came up a couple of weeks ago. >> So first we want to see the question is does an AI generate

everything or is there copying? And I just have some examples on how this really works. So on the left hand side of the screen you will see uh what I uh the prompt that I will put into an AI system and here we have copilot that I used and I just wanted to do a matrix transpose. So it's a mathematical algorithm and the AI is really great

in that. So it will quickly generate some code to turn around the matrix. You see it's quite complicated but it just takes minutes to write that. Why can it do so? because of the EI was trained by some code and very often it's open source. So if it would be like that, we should be able to find the source of that code somewhere. So we looked around

and we found something in GitHub and so it's a question to all of you. Can you spot the difference? And I know it's a little bit small from the back, but I can tell you in here it's all the same. The variables are the same. Even the commands that you see are the same, but there is one big difference. Maybe try and you can see. >> Yeah,

there's a license up there. LGPL license. So, this was licensed code that was reproduced by GitHub and there's no more license there anymore. Well, that's good news, right? That's how to get rid of licenses. >> Yeah. So, does that matter? Uh well to the people who use GitHub maybe not but to us lawyers it does matter a lot because obviously this is a copyright infringement because I

don't and we don't have to explain what the LGPL says but obviously we have an issue here. >> Mhm. Okay, I have another example. Again in copilot if you try to write a function is even. So you just want to check for a number is it even or is it odd? Again here's the prompt. I just start writing like that and it's great. The AI will kick

in immediately and it will write an check. It looks a little bit strange for those of you who can develop code. So it's first testing is it I give it a number then it's testing is it zero? If yes then it's even. Is it one then it will return false. If not it will first test is it negative number then it will turn it around and otherwise

it calls itself by subtracting two. If you look at the algorithm it will work but it may not be the ideal algorithm. So it's not really performant. So everybody would ask why would an AI do that? Maybe we can find a source again and this time it's a book Java elegant JavaScript and what we find in this book there is um a whole section it's explaining recursion

and how it works. So you can imagine where this comes from. This is a very nice example to explain recursion to somebody who doesn't know how it works. And many people must have typed this example and the AI found the solution. hundreds and thousands of times in the internet. So it thought it must be good. So this is what you get. It's not ideal. >> Now I

understand your example. >> So if you are an experienced developer, you would just write it like this. So this would be much easier and much faster. >> And I have one last example that I want to show you. Who remembers the game Quake 3 from 1999? >> The old guys here. >> a 386 processor, right? >> Yeah. And you know it has lots of 3D graphics and

to present those you need lots of mathematical functions. One of those is the inverse square root and that's a function which is very expensive to calculate. So it will take lots of time. And there was a very smart developer at that time who invented an algorithm that is very quick by shifting some bits in the register. And this is a very good approximation for a square >>

with a magic number. You >> magic number magic number. Yeah. >> So what you see here um again the prompt I want to have this inver square root. And what is very good here I get a copyright holder I get a license and then I get this great algorithm where you see you have this crazy magic number which works and it's still used today. So in many

engines that need to generate 3D graphics, you will have exactly that >> Again, we looked for the source and we found this. So it was written by ID software. We find this Quake 3 original. There's a different copyright holder, a different license. >> Oops. GPL. >> And again, the code is exactly the same. And again you find the same magic number, the same commands. John, is it

okay? >> No, of course obviously not. Unless of course you would say that this is an algorithm that should belong into the public domain because it would then else monopolize and that's something we can talk about but um other than that this is a GPL violation. Anything that's derived from that is covered by the copy left effect. So the entire software has to be published on in

under the open source license of GPL. Yeah. Okay. So what we see here is a typical hallucination of the AI system. It knows there should be a copyright holder. It knows there should be a license. So it's giving all of that. As a developer, this is very difficult to see. You think everything is fine and it's okay. It looks great, but it's not correct. >> So forget

aboutology and scan code and all the other ones who just look at the headers because you will be fooled by wrong license information there. H >> yeah. And here's one new example that you may have heard about charted. So this was now it's a very interesting case that we >> just had a hands up. Who has heard about the charted case? Oh, whoa. Wow. Okay. Couple of

people here, right? All right. >> And charted what what happened? Um, some developer had written a library a long time ago and released it under LGPL. He stopped working on it. there was a new maintainer and since long he wanted to have a more permissive license to distribute the library more easily. So what he uh he had an argument with the original owner and he said no

no I don't want to change it. So what he did he took the whole charted put it into an got a rewrite which is completely different from the original code. I think he also did a check that was just 1% or so is the same of the code and now he released it under MIT >> still with the name charted under the version seven now. So uh

version six is still in our LGPL the rewrite it's much faster quicker and it's liberal under MIT. >> So that means we don't need copyright licenses anymore do we? >> Well it could mean uh let's do it with charted or the Linux kernel. What comes next? Can we just use any software rewrite it with AI within couple of hours and then republishes under commercial license or uh

MIT license or whatever. Now that's really exciting case because here we don't have a literal uh copy of the original code. If you if you run scan code or even if you run black duck over the the new code, you will have zero findings. It's a completely new software. But yet we believe this is a copyright infringement. First of all, there is no such thing as a

clean room implementation when you use a large language model because the large language model has been trained with all software that's out there. So it has known charted and I tried it out. I mean I don't write a single line of code. I asked clo code hey can you rewrite charted for me? And he only needs that one word and it took him 50 minutes to write

a new implementation of of charted. So in this new implementation was derived from the original software and even if I tell them don't derive from it, don't use it, don't think of it, you can't tell an L&M not to think of something when you tell them what to think not to think about. So obviously clean room does not work. But I think it's not the end of

it. I think we are seeing something big coming up here because if you would rewrite it in a real clean room process, if you would write uh see the specifications and have a new code written by a human or by a AI assisted human, we would later we lawyers would look at the code and see does it have the same architecture? Is this the same SSO uh

structure um sequence organization? Is the core still the the copyrighted uh functionality architecture? Is that the same or is it not? If you somehow manage to create a new architecture with AI or without AI, you could be able to rewrite legally and get rid of the former license. H >> so that's great future. So great future future for people who don't make their living on open source

compliance. Yeah. But uh until that maybe for another one or two year we can chase those people who do a bad job at rewriting because they would just putting it into claw. But um just make sure just because we say this may not have been legal, it doesn't mean it's not possible to legally rewrite with AI. And maybe it's a business model. I wrote you an email

yesterday. Hey, should we just uh shift over, start something new and either hunt those who do a bad job or create a service that creates new architecture for legal rewrite. Shall we start with G Linux kernel or something else? >> And today we heard there is a website already that does that. >> So a lot of questions coming up and I had a lot of discussions today

with people like with Matias out here and arguing about the legal situation of it. We always have to take two different approaches. I'm a lawyer. I don't create law. I don't say what is right or wrong. I just give predictions on what courts may say in the future. And courts tend to stay with what they have learned the last 10 years. For example, that everything that's complicated

must be humanmade and everything that's simple must be machine-made. And everything that's machine-made cannot be complicated. and therefore nothing that comes out of a machine can create a new copyright or can be protected or could be distanced from a former work. Now that's obviously technology But my guess is that's what will happen in the future that it will be really hard. We we get shall we take

questions now? Do we have time for that? But yes, you want >> going back to the inverse square root formula and the code you showed >> is there to discuss thresholdity >> definitely yes >> does it change the outcome of your statement >> it could I mean the uh quake 3 3D because it's a very short algorithm one could argue that the idea which is not protected

should not be protected if you have an different expression of that. Uh if you just rewrite it from the idea, you would be able to create an algorithm that's not a copyright infringement. And that's what we will be talking about in the future with a lot of algorithms probably because we will not just be looking at licensed headers and literal expressions but ideas and concept. And that

means we lawyers will have to learn a lot about algorithms and you people will have to learn about law. Yeah. >> Yeah. So that brings us to we've just mentioned some examples now that's my part again so much what is the what are the problems >> okay it could be cynical a bit we we start ended with a cynical note in the uh last session if you

can not solve the problem make your problem someone else's problem right okay you either solve the problem or you push the problem to someone else if you can and that's reminded us of the game I don't know if there's an expression in that in English. Just make sure someone else has a problem. So if you create the software, tell your customer, hey, I created the software with

AI. It's probably full of copyright infringement. That's all right for you, isn't it? Maybe you can put it a little more nicely. Maybe you could you could uh hide it somewhere in the small print or something. But actually if you can't solve the problem which you should and you at least will help you with solving the problem that's one way to using using contracts to just tell

the other person that you are producing AI created code on the other hand if you are buying the code if you are the OEM I saw that ah 10 minutes I if you are buying the software just make your uh supplier promise with blood that uh whenever there should be any AI written code in there in there and you have to call back all your products, your

cars will taken off the street because you created something with KitHub that he will have to pay for that and he will just pass on this problem to to the last person who cannot afford a complete recall of 20,000 cars. >> So problem solved. >> so we have one nice example from >> you know people big things are happening on the one hand we see more and

more copyright infringements but on the other hand we see and those of you who've been doing open source compliance the last few years may have noticed that you run into an argument all right I understand there's compliance but who cares where are the cases that's what happened to us so I kind of thought let's change sides for for a moment just for for game just for to

try out. Especially when we found out that there was a car in our parking lot from our law firm that had no license um um compulsory statement. We asked them, hey, you forgot to put them in the car. They must be somewhere other where are they? Could you send them over? And by the way, the source code as well probably you have Linux in there. They did

not respond. They had nothing on the website. And uh basically it just said leave me alone in a nice way. So that's that's MG. we can call the name because maybe you will have may have followed our case on LinkedIn or or YouTube. So instead of looking for a copyright holder or a customer, we took our own case to court to the uh district court land of

Munich one. The same court by the way and the same chamber who made the ruling of uh GMA versus open AI. So the people who at least think they have an idea of what technology and and open source is about. So, uh, we, um, filed that lawsuit and, um, maybe you followed up. There was this video where this car was loaded on a truck and they took

it away and brought it back two days later and the only difference was there's a QR code inside now. So, so if you are waiting for a court decision on the question is does along with the software really mean is in the car or is it okay to just have a QR code? We'll have a solution to that, an answer to that. But first of all, can

we have Linux operating system probably 10 times on a car and not provide source code? And could it be a good good decision for a company to rather not provide the source code instead of giving a source code and showing everything what's inside the source code because that's what they decided on. They are working really hard on open source um compliance now. I think they have to

make up for a couple of years and on the way probably they found out oh >> we cannot months already. >> Yeah. um they they try to uh um prolong every uh time frame they have and uh still they have not provided source code and uh uh we have a long list of software that's in there GPL 3, GPL2, LGPL, everything commercialized everything inside because that's that's

what happens usually when people start over they just present a long list with all the brutal uh truth that's inside which is really helpful because this way we we learn that they are using GPL3 software which has nothing to do in in a car because you cannot exchange the software in a car because else you would lose your uh license to operate the car on street. So

that's the case. We have no verdict yet. Um there's a public hearing coming up in fall. >> So we have one more slide from >> Yeah. Yeah. Yeah. We we went to the aaa whoever came up with that stupid name mobility international automobile a de mo mobility in Munich and we crawled into a lot of cars and and uh we I mean that's no big surprise uh

we what we found in those cars and and some people tell us yeah I we would really like to do uh compliance but we have no money for that so please don't sue us. So, so that's what we did. A lot of cars and um and uh actually we found some positive examples. Avatar was one of those and also talk I didn't didn't even know the names

but there are some who are doing compliance and others who don't. It turns out batteries also have a Linux operating system and have not seen the source code of those yet. >> So, okay. So, we see >> we see it everywhere. So one of the biggest questions then how can I detect AI code? So if you get some code base do you know where is AI code

and where is humanly written code and it's extremely difficult to see that technically what you could do is look at the structure of the code. Are there very generic generic variables? Um is it does the code do more than it should do? is there's some overhead or you can look at these semantic very often AI leaves out some edge cases but all of that is just very

vague so will not give you a 100% answer what you could do for example is ask your developers whenever they use AI code please use a tech so we can see in GitHub for example generated by co-pilot or something like that that are all indicators but to see AI generated code is getting more and more difficult as the code gets better and better. >> Then the question

is what can you do in a company or as a developer with this AI generated code? So first of all we recommend have some guidelines. So make developers aware of the risks that you have in AI generated code and then mark the code which parts are AI generated idially which LLM was used which version and also the prompt that you use to generate the code just to

make it reproducible. Uh that's one part of course there are many technologies how you can find it. There's for example um GPT detect or um clean GPT several open source tools that you can use just to find um the code that was generated. You can put the code into that machines and they will tell you there is a chance of 60 70 10% that the code was

AI generated. I did it with some different samples and parts. Sometimes it was very accurate, sometimes it was way off. Um what you can also do is to use snipplet detectors that can detect the code which was originally used to train the machines. That is a one mechanism that you can use to detect that AI generated code and of course you should look at your contracts and

you mentioned that already. What do I need to do in my SLAs and the contracts with my suppliers and with my customers? >> Well honestly we don't have good technical solutions right now to be really sure. I mean if you have uh unlimited funding of course I could uh I could analyze every line of code and see if it has appeared somewhere else literally or also if

the structure has appeared somewhere else. I mean basically I think I could take every product from the street and find some sort sort of copyright infringement. Usually it's easier than that. Usually you just uh look for the GPL tags and know um there's copyright infringement. But that's something coming up and probably tools will be smarter also in the future. So exciting times to come >> Okay, so

that's our short summary of the risks of AI generated code already >> and we ran out of time for questions now. Or do we have a couple? Okay, one minute left. >> We'll have five minutes left. No >> Thank you very much for this really great talk. >> Thanks guys. I love the energy. uh the question about the proprietary code or ownership of the code at least

in some areas right now uh for example in creative space start to feel a bit obsolete and I've heard arguments that in the future we shouldn't even bother ourselves with the copyright of the code because you want to be able to tell but we should shift our focus from the ownership of the code into the ownership of the process >> like So for example, you you cannot

put the copyright on the code that you produce, but you can put the copyright on the process on the prompts and the whole I don't know you know the steps that you take. >> What's your opinion on that? >> I think you're on the right way. It's not the process. It's more what currently they call structure and architecture. That's what receives and deserves protection. But it will

take a time for us lawyers to understand what it's about. for right now we are just looking at the code just literal >> yeah because that's the example uh I saw in the creative space that for for artists generating in AI space a lot of them do not claim any copyright over the >> I don't know the picture something but they claim copyright over the whole >>

could be the future >> yeah so I think it's an important question you could in future use any book that you like you just put it into an AI and say please rewrite >> what is the result it's the same with code or a picture. >> Uh, thank you for the talk. I have a question related to charted. You said like maybe that's a way to rewrite

the license. Maybe that's coming and I'm wondering is the output actually copyrightable at all so that I can choose a license. Is it what because that's the other alternative right so what's your take on that? Definitely we always have to see is there an infringement creating it and do you have protection of what comes out of it and uh usually whatever comes out of a machine is

said to not be copyrightable because it's not a humanmade product however there was I mean he worked five days on it at least five days and so prompting could be a generative human process so and here we are again at the at the process could be but that does not mean that the literal expression the code is really copyrighted. So it strips down to something else. The

architecture again. More questions. >> Yeah, the one in the very back. Uh yeah, architecture is a good point because um I believe that um it comes to architecture again and again and uh you cannot write good code if your architecture isn't available or just uh not the right architecture or enough architecture for what you want to achieve. Um so um yeah my question um how do you

um uh feel like um what is important to um to to to make the architectural >> you know um >> um let us not think technologically for a moment. Um, as lawyers, I would really welcome it if we could say any architecture that's humanmade is good and protectable. Let's say that good architecture can only be made by human. I mean, let's think of it. That would be

a solution. That would mean all architecture is a creative process only capable to be made by human and then we're on the safe side. However, we all know that architectural designs can also be made by machines and they could be created newly and creatively. But um for right now, we lawyers will believe that any architecture is humanmade and therefore that's that's our retreat line from code we

go back to architecture and maybe process something else. But I don't know where this will lead us in >> Thanks guys for this really futuristic >> Thank you very much.