Juliette van der Laarse - How AI is Re-shaping Engineering Work (And How to Deal With It)

In this session, Juliette van der Laarse discusses the transformative impact of artificial intelligence on engineering work. She introduces the AI flower framework, which provides insights into how AI is changing the landscape of IT activities and the skills required. Drawing from her experience as an engineering manager, she highlights the importance of defining IT activities and understanding AI capabilities, emphasizing that current frameworks do not adequately prepare professionals for future changes. Juliette outlines a process for evaluating AI's role in IT tasks, from identifying work activities to predicting shifts in job roles, ultimately arguing for the necessity of 'engineering philosophers' to navigate these emerging challenges. Her goal is to democratize access to her findings, ensuring the framework remains free for all engineers.

Please welcome Juliette van der Laarse. Um I think she's working double jobs recently now >> [laughter] >> cuz she's been working a lot in her own time. Uh and in her day job she works for the NN Group um the insurance and investment company. because as an engineering manager she realized how fast the change is moving. Um very exciting but also a little bit scary as we

noticed in the audience this morning when we're asking questions. Um so what does this actually mean for your work, teams, and future? looking for clarity Juliette has developed the AI flower framework. An open framework that makes visible how AI is changing engineering work, skills, quality, and risks. Combining the skills fossilization model You have to explain to me what that is. >> I will explain it. >> Continuous

change becomes inevitable. She won multiple awards. She's a leader for women in tech. And even her own employer is now looking at how to implement her framework. Give it away. Thank you. I I have my mic, yeah. >> [applause] >> So first of all, I am really happy to be here today. Especially because I will actually be presenting a hobby project that got way way bigger than

I anticipated. Now we are nearing the end of the first day of DevWorld. I hope that today you have sat through some presentations, followed the workshop, maybe looked around at the booths. And I hope that by now you already have some clarity on what AI will actually mean for your work. And I hope that my presentation here today will be the cherry on top of that cake.

I will bring you a bit more long-term. I will bring you along the transformations that we can expect in the future. And as mentioned before, I will share how AI will change and reshape our and how we can deal with it. So, my name is Juliette van der Laarsen. I work as engineering manager at NN Group. And my own IT journey started when I was 12 years

old. I'm currently 34 years old. So, for 22 my life revolved around anything computers. I absolutely love computers. I love IT. I love working in IT. And when AI first came around, initially really really excited. Now, I actually have a background in game technology. So, I played around with some early AI with building MPCs. That was already quite cool. But then the AI boom actually came. And

suddenly everything seems to be possible with AI. Now again, the first reaction was excitement. Suddenly we can do so many cool things and it feels like this whole sci-fi future is here. But then I started to realize how extremely disruptive AI actually is. And I was looking at this field of IT that I'm so absolutely in love with. And I started to get very afraid that I

would lose all of that. Because everyone can be an ITer now. Everyone can be an engineer. We don't need engineers in the future. This is This is all the fear-mongering that I also read on my feet. And not only did it make me a bit uncertain, what will it do to IT, to the work that I do, but also to my actual job. And as engineering manager,

I don't only deal with my own nerves, I also deal with the nerves of those that I manage. And I realized that I could not find any answers that were satisfying me. And I know that I'm resilient. I know that [snorts] I can find other jobs. But frankly, I do not want to leave the field of IT. And if you like IT as much as I do,

you probably feel the same. So at that time, I realized, okay, if I do not get any answers to this out there, because when I was reading articles or I was listening to podcast, there were two camps. >> [snorts] >> One said, nope. You can never do this with AI. I used AI. It is horrible. It hallucinates. This will never be an AI job. This will remain

a human job. And then Metcalfe's launched. And that changed the playing field again. So clearly, this way of thinking does not work. If you look at current AI capability and you try to use that to explain what it does to our work in IT, by definition, your conclusion is outdated. Now there is this whole other camp who's saying that IT will not exist in But this is

more philosophy. I'm not against philosophy, but I work in IT. I want facts. I want numbers and I want concrete information. And although I don't have the answers myself, or I should say didn't have the I realized that I'm actually very good at building systems that create insights. I'm good at building frameworks that do provide us with the answers that I was looking for. So, outside of

working hours, in my days off, in my weekends, I thought, "Fun little learning project. Let's see if I can build such a system." Now, why I was building this, and it evolved, and I talked to friends, and they gave [snorts] me inputs, and it evolved again, and at some point in time, I realized, "Whoa. Now, it is starting to become something that can actually answer these questions."

And the second thought that I had was, "This should be freely available." I want every engineer out there, every company out there, to be able to use this framework that I'm building for free. And I did not want this to hide behind a paywall, because this impacts every single engineer. >> [sighs] >> So, I figured, "Okay, let's make this a bit more serious." What I'm presenting to

you here today is the framework that I mentioned. And I call this the AI flower. Initially, because the shape kind of looks like a flower, and I don't know if you see, but I'm fully tatted with flower tattoos, so it's kind of a theme that I'm into. I decided to that I would take this serious, so I decided to approach O'Reilly. I wanted this to be freely

available, and I basically sent out an email, "Framework looking for owner." And I told them, "I want a platform where this can be published, but it should remain free. As long as you do not pay place a paywall on there, you can have framework. Now, apart from publishing the actual framework, I will also be publishing my whole thought process of getting there. I'm feeling quite confident about

what I did, but I'm also probably biased in a lot of ways. Maybe I may thinking mistakes, maybe I'm missing interesting perspectives. So, not only am I now able to hopefully help a lot of people, but I hope that all of you here today can also help me. Look at the framework, give me input, and then collectively as an industry, we can make this work. So, I

will be publishing my thought process on O'Reilly as well. And in this presentation today, I will take you along a part of this thought process. I cannot share all the details because I would need multiple hours. So, I tried to condense it a bit. And in essence, I will be taking you through three major thinking steps. So, the first step is we will dive into IT activities.

Second, we will dive into what AI capability actually means, and then we will translate it to what that means about the impact that it will have on Okay, I've already shared a lot of I want you to relax, open your mind, and please go along with the thinking process that I'm going to share with you here Now, if we are talking about how AI will change our

work, the first step is thinking about what actually is that work that we are talking about. What actually is the work that AI can potentially start replacing? And I defined roughly 150 IT activities. So, these are activities like writing codes, managing CICD pipeline health, and many, many more. Now, for now, for this presentation, you will have to trust that I did a great But again, I will

also share my reasoning. So, I identified these activities because I was able to internally deduce them from the framework that I made, and I then externally validated them by comparing it with job openings, programs, and IT projects. And I was able to group them in technical domains. I will not dive into the technical domains here because it will just take too long, but you can think about

something like code engineering, something most of us actually touch upon in their work. And as you can see, it started to look like a flower. By now, I also had so many documents that I could not manually handle it, so I decided to use Neo4j, and perfect metaphor, it now came to bloom. For those of you who don't know Neo4j, this is actually called a bloom graph,

everything seemed to fit. Okay, so I now identified the work activities that we work on. These are 150 activities. Not every engineer does all 150. If you are a DevOps engineer, 40 of them will apply. If you are an infra engineer, let's say 30 of them apply. So, you can identify the activities that are relevant for you. Now, next, I started to define what it looks like

if you perform these activities at the highest standard. I was not innovative in this approach. Everyone who studied IT knows about this. Well, actually I I hope you know So, I took all the known industry standards and best practices that were and I started to identify which IT activities are impacted by them. And that's how I was able to generate North Stars or a definition of excellence.

Now, I used AI for the generation and I have to be honest, the output was It hallucinated a lot. So, the next step was months of manual reviewing, going through it, making sure that it was actually correct. But at least I didn't have to write everything from scratch. Now, if you look at the North Star, this is what you can expect. You have a general description, definition

of excellence, a definition of excellence checklist so that you can actually see if the activity was performed according to the highest standard and anti-patterns. And it also links to all the standards that this was based on. So, if a standard is updated or changed, the whole framework can be regenerated. Now, I placed writing code here just for you in the audience to get a feel of what

that might look like. Here we have a description of writing code. I've linked all the frameworks that I based this on and here you can see roughly some of the content that is in there. Now, mind you, in practice this is 16 pages because I wanted to be very detailed, very specific because if you are a little fake to AI, it ruins everything. So, this needed to

be very detailed. Okay, so this is what we have right now. We have 150 IT activities. All of them have their own North Star. And these North Stars are grounded in these industry standards. And I was now able to also derive the required input for each activity. If you know what activity you are performing and you know the highest level, you also can derive what input is

required. And there popped up that quite a lot of the input that's needed is actually output of other activities. There you have your dependencies. So by now it started looking something like this. We have defined all the IT activities. We have defined the North Star per activity and we have defined the required input per activity. Now I'm going to make a small side step in the presentation

because once I had this, I realized that two potentially interesting things started to pop up. One, for 150 activities, I now have input requirements. And that means that you as an engineer, as a team, as a company can select the activities that are relevant and you now know all the required input to perform those activities. This is your list of information that you have to start gathering

to build an internal knowledge base. Either to be used by humans or in the future potentially by agents. Now I started experimenting with something else as well because for those of you who have experimented with AI, the information that I had by now is starting to get very close to an AI constitution. So I linked my Neo4j to Claude and I asked it, "Please build me a

skill for every single activity in this overview. And it was able to do so. Is it perfect? No, but I think once you tweak it, once you improve it, and once AI get better, there is a lot of potential here. >> Okay, so so far, we have identified the work that we do. Nothing AI related yet. This is just the work that we do. Now, I know

quite a few universities that are quite happy with this already because they can actually use it for their students to identify what IT work they need to learn and at what quality. Now, we dive into the second section. If we want to know how AI impacts our work, we need to know what AI is actually capable of. And I do not want to know what it's capable

of today because this will be outdated tomorrow. I want to know what it can potentially be capable of in the future. And in order to answer this question, there are few sub-questions. What does it mean to be capable at something? How does that translate to LLM context? And how will this change over time? So, we will first dive into the first one. Let's define capability. And usually,

taking AI out of account for capability is in a human being, you have knowledge and skill in a topic. You can use that knowledge and skill in practice. And you can use it as a certain operating range with a certain level of independence. So, either you're able to write a small piece of code, or you're able to write build a whole application, understanding the whole range of

activities that is involved. I think level of independence you can guess. Either you need a mentor helping you or you're able to do all of this yourself. what's next? We expect that you perform it at a certain level of quality. These three topics together is what defines capability in general. So, how do we translate that to AI? Now, the first step, the knowledge and skill, we assume

that AI knows And we can consider this a pre-adoption check. If AI does not have the needed skills or knowledge to perform an activity, you cannot use AI for this And of course, as AI evolves, this will change as well. Now, we have acceptance here as well. This is basically your pre-flight Did it actually meet the acceptance criteria in order to go live? So, these are two

I don't want to say static points in time because if AI evolves, you have to run these checks again and the output might be different, but this is not where the interesting change is happening. It's actually happening in the middle. So, this is what I would consider AI capability, the operating range in which you can perform something and the dependence with what you can perform something. So,

let's dive into that. Because how will this change over time? I define two skills. won't go to all the details because it is a bit too much, but for operating range, let's say you have the smallest range where you work on atomic level all the way to comprehensive where you can walk where you can work on the whole range of things. Second skill that I defined was

the skill of independence. [snorts] One, fully instructed baby steps, you need full guidance, versus you're able to do something completely autonomous. And I'm now able to turn this into an AI capability matrix. And well, at least for me, this is where the fun begins. Because we can zoom into every single cell. If we look at AI capability that is atomic in range and it needs direct instructions,

the AI capability is that it only knows what's in the current prompt. It follows directly the how and the what designed by humans. Cannot offer any suggestions itself. It's unit-based output only produces a single artifact. And this translates to basic prompt engineering. What we had quite a while ago. And for those of you who've paid attention, this is now starting to work like a periodic table where

we can move across all of these cells and start defining it. So, for functional and guided we're looking at things like prompt engineering, context engineering, feature-specific agents. We can do this for all 25 cells. And that means that we can now identify how AI capability changes over time even for those cells that we have not encountered in person yet. This technology does not yet exist. But who

knows, it could be published next week. It's going quite fast. And while working on this, I identified a few what I call AI eras. So, we have the deterministic era, interpretive era, investigative era, adaptive era, and the last cell where AI is fully sovereign. let's look at the last cell, but because honestly every other cell we need quite a few human beings. This requires a lot of

transformation. This requires a lot of work. Personally, I was most afraid for this last cell. So, what would that look like? If we If we ever enter this last cell AI is able to work fully autonomous and comprehensive, and that means that it can define strategic goals and KPIs. It can operate across all domains. And it executes recursive self-improvement. This is potentially what we are moving towards.

Okay, so we have now first defined what our work actually entails. We defined IT activities. Next, we defined AI capabilities. So, what is AI actually capable of right now and potentially in the future? And we will now bring that all together into one big insight. What is the actual impact and meaning? So, we will start bringing it all together because each cell I was now able to

plot on those 150 And I started to ask these questions. we have this AI capability, how does it impact this activity? Will the activity still exist? Yes or no? Maybe it will have evolved into something different. Maybe it will have fossilized. And with fossilized, I mean that it went to the background. It became part of our foundation, but we no longer touch it. And what does this

tell us about how this activity will now be performed? So, if we look at writing code at the cell where it's the least interesting, this is what we have. Atomic direct instruction basic prompt engineering. Cool. This now translates to the system requirements for this specific cell and to what I call the human requirements. So, if you want to work in this cell, you need to have an

AI extension in your IDE and as a human being, you need to know how to do prompt engineering, how to check the output, and how to decompose your work into AI tasks. Now, if we look at the final cell, the we will no longer touch writing code. Again, I did this for 150 activities, so unfortunately not have enough time to go through all of this, but I

think you're starting to get the point. That means that per cell and per era, I'm now able to identify the impact it has on IT activities. We can now have a system readiness checklist per cell. We can have human learning plans per and we now have our transformation plan. Either for you as an individual or for your organization as a whole. You can also identify what cell

or what AI era you're currently at and if you identify where you want to move next, you have all the information here to start making that change. what does this actually mean? This is a long thinking process, but I started with one question. How will AI reshape our engineering work? And in this presentation, I will focus on the last sales scenario because this I personally was most

afraid of. Not because I don't find other jobs exciting, but because I really like IT and I don't want to see it go away. So, what I derived from this method is this. In the future, writing code is done by agents, if testing is done by agents, if all of this is done by AI in some way or form, maybe agents will be outdated in a year,

who knows. We will need product thinkers. You have to identify what you want to build and AI is able to build it. Now, I think we've already started to feel that this is probably where we are heading. Now, there's one argument that I hear a lot. And it is that AI cannot take accountability. Like, no, AI will not replace our work because humans are the only ones

that can be held accountable for mistakes. AI cannot be held accountable, and thus we need humans to review code, for example. But if AI is writing a thousand lines per minute, code that you have not written yourself, there will come a point in time where you are not able to review this code anymore. In the future, probably you have an agent writing code, an agent reviewing code.

But then you end up with a statistic. So, how big is the chance that AI did it wrong? Maybe there's a 40% chance that it hallucinated. Cool. You build a second opinion reviewer, and you build more, more until you lower that percentage of risk. So, another role that we will see in the is risk acceptance. Someone needs to understand the statistical chance of AI doing it wrong,

understand the impact that that will have on the [snorts] work that's being performed, and sign off on it. Take accountability, and state, "Yes, we know the risk. We know the potential impact, and we are okay with this." Now, these two roles can be in one human being. So far, I mean, it's great that I identified this, but as someone who likes IT, was not happy with this.

Um this did not get me excited, but luckily, along the way, I identified a third role that will be extremely important. And I'm calling them engineering philosophers. Everything that I've shown you in this presentation is based on a framework. It's based on definitions and on skills from which you can derive all of this content. In the future, you might not be touching the actual work with your

hands anymore. What we do need people to keep redefining what it means to do strong engineering. We need people that have to keep building these frameworks from which we can generate input for agents. So this is something that will have major impact, and I think you can imagine that being aware of the ethics of using AI is part of this as well. How long then will it

take to reach this last cell scenario? I don't know. Maybe the capability is available But this is not the speed we have to look at. The speed we have to look at is how quickly we can transform, because even if this almighty AI exists already, our systems are not ready. We have a lot of shaky systems, low data quality, things that are fully connect, code bases that

were merged but are not actually perfect. So, we will have to spend a lot of time in system readiness. Not just in actually building the AI capabilities, also ensuring that the performance is going well, um mitigating the AI risk, and automation. So, I want to give you one critical question before I go to the final slide. If this god-like AI exists, and we need to be system

ready for it. And that means that our systems need to be of high quality, and our data needs to be of high quality, and we have gotten all of that in place, either by humans or by AI Once we have this perfect IT landscape, please don't choose AI. Please go for cheaper automation. I I bet you there are a lot of things that we're now trying to

do with AI because we were too lazy in the past to actually fix the systems underneath. Please don't work yourself in bankruptcy and go for cheaper automation. Then the final question, we've talked about the last cell scenario, but will we actually ever get there? Now, this is the only cell in which humans are no longer doing IT work. This is the only cell in which we're talking

about the three roles that I just described. But will we actually get there? Well, that depends. If AI is able to be truly trustworthy and reliable, maybe the answer is that it will never be able to. We have to see. If AI is affordable, and that is an issue at the moment, and if AI is regulated or not. Maybe in the future the European Union decides the

impact of doing this with AI is so high, we forbid ever doing this with AI. So that activity will always remain human work. So this is what that depends on. And then we can use our pre-adoption and our pre-flight checklist to see if you're ready for that change. Okay, so I hope that in this presentation and I realize that I gave you a lot of information. But

if I would have only shown you the final slides, if I was in the audience and I only saw the last slide, I would not have been satisfied. So I wanted to take you along with this thought process of how I derived this conclusion. And uh I did not come here alone. As I mentioned, this is a hobby project that I've been doing outside of working And

I want to give special thanks to all of these people on the slides who actually asked me critical questions, provided me with input, and then I would come home and think, "Wow, I did it all wrong. They were right." So special thanks to these people. I also want to give special thanks to the O'Reilly team and to all of my NN colleagues that are also here today.

And with that, you can find QR code to my LinkedIn. I hope that you will connect with me after this talk. I'm open for coffee, I'm open for lunch, I'm open to talk about this. On my LinkedIn I will also share all of the slides that I shared And once it is published on O'Reilly, you will be the first to know. And with that, I hope I

was able to give you some answer to the question of how AI will reshape your engineering work.