Roy Wasse: The Definitive Answer to Measuring Developer Productivity

This talk discusses the challenges of measuring developer productivity and skill level in the software industry. The speaker shares insights from research, including arguments against traditional metrics such as aptitude tests and the significance of experience and educational background. Various hiring methods are examined, including open hiring and the use of coding interviews, while emphasizing the importance of objective assessments of skills. The presentation highlights the development of a platform called 'reps', which incorporates specific tasks to measure a developer's abilities more accurately. Additionally, the speaker addresses how generative AI impacts developer roles and suggests that while technical skills are crucial, the broader factors influencing team performance are complex.

[Music] ladies and Gentlemen please welcome our next speaker rwas presenting the topic the definitive answer to measuring developer productivity so I'm standing between you and the drinks so thanks uh for showing up on the last Talk of the day um I'm going to talk about a subject uh that I think many people here probably have an opinion about how to measure developer productivity how to determine skill

level of a prod of a developer um these are questions that are not straightforward to ask and everybody has well ways maybe to test this and we're going to talk about some some ways that we can maybe do this in a better way so who am I um well I put some background up in the slide here once upon a time long ago I studed computer science

then I studied uh philosophy of Technology with a specialization in AI started working as a developer at the consultancy company daughter company of Cap Gemini worked there for a number of years uh became an engineering manager but always um well kept track of what was going on in the Java space especially was a jug leader for many years and started my own company uh together with a

co-founder in 2017 it's called open value and I've been doing lots of hiring in that period work together with lots of developers also so I also had a lot of opinions about this subject and I'm sure uh that you also have like tricks to determine how well good somebody may be how well of a fit somebody might be for your team and I have a few of

these on the slide so maybe you ask like are they familiar with well acronyms that are well used a lot in the industry like you aren't going to need it of hyper media as the engine of state and so on and you want to make sure that people are familiar with those maybe you think it's important that they understand certain algorithms I used to work with a

CT back in the days um if you didn't get an answer right um about how to implement the binary Tre search algorithm he would stop the interview there but it was kind of harsh so yeah um I don't know if that actually was a good idea probably not because some people were let go at that moment who I thought had a great background and maybe could really

add something to the company maybe you just go to the internet and see what other people are asking in a technical interview maybe that works for you but maybe somebody that's then taken interview has read the answers there as well so maybe not a really genius idea then there's this more wild idea of open hiring I don't know who is familiar with that but the idea is

that we all have biases and you want to prevent your bias from hiring the right kind of person so what you do you just put out a job posting and you say anybody can start on this job and then after like a week or two you see how somebody is doing and if you're both satisfied well you turn uh the probation period into a contract so it's

pretty pretty wild I'm not sure that I would want to do that but the funny thing is um I ran into a company that's actually doing this in the space of software engineering a while back I must say they were doing this with more the junior profiles and it worked well for them uh C said okay then I think what most people do is that they use

proxy variables as well in the selection process you look at somebody's CV uh their prior experiences educational background references from other people maybe you let them do some questionnaires online etc etc and you get some insights from that so this these might all be good ideas oh that's of course the Whiteboard coding a really important one I think that's a good way to test how how somebody

can perform under stress if you're in a job interview you need to write some code on the Whiteboard there's a lot of stress at that moment and if you can perform really well then without your IDE the normal environment in which you're normally writing code well that says more about how you can handle stress maybe than about anything else in my opinion so it's a tricky way

I think to uh select anyone and then there's the lengthy assignments maybe you have like a homework assignment that you ask someone to do that can maybe sometimes take up like two or three days and then you need to be lucky enough that somebody's willing to complete that assignment for you and that is not going to maybe a competitor that has a more well stream lined hiring

process so um I saw a customer of ours and they have been using the same assignment for a number of years and there are multiple answers for that assignment on get up so also a little bit tricky and who says that what what's in the assignment is actually the right thing to measure so a lot of questions around that as well so you can also look at

Google how's Google doing it well they're basically doing everything that was on the previous slide so they have a couple of interviews online to see what your motivation is how the fit could potentially be what your background is then you need to do like an assignment you need to complete a small project and if you go through all that then you get a round of interviews so

that's a lot and I don't think a lot of companies uh can afford to do all these kinds of things and is it actually really needed because who knows what actually works and what's really interesting is that there's actually a bunch of research around this subject and I ran into this uh after I contact Ed a guy at Oslo university called gunar bergerson who basically dedicated his

entire career to measuring um developer productivity and I went into his PhD and I read it and I saw some references and I learned a lot and from those papers I read other papers and I got quite a few interesting insights from all of that and I want to share some of the things that I found out so first the use of aptitudes test so a test

that somebody takes online hey maybe you need to solve some puzzles there and it's supposed to say something about your potential fit for a development role well since the 70s it's already known that these aptitude tests don't really work and um companies have been using them for a long while even when I started my career I needed to do an aptitude test and I know for a

fact that based on the outcome of the test you were either giving uh well no job offer a job offer as a test engineer or a job offer as a development engineer or Dev engineer so that's uh uh pretty wild I would say there's an interesting study uh done in the 90s where a group of students was asked to do either the general aptitude test for the

United States Air Force or a specialized test that should be really good at selecting developers and turned out that both these test worked equally well and that the predictive value of those tests was almost zero so better think again if you want to use an aptitude test in your selection process or if you want to measure developed productivity and so on so I got a question for

you um so who here thinks it's really important to have a computer science background if you're going to be in the role as a software engineer so please raise your hand if you think it's important well I think about 20% of the people rais their hand and I for one said for many years in my well I would say long career I'm getting a bit old um

I I thought that for a long time um it's really important to understand how the fundamentals under the hood work because then you're a better Problem Solver and so on I would use this argument all the time but then I work together with a lot of really good software Engineers Architects and so on who studied like philosophy or physics or well something that had nothing to do

with computer science so I changed my opinion a bit and there's actually a study around the subject that shows this as well so a lastar meta study that was being done showed that shortly after graduation there's some correlation between job performance of the developer and their educational background computer science study but after a few years it's almost zero the correlation zero 05 is well equally equal to

there's no real correlation between the computer science background and job performance so it doesn't make any sense according to this piece of research to ask for a computer science degree if you're also asking for like five years of experience in the field that's something that you can take away another question then um how important is that experience then who thinks that a senior developer should have at

least five years of IND of Industry experience again ra your hand well it's a bit more bit hard to see it's a bit more than the other question I would say almost half of you so at least five years of experience before you can call yourself a senior so let's again look at a research or few research paper actually around this subject and surely there's a positive

effect of having prior programming experience and job performance and that's that was also um being researched but beyond the first few years there's actually no real effect of experience according to a two different studies so think about that after a few years it doesn't really matter if you have four years of experience or eight years of experience and so on and I think if you're honest you

can maybe recognize this so for my for me this was also a little bit of an eye opener but if you look at well the really smart guys that you work with smart girls that you work with and you take into account how many years of experience they had well sometimes somebody is really really good after just a year two years three years and sometimes you work

with somebody who have had like 10 years of experience but you know um they don't bring something extra to the table in your opinion so also again if you are searching for a new team member maybe not necessarily think about at least five years of experience because you're going to miss out on some people who might have been a really good fit it's probably not something that

you want to do in this market okay then uh my last question who here thinks that gen AI is um going to replace us as developers within the next few years for the better part of our job I see one two three four see four hands so that's really cool so be nice subject to chat about during the drinks for sure so I don't know the answer

to that but I do know that it has some some impact uh um on us as Developers for sure there's an interesting uh paper called refactoring versus refactoring from the beginning of uh this year and what that paper basically says is that it's becoming harder to progress well basically from a junior to medor to a senior developer due to these llms and I think it makes a

lot of sense if you want to be really good in software development learning Details Matter if you want to be good at writing regular Expressions but you better do a lot of practice but if you don't do it that often you probably go to Jet gpg and you ask please write me this recx right and this goes the same for if you want to work maybe you

don't regularly work with plsql and you just ask jet gbt to generate the query for you then you don't really learn it right so this can be uh this can be a problem uh there and the question is also for whom is it ideal is it ideal for a junior well that's the question it can help and Coach maybe the junior but can also prevent them from

learning the deal details as just mentioned for a metor it can be a really handy tool and a senior usually already knows the kind of output that uh um well jna gives you so that's a bit about the role of llms oh I got one more question that's right um we have uh certification in our field so the question is is certification important for a developer do

you should you have the proper certification if you're well maybe working as a net developer or a Java developer who thinks that's the case this is really great just one hand I love this I totally agree um so being able to write software is a skill knowing how to do it and not knowing what to type in it's not about the factual knowledge it's not about knowing

what's in the API it's not about being able to spell out what the compiler does it's something that you internalize it's more like a holistic uh thing that you acquire over time like learning how to ride a bike you can read the manual about how to ride a bike but you need to practice and I think the same goes for coding but Ness it turns out that

having the right knowledge is a really strong predictor um a central predictor as CH says in his paper 1998 for how well you be you will be performing as a developer so maybe therefore if somebody puts on a lot of well programming certifications on his or her CV it does say something probably but that's also uh interesting so I talked about a skill so what is a

skill a skill is an ability something a person can do like maybe winking your eye but no matter how much you practice you can either wink your eye or you can't it's not something you can really improve upon in time a skill is a special type of ability it's something that you can actually improve over in time by practicing and there's there's a nice model here that

I actually used in my graduation assignment back in the days that you can use to measure skill level and this is also something that's really interesting for our subject today of course and this model is being created by drivers and drivers it says there are five different levels that you can uh go through from beginner to expert from noice to master and you can think about this

level as follows if you want to learn how to drive a car you start out as an office and you well need to understand where the steering wheel is where the clutch is how to shift gears and you need to actively think about how to do all these steps and you try to internalize it as good as you can and the first time you try to drive

away with a car but the engine turns off you practice a bit more and you get used to the procedures you understand put the clutch down shift gears gently let the clutch go put down the gas and you practice and practice and you know the procedures by hard but it's really hard to get it right and get the timing in a good way but at a certain

point in time it becomes more easy and you go from comp competent to proficient if you're proficient you still need to actively think about what what you're doing but you learn how to well break in time before you turn the corner uh and so on and a certain moment in time you know it becomes more automatic no longer actively thinking about it it's more shifting to your

procedural memory and you can even maybe talk to somebody sitting next to you while driving through a busy City Center then you have become an expert and some people can progress to the next level and they become a true master so think about maybe a formul One driver that can drive very very fast through corners and talk on the radio at the same time think about a

guy that you know that drives into his garage at a high speed and stops the car one cimeter before the wall and it's C him well so it seems no effort at all and it would be great if you would U well could be working with developers who also reach this master level and I think maybe if you ever experienced flow while writing code you may be

operating at um the the Master Level you're no longer actively thinking about the programming constructs that you're going to use you're just reading specifications or you're thinking about the problem you're trying to solve you're not really thinking about how to actually use your IDE just using the shortcuts that you have internalized and you're actively thinking about the problem and you're typing the solution at the same time

this is a great feeling and it would be great if you can well work in that state the whole day with others that also have that state so it would be good if you can find people who have that high skill level or is there more to it do we need to take more things into account something was also mentioned in the keynote this morning what about

soft skills well yeah um I think it's definitely important but that's also again look a bit about some research around this subject um so first of all you cannot say that somebody's really skilled so there will be good performance research from Campbell McLoy and oper and S says and you also have things like knowledge and motivation so if you're not motivated well nothing's going to happen right

and if you don't have any knowledge the same thing Campell says there are eight different dimensions that determines how well somebody will perform in a certain job and depending on the type of job some of these factors are more or less important okay eight different things and skill is just one of them okay well maybe not that important then well McGill looked into the study of kampbell

and looked specifically at what what of which of these dimensions are most important for a software developer and his paper he says technical skill is definitely the most important one of all those factors okay so maybe do focus on that a bit then there's another interesting study done by jensson he investigated almost five million bits that were being done on approximately 800 projects that were being put

there by clients and um service providers could do could do a bit and he looked at the project results that were executed by these service providers that won the bits and he found out that if the service provider would have a lower skill set and and if he would also offer the lower price there was a 9% increase in Filly rate so there's some correlation there as

well well directly between the skill level of a service provider and the success of a project all nice we're talking about individual developers here but what about teamwork isn't it about how well you fit into a team well it turns out um that is actually quite hard to measure team performance it's not so easy to say what what makes it up there's a complex um combination of

factors that are influencing each other but Accord according to Lang Wong and Jeffrey individual skill level of a team member is a central predictor to team okay again skill is is actually very important to say something about the productivity even inside a team and there's a bunch of other research mentioned here that basically say the same thing so uh famous book by Bo he looked at many

different projects and he said that the people factor in any software development project is the most important predictor of success and that the developer attributes of those people are the best predictor for well um improving the software productivity there's also a study that looked at technology Stacks that were being used and the turn out it doesn't really matter if you build your software project with java.net or

even PHP I was a bit disappointed by that being uh having a Java background but it is what the study says so it's more about the skill level than about the stack for instance uh small side step um it turned out that um um PA programming is also a nice way to boost productivity but what we usually do at least what I see a lot as well

is that you pair a junior with a senior to get the junior up to speed but it turns out that if you match people who have equal skill level that pair programming gives the best results so that's maybe something you can can think about um if you um P programming your project you know is a lot of the this what I'm telling you not just also Common

Sense the US Bureau of Labor Statistics they actually got it right I would say in the definition of a software developer it's someone who usually holds a bachelor degree in computer science so knowledge is important it is a central predictor right and has strong computer programming skills again it's about so why not higher based on skill level and if you're a developer and you want to well

progress maybe would be really nice to know what your current skill level is and how you can improve the level so this is where uh the work of gunar uh comes in again so a lot of the things I just told you you can find some references to it in his PhD study and uh he he wrote this uh in 2015 and after this he went on

to create a platform that Incorporated a lot of the insights I just shared with you and now it's going to get a bit more interesting so this is some of the work that g published you can look it up at Google Scholar and these are specific papers that he wrote um well basically because he needed to do some research for the platform that he wanted to build

and he needed to understand how he can measure skill in an objective way we talked about a few approaches already the homework assignment the why whiteboard coding and online questionnaires well there are some issues with that and I would say they're definitely subjective I mean who determines that a specific whiteboard whiteboard coding assignment is the best way to determine the skill level of a developer that you

want to work with so we need something more objective or at least backed by research one of the things that gunar could use is the Dunning Krueger effect you probably heard about this effect it's about uh the idea is that if you ask somebody how good do you think you're going to perform at any given task people tend to rate themselves too high unless they're an expert

then they tend to rate themselves a bit too low but normally people rate themselves a bit too high but if you ask the same person how well do you think your peer is going to say that you're going to perform on the same task they tend to rate themselves more fairly but that's an interesting effect something that Guna could use in a way to somehow determin the

skill level of a developers and he looked uh in specific into specific research about measuring skill level and he saw that what started already in the 1950s people wanted to understand which traits of punch card operators made them stand out that were huge productivity differences and 60s this went on and and the 70s a lot of papers were published and there's actually a nice meta study that

looks at 20 different papers and they basically have kind of the same inside if you want to say something about productivity developer productivity and you want to have like an objective way to measure that you need to look at specific tasks not just any task but a task that a person does um in the normal day job in the environment that they're also normally using so in

our case that would be our regular IDE on our regular operating system that we using and the task should be about something that you also normally do in practice so probably that's not implementing dyra search algorithm right and don't think too many people do that on a daily basis it's more like solving a bu integrating something and so on so the task should also be about those

kinds of things so again something that gunar could use he found an interesting study done in the 80s around lisp who here has worked with nobody oh one sorry one one guy yeah so um lisp has uh I never did any lisp uh but apparently this has a a limited amount of way that you can solve a problem around 30 different ways to solve different kind of

programming problems that you usually encounter and this study found out that if you are really good in solving uh certain problems with strategy a b and c it says a lot about how you're going to perform in De andf so you need to find the right kind of tasks in a given programming language and find the common problems then you can say something predictive about other tasks

in the same programming language so with that idea G now needed to do a few extra things to be able to create a system to measure um skills in an objective way he wanted to incorporate the factor of time if you ask somebody to take a test you want time to be a factor otherwise you can wait forever maybe so he did some research around that he

also used some research from aert around how to measure quality of software so there's even an ISO standard for um software quality and he could use a lot of those insights as well so he could use that to see if the code that was being delivered was of the right quality and then he needed to understand what the right kind of tasks were and he used research

from deba for for that so next step um he needed to have some kind of model that he could use to score different tasks and he used the rush model for that it's actually a quite popular model in um um scientists that try to understand how the brain works how to rure reading measure reading performance for instance has been used for a long time already and he

thought I can use also the same thing for the meas for measuring developer measuring how well somebody wrote a code so now now he picked some existing tasks from lature he reanalyzed those data sets based on the additional research that he did for time and quality and now he needed to create like a control group and the nice thing here is that he actually got a subsidy

of almost half a million euros and he used that to hire developers basically from different countries from different skill levels and he used that group to bootst step the model so he used the D Kruger effect to initially score the developers he asked them to do different tasks they manually reviewed that they tried to determine what the proper Solutions were and then determined how well people did

the task and that bootstrapped the model and this platform is actually uh online nowadays it's called reps measure and analyze software engineering skills and if you take a test on the platform you get an outcome like this like this uh you see how you scored compared to other people that did a test on the platform as well now the first person scored around a seven which is

quite a high score I can say and you can also see that he performed really well on the AIS of object orientation and roring you also have unit testing there coding speed implementation and regression as things that are being measured and you can see the other person have having a slightly lower uh rating but still uh scored pretty decent but in different areas this gives you some

really good feedback on how how you improve and also why you scored that way I will give you a short demo of this platform so if you go to the Cs platform you type in your credentials and it tries to figure out where you are and locates an AWS uh machine near you spin it up you go into a virtual machine that hosts the platform here you

go speed it up a little bit there we go spinning up we're going into a Linux box and the wizard will be started in a few moments so here we are we're now in the in the Wizard so it says welcome and you're going to be measured these are the areas that we're going to measure you in and just going to ask you to uh test if

the environment works as you uh expect so explains to you um what the different things are are again how many tasks you're going to do this person is asked to do five tasks and the next step is uh to check out the platform pick one of the two available idees inj or Eclipse who is using Eclipse here one person okay most most people uh are using intellig

I think nowadays there we go you check out if you can work uh with it if all goes well you run a few tests here and if this all goes well you go on and then we go to the first task and this is correcting a logical bug in a find function so you get some explanation about what the task is the idea is that there's a

graph that holds uh a value in each note and the function needs to check if every value is actually present in one of the nodes the vales one to five pretty easy stuff so you you read this you try to understand it and if you're ready for it you hit begin then you go to T and here we go now we need to find the BG so

we can run the unit test you can see that we have an error we can find all the five values and now we go through the code really really fast so let me speed up a bit and what we're going to see is that there's a function that's not returning parameter and we need to correct that and give back integer K and now we rerun the test

and then we see that the test goes well and now we solved it and now we can go on to the next test NE next next task so we do this for five different tasks and then we get a score like you just saw on the previous slide so at open value we've been using this platform for a number of years and I must honestly say that

it was quite Bucky in the beginning um and we helped uh we helped gunar a bit improve uh improve the software but we find that this software works well I would say really really well surprisingly well even so we always ask somebody to take a test and if somebody scores below a four that's for us a filter but if they score like above six they're always really

good and this really surprising because we only test like core Java skills um by the way other languages can also be used with the platform but we only use it for Java if somebody scores higher than sticks on this platform they always know everything about front end about Cloud Technologies infrastructure well maybe not everything but a lot at least enough for us and we will regularly do

those people a job offer and we did a lot of tests already here as you uh as you can see and so I'm always always a little bit surprised um that you just take this online test and by the way there's a bit of manual grading in the test task is not graded automatically but most of it is and then you already kind of know how the

technical interview will go and by the way in our case we do have like a 90 minute uh Q&A where we try to dig really deep in well where's your knowledge coming from how well do you know certain Concepts and so on but it never really changes uh well what we are guessing based on the interview um so that's uh based on the test so that's quite

amazing so um Switching gears a little bit um want to talk a bit more about the certification I mentioned this uh already and well certification is definitely not a great way to test programming skills as I already argued but wouldn't it be really nice that we can use well this platform that I just showed you in some kind of way that we can also like standardize this

for programming languages so that you only need to take a test like one time in your career or maybe a few times if you think that you progressed and then just show that result to a company that you want to work for or to to a team that you want with and then they say oh if you scored this well we're definitely going to invite you at

least for an interview and they're not going to give you like those lengthy homework assignments wouldn't that be awesome but this is something that we are trying to accomplish for Java uh with the fuj community so the fuj community is a global Java Community where a lot of people participate quite actively there's a f J.O website where there's like two articles uh around Java subjects online every

weekday there's a podcast um and you can find the fuj boot at many conferences as well it's great place also to ask any technical questions and what we did is also start like a food J certification initiative here basically um using the insights from the C platform and also the platform itself there's an article online about this and it turned out that there's a lot of enthusiasm

for this with over 80 well community members Java Champions other people who are involved in conferences people are involved in all kinds of Open Source projects we had a discussion uh around the subject would this be a really good idea to um well try and promote amongst our community as a standard way to deter skill level and people were well of course uh um not convinced straight

away but uh they were definitely interested in taking a loop and what we did uh with work group we created specific tasks for the grass platform and we for and these task uh should be the kind of task that measure things in the Java API that you want people to be proficient with at least that's what the fuj community thinks we did like a poll for this

and we're currently in the beta phase so we had a quite a few people doing a test run now with this task to see if he feel comfortable with well saying that this is a great way to determine not only the skill level but also specifically the Java skill level so if you're curious and uh want to know more about this let me know then you can

do a test drive so we're currently in the beta testing uh phase and be happy to share this this you okay this is my last slide small recap so coding is mostly about skills and research actually shows how you can measure these skills and why not also use this well for uh more honest Java certification this you can also like reuse um if you go and apply

for a job somewhere or maybe if you want to know what your skill level is and want to see if you progressed over time and so on this is it I don't know if there are any questions thank you thank you Roy um Round of Applause please you got a woo uh any questions from the audience there one here uh do you have suggestions how to measure

not only Java knowledge but also some framework like bring knowledge because I assume that's also important in a lot of roles because pure Java these days are probably less important I 100% agree uh one of the statements that I did is that the task should resemble your normal day job as much as possible so most Java developers nowadays are definitely working with with spring so um having

said that um like finding the bug example that you saw is necessarily A a whole lot difference in Spring while you can of course come up with some specific things that are inherent to Spring uh this could potentially be added there's no way it could not be added but it's just not available um so the platform is nowadays also compatible with cotlin for instance so that's also

something you can use what we found again um is that if somebody scores well quite well on on this test and they're a Java developer they normally know enough about spring as well you don't learn these coding skills by accident and you probably worked with spring then apparently already and then therefore there's a correlation probably between your Java knowledge and your spring knowledge but it would be

definitely a great add-on I think thank you any other questions from audience okay then we can go to the questions on slid uh the first one is um how do you assess if the person is a team player versus a lone wolf yeah so what I can can do here is hide a bit behind the research um it didn't say really anything about this what it did

say is that skill level is an important predictor of how well somebody will perform in a team but that doesn't necessarily mean that uh well they're a good fit for a team so there's of course other things also implementing that so uh how I would assess that personally well yeah I would just ask questions of course how how how well did you work together and yeah you

probably if you ask them about practical examples you get some kind of idea but it wasn't really something about this in the research that I looked at yeah but you can also do HR tests um during the 90 minute yeah for sure that's uh that's also a way to go yeah makes sense uh the next question is if five years of experience is not necessarily required to

be a senior how many years of experience is then a few yeah that's a good question so three or four um I would say uh I think that's also what they couldn't put an exact number of it just that the number of years of experience was not a really good predictor beyond the first few years so yeah it's somewhere between two and five uh the next question

is for the Donner Krueger effect does it also apply for women that women rate themselves to question I don't know I I I think uh there's not a did the what was being uh mentioned in the study they didn't differentiate between men and women that's that's that's all I know so I I guess uh it goes for both for men and then okay this is really interesting

I will repeat for the rest of the audience that might not have heard this so men 16% too high and you said women 60 60 16 right yeah and women how how much lower nine so but still a bit higher but it's not not as strong lower okay so that's that's interesting should definitely look into that the confidence Gap okay that's how it's called in interesting interesting

yeah uh the next question lots of studies are 20 plus years old it has changed can we still use those results to have meaningful meaningful takeaways well uh gunar H PhD um was in 2015 so that's not too old I would say that that that's one thing um and the specific work that he put into it is all from the last 10 15 years and uh yes

there are a bunch of studies uh that I made references to that are way older um but a lot of the things mentioned in those studies are also like general concepts I think the the lisp study does play a central role in his work but if you um look at this the central claim of the study that you can uh learn something about productivity of a developer

in general in a certain Language by looking how well somebody's able to solve problem a b and c and that that is going to say something about DF it also sounds kind of logical right so um I'm not definitely a science expert who can make the Judgment call here um but I do think a lot of those takeaways uh have still have some value we have um

time for just one more question even though there are a lot more so my suggestion is that every other question you have you should ask Roy at the ask me anything part and I'm going to read one more U with the growing integration of llms in development how should we adapt methods for measuring a software developers abilities um yeah so what we uh did at open value

we we used of course we used jet GPT to do the test and you know what did quite well so that was a bit disappointing so yes you can hack this test um that's that's something we learned so I guess you can hack any test if you really want to I mean you can always ask somebody else to take a test for you if you have an

online test so in the end is is really a bad thing and if you want to be sure that this is not happening well maybe you want to ask somebody to come to your office location and you put your um laptop there with the the CBS test on it so you make sure that they're not accessing jet GPT um so that would be an adaptation you can

certainly apply and I would even though I'm well really convinced about the value of this test I would never only use the test so I do recall we had a computer science teacher who scored like a nine on this test so it also comes back to the point about the spring framework question so we had high expectations of this person but uh in the technical interview it

turned out he didn't knew a lot about spring or about basically any other stuff than purely the Java language so you never use I would say any test like this as the only uh selection Criterium so that's maybe something to take into account that you use this as one of the things I would say okay thank you