An Engineer and an LLM Log into a Tech Interview Are They Cheating

In this talk, Anna McDougall, the field CTO at HashiCorp and IBM, addresses the challenges in technical interviews, arguing that traditional methods are outdated and ineffective. She introduces the 'McDougall method,' an AI-friendly technical interview framework designed to create a more realistic and less stressful interview experience. The method emphasizes collaboration, requiring candidates to engage with a prepared coding environment akin to their future workplace. Anna argues that technical skills should no longer be the sole focus; instead, evaluating planning, communication, and the ability to work with AI tools is crucial. Through this approach, interviews resemble a candidate's first day on the job, allowing for a more accurate assessment of their fit within a team and their problem-solving abilities.

We've got a packed uh rest of the afternoon. I'm really excited to get started with Anna McDougall. She's the field CTO at Hashi Corp and IBM. This is a topic that I know almost every one of you probably struggling with these technical interviews. The whole industry is struggling with this. Technical interviews are broken and AI has made the old formats obsolete here. So Anna's actually rebuilt this

process from scratch. And she's going to share with you what engineering can look like on day one of the job. So it's not a, you know, not a take-home test, not a tool that they can pass in 30 seconds. So Anna's going to explain what she's doing and what's making it work. So Anna, thank you so much for joining us and let's get that energy up. No

post lunch snoozing over here. >> Thank you so much. >> Also, we we canvas. >> No. >> Uh, hi everyone. Yes, thank you for joining my after lunch session. I know energy is often a little bit low after lunch. I'm a high energy person, so I hope that I can kind of lift you up and carry you through and keep you interested in this topic because I

do think that it's kind of crucial. Um, and I'm going to dig into why what I think has been broken and what I came up with. It's called the McDougall method. um and why I think it's good and also how you can do it differently. If you don't like the McDougall method, that's totally cool, but I am highly opinionated. So, I'm going to give out a lot

of opinions. I hope you disagree with some because that makes it more fun. All right, so this time last year was about when I created the McDougall method. And at that stage, the idea of using an LLM in a technical interview was controversial. And I was often hearing people calling it cheating. Yeah. that these people are cheating if they want to use an LLM. And this is

kind of coming from the background of people not wanting candidates to use Google in technical interviews, not wanting to, you know, be able to use code completion or anything like that. It was kind of this idea of keeping technical rounds pure. And I kind of disagree very strongly with that idea. I'll explain why in a second. Um, so I created the McDougall method, which is an AI

friendly technical interview framework. So I believe technical interviews were actually already broken before AI. But I think that AI has amplified a lot of the things that made them broken. So firstly, I'm just going to put it out there. Data structures and algorithms are a terrible way to test if someone is a good engineer. There is only one good use case for DSA and that is if

you are a mega corporation who needs to cut down the number of candidates from like 10,000 to 1,000. That is the only decent reason that I will personally accept for you using DSA and any other reason to use data structures and algorithms or leap code is in my opinion not good. Why is it not good? Because it's testing memory. Back in the 90s, the early 2000s, using

data structures and algorithms was kind of an okay way to test people because usually they were seeing a problem for the first time. Now they are not. This information is everywhere. It's on YouTube. There are guides. There are books. So it's really just has this candidate happened to have come across this particular data structure algorithm problem and memorized the solution. Is that a good test of a

good engineer? No. Thank you. No, I don't think it is and I don't think you should think it is. Again, highly opinionated over here. Um, so then we get to takehomes. Takehomes are better than DSA. I will grant them that. There are there are a few problems with takehomes and it's the same problem as saying just bring us your own project and show us what you've built.

And the problem is this. They are what I call a time test. Who here has children or care responsibilities? Uhhuh. Okay. Who here had children or care responsibilities when you were fresh out of university in your early 20s? Right. So, who do you think has an advantage when it comes to a take-home test or an own project in terms of doing this stuff in your own time,

right? People who are younger, people who don't have care responsibilities. And by that, I don't just mean children. I mean maybe ailing parents, um sometimes there are sick pets or even people who are very highly um influenced or very deeply ingrained in their hobbies. So, I'm thinking people who are doing professional dance on the side, things like this where it takes a lot of their time and

effort and I, you know, I often hear this, oh, don't spend more than two hours on your take-home. Does anyone actually think that a candidate is going to go home and only spend two hours on a takehome just because you told them to? No. Of course, they're going to spend as much time as they possibly can. So, we don't necessarily just want to filter for candidates who

have the most time on their hands. I don't think that's a good indicator either, especially if they've also got a full-time job on top of it. And it's kind of the same with an own project. I know that AI makes creating your own projects a lot easier nowadays. But again, it tends to be a time test. Who has the most time to create their own projects on

the side? People who have time I I have two kids. I don't I don't have time. I'm just going to put it out there. They're three and six, by the way. It's mad. Um so so the other thing is that I believe these two things are also kind of broken by AI in terms of signaling level. So when we think about the the way we thought about

engineers and leveling engineers we have like junior mid-level senior and the gaps here were often filled by how well they could code or how well they could solve problems with code. And now that we have LLMs assisting, I believe the gaps between these in terms of coding ability are much smaller. So I don't also think anything that involves them bringing code to you is necessarily a good

indicator of what their level is. There's also a bit of a mismatch how we expect engineers to work and what we're testing in a technical interview. Right? I'm not going to go through all of this, but the basic gist is we expect engineers to have their own laptops. They're working with their own setup. They have time to think through a ticket, how to plan it, how to

solve it. They talk to their teammates or someone in product. They try to work out how to solve it. Then they go away and they get it done usually with an LLM. Right? Nowadays, we want them to use LLMs. So why would we try to test their ability to do that job by giving them this very kind of uh clinically sterile coding environment? Why do we try

to test their planning and their ability to reason through problems by surprising them with a DSA that they've never seen before? You know, it it doesn't to me it doesn't make any sense as to how we can actually find the types of engineers we want to find by creating an environment that is so antithetical to how they actually There's also this, of course, I kind of alluded

to it a little bit already, but engineering skills are shifting and changing. What we want from engineers is shifting and changing. So, we need to change, too. And this has an impact on hiring priorities. So, we're no longer wanting to test, can they code? Because that's probably one of the least important questions that we have now for our candidates. What we want to test is can they

plan, review, and reason with an agent. Can they take a business need and translate that into a plan for an LLM to mostly execute and then review that code and tell us why it's necessarily good, bad, terrible, amazing, whatever. Right? So these are different skills. These are not the same skills that we used to test for in technical interviews. So in a way we are trying to

find people who are intellectually ambitious technical generalists right we don't necessarily need as much depth in every single candidate especially not in terms of a specific programming language right I I do have a caveat there which that teams still thrive with diverse skill sets and diverse personalities I don't think it's like the era of having deep knowledge is over but I think you don't need the whole

team to have deep technical knowledge anymore. What you need, as I said, are people who can translate business and product into these plans. But the problem is you can't translate business and product requirements into a plan if you're not a good communicator who can't understand what the business and product needs are, right? There's a mismatch there. So, you need to find people who can do these things.

And we need to find people who can understand generally how to communicate especially with the written word because now pretty much every engineer is a prompt engineer in some way shape or form. So they need to be able to do that. Yeah. The other problem is this. Candidates hate technical rounds. I don't think this is like blowing your mind or anything. Um but why why do they

hate technical rounds? Yeah. Because they feel judgmental, unfamiliar. They don't have time to reflect and think. It has little relation to how they actually work. And of course, anxiety reduces performance. No one is performing their best when they're anxious. And there is zero way, I'm not here to claim that there is any way, shape, or form that you can remove stress from any interview process. It will

always be there. People will always be a little bit freaked out to some degree or another. Um, so, you know, you're not getting rid of anxiety completely. But what if we could systemically try to create a nicer interview environment without just telling our interviewers to smile more? Has anyone here actually had to have this conversation with their engineers about like how to be nicer in interviews? Yeah.

Yeah, I have too. It's it's a fun conversation, but it often involves literally training them to smile and like say hi. It's it's quite amazing. Um, but that's only one part of the equation in terms of reducing stress into or in order to create this this safe environment for candidates. Maybe there's a way that we can create a system that helps them to feel more at ease.

As I said, you won't eliminate it completely, but just kind of make it a little bit nicer for them. And what if at the same time we could find out how they work and how they think about problems in code while still being able to level them successfully? Because as said this this gap between junior mid-level senior is getting smaller. So if we're getting kind of similar

code from our candidates, how do we actually know how to level them, how to compensate them, how to place them in a team, right? That's the question. So these are I know it's a lot of like negative talk here but but I wanted to give you the background for why I think there's a problem with technical So then we get to like what I think that the

answer is or or how I got to the point of forming the McDougall method. It's based around this idea. It's this first day of work concept that the technical interview should not feel like some sort of make it or break it singular moment that is completely separate from the work they're going to do. Instead, it should feel like they're arriving on their first day of work. They're

just meeting the team and they've got their first ticket and now they need to work it out. Now they need to solve it, right? I mean, I think it's a pretty good concept because what is it? very close to the work that they're actually doing. They're there with the team. They've got a little bit more time to think about it and I'll explain how. I'll just give

you a second to read through this. This is the actual structure of the method and then I'll explain why each part solves some of these problems that I spoke about before. Okay, I'm looking for the moment when people start looking back at me and not at the screen anymore. That's how I know most people have finished reading it. Okay. Uh, so the basic structure is it's a

90-minute interview round, right? 15 minutes of introductions, not five, not 10, 15. There's a reason. Um, you want to get them comfortable. You want to talk a bit about them and you want to throw them some some nice easy balls. You want to say, "Tell me about yourself." You know, just the classic interview question. You want to say, "Tell me a bit about your current workplace or

where you worked last. What are some problems you've solved? What do you like about your job? What are you looking for next?" You know, you just all the nice kind of easy questions that people have answered by this point in the interview process several times so that they get a bit more comfortable. They get to know the interviewers a little more. They feel a little more at

ease. But you don't want to go too long on this. And the reason you don't want to go too long on this is because everyone knows there's a technical part coming and they also don't want to be sitting there like when am I going to have to do the code thing. Um so we want to get to that in a reasonable time frame. So then you get

to the pairing itself. Now the key part is this. I mentioned that one of the problems with technical interviews is that they don't have time to think or plan. So what the idea is is that you get one of your engineers who's working in the area you're interviewing for to create a miniature codebase, a miniature repository based on your real actual codebase, but maybe just pulling out

one or two components of it. Right? So not not the full deal, no API keys, nothing that requires some sort of crazy setup, just something that is easy to deploy or easy to run on their local setup. uh and inside this repository which you send to the candidate 1 hour before the interview. So they have a little bit of time to look at it. They will block

that hour out because you will tell them this is going to happen. So they will block that hour out. They will take that hour to look at it already. That's fine. That's good. That's expected, right? It's close as possible to a real codebase. And the key part is that you create failing tests. So you use whatever your actual testing framework is and you create some tests that

are failing and those are kind of graded at different levels. So for example a junior engineer you might start with like the box is the wrong color. So the test is checking is this box green? No, the test fails. They have to go find where that box is and where the styling is and then change it to the right color. Right? Something simple like that. Um and

you could work all the way up here to like wildly large functions. You could work to whole like refactors, whatever you want that kind of helps you indicate junior, mid-level, senior, principal, staff, whatever it is that you want to hire for. And again, you can work with your team to do this. You don't have to be the one to decide. You ask them, what would you expect

to be able to do here? And they can come up with it together. Um, so that's the basic idea. And then it's a pairing session. So one of the interviewers is there with them solving it together. And a candidate is using any tool they want. There's no restrictions. They've got their setup. They know how they like to work. You want them to feel comfortable with their tools

because when they're working for you, they should be comfortable with your tools. So by the time they are in this interview, you want them to be working with the tool they're comfortable with. You want to see how they use AI, right? You don't want to hire one someone and then find out one, two, three months later that they don't like AI or that they accept everything the

LLM gives them without any question. You don't want to know that after you've hired them. You want to know that beforehand so that you can filter them out. Right? So this is the idea. Let them use it. It's it's fine. And then the core part is actually not the pairing itself but the discussion that happens afterwards. Why did you make that decision? I saw that you rejected

the suggestion that the LLM made here. Why did you reject that? What were you trying to achieve when you asked it to do this? Or I liked how you broke down the problems in this way. I also encouraged my um team when they were doing this to uh purposefully disagree with something that the candidate did. And this is kind of a little bit of a test of

how they take feedback. So to kind of say, "Oh, I wasn't sure that I liked how you let it do XY Z or I didn't like the prompt you gave it here. I thought it was too much context or something." And then you kind of see how they react to it. It can be interesting to see. Um, and again, this is trying to mimic this idea that

it's their first day of work and they have these tickets that they actually have to solve. And you know, if they make it through this and everyone's liking them and they're going, "Well, you know how they use AI. You know how they work with your team. You know if they get along well, you know if they can take feedback, you know if they can plan. All of

these questions are being answered that will actually give you good indicators of how they'll be inside your organization. So training the interviewers, I mentioned earlier just um you know training them to be friendly. Unfortunately, you do still need to do that. Uh so uh they do still need to be nice. Uh that's kind of just part of helping people to feel at ease. You know, no one

wants to sit there with two stone faces kind of staring at them. Uh you want a little bit of friendliness. Um, but there's two and I think the number two is really important here. With one, it's open to too much bias, right? One person can have a strong opinion and it's just they might not have the same opinion as someone else or it might be they might

be um, you know, unfortunately biased against certain groups or certain people. Um, maybe someone went to the wrong school or something like that or they came from a rival city or you know, you never know. Like people have weird prejudices. So it helps to have two people in the room at least so you can kind of check each other. Um the other part of this is that

three is not good. So three is not good because it's too many. Then it starts becoming like a panel. Then it starts becoming a real feeling of judgment. You would very rarely have a new engineer jump in a call with three of your senior engineers to solve a problem. Right? It's a pretty again unrealistic scenario. This might sometimes happen if it's a bit more urgent, but generally

speaking, it won't. So again, you want it to feel comfortable. You want it to feel normal, right? It doesn't need to be weird. Um, and of course, the idea is that you are looking at their use of AI with curiosity, not skepticism. So this idea that using an LLM is cheating, that basically needs to go away. You wouldn't say that of an engineer nowadays who's in your

organization. I like to think that we're already past that point. Um when I submitted this talk at the end of last year, uh December hadn't happened yet and I think there was still this kind of real struggle like do we want engineers using LLM? To what extent? Uh uh and then December happened and suddenly like over the summer everyone was just like oh no we do. Um

so it's we need to know how they're using it before we hire them. And the two roles for these two interviewers, one is the welcomer, the talker, the timekeeper. They're the person who says, "Okay, we need to move on to the next part now." They're the ones who say, "I'd like to ask you to do this now." They're the ones who say, "It was so lovely having

you here today." You know, this person, the the facilitator. And then the second is the pairing partner who takes on that coaching role who they're kind of like the the person who helps them to navigate this code base that they may or may not have actually looked at yet but they should have because they should have that time. So I said this is like hang on there

we are this is my very strong opinion on how things can be done. I don't think it's perfect. I think there's still problems in general. Um but that's my personal opinion. You might be thinking ah it's a bit too much effort because yeah you have to create you know you have to create a separate repository. You have to remember to send it to them one hour before

GitHub does not have the ability to um to time when you send an invitation to a repository asking how I know. Uh so you must set a uh an alarm for yourself or one of your teammates to before. It's a little bit annoying, but I would argue the cost both in terms of time and money to actually create this is worth it in order to get a

good strong signal on who you're hiring, especially nowadays when you want to hire for these skills. Um, so what else can be done though? You might disagree with me still. Fine, totally cool. Um, so then I I want to give you some good options and I want to give you some bad options. um the good options again pair programming is the one that I've kind of outlined

of what I think um is a is a nice way to do it. But I've also I also think the strongest argument against my approach is actually doing graded PR reviews. So this is a system where you create fake pull requests for reposi for your real repositories sometimes um where you actually include purposefully bad code and this could be from an LLM or just from another engineer

and you kind of see what the candidate picks up like what are the problems that they're picking up but also how do they communicate about it right like are they because you know some some engineers are maybe not so nice in the comments you know so it's kind of also testing to see if they can be a bit diplomatic about the problems they find or how they

approach these things especially for more senior candidates. Um I think it's a good system especially nowadays um again with LLM increasingly being the ones writing and pushing the code that you are actually testing how they review code. It's a pretty good option. System design rounds are pretty good. It kind of suffering a little bit from the same problem as DSA in terms of being able to study

for certain problems in advance and memorize certain solutions in advance. Um but nevertheless if we're looking at the skills that we value in engineers nowadays yeah I mean architectural knowledge is moving up you know and we need to understand do they get what they're actually building. Do they understand the ecosystem within which they're working and how this fits into the bigger picture. Uh so system design rounds

I think there's still some place for them. And this other one which I've not seen as often but I have heard one or two places using which is that you actually send what I call starter questions in advance. So, you basically say, "Hey, we're going to have this technical interview round. It's going to be based around this concept, and we're going to ask you these kinds of

questions." And you include five or six questions that will kind of kick off a discussion. They're not all the questions you're going to ask in the interview round, but they kind of get them thinking about what what um code philosophies they need to bring, how do they think about problems, these kinds of things. You give them a chance to think in advance about it. I I can

see pros and cons here, especially in terms of how they could use AI to to prep a lot for this. Um, and not necessarily being bring any of their own opinions. Um, but I do think if they're just starter questions, you have space to then probe into more detail to see if these are actually their own opinions or not. Um, plus in general, I think we need

to wait the behavioral rounds and written communication, especially when, as said, prompt engineering is kind of becoming a core part of the the engineering role. So we need to be looking at can they do systems thinking, can they communicate in text? Can they communicate with product? Can they understand business? Will they work well in a team? These are questions that are of increasing importance. I think previously

at least is my impression that the technical round used to be the make orb breakak round. You know that it was always like that had the heaviest waiting for an engineer. And I'm not sure that needs to be the case anymore. I think that there should be a like it needs to be there. Of course, I'm actually not I don't disagree, but um I think there needs

to be a heavier waiting into these other rounds. Bad options. Um the that's an ostrich, by the way. Um that's the one who's avoiding AI completely. That's the one who's trying to maintain this sterile technical interview round environment where there's no outside influence. Um DSA and leak code bad. Yeah, cool. I explained that earlier. You got it. Mathematics tests is another one that is unfortunately still happening.

I really hope I don't have to explain to you that not every engineer is good at mathematics. The they there's a correlation but there's not a causation here. Um there are some engineers who are better at music or languages or other kind of systems thinking that comes about through different desires and likes and dislikes in their teenage years. It's not always maths. and to test an engineer

on their mathematics skills. It just to me it it doesn't make any sense. Strongly held opinion. Um and take homeown projects. I explained that before why I don't think that's a great idea either. Um there's also this thing of people and and I know deep fakes are a thing. Um and there's this thing of people bringing candidates into the office. Bringing candidates into the office is only

good for avoiding deep fakes or showing off your coffee machine. That's it. Right? Otherwise, we're getting the same problem. Yeah. That we're pulling them out of their normal environment. Eventually, if you bring them on, they will get used to the office and it will be a normal environment for them. But bringing them in there for a technical round, it is just going to freak them out. You

there's a lot of stuff going on. There's lots of people. It's, you know, where is the meeting room? Oh, I'm 10 minutes early or I'm 10 minutes late, whatever. Like, yeah. So, this is my call to action. Try it. Yeah. Take it away. Talk to your teams about it. See what they think. See if you can find a volunteer or two who's willing to create these little

um pretend code bases. It can be quite fun. Um and adapt it as you need. Yeah, try some of the other options out. Let me know how it goes. I'm on uh LinkedIn. Um so when I when we implemented this at Hello Better, which by the way, I I created this process with six engineers from Hello Better when I was an engineering manager there. Um and I've

got all of their names and stuff on the blog post that I'll show you in a second. Um but yeah, we got some really good feedback. everyone felt it was really fair. Um they one person the one who passed said that it was the best technical interview um that I've ever had. You know there people feel more relaxed with this as long as you're upfront about what's

going to happen and when and how. So that's it for me. Google McDougall method tech if you want to find uh my blog post about this in detail, how we did it, how I workshopped it, why we came to the conclusions we came to. If you don't include tech, you get a low-fat diet. So, you know, best of both worlds, I guess. And um please feel free

to add me on LinkedIn as well. I'm very friendly. >> And oh, we've got some questions. Nice. >> How can this method scale to more senior roles including management, strategy, and design? I think it's a good question. line management. I don't know if you need a technical round so much as just a really deep discussion about their philosophies. I I feel like once you I mean I

I know that there's a tendency for line management and even CTO's to be more like hands-on at the moment. I think right now we need to be a little bit more hands-on in order to understand what's actually happening with our teams. But I think in terms of like long-term, I still think that people management is the number one most important part of line management of any kind.

Um, and if they've got a good CV that shows that they've done technical roles before and you can have a conversation with them about the tech that you're building and they're asking good questions about it, they're probably fine. You know, I I don't feel like you need a um a technical interview round for it if you really want one. Um when I was um I've been in

this job for one week by the way. So uh when I was interviewing recently for for CTOVP kind of level roles um I did quite a few system design rounds. That seems to be the industry standard at the moment for for our kind of levels. Um but yeah, I'm not sure that you really need a technical interview structure like mine for that level. I think this is

more for people who are actually going to be in there working with the code talking to the LLMs every single day. I think that's Yeah, it's more for them. But for more senior roles, like I'm thinking staff engineer kind of level, u I think you could still do something like this, but again, I would weight it quite low because I think once you get to staff engineer,

it's again more about how they can bring everyone in on their technical vision than necessarily how they work with the codebase itself. And in that case, maybe I would even say don't use my framework and instead go with something more like the review, like doing PR reviews instead. That would be my my opinion. Yeah, but I'm happy to answer other questions on LinkedIn if you're if you're

interested because I don't think we have time. >> Yeah. >> Tech McDougall method tech. Yes. Otherwise, you get a diet. >> Thank you.