Flipping the Curve: A Platform Engineer's Guide to Unlocking the Silent 80% - Michael Reichenbach

This talk focuses on platform engineering and the importance of building solutions for the average software engineer, rather than just catering to experts. The speaker shares personal experiences of developing a command-line interface that initially gained little traction despite its perceived utility. Emphasizing the need to understand user needs, they propose conducting exploratory interviews with the 'silent majority' of users, those who might not vocalize their challenges. The speaker also highlights the significance of collecting feedback through surveys and iterative testing using the build, measure, learn cycle. They advocate for creating products that evolve continuously, rather than completing projects with fixed endpoints, in order to better align with user needs and foster a more effective development culture.

Thank you everyone for coming. A full room. That's really great to see. Um today, I'm going to share a bit about my story and experience for platform engineering and how we can start building for the silent 80%. But what does that even mean? Before I start, I want to share a short story. Once upon a time, I built this awesome CLI. Could do everything. Connect to Jira,

Kubernetes, deploy our services, read from Backstage. It was so cool. This was pre-AI, so built it with sweat and tears. And the users loved it. The one user loved it. I loved it. But yeah, um that's how it goes sometimes. And who here knows this that you build something you're really are sure that it's going to work, that's people need it, but then in the end you

keep on pushing it to people and nobody really uses it or only a fraction. So, quick raise of hands, who's experienced this before? A lot. I'm not alone, that's good. >> [laughter] >> So, but why does it actually happen? Why do we keep sometimes building things nobody ends up using or only a few? For this, let's look at who are we actually building for. This is oversimplified.

So, we have platform engineers. Us, DevOps engineers. They're experts. Then on the other side of the spectrum, we have the other experts. This can be users who are proficient in Terraform, know Kubernetes, love Kubernetes, and can't imagine using anything else. Then in the middle, we have the average software engineer. And if we put them on a bell curve, the distribution in a company, you usually have the

average engineers in the middle and on the sides you have only a fraction of those stupid people. Platform engineers and the experts. And the ones we actually want to build for are the ones in the middle. Right? So, the 80% in the middle are the ones that they need our platform the most because those are the ones that are not the experts, that are not proficient in

Kubernetes or Terraform. when we look at this, why do we keep building for ourselves and experts? That's because they're the loudest. That's the issue here. The experts and we ourselves are very convinced what a problem is, what a good problem should be solved. The experts are very loud in telling us what they need, what they want. And that's where you can get biased and pulled towards the

edges and forget about the silent in the middle. This creates what I call an inverse productivity curve where you have the edges, the platform engineers, the experts are highly productive in their very specialized platform, but the average software engineers fall behind because we have optimized the platform for the experts, not for the average. So, what is the third thing here? We should start building for the many,

not for the few. how do we start, right? So, it's it's ambiguous. It's um very large goal. You know there are 80% and by the way, you can um if you after this talk, you can already start mapping out. Maybe you put some names on this curve, right? So, put some names in your company on this bell curve. Think about who fits in what segment and who

are the people I can actually start talking to. That's what I did, we did and was really helpful to find out who to talk to. we need to talk to people. Sounds hard. Uh it is. We need to bring the ring to Mordor and actually start talking to our users. The first question is who do we talk to? Well, we should talk to the silent ones, not

the ones that want to talk to us, but the ones that are actually seem happy. They everything works fine. They have the workarounds. They just use the platform day-to-day. Pull out those and really talk to them. Start talking to them. And how I'm going to get to in a minute but also take in the opinion of the loud ones and the experts, right? They have very valuable

insights, but don't get biased towards those. So, what do we actually talk about? So, I want to focus on two kind of user interviews you can actually do. Um there are many mores. I call them exploratory interviews where we go broad and wide without a specific goal. Then you have targeted interviews. The targeted one we'll look on later. And the exploratory ones are the ones to get

an overview. They're really broad. you they're great if you're new to a team and you come to a company to actually get to know what are they using, what are the CI flows, how does Terraform look alike, how is the Kubernetes set up, and so on. So, really get an idea of how people are using the existing platform. And they're really, really great at uncovering blind spots,

the unknown's unknowns. These are the interviews you do to get to know the problems you don't know about. And they're good at observing day-to-day work. So, you want to throw the people in there, get them to know how they work daily on a daily basis. And I would recommend, when we do this ourselves, do them every 6 to 12 month so you stay up to date with

the unknown unknowns and actually keep continuously discovering the blind spots. So, but what exactly is an exploratory interview? Sounds fancy, but how do you do one? Well, it's um Yeah. You can do them um across multiple teams, so I would pick out three to five people across teams, have one expert, maybe a platform engineer. You bring the the points yourself, right? So, you know the problems your

platform has probably. Go pick one to two experts and then three three normal silent users and start talking to them. And really um have them walk you have them walk you through their typical Um just share have them share their screen and to say, "Okay, how do you work? What do you do? Um just share your screen, show me what you do every day. How do you

develop a feature?" And think out loud while you do. And then watch them how they use your platform. Watch them how they deploy a new release, how they push code, how they review code, how they update the infrastructure. And try to cover the full software delivery cycle from end from beginning to the end. And maybe even start at the spec phase already. How do they spec out

features? How does product give it to them? These are great signals you can also pass on to other parts, engineering manager in the company, when you discover bottlenecks, for example, in one of the earlier stages of the workflow. And now, the most important part. This is really, really crucial for these Understand the why and do not correct Don't try to teach them your platform. Don't try to

make them use it the correct way. Just understand why they're using your platform as they are. It's 95% listening and 5% talking. And your role is not to explain it to them. It will hinder you from actually understanding why they're using the platform as they do because they start to feel, "Oh yeah, I'm doing this the wrong way." Then they stop telling you why they're doing it

or even showing you. And if you just listen there and understand the why, you will uncover many, many things you might have known before, not known before. Okay, good. So, we talked to a handful of users. Say we have 100 software engineers or something or more. And we have blind spots. Let's go build No, wait. Before we start building, we actually want to know if there is

more than five users that have these problems. So, let's do surveys. Who here likes to ask surveys? Who sees value in Okay. Who likes to fill them out? >> Oh, there. There. Three people. Four. Yeah, that's the crucial part. How do you actually get people to fill out surveys? I mean, they provide value, but how do you get people to fill them out? Here are some quick

tricks that work for me and us really well. So, we have an average accept fill out rate of 70 to 80% for every survey we sent out. >> [snorts] >> And let's try to not make them anonymous so you know who filled them out. And then use cursor or Claude or some MCP in Slack or whatever and send out targeted direct messages to them, personalize and say,

"Hey, why didn't you fill out the survey? Here's the link. Just takes 5 minutes." And people will start to actually fill out the surveys much more reliable than if you just posted to a void channel somewhere at devnull in your company's teams or Slack. And then, a method yeah, that works, but should be used sparingly. You can lock them in a meeting and don't let them go

until they filled out the survey. >> It works, but I wouldn't use it so often. We used this twice where we really had time pressure on getting the data. And usually get a 90 to 100% fill out rate. It works great in global meetings, but please warn the manager who your meeting your busting first. It helps. how do we actually get good quality and data out of

service? Um that's where our user interviews come in. When you're done these three to five user interviews, you should have a good understanding or list of hypotheses of problems in your platform, things you want to validate. And these are the basis for the questions in your survey. You want to build a survey around a list of hypotheses you want to validate or invalidate. Don't just ask questions

blindly. Try to make up a list of things you want to validate or invalidate with the survey. Here's an example from um case study of ours about 1 year ago. Um we've refactored our ISC deployment pipeline with Terraform. We've had a bunch of micro Terraform stacks where each resource or each few resources were in their own unit of deployment and stack were scattered around the repository. And

we then pulled them together in a more coherent unit where each stack was related to a backstage component. And before we did the survey, we actually listed did some user interviews and then listed out these hypotheses we want to validate. This was after we did the component stack. We wanted to know do we need to do more or are users happy now? So for example here, infrastructure

is not the biggest pain point right now. Want to know that. Um modifying existing stacks is painful and complex or time consuming. So contradicting these are hypotheses we want to And surprisingly our users actually said that Terraform is easy now and they modify resources once a month or less. And it was really crucial signal for us because we could move on and build something else. We didn't

need to further improve what we've already built. And if you want to know more about the component stack and what we did there and how we are keeping resource labels in sync with backstage and Terraform, check out my talk from backstage con last year. It's in the QR code. >> There I go into detail about this and you will also find a list of resources at the

end or have a QR code with the notion page where you find everything. now we can start building, measuring, learning. So classical build, measure, learn cycle. We start building, then we want to measure if we're successful, then we learn. >> my colleague Klaus, our product manager, had this idea to flip this on its head and we were quite successful with this and I want to give a

quick breakdown and have a link to his talk from fall last year as well. Actually flipping it around. Before we we could measure when we're successful. So quick question here. Who establishes success metrics and actually data they measure when they before they start building? A few. That's good. establishing clear success metrics before you start building a feature or product or part of your platform really helps you

in driving as for giving it momentum and checking if you're actually building the right thing. And this doesn't need to be fancy Dora metrics. You don't need to measure the full Dora pipeline and change lead time, change failure rate, which are lagging anyway. Make them highly specific on what you're building. For example, here this from a case study beginning this year where we wanted to refactor our

scaffolding for Terraform resources. We select really simple metrics where we just said, "Okay, there should be zero manual steps." We just did a run through once, looked, counted the manual steps. There were nine. We said they should be zero. Should be completely automatic. There should be zero failure rates. It was like seven failures or something. Not good. And we wanted to have it up and running in

production under 15 minutes. These are really easy metrics. just think about, "Okay, how does success look like?" And then start tracking those metrics and they don't they don't need to be automatic. We just had an um notion page where we track them and I will show how that looks like in a second. Um And one crucial part I will go into detail in a few minutes is

the users rate experience nine on 10 on average. So here we're pulling in the user again because we don't only want objective metrics, but we also want these subjective metrics of the user. Are we actually building something with a great developer experience? Then learn. And this is about the user. We have the metrics and now we have a list of solutions, ideas that we think are good

and we want to build now. And here comes the second interview I teased before, targeted interviews. This the part where you go talk to your user about the very specific feature solution you have in mind and start showing it to And we usually do two to three interviews per week. So we pick out users who are willing to actually listen to us and talk to us. And

then challenge our idea or solutions with them and actually find out before we start building if this is the correct path. If we're actually building the right thing. And this saves so much time. If you just show them a mock-up, show them a Miro scribble, show them a flow chart or just some mock folders you put in there, how will the solution look >> And this is

for a specific feature. So you want to really be specific. Give them a scenario they want to should accomplish. For example, in the case study I just mentioned with the scaffolding, we said them, "Okay, imagine you want to scaffold a new service. Um you need one for production. Where do you start?" So we kept it really open to find out can they actually discover the entry point?

And then they started to use the scaffolding, which is in backstage for our case. And in the background, this is really different from the other part. From the other interview, in the background we fixed the problems ad hoc. So if an error occurred, we tried to get the scenario to the end. So we were two user two engineers in the interview and one was constantly fixing stuff

in the background. Remember the seven errors? We had to fix them to actually get them And then we had them rate their experience on multiple dimensions. And this is um really crucial here to actually ask them what they think. So we had the dimension clarity. We asked them, "How clear was it what you needed to do at each stage from a one to 10?" And then explain

a bit why. And how well did you understand what was being created for you from a one to 10? And then how easy would it be for you to do this again one to 10 again? And one key question here you should ask them after each step, regardless of what the questions are, "What would make this a 10 out of 10 for you?" And you will be

surprised what answers you get there because this is the indicator that gives you what pushes your product, your platform forward. What features should you build next? What's the thing that's actually missing for the user? And [gasps] now we can start building. on it. And this is how it looked like for the scaffolding um test. So we had a big notion table and started at what was it?

1,248 Our goal was 15 minutes. And nine manual steps, five failures. Then we iterated and we put ourselves in the shoes there as well and really went through the cycle multiple times a day, multiple times a week and then at crucial points pulled in users, got their feedback and then quickly iterated. And this way in two to three weeks, we actually brought it down to our goals.

Zero manual steps, zero failures, under 15 minutes. If you want to know more about why flipping build, measure, learn on its head, this the talk from Klaus from last year's Cloud Native Summit in Munich. Really great and he goes into much more detail there about platform engineering and product management. Highly recommended. Now before I um come to the end, I want to spend a few minutes to

talk about projects versus products because I think there's a key difference Projects have an end date. You deliver them, then you forgot forget about them. Products don't. But why is this important? To understand this, I want to look at startups and what makes them move fast. Startups try to find product market fit. For us as platform engineers, if we look at this, what does it mean? It

means finding the right problems to solve. Looking back at the product discovery interviews, the surveys, this is all there to actually find the problems the user need to be solved. So, instead of pushing ideas to them, we want them to come to a state where they pull our solutions because we're solving the right problem for them, just like a startup. Or any company, so to say. If

you If a company doesn't solve the actual problems of the users, the company doesn't survive. And I at least think of platform teams the same way. If you don't solve the right problems, no one will use your solutions. And listen to customers. Startups listen to customers, and we as platform teams can also listen to customers doing user interviews, targeted feature interviews, surveys. And they iterate incrementally and

quickly using the measure learn build cycle. We can do quick iterations, adjust course midway, and build towards a product. this is also a key differentiator. Projects end, products don't. They continuously evolve. So, for example, in our case for the scaffolding, right now, it's usable, it's okay, but it doesn't have the full feature set, and that's okay. But nobody asks about it right now. They might in a

few months. Somebody comes along, "Hey, I also want to scaffold this and that." Okay, then let's plan it in and add to our product, the scaffolding. But we don't need to build everything in the beginning right from the start. And this is how I see platform teams relate to this. We build platforms, we listen to our users, we find the problems to solve, we continuously evolve our

platform. We don't build it 100% from the beginning and then it's done. And we iterate incrementally and quickly. So, ship products, not projects, and evolve your platform. doing all of this, for us, it caused a real change in the dynamics of people actually pulling our solutions and flipping the productivity curve. Cuz we started to build for the middle, for the 80%, instead of just the Their knowledge

and ideas are still valuable and they help develop that middle pillar, but the key inside here is really to start listening to everyone and flip the curve. So, start building before we go into Q&A, here's uh some There are slides, contact, resources, also to the other talks, some blog posts, and yeah. That's it from me. Make sure to join my keynote tomorrow where I tell you how

1.5 is decentralizing the power grid. If you haven't seen that, um it's in tomorrow at the end of the keynote. It's going to be a really nice interactive live demo, so make sure to join. It's going to be fun. Yeah, that's Thank you. >> [applause] >> Any questions? So, thank you for the insights. Very interesting. So, you said like build for the many, not for the few,

but we still have to keep those few happy to use the platform because they are the loudest. So, what is your experience in in this regard because it's always about platform adoption? And if the loudest are not happy? Yeah, that's a really tough choice to make. And here we try to not block them, but allow them to do their customization, their things that keeps them happy, but

still not build it for them, but rather enable them to build the parts they want to build themselves, right? For Terraform, for example, we could build a YAML abstraction interface for our normal builders, but the experts would want to use Terraform. So, we allow them to use Terraform. So, might cause some snowflakes and some disruption there, but for us, this keeps the middle happy path and still

enables them to enable the others. So, that's how we do it at least. Thanks a lot. And are you using personas basically from the exploitation exploitation interviews and then create personas so then use them as a reference? Um not yet, no, but it's a good idea. Thank you. Yeah, thanks. Um yeah, thanks for the talk. Um quick question about value assessment. So, all of us sitting here

today, we go back to our engineering teams, whether you're a product manager or an engineer, how do you convince the platform engineers that this is a worthwhile investment of time and resources? Mhm. Yeah, great question. Um I would start small. Try to get them to one interview and then start seeing them the Let them discover that there are things they don't know, that there are things the

user uncovers for them where they had blind spots. And I think this is already a very eye-opening, right? So, do it, slice it up, do it incrementally, don't push it all to the onto them, and then actually start to experience it for themselves and then do a small iteration on a small feature doing it this way. Oh, there's one more. Thanks for the session. So, on the

point about product versus project, so a lot of the time we are forced to use projects because leadership and management wants something measurable. So, how do you communicate this to leadership to like protect your team from doing this product iteration and not being like reorg, etc.? Yeah. Yeah, I think the the key here would be to make products measurable as well. So, establishing clear success metrics for

your products, and what does success for the product look like? And then I think leadership in the end cares for results, right? So, delivering value for users, unblocking, making the engineering organization faster. And if you can measure that and prove that, and then I think they would be fine with it and you can do use even surveys to measure that. For example, we did a huge survey

about introducing AI to the whole organization. And we not just measured objective metrics, but also the subjective ones. Does it answer the question? >> Yeah, thanks. Okay, behind you there's one more. Hi. How do you deal with low adoption? I mean, in general, if you have some libraries, some of the of the things that people stopped using or you see that they are used by 5% of

the population, how do you deal with, I don't know, end of life or how do you still make them happy or you don't and you say we won't support those libraries because of the low adoption and we want to basically focus on the high adoption basically. Yeah. Um So, first I would want to understand why they're still using it, what keeps them from using the newer one,

the newer library. I would just just stop supporting it. Really uh radical there to say, "Okay, sorry, uh out of support, uh move on." Yeah. Sorry, but it's a critical service, we need it. Then you need to migrate. I'm Yeah, I know it's probably not the right answer and not the I think it comes down to understanding why, right? figuring it out together with the user. I

mean, it's the would be very as see for me to just say, "Hey, shut down >> understanding why and then I think you can come to a conclusion and actually start. Any practical tips? I don't know, like uh how do you communicate? Are you give like a half of a year to migrate? Like what migration plans? What's your approach for such situations? Or you say, "Sorry, next

month." How many how many users are we talking about? I mean, regarding developers, I would say like 50, let's say. 50? Okay. Yeah, in this case, I would try to figure out, so first understand why, what keeps them, and then can we build something else that pulls them towards it? And if not help with the migration really, then maybe pull in some other teams, enable them to

do the migration, automate it. I mean, we have AI now. We'll probably try to automate the most of it. Yeah, but that's the case. Help them migrate. But I don't care, it's 10% of people. I don't know. There's one in the back. Hi. Thank you. speaking about tracking volume and feedback loops, all all that stuff, when you put priorities into your backlog items, do you connect it

some in some way to four key metrics? If if your platform is about accelerating software delivery and uh operations pipelines, all that stuff. Um do do you do you track it? Um what metrics key >> Four four key Dora. Dora. Yeah, we track we actually have change lead time as our north star metric, we call it, where we keep a like broader overview and track the other

metrics as well to see how our platform develops over time. Does it always align? I mean, your decision to add like new features, fix things. No. Do you always see the results? Because I find myself um sometimes like different things, interviews, what I collect from users, and what I see as the result of four um Dora metrics, sometimes like I don't see results or the things are

competing with each other. Yeah. Yeah, totally. So, Dora metrics, I personally, they're okay to get a broad overview, but I don't like them so much because they're so lagging, they're so broad, and so many factors can have an impact on them. Even if a product team a product team is under pressure to deliver, suddenly your change lead time goes down, right? So, yeah. Yeah. It might be

because like as I said, like when I speak to my manager, to stakeholders, sometimes I don't have uh like the right metrics to show that the things are improved. And the only thing I can say like, "Hey, but but the my users are happy now, you know?" Yeah. So, yeah. Yeah, so find for us finding the right metrics is always hard. Um I mean, in the scaffolding

part was relatively easy because it's concrete to measure, but right now, for example, we are actually building a dashboard for users to measure these dev metrics, and there it's hard to measure metrics of the metrics, right? Um so, it's not always easy, and there I usually default to surveys really, so asking them and then to say, "Okay, here." Um user happiness is also very important, and trying

to figure out the closest approximate metric that could show us, but there isn't always one. Yeah. >> Unfortunately. Hi, yeah. Thank you for the uh for the Um I have a question regards to the uh the interviewing process of users. Uh do you adapt it as well based on the maturity of the platform? Um can you elaborate what you Um in the case of for instance, you

gave the example of uh how startups uh build their platforms, uh how about like an enterprise mature platform that might have already kind of the level of standardization that is expected, um and is trying to kind of evolve, and for instance, might have as well reach to point where some capabilities needs to be sunsetted and be able to identify which ones to sunset, which ones to expand,

which one which ones to build Yeah. Um yeah, it's probably a I would say it's something in the middle, right? You have product discovery interviews where you get a overview, then you have the narrow targeted interviews. What see this probably as a mix in the middle somewhere. So, those two interviews aren't the only kind you can do. You can specifically for something you already have in place,

you can start interviewing people and then give them a of something you have in your mature platform, and then do a mix observe without telling them what to do, and then you can start to see patterns how they use that specific feature in the platform. So, that's how I would do it. One more over there. Okay. Do you struggle to find the voice of the quiet 80%?

Like are there reluctance to get into a room to do the interview, or do you have tricks for that? Some, yes. >> Um there I'd say 80% of the 80% don't want to talk, uh but there are a few that if you start to talk to them, it can be over coffee or chat in the beginning, and then you start to pull them in. Um it usually

works, but not all are are willing to really do the interview and usually uh for us, it's our product manager who does these things and finds the people and that come into the room, but uh doing some getting to know them and then usually works, yeah. And then I guess between projects, it could be quite common once you have then they're more >> Yeah, exactly, yeah. Yeah,

sometimes we also do like a teaser where we say, "Hey, this is we know the problem and this is the problem we want to do. Who wants to be alpha tester?" And we get a Slack channel just of the alpha users, and those are very willing then to give feedback and constantly do interviews, yeah. Thank you all for staying. Have a great rest of you. Yeah, well,

I have also a just last question, but we can do this uh here. Yeah. Um do you have you thought about uh portfolio management for your products you're creating inside of uh the platform team? Uh yeah, we are actually keeping track of the products we have and then writing down like the features and managing them. We're not so good at it yet because we're still very um

let's say early in the platform. I mean, it's a startup 3 years old now, and we're still building some foundational points, but we're slowly sharpening the products. Um Okay, so so you think it's worth it like you do on the on the whole company level. So, in the end, it's the same what you're doing to your internal customers what you're doing to the external customers. Exactly, yeah.

Yeah. And just having the platform team know the products helps already a lot. If you can talk about the same thing with the same names, already helps a lot. Okay. Thank you.