How Much Platform Is Enough Platform? - John Keates, Wehkamp Retail Group

In this talk, the speaker discusses the evolution of platform engineering within the context of Veto Group, a company akin to Target in the Netherlands. The focus is on creating a platform that effectively reduces cognitive load for developers, facilitating easier software development and deployment. The speaker elaborates on the historical challenges faced with their technology stack, which previously required excessive coordination among various teams, leading to inefficiencies. An internal drive from the marketing department prompted significant changes, resulting in a more streamlined platform that allowed for frequent releases. Despite some setbacks with adopting external tools like Backstage, the speaker emphasizes the importance of adapting solutions to fit organizational needs, highlighting that simpler, tailored systems can often outperform more complex ones. Overall, the session underscores the necessity of balancing platform capabilities against actual team capabilities and supports a continuous evolution of internal tooling.

Welcome. Hello. Uh my name is John. I uh work at the VA retail group uh where we do platform engineering and I also do a little bit of consultants on the side because I find that if I work on the same thing uh a lot then I tend to get leopic zoom in on only that work and then you forget the world around you exists. Uh what

also helps is doing community stuff including standing here uh but also uh contribute to open source when possible. Um and that's enough about me. If you need to reach me, I'm usually very easy to find with the same username handle everywhere and the same 10year-old picture. So, Veto Group, if you're from the Netherlands, you already know what it is. Um, if you are not from the Netherlands,

it's like Target in the United States, but much smaller, like sized to fit in the Netherlands. Um, we do clothing and shoes and furniture, stuff like that. Now that is important for context because uh why we not well we're not essentially u um a a mega corpore but we do have our own logistics our own mobile app our own website and all of the software that sits

in between to make sure that when you do an order couple of minutes later it should go out the door. Uh lots of software means uh lots of uh systems developers uh and to make it all work you kind of want a platform not too big not too small. So in this case you also need to know what a platform is supposed to be and it used

to be that if you ask 10 people you would get 10 different answers. Uh it has gotten a lot better. There's a lot bigger mind share. Um but it tends to be that it is software that is used to make other software go both from the runtime perspective and the development perspective. So it's not software that exists for its own sake but to help and it should

help in multiple ways. One of the most important ones is cognitive load. So if you would have every developer need to know every detail about every place in your stack of software that would be a lot of uh well a lot of mental uh load to carry even if you just want to do a small and simple thing. Uh it also tends to be a product and

products have logos and branding and customers which usually are just your internal engineers. Now I didn't come up with the entire definition myself of course. Uh there were some great talks in CubeCon uh EU and NA last year. Uh the first one uh Camil and Ian they were mostly talking about creating and scaling a platform engineering team for which they of course also needed to say well

what is the context of platform in this case and Chris and Corbin had the talk about platform maturity uh models and they also have an assessment which you can also take yourself. Um and there were some others I mean Martin Fowl of course there's a uh a post on on there and we had our own research student a couple of years back who essentially did the same

thing because it is important to know your own context know your own rules essentially now how much is enough that's the second part of the deal because how much platform is enough platform you need to know what a platform is we kind of have some idea of what it's supposed to be uh but then how do you find out what is enough and you could have a

very cynical take you say well I'm going to pull pull pieces out of your stack until it falls over and then I'm going to put the last piece back and then that is enough. But that's like the bare minimum and only for the happy path. So it means that if something that is not in your happy path happens then it will still fall over. So that kind

of cynical kind of minimalistic take that doesn't really work. Uh so how then how do you measure what is enough? Is it some measurement of cognitive load and how do you measure it and is it supposed to make it low or when is low low enough? So those are all kind of vague and difficult to measure uh ways to reason about it. Uh you can also just

look into your past and see well how are we doing things now compared to how we did things back then and that usually helps put your own situation into perspective because it is kind of hard to predict the future and it's kind of hard to make those say softer and more uh vague uh questions to get them answers especially when you're not a big corporation which can

set up an entire R&D team for that because I've seen that happen that's really cool we didn't have the people for that so if you go back uh uh uh into the history of ACMP um you had a problem which well it wasn't really a problem back then but it is a problem today where if you spend all of your time on your operational work so in

this case it would be like analog work or the mainframe equivalent of your analog work uh you don't have time to do anything else you don't have time to do anything about it you start the day you have your entire list of tasks you have to get through at the end of the day you might have completed it you go home you come back and that list

is just replenished and go about your work again. So there's no breathing room, no time to do anything about it. And to be honest, back then, this might have been in the 60s, maybe it also wasn't really expected that you could say, well, I have maybe a a better way to do things. It was more about doing the work. So if we fast forward a bit, uh

let's say go to about 10 years ago, our stack was um not really a platform. It was literally a stack. like if you were to go to a shop and buy a stack of DVDs and stack it up, that kind of a stack. So, it was lots of off-the-shelf software, off-the-shelf hardware, enterprise contracts, and everything had to be done separately. So, if you were a database team,

you would have your database servers on your database hardware with your database operating system and your database management system and an entire team to manage it. It would be the same for storage, for backups, for servers, for networking, but also simple things like load balancing. Right now, you would take it for granted. If you run Kubernetes, you have a billion mini load balancers. Cube proxy mean technically

not a load balancer, but that was a dedicated job back then. So, this also had some other problems. It means that if you have many people who all are like experts on a piece of the stack, you end up having to talk to each other, which is fun. But it also means that your communication goes uh well not necessarily exponentially but it tends to go exponentially. So

every change which might be one simple change ends up talking to someone else which means that it's two simple changes and now you have 10 teams which means that the person who asked for the change might not know at the end of the chain who is also getting that question that you asked. So this causes all sorts of problems. Lots of handoffs. This is also like pre-devops

era. So it was really not ideal. Now, it might seem like, well, that's a stack. Um, it was more like multiple stacks. And it wasn't even like multiple nicely similar shaped stacks. It was stacks that were all different, all incompatible, all unique. And in their own local specialism, that works because if you say, I need a network and I need to to take packets from one end

and make it go to the other end and I need to do it well. You can optimize for that. You can do that. Um but if you then zoom out and you have like these 20 other departments and you have the business goals uh that little part that was super optimized suddenly doesn't seem that important anymore. I mean it's critical but the super local optimization doesn't necessarily

help you. Now the same goes for your cognitive load at that point. So if you're testing something you end up with maybe a spreadsheet or a wiki page and there's like this this correlation sheet. Well, well, if you're accessing this thing, then you go to that team which has this server which is managed by the other team, but they use a different name for the same server.

And this sheet would essentially require a secondary screen to expand to the side. But yeah, that would be problematic. You would have to spend a lot of time figuring out who to talk to, uh, what to talk to them about. Not ideal. Same goes for documentation about how the applications work or how you would debug them. One team might be doing it in a shape of a

how-to article or a tutorial and another team would do it more like an API reference dump. So it means that if you need to do something that integrates both of them, you now have to mentally load them into your brain and format them so they make sense together, which again that's time you would rather spend on doing something productive. So how do you make this better? Well,

you could say, well, we're just going to try harder, but that just creates local optimization. and if you try hard enough, you will work yourself into a burnout. Don't do that. Um, there are other ways to do it, but right at the moment where we recognized this is probably not the best way to do things, we had an amazing internal champion step up from a corner that

we didn't necessarily expect. So, the marketing department of all department said the change rate is too low. We want if uh if we have an idea, if you want to do a campaign, that software that the idea needs to be out the door within a week, not three times a year. No, once a week. And that was amazing as a as a concept back then. Like how

are you going to do that? That's not possible. You need to talk to so many people. You need to have a change advisory board for your change advisory board. That's never going to work. Of course, that's not what we did. uh because the marketing department they had uh uh enough willpower and enough budget and enough let's say political capital to say we're going to fix it. We're

going to change it. We're going to have our own party and we're going to have containers and clouds. And that's what they did. Got an entire new department. You would perhaps call it a shadow IT department. Uh which ended up eating the previous IT department, but that's okay. It all worked out in the end. Um, but they ended up not doing it once a week, but multiple

times a day. So, your release cadence essentially goes 3,000 times as fast as it used to. That's a pretty amazing uh achievement. So, that was started 10 years ago and we are still reaping the benefits of that today. Right now, uh, at this time we have an internal platform which has a name and a logo as you do. Um, and it is much more platform oriented than

it has ever been, which is important. Um, uh, and we'll get to why because just because it's more platform shaped doesn't mean it is the ultimate platform. Um but if you are a developer and you are writing some features, some software and you are in the happy path. So you're not doing anything too exotic, you're not doing anything too to too to too to too to too

to too to too to too to too to too to too too big. Uh your service be a micros service or a moderately sized service. Uh you can write it, test it, then deploy it without ever leaving your IDE. And that's not because we have some magic ID integration. That's because you sit in your ID, you write your code, maybe do some local tests, and then you

go comet and then everything happens for you. It's not required to not be involved. If you want to, you can just make your comments, do a manual merge, and then do a manual CI run and do a manual CD run if you want to, but you don't have to, which essentially means that you can spend all of your time on the things that you actually want to.

Bug fixes, features, improvements. So that is overall a major improvement. Now, this only works if you have some sort of ownership and some sort of um essentially a way for people to not be able to just write the stuff, but also see how it runs. See how well it runs. Does it conform to the the principles or the measurements or the ideas that you had about how

this thing was supposed to behave. Now, to do that, we have this amazing uh uh stack of CNCF projects which without them, we could not have done this. Um the main drivers of course being kubernetes but also argu because it means that if as a developer you make a service and it needs some traffic from the outside from a third party maybe inside some internal traffic uh

you don't need to know aboutto or kubernetes you might live your entire development or engineering career without ever having to touch it. Um but those things allow us to both supply the capabilities and delegate the capabilities without enforcing some sort of thou shalt be a YAML engineer rule. So this is kind of what would happen there. They use the richest concept. I think it's still relevant. The

12 factor app which says a bunch of things about the qualities of your software if you were to build well essentially it's a list of like you should have observability and it should have scalability and it shouldn't have local state that's not replicated and stuff like that. Um if you were to build the back end for that concept it would kind of look like this. Um, and

we didn't necessarily know it at the time, but you know, taking a few steps back and taking a look at your work, you're figuring out, well, we probably did the right thing. And it's also pretty cool, too. Of course, be at CubeCon and then see other companies do similar things. Uh, which uh it's interesting to see that we all essentially go on the same track, arrive at

the same destination, make something that works. Now if we look at the YAML uh uh YAML of it all u you are of course uh going to have some YAML and if if you make abstractions those abstractions are going to turn into YAML at some point and that's okay because compared to having let's say a wiki with 20 different flavors of documentations on how to do things

or having like both ARM and Intel and IBM power and also having proprietary but also open source systems that you need to weave together to make something. This isn't that much of an issue. I mean, sometimes we love to complain about it, but it's pretty amazing what you can get done with just a bunch of YAML. Now, this of course uh requires some automation and some self-service.

So, if you say, well, no more YAML. Well, how how are you going to do it? Write some custom software with emit YAML. Then you just need to learn the custom software. So, shifting the problem around doesn't necessarily help. Uh so we ended up uh at first just thinking about the the happy flow the minimum things that you need to get uh running. So you might have

a couple of of fundamental dependencies for your application. It needs to run somewhere maybe it needs a database stuff like that. So what do you put in the happy flow uh uh those things and no more because that all requires upkeep. If you template or automate something, you're going to have to maintain that, which is okay. But if you make that maintenance problem very big, then you

might not have enough people to carry that load. Uh, and of course, you need to be able to deliver that to your users in a way that they will accept, which is usually APIs, sometimes forms, stuff like that. Now, internally, uh, uh, it was started as a different project, but it ended up being a provisioner, which is laid out something like this. And it means that if

you have a resource as an engineer, you can just say, well, I don't know everything about my database, but I do know that it needs to be big enough to hold my data and it belongs to my application and the application belongs to me. Which means that this can just make sure that it's tagged correctly. So your cost allocation is automatically done right. It also means that

if you find a dangling resource somewhere, if something went wrong, you can just look at the tags or look at the metadata and find out who it belongs to and talk to them. Hey, was this on purpose? Was this an accident? Stuff like that. Now, me being uh uh coming up from software engineering, but also Linux uh administration. When I first heard about this, the first 5

seconds, I was like, that's dumb. I mean, that's like a glorified version of set and grap and you know, there's no schema. Why don't you just make a PR and do it yourself? Um but literally 5 seconds into that thought train, I was like, well, wait a second. If that is a hurdle and you can take that hurdle away and it means that everyone now doesn't have

to spend any time on this part of their job, that's much better. Of course, we weren't the only one that thought that and um right at the same time uh this became a thing and it was already announced and it became open source and later of course it got adopted by the CNCF which is great. That's a sign that you know well this is a good project.

Um and we loved it. We read all of the uh the documentation. We read all of the blog posts, the videos, the examples. We're like, "Oh, we want this. This will solve everything. We'll make everything better." And uh once we have this, then uh everything will be great. Rainbows, ponies, the works. High five around the office because it means that we could tell ourselves that we are

already on the right track, but there is this other project which already does the thing, but does it better like an entire portal and everything is great. Um so what do you do? Well, instead of doing your homework, you just get implementing. And the problem with that is that you find out later down the line that it might not be that simple, especially if you don't have

that many engineers. So, we ended up just trying it out. We borrowed some people from some front end development teams because, you know, they're familiar with TypeScript and Node.js and, you know, that's how backstage works. And to get it to do the things you want it to do, you have to recompile it or essentially repackage it. um which is not necessarily very difficult but it's something that

you have to do on an ongoing basis. And in our platform engineering team we mostly had people who are wellversed in systems and engineering and making sure the stuff runs not necessarily in creating or maintaining new software. So this kind of worked a little bit but also mostly didn't. Um, and at some point the product owners of the teams where we borrowed some people, they got a

little bit anxious like, well, we kind of want our people back because our backlog is getting longer and longer and longer. So, they got the people back. Uh, it fizzed out a little bit. Um, but it doesn't stop us from trying it again anyway. So, that's what you do. You try it again and you still run into the same issues because the underlying problem is um, it

might be too heavy of a lift for a too small of an organization to do this. Of course, we didn't learn from the first time. So, we did it another time. And then we did it yet another time for good measure. Uh, which also didn't end up working. Uh, and this is all to no fault of backstage, right? This is just um us trying to shoehorn something

into a place where it might not fit the best, at least not in this point in time. Um, but we did get very far. So, if you look at what it looks like, you know, you have your your catalog and all of your stuff is in there. That's amazing. Uh but then when you go and click on things like for example you take this surface in your

tech docs and you click on it and it just says no that doesn't exist and you spend a lot of time uh uh debugging this because you of course want okay why doesn't it work even if you're not going to solve it if you're not going to use it you have this itch you want to debug it and find out why. Of course, it turns out it's

not our fault. This was a bug in backstage, but because the people working on it weren't necessarily that much open- source uh project uh uh uh aware, they didn't check the issues first. They were just used to debugging the software first and then go and check the issues, which would have saved some time. You also had like weird stuff where your schema would say, well, you have

a text field, but it has to be an integer, but how is your text field supposed to be translated into an integer? So, that doesn't work. So at the end of the day, this all ended up being a skill issue. We tried to do something that perhaps was a little bit too big for us to handle at that point in time. And this of course has been

uh a thing with other companies. Some people which like outright complain, oh it's too big, it's too problematic. But that's kind of because the scope of backstage is big and you can't really make it smaller because then it wouldn't be an IDP anymore. You can of course disable some features if you don't need them, but we were very uh keen on trying out the scaffolder and making

the documentation all come together and having this nice entity relationship thing where if you have your application, you also know which dependencies exist and how they relate to each other and how the documentation might also link across projects. Um but unfortunately uh that didn't work out for us which is a big issue. We did have a sad moment where we had to archive our internal uh clone

because as you do you clone it and then you configure it and you have to get your upstream changes merged in. So that's a very software engineering workflow which I personally don't mind but uh such a project cannot be carried by one person. So one of the rules that we also had is like I might be very excited about it but if I'm the one working on

it then nobody else will be able to work on it because then I'm the one with all the knowledge and that's not sustainable. And I had other things to was hacking on STO which is also a lot of fun. So it might be too much platform. That's essentially the conclusion for the backstage part which is okay you know maybe you don't well you kind of need it

but you don't need all of it. So it's okay to come up with a smaller scope a smaller scale and that's what we uh ended up uh doing. Um but before uh uh we skip this is a wall of text I would say read it on your own time if you want to. the slides are uploaded on scan. Um but uh what we I think the main

problem that we have is backstage is designed for collaboration. So if you have a software engineering team that says well I need to scaffold this new thing that you don't have yet and we know that you guys don't have a lot of time to implement it so we will bring you the scaffolder and then you can just approve the PR and now we can scaffold this new

thing. Uh that works well but our teams don't really work that way. Perhaps it's because we're just not big enough. Perhaps our teams are too small to do more than just the tasks that they are uh assigned on. Um so yeah, that that's how that works. We of course uh we have learned from the past but not enough because we're probably going to try it again anyway.

Um that's also because um complexity and needs they don't disappear. So just because you have a internal tool that will preision provision stuff for us um doesn't mean that the scaffolder isn't a better solution. We also looked at uh another project it was I think deprecated. It was clutch from Lyft. I don't know if anyone has heard about that. Um but it is a much simpler thing.

It mostly focuses around uh runbooks and um also little scaffolding. Uh but it doesn't do most of the other stuff which would also be fine because that's an open source project. you can contribute. Uh but sadly, I imagine they also switched to backstage and then they don't maintain their own system anymore. So just because backstage didn't end up working out doesn't mean that we didn't get anything

out of it. So you're golden pass the concept of platform engineering. Uh backstage has that as a native component. Um but the idea of that is still super valid. So if you have a uh a couple of programming language that you use mostly in your company and uh your organization says well we might be spinning up services every now and then it would be great if we

just have a template be it a simple cookie cutter template or something slightly more advanced where you already get everything that you need uh because that that essentially saves a lot of time. It means that anyone making a service will have the same defaults will have the same valuable insights that other people already have learned before them. So you still get the ball rolling on improvements um

but perhaps not in a nice uh portal. Uh so uh it looks like we have a little bit more time. So what I am going to do is uh do a slightly more drill down into what we did internally. Um because you know I've shown this slide um the thing that it mostly does is string templating and string uh replacement. So if you are mostly a YAML

engineer then you will find that uh just replacing some strings in YAML can turn a template into a thousand useful things. Uh so if you can do that by just replacing some strings well it's not too hard to have a computer replace the strings for you. So what does it look like? Well um as a developer you just go to Slack and you go here and you

click the buttons of the stuff that you need and you just get it. it's there and you don't really have to to to wait or ask for permission or anything like that. It just works and the amount of resources is somewhat limited. But if we look at our landscape like 80 maybe 90% of our software all uses this. It doesn't use anything more than this because as

it turns out Postgress is great at almost everything. And if you need some sort of integration system where you pass messages on a queue uh well maybe Kafka solves like 90% of that as well. And as it turns out, if you have Argo, then you don't really need to reason about all the other things, all the intricacies of your YAML because you have a Helm chart. Argo

dumps in all your variables and rounds it out and you have your entire STO configuration and deployment and your KA configuration and all the other stuff that you need. So if you are on the golden path uh as a developer uh you don't need all that other stuff at least not in most cases and for us that's perhaps because we are not that big because we don't

have that many different archetypes of projects uh but it works very well. Uh of course the downside is that you have to maintain it yourself. So even though it's mostly just working on strings and bits of text, um yeah, that's something that you have to do yourself, but that that that that load is a little bit lower than backstage, which means that it's feasible for us to

do and keep doing. Of course, we have a local hero. I don't know where he's sitting, but uh he uh does a lot of the maintenance. Uh anyway, uh so what does that look like on the back end? Well, the interesting thing is we didn't want to skip our normal workflow. So when you have automation, you should pro probably also have some sort of graceful degradation. Uh

so if there's something that you do need or you need to customize something that's beyond what's in the well the realm of just text templates and string replacement, uh it would be great if you can also just do it manually using pull request. And because we don't want to have two channels where your source of events, source of changes and source of truth might stream differently, we

just have this string placement thing just do the PR for you. And because it is assigned by a service account where we know this must be string templated, you don't need to re-review it because it's already been reviewed by the provisioning system. Of course, we do still have rules in our repository. So you must have a review from someone who is not you. So you end up

with three bots talking to each other. So one does DPR the other one approves it and then there's a third one that will do for example if you use Terraform because we use that as well uh use some string replacement and now you have your database well that needs to be applied by something or someone so in this case that's uh Atlantis doing that work so all

in all uh you have your provisioning or your scaffolding uh phase with all your guardrails and your golden paths kind of already worked out at least in a minimal fashion that fits our needs. so far documentation wise or essentially where do I go to get something uh our same uh local hero at a company made this uh which is um essentially equivalent to sharing your bookmarks of

HTML but a little better. Um, so if you are a developer and you do need to debug something, you maybe you have some metrics that say that your service is having a bad time or some logs that you want to dig through because perhaps you did an oopsie and you just want to know is it my oopsy or is it the dependency that did it. Um, you

can just click on the thing and you go there and you will have your logs. It's not as fancy as having a portal where you go to your application and you get all of your observability in one place, but it does the job. And because you don't need to go there all day every day, it's not like you have to wrangle your code and look at the

metrics at the same time because it breaks all the time. Not nothing like that. So if you have a runtime platform that is stable enough so you don't need to dig around in your tools all the time, that's also kind of okay to not have a internal platform where you have a portal that does all of that for you. But we still want it. So this is

kind of still a success at the end of the day. We couldn't get at this stage at least uh backstage to do it for you. We still really badly want to. Um but it also uh told us something other uh something interesting. Uh with our history in mind uh you might solve one bottleneck. Um but there's also always a new bottleneck over the horizon. Thank you.