From Projects to Products: The Sociotechnical Journey Behind Sony... Eugenia Bergman & Hagen Tonnies

This talk covers the journey of building and scaling an internal cloud platform at Sony PlayStation. The speakers, Eugene Bergman and Hagen, discuss challenges faced by platform teams managing Kubernetes in production, including team requests for custom solutions and growing backlogs. They explain the transition from a project-oriented approach to a product-oriented mindset, emphasizing the importance of understanding user needs and experiences. The speakers highlight strategic changes in architecture and team coordination, notably mapping team interactions and refining team responsibilities. They advocate for shifting success metrics to focus on user adoption and satisfaction rather than just project completion, ultimately aiming to deliver a product that meets the needs of internal users.

Good morning everyone. I hope that you've been enjoying the conference so far. Um we are very happy and excited to have you here today and take you on our journey of building uh our sociote technical journey on building the internal cloud platform at Sony PlayStation. Before we start, uh, I would actually like you to take a journey with me and to take you to a situation that

I'm sure a lot of you are familiar with. You are part of a platform team. You've been running Kubernetes in production for years. You've built operators, pipelines, golden paths. You've standardized everything you could have standardized. So from the outside your platform looks very mature and yet somehow your backlog keeps on growing. Um, every team seems to need something slightly different and some ask for exceptions, others bypass

your platform entirely. And you start seeing things like custom deployments, direct access to clusters, uh, teams reinventing parts of your stack and not because they want to break the rules, but simply because your platform doesn't fit their reality And this is exactly where we ended ourselves in this situation. And Hagen and I are here today uh to tell you about our journey scaling both our internal cloud

platform as well as the organization operating it uh Sony Interactive Entertainment which is the company behind PlayStation. So our goal today is not to tell you what you should be doing in your own scope, but rather share some of the learnings that we acquired over time, some signals that we learned to recognize, and hopefully some takeaways um that you can take out of this talk, some practical

things. Um I would also like to share a message of hope. Moving from projects to products doesn't mean that you need an army of product managers. As a matter of fact, it's not a function that we have access to internally. And I think it's could be something that is quite advantageous because it requires you to put yourselves into the customer's shoes and really understand what their needs

are and how do they experience your platform. And so starting as well with introducing ourselves a little bit more. I'm Eugene Bergman. I work in the uh what we call the technical operations department within the uh gaming developer and future technology group at Sony PlayStation and I have been running for several years now large programs across infrastructure and platform engineering and recently I shifted a little bit

my focus uh in helping teams to move from a projectoriented delivery approach towards more of a product oriented approach and ways of working And today I'm here with Hagen. >> Hi. Yeah, I'm a platform architect. I uh been 10 years with Sony working out of the Berlin office and I have multiple like it's a typical staff career I would argue. I was data engineering, software engineering and

now platform engineering and um yeah I work with CTFs and help uh facilitate their objectives obviously but making it also within a harmonized architecture and I bring a little bit the architecture perspective into the talk because we figured that we both together have quite a somewhat similar things to be concerned with but also like the normal aspects of architecting and organizational work um needs to come closer

together and we had a good opportunity here that we want to share. >> And just a disclaimer as well, we're sitting here on the stage today, but this is the story of a large group of people of different teams that work across different domains. So, I just hear the voice of all the common effort and collective effort that we put into this. So, what is the talk

going to be exactly about? Uh, we'll take you through some of the moments of our journey. Obviously, we're not going to go through all of it. Um, but where to through the moments where things looked like they were working but didn't exactly scale as we were expecting. We'll show you how we started by trying to solve this as a technical problem. Uh, by introducing things like operators,

contracts, compositions, uh trying to make the platform more modular and easier to consume. We then uh moved on to a part where things started to break down at some point because even as the architecture itself improved the coordination between didn't quite scale as um following the and finally how we had to rethink the problem entirely uh and stop treating this as just an infrastructure problem and started

looking into it as a system of teams interactions and feedback loops. So let's dive into this for a little bit of context and to tell you exactly where we started. Uh for our a long time our platform had a relatively simple uh context. We primarily supported one tenant our game streaming service. if there are any any players with a streaming plus account. Um we are actually the

team that is behind it and we so we operated everything from global data centers to kubernetes clusters storage network the operational support that um is supporting all these workloads uh organized in long lived cross functional teams that were defined around or had a defined product scope or system scope. So our platform behaved essentially like an internal runtime environment and this worked really well for a certain time.

At some point we started moving toward a model where multiple internal teams needed to consume the platform and they came with different uh workloads patterns, different security requirements, um different levels of Kubernetes maturity and while at the same time well and so just just to give you an example in terms of where we were situated sometimes we would have ask or have questions that would be things

like can I have direct access to specific rack, can I um uh deploy directly onto a specific node or into a specific cluster? Um sometimes questions like can I bypass your platform entirely, do things my way? Um which usually resulted in just people doing it. Um so while this was happening we were actually trying to move into the opposite direction and trying to invest in higher levels

of abstraction having standardized deployment paths uh having a unified interface that people can then consume the uh the platform from. And so we had this clear tension between teams wanting to have control and direct access to our infrastructure while we were trying to actually abstract the complexity. And the problem shifted for us from how do we operate Kubernetes reliably to how do we run or how do

we turn Kubernetes into a set of consumable um and self-service platform capabilities. And that's when it became obvious to us that we needed to turn our platform into a product offering. With all of this, uh we also had a certain context in how we were working. Um obviously these requests from our uh users of our platform didn't come once a year, but yet the way we were

working assumed something entirely different. We essentially had yearly a yearly rhythm that we followed. Uh we would allocate budgets to the work that we needed to deliver on define projects up front based on what we assumed needed to be built. We planned road maps with these projects based on the defined end dates deliver on milestones uh based on these project plans. And this delivered value for many

years. But um over time we also noticed a certain pattern where every single bucket of work and any every single capability that we were trying to deliver on was turning into a project with again this long cycle of planning and trying to plan it ahead of time. Not really a lot of wiggle room to pivot uh when needed. And so even though we had long lived teams,

a project would start, we would deliver on something, we would close it and then move on to the next thing. uh while we just started to accumulate a backlog of backlog of systems that we're just adding onto uh onto our stack which essentially meant that over time the platform became something more of a like black box missing insights from uh our user experience and our user base.

And to add yet another level level of complexity uh while we had all of this um our teams our teams are globally distributed. We are operating across many different time zones. Um this is here on the map. What you see here are all the offices locations that we have in our uh department. And so obviously lot of uh time zone differences. um we cannot afford um spending

a lot of time coordinating things amongst ourselves. And while we had all these distributed teams uh we also accumulated over time a mix of modern cloud native components that were living with legacy systems and evolving uh infrastructure standards. So if you look at this, we essentially had an increasing demand uh of becoming more that was becoming more unpredictable. Our delivery model was not really designed to follow

the uh the our scaling up and the environment of it in it of itself became more and more complex. So what did we do? We started reading books and books that I'm sure many of you already read and I'm not going to go through uh all of them. Uh we actually also had the uh the author of platform engineering for architects here at the conference. Um but

really what we try to do is there are smart people out there. Those are not common problems. Um we just started to look outwards to not reinventing the wheel. One thing that I will say is that these books all excellent but they tell you the end like the end goal of your journey. They don't really tell you the steps how to get there necessarily which are not

always very easy to apply in your own context and that's what we want to cover next. >> Cool. Thank you Eugenia. Um yeah, platform strategy by Grigo Hopper and platform engineering for architects by um Max. Um that was really insightful for me again coming from data engineering, platform engineering, ML backgrounds and other situations. I think what helped me the most uh for driving conversations, workshops and also

um evangelize upstairs but also downstairs if you will or engine room and penthouse like Rio Hopper wants to put it. Sometimes this repeating pattern of when you look at your systems and you want to you want to talk with your teams all the teams have some control concerns. They they want they have a boundary of sort where you where you negotiate with. Most of the times these

negotiations are your best people like the staff engineering your good senior engineers and really capable project managers that facilitate that for you. But um it depends on where you look and in our stack where we have the super big bone backbone network uh in the data centers these ecosystems also evolve quite differently that what you are used to when you just go to the AWS console. there

are more requirements in that and slowly are moving gears. Um and that was really insightful to like say yes we are all part of this at the different abstraction level of our whole stack. Uh the second thing obviously is the power of the reconciliation loop and the controller and this continuous loop of observing something analyzing it acting and go on repeat. That is a really powerful means

but this is something where this our cross functionality teams may um may have been incentivized to do shortcuts and uh by then using tools that may not fall into this pattern really well and by that creating drift. And so we we were trying to establish with the CTFs that when we do these product boundaries um please use controller logic. Please try to go through your states of

validity, your tests, your validation to transition from one state to the other and report a controller loop, right? And uh we hoped to fight the imperative snake um because well certain automation tooling and config management systems from the past and PlayStation has a legacy 30 anniversary and we didn't start there right like it's not green field so we have an ecosystem of architectures but uh that enemy

is outward I want to call that out because it's I think it's it's super important there are many more important things but imperative calls means basically you threw it over the fence like who should now reconcile what you just created as a drift like who who's doing that it's it's not really well answered another strong uh pattern that I found fruitful is um this notion of decompacting

it's an old talk from rich hiki about datomic um but always decompacting component like make components make these boundaries you learn by reduction obviously what one component is but also what is the thing that makes the two things together work, right? There's an information gain here, which is also in the light of AI training something you want to maybe talk longer about. Um, but yeah, taking our

streaming service for example, amazing. We do the whole SRE practice, right? We're doing amazing like we're having an error rate below 2% or what it's it's it's really good. Um, and people know what they are doing. We have these objectives over time. We measure that. we are doing quite decent job here. Um but then when it comes to the next level where is about how do I

bring a slice of my value into the company? How do I model this multi-tenency such that I'm cost effective as well but also user friendly? So what is my reconciliation concern here? It's not anymore that you don't need SLI and SLOs's. You probably need them but your agency is a different one. You you want to you want to know how much do I need to pre-provision in

order to even onboard the tenant? What am I spending? What is your node class? What is your placement strategy? Where is the metric that feeds back to the learning algorithm telling you to optimize this? How does this information flow? Um and yeah, the money aspect of scaling these architectures is important and it is just one loss signal that you want to train and optimize over. And then

um this is something which is um yeah not common for all CTFs. Again depends a little bit of the abstraction level. If you're down in the data center or if you are up meeting uh the streaming session with a customer um or now newly like maybe you put out S3 buckets for your internal developer teams or you provide studios backbone infrastructure for services. Um you need to

onboard them like what does it take your tenant to be part of your story like do you have user experiences metrics for that? What is the uh time to value metric for them looking like? What is when they have problems how do you act with that? How does that reconcile against your infrastructure? So just for instance, studios move around the like people or like you may have

a new tenant but it's actually the same company and you don't want to undeploy the infrastructure just somebody bought a studio. So this is important and what does it mean when these things happen and how do you support the teams that are using these infrastructures when these things happen? Um legal compliance as well as governance and policies. We heard this in a lot of talks already, but

it obviously is super important and with the new ecosystems here. Um, this is has been also very helpful. But we you put these concerns basically at these boxes for now, which doesn't mean that they shouldn't be talking to each other. It's quite the opposite, but it's about constraining the questions we're trying to ask at each abstraction level. Um, so then there is this never written contract and

we have all about this crossplay crow. Um, but it's a good idea. But there's more to the story. I think it's this when developer teams come together, they want to know and understand what they actually need. And it has to be an active conversation. And you you come into this game of um promises and obligations. There are some obligations that makes you create that final resource that

your team really needs. It might be that environment and that environment looks different for this team than it looks for the other team. And composition is a great way obviously but nevertheless the linting the compiling the whole trust the establishment between the what I have in my team and what I hand over to another one's obligation and promises set is a is a good conversation to have

as well as this what is this final resource even and then like you could put also like instead of developer platform or cloud platform you could have a customer relationship if you will Um but then the team on the platform side would take that contract of BSV and yet apply its own policy linting tests and all of that against it and then compile it down to the

actual cloud, right? And that could be one of the many or your on-remise infrastructure when you want to be more sovereign. Um and having that gives you this happy space where both sides of the story can trust that contract. You learned your obligations. You probably had errors here and you have an error budget, but you have a framework of talking to one another about this. And once

this reconcileides to a stable equilibrium, if you will, you should be happy about this because now everybody knows where they are and you can just relax. Um, so yeah, we talked about this uh before, but I also heard it here a lot. But this is not only about having to deal with many things and being sad about the cognitive load. Um it's you can like with composition

you can do obviously much more and from functional programming we know this uh as a very powerful tool to create this like pure functions right like you have no side effects and you can totally repeat it it behaves item potently all the good stuff and I hope that where we talk more about these compositions that we now use and hear a lot about the conference will take

more place and practice um and that should make us skate again and uh move forward and uh with that >> thank you. Yes. So while all this was happening and while we were scaling the architecture um we also obviously needed to make some changes from an organizational perspective. Um we had so far a fairly simple system in place. Um essentially um we had a very simple onetoone

communications and system in place. uh if a team A uh needed something from a team B, they just went on and just talked to to that team. There were very clear responsibilities that were defined, very clear boundaries. Um coordination was fairly minimal. Um the thing is as we started to expand and adding more teams uh in the mix, it expanded as well the relationships and the dependencies

between them. Um this is a very simplified version of what we started happening uh where everyone started talking to everyone every new capability required to coordinate across multiple teams. It was not uncommon when for a single feature release we had to put five different teams into a room together. Um dependencies also became very unclear at some point. Um we started to see things like dependencies being uncovered

last minute. teams needed to have exceptions, escalations happened um relating into a lot of delays in releases, a lot of frustration and uh some simple changes essentially took sometimes weeks uh to get out. So we tried to solve this and I showed earlier uh team topologies. Um that was also something that we essentially looked into and that we are still adopting. Uh but one of the first

things that we did is that we essentially tried to map out the teams and the interaction paths. So essentially for delivering a platform capability, we just mapped it out, drew the lines of who must which team needs to talk to uh which other team with the idea being simply to help teams navigate um the the whole system, connect the right people, understand uh who is into play

to um to deliver something. But essentially as well for us to understand it to see what is actually happening in the whole flow. And so at at the beginning it helped a lot uh with also the implementation and the creation of uh an enablement team. But at some point we also noticed a certain pattern that started to emerge where people and teams would just simply start talking

about oh we need this capability let's talk to the enablement team we need that let's schedule time with the enablement team and suddenly everything started to converge uh through this team and while we also mapped out um the whole flow we noticed that all the arrows started to point towards the enablement team which not exactly what we not the outcome that we tried to achieve. Um we

essentially solved somehow the coordination problem but simply by centralizing it not really the expected outcome. So we went back to the drawing board again and we tried to essentially map things out a little bit differently. Um we tried to define clear capability boundaries between teams also leveraging things like APIs CRDs um some common standards um to try to make the interfaces more explicit and as well as

trying to navigate and guide the teams in terms of okay you actually don't need to coordinate with all of them just focus on this specific scope and so we essentially redrew the communication path with the goal to try to empower them to focus on uh or essentially to make decisions locally locally while still aligning on certain higher level goals such as for instance with something that we

are doing our OKRs uh product goals that we also started to implement uh to essentially try to kind of have them have the decision making done more locally um while still having a certain way to map this out to larger um larger goals. And finally, another thing that we try to do as well um is to kind of start thinking and introducing the concept of um producers

and consumers. Obviously, not something new, but this is essentially something that helped us or that we tried to to implement to to help us answer three questions. Who owns what? Who depends on whom? And when interaction is actually required. I mentioned earlier we're distributed across different time zones. We cannot afford time to spending hours in meetings and in conversations. We need to have things moving fast. And

essentially this kind of concept and this mindset is something that we started to introduce in in in discussions. If team A needs or is consuming a a capability from team B in this particular concept, they are the consumer and the other way around as well. essentially this also helped us to kind of really to close this part off to to use it as a good indicator. Essentially

if a team needs a standing meeting to coordinate work that means that they're probably their boundaries are not really well defined and that's not something that we want to have. And so while all of this helped us scale the coordination we started to notice another problem. There is always a problem to solve. Um, we essentially were getting better at delivering platform capabilities, but we still didn't actually

have any idea about whether what we were delivering was of any use or useful. We were just constantly busy uh being in a rabbit hole of delivering on projects, closing out tickets, hitting milestones, reducing the backlog. So, if you looked at our scoreboard, everything looked green. But despite doing that, the number of requests and workarounds, the platform just kept on growing and it made obvious to us

that we were just focusing on the wrong signals. We essentially had a very good observability of our infrastructure but we we could tell you what is the um the CPU usage of every cluster uh the health of every node, the latency of every API call but we cannot tell actually and answer the question of whether a capability that we just delivered and just spent six months on

building was actually adopted and used and whether it was even useful. So we started changing the control loop. Um we essentially turn our road map into bets and user capabilities that we wanted to unlock rather than just commitments and yearly commitments to essentially move to a different different set of success metrics. We started looking into things like how fast teams can get value out of our platform,

whether they actually adopt what we built, how easy it is to use, how efficient can we actually operate at scale. So essentially a mix of flow metrics, adoption metrics, user satisfaction, ops metrics and we shifted entirely the questions that we would ask the teams from when will this project be completed to which capability will this actually enable for our users. And finally to to close this off

uh one thing that we actually adopted that was really helpful for us as well was changing what done meant and I'm not specifically talking about done in the context of scrum but what actually done means before that done we would qualify as done as the feature is completed or project especially is done and is completed whether it was on time whether it was on scope whether it

actually committed at all. But we started shifting this towards a different meaning. Um done essentially for us now today means that it's something that someone is able to rely on a given outcome that our users are able to rely on whether that's unlocking a new capability, whether that's unlocking or something that is useful for our users. And it also therefore means that it's something that is integrated,

documented, supported and is most importantly that someone actually uses it. Because throughout this whole journey we essentially were yes focusing focusing on the wrong things and not really taking into account whether our users or customer base was happy. So essentially what we what we did is just scaling all throughout all these three layers. the architecture layer, the interaction layer, and the feedback loop layer. And this is

only then that we finally when we started to align all these three concepts that we were able to transition from shipping solely projects to actually building platform as a product. >> Thank you, Eugina. Um, yeah, trying to wrap this up a little bit. Um so the learnings from the operator pattern for us go a little bit beyond the what it's actually in the spec right it's not

just about the golden loop there but it what does observing mean like is it the metric which signals are you choosing to to defer what actually do you know that um you want to know that and you want to also know that the other people you work with know what you're asking for and what you're working towards and this is not something necessarily which is just in

the technical system itself I mean the technical system is absolutely the facilitator and has to be and with the AI context we have a lot more things to reason about and that's that's crucial stuff but it it means that the technical systems need to converge also with a social plane because this observation also goes for the managers the TPMS the people who work with other humans are

you a facilitating character are you a value stream person are you a platform person like what are the teams that work together how could they yell with one another and you how do you create that product where you have a work environment we where us like to be and and want to be very productive in so this analyzing is not only about your service being unhappy and

your SLA is is is indicating that something is obviously not quite right and that's a good thing and you want to do this retrospective uh stuff and the whole SRE practice but you also want to look in your engineering manager roles your staff engineering your principles in order to define this environment in which you work because the social sphere has so much influence of our day-to-day business

conduction. And it also right it costs some energy for some people more for some less but it is it is value to do and when you act you want to stretch yourself right be ambitious try to go over try to take a risk because without a little bit of shakiness and about like trying to fail where is your error margin right like the whole point of AI

is this powerful back propagation algorithm and the loss function and if you have a derivative you can optimize it right but you need to have the difference and the difference is most of the time an error or something went wrong and you need to be comfortable going there. Um yeah, so it takes this redefinition and constant reconciliation in the organization to define your roles, responsibilities, the promises

and obligations we do with one another in our teams. Um, we want reasonable architectures like stop doing BS and and most software engineers know that the disciplines are well advanced but we're still struggling sometimes in the industry with basics language frameworks just inherently bad ideas. Um, but there are a lot of good ideas in the last 50 years I would argue. So reasonability is about understanding what

you're doing having a transparent like ask everyone how does that work and she should go back. Yeah, it works because it's like that you have a model about it. It's not like, oh, let me go and find the commit. No, it needs it needs to be reasonable. You need to be concluding it because it's clear. Um, leadership support. This would not have happened if the leadership is

not comfortable with us making mistakes and we had that support and I would argue without that like we shouldn't even start. You want full buy in from your exact leadership team in supporting you doing the doing that thing and it's also okay to be wrong at times and readjust that has to be a comfortable situation for everyone in the room when that takes conversation takes place and

for us it's the case and I feel lucky about it so it's really a nice place to work and then obviously without you guys and the whole industry this would anyways not work so um being part and a member of these organizations is obviously super valuable to us um hopefully to everyone else as well. And um yeah, with that I guess thank you community and have a

great rest of your uh CubeCon. Thanks for listening.