The Accidental Platform Team: Kubernetes Operators at Swisscom - Fabian Schulz & Jelena Malic

This talk discusses the transformation of a team at Swisscom from building Kubernetes operators to establishing a platform team. The speaker outlines their mission to modernize automation within Telco by leveraging Kubernetes for the 5G mobile data core. They explain the complexities faced by network engineers, who previously managed configurations manually with Excel and various tools, and describe how their team implemented high-level network intents to automate these processes using Kubernetes operators. This shift towards automation has significantly reduced setup times from days to hours, and the talk emphasizes the importance of user engagement, gradual adoption of technology, and building trust with stakeholders. The session wraps up with reflections on the ongoing journey of integrating AI and further automation into the platform.

Hello, hello. Hey, so welcome to this last afternoon of KubeCon to our talk about how we went from building Kubernetes operators to becoming a platform team. So, in this team we have a Yelena here with me who is the product owners of the team and I'm a DevOps engineer writing the Kubernetes operators. So, we work for Swisscom. Swisscom is the leading Telco in Switzerland and a large

IT solutions provider. So, our team got created 2 years ago and the goal was basically bring cloud native to Telco and modernize a bit the automation of the generation of our networks. So, we had to generate networks for the mobile data core. Let's have a look where the mobile data core is. So, on the left hand side you can see the phone, the user equipment. It connects

to the closest antenna. From there the data then gets routed to the mobile data core and then most probably out to the internet. So, let's have a look at this mobile data core. So, this is now the 5G mobile data core and just as a side note, just because your phone has the 5G icon doesn't mean it's using a 5G mobile data core. That just means it

connects to the antenna using 5G. So, lots of Telcos still have the 4G core and now we have the 5G core and this one is fully running on Kubernetes. So, you can see here in blue the pods and the mobile data core it's structured in those cloud native functions. You have different functions, some are there for the session management, some for the antenna to connect to it.

Manage a subscriber data, data analytics, there's lots of things. So, in one dev environment we have around 2,000 pods. Of course, for this we use around 32 big servers for one dev environment. And since we are Telco, we want redundancy. So, we have three to four data centers, our own data centers in Switzerland where we operate it. Of course, Switzerland we have 8 million inhabitants. If you

think about the bigger country like Germany or the Netherlands, it scales. there's quite some needs for the network. So, who operates, who creates these networks? It's basically our 5G network engineers. I will call them wizard from here on and not this type of wizard, maybe you know this network wizard, but I'm talking about this type of wizard because to us it sometimes feels like magic that the

phone just can connect to the internet over these Kubernetes things here. So, let's have a look what he needs to do to deploy a new 5G network function. We'll select UPF, that's the user plane function, that's basically the router. So, all your data from your phone goes through these pods before it goes to the internet. So, what do we need? We need the server hardware, we need

operating system on top, the Kubernetes distribution, and then we can have our 5G core deployment. Now, we want connectivity between the pods. I mean, they can be in one data center or multiple, and we need connectivity from the phone to the 5G core and out to So, now let's go a bit through the steps of what this 5G network wizard has to do to create these networks,

how it's currently done. So, first, oh, I forgot the routers. Of course, we have the backbone, the Swisscom internet backbone, so we need to route the data. Now, to create these networks, he starts by creating the IP assignments. So, basically, the 5G network engineer needs to know which prefixes are needed, IP addresses, static routes, prefix filters, VLAN IDs, everything he needs write it down in this Excel

sheet and do the generate what is needed for this new network. That can take him up to 2 days. Once he has done this, he will have to collect the IPs in the IPAM. So, we use NetBox for the 5G core, so he has to go to Netbox and reserve the IPs there and then he has to populate the inventory. So, basically he writes those what the

the the IP addresses and everything in variable files, which are then used, for example, in Ansible playbooks to create the CRs, which we um for example, MetalLB CRs, since we use MetalLB for load balancing. Then, we also need to configure the routers and the switches. Luckily, he doesn't have to go on each router and switch and configure it himself, but we have this network as a service

in Swisscom, where you can config configure it using a UI. Once this is done, we need some integration testing and observe how it goes. We use Landslide to create simulated traffic and of course also want to test the handover. So, you can see that it doesn't look so nice all that time is used and also, while he is doing these designs, he needs to talk to multiple

people. I mean, he needs to talk to the operating system people, to the people operating the Kubernetes distribution and to all the different other network engineers that manage our backbone. basically, that's him trying to do this network. It's quite complex. So, let's summarize it. He needs cross-domain knowledge, needs to understand how the 5G core works, how the routing works, also the network hardware. Maybe some of you

went to talk this morning about SR-IOV. So, these are also things he needs to know about and also understand the Kubernetes networking. It's a manual process and in the end we have a static config. So, it doesn't catch any drift. So, now of course, we're in we use Kubernetes already, so why not leverage this? So, our team's idea is our 5G network wizard comes up with this

high-level network intent. This is then read by our operators. We have network we have Kubernetes operators, which read this first intent, then generate all the other intents, which are then read by our other operators. IP addresses, secrets, router config deployments, and certificates. So, where do we put these operators? Um we have the stack from before. We had the routers at the bottom. We have our network as

a service, which configures And we have our IPAM inventory. So, just quickly to recap how it's now. He has to use Excel files, manual user interfaces, and even something called Netconf. So, this is a protocol that is used in Telco to configure the 5G core. So, he has to use different of these interfaces. Now, let's just put our Kubernetes operators here in the middle. They will talk

to the IPAM. They will talk to the network as a service, and they will configure our 5G core And now, of course, you'll know the Kubernetes operators interface is the custom resources. So, he just has to talk to the Kubernetes API. And now, the only thing he needs to know is YAML and create this high-level intent. His life seems much simpler. So, to summarize, cross-domain knowledge is

now in the operators. So, we have some high-level operators with which have these knowledge for what is needed for these network functions. And then, of course, there's the lower-level operators, which then do the IP management and those things. We have now automation, so it's not manual anymore. And thanks to the constant reconciliation, um we detect drift and can easily adapt. Now, this being a beginner session, we'll

not go more into details of this architecture. If you're interested, a week ago we published this cloud-native Telco reference architecture. So, it's uh under the CNCF architecture, where you can see all the different tools, also open-source tools we used, and also which parts we open-sourced ourselves. Now, this might seem a bit unrelated, but if during KubeCon now a wizard approaches you and gives you this potion and

says if you drink this potion you can go to all the sessions at the same time, would you just take it? Yes. You're brave. I think not everyone would take it. Um so just we see our automation like automation potions. So it can look scary to them. They see this new black box. We say, "Hey, we do everything now that you did before. We do it." could

be a bit uh difficult to sell it that way. So from the beginning we had this gradual approach basically where we say we select the first easy use case. We ship the first prototype early and then we iterate on the feedback. Now, what does this look like concretely? So in our architecture here, our first use case was just the routers. So we decided, "Let's just automate the

routers and Okay, let's also take the IP management. So we have the IPs for the routers." Basically we shrink a bit our potions. We still let the 5G network engineers use NetConf and the things they know and they can start to get used to YAMLs. So let's go. It's easy, right? YAMLs, run the automation and you have the healthy network. So to write the YAMLs, he has

to go look It's quite easy, right? He has to go look at our custom resource definition. I mean, you write it in Go. You can see here it's our secondary network. You need the NetBox config, the router config, and then you have some parameters for BFDs and network parameters. If you look at the NetBox config, you have a tenant, IPV6, IPV4. And then also you have So

the here is an example of the IPV6 config. You can enable disable it, parent prefix, prefix length. I think our wizard should be able to do that. What do you think, Yelena? Well, from our perspective and from product perspective, we agree. There are some complexities related to our solution, but we have done what we promised. The software works. It does what it's supposed to, so it must

mean we're done, right? Right? Well, our users did not quite feel that way. So, they ended up bringing quite a few questions and concerns to our table. And when we face these, of course, it was first scary, but when we thought about it a little bit further, we figured out that the reality is that we accidentally built a platform. And that simply what we have built cannot

be just software that you shoot and you forget, but we will need to do some further steps to make sure users can actually use it. So, in order to do that, we decided, "Okay, there are many questions and concerns. Let's go through them step by step." So, we first focused on the fact that really most of our users did not necessarily have time to play around with

any software that is given to them. They're operating this very complex system that is 5G network core, and they cannot just check any single thing that pops on their table. So, we followed the approach that Fabian already mentioned, which is to go gradually, to start with a couple of brave ones that could really dedicate the time to to playing around, and then build from there. So, in

the architecture view that you have seen before, yes, we focused on the early adopters. We worked with them. We got them to test our solution to actually make sure it works. And then we could gradually adopt more use cases from there, and we already had these early adopters to act as our ambassadors and convince other users that what we built actually works. Then, we focused on the

next couple of concerns. And the reality is, even if some of the things were done in Excel. Our network engineers were used to having a UI to do their standard operations. And the terminal was having quite a bit of a steep learning curve for them. It wasn't easy for them to adjust. But then we looked into it and we decided if UI is what they need, well,

we can give them the same. worked in several ways to provide different UIs to them. First one is the CRD wizards, which was really cool because it comes out of the box as long as you put really good comments nice and descriptive in your code. The users can now immediately see how the schemas are and better understand how different fields correspond to the terms that they already

know. On the other hand, we also use the open source open source tool headlamp to have the UI for the Kubernetes cluster where things are running. And there are users could more easily see how things are currently working, whether something is in error or a warning and be able to address it quicker, of course. And on the other hand, we also contributed to this open source tool

and created this view because we have a lot of custom resources that are included in our platform. And so we wanted to make sure that our users can see them in this map view and better understand how things are connected and what will happen if they touch one and how it reflects on another, right? Then the next group were quite hesitant users. The ones that were still

really not certain if our solution really works, if our platform platform is making them quicker or maybe the old way of doing things that they're familiar with was easier for them, right? And if a change happens, how will they be able to detect where it came from, how will they be able be able to fix if something goes wrong? And we looked into this as well and

realized that the place to invest would be into observability and traceability because of course if we have information of how the system is behaving we have different metrics we can more easily detect issues and if we have traceability we will also be able to realize why this happened and revert it if necessary. Luckily in our architecture this kind of came out of the box as well because

all of these high level intent YAMLs that Fabian was mentioning are already stored in Git. So we have the commit history that tells us who did things why they did it when they did it and that makes it immediately like we have this recipe book of saying what happened when and then we something goes wrong we can more easily trace it back and revert it if necessary

and it also comes with audit and compliance information that we need for our organization. We also invested in more traditional observability methods so you here see one of our Grafana boards where you can see all of these different metrics that we created and additionally we specifically focused on the ones that correspond to speed so that we can prove that what we are claiming is not only living

in the PowerPoint slides but actually we have some data backing out backing our claims and ensuring that our users also might start believing in what we're giving them. In the end we had the last group which was simply lost in this sea of information in all of these different talks different videos different tutorials that might they might be asked to follow and they couldn't find the right

way to start using our platform. They sometimes didn't even really understand how to start with Kubernetes. And there we decided to not leave these ones behind either but try to bring them along and invest into the relationship with them to explain to them how our solution works but also give some knowledge tips and tricks that they can start with to make their start with our platform a

little bit easier, right? So, we started with several initiatives. We tried to really establish the trust with our users from the start and involve them already in the design phase to understand what problems we are really trying to solve and include their feedback if they notice that we are somehow getting a little bit off track comparing to what they actually need. Additionally, we created a focus communication

channels which made things much more easy for them because they could directly talk to our engineers and get the answers very quickly and correctly instead of having to navigate the complex system of different ticketing tools that exist around the company. And additionally, what the added benefit of this thing is is that even the other users could answer their questions and they could create a small community that

was already using our Of course, now that we have the platform, we are not done. We are in regular check-ins with our users. We try to understand how they're feeling, if they're feeling blocked, if something is preventing them from using our tool and we try to address it as soon as possible, preferably before it escalates in any manner. And when it comes to knowledge, we invested in

several initiatives. First one was really focused on Kubernetes itself. We created a small community of practice where we had experienced engineers making some hands-on workshops where the networking engineers could actually play around with Kubernetes, see how it works and apply this knowledge not only to our platform but many different platforms and systems around Swisscom. Additionally, a thing that really worked well for us was the initiative where

a networking engineer would come and join our team for a couple of weeks or months and then they would get to better understand how our tool is actually being built, what is happening behind the scenes and once they go back to their original team, they can spread the knowledge to all of their colleagues. So, we can see that even though it took several steps to get there,

step-by-step we were able to address all of these questions and concerns that popped on our table so far. So, this is in fact the story of how we have become the before-mentioned accidental platform team. And while we can say that we successfully addressed the topics that popped up, probably you're still wondering, "Okay, there is a platform, but does it actually work? What does your data that you

gathered actually say?" And we have that information for you as well. So, we will start with the old workflow that Fabian has already shown you. So, as you remember, the networking wizard had to go through Excel sheets, create different IP tables there, then populate the management systems inventories, create these change requests, and tediously test every single change that they he or she made. And this would take

days, if not weeks, to achieve. With our platform, we luckily managed to automate most of it. So, of course, there is still the initial investment needed, where our network engineers have to now create these high-level intent YAML files. We have provided them with UIs that help them be a little bit quicker with this And this takes a couple of hours now. And everything after that is more

or less automated. The lower-level intents get created from there, and they get realized, the corresponding CRs get created in just a matter of minutes. And very importantly, we have a whole set of open-source tools that we integrated when it comes to two two testing and to observability. So, we very quickly can provide them all the information they need to know that things are working. and even more

importantly these are available continuously. So we don't need to redo all of the testing every time when you change this and that. So we don't need to redo it, invest 10 days again. No, this comes out of the box. So we can see that we went from a matter of days and weeks to really a matter of hours, which we really do consider a success for our

platform. But now we have a question back to you. If we were to come to you now with this automation potion that you have heard all the details, how we make it, how it works, would you be more likely to try it? Do you feel more brave? Still yes, good. Well, luckily some of our users really do feel that way as well. And it has taken us

years to get to this point. It really is a process that needs investing in, but we are really happy to say that we now save weeks of toil on every change that they are making. And very importantly for the nature of the Telco industry, we are we are able to help them make these changes without any errors, which as you can imagine imagine is very important since

we're working with 5G network itself. This does not mean that we have this magic crystal ball and we have answers to everything that the future holds, of course. We still have some open topics and open questions that we are exploring. And we are actively trying to figure out what the next steps for us are. And one of the topics we are looking into is the fact that

there might always be some manual steps that the network engineers need to do. It will possibly never be We will never be able to have 100% automation. But we are not letting that stop us because already with this partial automation we have achieved so so efficiency gains that it is really already worth it and we don't let the thought that we will never be 100% discourage us

from proceeding. On the other hand, there are different types of systems that we are seeing in this network core area and some of them have a more imperative way of working, which is not really compatible with the declarative way we have taken so far, right? So, there we are looking into whether there are ways to reconcile these two ways of working or whether maybe for the imperative

systems another way for automating things should be taken. And in the end, the most mysterious potion of all that you have certainly heard a lot about throughout this conference and before, the topic of AI, we are also looking into it and we are exploring in what ways we can integrate it into our platform so that it meaningfully brings value to our users and it doesn't just introduce

unpredictable or not non-deterministic behavior, which would cause them to trust our platform less. So, that is very important and we will keep exploring how to do it. In the end, if you so happen to find yourself in a similar situation of accidentally becoming a platform team, here we have a couple of last takeaways that you can take with that should help you survive the process. And from

our experience, we can tell you that really we saw that investing into empowering our users, into creating trust with them is what makes things move forward. And now you can empower them through technical means like the graphical UI, like different metrics and traceability, but you should also invest in the human relationship and try to focus on the people who are ready to test your solution and focus

on this gradual adoption, but also invest into knowledge sharing and making sure that everyone can follow along with you down this journey. This is not the first talk that we have given on how we have come to automate this network core in Swisscom. Here you can see a couple of talks that we have given in previous years in KubeCon and other conferences. You can check them out.

You can also check out the open-source repositories that we either contribute to or we have open-sourced ourselves, like the Netbox operator or the schema-driven SBC. And you can find all of those and soon today's slides in this repo in GitHub. So, if you scan the QR code, you will immediately end up there. And with that, we thank you a lot for your attention and we are now

here for your questions. Hi. Thank you for the talk. It was great. I have a question that comes to domain knowledge. You said that before the wizards need to have a cross-domain knowledge. And now it's in the operator. How do you make sure that knowledge stays in the company? It's not forgotten as the wizards don't need to know about it anymore. So, I mean, they cannot completely

forget how the network works. They're still the same concepts and everything we do is still inside the company. So, all of these operators that we are building, that's also inside. So, it's not completely leaving the company. So, that's definitely the case, but hopefully they do not need to manually know which IPs to put where, how to connect the routers if it comes happens automatically. I think that

is not hopefully going to be necessary in the future to that level of detail. I don't know, Fabian, if you have anything to add. No, I think it's a good point. Thank you. Yeah, I have a good question also since I'm here. Yeah. So, thank you for the And they're inspiring. And uh thanks for us who's come to share since uh 2023 all this element. So, you

insist a lot on the fact that you you save a lot of time with automation. I was wondering is for your management so the the option for the main argument was to save time or to bring trust? Because at the end the 5G core we don't redeploy 5G core every every 2 weeks. It's it's relatively static, I would say, even if we security on CRA coming that

it will be a probably more upgrade uh and but at the end what is interesting I think it's more of the fact that you bring determinisms in this overprovision uh process. And at the end goal is really to to to building trust between the operations and the the the platform engineering. So, I think it's more important that that the speed of deployment even if it's more consequence

at the end that we put all this in place. From my point of view, I fully agree. And I think really the value of this saving time is not to drop these engineers or anything like, but to give them time on focus to focus on the important things, to maybe focus on further things they need to do, on improving the network core further, and we just give

them more time to focus on what actually matters from my point of view. And yes, they know that what they have done is stable and can be trusted, so they don't need to constantly keep their eyes on it. I fully agree. Yeah. But there's a valid point you said that we don't need to deploy a network every day. So, of course, for everything we need to decide

is it worth it to build automation for it or not. And there's also the fact like you say you have repeat you can repeat the same thing, there's no errors, but there's also things which we don't automate because like it doesn't make sense. We set it up we set it up once and usually don't touch it anymore. Right. Hi. With regards to the human perspective, how many

times uh did you almost give up on trying to convince people uh to use your solution? I I almost give up fully we didn't really, but are there days when it's difficult? Yes, there are. And simply you have to you have to manage, you have to come back when they're ready for it. And usually there were some users that were ready to proceed. We were never fully

blocked from all the sites, so that was good. And then we just go with the group and the use case that is ready to try our solution at the time. And we wait for the other ones to well, catch up. Thank you. Yeah, there's always going to be people that complain or not happy or have a different opinions. I mean, but if you have you can show

them that it works, this I think is the best to convince. First of all, thank you for the great presentation and for sharing your story. Like uh especially among us who like to introduce new technologies to the companies, it's often very difficult to convince people that the old ways are not really optimal nowadays. And especially in the companies that have multiple layers of ownership where every single

change has to be escalated upper sometimes to a different country. It's uh very often difficult to push changes. Like believe it or not, there is still companies that are discovering it's possible to split CICD and use GitHubs to reduce deployments from deployment time from half an hour to few seconds when our configuration is changed. Um I'm just curious uh where in the Swisscom at what point this

moment came where you decided that it would be beneficial to start optimizing these things. Did it come upper layers of management? Like did they decide that it's something uh that you should tackle? Or is it rather something that the engineers uh came with that would be nice to have and also fun to work with? I think I think we had both and you need both. So, basically

Swisscom had this thing where they say we want to go from a traditional telco to a techco, this was all at the top, and then we engineers at the bottom could start building this up, and I think it was very nice that we could always reference to that. That's also one reason why we can stand here. We're like, "Hey, you want to become a techco, so we

do techco things. We go to conferences and we automate things." So, uh I think we had it on both levels. I don't know if Yes, I agree. I think it's important that you have support from the higher levels and their understanding of why this matters, but then also you need the excited engineers who can really make it work. And in case the management was not as optimistic

or did not have enough budget or time to allocate for that, what would be What do you think would be a good way to convince them? I mean, usually you really need to explain what the value is, right? Just saying something is cool and new and most modern technology usually doesn't work. So, these are numbers that we have shown we really have used to explain around the

company why what we do matters as well, even if they're not so supportive immediately. And I think having some metrics, some explanations of why this works that they can follow, of course, not showing all of the details of the code or YAMLs, and then that really helps. And of course, always communicating with them so you understand what is actually the problem or what are they scared of.

I mean, is it It can also just be money, but maybe it's also something else. Maybe also they don't trust it. So, for example, we also built between our operators and actually the workload clusters, we built Git in between like for certain periods to gain trust. So, they're like, "Okay, they're maybe scared that we break stuff." So, we're like, "Okay, we first push to Git at one

point so they can review it, and once it works, we can then push directly to the cluster." So, depending, you need to understand what's the the problem is or what they're scared of, I think, and work with that. >> being transparent. >> Yes. All right. Thank you very much. Only for me should advise there is a CNTF documentations with rationals that are public and can be shared

also. So what you initiate that can be used also and transfer to to your management. Can be transferred. Yeah, maybe we can >> Yes. Yeah, there's also all the slides online and yeah. And if you didn't understand the slides, I saw that the gray boxes were not visible. So if you make slides, don't put gray boxes and you can check them online then you'll the nice graphs

with the gray boxes. And if you have any further questions, we're still around today so you can catch us afterwards as well. I think we're slowly running out of time here. So thank you again and see you around. Thanks.