Confidential Platforms for Regulated Industries - William Rizzo, Mirantis

This talk presents a solution for maximizing the utilization of expensive trading hardware by creating ephemeral Kubernetes clusters using K0S. The speaker, William Ritzo, discusses the challenge of ensuring these clusters remain compliant during audits while operating efficiently during trading hours. He outlines the architecture involving several tools like Cluster API, Metal LB, and Cordant for unified state management and observability. The session also covers implementing a mutating admission webhook to ensure confidential runtime enforcement for container deployments. Additionally, the concept of 'spot scheduling' is introduced, allowing teams to allocate idle hardware for temporary workloads. The talk emphasizes the importance of security in the development process and introduces the Five Spot project for time-based machine scheduling.

So what we are going to go through is um what started as an experiment for a challenge. So the challenge was to well the initial request was to maximize the utilization of very expensive hardware destined to trading uh during something like six hours or so. dur uh this talk was I apologize this talk if you looked at it at the beginning of of the when the schedule

was announced uh we were supposed to be two um myself and another person uh and this other person a good friend and a fantastic engineer uh worked for the regulated industry for the regulator firm that we are talking about and we designed this pretty much at forens uh for reasons that we're not going to go through he could not make it. Uh but Eric, we wish you

were here. So the initial challenge was to maximize the utilization of this very expensive hardware and find a way to uh to en encapsulate K0. Uh everyone knows K0. No really lightweight Kubernetes engine like it's the best there is. Okay, whatever. CNCF project. um the to encapsulate the the K0S binary and everything that is used to use these systems that are under audit. So uh and then

somehow uh remove this this encapsulation let's call it like this now and um uh for the for the for the software that does the audit before the start of the trading that everything is clean. So this is this is the premise. So my name is William Ritzo. Uh I am a a global field CTO at Mirantis. I'm in the Kubernetes release team. Uh I'm a CNCF ambassador

linker ambassador and I'm a Chyros maintainer. Specifically I work in Chyros. I work on the uh cluster API provider integration. Uh and I'm a member contributor in Fenos the open source in finance foundation. So as I said uh we had uh something like 10,000 systems approximately we still have something like 10,000 systems that are very expensive and the re the demand was how can we maximize the

capital expenditure that we have on these systems. These systems are working for six hours a day and then for the rest of the day they're doing nothing but they are under audit. So if something is uh not aligned with how the auditor are expected to find the systems then they have to be t taken out of the trading pool. Sounds familiar to anyone? Not at all. Okay.

One guy. Okay. So the uh so there is obviously regulatory burden as I said there is also a bunch of uh problem with fragmented tooling that we might have we might might not have in financial uh services for but this this last two I would say that is not necessarily true for necessarily true for this environment but we decided to add them as as additional things that

we want to work through in this solution. So we came up with this idea what if we what what if we build these clusters in a in a ephemeral way. So what if once the the trading time the the work the daily work of these systems uh is is done what if we turn these systems into clusters. We expand clusters or we expand the worker uh pool

of these clusters into the the metal and and we make we make everything disappear. How can we achieve this? So we we found this way. So for a moment let's pause for a moment on the spot scheduling. Let's focus on the on the ephemereral cluster and the confidential part for the ephemeral clusters. We decided to use the hosted control plane pattern would uh would play uh would

play a role. So in a in a cluster API for those that are not aware there are two patterns. There is the management cluster which deploy managed or child or downstream clusters via cluster API. Who doesn't know cluster API? Beautiful. Uh, pretty much everyone. So, cluster API is an API for clusters. So it's a is a project is a project within the kubernetes sig uh and is

to provide um a set of API namely tree infrastructure provider control plane provider and bootstrap provider for declarative declaratively uh create clusters from a managed a managed or unmanaged Kubernetes cluster. So let's let's u summarize it. We can have control planes which are in the child clusters or we can have control planes which are hosted in the manage cluster as pods in the management cluster as pod.

So we we thought that if we to have this something like uh this manage cluster what what we end up having I'm going to push one slide forward. So in the hosted control plane you would have the management cluster which uh retains pods that are only for the that that retains the control plane of the managed clusters in uh as pods. So we would have CD uh

the controller manager, the cloud controller, the API server uh and so on and so forth of the child cluster which can be on cloud on prem or you can have machine pools on prem and on cloud whatever you feel creative to do. So going back in the architecture we have uh Amirant's uh open source project called Cordant uh which is what we like to call a platform

control plane uh and cordant uh unify the man the state management and the cluster management and optionally the observability of the entire state uh through copy. Then we obviously use cluster API for this solution in particular. We decided to use Kyao for uh policy enforcement in the in the entire cluster estate. Uh Flux uh that we all know and love. Who knows Flux? Okay, take your hand

down. Who uses Way less than the people that actually raise the hand. Are there flux maintainers here? I know there is one. Okay. So, and then uh last but not least, we use Metal LB. The reason that we needed Metal LB um is well, we need a lot we are doing everything on prem. So, we need something that provides us with the load balancer service uh uh

for the for the for the control plane of the child cluster. So metalb will have a set of IPs uh IP pool and IP ranges defined for the control planes that are going to be uh created and for the clusters that are going to be created. So when on the other side we see what a child cluster baseline looks like. So the green one is something that

we let's call it um a nonspot cluster. So we would have containerd obviously and we have we have focused on k0 but we can do this with any type of kubernetes engine. So we have container d uh which and a mutating admission webbook I will get to that in a minute and we have the kataka kokco runtime. So the confidential containers runtime. So what we do we

do a transparent in uh a transparent injection of the runtime on any deployment that is coming into the cluster. So as as I said we use a webbook uh ad mutating admission webbook that that I that well we have written specifically for this. So whatever the developer pushes to the to the to the to the cluster, whatever deployment pu gets pushed, the runtime class gets switched and

we use the kata uh the kata co confidential container run class. So from the uh confidential container operator, sorry my tongue went everywhere. So the that's what the mutating webbook does. Now this is not ideal. The way you want to do this with with something that is it's has the job of only doing that and it's a first class citizen in uh in Kubernetes. Nowadays there is

when we did when we wrote the these mutating and web book there was no uh native way of doing it in Kubernetes. We well it was Kubernetes 133. So you needed to enable the feature flag mutating admission and you do that through Kubernetes. So with writing a mutating admission webbook with Kubernetes 135 the uh mutating admission mutating admission is a first class citizen and is enabled and

we can do that natively with Kubernetes and that's the way to do it. It's for for day two operations. The best way to do is to use native what Kubernetes brings you. And we have decided specifically not to do this with Kyerno. Again, we uh if Kyivero would fail, it doesn't fail, but if Kyivero would fail, uh nothing would get uh would get scheduled on the cluster,

which might be a good thing, but that means that nothing would get mutated. So and there is no policy control so nothing will be u running. Uh so this um this is how they the run actually I think I can do this. Yeah I can. So okay Alex doesn't like to do that. So here we you see the runtime class how it gets mutated. This gets added

by the webbook admission uh mutation. Okay. So what do we actually do with Kyberno? These are policies that we just check. The first one we validate that this is the condition that any runtime any uh deployment as the runtime uh kata for the the runtime the runtime class uh kata the the we we one thing that I didn't say there are certain name spaces that we don't

want to touch like the cube system uh we don't want to touch flux of uh nobody wants to mess around with flux. It's already complex as it is. Uh the the image we we make sure that the image pool policy must be uh must be always the images must come from in this case artifactory but we can we can use any uh image registry. uh and we

can the certain resource limits are uh applied as they uh as they we always want them to have and in certain cases we always we also make sure that the name of resources match the what we expect by the from from audit. So the obvious uh the I don't need to I don't need to take to say this to anyone that is in a in an air

gap envir in a regulated environment everything is air gap you cannot have a cluster that goes out so all the images all the binaries all the uh providers whatever we use the cosmotron whatever um sorry cosmotron is the um bootstrap provider that is embedded in coordinant that is part also k0 is also CNCF object which provides the hosted control plane. Also these images are in the registry.

Also the K0 images that are pulled and the binaries that are pulled are in the registry. Now how does the spot confident this is we go we go back to this uh to this point. How does the spotuling works? So at 6 pm the markets close and at around 7 p.m. we begin the claims. The claim looked like something the there are several teams that have hardware

that they can spare. Maybe production hardware, maybe no production hardware. I know it sounds very horrible. Uh for the moment we are not doing it on production, but we have staging and development environment and very expensive systems that can stay maybe 12 14 hours without doing anything. uh so we they can submit this uh this hardware to something that will show you I will show to the

end is a spotuling uh and the workloads and the clusters get formed the workload will start then around 5:00 a.m. the the servers get drained and get released and at 6:00 a.m. everything returns to normal that's pretty straightforward. The challenges uh well for the moment I I even if it sounds a bit odd we are not doing we are not running on trusted execution environments. First of

all these systems were not were not designed with the TE chip where we know what the TE chip is. Yeah. Okay. So to to be a a real confidential platform or a confidential uh system on the hardware there is something called trusted execution environment. That's an actual chip. So the workloads that are um that are running on this system get targeted get targeted to run on that

chip by the container uh by the container class run the runtime class. So in our case we are not using a runtime class that can do that but we'll we'll this we'll use the QMO um hypervisor on the host which is used anyway because not all systems have a tusted ex execution environment. Uh all of these already expressed. So what we learned from there are some lessons

lessons that we learned the the one that I want the two that I want to bring forward is the security as an enabler. We see security as a problem. So let okay let's scratch that from the recording. Security is never a problem. So we see security as something that might slow us down or we need we have the compliance. We have the regulators. We have the auditors

and this is not necessarily part of security. Security begins way before that right. So we have practices we have how we build our platforms. We have our deploy how we deploy our software and necessarily how we develop our software and uh how we do the build of the images so on and so forth. So we are not addressing that that has been addressed already and to be

honest a lot of this has been already addressed by certain good practices that they have in that environment with flux. So but in this case we have certain uh compliance the compliance that we have to stick uh with which the confidential containers already give us for free. We already we already get it. So providing to the developers this transparent injection of runtime, they don't have to think

about it. They don't even have to necessarily know. Uh and that's that's a that's a big win because I don't know if if any of you has been following the platform engineering talk, but I'm 100% sure that they have been talking about accelerating developers, about reducing cognitive load and all these good things and these type of uh solutions. I think that that's what going to is going

to bring us there. It does. It did. Yeah. Okay. So, what is the the next phase? How many times did I jump? Okay. What is the next phase? Uh so, that's where we are now. Where where we were a couple of months ago. Where we are now is we're building um how many people know Chyros? I'm a caros maintainer. If you don't if if you don't know

it, please raise your hands. It makes me feel much better. Thank you. Thank you. Wow. Okay. So, Chyros is a framework uh is what we some people call Chyros a Linux distribution, an immutable Linux distribution, but it isn't. It's a framework uh to build is a meta distribution which means it's a framework to build immutable uh distributions from OCI uh artifacts from containers. So uh the great

thing about chyros is among many other things is that it's uh makes makes the operating system soro untouchable meaning immutable meaning the this the hardware we can use TPM to trusted uh trusted boot uh trusted sign boot to um describe what is the operating system that needs to be boot and nothing else. So that's what we want to boot. We don't want to boot anything nothing else.

We are not going going to go there with this um with this solution with the TPM because the system might not be obviously are not going to be used just to serve our uh Kubernetes workloads. We uh we have built a pipeline for uh we're building a pipeline for to to have these images these operating system images declare uh declared via the um via pipeline of course

and create golden images that do not drift so on and so forth. We want to use pify at test station for the workloads as well as the the operating system itself. So we are not it's not enough for us that the hardware can can boot that operating system. We want to know that that's the one that we intended to boot in first place and we want to

go to to the hard trusted execution environment but that's something of course that is not uh so straightforward to achieve. How does the attestation will look like? So first of all we have a hardware boot with TPM. That's not always the case, but we want to do the measurement uh for at least for virtual machines. We want to measure that TPM and attest it to the attestation

server. Then we want to deploy the immutable OS and the work get a workload ids vpy. Uh we attest the confidential runtime and then the auditors and compliance regulators are all happy. AI. Okay. So, we want to obviously we want to obviously uh try to use um uh confidential runtime as well with uh with GPUs. But this is something that is so so far away. we are

more interested in optimizing this this platform and the u confidential runtime is just experimental at the moment as far as I know with Nvidia. this spot scheduling how do we do that part of spot scheduling? Uh this is in my opinion is pretty cool. Do you know? Does anyone of you what know what it means? Five spot. Five spot. Nothing. Spot five. Spot me. Five. No. Spot

me five. Uh, okay. Well, that's that's a name that uh uh great engineer Eric Bourgeoa, unfortunat. The idea is that you let spot me five is spot me five dollars. Uh so the the idea is to create this uh this timebased scheduling for machines. So the develop the the teams that own these hardware can u donate this these the this these machines and specify until what time

from what time until what time they can be used or they or they when they should be released. uh fivespot is a project that he wrote again and uh it's uh it supports graceful uh graceful life cycle of the machine. So graceful shutdown graceful pause but also emergency kill switch time based scheduling. So until uh uh 6 p.m. nothing can happen to that machine and from 700

p.m. on onward it can be pulled into the the pool for scheduling uh the cluster expansion. I say expansion because we don't necessarily need to destroy the clusters that we create. We can reduce them to only a control plane in uh the management cluster and then expand into the spot uh pool um as we need it. Uh the great thing I put that QR code there because

you're not going to get to the project. You're going to get to a virus that gets installed in your phone, but you can scan it. So no the the it's uh this five spot the good thing is that it's been donated uh to the uh open source in finance foundation and uh thanks to Eric and and his team that has been working very hard with it and

it has been accepted they are going through the governance and you know you know regulated environments they have to check all the boxes and 15 people have to go through it and then it's going to be reviewed again for by other 15 people. So that's what the process that is going to go but it's going to is it's in the process of being open sourced accepted and

we will can all enjoy it also for use cases that are not necessarily for regulated industries and uh this has been great and that's um all I had if you have any questions I think we have six minutes more. Any questions? >> Yeah. Uh there is a microphone there. If you can please do it. So with the example that you gave um what type of workloads would

run uh on those clusters that you would run overnight was just internal to the company or also for outside and what are the plans for that with spot 5? >> So the the the the the clients of these platforms are all internal to the organization. uh they from what I understand they run experiments and tests and development. Uh normally they they want to they need uh they

need some uh uh compute capacity to to run their uh their tests and um and the this is what's running. So it's not something for the moment is not something that is going to be dedicated to any sort of production also because it will die in in matter of hours. more questions? >> Okay. Well, thank you very much and I wish you a good rest of CubeCon.