Beyond the Edge: Cloud Native Application Management Under Ex... Tobias Nöthlich & Maximilian Nitsch

This talk discusses the challenges and solutions for implementing cloud-native application management in space environments, specifically focusing on the use of Kubernetes in conjunction with delay and disruption tolerant networking (DTN). The speakers, R&D engineers Tubius and Maximan, introduce the concept of deploying applications on celestial bodies like Mars, where existing cloud solutions falter due to unreliable communication links. They elaborate on the limitations of traditional Kubernetes, which require stable connections, and detail their development of a system that utilizes DTN for application deployment and data communication. The session also covers the Dark Soul project, aimed at creating a delay-tolerant cloud infrastructure and improving tenant separation in space operations. The speakers conclude with a discussion of future work, highlighting the need for enhanced security measures and the potential for opportunistic networking among space clusters.

All right. Uh thank you very much for coming today. Uh it's the last day. We're happy to see such a turnout. Um before starting with our presentation, uh maybe a short introduction is in order. So my name is Tubius. This is my colleague Maximan. And uh we are R&D engineers at a small Dston based company uh that tries to push the boundaries of communication forward every day

uh a little bit at a time. And uh this is why we are here today to try and bring cloudnative application management to space and to you and obviously also to other extreme network environments. Now a quick view of the agenda. Um I'll give a short motivation on why we're doing this. Uh then I'll pass the stage to my colleague Maxan who's our resident DTN expert. Um

to briefly introduce to you the background uh why we not just use Kubernetes. Uh what we use instead, how we make all of this work in space before I'll take over again and um well we'll get deep into the nitty-gritty of how we implemented all these things in Kubernetes in Helm and uh how we got it to work in the end. Uh then there's a slide on

key takeaways and future work. And on the last slide, there's going to be a few QR codes. Uh one for rating our session and a few for following us on LinkedIn and yeah the project website. Now without further ado, let's get started. Um maybe with a bit of a provocative question. Um who even cares about cloudnative applications in space? Well, it turns out quite a few people

do actually. Now why don't we see any cloudnative deployments in space? Uh it's a bit harder than it sounds. Uh especially if we look into the deep space context there. Um I've brought an example use case as you can see here on the slide. Um very simplistic. We have a cluster on earth uh which is used by some researchers and operator and is connected to a ground

station via well stable links as we have them on Earth usually. And then this cluster is connected to a cluster on Mars uh quite far away. The link is not as stable. uh there's quite a bit of one-way delay 13 minutes on average to be exact and then we have a satellite which is on around Mars orbits is connected to a ground station there and we have

a rover which runs some research applications now with this we have a few problems um the first problem I briefly mentioned before um deep space links are not continuous and they're also very frequent for disruptions very delayed very low bandwidth uh so we do not have the continuous network connectivity like we have it on earth and This makes it very hard to use established cloud solutions. More

on that in a few minutes. But uh just for now, it's basically impossible to have a Kubernetes cluster span Earth and Mars. The second point is that there are no flexible and easy to use cloudnative execution a uh environments for space agencies. They all have their own things. There is standardization ongoing. But uh even if they have their own standardized execution environments, obviously operators on Earth wouldn't

know how to use it. So there's a bit of a disjointment there uh which we're trying to address. So not a lot of adaption. The third point uh is not that important in the moment but we can foresee it becoming important in the future. Uh tenant separation. Tenant separation for space hardware software platforms is insufficient and just imagine you have your own research payload on a rover

and there's a another workload located on the same node and they can read your data. Then obviously that's not something we want. The last point is that deploying new software to existing satellite or rover platforms is done via dedicated tool chains. Now this goes hand in hand with the second point. Um usually missions are pre-planned. Uh the hardware is purpose-built, the software is purpose-built and if you

do need to run an update for some reason, then there's also a more or less purpose-built tool chain or at least a tool chain that's not available for public use. Now if we want to build an application deployment pipeline and obviously people that aren't part of space agencies should be able to use it as well. So that is obviously not the case at the moment. Now let's

tackle these problems one by one and uh since we're at CubeCon let's start with the execution environment and ask ourselves the question why can't we just use standard Kubernetes on the satellite. Okay. And that's quite easy because it's built for terrestrial use cases and optimized for stable low latency networks as we usually have in data centers. So for example, image pools uh like after a deployment requires

a stable internet connection to fetch the data and also the application data flows assumes an uninterrupted connectivity. So these requirements are rather easy to realize in a data center but very hard to do on satellites or space probes due to high delays and disruptions. So but there's a solution for this. It's called delay and disruption tolerant networking. Compared to traditional networks with permanent endto-end connections, DTN uses

a hopby hop transfer including physical movements of the nodes. To do so, they have to buffer the data in their internal memory or on persistent storage to deal with power outages for a potential arbitrary amount of time. So, DTNS are designed as an overlay network by introducing the bundle layer which allows DTNa applications to communicate over DTN with with other DTN applications over a potentially hoggeneous networks.

This is done by implementing so-called convergence layers to transfer the messages over different underlying network layer protocols. As already mentioned, the data transmission is hop by hop based including the including the physical movement of the nodes. Um and this adds an additional access to in addition to the time t because nodes may travel to another space location in space. um and have then other contacts to other

nodes. This can be either done by pre-calculated uh scheduled contacts or also opportunistic contacts are possible with then um an opportunistic discovery and transmission. But let's have a quick look at the example. So if node A wants to transmit a message to node C and they are never in range, it has to send the message first to node B as an immediate hop which then may travels

at a later point in time closer to node C and eventually transmit the message to its target destination. To combine the concepts of delay tolerant networking with state-of-the-art cluster management solutions, we freeTN started together with the T University of Technology and Cyrus technology a research project called Dark Soul, short for DTN architectures for space operations and space cloud life cycle management. The main research areas of this

project are first build a delay and disruption tolerant cloud infrastructure and apply secure separation layer concepts for multi-tenant operations and provide also application operations. On this slide you can see an example how a potential solution could look like. we have multiple individual clusters um consisting of one but usually multiple nodes. These can be traditional computers but also satellites in space, space probes or ground stations. It is

required that that these nodes within the clusters are connected with a low latency connections and persistently connected to each other. one or multiple node of the cluster needs to be uh a cluster management node which can receive management informations from other clusters and um the administrator of the clusters. If the worker load has a DTN capabilities, it needs to be able to forward these management informations to

the management node. Um one of the uh the most easy sorry the most easy uh cluster you can imagine is an traditional cluster in the internet as we have on earth on data centers which is connected via the internet to the ground stations which can then can be forward messages over the delay tolerant networking link to other clusters. Another option for a cluster could be an constellation

of LEO satellites here marked with 32 um in a lower earth orbit um which have an episodic link only to some crown stations um and therefore the crown station is not part of the cluster because of the requirement of the low latent permanent connections. These satellite constellations can also be placed in other orbits of other celestial bodies such as the moon or the Mars. In case of

the Mars, it could also be required that there is an additional data relay in place which is then not part of the cluster but forwards the pro the data as a proxy transparently. We can also build clusters uh with devices in space and on ground considering we have and stable uh low latency connections as we can build with geo satellites and for example an uh satellite telephone.

And last but not least, it is also possible to build an totally isolated cluster which operates autonomously after an initial configuration phase because we are disruption tolerant. And as you might have already guessed, the cluster software solution we are using for this is Kubernetes. Right? So this is what such a cluster could look like. Um we have a management node which runs the control plane. Uh this

is running on one or multiple satellites in the cluster. In this example, it's just a single one. And then we have various worker nodes uh which run the actual well worker loads. Now with this, if this is provisioned, it runs on its own. But there is still something missing and the missing component is that well we still need to be able to communicate right we need to

communicate between administrators or users and clusters uh and this needs to be delay tolerant because this cluster might be in the orbit around Mars. Now what do we need to do to actually realize this kind of communication? Well that's easy. Uh we just need to build an application we want to send to Mars. Now luckily this does not require delay tolerant communications. it just requires a delay

torrent team lead. So we just take it off the list and uh look into more interesting things. Um once we have an application, we also need to deploy it to a cluster. And for this we need delay tolerance because well we need to send it to Mars, right? Um the way this works on Earth is probably known to all of you. So I'm just going to go

over it very quickly. Now this is a very very simplified view of what happens when you run helm install. Now I'm pretty sure everybody is familiar with Helm, but for everybody who isn't, it's basically the package management for cubit uh for Kubernetes. Now the way this works is Helm is invoked. Uh it does its thing validating and rendering the Helm charts and then sends everything to the

API server. This is the first connection to the cluster over a link which could potentially be a space link. Now inside of the cluster um well Kubernetes sees well I should deploy something. There is something missing. uh it starts a pot creation process and the container runtime thinks well I need an image for this pot right I need to I need something which I can build this

pot on and this will also need to be fetched so we could have a local registry but the more interesting case is if it's a remote registry so for example dockerhub then we would have a second request which goes outside of the cluster again now why is this a problem well we briefly talked about it but we will not have a stable connection in space there's delay

case, there's disruptions. I mentioned the 30-minute delay to Mars. Um, and this is especially problematic with the image fetch. Um, there's lots of talking back and forth there. We need to establish a connection. We need to do authentication. Uh, we need to do the actual image pool. Um, but the connection is slow and it's not stable. So, it might require retries and each retry again costs time,

right? So, for example, if we try to do this over TCP, the average single trip latency is 13 minutes. TCP has a three-way handshake. So even if it weren't working um this will be 39 minutes just for establishing a connection which would time out anyways because TCP is not configured to deal with like delays of 13 minutes in a single trip. So we need a different solution

for this and this is what we built. So we build an extension for helm we call aptly helm dtn and the way this works is you still invoke helm but now with a plugin helm dtn and then you run helm dtn install push in your helm chart and some extra configuration and helm invokes the helm dtn executable. Now what happens here is the helm dtn executable goes

through your charts goes through your values tries to find all images or other dependencies. It's very extensible. You can also I don't know fetch git repositories or do wget or curl calls and it pre-loads all this stuff. It loads it to your local machine and has it there. Now it then goes through the charts again through the values yl updates all your uh references to point to

a local registry running in the cluster for the images for example and packs everything in a tar. So we now have an archive containing all the dependencies the uh charts that should be deployed and need to send it to the cluster and this is where microdtn comes in which is our implementation of the DTN networking protocol the bundle and we have two microdtn instances one is running

on the internet enabled side which is your local computer for example and one is running in the remote cluster and with this we are able to bridge the space link using DTN which Maxmillian already explained. Now once the archive arrives on the other side in the space cluster on Mars wherever uh it is forwarded to an application life cycle management engine and all this does is basically

it takes the archive puts everything uh on your local drive or unzips it basically um pushes the images to the registry the local registry in this case and itself invokes helm install now this then works basically the same way as I explained two slides earlier except that we're not fetching from a local registry we're fetching from the registry uh in the cluster which has been pre-populated with

the images we need. So with this we can now deploy an application. This is very nice. We can take another thing off of our list. Just two more to go. Uh now we have an application running in the cluster but it is not able to communicate with us. It can do its thing. It can scan images from a rover. It can do atmospheric analysis or whatever but

there's no way for it to send the data back to Earth. Uh so we do need to send application data to other clusters. uh for example an earthbased cluster and we would also like to send commands back up. So the other way is also important. We need to receive data from other clusters. Now as a short throwback uh let's go back to the overview of the space

cloud. Now you can see several dashed lines in this image and each of these dashed lines is a link that is either high latency, unstable or only available episodically. And for these links, we need an ingress or egress solution that can cope with these conditions. So we can send application So we thought, well, let's let's make it as easy as possible, right? We need a BPA resource

microdtn which actually can communicate over these kind of networks. We need a component for getting data to the applications and ingress and we need a component for sending data out from the applications over the network to another cluster and egos. And all of this should be connected using a existing protocol which is the application or agent protocol which is uh also included in microtm. Now well let

me quickly say this is the first time we really work with Kubernetes. So the next slides are probably going to be weird for a few of you but uh without going or with going in there without any prior knowledge we thought let's just use the gateway API. We're building a gateway. Let's use the gateway API. uh first idea was well you know we have microt that is

required we have an ingress component let's make this the gateway so well that matches with the original sketch I showed earlier um we spawn our DTN ingress proxy this connects to microdtn registers agent and then forwards data to the applications so how does it forward data well I don't know maybe a TCP route maybe a plain TCP connection obviously please bear with me for a second here

that spoiler alert doesn't work like that Um the way the gateway API works or the way we understood it is that it wants to register listeners and expects to be the first point in the cluster where data is received. And this is obviously not the case here. We still have microdtn at the edge receiving and forwarding data. Uh we have it the gateway as the DTN proxy

which would then need to map the data we already receive to the listeners and forward it to the application. Uh which is obviously very confusing for people used to how the gateway API is supposed to be used. And on top of that, the second option is even worse. The plain TCP connection option is doesn't even use listeners. It doesn't even use routes. So, um, basically doesn't use

anything from the gateway API that's worth using. So, we threw that idea out. There's a second idea we had. Let's just make microt the gateway. Uh, this would have the benefit of actually using the listeners correctly, right? That would fix that issue. Uh, however, some problems remain. Um first of all obviously there's no DTN support or no protocol uh DTN support for the gateway API. Uh we

would need to build a custom route type AIP route which is very possible but uh the gateway API by its very design is built around the paradigm of realtime connection oriented network ingress and routing and its core concepts kind of reflect that right listeners protocol forwarding immediate routing to backends. um they're optimized for low latency always connected networks or environments like HTTP or typ typical TCP and

MicroDtn doesn't really do that. So MicroDtn is doing far more than just forwarding bytes. Uh it's performing bundle processing, right? We need to unpack all the data structures from from the bundle protocol. We need to handle persistence, right? We're working with a store carry forward protocol. There's a chance a bundle never makes it past the entry point which is microt. There's custody transfer. So we know which

node needs to retransmit in case a bundle is lost. And there's also opportunistic routing. And these DTN specific operations do not have a direct equivalent or abstraction within the gateways API a gateway API's listener or route model. So even if we were to use it correctly, it would still a be a massive undertaking to bring this somehow into the gateway API and it would still be confusing

for everyone. So we decided that doesn't work either. So we decided to build a CRDA custom resource definition which combines ingress and eress routing for DTNS instead. And this is where we are now. So we have the D gate uh which is short for dark soul gateway and this is a CRD which provides an ingress solution for us um which allows us to register agents at microdtn

which can then receive data from outside the cluster designed for a well particular application and forwards this data to the correct backend application. Uh the other way around is also possible. Applications can reuse the same connection to send data back out. So we have a combined ingress solution there. The way this works is you have your YAML for the for the D gate and you configure your

application. You tell it what's the service it's reachable at. What's the service microdn is reachable at what's the name of the agent that should be uh registered and what support we uh well the application listens for incoming traffic and uh then the dgate goes ahead registers the agent so it's reachable under for example microdtn.dtn/lication one. Once data is received, it's forwarded to the Dgate. Dgate knows, okay,

this is from the app one connection, forwards it to the application and the application is happy and received its command. It takes a picture. The rover does some pre-processing. The picture is sent back out using the same connection. No further uh well configuration needed. D8 forwards it to microt send out using the same agent so the other cluster also knows where it's coming from. So this works

very nice. Uh there's also a link to a previous tech talk I held on this topic uh or well on the predecessor of the gate the de the gate uh here on the slide if you're interested. Uh this goes more into detail of the how we did the ingress on egress. And uh that's actually all we wanted to show you. So we can now send data, we

can receive data, we can deploy our application, we can update it, we can delete it again and everybody's happy, right? So what are the key takeaways? Well, we found that networking in space poses challenges that are not usually encountered on Earth. Um and addressing these challenges is very important because otherwise we can't do networking. So what do we do? We use DTN and this opens up the

possibility for cloudnative application deployment even in deep space Mars beyond even deeper it doesn't matter. However uh we still can't span clusters over well multiple set multiple planets right the delays are too big. Um the Kubernetes components are tightly coupled. They're expecting heartbeats in a timely manner. And if we just, you know, wait 13 minutes for a heartbeat, we well don't need to send any heartbeat at

all because we just know it 13 minutes late, right? Doesn't make sense. So we have the application deployment and life cycle down so far. Now what's future work? Well, there's a lot of platform related work left. Uh we need to do autonomous deployment and upgrades of the Kubernetes environment. This is not touched at all. This is also uh not part of this presentation. Obviously, we need to

do security related work. The tenant separation is still not there where it's supposed to be. Uh we also wanted to note and workload at a station. There's also DTN related work. Um imagine a cluster that's spacecraft that were sent out for a I don't know discovery mission. What we would like to have is then be able to connect to clusters along the way to hand off findings

which were then transmitted back to Earth in the next possible case. So we need opportunistic discovery of clusters. And we also want to test this practically. There is a follow-up project uh where we want to explore project components in space terrestrial internet working scenarios using satellites deployed and operated by ISA to gain further insights in the applicability of our overall solution. That's the end of our presentation.

Here the oblig obligatory QR codes I mentioned earlier u on your left or yeah on your left feedback for the session project website is in the middle and if you want to you can also follow us on LinkedIn. We post updates there as well. Thank you very much again and hope you enjoyed it.