How To (Not) Fork Headlamp - Joaquim Rocha, Amutable

This talk introduces Headlamp, a graphical user interface designed for Kubernetes, emphasizing its extensibility through a plugin system. The speaker, Joaquim, who is a maintainer at Immutable, explains how Headlamp can be run as a desktop application or within a cluster, and highlights its vendor-independent architecture. Furthermore, Joaquim discusses the different strategies for extending Headlamp, including using JavaScript bundles to create plugins that can integrate seamlessly with Kubernetes workflows. He also covers the considerations for forking Headlamp, the implications of doing so, and best practices for maintaining forks, emphasizing the balance between flexibility and the complexity of managing separate codebases.

Thank you very much for being here. I'm here to give a talk about Headlamp and how to not fork Headlamp and to also fork it. We'll see. Uh my name is Joaquim. I work at Immutable and I am a maintainer of Headlamp together with other folks like Renee here. And yeah, I'm also a community C GUI lead. So what is this talk about? I was told by

somebody that the title was somehow edgy. It's It's not really supposed to be or maybe. So yeah, so so what we're going to see is what is Headlamp. What is the Headlamp plugin system? And then do you really need to fork Headlamp? And then spoiler alert, ways to to fork All right, so Headlamp. So who knew about Headlamp before this KubeCon? I should have said the other

way. Sorry. Who didn't know about Headlamp before the KubeCon? Okay. Okay, so so I don't have to do a big introduction. So let's do a quick one. Graphical user interface for Kubernetes. More from the operator point of view. So you see your workloads and everything. I actually thought okay, I'll just pull it up. Let's see. And live into my home lab. In my home. So yeah, so

it's a it's a Kubernetes graphical UI. Usually super fast, right? But we are at the conference Wi-Fi. You can see your workloads. You can open port forwards. All All of that thing All of those things. And we'll see also how it's extensible. So for example, I'm running Flux in my home lab. So I have this Flux plugin which allows me to uh you know, see what's going

on if everything is okay. I can also, you know, I push uh um I push an update and then I I can just sync or I can pause the sync. Tells me when it's going to be the next reconciliation. All of these sort of things, I mean the talk is not about flux but it's to show you that everything is kind of seamless seamlessly integrated as a

plug-in. Right, so uh besides that Headlamp is officially part of the Kubernetes project uh under the Uh you can run it as a desktop application like I just did or you can uh like in cluster or as a deployed service. We also have a an extension for Docker, an extension for Podman Desktop, Backstage extension. We try to try to get it everywhere, okay? Um It's vendor independent

so it's not I you know, it's not it's not depending on any flavor of Kubernetes and you don't have to install anything in Kubernetes uh to run it. Uh usually people want to know what about authentication and all that. We don't have an authentication layer. We expect you to to use um your Kubernetes are back with it so if someone connects over OIDC and stuff like that,

the UI even adapts to whether they can delete a resource. If they cannot, you won't see a delete button for example. So uh so yeah, so that's the philosophy uh and the big thing is also is that it's extensible via plug-ins. yeah, and I put it in there. I didn't know how to how to how to phrase this but I see it as a natural successor of

of Kubernetes dashboard being under the SIG UI again which is a project that uh that was archived uh earlier this year as many of you may know. So if you want to know know more that's the website down there. Now, extending Headlamp uh just to give you an idea as I was kind of a build up to to to forks and Uh so what how do you

extend Headlamp? So the plugins or sometimes we also call it UI plugins, they're essentially JS bundles, right? So we have a whole SDK, if you will, that you can create these these uh plugins. Uh you build them into into JS bundles, then Headlamp loads them. They are dynamically loaded into the into the front end. Uh yeah, and how do you publish them? Since we have a a

catalog, it's via Artifact Hub, so everything is kind of decentralized in that way. Uh and even the plugin catalog that we have in the desktop version, that's that's not even part of the core. That's also a plugin, right? So this allows you to allows us to be very flexible and if you want to deploy Headlamp without any any plugin catalog, you can just not ship the plugin

catalog, right? Which is the case in when you when you run Headlamp in cluster right now. What the plugins do? So you probably guessed that they do uh graphical user interface extensions. So you know, if you want to register a new route uh that points into a a page, an entire page, you can do. Uh head by actions like, okay, I want to create a button that

when you click it opens a ticket in my uh Jira system or whatever, you should you can do that. Uh branding is also possible, so you can change the the the logo and everything. Uh if you sorry. If you looked at the if you've seen the AKS desktop project, uh this is uh some of the of the functionality that it's uh using. Of course, there's uh besides

the the the GUI part, there's also data-related uh part. So Kubernetes API-related part. So we have a a nice way of using uh of of being able to list uh the pods, uh the CRDs and all that. Um yeah, and of course, we also have some APIs that are specific to knowing am I running in a desktop environment or am I running in cluster? Because depending on

those, maybe you want to restrict the functionality of your plug-in for and even stuff that is a bit more uh that is actually towards the desktop like running commands, it's also you know specific API in this case. if you have plug-ins why am I here to talk about forks or to advise against forks? Right? So one question is usually is everything changeable by default? Can you Can

you override everything by a plug-in? And the answer is no. When we do some When we implement something we don't we don't necessarily make it available to plug-ins right away. And the the the rationale here is that that requires extra implementation work of course from us. We do the implementation. Now how do we How do we expose it to plug-ins? Every time we expose something to that's

going to it's on us as the maintainers of the project to be able to carry this forward. So um and so far until now I I don't think we ever broke API uh for the plug-ins. At least not intentionally. So yeah, so there are also security implications. Sometimes we're like, "Okay, is this something that the plug-in should touch?" And it's an increased maintenance burden on us, right?

Because Because of what I just said, we want to be able to for people to not worry about breakage. So the new plug-in functionality is on a need basis. So if you if you're like, "Hey, I there's this part of Hadron but I really like to change this." And that's super neat niche is like a cave. Let's see if that makes sense or if that's easy to

maintain and all that. If we see like, "Oh yeah, this is actually very useful to change." Usually we you know we we try to get to to add How do you do that? You can of course open a PR, but you can also start a to and see uh you know, who's interested in in opening that functionality. Okay, so like I said, plugins allow implementations to focus

just on their functionality, right? You don't have to now care about the API that the because we already offer a way to list and to do many things that are expected to to be needed by plugins, right? And that the the question now is if we since we have plugins do you ever need to fork Helm? So, who's Do you or do you not? Who who thinks

you you do need? By show of hands. Sometimes. Sometimes you do. This was not the emotional drive that I was thinking it would be. But uh So so before we go of why why we may uh what is a a fork anyway, right? Because when I said I was talking to somebody um before the I guess that stop uh thing was talking to somebody about it. I

mentioned fork and they kind of freaked uh because fork can have this negative connotation, right? So so what do I mean by fork? By for By fork I mean I mean essentially what sometimes is called a friendly fork or a lean fork. It's basically it is a fork, right? You clone the project and you you're responsible for that clone. But the idea is to have a small

delta, right? So, let's say you want to change something that is very specific in Helm like the color of one letter uh somewhere. You have to fork the whole project. If that's just the the change you're doing, that should be fine. If you start changing more and more and more stuff, then it's going to be difficult. Uh it's also a fork where we expect that the changes

that we do that make sense to be given back to the community are indeed sent back to the community. That will help everybody. We'll make the the original project more valuable. We'll take that off your off your hands as a as as plugin developers or as fork maintainers, right? And it should be easy to update and rebase. The higher delta, the more difficult it to maintain. Now,

why would you need a fork? Very quickly. Plugins are not enough. That was very clear, I think. But there's also something to be said about speed. So if you if you um if you want to change something in Head Lamp and you send us the the PR, we maintainers have to make time to look at Most of the times we don't have time. Uh that can drag

a little bit or sometimes we're like, "Oh, actually this, you know, it's we go through the review process." That may require some iteration. and you just need that change out. So if the change is good for you, then just apply it. Security fixes, so uh we may have a security fix lined up, but we may have the release only in like a week depending on the security

fix, right? Uh but you may need to to release it tomorrow. So so that also gives you that flexibility if you have a fork. And then there is one thing to be said, which is not less important, I think, which is the organization practicality. What does this mean? It It's like if you have a uh if you have a a clone, if you try to avoid having

a fork of Head Lamp with ideas like, "Oh, yeah, but what if I just delete the tarball?" Um you know, extract the tarball, apply some patches. That's that's going to be a mess. Okay, you you need Git. Git exists for many good reasons, and and Git will help you if you have a fork. Um however, you know when you have a fork, as we'll we'll see, you

also have, of course, the the repo structure that comes with Head Lamp. And maybe that's not what you want. plugins or a fork, we should have both sometimes. Now, uh if you have both, basically you have full uh flexibility. Uh anything that you put in the plugin, and by the way, the the the point here is like, okay, so if you have a fork, why would you

need a plugin anyway? If you have anything in a plugin, basically you don't have to care about integrating that. If you have stuff in the fork, every time you you you rebase the fork, probably you're going to hit some conflicts, and then it's on you to fix them, right? Yeah, so that's essentially what I was already talked about. Having the plugins in the fork when you need

like full control of releases and and this and that will give you full So now we go to uh ways to fork, right? And uh we talked about how not to fork, like don't don't get a tarball, that's not a fork. Let's see because we may have different ways of working headlamp. So one quick uh and the most common and obvious approach would be you you do

a clone you have a clone of headlamp, and uh that's that's your repo. You call it whatever, my desktop or my dashboard, right? And uh and that's very very simple. You have the history from headlamp, you apply your your patches on top, you rebase and However, there are a few or many cons uh to that. The repo structure, like I was saying, is tied to in headlamp,

we have a uh folder called right? So now if you plan on having a uh your plugin inside the plugins folder, that's definitely possible. If later we change it to uh shipped plugins, then you you have to take care of that change. So so again, uh which is tied, of course, to get history, will will play a role here. Or does CI so you you cloned it

you we have GitHub GitHub actions the CI is in there you have to figure out like is that what you want? So this is what it usually looks like plugin repo file structure is essentially the the you know even though you apply I put plugin here it should be like fork but the the files without the the mark are green they are the headlamp original then you

have some plugin related files and you have some change headlamp files and everything is in is is mixed together but the git history is quite simple right? You have the headlamp stuff and then you have your own stuff. The the difficult part comes when you have to update headlamp space. Now let's look at something different who knows about git subtrees? Really? so git subtrees is is essentially

importing a git repository into a subdirectory and you get one you can have several commits I guess but usually you want to use a squash option so basically it you get one commit that brings in an entire directory with all the changes related to a ref of the of the project that you you importing like this. So what what does this mean? Now the the repo that

you have is your plugin repo it's your structure you decide and then you decide what to call to the actual headlamp folder where you're going to have the headlamp files. So it's a it's a very simple concept Um subtrees allow you to to pull and to push from the subdirectory. I actually never tried to do that but I was told it works. And who's familiar with submodules?

More people yeah. Exactly and uh it's supposedly less less of a burden than submodules. I I don't love sub mo- I don't like sub modules. But, uh but yeah, we will see. Uh I also have a slide about sub modules. Right? So, there are some cons, right? So, the the the history is mixed. Like I said, this is just a directory and Git does some internal magic

to know that, okay, when you're when you want to pull or push from inside this subdirectory, it it actually means the it actually looks at the at the remote from headlamp in the in in our example, right? Um but then you you still have a a history. Why don't I just do it here, right? Uh you have you have basically the the squashed commits uh that would

be the initial commit, if you want. So, you start with a subtree with headlamp. Then you you add your plugins plugin files. And uh and then at some point you want to bring in again new changes from headlamp, a new version, right? And that is another commit in your repo. uh I have uh some experience in in handling forks. I always prefer to have the original uh

code or the upstream code to be the first code in my Git history, so I can just focus on my changes. Because usually the case is that you have lots of changes from the from the upstream um project and the downstream changes are less, ideally. Right? So, so so I don't love that now I have a commit that is in here, so I have to rebase. How

do I rebase? Apparently, there's no subtree rebase. I ho- I hope I'm not wrong at this point. Uh I I know how to do this manually, but it's it's again extra burden. And the repo file structure is very clean, like I said, you have things separated, so that's the thing on the left. Now, we go to the sub module. So, a sub module uh is a way

to tell Git to tell Git, "Hey, uh I'm actually going to create here uh a directory, right? That is actually a pointer. It's not it's not that you import the whole history into your history. You just say, "This is a pointer to another repo." or to another remote, usually. So, so when you navigate to that folder, you're still going to have in in our example, that would

be the headlamp folder, you're still going to see the Sorry. But when you do get logged inside that folder, it's only going to be the the the you know, the headlamp history. So, you can you have a separation of your plugin history and the headlamp Even better, you can go there and you can apply your own changes to the sub module, right? And then when you do

get logged, you're going to see like headlamp history, your downstream changes, and those are different from the plugin changes as they should be. Now, the this is these are the pros. You have a clear separation of history. Headlamp rebases should be easy because you now you only don't know about any commits from the plugin, only the downstream commits you have on top of headlamp to change core

functionality. What are the cons? Well, if you have worked with get sub modules, I have opinions and and most people do about that. So, it's not it's not the most intuitive thing, at least not for me and I and I think I'm I know get fairly well. So, um Yeah, and the other thing is that it's not self-contained. So, you have Now you have two repos. Right?

Now you have two repos because you're not going to push your downstream changes that you do for headlamp, you're not going to push them to our repo of headlamp. You have to have a a repo that you are now pointing this sub module to. So, now you have two repos to maintain, your plugin and your headlamp And this is what I meant, so it's essentially the same

as in here, except the this is the the plugin repo at the in the sub dir. Sorry, in the sub tree. And now the plugin repo is one self-contained entity and now the headlamp clone is another one. And this is the Git history, like I was saying, you have a pointer to a to a to another history. Things are separated, but also related. So, this is kind

of ideal if it were not for submodules being a pain. Okay, but there's another thing that you can do with submodules. And this is actually the thing that that we end up doing with the Aegis desktop. That's That's how it ended up being done in there. which is you still use a submodule. And I said that there were two cons, right? One is that you get the

Git submodule uh pain. That won't go away, as you can see here in the spoiler. Uh and you have and you have the the problem that you have to rebase. But, there's nothing impeding you from having a submodule that points to the same repo, right? So, by the same repo, I'm talking about the same you know, remote, but different branches. So, what this essentially means is that

you can you can have your headlamp folder being a submodule that points to a branch in this own repo. And and this is nice because let's say that you want to uh add a a commit to your headlamp um uh fork again. So, you just enter that that folder, and you do the commit, and then you push it. It's going to push to the same uh repo,

but a different branch. And then all you have to do again is is um update the pointer in the in the plugin repo, but it's it's essentially the same So, this is what I'm talking about in terms of structure, right? You still have two clones. They are completely separate separate histories. It's a branch in the plugin repo, but it's it it has to be started. I didn't

mention it in the previous slide, but it It to be started as a as just a a branch you pull from another repo or an orphan branch and then you you apply the the headlamp commits. Right? The point is you have two you have two branches with completely different histories, right? And you still point one to the other. And the and the and the Git history is

essential it is exactly the same as we had in the in the submodule with two different repos. Hopefully this is not too um confusing. It sounds confusing more now than I practiced in my head before. but yeah, so what's what's remaining here? The the Git submodule pain. how do you work around the pain? You don't. You endure. And uh and you you can you can have some

commands. Of course, we um we can always scripts to to to help maintain the the modules like you know just update and then it does all the all the the things that the module needs to to be done. It's also annoying when push a change to to your uh headlamp fork and um and you need also to update the plugin uh history so it points to the

new thing, but that's kind of logical, right? That's kind of it's just what is expected. Every time you every time you do changes on the left uh history there, left tree, then of course you need a commit on the right one to point to the to that new commit. If you if you have been dealing with Git, you know exactly what I'm talking about. If you're not,

you're probably like super confused, but this is uh as easy as it gets. Um yeah, so this is just a table I put for comparison, not for I'm not going to repeat everything. so which approach is best? Out of these that I mentioned, rocks your boat. So, um you know, if you if you prefer to have a subtree and it's just one commit and you are able

to manage that, you don't want to touch submodules ever in your life, your go-to approach. If you just want to have a lean fork like, "Okay, I have all the headline commits and the in my plugin commits, everything in one in one tree." Go ahead. If you want to use the the submodule uh like EKS Desktop is doing, that's uh that's probably also it's it's something where

I lean to, uh but it's it's also an approach. But, in the end, it's like I'm showing you ideas, right? Maybe there are others that that I'm missing here, but I'm showing you ideas that you can um basically create your own Kubernetes uh um GUI app uh based on headlamp and hopefully with with with the least burden on maintenance. So, uh fo- fork fork's best practices, yeah,

when you're when you're main- maintaining a fork, there are a few uh things that you that you have to care about. This is the pres- presentation is not going to go deep into that. That's a whole presentation, I think. But, it's like I'm talking about stuff like if if in in the in the models that were not the headline sorry, the the approaches. If you do a

change that changes your plugin code and at the same time changes the headlamp code, and now you commit, now you're touching two different things. And next time you rebase, you're going to have a commit that touches those two different things. And the chances are that you have a conflict. And now your plugin code, which is not the culprit of the of the conflict, uh will have is

is is is in the same uh plugin sorry in the same changes table that you have to to handle. So so the best thing you can do for for for saving your time is to have atomic commit. So these are commits that only care about one thing. So in this case I mean the criteria of what one thing is is debatable, but if you have a fork

and you have your own source don't don't put commits that touch all the places, right? Um now the the the last commits you have in your fork like I said the the least burden on you for for maintaining it. Just offload that to us as maintainers. How do you keep track of those because it's very easy that you don't remember which which um changes you you thought

would be nice to send to the upstream folks that you did like 6 months ago. Just put a a label on the on the commits and then you know which ones you want to to push upstream or send upstream. Uh and the other thing is that I said you don't remember what you did 6 months ago. It's much better if you don't remember what you did 3

months ago and that means that if you rebase earlier and often then you're going to it's going to make your life easier. Right? And uh obvious I guess from the previous things, but just send us the changes uh as as as you know as frequently as possible and as close to what we expect as a project as possible, right? So so we we are able to move

fast. I have a whole article I wrote about my opinions on this if you want to check it out with more stuff. And that's essentially it. happy hacking on Headlamp. And if you have questions, I'm here.