"Will it Kubernetes?" A Tinkerer’s Journey From Curiosity To Career - Niklas Frick

In this talk, Nicholas shares his journey of discovering Docker and Kubernetes, detailing his initial struggles and eventual success with setting up a home lab environment. He describes how he started experimenting with Docker images after a night of deep exploration into the technology, which led him to build a clustered infrastructure using K3S. Nicholas candidly discusses the real-world challenges he faced, such as networking issues and managing certificates, as well as the learning curve associated with Kubernetes components like Helm and Longhorn for data persistence. He emphasizes the differences between working in a home lab versus a production environment, highlighting the skills and mindsets that translate into professional roles, particularly for aspiring platform engineers. The talk serves as both an introduction to Kubernetes for beginners and encouragement for seasoned engineers to further their exploration.

All right. So, picture this. It was a summer of 2023, late summer evening. I sit down at my laptop on evening and I've been hearing about this thing called uh Docker and containers for quite some time. Uh it popped up in my previous work often and it was always a question of like where does this thing run actually. Um, I heard that you can package applications somehow

magically together and uh run them on different kinds of systems across environments um and it should work anyways. So I think all right I just give it a go see what it's about. I try it for an hour. So of course I start to watch some YouTube videos. uh fire ships docker in 100 seconds or network chucks you need to learn docker right now that was actually

actually eventually the moment I started to dig down a rabbit hole first it was just a basic concept I got used to it what what it's all about but then I realized docker networking for example is not that easy as I thought it was and also um I then dabble more deeply into it and find myself building my own Docker images and really getting started. At the

end, I just briefly touched on the topic of um I could use containers in a more distributed way in a way for much larger infrastructures and deploy them um much more interconnected. And that's where I've first heard about Kubernetes. I look up and it's 4:00 a.m. in the morning. So, it's been all night and uh the better part of night I've spent just down that rabbit hole

and I didn't really re even realize how how late it was. That was the moment I I realized something had woken up inside of me that I didn't even know was sleeping. So, hello everyone. My name is Nicholas. I am from Likenstein which is a very tiny country in Europe. It's a micro state. And for the ones that didn't notice, that isn't Likenstein. That's Luxembourg. This is

Likenstein. It's located between Switzerland and Austria. That left picture is actually from my passport. So, they have to zoom in a little bit to see where Likenstein actually is and what the country looks like because normal webs don't really show it. On the right side, you can see some real pictures of the country. So who am I? My background is in IT systems engineering um at the

bank in Likenstein as well. I um went I did an apprenticeship when I was at the age of 15 until 19 just four years. Uh later I got a degree in business digital business management which then eventually led me into the work as a solutions architect and a product manager uh where I would uh develop uh digital products and uh also build some startups with that. Exactly.

Um I nearly spent a decade in building digital products and I always had a little still a little foot in the infrastructure side because of my background. uh but uh I kind of move moved away from from it more into software engineering and less infrastructure. It was until that summer night when I couldn't really stop. So will it Kubernetes my all nighter I just stopped with um

herring for of Kubernetes for the first time and uh I just touched on the overall concept on it. uh was intrigued actually but I didn't understand anything at all. So uh I thought that would be something worthwhile looking deeper into. So I picked up where I left off. Um, and I thought about for me the concept of Kubernetes was just that that big system of connected computers

that would do work. And I thought of all the old hardware I had laying around in my basement. Um, and thought to myself, will it Kubernetes? So, yeah, I used it as a verb because I didn't know any lingo at all at the moment. So in my in my um in my thinking of about Kubernetes was just something you would do rather than a system you call

it that way. let me introduce you to the pile. Uh these were all the notes I selected for my first cluster. U not realizing that way at that moment that you call it a bare metal cluster and you can run it differently. for me was just a distributed system of hardware. Um to the left we have uh an old workstation PC from my time at the apprenticeship

at the bank. Uh then we have my old MacBook then a friend's old MacBook Air and uh on the bottom right we have a old uh lab machine also from the the days back uh at my apprenticeship which I thought was broken but it turned out it was just the screen. So the the the computer itself worked perfectly fine. So it became a note. On the top

right we have a Raspberry Pi uh which I used at for a IoT project at uni which I thought why not put it also in the bunch. And the detail I really like about this is that some or most of them are um in or hardware that I had back from my apprenticeship days when I was doing system engineerings and uh now it's kind of a full

circle moment to like use them and uh bringing me back to infrastructure. Then the problem started to arise which first one was Ethernet cables, patch cables. Uh they were really sparse in my apartment. So I had to run around and find uh cables to connect them together. Uh I unplug uh my Apple TV for instance to get the last one. And then I thought I connected everything

together. The both MacBooks didn't have any Ethernet um slot. So I thought I just connect them via Wi-Fi. Uh, with the MacBook Air it worked, but with my MacBook Pro it didn't. I I don't know why. Still to this day, it's some some maybe it's a hardware defect, but I thought I need a an adapter to like connect it via Ethernet as well. So, I consulted my

group chats with my friends and asked if anybody has a USBC to Ethernet adapter and one did. So he had a docking station that would eventually work and I could connect my MacBook Pro and hook everything So so far this project was just an infrastructure project and I was thinking this is not exactly what I thought of. So where was I starting from at that point? I

want to re be really clear I had no idea about Kubernetes at all. uh I understood servers and uh networking on a physical layer uh for my background but um when it came to Kubernetes I had I had no idea what ETCD for example was and all the other components that uh make up Kubernetes but uh yeah and my mental model was basically was a computers that

work just together. So my idea of it was it would automatically do anything you want to throw at it like scaling and high availability. Also um I expected workloads and compute would distribute evenly across the nodes. So I deploy something and it splits apart the compute power so I can like really use the um all the compute power I have um available to me. But then I

pretty soon realized that wasn't the case. Uh at that point I had the cluster or like the infrastructure ready. I set everything up freshly with Ubuntu Linux. And now comes the part that I call failing forward because that is the path to then eventually a working cluster. So there were many obstacles along the way and I give you just one example that show highlights a little bit

uh where I was coming from and what was there to learn. Um I used K3S as a as a Kubernetes distribution for that project because I was pretty much the first thing that came up when I searched for it. So I chose it and um I set up my nodes. I installed K3S seems to be working especially exceptionally uh on the Raspberry Pi didn't unfortunately. So I

thought yeah there must be something with it wrong maybe a hardware defect start to search around. I Google and check GPT for any combination of Raspberry Pi not starting K3S and um unfortunately I went down the wrong rabbit hole at the moment debugging the problem. I thought it was the mismatch of like the ARM architecture versus the x86 architecture which was not the case but it turned

out the problem was there were cgroups disabled on the Raspberry Pi by default. So something in the back of my head felt familiar from from my apprenticeship days. Um had no idea anymore what croups were. So of course you then start to search what it is and the fix was a oneliner in a in in a config file. You just enable the croups for memory and all

of a sudden the Raspberry Pi were also running K3S. So the lesson here is like the problem isn't even on a higher level where you search from. Um the lesson is sometimes way way deeper underneath in infrastructure and uh yeah for me eventually it would lead to learn about croups which is of course very useful later on when you work with Kubernetes more intensively. Then my biggest

headache were certificates and tokens at that time. Uh I had also no idea what to use what for and as well here I went down the little bit the wrong rabbit hole as well and uh swapped them like I took tokens for certificates and vice versa. uh turning out not being able to join the nodes cor together correctly because at the time when I was looking at

the at um example how to like build a K3S cluster uh the guy who wrote the documentation did everything by hand also generating the tokens and the certificates but in the end I just deleted everything again and did nothing basically um and so it would generate the tokens and and the certificate themselves. So it's was way easier that that uh at that point. So it was really

frustrate frustrating at that moment. It wasn't that it was hard just once you find out how easy something could be um yeah you want to throw something somewhere around. So then what's when uh was starting to work more with the with Kubernetes to run a cluster um was when I had a little mindset shift of the one many people here which is uh treated as kettle not

pets and uh that was kind of a mindset shift for me with a background as in systems engineering and network. So something that is broken I wanted to fix immediately instead of um yeah being so so generous and throwing it away and trying to to just rebuild it. Uh that was the whole concept of it that was very new to me but um yeah fundamentally changed how

the way I looked at it and when I first realized what it's all about actually so I have a running cluster now at that moment so I feel like nice everything is green everything is running um looks cool I don't know Um, I just sat there, had no idea what to put on the cluster because I didn't know what what to run there. So, of course, I

Googled for it. Um, looked for Kubernetes examples and then I found uh the Kubernetes IO documentation and they had a simple startup uh example app and deployed it step by step and it it ran. I thought, "Wow, cool. I understand nothing at all." but it runs. Um, yeah, but shortly enough, of course, it wasn't enough anymore and I wanted more. I became I came I went down

another rabbit hole which is called self-hosting. So, the example app of a guestbook app uh wasn't really appealing to me anymore. I wanted to do something I would really use and that was a a w a wiki app. It's called outline wiki. At the moment there were only docker images available, no hm charts or anything. So um yeah I just started wanted to run it on my

cluster and immediately knew I don't know anything. I need way more um information how to do it. So for example, I want to persist my data, right? I would I want to write um my wiki put put information in. So I need to store it somewhere. I found out that Kubernetes isn't really made that way. you need something um for example Longhorn uh which was my first

um u Kubernetes storage solution I I touched on because um it's one of the easier ones to get started with uh I learn about it and install it and it works so I have data persistence then with learning about Longhorn I see um there's something called Helmcharts which is nice. At the up until the end moment I just thought everything is cubectl apply but uh I learned

about how you can bundle together standardize your deployments. So that was the next part I was was new to me and really helpful at that time. So the app needed the database and caching as well. So I hear about Postgress and radius. I tried to deploy them and eventually succeed. So next part done and then it also needs S3 storage. At the moment I had no idea

what S3 storage was. So I just Google for it how to run it. I find minio and install it on the cluster as well because now I have al also um persistence. So that works well. I want to exit from external externally of the cluster and that was really a concept I was not familiar with. I just thought I installed it on it and open the page

and um I can access it. But that wasn't the case. So I need something called ingress. So I installed traffic. And then there I had the notification that the page wasn't safe when I open it. So I want HTTPS. I learn about SER manager and let's encrypt and also that you can integrate it with uh DNS providers. So if you're in a home lab environment and it's

not public publicly accessible, uh you can automate it via DNS01 challenge. So which was nice. So I hit HTTPS. So what about looking at what what runs underneath or how it works like what are there any errors are I how much RAM is it consuming for example. So I search for how do I monitor that? I find Prometheus of course and Grafana to visualize. I felt like

that was really tedious work. You have to install everything by hand um like write everything in a CLI. So I searched how can I automatically deploy this? So I found Argo CD at the time and didn't even realize that I was doing GitHubs already. So this is the moment where I would want to highlight the community which is what actually made it possible for me to run

my home lab and um I I was physically alone doing this but I never really felt alone because there's so many channels you can reach out many people's out there putting their work in open source so something like this is possible and you guys are cubecon already so I guess you're also already part of it which is nice and uh yeah so okay now I have my

home lab but I want to it sparked something in me and I knew I wanted to dig in deeper I want to make this my career so the differences what does transfer from your home app to production how I call it like to your professional work so the basic core principles of Kubernetes of course um pod service, Singress, all that does really well apply into any environment.

The mental model of it, um it's a declarative uh mindset. So you store everything in code which is across environments the same. Um and of course very important your troubleshooting instinct. You learn the techniques how to freely find a problem, dig it deeper and and eventually fix it and learn from it. and what tools you use like the tool stack in is there's so many different things

of course um not all apply as well but you really use the tools that that the big uh companies and and environments also use which directly um yeah applies to production as well. So, it's really the hands-on skills and the engineering part that transforms. But there's lots of things different in production in in the in a professional environment. For example, better architecture planning. So you um need

to really be careful be before you start digging in the keyboard like what what to choose um what tool stack to choose what sizing of environments make sense because you always have to have scalability in your back in the back of your head not only from a technical standpoint but also from a operational standpoint. Also, of course, security. In a home lab, you usually don't have much

security at all to make your life as easy as possible. But in production, in in critical environments, of course, that's that's a key point. Then, uh, multi-tenency is also something usually at home you have maybe one maybe two clusters. Um, and it's just yourself on that cluster. in production there can be um many many different instances for for different teams for different uh clients on the same

infrastructure. So that's something that that you probably only learn in the Then uh of course reliability at home if it breaks it breaks. Um your your cell phone step is down. You try try to figure out uh or you can't watch your favorite movies. But uh in production that's pretty different. So uh in production you uh you start losing money when something's not up for example. So

that blast radius is way way bigger. Uh in production you have observability like in your home lab as well but alerting maybe you have it in your home lab as well but there stuff is firing and you don't care. Uh in production hopefully that is not not the case. So um alert fatigue didn't kick in there. And uh after alert like with alerting also then of course

things like on call 24/7 availability uh comes too. Day2 operations are completely different in the home lab and in uh production. In the home lab it's fun to like upgrade something. In production you usually have maintenance windows fallback options and uh yeah it's just way more more complicated than at home. And also cost in the end is a factor in the home lab. Maybe your electricity bill

is a problem. Uh in the production much more things like cloud um infrastructure provisioned comes um like adds to cost and you need to really be u aware of that and optimize for that as well. So basically at home you're a generalist so you're doing pretty much everything you can like everything on a very top level some things deeper some things less um in production you're a

platform builder at home it's shiny newest cutting edge technology newest versions nice production you want more stability so less less fancy new tools you always think about what to use. At home, you have toy, high availability. You think three nodes, nice. It's it's highly available. But in in uh production, you have different environments, different availability zones. Um you need to for example know what's what traffic redirection

between multiple locations uh works and how the application uh reacts to that. Yeah. So, as much as I want to say it's it completely applies from home lab to your professional environment, uh I wouldn't even say it's a 8020. I would think it's more 6040. So, you learn the basics, you learn your um troubleshooting skills in at the home lab, all your foundational, but uh people, processes,

consequences, regulations, you only learn in production, in real environments. So how did I get from the home lab to uh a job as a platform engineer actually? So in the first job interviews I thought uh something changed because uh now every question about Kubernetes I could answer was no problem. Um not not some theoretical knowledge but something I really had done. So um that that's actually the

difference what it made for me to like when it came down to technical questions I could answer them as well. Um because you can't really fake something you have broken uh and you fixed it when you have done it. So that was the key difference for me. So maybe some points that job interviews um interviews will probe you for. So definitely scalability is a point. Um at

home you have less nodes than in production usually um small environment versus a big one. uh you need to yeah be honest and say um I haven't operated at that scale but here's what I expect uh what happens at that um scale team and processes of course um at home you're solo you're alone uh and in production you work with many stakeholders together so that way you

um navigate more to I'm sorry. Um yeah, you have to really be mindful about who you're working together and uh handling change management and also push back from from um others when you decide to do something. So um also show your incident mindset. um when something happens you need to prove um how you would go ahead how you would try to solve it and uh one point

that's really important is uh the platform as a product part at home you are um the only one using the platform you're deploying everything and this is probably the biggest gap to production um you're building IDPS golden paths and developer portals for others to self-s serve themselves and you're not the users on because but uh the users are your developers using the platform and that's something really

So some tips for interviews. Uh be specific. Say exactly what what you're doing and uh go into detail uh what you did um about your key learnings from the time. Um obviously with why. um say why you're doing this and for example why you chose a specific uh product over the other and um yeah which which brings more detail into into your reasoning also bridge to production

u say I learned in the smaller scale at home something like that uh which taught me that and eventually in product I would do it differently because of that then also own your gaps there are gaps of course especially the the scalability part. So you just say you haven't managed that 200 nodes before, but um here's what I understand how I would expect it to work in

production. So this brings me to where I'm I am now. Um less than three years ago, I didn't know what a cluster was. Now I'm standing here at CubeCon, which is still a bit surreal for me. And um yeah, I work as a platform engineer uh together with some amazing people I can learn from every day, which is really nice. And I always al always dabble into

new things that I hear across uh during my professional um surroundings. So for example, AI workloads and Kubernetes as well. And of course, I still have my home lab. It looks a little different here on the right side than before. Uh, I got still a little bit messy, but less so than in the beginning. Still my playground for for everything I want to try out. And it's

not as uncritical anymore as it used to be. I run some websites as well on it and also some internal tools I really would hate to fix. So I I yeah, my goal is to keep it up as as long as possible. So now it's your turn. Uh, yeah. If you're just starting out, everything you need is actually just one laptop installed K3S or any other distribution

on it. Um, deploy something you really want to use because that's um, yeah, that makes you want to keep up the cluster and running. You learn a lot along the way. Join communities like you do already. Break things on purpose. That's where you learn and brace yourself for production. For the seniors in your in here in the room, uh maybe some early feelings and frustrations feel familiar,

but I would really encourage you to start dabbling into anything. Build something stupid stupid at home. Uh try it out. Also reserve the time for it. Uh not just to build a new CV skill or whatever, just to try it out. You never know where it leads you. So old stations from the work at the bank before uh some friends not books my own notebook yes it's

Kubernetes it did it was not about just will it work but something a question about me uh will I be willing to like put in the work will I be willing to following my intention um if something feels appealing would I go down the route so the answer was yes for me and the only question I have left is will you Kubernetes 2. Thank you. I have

one last thing. Actually, it's three last things. Uh I haven't uploaded the slides yet. Maybe it's in the process now. If not, here's uh the link to SKT. Would be also nice if you leave feedback there and if you want to connect down there. And the second one is I would really like to take a selfie with you guys because it's my first talk at CubeCon. I

hope that's okay for you. If you don't want to be on the picture, just cover And my first uh deployment was the cube cube um the example guest book and I thought it would be fun to do something different and also do the guest book for you guys what was attending my speech. So I have deployed something on my cluster running at home. It's called guest pods

and it would be really nice if you could enter your name there, where you're from and what's your home label. So you can take a picture right now or start entering. And the countries will be grouped into name spaces and your home lab level will be uh shown as the emoji. This will serve as a yes as a memory for me. So thank you guys.