Mario Fahlandt: Cloud Native Security: A New Paradigm for a New World

In this talk, Mario Fallend discusses cloud native security and its transformation in a rapidly evolving technical landscape. He highlights the limitations of traditional security approaches such as perimeter-centric defenses, manual processes, and the siloed interactions between development, operations, and security teams. By emphasizing the need for a zero-trust model, he argues for the integration of security practices into the development lifecycle through automated tools and processes. Fallend introduces concepts like Shift Left, which encourages early security testing in development, and stresses the importance of observability and automation for maintaining security in dynamic environments like Kubernetes. He also touches upon the necessity of collaboration among various roles in tech teams to create a cohesive security strategy.

[Music] ladies and Gentlemen please welcome our next speaker Mario falland presenting the topic Cloud native security a new paradigm for a new world so hi everyone how was lunch that's that that's that's the right answer so um now something light security so that the easy talk after lunch um I I I try to make it as entertaining as possible uh the first question is uh who am

I hi I'm Mario I live in northern Bavaria in Germany I work for a company called kuber medic and uh I'm also part of the kubernetes project itself there I'm the tech lead communications and I'm also part of the last week's in last week's incubes development newsletter hard hard word but uh it's a website you can check it out there you can basically see which pulus got

into kkp uh to cators and so on and so on I'm also a Google developer expert for cloud and I'm an arm arm Ambassador uh quickly what kuber does kuber has three open core products we run kubernetes inside of kubernetes we have a multi- l bancer we have a c kubernetes life cycle management tool and we do Consulting so who of you is part of a security

team four and a half who of you is a developer that's more that's more and who of you is an operations person that need to run stuff in cloud or on Prem okay so we have a mixed crowd the reason why I'm asking is you all don't talk to other most of the time because as we let's let's go back let's go back in history or like

10 years ago 15 years ago and we have like security team development team and the idiot that needs to run the stuff usually that was me um and the security team came up with great Concepts there are three great Concepts that we can talk about and the first one is we buil paramedic Centric defenses so we build a stone wall out of our in around our infrastructure

then it's safe because nothing can get inside as we know um we have a manual process if something goes bad then someone needs to take care of this and this will then be discussed in the meeting and then we slowly move forward and as uh as I told you the silo approach development I don't care where it runs on my PC um security like there was some

government letter that said we you need to do stuff like this way and you can't move away from this and that's how we do security and you need to do this and otherwise we don't allow it to you and the Ops guy is like I have servers I don't care so let's start with parameter security basically parametric security is exactly this I can I have my my

wall around my my my world and um everything is fine if I'm inside the world uh this also was uh the CIA thought also the same way in I think 2011 um and then someone dropped the USB stick on the parking lot and this USB stick was picked up by an employee and what is the best thing to do with an with an USB stick that you

find in the parking lot in front of your company yes you plug it into a PC inside of the network and see what's on the USB stick this was a bad idea because then you open basically the back door so if you go back to our Castle analogy it's like yeah we dig a tunnel uh underneath uh underneath the wall and then just got out in in

the inside world so the problem is that we all know best movie in the world hackers was released somewhere 1994 or something like this amazing movie so we know hackers are um skateboarding and going to telephone booths and connecting to telephone booths no they are not um hacking has changed of course because they are adopt as well and uh hacking is targeting everyone of you I mean

you see this basically daily like you get an email in the emails said like hey I'm George I'm the new colleague from the new office in yada y y can you please send me this information can you please send me that information so we move we have the problem that we the the angle where we we are getting attacked is the people so we attack layer eight

of our of our world and they try to get into your into your system by yeah people engineering so um social engineering this is the way how the most attacks are being carried out because Brute Force attacks there are services for this I can just root them array but the problem is every employee so we have different layers where our security needs to be so we have

we need to have like a layer that protects the employees and this is usually done by the thing that everyone of you really like to do when in in your company when you got onboarded and you have to do it every year is the uh yeah ask this qu uh answer this questions what would you do don't leave your work laptop unchecked and so on and yeah

then you have a Meetup in your company and um the pizza delivery guy enters the building because he's bringing pizza for the Meetup at 8 uh at 10:00 p.m. and people don't care and the pizza guy is just walking through your offices and yeah who's who's checking this person probably no one also we have the fancy things like you get a key card and you get a

key card to get into the into the system but those key cards can be lost and replicated and cracked so people can get in this world we have network security I mean when we we are already we have already neglected two layers of our security so when we go to now to network security it's like yeah we have everything protected to the outside world but not inside

so we have the problem that inside I have big networks and I can just do what whatever I want and the next step that I basically need to do is I need to take care about endpoint security I need to take care about applications and the the deeper I get the the more critical data I can I can basically I can basically grab so this is our

old world thing and the next thing that we have is manual processes who knows what a cve is that's not much so security is a manual process so something uh a security risk was found needs to be fixed so you need to fix it and you need to uh to fix the problems that that occur there and uh so we need to patch our systems we need

to update our Docker images we need to uh reschedule our containers that they are placed everywhere and this takes time and effort and this is usually manual and this is bad so when it comes to cves cves are common vulnerability vulnerability and exposure issues so they it's a closer that defines basically vulnerabilities in software currently we have 233,000 CVS were filled last year which is 5% more

than the year before which is 24 24% more than we had in 2021 and when we look at the graph they exploded by a lot and the problem that we have and this is basically a really really really new information for you this happened 24 hours ago so 24 hours ago um in January uh no February sorry I'm sorry February the um kernel.org so the organization for

the Linux kernel was added as a numbering Authority so the kernel.org can now file cves so they can create a new cve number create a new cve based on issues the problem is that they are now inflating the system so they opened up 700 cves in seven in the last seven days and they did it retro uh retroactive so every Buck fix that came into the Linux

kernel in the last years is now declarated as a cve which is utterly nonsense because sometimes it's all just a typo that was fixed or something like this and um this is a real interesting discussion because usually everyone in security says like okay a cve is released and I need to check my software if I'm affected by it for example there was a cve regarding that affected

container D container D everyone knows what container D is container D is the runtime uh is a container runtime that is the current default for kubernetes it's basically how containers are run uh in kubernetes and what we and the cves basically said that yeah there was a there was a bug that affect that could that's always the important thing that could affect and could be a security

issue in your system so we basically said that okay we need to fix this how did the fix work the kubernetes cont the uh container D contributors fixed the issue then 8 hours later the buck fix was released for the kubernetes repository and uh 7 Days Later the flat car image was updated in between Ubunto updated and so on and so on so when a cve is

published and released everyone is jumping on the oh train and we need to fix the stuff and this is basically what they what they did um and the problem now that we are inflating the system we are basically creating the main problem that I can't trust cves anymore more because I don't know if I need to fix this or if I have already fixed it like three

three years ago and this is a real problem because we are losing our way of in identifying problems so and the last point that we had in the old word thing is The Silo approach I already talked about this um in the past it was we or it's not the past it's the current uh still the current we have container we have our security teams we have

our Ops teams we have our uh development teams even development teams beneath each other don't talk with each other because they don't care I don't care what what front end does uh because I'm back end and so on and so on and um this Silo approach limits um the way how we can Implement security because security cannot be put on top of each of the steps security

needs to be throughout the system we fixed this a couple of years ago somewhat because we said like use Docker use containers because containers will fix everything it's uh we make we make uh your computer shippable the problem with this is that um they still rely on the underlying platform they still rely on uh the underlying uh systems and uh not every environment is the same environment

so that's a problem when I started and this is not so long ago hopefully so 2004 2005 um every server that we set up and this is probably you heard uh many times again was a pet because my database server had a name I identified my database server and uh in the cloud native world we move away from naming things because we don't treat them anymore as

pets because we don't care if they die the the goal is we want to have systems that can go away and that I don't need to care about anymore and we will get to this point later in time again so another example that we had in or another example that old security teams usually have is ports everyone knows ports right what is running on Port uh 2020

exactly s what is Port 25 smcp right everyone knows this this is this is a fact so I know a port is something where I can rely on the problem is there's for example if I want to expose a service via not ports and kubernetes we have a port range that we open uh the port range is 30,000 to 30,2 uh 32,000 something something and the security

team and this is not made up I had three different companies came to me and said like we can't open two 2,000 ports that's a risk no it's not it it's not it's like we don't use those ports these are the kubernetes default ports and you can you can you can reduce them so they asked us okay can we reduce it to 200 open ports okay we

reduce it to 200 open ports 30,000 30, 1999 guess what they deployed 205 what was not available five Services because no ports anymore because they don't understand that the concept in the new world is ports are still a part we still need ports but they are not like a specific thing and um I don't know if anyone saw the the talk of upd it's like upd short

an application that randomly created a port or used a random Crea a port to run the application because they don't care ports are aeral it's not um they get random assignment and uh I don't need to have ad as a web server because it doesn't matter anymore um micros servers and API is talk to each other inside of the cluster based on their inner cluster I uh

um IPS and um they talk to each other B based on the on the mapping inside of the inside of the inside of the cluster so the port is just like another identification number but it's not a fixed thing where I can say like this application goes to this uh server so I can open up this port there this doesn't make sense anymore because the problem that

we have is even if I block a port um or if I have an open port I don't can guarantee I can't guarantee that not a security issue is coming in there so basically what we move to is a zero trust we don't care anymore where uh what is the port where it's running we care who wants to access what so we need to identify all of

the different parts also another thing is everything needs to be encrypted so we need any traffic that goes anywhere need to be so what we do is we have in the cloud native world we have identity access manage me which means that we need to identify and manage all of the uh the things that that go on we need to have a behavior analyst which means that

if we have like a service that constantly talks to another service I know what is going on there I know the behavior of this and it's uh I can identify a security Risk by seeing what is if there's a change in the in the traffic in the traffic patterns or if there's uh a strange be behavior in it so this is something that we need to implement

for uh watch increasing security also networks in the past we had like this huge networks but the problem is we you don't need a huge Network I know what who needs to talk to whom for example if I have kubernetes nodes and I put them um a kubernetes node don't need uh uh don't need to talk to the other nodes they only need to talk to their

control plane which means that we only need to have um what is called connectivity for example in kubernetes uh which connects the worker node to the control plane and that's the only thing that's a tunnel onetoone connection and they don't know which other servers are in the network because they don't care it's not their it's not their um their job to care or to see anything else

in the network so what we do is basically we we take every environment gets its own network separation based on its criticality so we have more smaller networks that are just there to host nodes and everything else is managed inside of clusters with network separate uh Network policies and clusters and so on and so on and so on also we have Central gateways where traffic needs to

go through which is uh API gateways where we can say like I can say where do we need to go where so that we can basically uh see that who what's the what's the goal where we want to go so it's all we still have ports but they are not as important any anymore and all of this come together that's and now I'm boring you basically this

is all called devops it's called has been called many things devops is around since forever um but it the idea is basically that we want to have a world where each and everyone talks uh where where each and everyone talks to each other it's not that this is not a job role because if anyone says to you that deaths is a job that's a lie because this

is a concept of different people in a team that are working together so I have my developers and my de gu my my Ops people working together in a constant stream that everything is uh always repeating each other and this is basically how devops was born and this was 2003 so this is old news um the next thing that we had is clusters are becoming cattle which

means that clusters are no longer as we had with our uh VMS back then and our database servers we don't care about clusters anymore because we can easily exchange clusters um and they can die and we need to move stuff from one to each other so how do we ensure this whole thing so how we make sure that this is actually working there are three uh three

constants that we need to uh to adopt in Cloud native world so we always need to have automation everything that I do needs to be automated in any way why because I can leave the company next week and I was the only one that did this who who will Who will uh my my computer my bash history good luck getting this so um this needs to be

anywhere in any envir uh in a in a stable environment that I can that can change immutability means I have my whole infrastructure in a repository and no one is touching a server anymore why because you can't track what has changed inside of a uh inside of a manual change so this means that changes to your infrastructure are pull requests because people can review it people can

double check it and um you can rever it and you have a history you can know why why something was changed and what we also need is we need every time 247 we need to observe how our infrastructure is behaving the good thing is observability can also be done with automation so that's why everything basically ties in together this brings us to a next ugly buzz word

that everyone probably hear it's shift left what is shift left shift left means we move security more left in those death Ops thingy thing why so that we have different topics in the uh or that we have different parts in our environment where we can basically Implement different matters who knows or who is using sust in in their company three people four people five people that's good

who's using dust two that's not so much um so to explain a little bit what what sust and dust means it's basically Su is a white box testing white box testing means I actively test my code or my to my code base is being actively tested because the program knows the code base and can look into it and see like okay this package has a vulnerability this

package needs to be patched and so on and so on so this is basically like we Implement security uh with closing CV or patching infected versions early on Dust is a little bit more complicated because dust is a blackbox testing dust means that we need to test uh or we need we test end to end so we we know we we see if the application or application

stack is behaving normal and if there's any risk that goes into this um this makes scanning of course way more complicated but this is on the development level so this is something where the developers need to take care of and not the Ops people because the Ops people don't care what code is inside of their container because they don't know in the worst case I I don't

know what if if nodejs needs a package update or not also we talked about containers people you you know the first line in the container is usually from Docker blah blop the problem is from blah blop means I trust that the that the image that I have above is patched and I have like all of the stuff is updated and everyone and someone car uh guess what

no nobody cares there are tools that automated I mean you can have uh security scanners that check uh for vulnerabilities and scan your container images but this also needs to be done so every time you publish something or you pack something in a container this needs to be this needs to be checked and the last thing that we have compliance we are in Europe and in Europe

it's always great because we get a lot of new uh laws and the new new guidelines that we need to apply and uh those compliancy scans if anything critical is in there need to basically go into uh into this as well so this is something something that is really critical that people are taking care of uh compliance like gdpr non gdpr violations for data handling and so

on and so on so um this is also part of your security process because no one of you wants to be uh the person that said like yeah we accidentally leaked two million customer data it happens so what what go we what's the goal to to build the the architecture around it so we basically what we aim is we we we stated that we had all of

the security tool sets and we we build our infrastructure so the goal that we want to achieve is basically we don't trust anyone I don't trust you I don't trust my uh my employer I don't trust the guy that built an open source package I don't trust any Docker image so what we uh and foremost is I don't trust my application itself because we are all living

in this microservice world and microservices are now going back to monoliths because AWS published a paper uh that they moved to monoliths and now everything is great again in monoliths um we will see about this uh but the problem is I need to make sure that only the specific parts of my application stack can talk to the other to the very application that it needs to talk

to for example um a friend of mine runs a and they accidentally published to the um to the production database instead of the development database so they did the migration on the production not on the development that's not so great and this could have been fixed if you would say like my upgrading Ser or my my migration service can only talk to my development if I deploy

into development and not accidentally to to the database it would have been even better if you just separate the whole environment but that's another topic that's basically the same for micro segmentation so if anything goes to nuts then it's only a small part that is exploding so our new parameter when we go back to our first slide with the castle um is identity so I basically go

ahead and say like everything needs to authenticate against each other even if I am in inside of a cluster so we need to have a strong parameter with uh identification we have to uh I need always to implement myself that I go ahead and only talk to a service where I know that I'm allowed to talk to kubernetes let's go now into the whole platform thing so

basically in cuetes we need uh we have containers and containers as I said need need to be scanned so there are a couple of tools as we go back to automation because I don't want to do this by myself um that does this for you the problem with this is I they are commercial tools but uh I am not a fan of commercial tools because I do

mostly open source that's the reason why I use open source tools so some of those tools basically go ahead and um give you options to automatically run inside of your build process uh to find vulnerabilities to figure out they uh Implement into your um Registries that you can just run with in inside of your Registries in your automated deployment uh process uh and that take care of

this another point that I didn't add there is when you run Q in qes clusters how old is your longest running pot exactly hundreds of days what's the problem when a pot is 100 days old exactly and the problem is that we have um inside of this image there are vulnerabilities that are more than 100 old maybe the image is already updated because our system already took

care of it but the pot is not restarted in the in the ques cluster so this is also this is um I saw it a lot that we have they had everything in place they updated everything but they didn't update their their life cycle inside of the cluster and uh because they said like yeah but then the service is breaking because we can't restart this pot that's

not how kubernetes Works kubernetes uh the the idea is everything and then we go back to Kettle everything can die at any given point in time because it doesn't matter because we have fallback mechanisms to reconcile this that's the idea behind kubernetes it if something fails even down to a node level another node can take over and uh we have are even at the point with cluster

autoscaler and so on and so on that we can just provide new notes where the stuff is going on top of this so inside of the kubernetes cluster we also need to double check everything we also need to um have a check that helps us to find those things and for example they are Cube they are tools with for example FAL Cod tells you and says it's

like hey um this container runs more than 30 days restart even cueno has a function that basically warns you who uh cueno knows okay uh cuan is a policy management tool for kubernetes um it's also open source of course and uh you can basically Implement policies to say like hey you can only pull from this registry you uh only traffic can go from A to B um

only uh I can only use images that uh are listed in this specific thing um only pots uh pots can only run for maximum of 30 days until they get rescheduled and so on and so on so basically uh Ceno helps you automate a lot of this stuff inside of and because AI everything is AI now uh there's another tool which is called KS GPT it's also

open source and KS GPT is basically uh either can run as a Ci or as an operator inside of your cluster and uh it's constantly checking your cluster and sending the data to an llm you can either use it uh you can either either use local LMS which I would prefer because I don't trust any of the rust no one let's get back to the rust no

one thing um so we don't want to send it out even though you can anonymize the data and it basically tells you what's wrong inside of your cluster and gives you an idea how to fix this which is nice and there are a lot of tools that can basically also help you automate all the stuff inside of the inside of the environment and this is what you

should always have to keep in mind that automation is the key because you won't get more resources to you won't hire more people but your infrastructure that's it just gets bigger um which brings us to the last Point um what we need and it this is observability um I won't go deep into observability because there are talks just around observability um it basically means that we need

an alltime deep insight into our environment at any given time because the more that we know about our environment the easier we can find a problem in our environment and basically say like how can I fix it the the goal there is also to centralize it and again AI can take a role there because it makes ugly locks human readable and with this the new title was

born which is death SE Ops and death Z Ops basically just enhances the death Ops thing into the new with the new parts that we just discussed so it's still this is multiple people this is not one person this should never be one person and we basically go ahead now and say like okay we have our layer of defense against any uh against any any Intruder already

on the code level where we have sust and dust we have penetration tests we have our code review principles that at least you have four eyes um we have the same on the on the op side so we have uh we we create threat models what could be a potential uh attack Vector on our system we have locks we have uh the uh we have an automated

patch system and so on and so on so this is basically the way how we should go forward and the main problem that we have and this is always a problem this is the problem that Humanity had since the beginning um this is the problem that I have at home it's collaboration and communication because people need to talk to each other and we have different people from

different areas and they need to constantly talk to each other because we need to have the um the de guys talking to the Ops guys talking to the security guys and the the the matter is not like saying no the idea is to stop uh to say like stop okay let's talk about what are solutions that work together and actually work and empty yeah we can't do

this because no because no is not an argument so we have basically devops def SEC Ops and platform engineering which in the end is everything together it's just like an enhancement of how we work together that's a collaboration and it basically it's key parts that just enhances the idea of Def Ops or s or how you ever want to call it and the good thing is this

is evolving like a lot um so uh the funny thing is we we uh we reached def SE mlops when we have all of this AI stuff in it um Triceratops we haven't yet reached so this is we we still need to get there but uh we will eventually and that's it questions anyone oh by the way we have already one question on top of here so

if you don't use the wop's description how do you describe distinguish between operation and compon internal process stuff and operational V stuff I mean it's this is debatable this is a debatable thing uh how you use those terms right um in the end it's um I have operations I have security and I have developers and these are the three roles and they should go into teams based

on where they operate together because they need to exchange each with each other what what is what is what is going on and um I mean the distinguishment is you you still have those specific roles right um uh and how you organize your teams is just an organization topic yes was this your okay then we just we do the other one yes but just I really trust

that someone's posting it anonymously all these questions yeah when it comes to the collaboration who should lead the conversation would happen in the um that's an interesting one the lead should be the topic of the person that created the requirement which means the requirement can come either internally externally or by a customer so basically we would go um a customer or uh uh we want to run

uh we need a department wants to run stuff inside of containers we don't have a container platform so we need kubernetes for example so we decided to go for kubernetes which means that the operations folks they uh uh go okay we need we go to the infrastructure folks and say like we need the hardware and we go to the um we go to the security folks and

we need the requirements for our security team so uh basically they would lead in the internal technical discussion but the overall process is led by the people that brought in the requirement and uh this maybe be a department they need to push okay these departments now need to work each in each other but I would basically go that Ops is the one that at least for the

platform part should go into the lead okay okay we have a question from the room uh thank you for the presentation uh if you told that the trust no one and uh everything should be encrypted uh what what is the gain versus the cost uh comparative if I have load balancer before my kubernetes cluster doing SSL offloading and inside cluster everything is open and every every service

connects to the other service through the inside cluster to the gate phas to the proxy Services compared to the everything is encrypted to with tens or hundreds of services inside the cluster I mean um the good thing is that you can also encryp traffic inside of a cluster yes Tech technically I can the question is what is the cost compared to gain oh uh that that's really

easy uh if you use for example s cni uh it's uh the the cost is neglectable because uh it's the encryption happens on the um on the EF layer or on the Kernel layer and uh the overhead is really small so there are tools that makes this way easier and they they just ship with encryption more or less so um this is basically what we tell most

of our customers if what cni to choose yeah go with syum because you have encryption already integrated that's it's it's like a yeah check mark done okay if you don't have any more questions thank you for this energetic uh presentation after the lunch I see that someone still uh somewhere in the lunch mood but okay thank you and big Applause to Mr Mario thank you thanks thank

you thank you very much