Thibault Lengagne: The Nightmare for Attackers: An Infrastructure without Secrets

This presentation discusses the concept of a zero credential infrastructure, which aims to eliminate long-term credentials to enhance cybersecurity. The speaker, Tibo, shares his experience in penetration testing and emphasizes the importance of effective secret management to mitigate attacks commonly associated with credential leakage. The talk elaborates on solutions like single sign-on (SSO), token-based authentication, and the automation of credential generation, all designed to reduce the administrator's workload while significantly lowering security risks. Tibo also highlights the benefits of short-lived credentials for reducing the likelihood of cyberattacks and improving compliance and auditability. Technical demonstrations illustrate how developers can implement these practices effectively.

[Music] hi I'd like to uh introduce UC today uh TBO who's going to be speaking about uh an infrastructure without secrets and uh a nightmare for attackers so uh tibo if you want to introduce yourself and uh take the stage good morning everybody uh so I'm tibo I'm a French engineer I've been pentesting companies for four years and after I've joined padok and I'm glad to be

here today should I start yes of course sorry yeah have full control okay okay so welcome everybody I hope uh you will like this presentation um I'd like to talk to uh zero credentials infrastructures it's a concept that we have developed at padok and it's it's working very efficiently to prevent attackers from being interesting in your in your infrastructure so just a quick word about uh about

padok um padok is part of theodo group it's a French company um we address all sorts of uh technological problems and at padok we focused on devops cloud and cyber security and I'm responsible for the services um okay so we will start with uh explaining what is a zero credential infrastructure and how to REM remove every long-term credentials from your infrastructure so we will start to developer

Secrets every secret that's used by humans cicd Secrets um more importantly the workload Secrets uh and after we will talk a little bit about non- regression and conclus and conclusion and there will be some technical demos so don't fall asleep before I I start the demot okay so if if we take a step back um secret management has always been a problem uh since secret exist it's

been a problem how to manage it how to make rotations uh who is using Secrets when um it's a pain for every everybody and it causes security problems too uh for example in January Mercedes uh has leaked some source code uh after exposure of a gab token so it was a simple human mistake and all the security that has been built by Mercedes crash instantly and all

the source code of mercedesbenz was leaked um and more recently in February uh an ex employees admin credentials for US government was leaked and used um for an attack we don't know yet the blast Rus of this attack but it's it's a major one so even big companies have problems resolving this Secret management thing there are also some reports uh very recent reports from Verizon and Google

threat they whole say that um most cber attack involves a human maybe it's social engineering or it's just an error and also credentials continue to be um involved in attacks so it's not me that it's saying it and also data dog has say that secret rotation um is not a good thing in here we can see that in AWS Google cloud and Asia there are lots of

long LIF credentials and these Cloud credentials are extremely important because as the API of the cloud provider is always public if someone finds these credentials um he will be able to connect to your infrastructure and do a lot of Damages so um in the past years some tools have popped up and they are acting uh after a secret is created or leaked uh the most wellknown is

G Guardian they will tell you if you have leaked the secrets in your resource code so um sometime it's after the secret is created or leaked and cyber engine is monitoring if your secret is somewhere on the dark with zero credentials infrastructure we will try to eradicate the source of the problem by removing any long-term credentials so this is the paradig so what does it change on

the left you have Legacy Concepts and on the right Z credentials um so nowadays uh developers still have many passwords to handle and with the recal we will try to uh switch it with a single sign this is something that most of you have normally already done it's the first step of zenal infrastructure we will try to replace authentication by authorization um secrets with tokens avoid any

manual rotation with refresh TOS um and try to generate on the Fly credential when it's needed uh instead of storing it in a long-term fashion and even if awareness for developers is still something important um if you have a lot of mechanisms that Implement security by Design you will avoid most human errors even if they are aware of security problems and and that important so what are

the benefits of a zero credential infrastructure first it will reduce the risk of tax by approximately 60% um because 60% of the involves long-term credentials uh in the kill chain of an attack first an attacker will take control of part of your infrastructure get credentials and he will try to resell it to another malicious attacker who will um do some threats in ransomware cryptojacking everything so if

the credentials are shortlived an attacker cannot resell it to another one so it will make you a less interesting Target uh also you have to consider the fact that most attackers they are looking for for for the low hanging fruit in the environment and so they will look for the less secure companies if you have done some efforts to secure your infrastructure uh you may avoid a

lot of uh attackers attention and also uh we will dig this deeper in the in the f of this presentation but with short live credentials everything is automated so an attacker will have to go through the same path and developers does to do production release or anything it will leave traces in the logs and attackers don't like it on the developer side it makes the developer experience

much better we have uh we have seen it in the project that we' have done um the Developers very happy with zero credentials infrastructure because uh they will have only one password to enter at the beginning of the day they they won't have to do anything related to Secrets everything will be automated they won't have to handle rotation also and they will strictly keep to what developers

does best is writing code and the configuration will be handled by uh by process and lastly for security teams uh it's a huge Improvement uh you will have a lot of compliance by Design every compliance step related to password management creation rotation everything will be done by Design uh it will greatly improve also the auditability of your infra because as credentials are generated on the fly when

it's needed in the logs you can see that this credential was generated for this task for 15 minutes and and and that's it so it's much better than having a password shared by a whole team of developers it facilitate also onboarding and offboarding you just have to create one access on the S sign and give permission and that's it and we we've seen it a lot also

at padok when someone is attacked there is a lot of stress generated and with zero credentials it it simplifies emergency procedure because you know that if you have fixed the vulnerability the haers used to enter your infra you just have to wait 15 minutes and you won't have any CRS anymore to to use so I hope this is clear do not hit take to to ask question

in in the tab I will have some times at the end to uh to answer it so where are credential stores in an infrastructure this is a very simplified uh schema just for for me to to be able to uh to show where credentials are stor most of the time on the employees workstation you will have the password manager some text files uh the history files of

slack teams every one once um once a month or it depends but he will share credentials when it's um when he has to rush when something is needed very quickly he will send cred through slack teams any any messaging app on the cicd they are mainly stored uh in the code in the code base this is a very bad practice but um we we still see it

nowadays or it is told as cicd secrets so the cicd is able to deploy um any assets to the infrastructure and on the infrastructure itself for example infrastructure it will be stored in the secret managers uh software and it will be also stored in the runtime environment in the virtual machine containers um serverless function anything in storage bucket also so you have lots of credential stored in

many places and sometimes there is some replay of credentials it's a very common pattern to see developers use credentials that are used by machines in the development environment to connect to database or even production environment so at the end you don't know when an access is made on the database who has done the initial access is it your runtime environment is it your developers you don't know

so the method that we have developed to reduce this long-term credentials is in five easy steps first detection um just map every secrets that you can find uh Define by who it is used and after you will be able to eradicate it and we will reduce the developer Secrets the cicd secrets the workload secrets and after there there are some useful steps that you can do to

avoid regression so let's start with the secret um as I said earlier this step may have already been done by you connected the single sign um before single sign you had to have credentials for each asset you want to connect to a web application Cloud infrastructure active directory any internal web app tooling with SSO you only have one identity provider that is federating your identity and allowing

you to connect to um different things like web application infra everything uh just a quick note it's very useful to enforce your MFA on your single sign as it's the main point to connect to everything this became some critical assets and must be handled very carefully of course so when we do this kind of mission for our clients we have some objections um I have have a

custom application it doesn't support SSO for example it's my main product in the development environment I don't want to configure anything else that is not in production so I cannot connect to it SSO sometimes you have tools that you don't pay for the license and there is this SSO tax things with lots of tools so you if you don't pay you cannot connect your company SSO to

the tool and sometimes you said that it's it's a pain but my database for example my my es database doesn't support SSO and so I cannot plug s to it fortunately there are some solution that exist uh to address these issues first is a product that I like a lot it's an open- Source product it's called O2 proxy it allows you to act as a reverse proxy

in front of your web applications connect to your identity provider so you can add a layer of authentication and avoid expose your internal tooling or your developer applications uh development environment application sorry uh for everyone you can just put your O2 proxy in front of it and use SSO to uh to add a layer of security another good solution um is to have a basan host I

like the setup with ashop bundar and volt it allows you also with open source tools to uh plug to your identity provider and access all sorts of assets that are not supporting uh single signon natively kubernetes clusters um SSH key to connect to VMS database credentials so with O2 proxy and bunder you will be able to address most of the assets that does not natively support SSO

um it will remains unfortunately um the SSO the S A sorry s AAS SAS application that you are not paying for SSO uh but you will have reduced the number of credentials to the maximum minimum you can have Okay so this was the easy part secret um now you we can talk about cicd secret this is also not so complicated but um I see that in most

companies that I go to um if the developer secret has been handled the cicd secret is is not something like I see usually even if it's very easy to to set up so two parts for the cicd secrets first is the secrets in the source code so how to have zero secrets in your source code first thing is to be sure that you don't have secrets you

may have some surprises and it's a good thing to try to detect scan all your repositories and see if you have some clear Tex secret um there are two open source tools that are well known it's trle hog and uh git licks at padok we have created a tool that mix both tools so it takes the best of both tool to to scan your repositories and output

an Excel files with every alert that has been triggered sometimes it's HSE positive of course but it allows you to automatically scan everything in your GitHub or gitlab organization um so please use it if you have some issues to uh to tell us or some remarks uh it it's welcome after you have detected all your secrets what what do you do so these secrets must must be

gone they must be moved to secret manager to vult to anything that's more secure and if cannot avoid storing it in the code base for whatever reason maybe you are deploying to A system that is deconned from your infrastructure that doesn't have ACC access to uh to your Vault or your secret manager um in that case you can use Subs it's a Modzilla product open source also

that allows you to store encrypted files and to decrypt it at R time I will show you this just after um if you are rotating these ensure each credentials has a unique usage for example if you have a bunch of microservice and you realize each micros services use the same password to connect to the database it's now it's a good time to create a unique credentials for

for each service and of course all the clear text secrets that has been found must be rotated because they may have already leaked and you maybe you don't know it so Subs um how is it working a developers has a clear text configuration and is able to encrypt it using any key solution Management Services and push the encrypted version in the gitlab OR gab repository and it

will be decrypted when it's deployed on the workload so you allow developers to encrypt and you allow your workload to decrypt it and at least you don't have clear text credentials in your code base how to decrypt it you can use an init container you can use a a job that runs before your application it's not so complicated to to implement okay so that was for uh

secrets in the code base now let's talk about Secrets the cicd secrets itself so a lot of major investment have been done in the past few years it's now possible to allow a repository or a branch to connect to your Cloud infrastructure or to your active directory infrastructure without having credentials just relying on authorization so uh first step is you will create a no IDC trust so

on your infer you will trust GitHub or gitlab or anything bit buckets um to connect to your infrastructure and whenever a repository for for example repo a will try to connect it will first generate a GWT it will present it to the cloud infrastructure if it's allowed if it has been allowed in the policy to connect um after the G GWT has been verified he will be

able to get a short term credentials to connect and interact with your with your infrastructure to deploy Things Push Docker container access a database connect to a kubernetes cluster whatever so I will show you uh how is it how it works so this is theform code that I've been using uh to create this oidc trust so here we create a trust in AWS this is just an

example but we will trust Gab after we will create a dedicated role for for my giab repository and I will allow it to be assumed by who by the GitHub organization called Padu team and only this repository called demo giab action o IDC and only the main branch will be able to assume it so this is a great Improvement before you had some secrets in your repositories

and anyone able to trigger this ACD will be able to print the secrets uh to export it or to use it maliciously now you know that only the main branch will be able to access your production environment so if you have a protected Branch if you have a two high review or anything you know that um nobody will be able to access it uh directly and at

and each time a new credit credentials will be created because each time this for will will be assumed credentials will be created and each run of your CCD will have dedicated credentials so I will just allow the role to list A3 bucket for example and I attach it to the RO so that's it you just have to create your identity and to allow your identity to be

used by a repository or a branch um and so if I use it in my cicd what will I do I will first allow my repository to uh create a GWT and after instead of using hardcoded credentials I will just say okay this is the role you need to assume and after you will be able to use it and I can conf figure how long a credentials

is available so I will show you right away I will create a reverse shell uh inside the GitHub worker to show you uh concretely how it works so I I will just push to the repository okay so that's it I've pushed and as every commit and Main Branch will trigger this job normally I can go to GitHub and see how it works so the job has been

triggered so the worker will create a JWT present it to AWS AWS will check that the GWT has been signed by gab after it will check in its identity and access management if this repository and the main branch is allowed um is allowed to to assume the role and by so I may have some issue with the running action I will get back to this tab after

the J has been has been run okay just be patient it will it will eventually okay so what are the objections for this cicd replacement uh clients tell tell me my pipeline um the assets my pipeline is accessing are not connected to Cloud am am so it doesn't work uh even if my pipeline is able to access a credentials it's not useful to access my database for

example or uh it require ire some external credentials that cannot be generated in the flow so what do we do and we have these problems possible solution is at least to use this oidc mechanism so your workload access some remote storage a secret manager or vault to retrieve Secrets why should we do that why add another step instead of storing the secrets store it square and authorizing

your repository to access the secret it's better because you will store your secrets in one place you won't have two places where your secrets will be stored it will be easier to do rotation and if an attacker wants to use it he will have to leave traces in this ACD pipeline another good point is we will see it is if you can use Dynamic secrets for example

for database it's possible for Vault to generate Secrets every time someone wants to access the credentials it will it will be better than nothing each time a repository wants to access the database it will connect to the cloud ask volt and have one time credum shorts to connect to the database so last part let's talk about workload Secrets uh in the meantime we just check if the

job has been uh run it's a shame it's it's not running um I would just check my previous job if it's possible to uh to access the the worker but I don't think so but in in the previous job I just launched before the presentation um I try to access AWS oh know it's not working maybe I have some connectivity issue I will just wait for sop

to to be launched okay let's talk about workload secrets so this is a much more much more complicated thing to do um what you need to consider is it it will have some friction with your application when you try to have automated rotation it will it will not be as easy as cicd or developer secrets so you have to ask yourselves what is the type of workload

how does how do applications load Secrets this is a very important question is it through a file and environment variables is it directly accessing some Vault or or some secret managers is your application compatible with some mechanism for example in AWS gcp or Azure you can authorize an application to access a database directly through I am but sometimes you will use libraries that are not compatible with

so uh this is some questions and last what type of secret is it is it an asset that is stored on cloud that supports IM connections is it some assets that are supported by volt Dynamic Secrets is it a simple I API that I not I cannot rotate easily these are the important questions um in in the following of this presentation I would like to zoom on

uh various volt kubernetes uh setup that are useful to uh to do this automatic rotation of credentials but of course uh a lot of solution exist and I invite you to uh to contact me if you want to discuss further your particular use case I will be happy to to discuss it with you so the first setup that we can see is using volt as a back

end and using using external secrets in communities to uh to get these secrets so in each of your kuties Nam space you will use external secrets to connect to a Secret store that is connected to Vault so each time your application wants uh to use a secret this Secrets is automatically synchronized by external secrets so if you do some rotation in volt external Secrets will detect it

detect it and uh it will reload the secret object in kubernetes but you will have to roll out your application yourself so they take into account that the secrets has changed otherwise it it will just keep using the old secrets and uh and it won't work anymore so this is a nice and easy setup another setup that is provided by volt is a secret operator so it's

an agent an operator that will automatically synchronize Secrets just as external Secrets does it's slightly better because the V secret operator is able to automatically roll out application whenever a secret rotated so it will detect this rotation and roll out the your deployment if you have multiple occurrence of your application the old ones will still have active connections with the old password it still works and the

new ones we start using the new new centures so this is working nicely the only problem is that it is using um a secret object just as external secret so the granularity of your secret is a deployment every pod of your deployment will use the same another solution is a CSI provider so it it works a little bit like the operator but this time the volt will

create a amount on your volume on your deployment on your CES and store credentials in it so if your application um just needs some secrets in your in its file systems to load uh but it does not support uh secret rotation so if the secret changes in VT um the CSI provider will not create a new volume it's not possible of course because it is already mounted

on the Pod and that's so you you will have to Ender too and another setup that I will show you with a technical demo is is this one I like it very much uh it's to use a agent so a volt agent will create an init container in each of your pod I will show you concretely how it works and the nice thing is that each pod

will get a dynamic credentials because your agent will try to read a path in volt for example SL database sled readly and if it's a dynamic Secret volt will generate the secrets on the fly in the database so each of your pods will have different credentials and it makes auditability much more much better because you will know which pod has done what on your database the only

problem is that the volt agent will create a file on your pod so you will need to develop a library to watch the file system for example with I notify system codes and to reload uh the connection to the database for example whenever the credal is rotated uh this is a QR code that links to ashop uh blog where they talk about these different solutions um I

was not able to to speak to every pros and cons here because our time is limit limited but um you can go by yourself and check it out all right so let's do a a demo I will just check if this uh this job is is done or not no it's not so it's demo demo effect sorry for that um so let's talk about volt and the

agent so I have created the local clusters um and I will uh create first uh database so we create a post gray SQL database so I've created the Nam space and uh I've created a deployment Pog and there is still a demo effect what's happening I need to delete this persistent volume I'm sorry for that just do it back okay so this time the post gr SC

database is creating after I will create a volt and I will connect everything and push a demo application so you can see action so now I will execute a shell in the postgre SQL pod and I will connect to the database so here I am I need to create long-term credentials for volt this is the only long-term credentials that will be created in the database and volt

will use it to create short lft credentials as a note it's possible to automatically rotate this password um but I won't have time to show it to you uh today so now I will create a role for and I will create also a role for my applications just for the demo a readly rooll so volt will be super admin and it will have access to a readly

role that it will use uh to create short lft credentials for my applications so I can go back uh here and create the vot Vol so I will install install volt okay so normally I should have the volt here and the agent the volt agent so I will execute a shell in The Volt pod and now I will be able to uh create the the path for

datab secret so enable the secret engine called database and I will create one configuration here so let me expose it to you so I want to connect to a postc database I will use uh my long-term cred volt credentials to connect to it so volt will be able to uh to interact with the database and now I will create the dynamic secret so what I will do

it's I will create a path called database Road rly and anytime someone will try to access this volt path it will execute this SQL sentence so it will create a dynamic short RO with a custom password for a certain amount of time which is here um 1 hour so I will create this file and after I will use it to create my pass to to vult okay

so let's check it how it works if I try to read this pass I will get for one hour this username and this password it's been created automatically if I do it again I get different credentials and I can do it anytime I want it doesn't mean that this one is not available anymore they are have just created two new credentials for the database for one now

how do I connect my kubernetes application how do I do this interaction between the application and v i will allow kubernetes Authentication will connect volt to kubernetes and I will just say okay uh this is the policies someone will be able to read pass and I will say who will be able I allow the service account name internal app in the namespace default so I we this

this policy I think I've just missed okay okay so now I can deploy an application and if it's default name space and if it's using the internal app it will be able to have access to this pass the readly pass and whenever it access it it will execute this SQL statement and generate a new credentials for it so I will deploy a simple three pod uh I

will restart it uh I don't remember do that unfortunately okay so these are freshly new Poe and if we if we check the logs of the cube of the volt agent you can see that the volt agent has detected that some application needed volt secret so this is something I have not shown you but here in the deployment of the application I've created some annotation I said

okay can you inject credentials can you use the RO internal app and can you access this path in the vault so when the application started the volt agent created an unit container and mapped this the credentials return by volt here uh to the application and so now if I checked what credentials have been loaded in the application for the three pods I can see that each pod

has a login so it's not so complicated to to set up and after you know that all your that the volt agent will be responsible to reload it frequently you just need to code a small development library that watch the file where the credentials are stored and notify the application to reload the that datase PR shows the okay um so this is it for um workload Secrets

again there are lot of configurations possible lots of interaction with application so I've just shown you some examples but lots of solutions are available to you even if it's each ecosystem is different you can ask yourself uh can I use authorization instead of authentication if you use cloud assets normally you can just give IM roles to your application to to be able to it does my asset

support Dynamic secret is volt able to generate SSH key on the Fly kubernetes authentication um database crur they have a lots of supports for dynamic credential so check it out and and finally for other assets how can I Implement automatic rotation if it's a third party API key maybe you will be able to uh to just do some API calls to uh to rotate the API key

so finally zero cut infrastructure it should buge down to one secret per person Zero Secret in sayd and Zer Secret at R time which will avoid human errors and for for the attackers to traces um okay thanks for listening to me if you have any questions you can ask it here and I have my LinkedIn here on the right if you want to to chat with me

talk okay so there may not be any questions uh please contact me on LinkedIn I will be happy to discuss the particular use cases with you and uh enjoy the rest of your cyber wise conference too bad for the first demo for still no questions in the in the chat