Halil Ibrahim Kalkan: Building a Kubernetes Integrated Local Development Environment

This talk covers the development of an efficient local environment for creating distributed systems, particularly microservices, using Kubernetes. The speaker begins by addressing the challenges faced in developing microservice solutions, including the complexity of dependencies and the difficulties in setting up a working local environment. He introduces a concept called Kubernetes tunneling, which utilizes tools like Telepresence and Helm to facilitate communication between local development environments and Kubernetes clusters. The speaker demonstrates how developers can configure their local services to interact with Kubernetes infrastructure while using their preferred IDEs. He also presents a custom-built tool, ABP Studio, that establishes a secure VPN connection allowing seamless access to Kubernetes services and integrates local development with remote resources. The session culminates with a live demonstration of the solution in action, showcasing how to debug and test microservices efficiently.

[Music] ladies and Gentlemen please welcome our next speaker Hal ibraim kin presenting the topic building a Cuban netus integrated local development environment uh hello uh welcome everybody uh I am halim kkan um today I will talk about how to create an efficient way efficient local uh to develop distributed systems like microservice systems and to do that we will get help from cuberes uh let's talk about the

um agenda first uh I will start by defining the problem why def building a microservice solution or set up a development environment to create such a solution is hard uh then I will mention about some primitive Solutions and then introduce uh a solution uh I describe it as kubernetes tunneling uh I will show some uh existing products that is already available in the market some of them

are already free on open source uh to solve this problem and um then I will uh explain how we have designed and implemented a solution to this same problem just similar to existing products uh and I will mention uh about the tools I have used us it so maybe you want to implement uh in your own company without using an existing tool existing U product and I

will finally show a demo uh of the system we have created so you will see it in action uh let's uh talk about me a little bit I'm computer engineer uh like uh almost 20 years uh and um I have been worked in uh different companies as software developer architect and team leader then I have established it uh woft uh currently can be call it as a

startup uh that produce tools for developers uh or most common known uh product is the ABP framework it's an open source framework for net uh developers uh and I typically work on multitrade systems distributed systems and uh I'm a coder so I work on objectoriented programming uh I sometimes talk about DDD uh some software architectures and so on so I'm very uh active coder mostly working on

open source projects the AP framework I mentioned here uh is already open source the core part is open source actually there there is some uh also commercial parts of it uh so we can survive the company uh now you know a little bit about me I want to know about you uh anyone worked in a microservice solution here okay good uh anyone used kubernetes in development or

production environment okay most of people have already used kubernetes great uh devops Engineers okay developers or especially backand developers great I'm also backand developer um backand focused full St developer or some any net developer okay not too much I'm net developer uh okay let's start by uh seeing a simple microservice solution so to realiz in my in our minds uh for a typical uh microservice generally we

have one or multiple application for for this example we have a backend application as uh like for back office users or your employee and a public website for example in e-commerce website uh website uh that uses by uh the end users uh if you are using uh back end for front end pattern you may have uh two API gateways in this setup and then typically an Authentication

server uh that is separated uh in the system uh and some microservices Behind These API gateways these are just three example uh like a product service ordering service and an identity service to uh manage user accounts uh you will typically as a event bus maybe rabbit mq CF or whatever you want whatever you are using some other uh supporting infrastructure Services uh this a complicated system but

actually this is very simple uh when you uh in real life you will have much more microservices uh but think that you are only responsible to develop the product microservice here so how to run this product microservice locally uh in your own environment uh while it has many dependencies on the other systems uh think that you have a local computer uh using net visual Studio doesn't matter

can be Java uh when you run it locally you will get many exceptions because no red disc connection no SQL Server database available maybe it will uh make request to other dependencies like authentication server some other services so uh to work this product microservice in your local environment all these dependencies should be ready and available if you uh if you want to run and debug in your

own local environment uh that was a simple example but this is for example Ubers microservice CB how you can run all these systems locally to be able to develop only a single service or a few Services uh one way can be maybe uh you may think create unit tests and maybe some integration tests uh then to uh then you isolate your uh all service from all others

like for example you may mock dep mock the dependencies uh for database you may run a local database or just in memory database uh to simulate the database uh for infrastructures you may create obstructions and create mock implementations for all these systems like C file system maybe you are using third party services to send SMS email and so on you can uh maybe somehow mock dependencies uh

you may uh mock uh other uh service calls like for example ordering service may need to call a uh product service to check the stock count for example uh you may need to uh create obstructions and uh fake implementations for example and also you need to deal with events uh in a isolated unit test environment uh and also your service sometimes will be triggered in different ways

for example it may receive events from other services or uh or it may receive API calls from the user interface or other services so if you can think all these and create a unit test environment this can be a good way of developing a single service in your local environment however to be honest it's hard to implement and uh doesn't solve all the problems for example uh

you may if you want to uh test everything integrated if you want to trigger the your service from the user interface or you may want to use the user interface and see how your uh service behave in this uh environment so uh unit testing is good but uh you working your production uh product service in a realistic environment would be better actually uh to see the problems

in production and also you need to simulate some Network delays some errors some retries or problems of distribut to transactions uh if you don't know anything if you don't use any tools you may uh try to run all these on local computer you may open your ID in and uh run all the services one by one or create a uh Powershell script for example uh to to

uh to run all the services with just uh single uh CLI command but uh this is this will not be efficient because it will um consume your local resources first of all and also uh it's not easy to run all these services in your local computer uh since I a net developer uh Microsoft provided some solutions to before uh they tried to introduce the Ty project before

it was an experimental project now it's retired and replaced by the net asire in the latest net 9 uh it's a still in preview net Aspire is U an opiated cloth ready stti for building observable distributed applications uh it has two main comp uh three main components uh the first one is orchestration uh you set up all the dependencies of your projects and then uh it can

run all this system in your local environment and uh sometimes in kubernetes or uh Ducker compost the uh second uh part is net asire is uh comp some components like it has some nuget package if you use this nugget package they provide some obstructions and Integrations with this uh post SQL uh and some other commonly used tools the uh third part of net asire is some kind

of CLI commands that uh make that makes some automation uh while running all these systems in your environment uh it provides also a dashboard when you run all the applications in your local machine for example here we have just three applications uh a web application and an API and a red cetch uh the redis is working on a container uh typically in Ducker and uh these are

running locally and you see uh all their status you can view their logs uh for example and you can see some metrics and traces uh I will not go deeper with net Aspire since most of the people are already not net developer but uh this topic is uh mostly focused on kubernetes solution so what's kubernetes you already know what is kubernetes actually kubernetes system or a tool

uh that is used mostly in production environment to deploy Your solution scale Your solution in a uh cont containerized uh system uh like you may have multiple nodes uh each not the uh can be a physical or virtual server uh you you have ports running in different computers you just describe your system you I want uh five copies of this application that's all you describe the state

of your system kubernetes uh does everything when a no no is goes down it can restart uh services in many in different computers and so on so uh this talk is not actually describing kubernetes but I wanted to introduce kubernetes uh best for uh running uh system in test or maybe QA or production environments uh and us it by mostly by devops uh Engineers uh but for

develop developers also uh uses because um they need to work with the Ws team uh to set up the all yl files and so on there is also a tool called it Helm you probably know that uh when you use Helm you can deploy all the kubernetes system in a single uh kubernetes cluster in a single command maybe C command hel install or uninstall uh you can

uninstall we will uh also that uh okay kubernetes is good for running uh your entire system in a uh production and uh like a blackbox maybe but how it it can help us uh in development uh the idea is to run all the system in a kubernetes cluster all the applications Services infrastructure Services gateways whatever you want and run only the Production Service product service that's responsible

to build in your local computer uh so when you run in your local computer you can use your favorite IDE you can use your tools and uh any other uh tools you are using in computer and finally connect two environments together so they can run as a single environment and work as an as an integrated solution so the question is how to establish such a system uh

this is a little bit virtualized I mean you product uh you run the product service in Visual Studio uh run it locally uh when you need when the product service need infrastructure like database uh cash or whatever it will use the services in kubernetes uh normally these services are not exposed to outside of the cluster because they are uh for security they are uh like behind the

firewall but your local computer uh the product service in your local computer should access these services uh and also product service may make some uh async or sync communication with other services or applications in your cluster some of these Services may not be exposed to outside again uh and uh it's best to uh run the product service just like it is running in kubernetes cluster by using

the same configuration and environment variables and wi Force for example uh ordering service may need need to uh make a call to product Service uh it should redirect all the communication to your local computer so uh your uh local running instance it is triggered by the events and rest API calls for example uh to do that there are some existing Solutions I will uh first mention about

this solutions then I will explain how we have built our solution uh the first solution I mention I can mention is uh bridge to kubernetes extension of visual studio uh it's very limited solution it can redirect uh calls right uh like rest calls to your local computer but that's all that that has not much abilities telepresence uh actually the best solution uh of I I will uh

I will mention uh it does actually all it can connect to uh your computer to kubernetes Cluster redirect API calls and everything between the computers another solution is Garden it has much more actually U not only for kubernetes tunneling that has more features all these two products is mainly open sourc but not free uh when you want to use in production you should pay uh Gea on

the other hand is um an complete open source and free Tool uh to be honest I worked uh with gria but couldn't able to properly work uh by it looking in documentation uh so it has some problems I think maybe it is fixed in U next releases however the purpose is the same um running your local code in your uh running your service in your local environment

and uh integrate with the is uh let's uh mention about uh the telepress solution a little bit uh as I said it's an open- Source core frame uh system uh but uh when you want to use uh in production you you will need uh an Enterprise license it basically install some tools like a traffic manager or interception services to your kubernetes cluster when you want to connect

uh uh and tie your environments then redirect calls to your local computer and when your local computer needs resources in the kubernetes it redirects all the uh service calls uh and connections to to kubernetes uh to be able to use telepresence you first need to install the telepresence CLI to your local computer then uh install the telepresence in your kubernetes cluster actually this is a connection uh

step uh you first Target a kubernetes cluster uh then uh use telepresence Helm install uh then it actually first installs all the traffic manager and uh other tools to your local computer uh to your kubernetes cluster uh when you want to connect you just uh type the connect command when you do that you will a ble to access all the services inside the kubernetes just they are

running in your computer uh then when you want to redirect calls API calls to your local computer you will need to use the command uh finally The Intercept command is also extracting the all the uh environment variables into a file then you can use file that file to overwrite your uh environment variables in your local computer so when you restart when you uh start your local service

it just feel itself itself just running in a kubernetes cluster and communicate all services uh just like they are uh a environment okay now I will show how we have implemented uh this solution uh we have uh also investigated all the solutions and uh now implemented our uh we have a local development environment like a just a laptop and a kubernetes cluster that um our solution is

already running and up so we can uh for example brow uh a user interface in this solution uh it's typically um handled by an Ingress controller in the kubernetes cluster then uh redirected to a web application for example uh the web application typically uses an API Gateway to access the uh underlying microservices for example these microservices uh can also communicate internally and uses some uh infrastructure tools

like cql server rabbit mq and so on uh and the for this product service you want to uh uh develop it locally so you open this product service in your Visual Studio or favorite IDE uh the tool we have created is named as AVP studio uh so I will explain how works uh when you connect to uh the kubernetes cluster it first uh installs and VPN server

to to the kubernetes cluster and the VPN client to your local environment then establishes a VPN tunnel between two environments with the help of this uh VPN tunnel whenever your service uh requires uh needs to uh use a service or infrastructure service a database and so on it just can run uh it can it can just use just um like it's running in local computer because there

is a VPN connection and redirects all the TCP connections to that uh cluster and makes it available in this cluster and also it uh setups a proxy server to collect some metrics and a proxy client uh it collects uh all the HTTP calls exceptions logs and some um uh some other information from all the services and infrastructure and sends to your local computer uh in in the

AP Studio user interface I will make soon uh then ABP Studio makes changes on your host file so in this way um you can just type a service name in your browser and access to the service uh in the kubernetes cluster even if it it is not exposed to the outside okay when you want to intercept a service intercepting a service means uh when for example ordering

service makes a call to that product microservice in the kubernetes it redirects the call to the local instance in your running in your local uh machine uh so it setups a proxy server and proxy client uh change some service definition in the kubernetes and redirects calls to your uh running instance in visual studio for example uh when your service uh wants to use S other services it

can already use uh the VPN connection and finally it uh manipulates the environment variables for the running instance so overrides all environment variables that that are copied from the uh pot running in the kubernetes cluster and finally when you run the application uh in your local machine it behaves just like running in cluster so we I I mentioned uh by the uh about two operations the first

one connects establishes a VPN tunnel so you can access all the internal services in the cluster and can start to collect uh some monitoring data uh to track problems or get exceptions in the cluster the second operation is interception uh when you do that it set ups a proxy uh server client and redirect all the traffic to your local environment including distributed events uh it overrides the

environment variables and it's some DNS records to host file so everything is ready uh we have implemented many custom logic or uh code as you can expect but we have used two main tools so you can check these tools to create your OV solution the first one is wire guard it's a tool to create a secure VPN connection between two environments uh with this uh wire guard

we can able to access internal services or kubernetes and the second one is FP it's a reverse proxy to redirect traffic to your uh local environment normally you know when you run run your a service in your local environment it starts to listen a TCP Port but this TCP Port is not open to uh outside unless you to uh you configure your modem for not connection uh

not uh settings uh but uh Tool uh can uh receive uh TCP connections from outside without needing to configure your firewall or nut Services now great timing now see a demo to see how it works in real life so you decide if it is easy to run uh make such a development or not uh first of all uh to not lose time I already uh deployed my

uh solution to kubernetes you see uh the services uh like Administration server authentication server a web application Gateway uh some tools like grafana elastic search and um SQ server uh Services deployments everything is ready so we can browse the application this is the application I'm logging in the application uh for just this demonstration I have created two uh microservices products and orders uh with just simple implementation

products we can create some products we have an orders page uh and uh you know it's it's running in cuberes for demonstration I running it uh on local uh Ducker however it can run on um Azure or any environment this is standard kubernetes so I'm now opening the ab studio tool uh and uh I open it the same solution in my uh ABP studio tool uh I

will not me uh much about ABP Studio because it's a like product uh but for the kubernetes uh top you see the all the services uh sorry all the hel charts uh that's deployed to kubernetes now now we have a connect button here uh in the right part you see all the services now they seem stop it because we are not connected to kubernetes when I uh

click the connect button it just um installs VPN to kubernetes and establishes a secure VPN connection so now we see all the services running in the kubernetes uh UC see Services uh up time we can browse any application for example let's start by browsing the product service for example you see we can you see that okay we can browse the product service just with the service name

just like it's running in our local environment so let's open the web application and log in here okay I will create a new product to demonstrate the development I will prepare some data okay I have created a new product it's called Product 3 and I will create a new order from the product tree you see the order when you go to go here you see the stock

count is decreased to uh 95 because of uh the when we place an order an event is published it and product service catched it and decreased the stock count uh now create another product with the same name you see the system doesn't allow to create create a new product with the uh with an existing name okay fine for now let's see HT all the HTTP request uh

performed during the all these operations like you see we uh this is product service sorry this is the product service call to create the product you see the request details and the response so you see the error message here so you can explore all the request headers and response headers here uh the good point is that uh this is not like developer tool in developer tools you

see all only the U request from the web application user interface to the API Gateway but uh here you see all the request including inter service rest API calls you can also see events for example uh when we place an order uh there is an uh event publish it name it order plac it uh with the product ID and stock count uh it's received by the product

service you see so we can also track all the events sent and received by the services and we can also see all the exceptions including the uh exceptions throwing throwing by uh the services uh we are using uh you see all the St stake trays here for detailed exception loog you see the same exception actually repeated two times because uh the first one is thrown by the

product service and it is rethrown by the web application but the uh but the uh exception type is different because for the product service it's kind of user friendly exception for example for the web application it's a remote API call exception okay if you also want to see some logs for fure details you can see product service logs the logs written to console is available as real

time here so you you as you see uh we can see all the applications get uh bro stem in kubernetes we can see request events exception logs and uh dets okay uh let's make a development uh until now we haven't set up any actually in the client site in our local computer everything's working in kubernetes now uh I will intercept the service and redirect traffic to my

machine I'm right clicking here and enable intersection in uh interception okay now it's done you have a you see a small symbol let's um try to brow the product service when I do that let's do it on Chrome okay it is not working because the product service are now redirected to or my local machine but it is not working I'm opening the visual studio and starting the

product service in thebug mode now okay now it's running in my local machine when I return to previous page if I press F5 for refresh now we see it's running uh when I browse this page it the request is actually uh sent to the kubernetes cluster it is then redirected back to my local machine and handled by the uh locally running instance of the product service okay

uh let's prove that it's really handled by my local computer I'm adding a break point here and trying to create the same again you see that it's uh it hits the break point point in my visual studio I can step Thro I can uh handle bux uh Trace problems and so on okay uh we got the same exception until now we didn't have any development however we

see that we can now debug uh a service that is normally running in kubernetes but we can debug it in our local environment now let's make some for example I want to change that instead of throwing an exception I want just change the name by adding some random number after the product name okay the code change is done now I normally should stop the application and restart

because this is a compiled application however Visual Studio has a hot reload system if you uh only change the body of a method you can just press hotload that's all it's now uh in live okay let's go to uh the same page again now uh I haven't uh you see closed the dialogue pressing the save button you see that new code is running and it added uh

850 to end of the product name just running it fastly you see that that product uh now available so uh the new version in our local computer is working instead of the uh product service in the cluster okay now my development is done I can stop the application I can disable the interception okay refreshing the page it is still working uh because it now uh return to

the product service in the kubernetes it now runs the kubernetes service to prove that let's create the same product again you see that we are getting the same previous exception because the version in the uh kubernetes was not updated okay let's update it right clicking the service and red deploying it when I do that uh it uh uses net CLI uh to build and publish the services

then creates the Ducker image and uses hel uh commands to redeploy the service it's completed you see now uh the new version uh working in kubernetes to prove that let's click the save button you see that we didn't get an exception uh we see the new product here uh working as normally uh anything else okay actually demo was like that you see that we we we can

just redirect uh we select a service intercept it redirect all the uh API calls to our local machine and we can debug develop test whatever you want just like in uh local development when uh our development finish just we uh uh disable the interception if we want commit or code uh then uh we we can redeploy the service to see in uh Real Environment uh if you

want uh then complete the development cycle it's actually pretty simple okay my talk is over actually um thank you for listening if you have questions h l Applause for you but we have a lot of question for here from the Let's uh ask it verbally first of all uh hello thank you for presentation I wanted to ask uh only one question uh while you press uh redeploy

uh it will be just rebuilt on your computer and then you will uh like uh it will run locally but it will have connect with kubernetes or under the hood you have functionality to push it to some artifactory and to trigger deploying to Pro to kubernetes yeah good question uh okay um normally you should uh deploy to an artifact then deploy from there uh but it depends

on the U like environment you are using for aure it requires like that you need an Azure container registry and so on but uh for this demo I use it uh local CU local uh Ducker desktop uh then we didn't require that we just created a Ducker local image and uh targeted that image as a uh as a cont container source and it says deployed so yes

you should do that uh in or tool we have created a system to customize this kind of things with some Powershell scripts so you can finally yes yes definitely uh if you are using for example Azure it will be Azure okay we have quite a lot of questions here and uh can the solution support multiple users actually deps to walk at one because it cluster at the

same time great question that's something we are currently working on um it the current solution is that uh to create multip to deploy multiple instance uh of the same solution to the same kubernetes cluster but with different name spaces so for every developer you may need you may use a different name kubernetes name space uh then use this name space to the developer and they don't conflict

the current solution is that but it has a cost because if you have 100 microservices uh and five developers then 500 microservice copies will work uh we we will be working uh to improve this solution to understand the request is coming from a uh a developer machine and redirect result to the same developer machine but it has some more difficulties we are working on that currently we

can do it like Nam different names spacing one more question uh how to develop and test front endent applications locally avoiding course issue course issue uh actually we haven't any uh the course uh and all set up in this solution for example we didn't have any uh problem for that uh for local testing we are just set up uh the course uh configuration to allow local uh

cost actually it's not a real problem uh all these tools are fine and Dy but how do you develop and test your services against different branches releases versions and different dependency Services uh uh again good question uh for the demo I have presented we I maybe it may dep it may work with the dev Branch for example uh normally for every Branch yes you should do that

however uh for example if you are using a uh if you are developing a single U service for example it's not depending other developers then you can use uh the main branch in the kubernetes cluster because the uh changent uh changent code is already running in your local computer not need to deploy it to kubernetes so all the uh new command or all the new Branch uh

can be check out in your local machine and the the uh the cluster can work with the main machine uh main branch so it's it's not actually a problem uh how to avoid data pollution issues if everyone are essentially working on the same death environment constantly testing and deleting data during the day okay uh good uh if you use for data is uh uh I think SQL

server for example you are using SQL Server creating products deleting products and other developers do the same so you will conflict in the uh finally however there is a good solution for that you can run SQL server in your local machine for example and redirect all the calls to SQL Server to your local machine then conflict uh okay one more question from Bernard I really like when

guys you are naming your names here in the questions because the anonymous group you know it had some different meanings uh if you run an application locally which connects to your database we need an active VPN connection with wi go can PR still use its internal VPN client if run application loc which connects to lo you need an active VPN connection okay still use it internal vpa

Clan I really didn't understand the problem but uh as I sh any TCP Port can be redirected to local computer without any EX for just with the same uh VPN connection okay one more question from uh AP any plans to support open shift open shift not not yet we haven't worked that uh okay and the last one from anonymous group what are possibilities to integrate this into

the cicd PIP for the testing the service while using the actual kubernetes cluster uh actually for our solution for develop not uh cicd pipeline it's a different story I think this is for simplify development workflow that's all okay thank you again thank you one more time and I would like to warm to to our speaker and here it's a small present from us thank you one