Adam Furmanek: Platform Engineers Must Change Developers and Databases - And Here Is How

In this talk, Adam discusses the importance of integrating database management into modern DevOps practices. He explains that while significant advancements have been made in software development and deployment, databases remain largely disconnected from these processes. The speaker emphasizes the need for platform engineers to facilitate a shift-left approach, where database issues are addressed early in the development cycle. He highlights common challenges such as code changes, schema modifications, and query performance degradation, which traditional testing methods often overlook. To combat these issues, Adam advocates for the adoption of database guardrails that empower developers to take ownership of their databases, fostering a proactive and efficient software delivery process. He concludes with the notion that changing mindsets around database responsibilities is essential for maintaining high performance and reliability in application ecosystems.

[Music] and we are back we have our and us uh platform Engineers must chain developers and database and his how so Adam take the stage and please te us how to do it cool thank you so much and hello everyone thank you for coming to this talk my name is Adam fman and I'm a head of Devo at metis where we work with databases and today for

the next some minutes we are going to discuss how to improve the state of the database we are going to see how platform Engineers should reshape the world today to make it better and more adjusted and more tuned to the modern devops we are going to see that while we learned a lot when it comes to devops and software development life cycle there are still pieces that

we are completely missing and lacking when it comes to automating our pipelines reasoning about our systems and generally doing the work efficiently So the plan for today is just to understand what we miss what to build and how platform Engineers can reshape the world so let's jump straight to it so we are going to go through a couple of ideas first we are going to understand what

is the current state of sdlc and why Conway's law still is still like very painful for our organizations then we are going to briefly take a look at what breaks today especially in devops databases and throughout our pipelines and then we are going to see what we need and how to do it let's go so the very first thing is we need to realize that we came

we finished or we actually moved through a significant transformation over the last years so 10 years ago or 15 years ago whenever we were working working on applications it was generally rather straightforward we had like monolitic applications when there was an app there was just one database maybe a couple of like nodes for for load balancing and generally a user touching that whenever we had to change

something we were generally modifying this single application and when something broke it was rather straightforward to understand where to look for the issues and where to look for the answers what could go wrong in very rare cases we decided okay let's build a completely independent application on the side it was a big thing for us we didn't want to build something completely new because well deploying and

generally starting from scratch was super expensive and super slow and we learned that that might not be the best case what we learned and what we did over the years is we moved from that very simple world to much more complex one a world where we have multiple databases be SQL no SQL Vector databases things for machine learning we have various ways of Hosting our code we

have serverless Solutions we have kubernetes we have on Prem we have cloud and many other things and most importantly is all those elements they talk to each other this is the hardest part about what we need to deal with today it's not only that we have plenty more elements but also they constantly chat with each other and when there is a problem we do not necessarily know

how to debug it it might be that our database is slow but it's not the database what is the issue it might be that our application is crushing but we can't find anything about that in the logs of the application because it's due to some other microservice or the traffic that comes to application looking for logs looking for traces and all the signals around is way harder

now than it has ever been before and this generally is like a good sign for us because it means that yes we learned how to tame this whole complexity how to develop things fast how to deploy them more often many times a day and how to generally keep all that ecos system under our control the problem is though that we completely left out databases we do not

cover them in our landscape and we are going to see in this talk why that's the case why our cicd pipelines do not cover the problems we have with databases and how to change that how to let developers own their databases the same way they own their applications no matter whether they are small startup or Fortune 500 Enterprise no matter how long they are in the market

this is something they need to do today the world has changed and the problem is we moved from doing things manually to doing things in this automated way with gitops with infrastructure code with anything as code but we are still running in silos yes the devops idea was that we realized okay devop development team and operations team they need to be much closer together so initially we

wanted to put them like side by side and ultimately we kind of Blended them together so now we have just one team responsible for both development and operations the problem is this one single team does not tackle everything that is important to successfully deliver the software nowadays it's a big thing for us to have def SEC Ops we try to mix in the security principles into this

devops world but databases are still outside and if we take a look at how the world looks like we often realize that hey on the left side on the screen what we see is like how we describe our applications there is a client of our apis that talk to a data store but if we Bend this image 90 degrees then we realize we are actually hitting the

silos we have one team responsible for like development of the application or the client another team responsible for building the apis and very often yet another team or separate like group of people responsible for working on the databases and if we take a look at that we may realize that in turn this is actually Conway's law in its Essence Conway's law is telling us that the applications

we build will closely resemble the organization the structure of our organization our orc chart and we still see that if we have DB Ops or if we have data like dpas sitting somewhere else in different room then we will have a separate layer for the database and we will have those silos actually there even research done showing that if some other team is further away than like

20 M from where you sit then you are most likely running into s and your performance will drop this is the emperia something we can measure in real world so moving on the problem with this approach that we have is when we have those separate teams the communication increases and we can draw fancy orc charts showing us that this team talks to that team y y but

the reality shows that the talk is what actually structures the organizations so no matter how we are s like between the teams or what silos we try to form within our organization the actual paths that we should follow is the communication paths and this may show us where we have those silos and how to deal with them and in order to deal with them we need to

understand that in order to automate everything in order to make our devops run faster increase velocity and Improvement of the stuff we need to shift every to the left that is the most important idea for all the automation we need to shift to the left move to the early stages of the pipeline everything that may slow us down this includes testing load testing security databases perform the

like configuration deployment and other stuff and we are going to see that through this talk to understand what we need to bring into the left to make things faster and the other super important part part to running devop successfully is self-servicing everything we need to let developers own their stuff and self-service their issues that's the important part because we can't let them or expect them to talk

to other teams because communication is slow we can't force them to raise issues in Jura tickets or whatever else because this is what slows them down so we need to shift everything to the left and make them self-service their issues this is what we need to do and we may say okay but developers should be able to do it right away right they own the infrastructure devops

Engineers own the infrastructure they can deploy stuff they can build their applications test them they can even see all the monitoring around right so they should be able to do it but is it really the case and let's see that it's not first thing about monitoring is that whenever we talk about monitoring stuff we typically think about dashboards like we see on the screen and they all

look great they show you metrics charts diagrams whatever you like the biggest issue with those metrics is that they do not give you the answer what to do and where the problem is they only show you that hey your CPU spikes somewhere there is fire burning out there and that's all you know you have no idea what's the cost you have no idea how to fix it

and more often you are swamped with tons of metrics that you have no idea how to act upon like those Monitoring Solutions they very often boast that they can bring hundreds and thousands of infrastructure metrics automatically they can get OS metrics uh runtime metrics CPU metrics memory etc etc but in reality that's not what we need we don't need to go for quantity we need to go

towards the actionable answers what to do and Monitoring Solutions do not give us that so let's try to understand what we need to build in order to actually give our developers those actionable insights and in order to do that we need to understand what actually can break today and what are those blind spots those gaps that our developers SM so whenever we talk about databases we can

all like typically we can group problems in those three areas so the first area is code changes so those are problems that we changed our application code we in turn changed the SQL query that has been sent we changed how the things are executed and no surprise things are now going like way slower okay this is the first area and this is the area that we pretty

often try to cover with like unit test obviously we changed the SQL query so we try to write run test that will tell us whether we are reading and writing the proper data the other area is schema changes so we change the database schema we add columns we change the data types and do whatever else and this may obviously change the the performance of our application because

our ORS may go crazy or whatever else so we try to cover that thing to some extent by testing those schema migrations like in our local environment we just run them but very often we do not have any idea how long such a schema migration may take in production when we run schema migrations locally we have like what 100 rows in our database those schema migrations will

complete very fast but once we go to the production it may be that the schema migration will complete in like minutes or even hours and this typically takes our database down and causes an aage there are always or sometimes there are even cases when a schema migration may take days or weeks to complete we know some of the protocols how to deal with that how to drop

columns safely and and in a way that we do not break things so we rename those columns and we drop them only a fiscal year after after we regenerate everything so we are sure no software has been impacted but generally those things they are still tricky but there is yet another area where we have no idea what to do and we do not cover that in our

devops pipelines and this area is query changes and I'm not talking about the way that we changed our SQL text no I am talking about the situation when our query slowly degrades and basically rots over time because our configuration changes our indexes change our uh windows or Linux or operating system updates change and we generally have a query that used to work very fast but now six

months after some index has been dropped or modified and the query changes the execution plan so the query still works still gets the data but does not do it in a way that it used to do it before so it changes the performance and effectively is too slow in our production case those are the things that we do not cover in our cicd pipeline at all and

our devops Engineers they do not see what's going on let's see a couple of examples imagine that we have an application code like here on the left side so we basically have some JavaScript code that actually comes from one of my production systems we basically want to get like an a a user and what happens is is like we we you know we get this user we

want to join many tables to bring the data and ultimately build just one single aggregate that would represent this this object right and in turn this code produces query like on the right hand side so we see that is Select statement with like many left joints and if we run thing like this in production then this runs for 25 seconds and extract something like 300,000 rows from

the database why that's the case that is because when we have like this many joints we increase the number of rows exponentially so in turn what is going on the database needs to produce so much data then it takes like nearly half a minute for the omm to process and be this one single entity what is the solution to this problem at least in this case the

solution was to basically drop the the the case of like taking all the data in one go and extracting those things like separately so this leads to many SQL statements being executed and one might say okay those SQL statements should be slower because we sent something like what eight queries instead of one but the reality is those statements are super fast and they bring much less data

and the important part here to notice is that unit tests for both of those cases for this query and for this set of queries those unit test will be green in both cases why is that the case that is because we only focus on whether we extract the correct data whether we read the data properly and whether we build the the the entity that we build at

the very end is the same is correct and those unit tests will cover those cases the same way they both will be green the problem is one scenario will be fast but the other will be way slower and our tests won't notice any difference yet another thingy is developers may be tempted to like rewrite the code to make it more readable so let's say that we have

like a query that tries to extract things from boarding passes and we have something like 8 million rows and we basically want to get the ticket number un calculate like double hash of the ticket number right so we calculate md5 twice and you may say okay we build this query We join like the table three times and then we need to repeat or duplicate the code to

basically calculate this double hash like three times as we can see in this code so how about we just use common table expression CTE that basically calculates this stick at number hash once and CT is just a conceptual temporary table so it shouldn't affect the performance and effectively we end up with much more readable query that we do not dupc replicate that the logic the problem is

though that the first query runs in 8 seconds and the other query that is now more readable finishes in 13 so nearly twice as long so just by making the code more readable we effectively decrease the performance significantly and once again the tests or the cicd pipelines that we have they won't notice this problem because they focus on the correctness of the data not on the performance

you might say okay we might have load test that will cover that we'll get to load test in a second but let's for now like realize that we don't capture problems early in our stages so whenever we talk about tests whether it's unit test integration test end to end tests there are so many things that may break and our tests won't capture that because the Pyramid of

tests is simply not covering those things but you may say Okay load tests load test should cover that right and you would be writing here load test would cover that but there are problems with load tests first they are very slow to complete they literally take hours to complete not to mention that it's hard to build them because we need to H anonymize the data deal with

gdpr California protection act and other stuff not to mention they are expensive we need to build a separate stuck on the site to run the load test and do the magic pay for those machines and if it involves like machine learning then we need to pay for crazy expensive GPU enabled machines but the biggest issue with load tests is that they happen nearly at the very end

of the pipeline so they happen far too late you already implemented your changes merged the code into the repository reviewed it with your teammate deployed it to non production environments and only then do you realize that there is a performance issue and that is the problem load tests they conceptually work they are hard to build expensive to maintain and run but they are way too late in

the pipeline and this is what cilles our performance we need something better so let's see what we need and solution is database guard rils so we need something that will let developers to own their databases end to endend and they need something developers need something that will show them that there are problems and it will show them those problems early in the pipeline it will let them

diagnose those problems and reason about the performance of the database developers often lack understanding of databases just because they are scared or maybe not using used to that however we need to we must let them own their databases end to end and the key part here is we want to let them own more but do less that is the idea they need to own bigger scope but

do less work and let's see how database GID can do that so database gers focus on couple things first thing is understanding what's going on with the databases and giving actionable insights so just like we had Telemetry early like couple years back so basically we had this ability to collect logs traces metrics whatever signals from our applications when we integrate Telemetry throughout our ecosystem we have something

that is called visibility so we can clearly say okay we can look into each aspect of our system and basically see what's going on there thanks to Tele we can use that to build the APM application Performance Management so we can say whether our application is green ecosystem is green or if there are problems or performance issues but we need to take it step further we need

to build the proper observability that tells us what is going on so the difference between monitoring and observability is monitoring focuses on the row signals on swamping us with data on metrics on alarms on things that we need to tune manually and basically adapt them to our cases observability on the other hand gives us actionable understanding of what's going on explains that we deployed this stuff and

this is how it affects this whole ecosystem observability captures the data throughout the sdlc starting from the developers environments throughout the cicd pipelines and with the production environment at the very end so the modern observability we need to build should look like this so we basically start with our developers writing some code they do that in their local environment and they run their applications locally what we

do next is we capture everything with like Telemetry open Telemetry whatever else and we capture how this application interacts with the database and we can then deliver that to the to a platform that will be able to reason about what's going on similarly we do the same for our cic pipelines we run the things we capture them with the collector and deploy to the pipeline and now

comes the tricky and The crucial part we want to do the same with our production database so we want to capture the interaction between like applications and the production database and also see what's going on what does this give us this lets us to do the projection so what we can do is we can take the application running locally and how it talks to the database and

then we can project that onto the production database to analyze whether the queries schema migrations and whatever other stuff that we run locally here at this point whether it's going to be fast enough when we run it against production so this way we can reason about what's going on early in the pipeline and tell our developers immediately that their Solutions won't scale well or won't work fast

enough so this is the very first part we want to do we want to bring the tools to the developers so they can reason about the performance early in the pipeline but the other thingy that we need to do is we want to capture the thingies throughout our collectors then understand how this things how they move from the local environment to production throughout cicd so we can

understand what is being changed when it comes to schema configurations indexes extensions and all the other stuff and how those things change throughout the pipeline so this way we can provide a full blown end to endend observability that gives actionable answers explaining that hey your production is now being slow because you deployed this thingy and it affected the production ecosystem in this way so this is the

idea with modern observability so the first part we need to do and the first part we need to provide to developers is the prevention part we would like to project the query onto the production into integrate with developers idees their environments and analyze how those queries execute and in order to do that we need to build the observability in the developers environments and also in cicd pipelines

we want to analyze those things as like a way of doing automated code reviews for our developers so this way they cannot only test their code with like unit test that will focus on whether we read and write the proper data but they will also get insights like okay your query works well when you run it in local environment because you have what 100 rows in your

database but the moment you go to production where you have like one million of rows your query is going to be way too slow you need to change it and this is how you change it because we know how the query is going to execute in production because we projected in the production and extracted the execution plan and we can even fix the code automatically by submitting

pool requests or giving like hints in the idees and the the other part we need to do is we need to build the observability that analyzes how things evolve over time how our indexes change how our configurations change how operating systems change our ecosystem and we can do that we can correlate the deployments with production changes we can correlate infrastructure as cope we can correlate gitops and

other stuff we can monitor database specific oriented metrics that will show things like transactions execution plans etc etc and visualize all of that and connect the dots and once we have that we can give our developers answers what to do and how to do it so this is how database God work they prevent the B code from reaching production they monitor the system end to endend and

turn raw data raw Telemetry into the actual observable knowledge and actionable insights and they can even automate the stuff they can troubleshoot fix automatically and give you things that you can just run so this is what it comes to tooling but the other part we need to do and this is where platform Engineers really need to shine we need to change the processes we need to reorganize

how our developer work we need to be proactive and push all the things to the left as early as possible our teams are slow because they lack permissions they can't access logs they do not have knowledge about databases they need to attend office hours to get help from other teams they need to do get explicit approvals and they cannot Self Serve their issues that is the most

important part we need to change the mindset of the databases of the developers cut the communication stop them from Raising tickets but instead own the databases and reason about them on their own so they can work within their teams much faster we need to build the knowledge and understanding to give them this way to reason about everything in the databases no matter whether it's like trivial stuff

or some lowlevel internals of the data basis so they can understand and realize that they are like now the owners and they can feel confident enough to reason about databases and finally we need to let them themselves serve their issues so when there are problems with the databases or whatever it must be them who tackles those issues not some dbas not some other teams but developers and

database God R is all about the mindset about the mindset that developers can own their software including all the parts of the software end to end from the very beginning to the very end and we need to build this mindset they can own their databases and it's not about giving them more things to to do it's about letting them own more but making them do less work

it's about replacing their communication with other teams raising tickets asking for permissions with things like getting actionable insights and getting true answers to what is going on it's about replacing part of their work with some other work that can be done much faster in summary databases may break in so many ways there are issues around like how queries execute around schema changes schema migrations around orms and

other stuff and we need to be proactive about that we need to not we can't wait for the low test to complete we need to push all these things to the left and be as fast as possible and we need to finally give the developers the database guard RS to link for actionable insights and understanding what's going on the understanding of all the internals and finally change

their mindset so they are owners and they don't need to ask for permissions and communicate with other teams but they can own everything end to end and not to mention there are tools like that mat is where I work is one of those so I encourage you to take a look at what we built if you like this idea and being all of that said it's time

for Q&A so if there any questions I am happy to take them now okay cool I don't see any questions out there maybe if there are then feel free to let me know thank you Adan yeah so that was a nice presentation very insightful do you see uh anything that that can be still uh right shift that is hard to be left shift well generally I don't

think we will be able to do this like light shift movement or left shift movement like in one step after all at the very end of the day we always deal with the live data right this is the data our customers most precious stuff that we can never break so automating fixes like automatically changing the code uh or tuning the configuration is probably something that we will

be hesitant to do at this point we may get like actionable Insight so the database gaers will show us hey this is what you need to fix but database gers will not fix that on their own just because we will be too scared to do so so I think this is something that will still be kind of right shift instead of left shift if we use this

wording and this is something we will get to probably over longer time but the point here is it's not going to be and we shouldn't expect that it's like one step and bang all is soft no it's going to be a journey and we will need to take many steps and the first step just like with devops First Step was to bring development and operations together now

let's bring databases together and then let's automate that so we won't need to to do it like manually or even approve the changes at all all will be deployed automatically that is something that will be slightly harder and will take longer in my opinion I have a second question uh I like the idea of the V I apply my daily basis but what I see when I'm

working full from the developers back in develop only developers is a resistance to uh include the the concepts anding they they work they has huge H how do you see that to apply the left shift and everything that you say that's amazing that uh the the biggest issue the biggest issue I see here and and it was cutting the internet was cutting so I didn't get full

question but I'll still try to answer the the biggest issue that breaks our reasoning about that is obviously the context switch right developers will not be happy to go to some yet another system to do this left shift they won't be happy to receive an email showing them that hey something is not right or whatever what we need to do is we need to integrate those things

into their environments as much as possible so just like when they are using visual studio code intellig or whatever if there is a performance issue it should pop up right there not somewhere else not yet another system it should pop up there similarly when they have code reviews if there are problems we ideally should show those issues inside the the the solutions they use for code reviews

so this is the first step to basically integrate with what they have now and just let them use it and overall I think developers will figure out how to make it better because they are very good at tuning and optimizing their environment so they will figure out how to you know put those pieces around so they are super comfy for them and convenient to use but the

very first thing is if we don't do that inside their environments but let expect them hey there is this yet another shiny system you need to use they will be hesitant to do so we need to integrate into what they have now to make the transition straightforward yeah nice yeah that's a very sightful thought yeah makes sense you cool thank you