Kalle Sirkesalo: What Made You Fast Yesterday, is Making You Slow Today

In this talk, Ker Calo discusses the challenges faced by development teams due to the increasing complexity of toolchains in software development. He emphasizes how a plethora of version control systems, CI tools, and other software solutions can hinder productivity rather than enhance it. The speaker illustrates through various statistics and examples how developers often spend a significant amount of time managing these tools instead of focusing on actual software creation. He argues that organizations need to reassess their tool usage, streamline processes, and prioritize communication to improve efficiency. Ultimately, Calo advocates for a balanced approach to tool selection that considers both immediate needs and long-term impacts on team dynamics and project outcomes.

[Music] ladies and Gentlemen please welcome our next speaker ker Calo presenting the topic what made you fast yesterday is making you slow today hello oh yeah so uh I'm Colo uh I'm here today hopefully it works yep uh to talk about uh yesterday is making new FL today and it's all about the fact that like if you have noticed we are kind of just adding more and

more to it like how many of you have more than one Version Control in use for example how many of you have seen more than two CIS in your company okay we are not in a mode of voting yet or then people just have a really good it infrastructure so who am I I'm K I work as a CTO of manag services for e code and this

trend that I'm going to be talking about is fantastic for us like amazing I get to talk to customers and I go there and I ask where is your source code and they tell me Well we have SVN bit bucket GitHub gitlab we probably still have perers some CVS and a few NFS discs that we put the code to and oh yeah Peter is writing to USB

stick because like he doesn't pre the internet yet and then you're like okay uh so we probably need to talk about this but anyway um I'm in charge of this product called rud which is devop as man service and during my Summers I sale and during my Winters I procrastinate like every single finished person and I've been doing 10 years of EIC devops so where this all

starts is the fact that like gitlab did a research that one4 of your developers time goes to different kind of tool chain issues developing tool chains or doing tool chains there was another research by Forester which is at this point almost 10 years old which was that 10% of it goes there so we can assume that at this point it's probably even higher than 25% but imagine

that you have a 5 person team one of those people is constantly doing kuber this Administration probably you have seen it someone's going to be doing Cloud thing terraforms or we need to tune some of the kid stuff all the other stuff like we're constantly spending time on things that are not creating the software so one of my colleagues did uh value stream mapping of what what

are is actually happening from creation of the ticket so identifying that we have something to getting it to production all the red boxes are manual steps that happen during the work so anything that has a red box stops everything and just everybody stop someone has to do something and this is the whole process of the customers software development and as you can see it always starts with

Exel like if you haven't seen an Exel driven organization you haven't seen any organization the big thing here is this this is where we create the value that goes to the customer like we have done value creation before starting the work because we have identified if the customer needs it probably we have done some uh mappings figuring out what is the specification but when the ticket arrives

this is where we are like this is what creates the value this is what the developer is going to do 25% of their time probably as as we remember remember from yesterday's keynote they're going to spend most of their times in meetings 50% if I remember correctly then they're going to spend from the left over 50% 50% of time tking their tools so you have 50% left

on doing development but then you have to also remember the cat pictures with ch CPT Etc so probably like 10% of the time we program which is good because lines of gold doesn't matter at all and we'll get so coming back to here value creation is not what we do it's value verification like we want to make sure that the system stay stable we keep releasing and

we keep doing things sa so that customers can keep working but why when we go to devop tools we focus on this part like every time we're buying a comp a tool like let's say you want to do something in kuus someone tells you hey let's take into use h b we're going to focus on this part like how do we deploy it what's the setup how

easy is it how fast can we deploy Walt into our kubernetes cluster I can have a Helm chart ready to go and then I I'm supposed to have a honeycomb here but like it's not working but then we have oh then we have all of the other stuff security best practices onboarding trainings how are we going to be upgrading it who is tracking the upgrades what are

we doing we are creating an uncont controlled cost that nobody is thinking about in the long term because we are just focusing on how fast is it to deploy which is fantastic in sense but it forgets the big picture that like hey why are we adding this and asking the why and this leads to the situation where complex tool chains slow you down so if you have

ever noticed having a complex tool chain doesn't actually speed you up so when you add more tools when we take into use more Tools in our cernus cluster when we take more tools into use in Cloud when we take into use even like a feature in certain tool we become more SC scarce and there is less and less people who know what is happening in our tool

chain like have you noticed that when you have uh more and more customized tool chain like you started with mayman for example for Java programmers I know there's a lot of java programmers here and then we keep adding stuff on top of that bit by bit you have less and less people who know what is happening and the own boarding time goes longer and we get farther

and farther away from focusing on generating the actual thing that we were talking about we're also building technical depth because every time we're adding more stuff into our tools our work working ways and things that we are working on we keep in coring that depth because we're not going to do it properly because if we add a tool every week every month every quarter we're probably going

to not do it so that everybody knows how to use it we're probably not going to have best practices we're probably not going to have a solution architecture for it and nobody's going to know why did we take it because we're taking them into use so fast that most of the team doesn't even know that we took it into use and then the question is should we

probably communicate about it but how do we communicate if we don't actually sit down and think about why are we taking it into use how many of you have ever seen a solution architecture description for example so sad document two people so sad documents are sad because they are super wrong text about the current situation that we are taking into use and why and they're incorporates constantly

news they have different names you also have star documents and so one but the key there is that you are trying to explain how are you managing these things and how do we avoid creating highly specific tools like when we're doing deployments to aure does everybody need to invent the wheel again or could we use the same templates again because our team is doing so different with

this VM deployments we need to develop this ourselves and like we can't use the other guys code no no you should probably just copy paste from one place and like have the encryptions enabled because now you forgot the dis encryptions because you were focusing on something else it affects everybody because you're keeping doing the basic things over and over again without thinking the future and then we

finally get to the last part like we have all these tools that are giving you metrics but if nobody like if one team is only using it you have a metric from here metric from here metric from here and they all collected differently they're all different and then you have what Darren in his previous security talk was talking about statistics and as we know statistics are just

someone's pick of the data that they have selected and most likely as we could look from Harvard Business study for example they might be lied or cherry-picked or cheated and no knows because who is who from us has ever gone through a 10,000 row Excel yeah so what this does this actually look like this like horrible situation this is your devop pipeline in 2023 I haven't done

2024 version yet but you start by development team here and you have different boxes that are yellow like Concepts that I've been trying to draw and then you have a bit more detailed Things That Go usually into one another and It All Leads to this like end user experience and application development over here and we have bunch of things that Mo most of you probably don't even

have so these are the concepts that we are working on when we are creating those pictures that I was sh the value stream map but in reality it doesn't really look like this it looks like this so you're probably finding your favorite tool and some of you are like where is my favorite tool from this I couldn't fit everything into one big picture but it's like the

whole idea is that like this is the landscape that we're working on this is the place where we're working on this is where every big Corporation is at the moment because every team is picking their tools doing things when I go to a customer I nowadays have a mirror Port where I start throwing based on their discussion about different tools that they mentioned to me during the

talk and my average is that I get like 10 different tools with that within one phone call and I'm like if I go deeper I usually find like 40 so we do all kinds of like assessments to figure this out and I hope one day I could actually release some data from it but not today but this is your devops pipeline so you might be to me

but we're doing platform engineering we're putting everything the Cooper need is everything is automated by backstage we're super far ahead and it's everything is easy well in the back end this is what it looks like when you have a platform team big boxes are missing still because I'm working on it that's why it's not artistic and so on so we have ridiculous amount of Concepts in operations

that we are all automating putting into a box and offering as a selfs service button and that's going to be fantastic for the 15 minutes that it works and then some platform engineer has to go and change something because Google just disabled Google DNS or something like that and now you're like okay we're going to move everything everybody through root 53 okay but now we need to

do a migration back to the old guys and oh and then you have a like guy who is like but how about my own premise kues I want the same features for it you're like okay I need to talk to the own premise uh and DNS and then the on premise DNS guys are like no no no no no you can just use the Route 53 you

go to the aw's team and they're like no no no no you need to go to AER because we are forwarding Dasher and you're like I just want a One DNS record how hard that can and then you repeat it to every one of these and you're in the Enterprise architecture but the good thing is we don't have to focus on all of these because like we

can think about our so software we can take a look back and think about what is actually critical so I'm not going to go to the Ops today hopefully someday but not today I have given some opinions here that like you could do server to remove bunch of boxes you could do manage applications to containerize it or you can do what most Enterprises do you just throw

it to someone else and make it someone else's problem why do you guys think sauce is such a big thing like why do we go to github.com why do we go to salesforce.com hubspot.com and so on it's because all of these become someone problem as said cloud is someone else's problem usually so what we can do in devops is we can about what does the developer actually

see so this is what your developer usually actually sees they see their ID so most likely everyone here has a different uh ID that they favor I know from yesterday's Java talk that equips is not really the favorite of everybody I know inj is high in Java developers but I myself moved completely to visual studio and visual code when they came to Mac cuz I like the

way of working with those but every one of us has our own opinion and most of my Engineers love whim it's it's like thankfully we don't have that many emx people with the B Because if you have heard emx pedals ever it is very noisy so I recommend putting the MX people into their own kopu instead of the open Office CU like it is it's wild what

it sounds like but it's really fast to program with those I tried the other thing that they love because like ID of course a developer loves their ID that's where they spend their time doing their favorite thing coding then they also like their instant messaging system like they might hate themes but they still use they might hate slack and they still use it and it's not because

it's easy it's not because it Mak makes them their communication easier it's because they keep sending J GPD generated cat pictures to another developer and laughing at them and if you haven't seen that happening then you most likely haven't talked to your developers enough but based on my knowledge most of my developers are not using jat CPD to generate code they're not using it to ask questions

they do that also with it but most of the time I just get CAG pictures generated by AI from them or anything else like incident ongoing at a customer someone generates an AI picture about the incident to make it funnier and that's fun like that is exactly what I need because like it makes the situation human even though it's completely AI but like it makes everyone relax

and think about what are we actually doing and that's nice CU then the people are actually having the conversation and not yelling about each other and then they have a third thing their Version Control System like I don't understand to this day how you can get such a big fight between people using gitlab or GitHub and they're using the same thing the other guys call it merge

other guys call it bull request but they can fight about it for an hour which is correct and why is it better and then the whole thing is just the same discussion goes constantly and you're just like it's all cold you're anyway going to pull it to your laptop to actually read it and build it and you're actually going to write it from your ID to Rie

just you can integrate your ID to your K if you didn't know but these are the three things that your developers most of the time see there is a fourth thing but I have for it here due to our slides being made for threee and the fourth thing is their browser and I don't want to know what they do with the browser like no but if you

think about this and you look at the picture that I showed you before this one how many tools here is the developer ever going to look at and how many of them are they going to Care the answer is three still so we could theoretically make it simpler and there's going to be a nice talk based on my talk yesterday with M uh gofman who is talking

on soundtrack about GitHub Advan security so we're not going to go deep deep into it but ideology is that you can simplify these things by looking at it and not remove the tool perhaps but make it so that the developer sees everything in his Pro request for example because he's not going to go and read the thing in sonar Cube we all know no developer is going

to go to sonar Cube and read about the code quality well okay I know there is someone here who is like I love my Sonar Cube asps and it's fantastic and I like this is a gold complexity growing over time and seeing the difference between BRS and so on I like that too but the reality is when we do changes for 6,000 developers it's not really that

they most of them are enthusiastic about one thing they are there to make code because they like coding or money that comes from coding but ideology is that you can integrate GitHub Advanced security for example to multitude of other security products and bring it visible to a developer who can then fix it without ever leaving GitHub their ID and working for there I could have removed jir

from here and Conference because you can also build smart comits has anybody ever used smart comits okay that's no hands so smart comits if you don't know which is based on the hands not drifting is that you type the EUR tiet uh short name and the number to the ticket and then you can put hash comment and it sends a comment to your year Version Control from

your Version Control to Y and you can actually never go to Y as a developer which is fantastic because I that's what I hope I don't need to go to I just go to pobi to read the reports out of Y That was supposed to be joke but anyway like ideologic like we need to think about more how do we get to developer to get the same

information we're trying to do here without them having to go across everything here and it needs to integrate to some of the things that they're doing here so you might be okay nice idea we could simplify it we can make it easier and that's going to be nice and how am I getting a funding to this well you do know this is not just an IT developer

problem to to this day I spent yesterday 15 minutes thinking about our our organization and the customer organizations and they're business people the people who are in charge of the business what are they doing this is how far I got in 15 minutes again Yellow Boxes are Concepts and they are connected by Pink boxes if I had enough time I'm pretty sure I would have blown this

thing up to the same level as the other ones and the thing here is all of this is going to be massive headache to someone most lik your security for example it's most likely going to slow them down because every time you're like okay where are the contracts well let me go to SharePoint to find the link to off Office 365 to find the link to uh

Google Drive where I find a link to an luminum or some comparable like contract management system and then you're just like what did you just say yeah yeah yeah yeah that's the fastest path I have found but this the link in Sal for yeah that doesn't work cuz I nobody ever updates it and then we get to the actual meat of thing it's not the tools that

are the problem it's the tools and processes being on alignment so when we take into the tools if you remember the beginning picture you need to actually think about why we are doing it what are we doing with it how are we doing things when are we doing things and where are we taking them because if we don't do that we're kind of in a problem so

if we focus on too few of the tools we're going to have a problem where everybody's going to be like but now we don't actually get the security ratings we don't actually cover XYZ but then if we take into use too many tools we have this situ situation that most of you probably have how many of you have over 10 tabs open on your browser yeah that's

what I assumed most of them are probably your different tools that you need to use during your day yeah some nothing yeah so the key is we have way too much swapping between our two day just to think about different tools that we need to use so think about the fact that like you're a newcomer to an organization that is a junior for example you're coming into

the organization and the first thing that they someone tells you yeah welcome to the organization so first of all let's go to year to find your own boarding tickets then we're going to go and update your data to our hatar system it's going to be in some hatar system like workday and we're going to go to the air earning platform where you're going to do your courses

on your compliance and so on and you need to do those every quarter or so then we're going to set up your VPN okay but to get the VPN we need to go to conference to get the information about the thing because that's where the guidances and by the way you probably need a phone so we should go to service man service now or your service management

to get like a ticket open for your phone because I forgot to order it how many of you have seen this in life yeah like and we see it as a completely normal thing like it's completely normal for us that when you go to a new company or even your company you might see five six tools coming into use during a one year and then you ask

like why did we do this exactly nobody knows like someone had an idea and like it was taken into use it was cheap so people were like yeah that sounds good and nobody actually did a business case for it nobody thought about what are we doing it and that led to that tool being just use so example of he's getting out of hand so we start like

think about this picture as an AI and you have J gbd it came out last January basically so Darren already bullied our CMO so I can bully him also so he started by taking into use chat GPT immediately he loves it and for good reason he's really good at using it he's really good at prompt engineering but he's made it look like this for everybody because last

time I talked to him he had seven steps I don't have the pictures because he didn't send me his workflow last time I talked to him he had seven steps to make up first prompt to your chat CPT so he started by writing the thing on a notepad then he took it to a uh tool that would generate based on that the estimated text that he would

need which he would then take to an automator that would uh like open up the terms that he had there which he would then take to widen the topics with another AI tool and then he would have this huge prompt that he could put into chat GPD that would generate him an answer that would actually give the proper answer that he was looking for and it is

like fantastic what he can get out of that pipeline because it really helps you like think about the topic and discuss about it and so on and and then you're there and you're like I just write this query into chat GPT like hey chat GPT can you tell me what's to what's the change management process but he may he like if that's the AI track that we're

going to have that everybody's going to do seven tools to AI like I don't I don't know what it's going to look like in over here at that point because if you haven't noticed every tool also has an own AI thing sorry I shouldn't rant about that but like our our recruitment system has an tool okay well no I don't have that anymore but it our recuitment

system has an AI tool that straight up lies to you when you read it and tells you for example that this person has 10 years of experience in axe you go to the CV and axe is not mentioned and he doesn't have 10 years in anything because he's a junior and you're sitting there and you're like well he's he's been alive for 10 years yeah and then

you're like this is the AI that I everybody's trusting so again coming back here we need to build some processes but if we build too many processes what happens most of you probably have never read all of your processes and policies has anybody actually read their company's policies and uh like policies yeah uh I guess we are 10 20 so some people have actually read them but

I like that for example Darren didn't with his hand who has been writing those policies in our so most people do not actually read these things because it's too long it's too many and we keep adding them so we need to actually also remove processes same as we do for tools so when you start your next transformation project around a tool CU someone told your CEO hey

we need to take into use let's say kit laab we are taking you to use kit laab you have to look into what can we remove then like what is it that we don't need then and then when you're doing that you need to also bring in some of the business people and ask them which proces can be removed because we're removing these tools but then because

it makes it sound nice and dandy and easy to change tools and remove processes yeah don't do it immediately like if you go into it and read like hey gitlab's features are these and you're like okay I can remove jir I can remove Confluence I can remove all my CIS I can remove my gitops platform because I'm going to use kit laab as a kops platform I'm

going to use uh klab for my uh feature flag and then then you go to the developers and you tell them we're going to remove all of these tools and you're going to put everything in gitlab you have six months let's go yeah it's not going to go well like uh probably better practice is let's start from slow let's think about what is the key things that

we want to get rid of first probably we would like every version control system to be the same or at least within the two things that we want them people to be we probably want those people to use the same security tools because otherwise our security doesn't have any idea what we're doing we probably want everybody to use the same authentication process also because I don't know

how many of you have but we have had in F code for example four to four authentications at the same time active due to mergers and Acquisitions it's down now because we have done good job of removing them but imagine having four different credentials for your normal work just to get to some few systems like okay I want to go read about atasan our latest year approaches

okay you have J One J2 j3 and 4 and now we go back to the original picture that I had and imagine those pink boxes duplicating or getting bigger and yeah that's what it is in like this picture if we don't do something to it it's going to BL up even bigger and this picture is going to be even worse than I have done here because I'm

not even done it looks like a mess already because Ops is not a pipeline it's actually like living thing that has bunch of things inide to it that we're trying to work out by putting everything in coer NES but then we're adding more boxes in the Cooper NES and then it becomes yeah it's not a rant about against coer because it's fantastic tool when you have a

professional team doing so thank you everybody I don't have any solutions to this like I like my solution is start the transformation like I've uh I sit in two different customers in large organizations one of them has a transformation reader who has an who has responsibility over all of their de Dev tools and now after two years he knows how many developers they have that was a

bit of a problem when we didn't know how many developers the company has just like when you start the discussion it's like I think we have developers and then the guy is like well well just called me and he has 3,000 more developers that we didn't about and you're like Bob just called you and you have 3,000 more developers yeah yeah yeah he had has this offshore

team in India that has been working for him and he's offloaded everything there but we don't know about them yeah yeah cuz they have a different email address because we did them on their own email Boxes Etc so we could control them better but nobody knows about them yeah yeah really really like really hard to start transforming when you don't know even what do you have then

we have another company that I work with and they had this situation where they were uh split off from another company and they had to bring in with them all of the old company stuff and when you split a massive Enterprise and you give them one year to do it so I get out of the systems in one year and take everything with you because you need

all of that IPR with your yourself how do you even do it well you take everything as is and deploy it as somewhere but now you became a smaller company with all of the Enterprise Dept that you had if you remember from yesterday Enterprise Dept also has an interest and now you have an Enterprise depth in a smaller company that has an interest of an Enterprise level

so we started killing systems quite fast with them because they realiz just like you know if it has less than 10 people we're going to kill it immediately and just migrate them to the new tool so it was quite quite a lot faster there in the beginning because we could just kill old systems immediately cuz nobody was is actually using them but actually the other customer that

I have is moving faster now that they got started because they had to build around the fact that hey we need to actually have a transformation lead he needs to communicate we need to build a communication pipeline we need to create adoption pipelines we need to create p factores and we need to start creating a community around this so people are actually inner sourcing if you don't

know what inner sourcing is it's basically your open source within your company and it's quite hard because nobody pays for anything open source just so you can imagine how much your company pays for inner sourcing so yeah as said I don't really have any solutions for you so good luck and yeah thank you go have a coffee we break oh I have questions uh yeah yeah yeah

you do have questions uh we will be having in I believe few minutes uh Kos I should say yeah and finish thank you Kus and Kos two very important words and I'm mistaking between those too uh so uh questions goes like that what is the balance between a tool to rule them all versus Liu way AKA lots of specialized little tools for single purpose so I would

say if you're a small company and you know the process you know know that the people know what they're going to use and when they're going to use them uh Linux way is okay but in large Enterprise if you don't actually like have a per like if you haven't made it so everybody knows the purpose you don't have proper communication you don't have proper management of it

and you just throw it to a team like most Enterprises do the single purpose tools get really complicated really fast in the bottom layer like for management it looks simple because you just buying different tools for different things but at the bottom if you need to implement it and manage it it becomes really hard so I think it's not about having a tool to rout them all

or Linux way it's about if I have Linux way of tools everywhere in Linux you can actually integrate all of them together by just putting a pipe or you can put an ad sign so you can actually have these tools integrating and working together in five minutes versus in software development if you take into use for example jrox artifact and you want to bring in son types

life cycle it's not going to go well if you don't know what you're doing and then okay but I want single reporting on artifact and there security FLW and getting it all running and you're like okay so now we start on data and now this project we're going to we're going to pull all the data oh we need to also bring the Kit data but we have

the secrets in kit we also have things that shouldn't be visible anywhere to everybody in the company so there is no Golden Rule for this it's more about if you look at it from the point of view of the developer if the developer cries when you look at their pipeline or can't describe to you how they do software development and releases you probably have too many tools

or you have too much uh like they are too far away from how you developing like deploying the system which means they don't actually know who customer so bad answer I know but I would say both but in uh like correct ratio and I would say I would go with 50% on the like tool to rout them all so most of the major things can be done

quickly and easily so if I want to start a new project I can do it with the tool to rot them all and when we decide to go to production and start doing like larer takeovers and more customers then we would have specialized with tools that we have decided on that hey we need this we need that we need to and we take them into use bit

by bit so we don't do I don't know if you know do N9 reports from cloud side but it is like this is everything you have fault in your uh cloud system and then user your management is yeah you need to fix everything that this report says and you look at it and you first time you see it you're like this is 300 things you probably want

to start by 10 10 most important things at the time because that's too too many already all right uh one more question from um slid platform but maybe there is questions from audience do you want to ask something quickly no if yes raise your hands if not maybe we'll see you at the ask me anything corner and uh yeah uh do you have any advice for situations

where your company has multiple uh subcontractors potentially with their own tool chains get all your source code to your Version Control System like that's my biggest CBE with any organization like if you have subcontractors that are bringing their own CIS or deployment tools and so on any data that you are paying for any IPR that you are creating or buying from them that they you own should

be in your version control system because if you're putting the version the data that you have and you're paying for in their Version Control System it's not really your IPR at that Point like yes legal it is but they might not really realize that it is your IP and they need to actually like tell you that hey we are intending to use this elsewhere so that's my

like number one thing then the other one is if they do all of the management and all of the things do you really have to matter how they are doing the deployments and so on as long as it's secure so you might want to bring in to that version control system that you brought in binary management where they bring in the binaries and dependency management so then

you can do source code tracking how good is the source code in general so that if you ever have to take someone else to use it you can actually know yeah the code is not completely horrible and you go know for example tests are being done and you want to know the artifacts are they safe and dependencies are they actually updating them and with that you actually

at least what's going on because you really don't want to take into any account like are you are they doing a I don't know are they doing Maven or C or comparable you don't care about that if you buy it from subcontractors okay and most probably last one um sonar cube is it worth it depends is the answer like in always in it so uh theoretically it

gives you a lot of feedback and the problem in software development often is that we get very little feedback on our work unless you're in very high efficiency team like you don't really see how many of you have met a customer for example in the aiio today yeah one guy two guys two people lifted their hands about meeting a customer like who are we doing this stuff

is the customer so we don't actually get any feedback from the our bosses don't give us any feedback like when have you last time gotten a feedback from your boss about your development yeah so sonar Cube at least it gives you some feedback about how you could be better but you need to remove the false positives