Lee Barnes: Continuous Performance Testing in DevOps

This talk focuses on the evolution of performance testing towards a continuous approach within DevOps practices. The speaker, who has extensive experience in automation and performance testing, explains the limitations of traditional performance testing, which often occurs at the end of release cycles, leading to delayed feedback and potential issues being discovered too late. He explores the necessity of incorporating performance testing earlier in the development process, advocating for a culture of continuous performance evaluation throughout the software delivery lifecycle. The speaker highlights the importance of establishing performance-related requirements from the inception of user stories, fostering accountability within development teams, and utilizing appropriate tooling for performance monitoring. He shares insights gained from a two-year journey of transitioning to continuous performance testing, emphasizing the need for smaller, unit-level tests and adaptable strategies to improve performance analysis across builds.

[Music] all right well thank you everyone one for joining my session today where I'll be talking about continuous performance testing in devops I want to give you a a quick introduction before I dive in um I'm CTO of Utopia Solutions and my focus over the past 30 years or so has been on Automation and testing and performance testing uh but relevant to today's topic excuse me most

of that time I've spent uh performance testing over the years uh has been focused on uh executing big system level performance tests that were really executed at the end of release cycles and uh we got really good at that we took the time to understand the user and load profile make sure we designed the right test we set up testing and monitoring infrastructure to ensure that the

the test was executed in an effective way um and and while that approach was very well accepted um certainly you could find similar scenarios in most organizations that had a mature performance testing function it had one serious flaw and that realization of that flaw has uh led me to declare that performance testing is dead so now hopefully that I have your attention let me qualify that statement

a little bit and add that traditional performance testing is dead now do I really believe that performance testing in general is dead no not really um but I needed a controversial uh statement really to grab your attention so let me blow up the fine uh print a little bit and qualify that a little bit further and say that traditional performance testing as the only evaluation of system

performance certainly should be dead so to understand why let's take a look at um what traditional performance testing looks like it was born out of the waterfall model where a single cycle stretched over many months and then after functional testing was completed we developed and executed a large system level performance test that often lasted weeks and I spent a good time uh portion of my career helping

organizations set up centers of excellence that became very effective at executing this type of uh performance testing um but they always had to enforce the often strict rules that went along with that um rules that that said if you want uh me to be able to test your application it better be complete uh because if it doesn't work in a single user environment how can I test

it under a multi-user um your code better be froze while I'm scripting and executing otherwise I'm going to have some uh some additional work to do and you need to provide me with a production-like environment if you want the results to be meaningful so this takes us back to that flaw that I mentioned earlier so let's say we do that um the organ organization decides to play

along with those rules and we design build and execute this system level performance test and it takes many weeks before we get any results what happens if we if we find issues at this point our options aren't good we either delay release which means we delay releasing value to our customers or we get hit by the freight train and that freight train is lost L Revenue lost

brand value lost Customer Loyalty um and it's also the increased cost of remediation uh for those issues compared to when they were uh or if they were uh found much earlier but either way our options aren't good and we need to find a way to evaluate uh the performance of our application much earlier and ideally as close uh as possible to the time when those issues were

introduced so The Logical Approach at least uh my first idea to approaching this was to try to perform this traditional performance testing at the end of each development Sprint the thought being that if I start relatively small and maintain that performance test as we build the application I can somehow uh keep up but of course it became very difficult to enforce those strict rules that I mentioned

uh when you take a more agile approach to to software uh development if you wait until new features are are functionally complete and tested there's really not time to enhance and maintain that performance test before the end of the Sprint and we certainly aren't going to freeze code uh for an entire Sprint or or worse uh multiple Sprints and certainly if we're early in our development it's

probably very unlikely that we have something that resembles a production-like environment so we end up combining those short Agile development Cycles with lengthy rigid performance tests and we end up with disaster and I stubbornly banged my head against the wall for quite a while um tried to solve the problem of shrinking down those traditional uh system level performance tests into something that could be executed uh effectively

and efficiently within a development Sprint um but after a while I realized I I was solving the wrong problem uh my goal ultimately was to get more frequent feedback on performance uh but I was I was assuming that the way to do that um was at the system level like I had been doing for many many many years um so I started doing a bit of research

uh and while there wasn't too much information on performance testing specifically there was a lot of work being done around this thing called continuous and really the rest of this talk is the lessons learned from an almost two-year Journey that I took within organization as we evolved from a a centralized performance testing center of excellence um that performed only system level performance test at the end of

releases uh to something that where we could evaluate performance more frequently more continuously uh as they built their software so let's let's talk about what continuous performance testing is and probably better to start with what it isn't um I learned the hardware that it's not just more frequent performance test uh it's not just moving performance test earlier in the development cycle um and it's not just faster

completely automated performance test although as we'll see automation certainly has a big part in it but it is the appropriate level of evaluation of performance at each stage of the delivery cycle is more frequent more visible feedback on performance um and an important aspect of this is continuously trending that feedback across builds so we can create a Continuum of performance data uh as we develop our so

let's see how we're doing if if doesn't fit in agile devops Cycles right but there's certainly a lot of organizations that are building software with with a devops approach agile devops approach let's take a look at at at how they're doing with respect to Performance testing and now this data here is a bit dated um but based on what I've seen at least uh things really haven't

evolved much so according to uh state of devops report even if we look at Elite performers only 28 % are performing some type of automated performance test or performance testing is part of their uh pipeline so if even the elite um aren't performing or aren't executing this continuous performance testing at a high rate there must be something missing so let's let's see what that might be and

I I've noticed that there's quite a few things um first is a lack of performance related requirements and user stories if we're not thinking about performance testing from the beginning it's difficult and and probably unfair to expect the team to think about it as as features begin to move through the uh delivery cycle and and really going hand inand with that is development teams not having the

accountability uh for performance testing analysis right especially in situ ations where uh traditional performance testing was handled by an outside team similar to that uh development teams uh often lack the knowledge and experience for performance testing and this is again especially true where it was someone else's problem right handled by someone else or or another team and finally um uh the concept of of performance analysis was

limited to running large scale system tests uh I'm sorry system level performance tests and the organization as a whole just didn't possess the skills or tools uh required to to analyze performance again right there was probably some other team that was doing this so what can we do about those those issues I'm going to go through each one here one by one and talk a little bit

about how we can address um uh those issues again one of the the huge holes in most organizations is the failure to think about performance from the beginning uh and again many times this hole was was created in older software development processes where understanding the performance of the system was someone else's responsibility um uh but I found a great way to address this is to consider performance

requirements and and I'll call them factors um when stories are being created so as you're creating your stories think about these factors uh from a performance perspective that are important to that story for example let's let's look at this one here uh related to uh the feature that describes the ability uh customer in in various ways maybe I know that I have a peak through put of

around 990,000 uh searches per hour and given that we need customer account information um and the products that they have subscribed to in less than 5 seconds I know that um that that has to be displayed to my my customer uh support agents within that time frame um although I want to mention that one thing that we found um when we started doing this we started calling

them performance requirements and we started to get into a situation where we were kind of paralyzed by the analysis um we were trying to get very detailed description of these performance requirements and the organization just wasn't ready to um to think about that information inform so we spent a lot of time uh and and kind of created our own bottleneck but when we started calling them factors

and took the formality away from it just hey you've got this story think about how it may be impacted from a performance perspective um and then things started to flow uh much better um of course at some at some point we have to incorporate uh these requirements more formally in uh in our work and I've seen this done uh a handful of ways uh probably the most

popular is as uh constraints right something that all stories or maybe most stories uh uh must meet so if you think about performance requirements response time support for a number of concurring users or some volume of throughput they often apply to many features or user stories so as we create these stories rather than copying the acceptance criteria from story to story we can just simply link them

to these uh uh these constraints of course you know uh we're going to have stories that have unique uh acceptance CR criteria related performance perfectly fine to document uh the performance requirements um uh per story um before I move on I I want to say that how you approach incorporating performance requirements uh should fit with the way your organization Works um the key is to develop the

awareness of the performance factors as you're building the software rather than waiting for someone else to think about it at the end of a release cycle so I I found the method by which you do that is less important um to start with what makes sense for your organization and uh you know adjust as you go to find your best way uh of working so next we

talked about um and it it's not surprising right that we see a lack of accountability for performance um there's still a lot of inertia around that traditional performance testing where was someone um we talked about awareness and um I believe awareness is an important prerequisite to accountability right if our teams aren't aware of what we're trying to accomplish and why it's hard to hold them accountable um

as we just discussed important ways to to build that awareness are to discuss performance factors and then incorporate uh performance requirements uh within the way we work but we also need visibility right awareness is one thing but if we're not constantly reminded how important performance is we're likely going to let it slip into the background and I found the best way to keep performance visible is to

ensure that our definition of done requires us to document those performance factors and performance related acceptance criteria uh as part of our stories um I've also seen some organizations uh create the role of performance owner one organization even called it a performance Zar and it's important to note that the responsibility of this performance owner um is not to own all things related to Performance but rather to

ensure that teams were adopting a culture of performance accountability um they were responsible really for clearing the way for performance by providing guidance and ensuring the teams had what they needed from a performance perspective um accountability is one thing right but how how can we expect our teams are going to magically have the skills and tools to implement performance related processes uh into Cycles um you have

to start building that ex uh expertise and I suggest one team at a time uh if you're part of an organization that has or or had a performance uh testing cu you can bring those experts into your teams to start uh seeding them with with expertise and once the team begins to self uh uh Su uh sustain themselves you can begin to expand that to uh to

other teams and in parallel I found it's a good idea to develop a virtual Coe that can provide guidance and oversight for performance related uh activities I'm not talking about heavy process or oversight but but just a virtual community that can work together to share ideas Lessons Learned um to ensure continued uh Improvement and and growth um you also need to think about bringing the appropriate tooling

into Upstream environments uh not just lad generation tools which may already be there but also performance monitoring and Diagnostics uh solutions that are going to help you understand uh more than just the user level uh metrics and I I found that this has um really helped us transition from a a situation where uh performance testing was a centralized responsibility to some to somewhere where uh it's now

part of the uh the delivery teams again this doesn't mean that everyone on the delivery team is a performance test or performance engineer but it does mean that everyone is aware of performance uh and has the appropriate responsibility relative uh to their role so um next I want to start taking a look um at what some of these performance uh test responsibilities might look like as we

go uh down our delivery pipeline so um uh let's take it I've set up here a relatively simple uh pipeline just so we can start to talk about what we can learn about performance um at each stage so of course we can learn about performance in the performance or load uh environment this is likely we're going to execute something that looks more like a traditional performance test

but as we'll discussed um we're we're likely not going to execute that on every build um because hopefully won't have to we'll be uh continuously analyzing performance um certainly learn uh about information at the integration stage this is where we're going to focus on individual components as we'll see or smaller groups of components um tests that can be executed relatively quickly and have uh few dependencies or

have their dependencies mocked um we'll also discuss how we can learn about performance and production and not from a just a traditional uh production monitoring perspective but using other techniques as well um we might even be able to learn something about performance when code is committed depending on what we're doing but um as we'll see the key is to evolve to a state regularly evaluating performance uh

so it becomes uh easier to track right regressions much closer to the commit where they were introduced and the old traditional way of performance testing much less frequently just leaves many possible culprits that require investigation so let's quickly uh look at the commit stage I don't see a lot of organizations doing this um but there are some things uh uh to consider I've even found that there's

some tools and libraries that were purpose-built uh for unit performance testing um this table here is something I gleaned from a relatively Recent research paper out of Charles University in Prague um that lists the most popular unit test uh Frameworks but popular again is a relative term here I think they reviewed something like a 100,000 Java Base projects and only found that about 3% of them uh

had some form of performance there are some important considerations um these should be true unit test right dependencies mocked we're just using multiple threads and multiple iterations to gather some performance uh uh related data um also some important considerations um probably the two most important here are controlling the length of the test um and how results are are presented um first test would have to run too

long to get enough data to give her high level of confidence in very small uh performance fluctuations therefore we need to understand that we're going to detect only the obvious changes in performance uh which should be fine for for most applications um second is the absolute value of the results is much less important than the trend from build to build right we're not going to be in

anything that resembles a performance I'm sorry a production environment at at this point uh so we just want to Trend those results from build to build and see when we start to uh uh to go in a in a bad Direction uh moving down the pipeline we definitely learn about performance at the integration stage and and this is where uh We've checked in code unit tests have

passed and now our CI server is deploying the new build into some uh integration environment so so what does that look like um well first of all it's test focused on individual components and these should be your most critical components at first especially if you're just getting started uh right we we can always scale once we've gained some experience and and learned about how to uh best

approach this in our environment um we're still early in our pipeline we want fast feedback um so these tests uh really just include individual calls uh to endpoints that are executed by a small number of concurrent threads for a small number of iterations just to get enough load and enough data that we can uh use to Trend across uh our builds um depending on the maturity of

our pipelines uh spec uh specifically how well you control environment configuration and data you may want to limit the complexity of these tests and mocking out the dependencies of the um we should also think about measuring not just response times at this point but also um application infrastructure level metrics by extending our our APM tooling uh to these lower environments and again there are some important things

to consider um um we want to execute these on every build um so we can Trend the results across time uh but similar to uh to unit performance testing right these tests are not deterministic like functional tests are right so we probably shouldn't have failed test uh break the build and it's also uh again right these results less important to think about the absolute results more important

to think about trending uh uh uh trending across builds so um now I you know the picture that I've painted so far and we'll and we'll talk about the the implementation approach in a second um but it's still relatively mature Implement implementation I I would recommend that you don't go back and try to recreate this immediately in your environment not because it's necessarily difficult but more because

you want to build it in a way that is sustainable by your organization provides value to your organization most organizations will start with just a single component test maybe even triggered manually at first just to get the feel for the mechanics of doing this the type of data it produces and and how to consume and and react to that data and we'll talk more about that in

uh in minute so before I move down the pipeline I want to dive in here a little bit we just talked about testing performance at the unit component level but you know let's not kid ourselves for many applications that's much easier said than done for microservices based architectures it's much more intuitive right certainly at the component level to design test each component is is likely relatively focused

in its scope and identifying its dependencies for the purposes of understanding mocking or service virtualization requirements is relatively straightforward and they lend themselves fairly nicely to very short focused test that could be excuse me that can be run efficiently um Upstream in your delivery cycle however for for monolithic architectures the idea of a component can sometimes get a bit fuzzy and and therefore the ability to execute

nice neat little tests that are focused on a single component uh can become more difficult um but while you may not be able to accomplish the same level of of compartmentalization there's still some things that we can uh uh do in in this type of situation uh a good place to start is to map user actions to specific areas of interest in the application this might be

a specific process in the application or business logic layer could be a specific stored procedure in the database um so rather than thinking about testing from a user perspective think about areas within the application that may have performance issues or something that you're interested in understanding um then identify the external inputs that will exercise uh those areas of the application um and your test script should only

include the minimum steps necessary to do that right avoid the traditional endtoend Journeys as as much as possible um and certainly for for monoliths the the scope of performance test you can do at this level um may be limited right because of the reasons I just mentioned the tests are likely to be a bit longer and they're likely to have more dependencies therefore the time to maintain

them is going to be greater um of course you the the ability to Mock dependencies and data might be limited as well and require more setup time so let's move on down to the acceptance stage um oftentimes it's common to have an inline environment where additional component tests can be run or or you know small subsystems of components um and also that out ofline environment where more

traditional system level performance tests uh are performed for the inline environment this is likely where you're going to be running your automated acceptance and regression tests as well um I recommend capturing and user level metric as part of those automated uh functional test and this can be added to most Frameworks fairly easily so typically will capture response time certainly um uh as well as uh tapping into

the the browser level performance metrics and and save these off uh you know across builds the same way we do the rest of our data uh it just gives you a good single user uh Baseline of uh of what performance looks like um again you'll ideally be executing your component level test on every deployment um with the dependencies mocked perhaps um uh if you've got good control

over your data and your environment uh I think it's best not to mock them at this uh at this um but the key is that you're evaluating performance throughout the delivery cycle right you can be more selective um uh uh with the larger system level performance test test that you would execute um uh in the out ofline environment um and certainly for those system level performance tests

all the recommended practices here that we know about that they they still apply there's really no magic um you need to make sure your load profile the number of users what they're doing their throughput is you need to understand the test data requirements uh and prepared uh for that um you should be prepared to monitor not only the user level metrics but the system and infrastructure metrics

as well um I have seen organizations execute this level of testing more frequently as part of automated deployment but it's rare right and you environment provisioning and configuration data setup need to be part process um and the hardest part really is making sure that the tests themselves are ready uh for execution and haven't uh been impacted by application changes all right so before I go and um

testing and production I want to give a a conceptual overview of the process and and some of the representative technologies that can be used to put this all together um as you'll see I've I've listed some tools here they're just examples not necessarily uh recommendations just to give you an idea of the types of tools uh that I see organizations using in this in this approach um

first we'll start out with whatever we're testing right we need to have some tool to generate load and and throughput um and remember that we want more than just uh user level metrics so ideally we're monitoring the application and infrastructure while the tests are uh are executing uh next we want to collect the results both from the load uh an APM tool perspective ideally in a Time

series database some of the load test tools do a good job of allowing you to uh collect results across builds and Trend them others don't have that uh that view that can be easily created uh so we'll we'll dump them to a Time series database uh right and then um uh have some mechanism to uh to display them um and of course we want the uh entire

process to be as automated as possible and you might be thinking at this point hey this seems like a lot of work and I don't see how I can possibly carve off the time with everything else I have going on to uh to implement this and you're right that this isn't a trivial amount of work and as I mentioned this was you know you're kind of seeing

the end result of a of a two-year Journey um so it's important that you understand you don't have to uh build it uh overnight you know again a good place to start is just executing a few simple component tests that are executed automatically as part of your um cicd process um but one important aspect of of doing the work is that this should be part of your

technical backlog or whatever you call uh the backlog that tracks kind of your continuous Improvement efforts right if this is off the book's work and we found this out the hard way meaning you know it's just not tracked anywhere it it will not be prioritized appropriately and you'll just never get to it even you know we had the best intentions but uh for a long time uh

we just weren't making much progress because we weren't incorporating it as part of the way we uh the way we worked so that was a lot of information that we just talked about and I I want to take a few minutes here just talk about some lessons learned and and around the implementation approach again this was a two-year journey and we did a lot of things wrong

and we had to Pivot and and and go in a different direction um and I think you that's just kind of that's the devops culture right um but if I can save you some time and and at least point out some of the the uh uh the very painful points and some of the the lessons learned that are are probably going to be more common I think

will be will be helpful um so I recommend first you start small um a good place to start is again running some component level tests on a few of your more critical components get comfortable with that process and the tools that it takes to execute um uh and get the information in a in a consumable state um you can then work out the Kinks and scale the

other components and eventually other teams um a a big lesson learned that we have when we first tried this we we tried to implement it across a large number of teams simultaneously and um it was just a disaster while we had the the the right number of people with the skills that we embedded in each of those teams they had different sets of problems and um you

know they they were depending on who was in the team different uh levels of Readiness to accept this and we just kind of ground to a halt in terms of progress so uh when we kind of pulled back started with a team that we felt was most ready uh to this um we got the process down got the Kinks worked out we were much more successful moving

um make this part of your technical backlog right um so it can be prioritized along with the rest of your work uh from experience this type of stuff just doesn't get done if it's something the team uh kind of just keeps in the back of their mind waiting for some some free um part of getting comfortable with the process is not overwhelming your team right again another

reason to to start small this is information that the team likely hasn't received before at least not with any uh regularity um you need to prepare them so they're not only ready to consume it but understand it and most importantly be ready to uh to act on it much of the value of performance testing is in the analysis and we can't assume that teams possess the ability

to understand the data that we're giving them and taking action on it this is going to be a learning uh a learning process um and probably you know again we we started just dumping a lot of data on the teams and they tried to keep up they tried to understand it but pretty soon they just started ignoring it um and then we pulled back and just started

feeding issues that we saw to them and then we would walk through how to analyze those issues with them and once they understood the process um they just started asking for more information we didn't have to push it on them um so I a good approach here again just feed the data slowly start with maybe only feeding issues um and then uh open it up as the

team becomes more comfortable um on um you need to take care of supporting factors right this isn't Magic you're just not creating scripts and executing them so um are you thinking about you know your test environments and are you thinking about how you're configuring them and how you're conf controlling data in those environments so are these tests ready to run um uh on demand um moving on

thank you all right oh this was this was a huge one I knew there was one more um when we started doing this for this particular organization we were part of a test performance testing center of excellence and we assumed that we were the experts we were just going to tell them what to do um but uh it it was also a new way of doing things

and when we started to solicit feedback both in of you know what did they understand about performance testing what types of things did they want to learn um we actually learned some things too and when we started to solicit their feedback take that feedback into account make some some tweaks to what we were trying to do we started to to move much more quickly um so as

you're implementing this against the teams ask them do they understand what they're doing do they have any ideas about what and uh as we found out you're likely going to find out that there's a lot of you know knowledge you didn't know about uh within these teams it just had never been used in that way before okay before I wrap up um let's talk quickly about uh

performance and production um I think most of us are probably monitoring our apps in production or at least we we should be um but testing production back when I first got started you didn't do that at all you could be thrown out of a room for even suggesting it but it's fairly common today as as an important part of a continuous performance testing strategy um from a

testing perspective the obvious benefit is the confirmation that production can support you know some expected scenarios um but also allows us to test the entire system not just what's behind firewall um and with the wide availability of cloud-based load tools we can evaluate the user experience from various locations and network conditions uh you know all over the world and we can combine uh testing and monitoring to

give a specific data we want by executing synthetic transactions and production um similarly we we uh should create continuous feedback loops between Upstream testing and monitoring uh to improve both test design and our configuration of course though right testing and production isn't as easy as pointing your test to the the production environment there's some things that we need to do um to maximize the benefits we get

but of course minimize the risk because we're we are in produ ction um one mistake I see especially from the performance testing side is creating this strategy in a vacuum right having some performance test engineer just come come up uh on their own with what they're going to do you need to get input from across the organization so you can put together a comprehensive plan um that

that is safe and adds value um if your apps hosted in a third party environment you're going to need to coordinate with them if you don't you you risk making them mad uh at a minimum or or likely being blocked because they think it's some type of uh uh denial of service attack um of course if your app is already live um you have customers and you

need to either find execution windows that minimize the impact um or possibly cordon off part of the environment to handle customer traffic while the test is run on the the remaining part of the environment um next and this is important the system system needs to know that it's being tested right we're in production we can easily um uh create issues with the business right depending on what

the application does um so the application should know what being tested whether it's seating test dat in production or some other means that the system knows hey this isn't a a real transaction um it makes this a lot easier um so if you're if you're building a new system thinking about this from the beginning makes it a lot easier um if not you're probably going to have

to evolve the system so it knows it's it's being tested in production um finally it's a good idea to eliminate or mock request to thirdparty Services uh so you don't unintentionally impact their customers as part of the uh as part of these tests so uh quick um quick wrap up um if you take nothing away from from this talk you know this is this is probably the

most important points first traditional end of cycle performance tests just don't work well with with more modern software development uh approaches um continuous testing is just not more frequent traditional performance test it's more of continually evaluating performance as appropriate as you go through your uh um this needs to be part of the way that your organization delivers software right again it's not the performance test team that

swoops in and executes this it's executed as part of the delivery teams you may have performance Engineers or performance test Engineers that are embedded in some of the teams and and perform some of the heavy lifting but this is this is needs to be part of the way that you work not some external function that that gets um finally again start small get feedback and of course

important part of the devops culture um continuously improve so let's uh let's move on to questions if we don't get to your questions today please feel free to reach out to me at any of these methods here um happy to talk about this stuff uh this stuff anytime if we got any questions here I don't see any coming in if anyone has any questions please put them

in the chat I'd be happy to answer or anything right hope there's no questions we can cut it off but again please feel free to uh reach out to me at any of these uh any of these methods and I will be happy to um talk about this stuff anytime thank you