To Measure or Not to Measure: The QA Metrics Dilemma - by Maryam Umar

This talk addresses the challenges of measuring software quality within QA teams, led by an experienced engineering manager from Booking.com. The speaker explores various metrics that can provide insight into quality, such as defect rates, test coverage, and automation levels. She emphasizes the importance of selecting metrics that reflect real quality, avoiding vanity metrics that may mislead stakeholders. Throughout the session, the speaker shares insights on assessing both product metrics and process effectiveness, advocating for a balanced approach to quality assurance that considers the perspectives of different stakeholders. Ultimately, she aims to equip teams with effective strategies to quantify and communicate software quality in a meaningful way.

Challenge accepted. [music] Challenge accepted. Challenge accepted. P2 [music] is always challenge accepted. Please take your seats. and welcome back. We've already talked about chaos, about AI, about prompts, and now it's time to talk about a topic that divides every QA team in the world and makes them argue >> the air conditioning temperature settings. >> No, metrics. >> Metrics. To measure or not to measure, that is the

question >> because sometimes numbers bring clarity and sometimes they bring only confusion. >> Is it the fact counts, test coverage, release quality or automation rates? Or is it just counting how many times your manager ask? >> Our next speaker knows this dilemma better than anyone. She has more than 18 years of QA experience across finance, mobile, and e-commerce. >> She's an engineering manager for quality of atbooking.com

in London. >> She's passionate about using metrics that actually reflect real quality, not vanity. >> Please give a big welcome to [screaming] >> Umar. Welcome. The stage is yours. >> Hello. I just noticed the mismatched socks. Good one. >> Right. When you say 18 years, it's customary to wear glasses on stage. This is what happens when you look at numbers for too long. Right. Everybody just let's

just come in quickly. Uh come to the front. I think they will this will be a good talk. All right. Um first time in Sophia for me. It's Sophia. Somebody told me and not to say Sophia like the name. So I will remember that it's Sophia. Um really excited to be here. Um I'm looking forward to doing some sightseeing tomorrow. Any tips please give them to me.

Right. So um this is what we will talk about. We will discuss how you can measure quality. What are some of the metrics you can do? What are the mistakes we all make? And then we want to make sure we're measuring the right items. Little bit about me. Thank you to the hosts for the welcome. I do work for Booking.com right now. Please do not come to

me for discounts. I don't get much anyway. Um yeah, I've been around for a while and these are some of the companies I have worked in. I came into management by choice. So I was a quality engineer and I really struggled to think about where do I go next. Um this was when I was at Expedia group again a travel company and I was told you can

become a tech lead you can become a developer you can become a scrum master but I am married to the world of quality assurance. So I did not want to go away and I said let me try and do some strategic work and do some managerial work. Nine years later, I'm still a manager. Thankfully, some fun times, some not so fun times. And the not so fun

times are when you're trying to The thing with quality is, let's talk about the ice cream downstairs. I think a few people are eating them. What flavors are you eating? Would you like to share? >> Salted caramelist. >> Pistachio. Same. Same. What's the last one? >> Double chocolate. >> Double chocolate. Right. For me, double chocolate is the quality is for you probably. Oh my god, it's amazing.

Quality is subjective. What's good for one person may not be good for the other person. And this is why we have a challenge as quality engineers and test engineers and test managers in this world. how I feel when I use Spotify and when I'm making my playlist is not how another person feels. Or sometimes I go in and I say, "I think the algorithm for Discover Weekly

is broken this week because my recommendations are really bad, but for some people the recommendations may be amazing." This is why we're challenged with measuring quality. And we're always asked, can we go to production? Yes or no? And some of the things you think about, oh, do we have blocker issues? No. Did we run all the automated tests? Yes. Can we go to production? Maybe. How do

we put a number behind the maybe? This is something we've all hear a lot about and sadly a lot of the people in the group. Everybody is a QA professional. Any person in product product manager, product owner, head of technology. Okay, these are usually the people who push us to run quickly. So if you know the speed, quality and cost triangle. I'm not making a heart trying

to make a triangle. Um, usually quality is the one which gets lost. We're thinking about speed and cost. Can you deliver faster? Can you test faster? Can you test quicker? Why is it taking so long to run the test? Well, it took long to write code, but that's something we are usually asked to um I'm forgetting my English. Usually asked to forget about and what we really

need to think about when we think about metrics is efficiency. How can we be efficient? So the quality is there, the cost is minimized and the speed is acceptable. Something we've all seen a lot of is sonar cube analysis. Have people use sonar cube? So sonar cube exposes some metrics like this which is test coverage, static code analysis, cyclomatic complexity and so on. And these are some

of the things we usually look at for measuring quality. So if you have sonar cube linked to your services um what you then see is okay this is what the code coverage is like it's 70% 50% this is what cyclomatic complexity is and so on and some people say okay whatever sonar cube reports we will use that as a quality metric and determine if this service is

production ready not so sure because you could have amazing tests written at a unit level, but writing assert true will also make a test pass. And we do not have the bandwidth to go through every single unit test and making sure that's not what is happening. Please tell me if I'm moving too much. Is that okay? Okay. These are some of the things we should think about

when you're thinking about metrics. Maintainability. How easy is it to maintain the software? Do we have a metric to measure that? How efficient is it? What's the performance metric? So if you're measuring if you're measuring how much how many transactions you can do in a minute for example, it's good to have a threshold and not just report the number. Does it make the software efficient or not?

Then we have usability which is how easy is it for the users to interact uh with the software. So for example, I was just doing a quiz in one of the booths and I realized whilst the quiz is usable, I had no reference to it because I don't live in Bulgaria and half more than half of the questions were based for um based on the um places

in Bulgaria. So I couldn't win anything. Sad. Um reliability. How reliable is our software? How much uptime do we have? Service health, all of those things. And then functionality. Do we have end to end business flows covered? Here is an example of some of the metrics I have used before. Yes, take your pictures please. These are useful metrics or metrics you all have seen. Defect rate. It

could be how many defects have we found, but then how many defects have we found in an environment? It's really up to you how much indepth you wanted to go with the metric. It could be just metrics based per service. It could be metrics uh sorry defects based per service. It could be defects based on a group of services which are used to deliver a piece of

the product together. [snorts] It could be how does the whole product behave and the defects reported in say a pre-prod environment. It's really up to the team and how we want to use it. Some metrics related to test automation. Something I'm a big fan of is how many scenarios did we actually automate. Automation is also something which I see goes out of the window. It is going

out of the window a lot in my current team and I am very perplexed because I have hired automation test engineers but because we have to deliver faster we do not have time to write automated tests but why is it taking so long to retest the bugs? because we don't have automation. So this is the data which is useful in those situations. How good are your tests?

Do you have flaky tests, non-flaky tests? Having badly written tests is as bad as having no tests. Then some metrics around regression tests, meaning how many tests do you have in your test suite. Why is this important? regression and smoke test in my head, at least from what I've learned so far, and again, I would love it if somebody thinks I'm wrong and we can have a

conversation about it. Um, the regression tests, smoke tests are a subset of regression tests. Smokes, smoke tests, you run them in your pipeline, and you should get a response within a few seconds to know if the product is solid. But regression tests again should also be a small set of tests. Then you have your functional tests. I was speaking to uh someone yesterday, day before, sorry, my

days are a bit of a blur and I can say why. Um where they were saying that they've just joined a company where they have 60,000 regression tests run manually over a two week cycle. And I said, "Your people are very fast." Um, and he was talking about how do I know which tests can I remove from the regression test suite. It's not a regression test suite.

I just think it's like a some sort of tough murder or some sort of exercise that they're doing every two weeks. Um, meantime to green. This is something which can really help you understand how quickly your team can fix bugs. The way to test is in your test environments first, but then it really helps you understand when there's a hot fix situation in production. How quickly can

the team fix the code? Some more metrics uh which are not all testing related but speed of development. Now this is some a metric which a lot of people currently um the place I'm in last year we had a big debate about this. This was about how many merge requests did a developer comment on and it was used for performance reviews. Not right. It is not right

to do something like that. But the speed of development can really help you um the size of the code and how quickly you're trying to develop it. something I read I think in one of the books I cannot remember the name right now which is very bad um is that the planning and design phase for any project when you do that that time is not finite but

development and testing and delivery is something which should always be predictable for the teams defect rate in relation to automated tests this is where you really understand if you're writing the right tests uh whether the defects you're finding are they being found by your automated tests or are they being found by manual tests and that's where you figure out okay do we need to change the test

data perhaps for our automated tests or not um acceptance criteria how good is the acceptance criteria again this is a very subjective uh metric if I again I'm standing in a country where English is not the first language I come from a country where English is not the first language uage but what clarity means to me when I read an acceptance criteria may not be the same

for someone who's a native English speaker and I think this is where it becomes really difficult but then for quality of acceptance criteria I actually go down to you see a lot of tickets in Jira I'm assuming people use Jira Trello quite commonly they don't even have acceptance criteria what is supposed to work as a result of this ticket being shipped we don't know it and those

are the things we need to understand as testers. And then the last one, code coverage. Very easy one. Unit test integration test coverage. If someone has found a way to measure integration test, please please please come to me afterwards because I'm really struggling to figure this one out and it's quite challenging. Um, swagger coverage is something which people think about but it doesn't tell you whether the

three or four services which will work together which will will they actually perform or not. How do you measure it? Do you write tests for every single endpoint and not and so on? This is far too many numbers and this is too much of a headache before lunch. Yeah. Um and this is how I feel a lot when I see numbers. Some numbers make sense to me

but the numbers don't make sense for stakeholders. All they really want to know is when will you ship this? How quickly can you ship this? Why are there so many defects? Why didn't we find this before? These are some of the things we are told. it's very important to think about these questions when you're thinking about Who is the recipient of the numbers? Who are your stakeholders?

How are you presenting the numbers? And what is the expectation from the recipients? The stakeholders. When you're specifying a number for quality, what does it mean? I have just realized I am on a timer so I will go quickly now. The real goal is to try and put one number behind the quality of your project or your product and to focus on the product rather than the

people. So don't say this team went into production and they had 10 blocker defects. No, you will not make any friends. What you really need to say is this set of services went into production with X number of services and this is how we could have stopped it. The narrative is extremely important Don't create a long document of metrics. I once created a 37page document. I will

I'm not kidding. It had lots of really nice graphs and I was really happy and so and I was like nobody's going to read this. It got my information together, but it didn't help the stakeholders. Um, and when you create metrics, it's important to say you can't just report, oh, we have 10% code coverage for this service. That means nothing. What you want to say is we

should try and aim for 80%, I see quality engineers and test engineers as people who are advocating for quality of the product rather than just putting numbers out. We need to encourage people to do this. I will go to an example. Wait, is this the wrong slide? Yes, it is. I need to switch laptops. I didn't test this clearly. One second. Happens when you do two conferences

in two days. all come up. I know it's a bit blurry, but I'll try and So, what can you do? We want to come up with one number representing quality. This is a first time for everything. I'm embarrassed. Please, I apologize. Uh, we have product and process metrics separately and then you can group these metrics and provide a value for each one of them. And I will

show you a working example very quickly. So what I've done right now is I've come up with a long list of metrics. One is a group of process metrics. One is a group of product metrics. Why do we want to group them? Because the product could be working well but the process to deliver that product is what is broken. Hence the quality of delivery is broken. Some

examples of process metrics. How many end to end tests pass the first time? This means the team was thinking or not thinking of integration tests. What is the automation coverage for end to end When did the test case design start? This doesn't mean putting in a date. This is more about did we start at the same time as when development started. Did we start after the code

was deployed to testing environment, pre-pro environment. This really helps us understand if you're actually functioning as agile or waterfall teams or water scrumfall or mini waterfall or whatever words you want to use. Um how stable is your system? Um did the development team complete deployment on time? Because if the development team gets delayed, we get delayed but we are the ones who are asked to rush. um

timeliness of uh the execution of the tests and then impact definition. When you do a requirements document, are you highlighting impact with other teams, other services in that document? What do those numbers mean? So what you can do is you can assign a weightage. You come up with a list of metrics and you can assign a weightage of which metric is the most important. So it will

hold the most value when you're coming coming up with one number for quality. So for example for us it was system stability is the highest when did uh the test case design commence because these two things really impact the timing of the delivery and hence the quality of the product. This is a group of product metrics. Similarly you can assign weightage for them. You have your code

health. You can have your defects here. pass rate, um, change failure rate, how many bugs were found by severity. These are the ones which we really hear about a lot and you can do the same for us. Code health which is code coverage is extremely important because this shows that the team has written some tests before delivering code. What do you do with this? So for each

of the metrics, you can then come up with a number from 1 to five. I've done this for how many test cases were automated and you can do it similarly for other metrics. So when you come up with a list of each of the numbers individually then you can say if I've written 0 to 20% test cases I'll give it a scoring of one which is bad.

But if I've written 80 to 100% of the end toend scenarios which were identified as automated tests I'll give it a high number because it means faster retesting and so on. So you can do that for every single item which I spoke about here and here. So for example impact definition with other areas. How can you do a one to five? This is either a yes or

a no. Some of the services was written down in the requirements document. Some of them were not. however you do them and give it a weightage because often we find that we find issues when we are doing a testing during a regression cycle and we discover oh they didn't mention service zed which is also impacted as part of this change and then it takes longer to fix

the bug hence delivery is delayed the product is working fine but the process isn't working so once you have the numbers this is what you can do with them so imagine we have these weightages assigned Can I do this? Yeah. So, you have these weightages assigned to each one of the parameters and then wait and then you can let's assume that these are the scores you get

for each of those metrics and then you can just multiply it and this 3.44 out of five is the quality number that you can give to your stakeholders. This is how the quality is. Is it good? It's mediocre. it's medium as a tester I'm not very happy with it but then what you can do is you can that's the email you're sending out to your stakeholders that

yes we're happy to deliver but this is the number what are the drivers for this number being low in this case we said the code coverage is really low um and the last one the change failure rate for our environment that's really bad these are things we can work on can we address them as part of the next release defects fixed before release again that was really

low uh this is why the number is low. If we can improve and give some action items based on this then we can improve the quality of our delivery and that's how we can try and visualize software quality for our organizations. So this is the slide you saw before. So in conclusion um you want to measure the product and not the teams. Do not put in team

names. Do not put in well project names yes and no. But then really it points at the PM. But I was told recently that it's okay to call out people specifically like the product manager because they didn't align the requirements well and this is why we have problems today. Who are your stakeholders? Who are the people who want to know what the software quality is? How do

you measure it? Is it good, bad, ugly? And if it's good or bad, that's the number you put behind it. And you put action items. And as I'm saying, action items are what our stakeholders really need. Um, I have done the 37page document before and I did not get much buyin from my stakeholders. They asked me to summarize it in one email which I ended up doing

which was great. But this confidence rating thing, I'm actually doing it now. I am getting some responses from the product owners at least because one of the emails I sent out the defect uh fix before release that was one because you went into production with a blocker defect and the PM immediately replied and said it was not a blocker we're going to fix it in phase two

of this project. Uh I said but you should have fixed the severity then we found it you fix it and that's what we're here for to help people understand that we're here to enable use of our products which are of high quality for us and for our stakeholders. Thank you. >> Thank you. Thank you Mariam. Are you going to to answer some questions now or in speaker

corner? You >> I we can do one or two. We have we have >> do we have any questions now? >> There's one. >> Yes, please. >> Please the mic here. >> Uh maybe one comment uh regarding an question. Uh normally if we have a metric we have a threshold and sometimes we go to optimizing for metrics. Is this >> sorry we optimize it? >> We optimize

for metrics not for the goal not for the quality that's an issue and also you concentrate currently mainly on product quality and process quality. What about metric for quality and use? Can we just skip the other and concentrate only into the quality and use in? >> So when you say quality changes meaning just for how well have we understood the product, what is the business flow? >>

I mean just the standard ISO standard. >> It really depends on what your organization needs. Like for me, and this is something I struggle with sometimes because as a tester personally, there are things I will see in the product and I'm not happy with saying yes to shipping it. But if the business is okay with taking that risk, then at the end of the day, you're working

for a business and that's what you're helping. It's the 60,000 tests I was talking about earlier. >> Yeah, I agree. At the end, testing doesn't increase quality. I mean the work of the programmers increase quality but we need to give enough data to the management that's the reality and sometimes business decisions are say strange >> yes this is the world we live in strange businesses thank you

any more questions >> but >> so I'll tell you about why I had the slide mixup >> the slides mix up >> I I don't understand You know the talk I was doing and suddenly I was like this is not the right talk. I need to change laptops. So I was speaking at a conference in oh my god >> my memory is not so bad. >> So

you can meet the next the to to hear next questions in the speaker corner. Is it okay? >> Yes. >> Okay. So you can find Mariam in the speakers corner during the next break. Now please send her off the stage with your biggest applause possible. Mariam