About this talk
In this talk, Dave Mallister discusses the importance of understanding statistics in monitoring systems. He explains the differences between mean, median, and mode, emphasizing that knowing how to interpret these statistics can significantly impact system reliability and alerting accuracy. Mallister critiques the common practice of blindly trusting metrics, urging attendees to analyze data appropriately to avoid misleading interpretations, especially regarding outliers and sudden changes. He highlights the strengths and weaknesses of different statistical approaches, such as harmonic and geometric means, and advocates for using median values to set alerts effectively. The speaker also delves into the significance of sampling in open telemetry and other monitoring contexts, pointing out how biases can distort the analysis. Ultimately, he stresses that careful thought and clear understanding of statistics are vital tools in effective system monitoring and troubleshooting.
Full transcript
How's that? Oh my gosh, I can hear myself. Oh wow. So, hi, I'm Dave Mallister. I work for EngineX. Engineex is owned by F5, but I don't talk about them very much. I work for EngineX. I'm an open source geek. Um I started with open source what is now called open source in 1992 got involved in Linux in 94 and have been open source since then. However
today I will be mentioning engineext a couple of times but this is really a kickback and have something very different and this is about the statistics behind the alerting system that we all depend on when we run our systems. So, let's start off with the fairly simple. What's the difference between a mean, a median, and a mode? How many people here think they know what a mean
is? Okay, more than more than 20%. Sorry, most of y'all probably have some idea. How about a median? Okay, how about a mode? Okay. Okay. For everybody who raised their hands three times, the extra credit question is, what's the ninth data kind number? Um, and I'll give you the answer at the end of the the talk. So, statistics and monitoring is a numbers game. Metrics are the
numbers that represent our best view of what's going on inside of our systems and inside of our applications. Generally, they're timestamped. Generally, they're key valued pairs. However, for data to be useful, it needs to be aggregated, analyzed, and usually visualized so we can see what's going on. sitting there with a tail minus F and looking at a stream of numbers coming from your system is probably meaningless.
But looking at this data as it as it goes forward, the picture over here is sped up, but this is a red monitoring system rate error duration based on open um telemetry tracing um showing the system and you can see over a 10-second period what's happening on a second by- second basis and you can see how error rates are going up and down and this gives us
an idea of what we're looking at. numbers lie. So keep that in mind. My other talk is about how to lie with statistics, but numbers lie. And so EngineX of course has monitoring. We have lots of different things in here. We're reverse proxy. We are web server. We're a mail server. We kind of can do any of those different pieces here. But again, monitoring is essential. And
the thing is that monitoring is actually really essential for you when you know what you want to monitor. your specific goals matter. The problem is is that we blindly trust computers. We blindly trust that what we're seeing on the screen is meaningful. So I started off by by talking about those other pieces. And what we are really concerned about is not usually the general behavior but the
outlier behavior. So how do you deal with outliers and and the spike activities and spikes can go down and spikes can go up here. How do you get a representative value when things build on top of each other? And then how do you for instance get a value that represents change over time? These are all different approaches to how we deal with our statistical lives. And honestly,
do you actually know what it's telling you? I have to tell a quick side story here. I got asked to do this for a large conference once, but they said you had to include AI. And so I said fine and I went up and I asked one of the AI systems to solve a very standard problem, exponential distributions. We'll cover those. It came back and did my
answers didn't agree with it. And of course, I blindly accepted that the computer was right and I was wrong. Turned out that when I did it again, my answer still matched. And so then I went and I asked a different AI system who came out with a third answer, a fourth answer, a fifth answer and so forth. So if you don't know what is actually being generated,
how do you know whether the answer is right or not? So anyway, mean, median, and mode, by the way, it's a trick question. So keep in mind when I get to that, but mean, median, and mode is a trick question for that. So here's a series of data. A mean is a measure of central tendency and in fact all of these items are central tendencies where something
is happening in the most common element here. So the mean is the central tendency of those different pieces. The median is the middle item of a sorted list and the mode is the most common item in that list. So it's very simple very straightforward. So given that data set the mean is 5.444 444 median six mode is one or is it the mean is 4.130 2.791 and
all three of those answers are correct. The reason behind that is that there's more than one kind of mean. There's arithmetic mean which is probably what most of y'all thought of when we were when I asked that question. Add the numbers together, divide them by the number of items gives you a central tendency number. So that arithmetic mean is very straightforward. Harmonic mean another one of the
classic means is multiply the numbers together and take the nth root. So if you multiply 100 numbers together, you take the hundth root. That's the geometric. And the harmon sorry that's geometric. harmonic is take the reciprocal of all the numbers add them together and then divide it into the number of items. Now they each have values and they each are used in different cases and so start
with arithmetic the most common this is the one everybody does it's the central point in that wonderful bell curve that normal distribution that we use Gaussian curve bell curve for this it is not usually the 50% point this is something everybody always ignores immediately unless your mean point is zero and your standard distribution is one. It's not the central points. So if you think of the number
coming out from mean being the middle. Nope, not going to happen for that. The time series we almost constantly are tracking new data. And so we now have time series data and we actually only look at part of that. And so our structure moves with us as we're moving across. So load balancing model, add the three together, divide by by that, we end up with 200 requests
per second. And this works really well in a single load balancer model with three systems who have pretty much the basically the same power and capabilities. Geometric allows me to compare different things together. So now I can look at request and request time and get a central tendency of those two things. This is where I multiply things together and where it becomes a very straightforward point in
DevOps number of deploys per unit of of time. How often I deploy things whether it's an MTR method throughput how fast things are going through the system becomes really point. This also allows me to build. So if I want to do an improvement on my sprints and I see a reduction in my sprint one, two, three, and four as the list shows you, then all of a
sudden I'm back to the point where it says they've got to build on each other. So what is my central tendency point for that build? You can't just add them together and divide it. The number is not correct. And so you do this from a geometric viewpoint and you will get that your your uh geometric mean of 0.937 or about a 6.3% incre increased improvement of time.
And then harmonic this is the divide by the reciprocals which divides into that number and then gives us a result. Those three numbers I showed you that the of the very first chart are the arithmetic, the geometric and the harmonic. Arithmetic will always give you the highest result always. And harmonic will always give you the lowest result. And so you can actually play games with your means
because very honestly nobody gets this by giving them a mean that matches what your needs are. And so in DevOps we look at this a lot from a range of performance are things staying within a range of behavior centralized tendency. Again it's incredibly useful. Harmonic is incredibly useful for looking at outliers because once again it's the smallest set of differences and therefore if it's an outlier space
and you can compare the two you get an outlier viewpoint. So harmonic is pretty So I always like this one. This is how you can lie with means and sorry my data is US for this. Average income is about 59 384. Average household is 2.63 people. Therefore the average household income is $156,179. No, it's not. It actually is um roughly $105,000. And this is how means can
be used to mislead you. If you are using a mean and then multiplying it back out, you've changed the rules of the game. And so keep that in mind. And by the way, given those data, if you want to earn a million dollars, you just need 14.8 kids. during one of these talks, somebody came up and says, "Well, what I do is I take latency and throughput
and multiply them together and then I compare that data." And this is the the data from one of their charts for this. And the suggested metric will show you that if you look up here, my suggested metric comes out to 10,000 in each of these cases. But my mean is drastically different in all three categories. in the GM category, the G geometric, because that's what you're actually
building. He's building a geometric mean out of this. And it might be useful, but it doesn't weight the right things. It's not actually telling me anything really useful. The three charts down the bottom again are the the metric categories, and you can see they follow the same pathway. That's to be expected. They actually centralize at one point or two points inside of this, but it's something that
not makes doesn't necessarily make sense. Okay, really cool. I just introduced you to geometric and harmonic. Why don't we use them? Well, there's this little thing called this nasty thing called a zero. And zero can't be used in either geometric or harmonic. Remember harmonic is dividing taking the reciprocal. One divided by 0 is meaningless. Geometric multiplies the numbers. However many numbers you multiply together times 0 is
zero. The square root of square root of zero is undefined once again. And so you cannot use them if you have a zero. So that's why we don't use them. But in certain cases they're still very very powerful. Um request timing is incredibly useful in these. Okay, median. This is actually the chart that started the entire talk that got put together for this. The median turns out
to be amazingly underutilized. Um, and it has some really interesting points. First of all, the median is always the 50% point. Always. By definition, it's the middle point of your sorted list. It has to be the middle point here. And that gives us some values. If we know the median, we actually always know exactly where center is. what happens though if you are using a historical out
um anomaly or a sudden change model inside of this and you're using the mean the mean can change drastically. This case it's it's I think 30 days worth of data something like that. I can't read my own set a little bit more than that for this. And we had a spike occur. And the spike is is substantially larger than what our mean was. And because of that,
our mean, our arithmetic mean jumped up. It does not return immediately to the next point. It slopes down. As the numbers return to normal, it slopes down. So if you're looking at sudden change, you're now looking at sudden change against something that's no longer valid. Hopefully, you fixed the problem that caused the spike. Whereas the median changed by one number which is already in the central tendency
line. So it's close to the middle point as it is. And so again using mean is useful for the central tendency but using median is best for using for alerts. This is actually really something that bugs the heck out of me because I when I go around and people talk to me about how they're using means not the best approach. And by the way, this is what
it looks like with a harmonic uh distribution. It follows pretty much the same trail. Harmonics do not respond to single number changes. U but again, you can't use harmonics with zeros. And so it has to be careful. So if you're using this model, this median model, you're actually using a 50% point. You're staying at the 50% point. If you're using P95 percentage 95, is it out of
range? 995. You're already using a percentage value. Congrats. I won't I won't yell at you for not using the median. But keep in mind that you're using a structure that does not react to sudden changes while at the same point giving you the ability to to control where your data lives. So slight try sidetrack here. This is called the measure of variability. If you don't know, if
I look at you and say, uh, you know, 63, the mean is 63, it doesn't tell you a lot. Am I telling you this with two numbers, 64 and 62? Am I telling you this with 10,000 numbers? Are my numbers widespread or they not? So, knowing how they vary becomes really important. And there's this thing called an interquartal range which basically takes that ordered list and says
here's the first quarter, here's the second quarter, here's the third quarter. And the reason that we are most interested in interquartile standard deviation is what everybody lives with. We'll talk a little I think about that. But inter quartile has a couple of advantages. So let's see where am I? That's okay. So this is a example Death Valley, one of the hottest spots on Earth. The mean temperature
is 61 degrees or 16.1 Celsius. Death Valley has a range from 40 Celsius to 15 Celsius. Either of those two ends, you don't want to live in there. Well, 15's not bad, but at 40, you don't probably want to do that. But in the middle 16 sounds pretty comfortable. So you have to know what your range is to understand what your behavior is. And again, two identical
data sets more or less, sample A, sample B, their means are exactly the same. Their deviations are drastic. And so how they change is really important for how your data is spread. And the data spread becomes really important because the more data you have the narrower your spread your deviation spread should be. So if you have one person you can have a lot of range. If you
have 50 people you have a different range. And so knowing again your data set makes it really important to to drive that. Talked about the anti-quortile. This is an outlier formula. It's visualized here. the outside edges are good candidates for outlier behavior. And in fact, there is this rule of thumb. There's no math theory behind this. It's just something that happens to work that says if you
take those two pieces from the U1 Q3, take the the length of that range, multiply it times one and a half, and add it on subtract it from each of those ends. If it's outside those two points, those red points on this line, it's an outlier. And it's almost perfect a point of saying it's an outlier. It may not be every outlier because you could have outliers
that are inside that range. If it's outside that point, it's probably an outlier. And so you need to consider what an outlier means to your your your structure. so mode mode is one of those underutilized things. Um the most common recurring set usually we see it as histograms for this and we don't use it a lot or at least we don't use it a lot in our
DevOps work here but we do look at it in log analysis. We look at it user behavior. What are people commonly doing with the system? What is their common pathway through the system? So user interfaces and user behavior become a really important part of a mode environment here. It can also be used for security monitoring. Are you seeing common behaviors and then seeing outliers to common behaviors?
It's not something that's easy to track from a pure metrics basis, but it's easy to track by accounting basis, which is what this what mode really is all about. we're going to get into the next next side track here. There's this thing called descriptive. And there's this thing called inferential. Descriptive statistics are using every piece of data. Inferential statistics are using a subset of the data to
represent the data as a whole. I can't tell you what every person in the world thinks, but I can grab a subset of people and infer from that what their behavior is. So both of them get used to draw conclusions. However, a um sorry, descriptive the conclusion is pretty precise. You don't have any of the data that can change around that. The other side can be really
interesting. You can get biases creeping into it drastically for this. What this does in statistics is leads us to sampling. Um I've worked in open telemetry for a while. between logs, metrics, and tracing. I have so much data that I can't handle it all. I can't visualize. I have one cl customer who does 43 terabytes of tracing data an hour. That's a lot of data. You cannot
do this. So, you have to look at sampling. But sampling has its own set of risk for here. Um, what happens when we do this is that we sample things and we can sample them in different ways. And the ways in which we sample affect how we get results. In open telemetry, we hear head-based and tailbased. Even though tailbased isn't really sampling, nonetheless, one says random pick.
One says pick it on a set of criteria that I want to to establish. I saw an error. I saw something that was an outlier space. And that makes us a set of data that is not necessarily purely representative. Here's an example. The first set up here is a sampled model. I forgot what my I forgotten literally forgotten what my sampling model looks like, but as you
can see, it doesn't look too bad. It's got one two problems that are showing up in here. Likewise, the latency distribution for that is working out that I've got a couple things at one to two seconds. That's my 95th percentile. When I look at the entire set of data, I have problems appearing all over the place. I have lots of things that are appearing. This is not
a random model and I have traces going out to 29 to 40 seconds. And so depending on how I sample changes the way my data is presented to me, changes the means, the outlier structures, all those different pieces and can make it really hard to find to do the forensic exercise to figure out what's causing that 29 to 40 seconds. Um, according to Google's surveys every year,
3.7 seconds is known as um walking away from a shopping cart online. 3.7 seconds. So if I've got something running 40 seconds and I'm expecting something under one, I have a problem I need to look at. Now the good news is metrics are seldom sampled. Um I have seen it but not often for that traces get sampled a lot. Um and traces can lead to metrics. So
hopefully you're looking at all the metrics we're tracing, not just some of them here. makes forensics. It can be a lack of direct correlation inside of here. It's a necessary evil. No one wants to store 43 terabytes of data per hour for any length of time. It's bloody expensive. And so we have to deal with this. But keep in mind that tracing changes the nature of the
statistics you're looking at on the screen. The alerts that you're looking at can also exhibit bias. And bias can manage a lot of ways. Cognitive bias, but the one that's actually most used found most often when we're looking at these is what's called recency bias. There has not been a problem recently, therefore there will not be a problem. That is a falsehood. The fact that there is
not been a problem recently doesn't change the impact that there could be a problem. It's like flipping a coin. You can get a heads or tails. You can flip three coins. What are the odds of getting three heads versus each coin being a head is very different. So keep in mind that looking at your data can have a recency bias and keep in mind historically that things
may be different. You know, you may not have had a um deployment in three months and then you deploy three things in three weeks. probably you're going to have a problem and it's probably related to that but that's secondary. There are lots of different distributions that can help inside of this. We talked about the normal one. That's a wonderful bell-shaped curve for this voicon which is lets
me look at the occurrence of events. And so that becomes a request timing model that comes in play for that u beta which is red um AB testing test two things one against the other for that exponential which is the time between asynchronous events. And we'll talk about that one because that one's fun. Wayable what's the odds of failure? Literally you can actually figure out what the
odds of failure are in a particular category and then log normal which is interestingly enough when your data you take the log of it it may become a normal curve. This data curve that you're seeing here which starts high at the right and moves down if you take the log of it is actually becomes a normal bell curve. And so that's what those two things are. Slight
sidetrack. Standard deviation because standard deviation plays a role in every one of these calculations. Standard deviation shows me my range. But at the same point in time in a standard deviation in that normal curve, which is the only place standard deviations really make sense, it shows me what the ranges are. All my values in the the first standard deviation one to one should be about 68% all
the way out to 99.7 when I go far enough out at three distributions. And so standard deviation can tell me what my ranges are with some chances of also looking at the outlier behavior. Those two pieces together again give me the ability to say there's a problem going on and I need to take advantage of it. Okay, exponential. By the way, this is the one that I
challenged all of the various AI tools with and they all got it wrong. Um, and I've checked my math more times than I care to think about. Okay, there is literally once you've taken your data, there is one variable. This is like one of the easiest statistical equations you will ever see in your life. Everything else is a fixed value for that. So E one over the
mean and then the equation is is one over mean time E times one over the mean times the X which is the single variable that you care about and so gives me a latency or throughput asynchronous events. This one I love dearly for that for that. And so you can see very quickly that I can come down with the exponential viewpoints and come out with a what
are the odds of a particular event occurring at a particular point in time. So my throughput probability iny at 58 just 58 is 62. However the cumulative something happening within that time frame is 58. And remember my discussion about median. It gives you a median. And so median can be used here to say, hey, something should have happened by now. I should have seen a request coming
in. Why is my system not talking to the outside world or why is the outside world not talking to me? And that becomes a really important aspect. You may stumble upon this on scale statistics are not your friend. Wrong. On scale probability is not your friend. Statistics are history. Probability is trying to predict the future. So if the odds come out and say one out of 10,000,
that's a probability structure. And by the way, probability can also be wrong. Coin flips are not 50/50. They're actually 5149. It has to do with procession. People can flip a coin. Generally, the coin will make a specific number of rotations. It doesn't matter what coin you're using. Doesn't matter who's doing this. A blind flip comes out at 5149. kind of interesting structure and the best way to
do this is not show the person whether what's up on this the face of the coin. And by the way, with practice, people can flip a coin and land on the same side every single time. So coin flips are an example of probability uh that we talk about all the time. finishing up uh as much as I can. This is known as the money hall problem. And
this gets us into this concept called basian, which every DevOps guy probably uses and doesn't know about it. You got three doors. Money problem says behind two of these doors are goats. Behind one of these doors is the new car. Pick the new car and you win the car. And you pick door number one. Monty would look at you and go, "Let me open door number two
for you." And behind door number two is a goat. Now Monteu knows where everything is. Should you switch doors or stay with the same door? How many people would switch doors? How many people stay with same door? Okay, everybody who raised their hands were switching. Well done. Here's why. Three doors, one-third probability on each door. One door is taken away. One door you pick, two doors left.
One of those doors goes away. the probability doesn't change. The probability between those two doors changes. And so now your original door is one-third. Your other two doors is a two-third probability. So one of those doors now has a two-3 This got argued about for years, by the way, and it's massively still fought about today. This is called basian basian theory. And basically what it says is
if you get new data, you are allowed to change your minds. Keep that in mind. In in my space, I constantly get new data. I constantly change what the underlying factor cause could be. Lots of pitfalls inside of this, but correlation is a big one. And this is my favorite sneaky correlation. This is correlation of marriage to the sale of beer. And they are almost a perfect
match. So, does drinking cause marriage or does marriage cause drinking? No, neither one. confounding variable. Something else is causing these. In this particular case, it's population growth and aging. As people, we got more population, we got more people growing up, more people got married, more people drank beer. And that's a confounding variable. Correlations are sneaky. And by the way, correlations can go both ways. So you can
have reverse correlations for this. And oh, by the way, I don't have the slide in here, but um if you go look at it, you'll find that Internet Explorer caused murders because as Internet Explorer use went down, the murder rate went down as well. So, statistics are how we analyze our metrics. They are great aggregations. They reveal central tendencies for this. We usually use only a few
things, but there's a wide world of other choices that we can make in here. And finally, the most effective debugging tool is still careful thought. Brian Kernigan said this in 1979. I think it's one of the best quotes I've ever seen. Coupled by judiciously placed print statements, metrics and the aggregations are giving you that information. And with that, that's the answer to what the ninth edi number
is. Um, so if anybody had it right, let me know afterwards. Um, because I certainly didn't. And statistics are history. History is written by the winners. So when somebody shows you statistics, be sure that you understand what you're reading. And with that, thank you very much. And I'll get off stage and let our next speaker get on board. Thanks.
