Lesson 2: Empirical Probabilities and Probability Trees

Theory

At the end of the last lesson, we learned how to calculate the probability of some event in a sample space by dividing the number of outcomes in the event by the number of outcomes in the sample space. What we calculated is technically called the theoretical probability, because we based it on nothing but mathematical reasoning. Assuming that each outcome had the same individual probability, all we needed to know was how many outcomes there were in total. Unfortunately, not every chance experiment in the real world will be this easy. Imagine a simple carnival game where you throw baseballs at a target. How can you possibly count all of the possible outcomes? There are an infinite number of points where the ball can hit, an infinite number of ways it can fly through the air... We need another way to calculate probability numbers for situations like this, if we're going to use mathematics to think about them.

Practice

Empirical (or experimental) probabilities are our answer. Instead of starting from nothing, and trying to predict what will happen, we can simply perform the chance experiment over and over and observe what actually does happen. Assuming nothing changes, we can still use the results of our experiments to calculate a probability number that we can use just like a theoretical one.

As an example of a simple experiment, let's consider coin flips again. While writing this lesson, Peter took out a quarter and flipped it fifteen times, and got these results:

Calculating empirical probabilities from these results works a lot like calculating theoretical probabilities, except instead of looking at the number of outcomes in the event and the number in the sample space, we look at the number of times we ran the experiment, and the number of times the event occured:

So in this experiment, the probabilities are:

Sample sizes, error, and statistical power

You might notice something funny about these probabilities, though: they're wrong! Unless there's something wrong with the coin, shouldn't each side have a probability of exactly ½? This is the most important problem with empirical probabilities: they won't give you the exact, correct answer. In fact, with the experiment Peter did, it's absolutely impossible to get exactly 0.5, because 15 is an odd number: no matter what, one of the two outcomes will occur at least one more time than the other one. This doesn't mean empirical probabilities aren't useful — in fact they're often the only way we can calculate probability numbers for a chance experiment — but it does mean that we have to be careful to remember that the number we're working with has some error built in. Error, in statistics, is simply the amount by which a measurement, such as an empirical probability, differs from the real, exact value. ^[1] In this case, because there are only two outcomes, each one has the same amount of error, but in opposite directions:

Statistical power is a very deep mathematical concept on its own, but all you need to know for now is that it's the likelihood (the probability, in fact!) that some prediction or conclusion we make with statistics (usually that we have estimated some number with no more that a certain amount of error) is correct. ^[2] The exact numbers aren't important for now — its enough to realize that more statistical power is good, and error is bad. We want our numbers to be as close to right as possible, and we want to be as sure of that as we can possibly be. Fortunately, there's only one thing we need to do to get more statistical power, and less error: more tests!

Sample size (not to be confused with the sample space!) is the number of tests we've run, or samples we've taken, before calculating our empirical probability. For the very simple kinds of statistics you'll be using to test your carnival game, all you really need to know is that the larger your sample size is, the closer your empirical probability calculations will be to the theoretical probability. More tests means we will have a smaller error, and we will have more statistical power to back up the claim that our empirical probability is good enough to use.

Review

• It's not always practical, or even possible, to do probability calculations like we did in Lesson 1, because we don't always know exactly how many outcomes are in the sample space, or in the event we're interested in.
• Empirical probabilites are calculated from the results of performing a chance experiment over and over again, so we can use probabilities even when we don't know the size of the sample space or the event. We still use division to calculate them, but instead of dividing the number of outcomes in the event by the number of outcomes in the sample space, we divide the number of times the event actually occured by the number of times we actually performed the experiment.
• Empirical probabilities are subject to error: they're almost always going to be at least a little bit off. Error is the difference between the empirical probability and the exact, theoretical probability, even if we can't even figure out what the exact answer is.
• Statistical power is the probability that some prediction we make with statistics is correct. For our purpose, we are informally predicting that our empirical probabilities will have errors small enough that we can use them to plan our carnival games.
• Error and statistical power are really complicated subjects, but what we need to know right now is that we can make the error go down, and the statistical power go up, by making the sample size bigger. The sample size (which is not related to the sample space) is the number of times we performed the experiment before calculating our empirical probability.