P-values are all about errors. Or, rather, about not making them.

Before we can really understand p-value, we need to talk about alpha, α

About Alpha

When we do research, we want to know if the results are meaningful. So we test our results to see if they are, and we hope we do not make a mistake in deciding our results are meaningful or not. You should know that totally eliminating mistakes is virtually impossible. So we have to recognize that, try as we might, we may make a mistake in our decision. How confident should we be that our decision is correct? 50 – 50? Would making a mistake half of the time be acceptable? Probably not for most people if the result of making a mistake is painful.

For example, most people would make a bet for a dollar on the toss of a coin if they could win, say, $5.00.
Remember how to calculate the expected value (EV) of a proposition? It is the sum of the values of the outcomes. In this case, the EV would be 0.5*-$1 + 0.5* $5 or $2. Not too shabby for a $1 bet. The pain of losing, for most people, would not outweigh the possible return.

But would you bet $10,000 to win $50,000 on those odds? I don’t know about you, but my budget could ill afford to lose $10,000. Before I bet $10k, I would want better odds in my favor. How about getting a 95% chance of winning? The EV in that bet would be $47k.

So it is with research. We want very good odds of not making a mistake. So we decide, most often that we will accept a 5% chance of making a mistake. We could also choose 10%, 6%, 1%, 0.1% or just about any value, though larger than 10% is risky and frowned upon. We call that our significance level and give it the name “alpha.” And, as statisticians are wont to do, we use the Greek letter α.

About Alpha

When we do research, we want to know if the results are meaningful. So we test our results to see if they are, and we hope we do not make a mistake in deciding our results are meaningful or not. You should know that totally eliminating mistakes is virtually impossible. So we have to recognize that, try as we might, we may make a mistake in our decision. How confident should we be that our decision is correct? 50 – 50? Would making a mistake half of the time be acceptable? Probably not for most people if the result of making a mistake is painful.

For example, most people would make a bet for a dollar on the toss of a coin if they could win, say, $5.00.
Remember how to calculate the expected value (EV) of a proposition? It is the sum of the values of the outcomes. In this case, the EV would be 0.5*-$1 + 0.5* $5 or $2. Not too shabby for a $1 bet. The pain of losing, for most people, would not outweigh the possible return.

But would you bet $10,000 to win $50,000 on those odds? I don’t know about you, but my budget could ill afford to lose $10,000. Before I bet $10k, I would want better odds in my favor. How about getting a 95% chance of winning? The EV in that bet would be $47k.

So it is with research. We want very good odds of not making a mistake. So we decide, most often that we will accept a 5% chance of making a mistake.We could also choose 10%, 6%, 1%, 0.1% or just about any value, though larger than 10% is risky and frowned upon. We call that our significance level and give it the name “alpha.” And, as statisticians are wont to do, we use the Greek letter α.