The Disease Management Care Blog Explains What p < .05 Means, or Everthing You Want to Know about Statistical Significance But Were Afraid to Ask

The prior post on Obesity and Fitness made reference to ‘statistical significance.’

What? Just what does this ‘p < .05’ mean? More importantly, can it be explained without resorting to opaque concepts that are outside the reach of the numerically challenged? Flushed with prior success over intrepidly tackling QALYs, the ever-confident Disease Management Care Blog volunteers to give the nettlesome 'p value' a try. It figures that some DMCB readers, from time to time, may need to grasp this notion when struggling with the meaning of some result being reported in the scientific literature.

It wishes it could take credit for the explanation that follows below. Rather, it witnessed a rather famous statistician use a simple yet instructive parable at a conference long long ago.

Imagine you are in a shady casino in the Wild West that fortunately lacks a check-your-gun-at-the-door policy. One of the dealers bets you that he can flip a coin and accurately call heads or tails. With each bet, the jackpot doubles. If he calls it right, he wins. If he calls it wrong, you win. Double or nothing.

On the first flip, he calls ‘heads!’ Sure enough, that’s what happens. He collects the money and you lose.

On the second flip, ‘heads’ is called again. He’s right and again you lose.

Third flip. Same thing happens. He calls heads and again you lose.

Fourth flip. Uh oh.

This keeps going until you conclude that you are being bamboozled. The question is when do you take out ‘Ol Bessie’ and plug the laying polecat cheater?

The statistician asked for a show of hands for voting yes, shoot the guy on the 1^st flip of the coin. A few hands went up, enabling the rest of the audience to spot the NRA members who were against gun control.

2^nd flip? Few hands went up. 3^rd flip? Few hands went up. The vote for the 4^th flip prompted a lot more hands. Most voted to serve up some Texas-style street justice when the huckster got it right 5 times in a row. No numbers, just intuition.

Those voting to give the dealer an acute case of lead poisoning on the 1^st flip had little reason to do so, because the chance of correctly calling heads vs. tails is 50-50. Calling it right twice in a row is also intuitively possible, though less likely. By the way, the chance of doing that is 1 in 4, or 25%. We’ve all seen some lucky people call it right after three flips. The chance of doing that is 1 in 8 or about 12.5%. Four times in a row and we begin to think something’s wrong. The chance of doing that is 1 in 16 or about 6%. Five times in a row is very unlikely and that’s when most people stop playing. That’s 1 in 32 or 3%. If it’s that unlikely, we naturally conclude something is wrong. The line has been crossed and it’s time for Mr. Shady Dealer to be introduced to the business end of Mr. Smith and his buddy, Mr. Wesson.

P < .05 is that line on the casino floor. It summarizes and standardizes that intuition using statistical processes. 0.05 or 5% is between 6% and 3%, that watershed region of probabilities when we naturally begin to discount randomness and start smelling a rat. That’s when we raise our hands at the conference and vote yes on the motion to provide some through and through ventilation.

The .05 means that there is only a 5% chance that the dealer, or in the case of the medical literature, the study authors, got the results they got because they happened to be lucky and randomly have the coin flips go their way. That low ‘p’ or ‘probability’ (less than 5%) tells us that there was something else that 'un' randomly caused the data to fall the way it did. The numbers were ‘pushed’ by something. Shady Dealer had a loaded coin, while scientists have the intervention such as a drug, some sort of surgery or a disease management intervention.