The Huffington Post

We had a simple rule of thumb that drove statistics experts crazy, and still does. If the background under the bump, estimated by looking at either side, or calculated from some model, contained N events, then sigma was set equal to the square root of N-1.

Now, here my fifty-year old memory gets hazy, and I have not been able to dig up any documentation. (If anyone has any, I would greatly appreciate getting a copy). As I recall, at first the journals were publishing 3-sigma results. But many were not being independently replicated. So, again according to what I remember, the primary physics journal for rapid publication, Physical Review Letters, asked Art Rosenfeld at Berkeley to come up with a criterion for publication. He used frequentist probability arguments, which advocates of Bayesian statistics despise but have served us particle physicists well over time.

Art counted up all the experiments being done, all the plots being looked at, all the bins on the plots, all the combinations of particles for which invariant masses were being measured, and came up with a rule that has been at least informally in use since: the probability of the bump being a statistical fluctuation must be less than 1 in 10,000. For a normal distribution, only one in 31,574 times will you get an upward statistical fluctuation of 4-sigma or greater. The observed 5-sigma fluctuation for the Higgs, or a larger one, would result only once in 3.5 million trials.

However, this method of analysis is open to question. Several observers have pointed out a flaw, which is known in the literature as “sampling to a foregone conclusion.” That is, the experimenters keep collecting data until the reach the level, in this case 5-sigma, where they then can reject the null hypothesis. The proper method according to the experts is to decide ahead of time what criterion you will use and also how much data you will take before rejecting the null hypothesis. Since that is not generally done, it is technically illegitimate to interpret the result as a probability.