Some Silly Thoughts on the Bonferroni Correction and Chance
Man, I hate a Type 1 error. Don’t we all? You can read a sensible account of the Bonferroni correction on Wikipedia: http://en.wikipedia.org/wiki/Bonferroni_correction. This is not a sensible account.
The exotic name for such an elegant and simple idea comes from Carlo Emilio Bonferroni, an Italian mathmetician. He did not devise the correction, apparently, though he did work on probability theory. The implication of the Bonferroni correction is that since chance permeates everything, if you use a 5% cut-off, 1 in 20 of your findings could in fact occur by chance. This is an idea which is devastating for an undergraduate, especially if the first time they encounter it is in interpreting their own data, which thanks to Prof. Bonferroni (actually Olive Jean Dunn*), is not that exciting at all.
Anyway, one thing that has always amused and troubled me is the idea of the ‘family’ of hypotheses. You should only need to correct for related analyses – typically, a big matrix of 12 correlations on variables from the same group of participants in the same testing session, for instance. But what are related analyses? My PhD supervisor and I used to joke about this idea. At what level should we correct for multiple comparisons? Since we all slave over work (pretty much) to achieve p<.05, we are undoubtedly acting as our own quality control – this is the old idea that null effects are harder to publish/do not get us so excited. But perhaps instead of using 1 in 20, I should look at the number of related experiments I had to do to get this effect?** But if this one-in-twenty idea permeates everything, perhaps we should correct for multiple corrections in science at the article level – 1 in 20 articles published has chance findings. (This is a silly idea.) Thus, the significance level cut-off of any one article in the edition should be divided by the number of articles in that edition.
Perhaps too, we should correct at the career level. 1 in 20 of our colleagues has published great work and got promoted merely through chance. (This is a very reassuring thought.) Or, perhaps, 1 in 20 of their findings is by chance alone: “Thanks for your application for this post but I am afraid your work has been randomly deleted from the scientific record at the 5% level.” Even better: “Your progression from PhD student to professor has been excellent. To make sure this is a genuine achievement, we would like you to go back and start your career again and repeat this success.'”**
These silly recurring thoughts have resurfaced thanks to the rejections that I receive intermittently from good journals who criticize my work for not being surprising or original enough. To wit, I am now suggesting, in my quest for publication glory, that I will turn my back on the slow accumulation of knowledge which is encumbered by my own intellect and ethical and practical considerations and instead I will use my Random Hypothesis Generator. We know that, as long as I design things probably, I should only need 20 experiments to turn up Psych Science gold. I can just randomly mix unexpected topics and dependent variables (carefully electing from a pool of sexy topics such as False Memory, Obesity, Religion, Stereotype activation, the Default Network and so on). Single experiment, unexpected finding, p<.05. Brilliant. No one will know about the 19 that didn’t work. Have that, editor! I bet you weren’t expecting that!
<sigh> Time to look more and more closely at Bayesian approaches.
*This is all according to Wikipedia, I’m afraid, but Olive Jean Dunn seems to be the unsung hero of Bonferroni corrections. I could find a lot of her books on Amazon: http://www.amazon.ca/Books/s?ie=UTF8&field-author=Olive%20Jean%20Dunn&page=1&rd=1&rh=n%3A916520%2Cp_27%3AOlive%20Jean%20Dunn Perhaps we should start referring to the Dunn correction now? ‘Have you Dunned it?’ is easier than the ‘Have you Bonferronied them?’ that is in common parlance.
**I’m too lazy do this, but I know it goes on. You can earnestly run and re-run the basic idea with lots of tweaks (or possibly no tweaks at all) until it actually works. When it finally turns up P<.05, which it can do by chance, bingo! Me, I fall in love with my ideas, but I am not loyal, in research, I am a serial monogamist. If my current love does not give me instant gratification, I move on, upset that it didn’t work. Unless, I can really pinpoint a critical design issue to change, I tend to move on. (A draft I am currently working on showed no effects as within subject design but it did as a between subjects design – I want to publish both. I imagine in the name of page lengths, the non-significant one will get erased from the record.)
*** This is my experience of moving to France in a nutshell.
concog.org 