Over winter break I worked on a small data analysis contract involving elementary school test scores and some numbers about an instructional coaching program that took place at the schools. I was following someone else’s analysis plan — a similar analysis had been run for the prior year’s data, and I had agreed to replicate the analyses.
One aspect of the analysis that bothered me was the correlations. At one point in the analysis, I was to run a matrix of almost 100 correlations, then highlight the significant ones (meaning the ones that are statistically significant from zero), identified as the ones with p value less than .05 (our alpha level). Well, you statisticians out there know that the problem of looking at many many correlations is that some of them are going to look significantly different from zero just due to chance.
With an alpha level of .05 and 100 correlations, you’d expect roughly 5 of the correlations to have a p-value less than .05, not because the population correlation was significantly different from zero but merely by chance.
This is known in statistics as the multiple comparisons problem, and one way to ameliorate it when examining a bunch of correlations is to use a Bonferroni correction. That’s what I advised my client to do, even though it hadn’t been done in the first analysis.
It turns out there are more ways to go wrong with correlation. The Neurocritic summarizes a paper about how researchers have been reporting suspiciously (in fact, mistakenly) high correlations between brain activity and behavioral measures.
All the deets in this article by Edward Vul, Christine Harris, Piotr Winkielman, and Harold Pashle. I’m hoping that soon I’ll get the time to read and summarize that article so I can ensure I don’t make the same mistakes.