Here’s a description of the data analysis project I’m working on.

I was so excited to push the button on my hierarchical linear analysis, hoping hoping hoping to see a statistically significant effect of the kind I wanted. And I did!

Here’s what I found:

- Per capita GDP, secular-rational values, and survivalist values all predict higher math achievement scores.
- Self-expressive values (at the opposite pole from survivalist) predict lower “returns” to liking math. That is, countries that are higher on self-expressive values show lower slopes for a proposed linear relationship between liking math and math achievement.

That second part was what I was really interested in. I hypothesized that a culture that valued self-expression would provide a worse context for math achievement, especially at higher levels of liking math. Students with higher liking for math would be relatively more disadvantaged by self-expressive values. In the graph below, you can see how lower values on the SURVSELF dimension (representing lower self-expressive values, higher survivalist) result in higher mean math achievement as well as a higher slope for math achievement related to liking math.

So why am I disappointed? The “effect size” — the practical magnitude of the effect — was small, even though it was statistically significant.The slopes just aren’t that different at different levels of self-expressive values.

One way of measuring effect size in hierarchical linear models is to report “proportion variance explained” or how much of the variation is accounted for when you add in the predictor of interest.

For finding #2 above, the PVE was just 5%. So self expressive values at the country level don’t explain much of the difference of the slopes of math achievement related to liking math. To use the technical term, those are some seriously puny results.

But still, it is a statistically significant effect in my model, and that at least, is a happy thing. I think there is something to what I’m exploring. The models I ran converged quickly, which my prof said is an indication of a highly informative model. GDP didn’t explain all the variance in mean math achievement — the two country-level value dimensions were statistically significant and practically significant also. That was a somewhat surprising result because both value dimensions are related to economic transition. That traditionalist vs. secular-rational values and survivalist vs. self-expressive values explain significant variance over and above GDP is important, and worthy of some more study.

Pingback: Adjusting for response styles in cross-cultural research « Anne Z.