psychometrics, statistics

Puny results for my model of math achievement related to liking math

Here’s a description of the data analysis project I’m working on.

I was so excited to push the button on my hierarchical linear analysis, hoping hoping hoping to see a statistically significant effect of the kind I wanted. And I did!

Here’s what I found:

  1. Per capita GDP, secular-rational values, and survivalist values all predict higher math achievement scores.
  2. Self-expressive values (at the opposite pole from survivalist) predict lower “returns” to liking math. That is, countries that are higher on self-expressive values show lower slopes for a proposed linear relationship between liking math and math achievement.

That second part was what I was really interested in. I hypothesized that a culture that valued self-expression would provide a worse context for math achievement, especially at higher levels of liking math. Students with higher liking for math would be relatively more disadvantaged by self-expressive values. In the graph below, you can see how lower values on the SURVSELF dimension (representing lower self-expressive values, higher survivalist) result in higher mean math achievement as well as a higher slope for math achievement related to liking math.

So why am I disappointed? The “effect size” — the practical magnitude of the effect — was small, even though it was statistically significant.The slopes just aren’t that different at different levels of self-expressive values.

One way of measuring effect size in hierarchical linear models is to report “proportion variance explained” or how much of the variation is accounted for when you add in the predictor of interest.

For finding #2 above, the PVE was just 5%. So self expressive values at the country level don’t explain much of the difference of the slopes of math achievement related to liking math. To use the technical term, those are some seriously puny results.

But still, it is a statistically significant effect in my model, and that at least, is a happy thing. I think there is something to what I’m exploring. The models I ran converged quickly, which my prof said is an indication of a highly informative model. GDP didn’t explain all the variance in mean math achievement — the two country-level value dimensions were statistically significant and practically significant also. That was a somewhat surprising result because both value dimensions are related to economic transition. That traditionalist vs. secular-rational values and survivalist vs. self-expressive values explain significant variance over and above GDP is important, and worthy of some more study.


OLS regression, student level: Math achievement on liking math and cultural values

Here’s a description of the project I’m working on.

I duplicated all country-level variables to student level, then ran a student level regression of math on liking for math (PATM), per capita GDP (GDP), and cultural value indexes (Rational and Self Expressive).

Collinearity was not too high — all variance inflation factors were less than 3.

Model Unstandardized Coefficients Standardized Coefficients t Sig.
B Std. Error Beta
(Constant) 382.428 .586 652.822 .000
PATM 4.969 .066 .163 75.521 .000
GDP 4.787 .027 .592 175.670 .000
Rational 23.244 .249 .235 93.291 .000
Self Expressive -33.841 .279 -.361 -121.273 .000
a. Dependent Variable: Math

These are all as I expected.

  • The more a student likes math, the better she does (this is a reciprocal relationship because if you do better at math, you like it better too)
  • Higher GDP is associated with higher math achievement
  • Higher secular rational values are associated with higher math achievement; in other words, traditional values are associated with lower math achievement
  • Higher self expressive values are associated with lower math achievement

The residuals looked fairly normal, though there were more positive ones than negative ones.This could represent an omitted variable (perhaps extrinsic valuing of math? which I could calculate from some other TIMSS questionnaire items) or some nonlinearity in the relationship.

The model has an R square of .321.

Now I’m ready to start running HLM analyses to see if countries with higher rationality and lower self expressiveness offer higher “returns” to liking math.


Data exploration: Cultural values and math achievement

I’ve put together a data set that includes eighth grade math achievement scores from TIMSS 2007, per-capita GDP from the CIA World Fact Book, and country-level cultural values from the Inglehart-Welzel Cultural Map of the World. I also constructed my own Positive Attitude Towards Math (PATM) index, using four self-report items from the TIMSS data set.

Before I model this in HLM, I’m getting to know the data set. I’ve learned from somewhat painful experience that it’s better to know beforehand what’s in the data rather than doing a bunch of regressions and then later having to go back and explore why you’re getting unexpected results.

One issue is potential multicollinearity across my predictors — if two predictors are highly correlated, regression coefficient estimates may not be stable. I expect GDP to be correlated with both the dimensions from the Inglehart-Welzel cultural map, which are these:

  • Traditional vs. Secular-rational – measures the extent to which religion is very important in a society. I’d expect more secular-rational countries to have higher math achievement.
  • Survival vs. Self-expressive – measures the transition from industrial society to post-industrial. As economic survival becomes more assured, people become more interested in expressing themselves. I’d expect this to be negatively correlated with math achievement, because putting effort into math is typically a means of achieving economic success in the job market not expressing your authentic self.

The transition from traditional to secular-rational occurs during industrialization, so of course you’d expect GDP to be positively correlated with the secular-rationality dimension. The transition from survival to self-expressive as well occurs alongside an economic transition.

Here’s a scattermatrix of country level variables (including country mean scores on PATM and math achievement), with lowess fit lines added. While linear fit lines might show me the correlation, I think lowess fits are more informative. Can’t assume every relationship is linear.

So what do I see here?

  • Rational and self expressive values don’t seem to have much relationship. That’s what you would expect, given that they are supposed to represent two distinctly different dimensions of cultural values.
  • GDP is positive related to both rational and self-expressive dimensions. That’s not surprising, given that development of both rational and self expressive values occur alongside economic transitions.
  • PATM is negatively related to GDP, except at the highest levels. That uptick at the end probably reflects the skewed distributions I saw on PATM for some regions with relatively low math scores (e.g., Latin America, the Middle East).
  • There doesn’t seem to be a clear relationship between self-expressive values and PATM, which is not what I would expect. I would think countries that had made the post-industrial shift would show lower mean liking for math.
  • There is a positive relationship between math scores and GDP but it’s not linear. The relationship seems to flatten out at high levels of GDP (which may represent the influence of the transition from survivalist to self-expressive values).
  • Liking for math and math scores are negatively correlated. Why? Countries with generally higher math scores report lower mean levels of liking for math. But if you look within countries, you’ll see that higher liking for math usually means higher math scores. This is exactly why a hierarchical linear model is needed — to be able to model what’s happening within clusters while still taking into account cluster level influences (in this case, cultural values).

As far as potential multicollinearity problems go, it seems I may have problems with having GDP along with the two cultural values indexes in the model.