Effort or expression

From Michael Foley’s The Age of Absurdity: Why Modern Life Makes It Hard to be Happy:

Difficulty has become repugnant because it denies entitlement, disenchants potential, limits mobility and flexibility, delays gratification, distracts from distraction and demands responsibility, commitment, attention and thought….

Why submit to mathematical rigour when you can do a degree in Surfing and Beach Management instead?

This is exactly what I was getting at with my cross-country study of math achievement. Countries that have higher self-expressive values tend to have lower mean math achievement. Not only that, but they are less likely to reward students with higher liking-for-math with math achievement, compared to countries with more survivalist values. (Note: my final results were not quite so puny as I had feared. Actually after I refined the model based on some diagnostics they were pretty good but until I solve the psychometric problem of measuring liking-for-math I cannot go further with it. So I suppose that will be what I attack this summer for my research practicum.)

I think this may explain why the curve of math achievement related to GDP flattens at higher levels of income — as countries make the shift from industrial to post-industrial, their values shift from survivalist to self-expressive, and self-expressive values are more likely to encourage a degree in Surfing and Beach Management than in something that requires multivariable calculus and differential equations. (Click for a bigger graph).

As countries industrialize, GDP goes up, and values shift from traditionalist to secular-rational. In this phase, math achievement is encouraged and supported, because an industrially-oriented economy needs, most of all, quantitatively skilled human capital. Mean math achievement will improve and students who like math will put even more effort into it. They will be encouraged by their parents and their peers and by future job opportunities.

But then industrialization provides so much wealth that the country shifts to a post-industrial economy, as has happened in many English-speaking countries. Now cultural values move from survivalist to self-expressive. It’s less important to prepare yourself for a high-paying job than to “do what you love.” The money will follow.

Crudely put, does a culture reward effort or expression? Math success requires effort and it doesn’t really help someone express themselves (just read some of my more statistically-oriented blog posts and you can see that!)

I’m not saying an emphasis on expression is wrong. But it doesn’t contribute to high math achievement. Improving teacher quality isn’t going to change that. Context and culture matters when it comes to academics.

psychometrics, statistics

Attitude towards math vs. confidence, liking, usefulness of math in TIMSS

Kadijevich, D. (2006). Developing trustworthy TIMSS Background Measures: A case study on mathematics attitude. The Teaching of Mathematics IX(2), 41-51.

Abstract. This study, which used a sample of 197,707 students from 46 countries that participated in the TIMSS 2003 project in eight grade, examined whether, for a large number of the TIMSS countries, trustworthy TIMSS measures of several dimensions of mathematics attitude can be developed. By focusing on self-confidence in learning mathematics, usefulness of mathematics, and liking mathematics, it was found that both factor validity and reliability of the measures of these three dimensions derived from the raw data was only attained for the students from the United States. However, when scores concerning the utilized attitudinal statements of all subjects were transformed into Guttman’s image form scores, the factor validity and reliability of the three measures utilizing such transformed data was attained for thirtythree countries (N = 137;346). It was found that for all these thirty-three countries mathematics attitude was mostly saturated by either usefulness of mathematics or self-confidence in learning mathematics. A higher mathematics achievement was found for countries where mathematics attitude was mostly saturated by self-confidence in learning mathematics.

It’s not mentioned in the abstract, but if you combine all the mathematics attitude items into one grand attitude-towards-math scale, you get decent internal reliability (alpha above .70) for almost all countries.

This makes me think maybe I ought to use an overall “attitude towards math” score in the next iteration of my model. Or try that Guttman transformation, which doesn’t make any sense to me, so will need to understand what’s going on with it first. How can it eliminate measurement error?

Also, interesting that attitude towards math is either saturated by self-confidence (something intrinsic) or usefulness (something extrinsic), and the intrinsic one predicts higher math achievement.

psychometrics, statistics

Puny results for my model of math achievement related to liking math

Here’s a description of the data analysis project I’m working on.

I was so excited to push the button on my hierarchical linear analysis, hoping hoping hoping to see a statistically significant effect of the kind I wanted. And I did!

Here’s what I found:

  1. Per capita GDP, secular-rational values, and survivalist values all predict higher math achievement scores.
  2. Self-expressive values (at the opposite pole from survivalist) predict lower “returns” to liking math. That is, countries that are higher on self-expressive values show lower slopes for a proposed linear relationship between liking math and math achievement.

That second part was what I was really interested in. I hypothesized that a culture that valued self-expression would provide a worse context for math achievement, especially at higher levels of liking math. Students with higher liking for math would be relatively more disadvantaged by self-expressive values. In the graph below, you can see how lower values on the SURVSELF dimension (representing lower self-expressive values, higher survivalist) result in higher mean math achievement as well as a higher slope for math achievement related to liking math.

So why am I disappointed? The “effect size” — the practical magnitude of the effect — was small, even though it was statistically significant.The slopes just aren’t that different at different levels of self-expressive values.

One way of measuring effect size in hierarchical linear models is to report “proportion variance explained” or how much of the variation is accounted for when you add in the predictor of interest.

For finding #2 above, the PVE was just 5%. So self expressive values at the country level don’t explain much of the difference of the slopes of math achievement related to liking math. To use the technical term, those are some seriously puny results.

But still, it is a statistically significant effect in my model, and that at least, is a happy thing. I think there is something to what I’m exploring. The models I ran converged quickly, which my prof said is an indication of a highly informative model. GDP didn’t explain all the variance in mean math achievement — the two country-level value dimensions were statistically significant and practically significant also. That was a somewhat surprising result because both value dimensions are related to economic transition. That traditionalist vs. secular-rational values and survivalist vs. self-expressive values explain significant variance over and above GDP is important, and worthy of some more study.


OLS regression, student level: Math achievement on liking math and cultural values

Here’s a description of the project I’m working on.

I duplicated all country-level variables to student level, then ran a student level regression of math on liking for math (PATM), per capita GDP (GDP), and cultural value indexes (Rational and Self Expressive).

Collinearity was not too high — all variance inflation factors were less than 3.

Model Unstandardized Coefficients Standardized Coefficients t Sig.
B Std. Error Beta
(Constant) 382.428 .586 652.822 .000
PATM 4.969 .066 .163 75.521 .000
GDP 4.787 .027 .592 175.670 .000
Rational 23.244 .249 .235 93.291 .000
Self Expressive -33.841 .279 -.361 -121.273 .000
a. Dependent Variable: Math

These are all as I expected.

  • The more a student likes math, the better she does (this is a reciprocal relationship because if you do better at math, you like it better too)
  • Higher GDP is associated with higher math achievement
  • Higher secular rational values are associated with higher math achievement; in other words, traditional values are associated with lower math achievement
  • Higher self expressive values are associated with lower math achievement

The residuals looked fairly normal, though there were more positive ones than negative ones.This could represent an omitted variable (perhaps extrinsic valuing of math? which I could calculate from some other TIMSS questionnaire items) or some nonlinearity in the relationship.

The model has an R square of .321.

Now I’m ready to start running HLM analyses to see if countries with higher rationality and lower self expressiveness offer higher “returns” to liking math.


Data exploration: Cultural values and math achievement

I’ve put together a data set that includes eighth grade math achievement scores from TIMSS 2007, per-capita GDP from the CIA World Fact Book, and country-level cultural values from the Inglehart-Welzel Cultural Map of the World. I also constructed my own Positive Attitude Towards Math (PATM) index, using four self-report items from the TIMSS data set.

Before I model this in HLM, I’m getting to know the data set. I’ve learned from somewhat painful experience that it’s better to know beforehand what’s in the data rather than doing a bunch of regressions and then later having to go back and explore why you’re getting unexpected results.

One issue is potential multicollinearity across my predictors — if two predictors are highly correlated, regression coefficient estimates may not be stable. I expect GDP to be correlated with both the dimensions from the Inglehart-Welzel cultural map, which are these:

  • Traditional vs. Secular-rational – measures the extent to which religion is very important in a society. I’d expect more secular-rational countries to have higher math achievement.
  • Survival vs. Self-expressive – measures the transition from industrial society to post-industrial. As economic survival becomes more assured, people become more interested in expressing themselves. I’d expect this to be negatively correlated with math achievement, because putting effort into math is typically a means of achieving economic success in the job market not expressing your authentic self.

The transition from traditional to secular-rational occurs during industrialization, so of course you’d expect GDP to be positively correlated with the secular-rationality dimension. The transition from survival to self-expressive as well occurs alongside an economic transition.

Here’s a scattermatrix of country level variables (including country mean scores on PATM and math achievement), with lowess fit lines added. While linear fit lines might show me the correlation, I think lowess fits are more informative. Can’t assume every relationship is linear.

So what do I see here?

  • Rational and self expressive values don’t seem to have much relationship. That’s what you would expect, given that they are supposed to represent two distinctly different dimensions of cultural values.
  • GDP is positive related to both rational and self-expressive dimensions. That’s not surprising, given that development of both rational and self expressive values occur alongside economic transitions.
  • PATM is negatively related to GDP, except at the highest levels. That uptick at the end probably reflects the skewed distributions I saw on PATM for some regions with relatively low math scores (e.g., Latin America, the Middle East).
  • There doesn’t seem to be a clear relationship between self-expressive values and PATM, which is not what I would expect. I would think countries that had made the post-industrial shift would show lower mean liking for math.
  • There is a positive relationship between math scores and GDP but it’s not linear. The relationship seems to flatten out at high levels of GDP (which may represent the influence of the transition from survivalist to self-expressive values).
  • Liking for math and math scores are negatively correlated. Why? Countries with generally higher math scores report lower mean levels of liking for math. But if you look within countries, you’ll see that higher liking for math usually means higher math scores. This is exactly why a hierarchical linear model is needed — to be able to model what’s happening within clusters while still taking into account cluster level influences (in this case, cultural values).

As far as potential multicollinearity problems go, it seems I may have problems with having GDP along with the two cultural values indexes in the model.


Measuring intrinsic motivation vs. positive attitude towards math

One comment I got on my research directions presentation was that my measure of intrinsic motivation to study math was not measuring motivation. I called it “liking math” in the chart I put together, but what is it really? It is composed of these four items:

  • I would like to take more math
  • I enjoy learning math
  • Math is boring (reversed)
  • I like math

I should probably call it “positive affect towards math” or better “positive attitudes towards math.” TIMSS calls it the latter (PATM) though they have created a transformed index based on the raw item scores. They report PATM as low, medium, high while I just averaged the scores. Their measure has less information than mine.

You might measure intrinsic motivation with items like this, from the Intrinsic Motivation Inventory:

  • I enjoyed doing this activity very much
  • This activity was fun to do.
  • I thought this was a boring activity.   (R)
  • This activity did not hold my attention at all.            (R)
  • I would describe this activity as very interesting.
  • I thought this activity was quite enjoyable.
  • While I was doing this activity, I was thinking about how much I enjoyed it.

Seems to me that the “positive attitudes towards math” and intrinsic motivation are going to be highly correlated. But maybe you could have positive affect towards math but not much intrinsic motivation? I can’t really see that.

I don’t think this distinction is critical to my project. What I’m investigating is how country-level cultural values influence “returns” to a positive attitude towards math that can be invested in math study as a sort of raw material. I would prefer to look at intrinsic motivation but I’m stuck with what’s in my data set.

Anyway, the issue I’m trying to explore is how different contexts (in this case different cultural values) better support or discourage higher achievement in math for students that have good raw material for learning math.

One potential problem is that positive attitudes towards math are partially caused by prior success in math. So if you’ve been unable to succeed in math in the past, you’re going to have lower PATM. Thus the distributions of PATM across countries could be very different; countries with poor educational systems might have positively skewed (left-leaning) PATM distributions while countries with good ones might have negatively skewed PATM distributions. I’m not sure how this would affect slopes of country-specific math achievement on PATM regressions.