Tag Archives: timss

Effort or expression

From Michael Foley’s The Age of Absurdity: Why Modern Life Makes It Hard to be Happy:

Difficulty has become repugnant because it denies entitlement, disenchants potential, limits mobility and flexibility, delays gratification, distracts from distraction and demands responsibility, commitment, attention and thought….

Why submit to mathematical rigour when you can do a degree in Surfing and Beach Management instead?

This is exactly what I was getting at with my cross-country study of math achievement. Countries that have higher self-expressive values tend to have lower mean math achievement. Not only that, but they are less likely to reward students with higher liking-for-math with math achievement, compared to countries with more survivalist values. (Note: my final results were not quite so puny as I had feared. Actually after I refined the model based on some diagnostics they were pretty good but until I solve the psychometric problem of measuring liking-for-math I cannot go further with it. So I suppose that will be what I attack this summer for my research practicum.)

I think this may explain why the curve of math achievement related to GDP flattens at higher levels of income — as countries make the shift from industrial to post-industrial, their values shift from survivalist to self-expressive, and self-expressive values are more likely to encourage a degree in Surfing and Beach Management than in something that requires multivariable calculus and differential equations. (Click for a bigger graph).

As countries industrialize, GDP goes up, and values shift from traditionalist to secular-rational. In this phase, math achievement is encouraged and supported, because an industrially-oriented economy needs, most of all, quantitatively skilled human capital. Mean math achievement will improve and students who like math will put even more effort into it. They will be encouraged by their parents and their peers and by future job opportunities.

But then industrialization provides so much wealth that the country shifts to a post-industrial economy, as has happened in many English-speaking countries. Now cultural values move from survivalist to self-expressive. It’s less important to prepare yourself for a high-paying job than to “do what you love.” The money will follow.

Crudely put, does a culture reward effort or expression? Math success requires effort and it doesn’t really help someone express themselves (just read some of my more statistically-oriented blog posts and you can see that!)

I’m not saying an emphasis on expression is wrong. But it doesn’t contribute to high math achievement. Improving teacher quality isn’t going to change that. Context and culture matters when it comes to academics.

Adjusting for response styles in cross-cultural research

I’m working on a cross-country study of math achievement scores related to liking-for-math and ran into some problems with the measure I’m using for liking-for-math. Some countries show extremely skewed distributions on the liking-for-math index I constructed.

Given the obvious differences in patterns of responses across countries, can I really make cross-country comparisons? I tried, and got statistically significant but weak results. But if my measure isn’t any good, I can’t trust those results.

Turns out this problem of “extreme response styles” is well-known in cross-cultural psychology. And there’s a related literature covering measurement invariance, asking the question whether you can compare psychometric results across cultures or other diverse groups of people.

I did a tiny bit of research this morning to see if there’s anything I can do to adjust for differences in response styles across students and countries on TIMSS math background items. I had already tried weeding out the countries that show extremely skewed response styles but then I didn’t have enough data to run the analysis. And anyway, throwing out a bunch of data isn’t a good solution.

Buckley (2006) suggests a Bayesian approach that estimates a posterior distribution for each student characterized by a location shift and a scale adjustment that represent how a student’s responses relate to his or her actual attitude. For example, a large positive location shift and a reduced scale would typify extreme acquiescence, as the student picked mostly “strongly agree” type items. Buckley also provides a quick-and-dirty linear regression tactic for estimating a student’s latent true score on the measure taking into account extreme or random response styles. I may give that a shot this morning — the class project is due next week so I have some time — and then later explore a Bayesian solution.

It’s so cool to see my two interests — cross-country psychological studies and Bayesian stats — colliding. Seems like a potential dissertation topic.

Reference

Buckley, J. (2006). Cross-national response styles in international educational assessments: Evidence from PISA 2006. Retrieved from https://edsurveys.rti.org/PISA/documents/Buckley_PISAresponsestyle.pdf

Attitude towards math vs. confidence, liking, usefulness of math in TIMSS

Kadijevich, D. (2006). Developing trustworthy TIMSS Background Measures: A case study on mathematics attitude. The Teaching of Mathematics IX(2), 41-51.

Abstract. This study, which used a sample of 197,707 students from 46 countries that participated in the TIMSS 2003 project in eight grade, examined whether, for a large number of the TIMSS countries, trustworthy TIMSS measures of several dimensions of mathematics attitude can be developed. By focusing on self-confidence in learning mathematics, usefulness of mathematics, and liking mathematics, it was found that both factor validity and reliability of the measures of these three dimensions derived from the raw data was only attained for the students from the United States. However, when scores concerning the utilized attitudinal statements of all subjects were transformed into Guttman’s image form scores, the factor validity and reliability of the three measures utilizing such transformed data was attained for thirtythree countries (N = 137;346). It was found that for all these thirty-three countries mathematics attitude was mostly saturated by either usefulness of mathematics or self-confidence in learning mathematics. A higher mathematics achievement was found for countries where mathematics attitude was mostly saturated by self-confidence in learning mathematics.

It’s not mentioned in the abstract, but if you combine all the mathematics attitude items into one grand attitude-towards-math scale, you get decent internal reliability (alpha above .70) for almost all countries.

This makes me think maybe I ought to use an overall “attitude towards math” score in the next iteration of my model. Or try that Guttman transformation, which doesn’t make any sense to me, so will need to understand what’s going on with it first. How can it eliminate measurement error?

Also, interesting that attitude towards math is either saturated by self-confidence (something intrinsic) or usefulness (something extrinsic), and the intrinsic one predicts higher math achievement.

Puny results for my model of math achievement related to liking math

Here’s a description of the data analysis project I’m working on.

I was so excited to push the button on my hierarchical linear analysis, hoping hoping hoping to see a statistically significant effect of the kind I wanted. And I did!

Here’s what I found:

  1. Per capita GDP, secular-rational values, and survivalist values all predict higher math achievement scores.
  2. Self-expressive values (at the opposite pole from survivalist) predict lower “returns” to liking math. That is, countries that are higher on self-expressive values show lower slopes for a proposed linear relationship between liking math and math achievement.

That second part was what I was really interested in. I hypothesized that a culture that valued self-expression would provide a worse context for math achievement, especially at higher levels of liking math. Students with higher liking for math would be relatively more disadvantaged by self-expressive values. In the graph below, you can see how lower values on the SURVSELF dimension (representing lower self-expressive values, higher survivalist) result in higher mean math achievement as well as a higher slope for math achievement related to liking math.

So why am I disappointed? The “effect size” — the practical magnitude of the effect — was small, even though it was statistically significant.The slopes just aren’t that different at different levels of self-expressive values.

One way of measuring effect size in hierarchical linear models is to report “proportion variance explained” or how much of the variation is accounted for when you add in the predictor of interest.

For finding #2 above, the PVE was just 5%. So self expressive values at the country level don’t explain much of the difference of the slopes of math achievement related to liking math. To use the technical term, those are some seriously puny results.

But still, it is a statistically significant effect in my model, and that at least, is a happy thing. I think there is something to what I’m exploring. The models I ran converged quickly, which my prof said is an indication of a highly informative model. GDP didn’t explain all the variance in mean math achievement — the two country-level value dimensions were statistically significant and practically significant also. That was a somewhat surprising result because both value dimensions are related to economic transition. That traditionalist vs. secular-rational values and survivalist vs. self-expressive values explain significant variance over and above GDP is important, and worthy of some more study.

OLS regression, student level: Math achievement on liking math and cultural values

Here’s a description of the project I’m working on.

I duplicated all country-level variables to student level, then ran a student level regression of math on liking for math (PATM), per capita GDP (GDP), and cultural value indexes (Rational and Self Expressive).

Collinearity was not too high — all variance inflation factors were less than 3.

Coefficientsa
Model Unstandardized Coefficients Standardized Coefficients t Sig.
B Std. Error Beta
(Constant) 382.428 .586 652.822 .000
PATM 4.969 .066 .163 75.521 .000
GDP 4.787 .027 .592 175.670 .000
Rational 23.244 .249 .235 93.291 .000
Self Expressive -33.841 .279 -.361 -121.273 .000
a. Dependent Variable: Math

These are all as I expected.

  • The more a student likes math, the better she does (this is a reciprocal relationship because if you do better at math, you like it better too)
  • Higher GDP is associated with higher math achievement
  • Higher secular rational values are associated with higher math achievement; in other words, traditional values are associated with lower math achievement
  • Higher self expressive values are associated with lower math achievement

The residuals looked fairly normal, though there were more positive ones than negative ones.This could represent an omitted variable (perhaps extrinsic valuing of math? which I could calculate from some other TIMSS questionnaire items) or some nonlinearity in the relationship.

The model has an R square of .321.

Now I’m ready to start running HLM analyses to see if countries with higher rationality and lower self expressiveness offer higher “returns” to liking math.

Data exploration: Cultural values and math achievement

I’ve put together a data set that includes eighth grade math achievement scores from TIMSS 2007, per-capita GDP from the CIA World Fact Book, and country-level cultural values from the Inglehart-Welzel Cultural Map of the World. I also constructed my own Positive Attitude Towards Math (PATM) index, using four self-report items from the TIMSS data set.

Before I model this in HLM, I’m getting to know the data set. I’ve learned from somewhat painful experience that it’s better to know beforehand what’s in the data rather than doing a bunch of regressions and then later having to go back and explore why you’re getting unexpected results.

One issue is potential multicollinearity across my predictors — if two predictors are highly correlated, regression coefficient estimates may not be stable. I expect GDP to be correlated with both the dimensions from the Inglehart-Welzel cultural map, which are these:

  • Traditional vs. Secular-rational – measures the extent to which religion is very important in a society. I’d expect more secular-rational countries to have higher math achievement.
  • Survival vs. Self-expressive – measures the transition from industrial society to post-industrial. As economic survival becomes more assured, people become more interested in expressing themselves. I’d expect this to be negatively correlated with math achievement, because putting effort into math is typically a means of achieving economic success in the job market not expressing your authentic self.

The transition from traditional to secular-rational occurs during industrialization, so of course you’d expect GDP to be positively correlated with the secular-rationality dimension. The transition from survival to self-expressive as well occurs alongside an economic transition.

Here’s a scattermatrix of country level variables (including country mean scores on PATM and math achievement), with lowess fit lines added. While linear fit lines might show me the correlation, I think lowess fits are more informative. Can’t assume every relationship is linear.

So what do I see here?

  • Rational and self expressive values don’t seem to have much relationship. That’s what you would expect, given that they are supposed to represent two distinctly different dimensions of cultural values.
  • GDP is positive related to both rational and self-expressive dimensions. That’s not surprising, given that development of both rational and self expressive values occur alongside economic transitions.
  • PATM is negatively related to GDP, except at the highest levels. That uptick at the end probably reflects the skewed distributions I saw on PATM for some regions with relatively low math scores (e.g., Latin America, the Middle East).
  • There doesn’t seem to be a clear relationship between self-expressive values and PATM, which is not what I would expect. I would think countries that had made the post-industrial shift would show lower mean liking for math.
  • There is a positive relationship between math scores and GDP but it’s not linear. The relationship seems to flatten out at high levels of GDP (which may represent the influence of the transition from survivalist to self-expressive values).
  • Liking for math and math scores are negatively correlated. Why? Countries with generally higher math scores report lower mean levels of liking for math. But if you look within countries, you’ll see that higher liking for math usually means higher math scores. This is exactly why a hierarchical linear model is needed — to be able to model what’s happening within clusters while still taking into account cluster level influences (in this case, cultural values).

As far as potential multicollinearity problems go, it seems I may have problems with having GDP along with the two cultural values indexes in the model.