Some attendees felt a disconnect that many vendors and researchers are looking at retention (students re-enrolling across terms) and course completion rather than learning itself. But aren’t we doing learning analytics, they asked?

At Pearson, I am working on exactly the problem of retention and course success. I work mainly with data from Pearson’s LearningStudio hosted learning management system. I sometimes have access to learning outcomes data but only very rarely; I can’t count on having that.

The data scientist is not drunk

Is my work a a case of the drunk looking for her keys under the streetlight? I don’t think so, and it’s not just do to with the profitability of institutions or their need to demonstrate adequate program completion rates and year-over-year retention to accreditation bodies.

Students enroll in higher education programs for many reasons, not all of which have to do with learning. We would all hope that after two or four or more years taking coursework a student will have better skills and cognitive capacity than when she started. But bachelor’s and associate’s degree programs today require students to take many classes that are not relevant to them or to their future work, that merely serve as hurdles to jump over to acquire the degree. The learning is not enough, because education serves a signaling purpose in addition to working to improve a student’s cognitive capacity.

As people working in education, we have to recognize the reality of higher education today, that it is only partially about learning. I’m talking from a purely U.S.-based perspective, as that’s my focus. In the U.S. today, bachelor’s degrees are a basic entry ticket to a decent job in many cases. In fact, credential inflation means that now a master’s degree is required for entry into many professions.

Some economists think higher ed degrees primarily function as signals to employers, that they are not mainly about increased learning or cognitive capacity. On this theory, a student’s degree shows that he has qualities valued by employers: conformity, conscientiousness, a willingness to defer gratification. If signaling is true in some situations or in some ways, that means it’s not enough for a student to take some classes and merely learn what they need to use on the job. They need to get through an entire program, arbitrary requirements and uninteresting or useless classes included.

There’s good evidence that signaling theories of education do not tell the whole story about higher education, that returns to education do indeed reflect the additional skills that graduates in various disciplines bring to the job market. The signaling theory may be true in part but as we might hope, education is also about actual preparation and learning–I don’t mean to say it is not.

I also don’t mean to reduce higher education to job market preparation, though I do question degrees that don’t represent a sound economic investment. The cost of post-secondary education today is such that we can’t divorce it from its role in linking people with economic opportunity.

Are learning and course completion orthogonal?

One presenter called completion and learning orthogonal outcomes. At least one attendee in that session took issue with that. We would all hope this is not the case — we hope that students don’t successfully complete classes without learning anything — but can’t everyone think of a class they were required to take but didn’t take anything away from it? I just completed a Ph.D. and virtually all of the cognate classes were worthless, annoying and time-consuming efforts that I had to complete if I wanted my degree. There was almost no learning taking place in those courses, and I am a highly motivated and engaged learner, taking courses that I thought would be interesting. Sometimes students do just need to complete a course, learning or no.

Certainly we should work toward making every course a worthwhile learning experience for students but in the real world there are always going to be some classes that aren’t that for one reason or another. Assuming that completing a particular program is a good thing for a particular student (a somewhat questionable assumption in this era of heavy student loan debt and low-value degrees), helping them get through all their courses successfully regardless of learning is a good in itself.

Background reading:

• Women and coding [Geeky Mom]
• Should all majors, not just computer science majors, learn to code? [Audrey Watters/Hack Higher Education]
• Some things to think about before you exhort everyone to code [Miriam Posner]
• Computer science for the rest of us [Randall Stross/NY Times]

On one hand, I think the answer is easy: if you need to know how to program to progress in your career and achieve your goals, then just do it. I went back to school after completing a bachelor’s degree in economics and philosophy so that I could gain programming skills. I did an M.S. in statistics but spent all my electives in the computer science department, because I knew that if (in the early nineties in Silicon Valley) I could demonstrate programming skills, I would be welcomed into the job market. And I was. Then I found many years later, that statistics + programming + domain knowledge = data science = $$$. Score!

On the other hand, I see serious obstacles for girls and women here. There are a set of reinforcing messages that girls/women receive that keep them from just doing it:

1. Math (… computer programming … physics … etc) is damn hard.
2. If you don’t have the brain for math (… computer programming … etc), don’t bother. It’s too damn hard!
3. Girls don’t have the brain for math (… computer programming … etc).

Our culture thinks that natural aptitude matters more than hard work in figuring out subjects like math and programming. This is not the case in other countries; in Asian countries it is thought that academic achievement including in math and science is primarily due to hard work. It is the performance mindset in action rather than the growth mindset, which says that hard work is almost everything. There’s just a short step from the performance mindset that dominates in the U.S. to thinking that certain groups have natural advantages in learning such subjects and that certain groups have natural disadvantages.

An inevitable outgrowth of this is generalizing from individual cases to a group:

How it Works (xkcd)

I face this pretty regularly even though I work in education, which has a decent balance of women and men, if not lopsided in favor of women in non-tech areas. I recently went to a “big data” internal meeting where I was the only female in attendance. This is not at all unusual for me in my position–when we’re talking tech it’s usually mostly or only men, besides me. I am aware in such meetings that when I say something I am representing female techiehood. If I say something dumb, I have shown that all females are technologically clueless. But if I say something smart I imagine people (/men) may be thinking “well sure, this one woman knows a little bit but she’s not a representative case.”

That sort of bothers me, because I know in some ways I come across as not a representative case for women’s capabilities at large.

I am, however, a representative case of why more women don’t study computer science in college. Despite having taught myself BASIC on my dad’s Apple IIe in middle school, I refused to take Intro Computer Science as one of my distribution requirements in college after hearing from (female) friends that the class was killer. I didn’t believe I could hack it. I came back and took it in my master’s program. To my shock, it was not only easy, it was fun. I got an A+.

If I consider where we need to aim, I’m thinking it’s not at the idea of whether girls are good at math/programming/etc or not but rather from this idea of whether such subjects require natural facility or not. They do not. They require a lot of hard work, for everyone, no matter their gender or their race or their obvious natural facility for things technical.

I do not want to dismiss the very real social challenges women meet in male-dominated environments. I cannot speak much to that as I’ve never felt unwelcome or harassed among male-dominated teams. Quite the opposite: I find working with men exciting.

So ladies/women/girls, dive in. The water’s cold and the current is strong: you’ll get a good workout. Enjoy!

Are undergraduates actually learning anything? [Richard Arum and Josipa Roksa/The Chronicle of Higher Education. For many students, college doesn't improve their critical thinking, complex reasoning, and written communications. 45+% of a sample of college students did not demonstrate any statistically significant improvement on the Collegiate Learning Assessment after two years of college. 36% of students did not show any significant improvement after four years. More disturbingly:

[We] find that learning in higher education is characterized by persistent and/or growing inequality. There are significant differences in critical thinking, complex reasoning, and writing skills when comparing groups of students from different family backgrounds and racial/ethnic groups. More important, not only do students enter college with unequal demonstrated abilities, but those inequalities tend to persist—or, in the case of African-American students relative to white students, increase—while they are enrolled in higher education.

An open letter to college admissions committees [Andrew F. Knight/Fairfax Times].

Consequently, the drive for high grades is blinding students and parents alike to the real purpose of education: learning. In parent-teacher conferences, “How can my child bring up her grade?” has replaced “How can my child better learn the material?” The system’s response to angry grade-obsessed parents and disgruntled students has been to fudge the indicator instead of improving the system in other words, to inflate grades in spite of worsening performance. I was routinely pressured by parents, students and even administrators to inflate grades in the form of curving scores, providing extra credit and retest opportunities, and more heavily weighting homework and projects that are easy to copy from friends. It is instructive to note that two-thirds of our students are on the honor roll. (That’s right.) When a majority of students routinely receive As and B’s in all their classes, the distinctions intended by a traditional A-F grading scale become hazy and meaningless.

What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study [Tamim, Bernard, Borkhovski, Abrami, & Schmid/Review of Educational Research]. A meta-meta-analysis of research on technology usage in education. Found random effects mean effect size of .35, statistically significantly different from zero. I have to wonder if that is meaningful in any way given the incredible variety of ways technology can be applied to learning. Have not read the full paper, only the abstract.

Health correlator: Calling self-experimentation N=1 is incorrect and misleading [Ned Kock/Health Correlator]. Self-experimentation is longitudinal, so n > 1. But results may not generalize to other people. Good for learning what works for you.

I love January, full with possibility and empty of regrets. Last year’s disappointments and failures–wiped clean. All January holds is promise for the coming year.

Spring, on the other hand, demands achievement, not just effort and inspiration. Daffodils and tulips should grow tall and green before bending down with heavy flowers. Crabapple trees must cover themselves with leaves then buds then an outrageously excessive display of flowers. Grass will shake off its winter chill to transform back into a thick blue-green carpet that provides the perfect backdrop for the wonders of the spring garden.

But that’s just the garden. It’s the same story everywhere. Work projects and commitments started in the bracing cold of January must blossom in the sunlight and lengthened days of April. Even then, summer threatens with child care arranging, summer camp provisioning, and family travel planning challenges. It’s not enough to just dream of possibility any more. Possibilities must be turned into executed reality. Or not, in some cases.

The difference between me and a bulb is that a bulb
has all that stored up energy plus a natural urge to do exactly what it is supposed to do. The Cheerfulness daffodils that grow among my magenta moss phlox will automatically produce double white flowers with pale yellow centers. The Claudia tulips that encircle my crabapple trees know to grow lily-flowered purple with white tipping.

I am neither so energetic nor so properly directed as a bulb. That is why every year I find spring a challenge even as I welcome the warmth and beauty it brings.

I am more like a crabapple tree than a tulip or daffodil, I suppose. I’m sure my husband and children would seize upon the “crab” in that admission. But there is more. I will bloom in the spring only if everything has gone right for me before that. Even then, I might refuse to do it if it’s not my year. I might throw out one solo branch of flowers, like one of my trees did one spring, taunting those around me with possibility if not aesthetic pleasure. I might get my buds ready to open right before a late Denver snowstorm then drop them unbloomed onto the ground, mocking those who anxiously awaited a flower show. I will only rarely do what a flowering crabapple tree should do–which is to flower, reliably and appropriately, each spring.

It’s hard to be a crabapple tree when you should be a daffodil instead.

The people of the petabyte [Venkatesh Rao/Forbes blogs]. An “informal taxonomy and anthropological survey of data-land” based on Rao’s observations at the Strata conference. Apparently everyone’s a data scientist now:

The taxonomy part is simple. Apparently the list of species in data land is very short. It has only one item:

• Data scientist

What is the value of big data research vs. good samples [from LinkedIn Advanced Business Analytics, Data Mining and Predictive Modeling group]. Interesting and lengthy discussion from LinkedIn’s Advanced Business Analytics, Data Mining, and Predictive Modeling group on whether/when sampling vs. big data sets should be used.

The real-world experiment: New application development paradigm in the age of big data [James Kobielus/Forrester].

This year and beyond, we will see enterprises place greater emphasis on real-world experiments as a fundamental best practice to be cultivated and enforced within their data science centers of excellence.  In a next best action program, real-world experiments involve iterative changes to the analytics, rules, orchestrations, and other process and decision logic embedded in operational applications. You should monitor the performance of these iterations to gauge which collections of business logic deliver the intended outcomes, such as improved customer retention or reduced fulfillment time on high-priority orders.

Social learning Analytics: Five approaches [PDF] [Rebecca Ferguson and Simon Buckingham]. Five categories of social learning analytics: social network analytics, discourse analytics, content analytics, disposition analytics, context analytics. All things I want to learn more about.

4chan’s Chris Poole: Facebook & Google are doing it wrong [Jon Mitchell/ReadWriteWeb]. Google and Facebook have a crude notion of identity.  4chan’s Chris Poole says we are like multi-faceted diamonds.

“The portrait of identity online is often painted in black and white,” Poole said. “Who you are online is who you are offline.” That rosy view of identity is complemented with a similarly oversimplified view of anonymity. People think of anonymity as dark and chaotic, Poole said.

But human identity doesn’t work like that online or offline. We present ourselves differently in different contexts, and that’s key to our creativity and self-expression. “It’s not ‘who you share with,’ it’s ‘who you share as,’” Poole told us. “Identity is prismatic.”

Permission to be horrible and other ways to generate creativity [Suzanne Axtell interview of Denise R. Jacobs/O'Reilly Radar].

“… there’s such a limited definition of creativity in our culture. People treat artists as if they’re off in their own world or put them on a pedestal. But it’s a misconception that technical people aren’t creative. Developers and coders and database architects are extremely creative, just as scientists are. They have to come up with solutions and code that have never been written before. If that’s not creativity, I don’t know what is.

I’m reading “A Whole New Mind” by Daniel H. Pink, which explores how right-brain is the new wave. We’re entering a new conceptual, high-touch era whereas before we were in a very analytical era. Our industry, the technical industry, is actually a perfect in-between point of left brain and right brain. You have to have both, a whole-brain approach, to be successful in our industry.”

Colleges misassign many to remedial classes, studies find [Tamar Lewin/NY Times]. This is something learning analytics ought to be able to fix.

M-theory vs. the x-y plane

Campbell drew an analogy to cosmology, contrasting 11-dimensional m-theory with the planar (two-dimensional) Cartesian coordinate system. He suggested that current work in learning analytics is like working in the x-y plane when we know that education and learning takes place in at least 11-dimensions.

Learning analytics, as practiced today, is reductionist to an extreme. We are reducing too many dimensions into too few. More than that, we are describing and analyzing only those things that we can describe and analyze, when what matters exists at a totally different level and complexity. We are missing emergent properties of educational and learning processes by focusing on the few things we can measure and by trying to automate what decisions and actions might be automated.

As I was writing this post, @webmink Simon Phipps tweeted about his post leaving room for mystery, in which he proposed that some problems will remain unsolved, some systems unanalyzed:

The real world is deliciously complex, and there will always be mysteries – systems too complex for us to analyse. It seems to me that one of the keys to maturing is learning to identify those systems and leave room for them to be mysteries, without discarding the rest of rational life.

Then Simon shared a definition of reductionism with me:

@annez Reductionism: the discarding of the parts you don't understand from a problem you can't solve until you're left with one you can.—
Simon Phipps (@webmink) March 04, 2012

This echoes exactly what Campbell said in his presentation:

My fear is that computers as they are used in learning analytics mean that people will work on simpler questions. They may be complicated in terms of the scale of the data but they’re not conceptually rich. They won’t be trying more concepts or playing with new ideas.

We’ll have a map that makes the territory far simpler than it truly is and we’ll design school to that, not to the true complexity.

Reductionism in analyzing online discussion threads

Last week in a meeting one of my colleagues pointed out the inherent reductionism of our approach to the problem of measuring and characterizing student interactivity and learning via discussion threads. He pointed this out not as a criticism but as recognition and acknowledgement. We are applying a custom-developed coding scheme to threaded discussion posts. We code each post into one of four categories based on the pattern of topics discussed in each post and across the thread. We capture what topics were introduced, how they relate to topics in previous posts, and how they relate to the main discussion topic. We cannot capture all the details and complexity of what people have written and how they have interacted. We certainly aren’t paying any attention to the broader experiences and connections that individual students bring to the discussion. But we are trying nevertheless to capture some important kinds of meaning and interaction in the posts via our coding scheme.

This is, at heart, the analytics endeavor: to take very messy humanly-meaningful information and transform it into numbers that a computer can manipulate. It can be done in more sophisticated and subtle ways or more crude and careless ways, but it is always reductionist. It does not fully capture the human experience of learning. We can’t model learning in all its complexity.

The math is not the territory

I see it as critical in data analysis to remember that our numbers are useful shorthand — easy to manipulate, summarize, visualize, and report upon — but they are not the thing we are interested in. We use them because there is something else non-quantitative we are interested in, something human (at least in social sciences like education).

Campbell said,

We tend to believe the math is the territory and we tend to organize ourselves around just what we’re able to measure instead of organizing ourselves around creating better measurements of what we know to be nearly unimaginably complex.

The math is not the territory — the codes and numbers we use to represent human understanding and action and connection are not the territory — the visualizations are not it either.

Learning as delicious mystery?

Simon suggested some things are too complex to be answerable and should be left as mysteries. Is learning something that should be left unanalyzed? Certainly not, although aspects of it are mysteriously wonderful and not amenable to quantitative or qualitative analysis. There’s too much at stake — for individual students who benefit from success defined in many different ways, for the government that funds or subsidizes much of their education, for the citizenry that benefits from an educated populace.

I believe analytics can help, but I feel humble about its possibilities, more so than ever after listening to Campbell speak. I used to call my stance “cynicism” but I think I will reframe it as “humbleness” which makes it seem like there is some chance of success. As uncomfortable as it was, I’m glad I sat in on Campbell’s talk and listened to it again this morning to think about it further.

The studies are both counter-intuitive and disturbing. The monkeys that enjoyed playing with wooden puzzles until given their favorite treat reward for solving the puzzles, at which time their puzzle-solving diminished. The kids given ribbons for their drawings then showed less interest in drawing. The writers shown a list of possible external reasons for writing immediately wrote less complex and interesting poems than those shown a list of intrinsically-rewarding reasons for writing. And on and on and on and on. Animals, humans, children, adults, across wide-ranging domains and in studies conducted by dozens of independent researchers.

If 99.9% of big data is irrelevant, why do we need it [Michael Wu/Lithium Lithosphere blogs] Lithium’s Principal Scientist of Analytics Wu says “Just because you can track, store, and analyze big data, doesn’t mean you should.” He argues that in many cases you can answer the questions you need to answer just by getting the relevant data — which might be able to be loaded and analyzed on a beefy computer.

Lazily musing about sharing [JP Rangaswami/Confused of Calcutta]. “Sharing is serious business” — it has serious consequences for businesses, especially for those built upon not-sharing. Five ideas about sharing:

1. For anything to be social, it must be shared.

2. Sharing, the act of making social, happens because people are made social.

3. Sharing is encouraged by good design.

4. When you share physical things like food, sharing reduces waste.

5. When you share non-physical things like ideas, sharing increases value.

Want to get value our of your data and analytics investment? Then deal with this issue before you buy the software [Maz Iqbal/B2C Business to Community]. People don’t think statistically correctly, even professional statisticians. Getting the right data into systems that can analyze it is the easy part. The hard part is:

Getting managers to give up their pet theories, their ideological convictions, their vested interests, their intuition, their past experience and use data and analytics to make decisions. That is the central issue that you have to and should deal with.

Now I’m convinced that’s all wrong. The self that matters isn’t some tightly defined, self-loving, individuated thing in the world. The self that matters is the mashed-up self, the networked self — the self made up of relationships and experiences and interactions and ideas. It’s way bigger and more powerful than the un-networked you.

These are some ideas I want to explore: combinatorial creativity, connectivist learning, the third person perspective in the creative process, and self-transcendence. What all these have in common is they all overturn the idea that the individuated self is primary.

Writer and artist Austin Kleon on how we are mashups:

We can pick our teachers and we can pick our friends and we can pick the books we read and the music we listen to and the movies we see, etcetera. You are a mashup of what you let into your life. [via Maria Popova]

Maria Popova on networked knowledge and combinatorial creativity:

Which is interesting, recognizing not only the absolute value of content but also its relational value, the value not just of information itself but also of information architecture, not just of content but also of content curation….

The idea that in order for us to truly create and contribute to the world, we have to be able to connect countless dots, to cross-pollinate ideas from a wealth of disciplines, to combine and recombine these pieces and build new castles.

This relates to how we conceive of ourselves. Are we distinct individuals with hard boundaries? Or are we somehow only ourselves when you consider how we fit into a network of experiences, people, and knowledge?

Robert Fritz, author of The Path of Least Resistance, is someone who might be called a creativity guru but instead of that I would call him a creation guru. He doesn’t write so much about being creative as about actually creating. In his view, the self that we want so much to develop and pin down should be set aside:

Don’t try to define yourself, instead, suspend the question. That gives you a better lens by which to create anything you want to be.

Here’s what Fritz discovered when he tried to help people become more creative by improving their self-image:

Questions of identity got in the way of the creative process, even when people thought well about themselves.  It took years to come to understand this.  Everything pointed to a fairly simple and, yet, revolutionary pair of insights.  The first was that what people thought about themselves, good, bad, or indifferent, wasn’t going to change.  All notions of self-esteem training are predicated on the idea that people can change how they see themselves.  This is one reason they don’t work.

The other insight was just as major.  It is that your view of yourself has no place in the creative process.  Simply put, the moment you make your success or failure about you, that’s the moment you can’t learn what you need to learn, experience what comes with the creative territory, and keep your focus where it needs to be, on the outcome you are working to create, and where you currently are in the creative process.

In his books, Fritz suggests that creators need to use a third-person perspective that takes themselves out of the equation rather than the first-person which makes creation all about the self, the I, the creator herself. Creation doesn’t grow out of some authentic, independent self. It launches from a networked self which is almost like no self at all.

Maybe the reason that thinking too much about our identities as distinct individuals stops us from creating is because creation comes through mashing up, through navigating networks of people and knowledge and ideas, not from the perspective of one isolated node in the network. The node alone is useless.

So we need to stop thinking so much about our individual selves — we need to transcend ourselves. Interesting that  some of the most satisfied people combine a love of the new with persistence and self-transcendence. These seem like exactly the traits you’d need to succeed in a networked world. Neophilia (novelty-seeking, love of the new) draws you to new ideas, new people, and new experiences, giving you more material for the mashup that is you and the mashups you create. Persistence keeps you from being merely a dilettante, flitting from one new thing to another. And self-transcendence stops you from thinking that it’s all about you.

It’s really freeing to realize your self alone is this puny, incompetent thing whose self-love or self-loathing matters not a bit. It’s your networked mashup self that matters, that’s capable of doing and creating great things.

* With credit to JP Rangaswami for his “musing lazily about” series. I like to be able to wander around a topic without reaching any conclusions or forcing it into some structure that might obscure the evolving ideas.

** Which shall remain unnamed and unlinked because I’m so beyond what I wrote then even though I feel like there were some really great ideas that I’d like to expand upon and refine. E.g., busy vs. bursty.

is this an iron rule of innovation in information technology—that the cheaper information becomes and the easier it becomes to manipulate it the greater will be the gap, productive and otherwise, between the informationally capable and the rest? …

We might well be in an intial phase of the information age in which technology amplifies cognitive gaps which gives way to a period in which technology mutes those gaps.

Our greedy colleges 2.0 [Andrew Gillen/Inside Higher Ed]. The Bennett Hypothesis says that increases in federal financial aid subsidies enable colleges to raise their tuition without concern for what students can actually afford. Study described here found that aid directed to low-income students is less likely to lead to tuition increases compared to aid directed at relatively affluent students.

A modeled student [Cathy O'Neil/mathbabe]. Do systems that recommend courses and majors for students reinforce discrimination?

Economics of the cold start problem in talent discovery [John Horton/Online Labor]. Novices can’t get hired if their talent won’t be revealed until after they get hired. Some empirical evidence. One possible help: “talent revealing sites like StackOverflow and Github as replacements for traditional resumes.”