Apache Hadoop is unquestionably the center of the latest iteration of big data solutions. At its heart, Hadoop is a system for distributing computation among commodity servers. It is often used with the Hadoop Hive project, which layers data warehouse technology on top of Hadoop, enabling ad-hoc analytical queries.

I’m starting my first ever project with Hadoop this week–a prototype of an analytics warehouse using Amazon Elastic MapReduce. Colleagues have told me EMR is a great way to get your head around Hadoop-based data processing.

CBO Report: Medicare pilot programs don’t control health-care costs [Megan McArdle/The Atlantic blogs]. McArdle describes what happened with a housing-project demolition program whose pilot studies suggested  much better effects than were actually seen at scale:

The initial study was small and involved highly screened people with a lot of support. And it seems to have suffered from publication bias–the most spectacular results got the most attention, even though these might just have been outliers.

This is distressingly common–not just in government or social-do-gooding research, but in organizations of all kinds–including corporations.

Programs at scale often don’t show results as good as pilot studies of those programs. More generally in program evaluation, it’s hard to find evidence of strong (or even weak) effects of interventions. Social systems are complex; factors other than those targeted by the intervention often determine outcomes. This is something I need to communicate regularly to my colleagues and our partners–student learning is largely determined by factors other than what we have control over. That’s not to say we shouldn’t improve our course design, teaching practices, and so forth but it is to say that there aren’t many easy pickings out there for improving student outcomes.

For-profits vs not-for-profits [Felix Salmon/Reuters blog].

I know full well that a lot of not-for-profit organizations are run in a dreadful fashion; I’m just not convinced that introducing a profit motive is always or even often the best way to fix that problem…. I very much doubt that for-profit education is ever a good idea. I just don’t see how the incentives there could possibly be aligned.

But the profit motive can’t provide optimal outcomes if there isn’t consumer discipline along with it. For-profit higher education is subsidized by the government in the form of grants and low-interest loans (and note that nonprofit education is subsidized in additional ways as well, in the case of public institutions). Would-be students do not have an incentive to seriously evaluate whether the education they are purchasing is worth what they pay, because there is a third-party payer involved. The situation is much like health care. Good discussion in post of the issues and controversy over for-profit higher education.

Virtually all American workers now spend time on teams and some 70 percent inhabit open-plan offices, in which no one has “a room of one’s own.” During the last decades, the average amount of space allotted to each employee shrank 300 square feet, from 500 square feet in the 1970s to 200 square feet in 2010….

Privacy also makes us productive. In a fascinating study known as the Coding War Games, consultants Tom DeMarco and Timothy Lister compared the work of more than 600 computer programmers at 92 companies. They found that people from the same companies performed at roughly the same level — but that there was an enormous performance gap between organizations. What distinguished programmers at the top-performing companies wasn’t greater experience or better pay. It was how much privacy, personal workspace and freedom from interruption they enjoyed. Sixty-two percent of the best performers said their workspace was sufficiently private compared with only 19 percent of the worst performers. Seventy-six percent of the worst programmers but only 38 percent of the best said that they were often interrupted needlessly.

I work in an open-plan office and I rather like it, mainly because my coworkers are fun and because my clean, small, mostly quiet work area is such a nice change from my sprawling, messy, mostly noisy house. We work on a puzzle together when we’re taking a break from work and wear headphones when we want uninterrupted time. I wonder, though, if I’d be more productive with a private office or even a cubicle. I don’t achieve flow as much I’d like at work. Not sure if that’s because the job is relatively new to me or because the work environment is an obstacle.

Hume, causation & science [Barry Ritholtz/The Big Picture]. “We humans love a grossly over-simplified narrative.” Determining when we can attribute causation to a correlation is one of the major challenges of research design and statistical analysis.

How to work from home like you mean it [Kevin Purdy/Fast Company]. I’m thinking of working one day a week at home to achieve some of that flow I’ve been missing. If I do, I’ll follow some of these tips so it doesn’t devolve into eight hours of Internet surfing.

Lack of interest and aptitude keeps students out of STEM majors [Olga Khazan/Washington Post On Small Business blog]. “A study released this week by Georgetown University’s Center on Education and the Workforce found that recent graduates in computer science, mathematics and engineering all had unemployment rates below 9 percent (with the rates dropping below 6 percent among those who had some experience.) Conversely, the rates for graduates in architecture and the arts were 13.9 and 11.1 percent, respectively.”

What is college for? (Part 2) [Gary Gutting/The New York Times].

Concretely, students graduating from high school should, to cite one plausible model, be able to read with understanding classic literature (from, say, Austen and Browning to Whitman and Hemingway) and write well-organized and grammatically sound essays; they should know the basic outlines of American and European history, have a good beginner’s grasp of at least two natural sciences as well as pre-calculus mathematics, along with a grounding in a foreign language.

Students with this sort of education would be excellent candidates for many satisfying and well-paying jobs in, for example, sales and service industries, except for those that require highly specialized skills. From the standpoint of employment, high school graduates would have no need of college unless they wanted to be accountants or engineers, pursue pre-professional programs leading to law or medical school or train for doctoral work in science or the humanities. Apart from this, the only good reason they would have for going to college would be for its intellectual culture.

Compelling idea, but seems unlikely to happen because (1) our high schools are mostly incapable of providing such an education and (2) our culture is overly invested in the idea of college as the basic ticket to success in today’s economy. E.g.: D.C. may require college application for all [Joanne Jacobs].

As much as it was like anything, magic was like a language. And like a language, textbooks and teachers treated it as an orderly system for the purposes of teaching it, but in reality it was complex and chaotic and organic. It obeyed rules only to the extent that it felt like it, and there were almost as many special cases and one-time variations as there were rules. These Exceptions were indicated by rows of asterisks and daggers and other more obscure typographical fauna which invited the reader to peruse the many footnotes that cluttered up the margins of magical reference books like Talmudic commentary.

It was Mayakovsky’s [the teacher's] intention to make them memorize all these minutiae, and not only to memorize them but to absorb and internalize them. The very best spellcasters had talent, he told his captive, silent audience, but they also had unusual under-the-hood mental machinery, the delicate but powerful correlating and cross-checking engines necessary to access and manipulate and manage this vast body of information. (p149)

To be a good data scientist, whether using traditional statistical techniques or machine learning algorithms (or both), you must know all the rules and approach it first as an orderly system. Then you begin to learn all the special cases and one-time variations and you study and study and practice and practice until you can almost unconsciously adjust to each unique situation that arises.

When I took ANOVA in my Ph.D. program, I could hardly believe there was entire course devoted to it. But it was much like Grossman’s description above. Each week we learned new special cases and one-time variations. I did ANOVA in so many different Circumstances that now I have absorbed and internalized its application as well as the design of studies that would usefully be analyzed with it or with some more flexible variation of it (e.g., hierarchical linear modeling). It felt cookbook at the beginning, but at the end of the course, I felt like I’d begun to develop that “unusual under-the-hood mental machinery” that Grossman suggested an effective magician in his imagined world would need.

That’s not to say that there aren’t important universal principles and practices and foundational knowledge to understand if you are to be an effective statistician or data miner or machine learner programmer; it’s not to say that awareness of Circumstances and methodical practice are all you need. It is to say that data science is ultimately a practice not a philosophy and you reach expertise in it through doing things over and over again, each time in slightly different ways.

In The Magicians, protagonist Quentin practices Legrand’s Hammer Charm, under thousands of different Circumstances:

Page by page the Circumstances listed in the book became more and more esoteric and counterfactual. He cast Legrand’s Hammer Charm at noon and at midnight, in summer and winter, on mountaintops and a thousand yards beneath the earth’s surface. He cast the spell underwater and on the surface of the moon. He cast it in early evening during a blizzard on a beach on the island of Mangareva, which would almost certainly never happen since Mangareva is part of French Polynesia, in the South Pacific. He cast the spell as a man, as a woman, and once–was this really relevant?–as a hermaphrodite. He cast it in anger, with ambivalence, and with bitter regret. (pp150-151)

Sometimes I feel like I have fit logistic regression in all these situations (perhaps not as a hermaphrodite). The next logistic regression I fit, I will say to myself “Wax on, wax off” as Quentin did when faced with a new spell that he had to practice according to each set of Circumstances.

[1]Highly recommended, but with caveats. Read it last summer — loved it — sent it to my 15-year-old son at camp. He loved it too and bought me the sequel for Christmas. After reading the second one, I had to re-read the first. It’s a polarizing book. Don’t pick it up if you are offended by heavy drinking, gratuitous sex, and a wandering plot. Do pick it up if you felt like your young adulthood was marked by heavy drinking, gratuitous sex, a wandering plot, and not nearly enough magic. My son tends to read adult books so I didn’t hesitate to share it with him, but it probably would not be appropriate for most teenagers.

Will Amazon offer analytics as a service? [Quentin Hardy/Bits]. Interesting to get an idea what that might look like. I don’t think, though, this would compete with SAS and similar software as the post implies. Would someone looking to implement a product recommendation engine implement it in SAS? Probably not. For example, Google is said to use R for model exploration and prototyping, then puts them into production using Python or C++. I feel a “choosing your analytics tool” post coming on.

Community college budget cuts drive students to for-profit school [Chris Kirkham/Huffington Post]. Balanced coverage of why students turn to for-profit schools and the pros and cons of such choices. My observation: community college tuition is artificially low due to government subsidization while for-profit tuition is artificially high, again because of government interference (in the form of financial aid). No market forces to bring about a reasonable balance between supply and demand. The big losers are students (and taxpayers).

Benchprep is codecademy for any subject, high school to med school [Josh Constine/TechCrunch]. “Eventually, publishers might get a clue that interactive digital education is going to destroy their paper book business. If they’re smart they’ll start developing their own courses or raise licensing fees. Until then though, BenchPrep will be the savior of anyone frustrated by the static book-learning experience.” I’m pretty certain some big textbook publishers see that already.

Forget dieting, try intermittent fasting [Josh Ozersky/Time Ideas]. “And that’s why instead of eating healthier, I’m going for longer stretches without eating so I can actually enjoy a whole meal. I don’t starve myself; I drink a protein shake if I get hungry and consume endless glasses of diet iced tea. People tell me this is bad, that I will soon gain back all the weight I’ve lost – and these rejoinders are always given with a smug malice, as if the people uttering them actually despise me for trying to compensate for the pleasures of the plate.”

I fast most days at work until about 2 or 3 pm, then have a small snack. I eat whatever I want once I get home from work around 5 pm. I find this allows me to eat generally what I want while maintaining my weight at a level I’m happy with. I have found, like Josh, that people get really upset about this plan, almost offended that I would eat this way. Funny how everyone thinks they know what is healthy and what is not, despite the difficulties in determining that (see first link in this post).

College has been oversold [Alex Tabarrok/Marginal Revolution]. The total number of students graduating from college is way up, but the numbers graduating with STEM degrees haven’t increased. That’s bad for individuals and bad for the economy. “An argument can be made for subsidizing students in fields with potentially large spillovers, such as microbiology, chemical engineering, nuclear physics and computer science. There is little justification for subsidizing sociology, dance and English majors.”

You have to break connections to get your ideas to spread [Tim Kastelle/Innovation Leadership Network. Innovation requires disruption. "When you come up with a great new idea, you need to think about this economic network in two ways. The first is: how can I connect to all of the complementary parts of the economy that are needed to get my idea to work? The second is: if I’m going to get my idea to spread, which of these existing connections need to be broken?"

The second economy [W. Brian Arthur/McKinsey Quarterly]. We are in the process of building out the economy’s neural system, what Arthur calls “the second economy” growing up alongside the first economy, the industrial economy. Downside: loss of jobs as computers take over.

Selecting amongst large classes of models [Brian D. Ripley] (pdf). We have the data and the computational resources to “trawl through literally thousands of models (and in some cases many more).” How to pick among them? A subject I intend to learn a lot more about in 2012.

Curing the big data storage fetish [Dan Woods/Forbes]. “One popular way to express lust for big data for its own sake is to create a gargantuan Hadoop cluster.” Not enough to just store the data, need to build a data-driven culture. “But how do you create  a company culture like CapitalOne or Google or eBay or Zynga or LinkedIn, where data is essentially part of the management team? At all of these companies there are data scientists, the elite professionals, but there are also swarms of data enthusiasts, people who are eager to use data to help do their jobs better.”

image courtesy Victorian Web

I did some program evaluation while I finished my degree and sometimes colleagues would lament, “we don’t have enough data!” Usually, we did. We could thank Sir Ronald Aylmer Fisher for that. He worked with small data in agricultural settings and gave us the venerable ANOVA technique, which works fine with just a handful of cases per group (assuming balanced group sizes, normality, and homoscedasticity). Maybe we might give a nod to William Sealy Gosset, too, for introducing Student’s t distribution, which helps when the Central Limit Theorem hasn’t kicked in yet and brought us to normality.

But Sir Ronald and Student can’t help me now. I’m down a rabbit hole… in some sort of web-scale wonderland of big data. I feel like Alice after drinking the magic potion, too small to reach the key on the table. The data is so much broader and bigger than I am, so much broader and bigger than my puny methods and my puny desktop R environment that wants to suck everything into memory in order to analyze it.

I stay awake at night thinking how to analyze all this data and deliver on its promise, how to analyze across schools and courses and so many, many students, not to mention all their clickstreams. How can I get through the locked door and experience the rest of wonderland when I’m so small and the data’s so big? I could sample the data, I think, and then I’d be in the realm where I’m comfortable, dealing with sampling distributions and generalizing to a population and applying the small-data methods I know already. Perhaps I can extract it by subset — by subsets of like courses, perhaps, or by school (I’m doing that already–not scalable and doesn’t address some of the most interesting questions). What about trying out Revolutions’ big data support for R? Or maybe I can apply haute big-data techniques: Hadoop-ify it (Hive, Pig, HBase???) then use simplistic (embarrassingly parallel) algorithms with MapReduce. Problem is, none of the methods I like to use and seem appropriate for educational settings (multilevel modeling for example) are easily parallelized. I’m stumped.

It’s okay to be stumped, I think — part of creation is living with uncertainty:

Most people who are not consummate creators avoid tension. They want quick answers. They don’t like living in the realm of not knowing something they want to know. They have an intolerance for those moments in the creative process in which you have no idea how to get from where you are to where you want to be. Actually, this is one of the very best moments there are. This is when something completely original can be born, when you go beyond your usual ways of addressing similar situations, where you can drive the creative process into high gear. [Robert Fritz on supercharging the creative process]

Alice ate some cake that made her bigger. Is there some cake that will make me and my methods big enough to answer the questions I want answered? For now I’m in the realm of not knowing but I hope in 2012 I will have some answers: first, answers about how to make myself big again, and second, answers from the data.

When I tell people my topic, I usually start with the domain area: social participation as related to life satisfaction in older U.S. adults (my data set is people age 65 and over from the Health and Retirement Study), but really, the topic is a statistical and measurement one. Participation happens to be something I’m personally interested in and fits the statistical problem area, but I could do this same project in a variety of domains with a range of constructs. Maybe I ought to change my elevator speech to start with the statistical/measurement part.

Most psychometrics concerns itself with the measurement of latent psychological constructs like attitudes, intelligence, academic achievement and so forth. Psychometricians have developed sophisticated means of constructing instruments (surveys or assessments, for example) that can measure these latent constructs. The approach taken is often based on either classical test theory or item response theory. Either way, the assumption is that observed data (such as a student’s answers to test questions or a subject’s survey responses) are caused by whatever unobserved trait is intended to be measured.

However, there are some things we want to measure that don’t fit this model. Social participation is one of them. Participation instruments generally ask the respondent to report his or her level of participation in various activities. In a latent factor setting, you would then assume some underlying level of participation that gave rise to the observed frequencies of participation. That’s not quite right though. If someone increases their participation in some area — say by joining an investment club — their overall level of participation goes up. The increase in participation in the investment club seems causally prior to the increase in overall participation. This is the opposite direction of causality than that proposed by traditional psychometric models.

Some people call a measurement instrument developed by some sort of summation of disparate items an index rather than a scale, where a scale follows the latent factor model. The development of such indexes follows a so-called formative measurement model, where what you’re trying to measure is formed of what you observe, in contrast to the development of scales that follows a reflective measurement model, where what you observe reflects the underlying latent factor of interest. In the diagram, the first figure represents formative measurement (observed indicators x1-x3 cause the latent construct eta 1) and the second figure represents reflective (observed indicators y1 to y3 reflect the level of the latent construct).

There has been plenty of criticism of formative measurement, but I think it can be made useful, and that’s the aim of my dissertation project. I’m now at the analysis stage and just beginning to really understand the usefulness and potential of formative indexes.

As an aside, I don’t like to call formative measurement “measurement.” I prefer to think of it as “modeling.” I think what you’re doing with index development is constructing a one- or few-number summary of a lot of individual data items in a way that predicts outcomes of interest. Think of the Apgar score as a good example. It gives you a one number summary of the health of the baby and its likelihood to survive and thrive, but you’re not measuring one thing in particular about the baby. Well, maybe you are measuring overall health. Hmmmm.

To be continued…

It is certainly no accident that the post-idea world has sprung up alongside the social networking world. Even though there are sites and blogs dedicated to ideas, Twitter, Facebook, Myspace, Flickr, etc., the most popular sites on the Web, are basically information exchanges, designed to feed the insatiable information hunger, though this is hardly the kind of information that generates ideas. It is largely useless except insofar as it makes the possessor of the information feel, well, informed. Of course, one could argue that these sites are no different than conversation was for previous generations, and that conversation seldom generated big ideas either, and one would be right.

But the analogy isn’t perfect. For one thing, social networking sites are the primary form of communication among young people, and they are supplanting print, which is where ideas have typically gestated. For another, social networking sites engender habits of mind that are inimical to the kind of deliberate discourse that gives rise to ideas. Instead of theories, hypotheses and grand arguments, we get instant 140-character tweets about eating a sandwich or watching a TV show. While social networking may enlarge one’s circle and even introduce one to strangers, this is not the same thing as enlarging one’s intellectual universe. Indeed, the gab of social networking tends to shrink one’s universe to oneself and one’s friends, while thoughts organized in words, whether online or on the page, enlarge one’s focus.

[via Stowe Boyd]

But big ideas have always rested on some sort of social connection across people and they haven’t always required the written word. Big ideas don’t issue forth from the head of one really smart person, working alone, reading, reading, reading then Eureka! No, they arise from the synthesis of multiple ideas and practical knowledge that together are greater than the sum of their parts. What’s required for this? That people be in some sort of social connection with each other.

Big idea: Cliff dwelling

Last weekend, we visited Mesa Verde, an archeological site and national park that shows how the Ancestral Puebloans lived in the time period from roughly 550 to 1300 A.D. While the cliff dwellings themselves inspired awe (and sometimes required special tickets and patience to tour), it was more interesting to me to see the progression of architecture at sites along the Mesa Top Loop Road. Here’s where you could see the Puebloans build towards the big idea of cliff dwellings. From 550 to 750 A.D., they mostly lived in pithouses on top of mesas but sometimes in cliff alcoves, then progressed to adobe houses clustered into villages. By 1000 A.D. they had developed stone masonry techniques for constructing buildings two or three stories high with 50 or more rooms. It was about 1200 A.D. that they moved their buildings into cliff alcoves, a spectacular form of architecture with practical and aesthetic benefits.

Who came up with the big idea to build a multi-story stone dwelling in the shaded and protected alcove of a rocky cliff? It certainly wasn’t one person acting in isolation and doing a lot of reading and writing. This big idea did not rest upon the written word and lots of deep thought but rather on social knowledge and practice of stone masonry construction and on ideas and experience about where the best place to live was (mesa top or cliff alcove?). If the Ancestral Puebloans did have Twitter, I could imagine them using it share tips about how best to construct a beautiful and sturdy multi-story dwelling, sited for protection and convenience. Most big ideas rest upon a wealth of little bits of knowledge that don’t need to be written up as a journal article or book.

Many ways to connect

How we connect differs depending on where the people we want to connect with are. In the 17^th and 18^th centuries, French intellectuals gathered in salons. In the 20^th century in the U.S. and other western countries, connecting through higher education channels and journal articles and at government-funded research centers was a good bet. Now, some people connect online using tools like Twitter and Facebook and Google+. This doesn’t necessarily replace long-form writing but it does complement it — making the generation of big ideas more, not less, likely.