The Economist on the data deluge

Been waiting for this week’s Economist ever since I heard that it included a special report on managing information. Got it today.

Three things I liked in the first article Data, data everywhere

1. We need to train people to work with data.

Alex Szalay, an astrophysicist at Johns Hopkins University, notes that the proliferation of data is making them increasingly inaccessible. “How to make sense of all these data? People should be worried about how we train the next generation, not just of scientists, but people in government and industry,” he says.

When will K-12 get the clue that top math students (really, all math students) need to learn probability and statistics in addition to or possibly in place of calculus? To be fair, the latest high school math standards I’m familiar with do include some coverage of probability and statistics. But at the AP level, the calculus class is much more rigorous and more respected than statistics. Yet many students would be better off learning statistics.

2. Statistics is sexy.

a new kind of professional has emerged, the data scientist, who combines the skills of software programmer, statistician and storyteller/artist to extract the nuggets of gold hidden under mountains of data. Hal Varian, Google’s chief economist, predicts that the job of statistician will become the “sexiest” around. Data, he explains, are widely available; what is scarce is the ability to extract wisdom from them.

oh yeah!

3. Social science is ripe for revolution.

More data and better techniques mean we can understand human behavior at every level better than ever before. 

“Revolutions in science have often been preceded by revolutions in measurement,” says Sinan Aral, a business professor at New York University. Just as the microscope transformed biology by exposing germs, and the electron microscope changed physics, all these data are turning the social sciences upside down, he explains. Researchers are now able to understand human behaviour at the population level rather than the individual level.

2 responses to “The Economist on the data deluge

  1. The pre-medical education requirements did not include statistics but did include calculus. I cannot recall any instance in medical school, residency or my medical practice that a knowledge of calculus was relevant. However, a knowledge of statistics is required almost daily in in the diagnosis and treatment of patients as well as the interpretation of the medical literature.

  2. I definitely see the need for these “data scientists”. While working in applied research I consistently saw vast, and expensive, data collection efforts only partially analyzed. While much data was left alone due to budget and time constraints, I know that applying more comprehensive and statistically-based analysis opened-up new insights. The handful of statistics and probability courses I took after my calculus-heavy engineering training while on the job as a researcher were immediately useful.