Tag Archives: experiments

Links for March 11, 2012

Depression: A genetic Faustian bargain with infection? [Emily Deans/Evolutionary Psychiatry]. Discusses the Pathogen Host Defense (PATHOS-D) theory of depression described by Raison and Miller [pdf]. Genes that make people susceptible to depression may also protect them from infection. Depression is associated with brain inflammation; inflammation is also part of the immune response that combats infectious disease. “Since infections in the developing world tend to preferentially kill young children, there is strong selection pressure for genes that will save you when you are young, even if those genes have a cost later in life.”

The people of the petabyte [Venkatesh Rao/Forbes blogs]. An “informal taxonomy and anthropological survey of data-land” based on Rao’s observations at the Strata conference. Apparently everyone’s a data scientist now:

The taxonomy part is simple. Apparently the list of species in data land is very short. It has only one item:

  • Data scientist

What is the value of big data research vs. good samples [from LinkedIn Advanced Business Analytics, Data Mining and Predictive Modeling group]. Interesting and lengthy discussion from LinkedIn’s Advanced Business Analytics, Data Mining, and Predictive Modeling group on whether/when sampling vs. big data sets should be used.

The real-world experiment: New application development paradigm in the age of big data [James Kobielus/Forrester].

This year and beyond, we will see enterprises place greater emphasis on real-world experiments as a fundamental best practice to be cultivated and enforced within their data science centers of excellence.  In a next best action program, real-world experiments involve iterative changes to the analytics, rules, orchestrations, and other process and decision logic embedded in operational applications. You should monitor the performance of these iterations to gauge which collections of business logic deliver the intended outcomes, such as improved customer retention or reduced fulfillment time on high-priority orders.

Campbell et al. on experimentation and quasi-experimentation

Ph.D. Topics : Research and Evaluation Methods

For my Ph.D. comprehensive exam, I not only have to respond thoroughly and knowledgeably to essay questions, I need to cite sources. This part of academic life feels odd to me, this reliance on citing someone else rather than making a good argument. I attended a dissertation defense spring quarter and found it strange that the defender spent a lot of time citing this or that book or article rather than actually intellectually arguing for particular positions. I guess when you’re talking about SEM fit index cutoffs that makes some sense, as one of the best intellectual arguments for them may be the results of a simulation study. But in many other cases, I think you’d want to back up your citation with some rhetoric.

I do agree you need both: you need expert works you can cite and you need to make good arguments. Anyway, if I want to pass my comps, I must learn and memorize the key authorities and works to cite. Ideally I would read and study all these works myself but in absence of the time to do that, at least I can learn more about them than just the authors and dates. It would feel intellectually dishonest to me to cite these works without having a really good idea what they are about and what is in them.

As far as experimental and quasi-experimental design goes, the key authority is clearly Donald Campbell and the three works I see cited over and over are:

  • Campbell, D. & Stanley, J. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand-McNally.
  • Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston, MA: Houghton Mifflin Company.
  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

These are actually three versions of the same seminal work that began with a book chapter in 1963, was published as a small book in 1966, “greatly expanded” in 1979, and issued in a new edition in 2002, that is “encyclopedic in its coverage” (Rosenthal & Rosnow, 2008).

Campbell & Stanley (1963, 1966) introduced the terms internal validity and external validity while the later Cook & Campbell (1979) edition added statistical conclusion validity and construct validity (Rosenthal & Rosnow, 2008). The first version of this work also introduced the term quasi-experiment.

Here is a pdf of chapters 1 and 14 from the 2002 edition, covering general topics in causation and experimentation as well as a self-critique of their work. I think I’ll print it out and read it.

Reference

Rosenthal, R. & Rosnow, R.L. (2008). Essentials of Behavioral Research: Methods and Data Analysis. Boston: McGraw Hill.

Experimental and quasi-experimental research designs

Ph.D. Topics : Research and Evaluation Methods

In an experimental design, subjects are randomly assigned to groups for different levels of treatment (or no treatment, i.e., the control group). In a quasi-experimental design, subjects are not randomly assigned to treatment; there is no randomization.

Random assignments of subjects helps control for participant differences, one of the main sources of threats to internal validity of a research study. Random assignment of subjects doesn’t guarantee that there are no participant differences; especially with smaller sample sizes, you may need to take steps to control for participant differences even after randomly assigning them to treatment levels. For example, administering a pre-test will control for different levels of ability or achievement prior to intervention. Measuring moderators such as demographics (gender, age, race, socioeconomic status) and including those in your analysis may help further isolate causal relationships between interventions and outcome. In some experimental designs, researchers may match participants across control and treatment so that each pair of participants can be treated as one virtual participant (Gliner & Morgan, 2000), giving a pseudo-within-subjects design.

A randomized experimental design with pre-test and post-test controls for threats to internal validity from participant characteristics but leaves some threats uncontrolled, specifically testing effects and bias from selective attrition. Testing effects–for example the possibility that taking a pre-test will help both control and treatment group participants do better on the post-test thus obscuring the actual treatment effect–can be controlled by a Solomon four-group design (Gliner & Morgan, 2000). In this design, there are two control groups and two treatment groups (assuming just one level of intervention). One control group and one treatment group takes a pre-test; the other control and treatment groups do not. This allows the potential testing effect to be teased out.

What if you can’t randomly assign subjects to treatments? This is a common problem in educational and other social settings. For example, if you are testing the introduction of a new curriculum it is unlikely you can randomly assign students to that curriculum. Students come to you in intact groups. In this case, you may choose to use a quasi-experimental design in which treatments are assigned to groups. The treatments may be assigned randomly (e.g., pick classrooms out of a hat to decide which intervention they will get) or purposively (e.g., principals at a school may have some say over which classrooms get an untested vs. the standard curriculum).

Like experimental designs, quasi-experimental designs may be improved by the use of a control group, measuring moderators and incorporating them into the analysis, or matching participants on factors that relate to the measured outcomes (Gliner & Morgan, 2000).

One type of quasi-experimental research design is the time series design, in which many observations are made over time, both without intervention and with intervention (Gliner & Morgan, 2000). Multiple observations are used to establish a baseline that shows an (ideally stable) level of the outcome of interest over time. Then multiple observations are made during intervention, ideally showing a change due to intervention. Then the treatment may be withdrawn, again in an attempt to isolate the relationship of treatment to observed outcome. This may be used with or without a control group. A single-subject design is a common time-series design, in which one or a very few subjects are followed through one or more baseline and treatment phases.

References

Gliner, J. A., & Morgan, G. A. (2000). Research Methods in Applied Settings: An Integrated Approach to Design and Analysis. Mahwah, N.J: Lawrence Erlbaum.