Studying Studies of Studying

I like reading about the Latest (and greatest) Education Research...but it's worth remembering that a large fraction of all research is wrong, and even if it's right it probably doesn't mean what it seems to mean, and even if it means what it seems to mean, it still probably doesn't have the sort of practical application we'd like to see. So it goes...here's an article about how to look at the data, even assuming that the research wasn't wrong to begin with, at How to Read Education Data Without Jumping to Conclusions (Jessica Lahey & Tim Lahey) | Larry Cuban on School Reform and Classroom Practice

1. Does the study prove the right point? It’s remarkable how often far-reaching education policy is shaped by studies that don’t really prove the benefit of the policy being implemented. ...[A study might show that "doing X works in situation Y" but then we commit to doing X in all situations]... A key step in interpreting a new study is to avoid extrapolating too much from a single study, even a well-conducted one like STAR.

2. Could the finding be a fluke? Small studies are notoriously fluky, and should be read skeptically. Recently Carnegie Mellon researchers looked at 24 kindergarteners and showed that those taking a science test in austere classrooms performed 13 percent better than those in a “highly decorated” setting....

3. Does the study have enough scale and power? Sometimes education studies get press when they find nothing. For instance, Robinson and Harris recently suggested that parental help with homework does not boost academic performance in kids. ... For example, ...if some parents who help their kids with homework actually do the kids’ homework for them while others give their kids flawed advice that leads them astray, then parental help with homework might appear to have no benefit because the good work of parents who help effectively is cancelled out by other parents’ missteps.

4. Is it causation, or just correlation? It turns out that the most important way for parents to raise successful children is to buy bookcases....

But none of these are about studies which generate results which simply aren't replicable even when we try to do just what the original authors did. Science Daily recently reported on an analysis of "the complete publication history of the current 100 education journals with the highest five-year `impact factor' (an indicator of how often a given journal's articles are cited in other scholarly work)" at Study details shortage of replication in education research

Replication studies that were conducted by completely new research teams were found to be successful 54 percent of the time. When replications were conducted by the original authors in the same publication as the original findings, 88.7 percent were successful. When replications were in a new publication, but at least one author was on both the original and replicating studies, 70.6 percent of replications were successful.

So when the original authors try to repeat, they report success about two-thirds of the time. Not great, but not really relevant if we want to base policy for other people on the techniques tested...when new people try to do whatever it was that worked for the original people, it works about half the time. And then we can start asking the four questions above.

So, is there a bottom line here? Yes, I think so. The bottom line is "Proceed With Caution."