Monday, November 5, 2012

Just by taking...

I feel aggravated. I just read High school rigor and good advice: Setting up students to succeed (At a glance)
Our analysis found that a student with above average SES and achievement had a 10 percent better chance of persisting in a four-year institution if that student had taken Pre-calculus or Calculus or math above Algebra II. Low SES/achievement students with high-level math were 22 percent more likely to persist.

That looks like a perfectly reasonable statement about the data. Then it continues with a statement about (cause and) effect:
Advanced Placement/International Baccalaureate courses: Taking an AP/IB course had a dramatic effect on students’ chance of persisting even when students fail the end-of-course test. Low achieving and low SES students who took an AP/IB course were 17 percent more likely to persist in four-year colleges and 30 percent more likely to persist in two-year institutions. The more of these courses a student took, the higher their persistence rates were.
And in the illustration's caption, I see a flat-out causal claim:
Just by taking these high-level courses, low-income, low-achieving students improved their college persistence rates close to their high-income, high-achieving peers.
Well, that's certainly one possibility. Another is that taking the high-level courses had some effect, partly explaining the improvement; a third possibility is that taking the high-level courses had no effect whatever.
Why do I say that? Because signaling is a factor here. Put it this way; some low-achieving students in high school will grow up a bit and start working harder in college. How can you tell in advance which ones? Well, the ones who sign up for hard courses, even if they don't do well and if the courses have no effect on them whatsoever, even if the courses themselves are a total waste of time for all concerned, might be the ones who are going to keep trying until they find something they can do well.
The researchers, it appears, never do group-by-group (e.g. school-by-school) comparisons, or if they did those comparisons are left out of the report. It would be nice to have a state pick two comparable schools lacking in these "tough courses" and offer funding for those courses to one of them, then compare outcomes ten years later (to let 9th graders go through the system and then six years to get through college). Short of that, you might get similar confidence by weighting the results for group comparisons, but the summary here looks pretty close to useless.
Or then again, I might be totally wrong.

No comments:

Post a Comment