Evidence-Based Urban Legends

Education treats the idea of "evidence-based" a little differently than the rest of the data-using world. We have fallen in love with the idea of a number. That number does not necessarily have to relate to the concept we are trying to support; if it is a number, it is good enough.

http://davidmlane.com/ben/

I learned my favorite evidence-based urban legend in a curriculum training. According to the slide (which was supposed to justify the county's choice in curriculum), 62% of students who took Advanced Placement (AP) courses at the time of the survey went to college.
Let's break this down, shall we? First of all, an AP course is a college level course. Any child who took the course and passed the test earned college credit. If 62% of the students passed the test and NOT ONE ever stepped foot on a college campus, some would still report the statistic the exact same way.
Now let's pretend that 62% of those students actually went to a college campus and even graduated from college. There's still a problem: the students who took the AP classes at the time of the survey were HAND PICKED for their performance and ambition. 100% of them were predicted to go to college. Only 62% of the students the school system designated as college bound actually made it.
Let's follow the extrapolation further. The curriculum publisher was telling us that these numbers meant that the AP courses CAUSED 62% of students to go to college. Therefore, if we put all of our students in AP courses, our college rate would rise substantially, to 62%. Suspiciously, even though the slide data was nearly 20 years old, there have been no progress monitoring reports  or updates since its original publication. I personally schedule each of my home children for 2 AP classes to give them a 124% chance of attending college.

http://davidmlane.com/ben/outlier.gif