I read an article yesterday about a new study which ‘proves’ how effective TFA teachers and TFA alumni teachers are. The study, which can be accessed here, claimed that middle school math TFA corps members get an extra half year of learning than non-TFA novice teachers, that middle school reading TFA alumni get an extra half year of learning than non-TFA experienced teachers, and that middle school math TFA alumni get an extra full year of learning than do their non-TFA experienced teacher counterparts.

Anytime I see some educational program being measured in ‘months of learning’ I get nervous. I just don’t think that learning is measured in months, and even if it were, I don’t think that a short multiple choice standardized test would be able to accurately measure it.

When I think of my own math teaching, my goal is to expose my students to as many ‘thought provoking’ questions each class as I can while still developing their skills. Maybe on a good day, with the right subject matter, I can get four or five really good questions that get the students thinking deeply and analyzing what is going on. [An example: If you fold a piece of paper it gets twice as thick. If you fold it again, it will be as thick as four sheets of paper. How many times would you have to repeat this for it to be as thick as the distance from the Earth to the Moon? (Answer: Way fewer than you think. Around 40.)] The purpose of these thought provoking questions is to try to evoke as many ‘aha’ moments as possible. These are the moments where math students make a connection that makes them want to jump up and yell ‘Eureka!’

So this paper was trumpeted by many people on Twitter including Wendy Kopp

So I downloaded the paper to see what it really said, and what it didn’t. What I learned is that there were eight different groups that were compared in the study: 1) Elementary TFA corps math teachers to non-TFA novice math teachers, 2) Elementary TFA corps reading teachers to non-TFA novice reading teachers, 3) Elementary alumni TFA math teachers to non-TFA experienced math teachers, 4) Elementary alumni TFA reading teachers to non-TFA experienced reading teachers, 5) Middle school TFA corps math teachers to non-TFA novice math teachers, 6) Middle school TFA corps reading teachers to non-TFA novice reading teachers, 7) Middle school alumni TFA math teachers to non-TFA experienced math teachers, 8) Middle school alumni TFA reading teachers to non-TFA experienced reading teachers

Of those eight comparisons, for five of them the standardized test score comparisons were not ‘statistically significant.’ But for the other three, well, they had the miracle where two groups outperformed their counterparts by a half year of learning while one group, the alumni middle school math teachers, outperformed their counterparts by a year of learning! Sounds impressive.

So I went through the report and learned in two footnotes on pages 62 and 64 that what most people think of as a ‘half year’ of learning and a ‘year’ of learning isn’t quite what they are talking about here.

For the middle school math corps members, the half year of learning was based on the students getting two extra questions correct out of about 40 on the TAKS test.

The half year of extra learning that the alumni middle school reading teachers got was because the students got one extra question correct on the test. And the amazing year of additional learning accomplished by the alumni math teachers was because they got three extra questions correct on the exam.

I think that TFA needs to back off on the miracle stories. The fact is that new TFA teachers are not much better, if they even are any better, than new non-TFA teachers. Neither are that good, really. Teaching has a big learning curve, but by the time you figure it out, you generally have to wait until next year to have a fresh start with a new group. As far as alumni teachers, yes, I think they are generally pretty good. I’d let an alum teach my kids. But as good as they might be, to ignore the fact that most of the comparisons were pretty neutral and then buy into the idea that when one group of students learns a year more than another group, they will only get, on average, three more questions correct on a multiple choice math test, well that’s the kind of thing that is going to keep me investigating these kinds of claims and spreading the word.

Quite a revelation, Gary. Well done!