Jun 09 2012

Do effective teachers teach three times as much as ineffective teachers?

I do not deny that some teachers are more talented than others.  My first year I was a very ineffective teacher who spent half of each class trying to maintain order.  My second through fifteenth years, I was much better.  But even within the very small sample size of myself, there is variation.  Some days I am better than other days, some weeks I am better than other weeks.

When I have a ‘bad’ week, my students don’t do as well on my test as they would have had I had a ‘good’ week.  The biggest variable, I think, for me, is that my ‘bad’ week tires me out and then I’m not able to give as much individualized attention for students who are struggling.

But aside from my first year where I really only ‘taught’ for half the time, there is not a giant difference, in terms of final test scores, between a ‘good’ year and a ‘bad’ year.  That doesn’t mean that I don’t inspire kids in different, unmeasurable ways, when I’m really on, but in this current ‘data driven’ climate, that doesn’t really matter anyway.

An often quoted ‘statistic’ by various ‘reformers’ is that an effective teacher is three times as good as an ineffective one.  Sometimes it is said that the ineffective teacher gets a half year of progress while the effective teacher gets one and a half years of progress.

I don’t doubt that there are a small percent of teachers who have little classroom control, mostly new teachers, who only manage to get a half a year of progress.  I also can imagine a rare ‘super-teacher’ who somehow gets one and a half years of progress.  (I think I’m a pretty good teacher, but I doubt I get a year and a half worth of progress.)  I don’t think there is a very accurate way to measure this nebulous ‘progress’ aside from test scores, but I could still imagine that there is a ‘true’ number, even if we will never be able to accurately calculate it.

As this statistic has been quoted by Melinda Gates recently on PBS and by Michelle Rhee in various places, including the StudentsFirst website I thought, in response to a recent post on Diane Ravitch’s blog I would investigate the source of this claim.

It seems they are quoting a 2010 paper by Stanford professor Eric Hanushek called ‘The Economic Value of Higher Teacher Quality.’  I had to waste $5 on this to find who he quotes when he writes “Some teachers year after year produce bigger gains in student learning than other teachers. The magnitude of the differences is truly large, with some teachers producing 1½ years of gain in achievement in an academic year while others with equivalent students produce only ½ year of gain”  So I look in the footnotes and he is quoting HIMSELF from 18 years earlier in a 1992 paper called ‘The Trade-off between Child Quantity and Quality’ which I downloaded from JSTOR.

The main point of this paper, it seems, is that when poor families have many children, it negatively affects the academic achievement of the kids.  So, ironically, this is a paper about how poverty ‘matters’ a lot.

He uses data from 1,920 student from Gary, Indiana (no relation to me), from reading comprehension and vocabulary test scores (no math!) from the years 1971 to 1974.  In his opening paragraph, he is clear about the limitations of his conclusions about variation in teacher quality.

An empirical investigation of trade-offs between number of children and their scholastic performance confirms that family size directly affects children’s achievement. Though parents show no favoritism to first-born children, being early in the birth order implies a distinct advantage, entirely because of the higher probability of being in a small family. Recent large changes in family size explain a portion of aggregate test score declines, but increased divorce rates and market work by mothers have no apparent impact. Finally, teachers are shown to differ enormously, even though performance differences are poorly captured by commonly measured teacher characteristics. (emphasis added) The evidence supports a teacher skill interpretation of differences in classroom achievement.

Other passages that indicate he admits his conclusions are not ‘strong’

(page 91) A subsequent section investigates the possibility that differences in classroom performance are not entirely attributable to differences in teacher skill but instead involve more complicated interactions of teachers and specific classes of students.

And again

(page 92) This finding of little bias from omitting school inputs, however, could well be an artifact of the specific sample. All children are drawn from the same school system, thus limiting the variation in school inputs and the range of correlation between family and school characteristics. The systematic effects of families are portrayed in the top portion.

On page 105, there he concludes something that is bad news for TFA

Within this sample, there is no indication of differences in performance for male and female teachers, but white teachers do significantly worse than black teachers. The sampled students are all black. Therefore, this result may reflect either that black students do better with teachers of their own race or that the white teachers that are attracted to this setting are otherwise poorer, given their measured characteristics.

For the very small sample size, and the admitted biases in the data, this paper has definitely not earned the right to be quoted 20 years later as ‘proof’ that firing ‘ineffective’ teachers is the answer to closing the achievement gap.

Even if there are a few teachers who get one and a half years of growth while other teachers get only one half a year of growth, he doesn’t indicate what percent of these superstars there are and what percent of incompetents there are.

Basically, this would be comical if this myth wasn’t being used to shut down schools and fire teachers.

18 Responses

  1. Sean

    “I think I’m a pretty good teacher.” Don’t be so modest! Gary Rubinstein once called you one of the best teachers in the country.

    • Gary Rubinstein

      Ha. I think he was talking about my ‘raw talent,’ which I rely on, but not my work ethic.

  2. Wonderful article. The research (and $5 spent) you did for this topic was exceptional. Teachers, public more so, these days face an unprecedented foe with an unlimited purse to spend. Evidence for which you delve into goes to show that this foe will stop at nothing to paint the picture, as inaccurate as it may be, that educators do not do their job well and are to blamed for the failure/s (debatable as well) of American Schols.

  3. sellario

    Tennessee has this proprietary VAM called TVAAS that’s been in use for ~20 years. Of course, now being used for employment decisions, compensation, school closings, etc.

    Explain this: my wife teaches 4th grade math. Really good teacher. A TVAAS score of 1 = one year of growth. She has a 14. So that means 14 years of growth. She’s not the only one. If you go here: http://edu.reportcard.state.tn.us/pls/apex/f?p=200:1:514547374837739

    you can find many TCAP grades with math gains in the double digits. Not sure about Language Arts.

    How can people have confidence in this measurement? Where’s Michelle Rhee to hold up my wife and say, “We need to have all teachers make 14 years of gains in 180 days!”

    The kids have only been in school 4 years prior to 4th grade. I’m under the impression that the VAM formula that TN uses is a secret, so they can sell it to other states.

    Is this a big, sick joke? Bring on the vouchers!

  4. Ravitch’s and especially Darling-Hammond’s bewilderment was interesting; if one has ever read Hanushek, one would be familiar with this claim.

    One thing to clear up: when Hanushek says that “performance differences are poorly captured by commonly measured teacher characteristics,” what he means (see pp. 109-111) is that having a master’s degree made no difference (a finding that has been overwhelmingly confirmed since 1992), and that class size made no difference either.

    • Gary Rubinstein

      Though he does make those two conclusions, I don’t think that is what he is getting at in this one paragraph summary of his whole paper. I think ‘poorly captured’ means just what it seems to mean, and not a specific thing about one or the other.

      • He means just what he says: there are striking performance differences in teachers, but those differences can’t be captured by pointing to a commonly measured characteristic, i.e., as if one could say, “all good teachers have master’s degrees.”

        That’s not a “limitation” of Hanushek’s conclusions; rather, Hanushek (like all other researchers since 1992) is finding that the performance differences of teachers are mostly due to something other than all the stuff you can read on their resumes.

  5. By the way, before you ever spend $5 on a Hanushek article, check out his website hanushek.stanford.edu, because most of the reprints can be found there. E.g.: http://hanushek.stanford.edu/sites/default/files/publications/Hanushek%202011%20EER%2030(3).pdf

  6. Carrie

    …and email me if you want some other articles. I teach at a university, and one of my privileges is free access to full articles in many journals online.

    • Gary Rubinstein

      Thanks, I will next time.

  7. anne

    Every single time this “study” is cited by Gates, Rhee, Duncan, etc., someone needs to speak the TRUTH.

    My principal repeats the mantra constantly, even though i’ve shared Hanushek’s work with him. It’s a meme that is going to be very difficult to reverse.

  8. Philip Pulley

    Awesome job Gary, I was working on this (in-between the classes I am taking and my family) when Diane blogged that you had tracked this down.

    Now we all need to Tweet, link and post this information so that those calling for change in their own interests will stop spreading incorrect information.

  9. Carole

    I knew there was some kind of Indiana connection. :)

  10. Brent Snavely

    I read it someplace … I wrote it down and then I read it. Thank you for pointing out that The Empress Has No Clothes.

  11. Tim Kozusko

    Ah, the wonderful word of covariates…
    Thanks for looking into this; I’ll keep it handy for my wife’s principal.

About this Blog

By a somewhat frustrated 1991 alum

High School

Subscribe to this blog (feed)

Subscribe via RSS


Reluctant Disciplinarian on Amazon

Beyond Survival On Amazon

RSS Feed