Sep 13 2012

What ‘value-added’ is and is not

Because of the Chicago teacher strike, more people than ever in this country are beginning to learn about some of the big issues in education reform.  The most important, in my estimation, is also, unfortunately, the trickiest to understand — that is, the use of standardized test scores in teacher evaluation.

There are a lot of people, I think, who when they first hear about this oppose it, but for the wrong reason.  They think it is unfair because poor kids do, um, poorly on these tests and so it seems to give an unfair advantage to teachers in the suburbs.

But then there is a different group of people who are a bit more informed.  They hear that the teacher evaluations are not based on having students achieve a particular score, but on ‘growth’ or ‘progress’ from whatever point the students started at.  In theory, this would make evaluations fair.  Based on the way this is described in the media (not necessarily because they purposely mischaracterize how it works, but because it is nearly impossible — as I will demonstrate, yet again, here — to explain all the subtleties in a few words), I can see why the union might seem unreasonable for opposing this.  The issue is that Race To The Top coerced states to agree to include what’s called ‘value-added’ into teacher evaluations.

What Is ‘Value-Added’?

‘Value-added’ is when information about students is fed into a computer and the computer uses a complicated math formula to try and determine what the standardized test scores should be at the end of the year if that class were to have an ‘average’ teacher.  After the end of year state test scores are calculated, a teachers ‘value-added’ is based on whether the class exceeds or falls short of the computer’s calculation.

The main problem with ‘value-added’ is that is is completely inaccurate.  Teachers who add a lot of ‘value’ one year often add little ‘value’ the next year.  Even within the same year, teachers who teach two different grades are often shocked to learn that they added a lot of ‘value’ to their seventh grade class first period, but little ‘value’ to their second period eighth grade class.

If your job was to help people lose weight, would you want your clients to be weighed on a scale that registers wildly different readings each hour?

What Isn’t ‘Value-Added’?

In an effort to get the ‘gist’ of ‘value-added,’ reporters often use descriptions that make it sound like something worthwhile and fair.  Here are the three most popular ways of doing this:

1)  Value-added is not ‘student learning.’

Often politicians defend their support of value-added being up to 50% of a teacher’s evaluation by saying that teacher evaluation is ‘broken’ because is does factor in how much the students ‘learn.’

Here are some recent examples:

Arne Duncan Speech ‘fighting the wrong education battles’

“We’re still learning about how to improve teacher evaluation and incorporate measures of student learning. ”

Christian Science Monitor

“Among other things, the striking teachers oppose plans to hold them accountable for what their students learn in the classroom.”

The Huffington Post

Tim Daly of TNTP is quoted as saying, “It’s very risky because they’re asking the public to support them in a strike that is about whether they should be evaluated on how much students are learning.”

2)  Value-added is not ‘student achievement’
The Denver Post

“But the key issue in the debate is a teacher evaluation system tied to student achievement.”

NBC Bay Area

“Rahm Emanuel is one of the president’s foot soldiers in that movement, which focuses on linking teacher pay to student achievement”

Duncan in Huffington Post

“Even so, many skeptics protested after Tennessee enacted a state law in 2010 that required 50 percent of a teacher’s evaluation be based on student achievement data — 35 percent on student growth, with the other 15 percent based on other measures of student achievement.”
3)  ‘Value-added’ is not ‘student growth’
NBC blog

“Twenty-four states now require teacher evaluations based on some measure of student growth, according to an analysis by the National Council on Teacher Quality, a research and policy group.”

Chicago Tribune

“Lewis was complaining about teacher evaluations that for the first time will be tied to student academic growth.”

L.A. Times

“Duncan also urged Los Angeles educators to use student growth as a factor in evaluations, something he and Obama have long advocated.”

In summary, ‘value-added’ is nothing but ‘value-added.’  I’d urge all reporters to call it that and not fall for the vocabulary of the reformers.

29 Responses

  1. To be clear, VAM is not “completely inaccurate”, it’s just *often* inaccurate. It also does appear to correlate with long-term outcomes for students.

    • Gary Rubinstein

      You really think so? It doesn’t correlate at all with principal evaluations. It correlates a bit better with itself, but not even that well. As far as long-term outcomes, if you mean the Chetty study, that was not compelling to me.

      • Educator

        The MET study published in 2010 is another large-scale respected study that demonstrates correlations between outcomes and value add. I think there are several very serious issues with VAM but whether you find these studies “compelling” or not, calling VAMS “completely inaccurate” is…well…completely inaccurate. It is largely regarded as the best system that we have to isolate teacher impact – not perfect by any means, but neither is principal observation or any other means we currently have.

  2. I do think that, rather than making blanket claims about “complete inaccuracy”, it’s worth putting actual numbers to the claims. If the correlations are really that low, all the better for the argument. And I certainly heard a lot of people express that they didn’t find the Chetty study compelling, but never heard much of a reason why.

  3. Matt

    What about the research suggesting that value-added correlates with student feedback of teachers? This, along with the Chetty study, provides some decent evidence that value-added is indicative of something meaningful, no? And although you say value-added is volatile from year to year – true – it is actually more reliable than other measures, such as principal evaluations.

    • meghank

      What research suggests that value-added correlates with student feedback of teachers? I understand that achievement test scores are likely to correlate with student feedback of teachers. This is easily explained with both factors relating to student socio-economic status. But what study suggests that value-added correlates with student feedback of teachers?

        • meghank

          “Teachers whose students described them as skillful at maintaining classroom order, at focusing their instruction and at helping their charges learn from their mistakes”

          Those are the only three survey items that correlated with value added. Seems pretty slim evidence to me.

        • meghank

          In addition, this quote, “One notable early finding, Ms. Phillips said, is that teachers who incessantly drill their students to prepare for standardized tests tend to have lower value-added learning gains than those who simply work their way methodically through the key concepts of literacy and mathematics, ” has been proven as a flat-out lie on Ms. Phillips’ part.

          It contradicts the numbers in the published research.

          Those numbers in their own study show that value-added positively correlates with drilling skills. As a result of this, I have doubts about any research conducted by the Gates Foundation.

        • meghank

          Jay Greene’s blog exposing the innaccuracy in the reporting:
          http://jaypgreene.com/2010/12/13/false-claim-on-drill-kill/

  4. It may be that a couple of bare-bones, speculative criticisms in Counterpunch (couched in numerous ad hominems) undermine all of the work of all of the scholars involved in the paper. And maybe they have no adequate response & it won’t get through peer review. But by itself it seems like a pretty thin reason to dismiss the research altogether. Especially given how many other researchers, including many not obviously enamored of VAM, think it looks basically sound.

    • PhillipMarlowe

      From CounterPunch:
      While the H-C study found a statistically significant, if meaningless, relationship between the “value added” of teachers and incomes at age 28, the authors did not find a statistically significant result at age 30. Why? In the study the authors explain this by the small number of 30 year olds in their sample. In their interviews with the media and in public presentations the authors do not mention this result at all. Yet the number of 30 year olds in their sample is 61,639, and these are all students who went to school in the same city. Is this a small sample?
      While Bruno may be an honest man, he is not an honest debater.

    • PhillipMarlowe

      More “ad hominems”, as identified by the honest man:
      The devil is in the details: the average wage and salary of a 28 year old in the H-C study who had an excellent teacher was $20,509 in 2010 dollars, $182 higher than the average annual pay of all 28 year olds in the study. How does this compare to the average salary and wage of a 28 year old in this country? The authors excluded from their study people whose income was higher than $100,000. As we shall see, this exclusion is problematic; but to do the comparison we must do the same. The average salary and wage in 2010 of a 28 year old who earned less than $100,000 a year was $29,041, 42% higher than the income of a 28 year old in the H-C who had an excellent teacher. In other words, even if we accept the numbers that the authors of the H-C study choose to spin, having an excellent teacher cannot pull people out of poverty

  5. madelyn roesch

    Are you people idiots???? Before you can measure anything, you need a baseline, which seems to be mysteriously absent in ascertaining value-added. You also need to determine your variables, which is a monumental job considering how many things impact student learning that have nothing to do with the school or the teacher. Finally, you need to validate the exams–what do they measure, how do they measure it, and how can we prove that the test actually does this? Education research, generally speaking, is severely lacking in rigor of any kind. Most of the papers that have been used to justify the “reforms” would be laughed out of a serious, peer-reviewed scholarly journal. What are the sample sizes? Where are the longitudinal data? This is a travesty of the worst kind and serves only to harm those students who most need help. With the ultimate agenda of desroying public education in this country.

  6. Michael Fiorillo

    Junk science, pseudo-science, scientism, VAM is all of that, but worse, exposing in neutral language the perverse worldview being imposed upon students.

    The promulgation of such a business term – “The enhancement a company gives its product or service before offering the product to customers” – into the schools demonstrates the underlying outlook and motives of so-called education reformers, and their view of children as commodities. After all, the kids are the “products” to be sold to “customers” (employers).

    Its use as a weapon against teachers aside, the term, its practice and the worldview encapsulated in it have no place in the public schools.

    • KrazyTA

      Short, concise, accurate.

      Thanks for posting.

  7. The assumption of the value-added model seems to be that the purpose of schools is to make kids better at standardized tests. What constitutes “value” in the “value-added” model? Bubble test scores. Ugh…of all the attributes that make a person a capable human being both as an individual and as a member of society, that’s the one that gets the attention? The other day I heard a teacher telling students they needed to behave well because they had to be ready for the state exams. This is so sad to me…it was not behave well so we can explore this topic, or behave well to show respect for the material and the instructors, or behave well to be at peace with the other people here…no, behave well so you will be ready for a test. All hail the test. Ugh.

    You cannot tell me with a straight face that a student of mine who speaks and reads three languages, holds down a good job, and shows genuine curiosity about literature, is showing less “value” than a slacker peer who happens to have only studied in English, and because of that English dominance does better on the state reading exam. But “value-added” says I did a better job working with the latter. The first student could probably get a better score by focusing only on English and by quitting work, but would that really be more valuable to her or to her community?

    Don’t get me wrong, standardized tests can be a useful diagnostic tool, and of course often poor scores indicate poor skills, and great skills indicate great skills, but this is not always so. Tests should not be an end goal in and of themselves!

    • Sorry, that should be “and great SCORES indicate great skills,” not “and great skills indicate great skills.”

  8. PhillipMarlowe

    Tim Daly of TNTP is quoted as saying, “It’s very risky because they’re asking the public to support them in a strike that is about whether they should be evaluated on how much students are learning.”
    Tim Daly believes Michelle Rhee’s lie about taking 63 of her 70 students from the 13th percentile to the 90th percentile on the CTBS.

  9. CM '11

    Gary,

    Thanks for this. I am on strike in Chicago, and this has been very helpful to share with people who think teachers just don’t want to be evaluated.

    On another note, can you review the following Op-Ed and post your thoughts? Greg Palast writes for The Guardian: http://www.nationofchange.org/worst-teacher-chicago-1347463241

  10. Chicago Educator

    I too am in Chicago. To all of you critical of VAMs and maintain that teachers want a fair evaluation (Gary, you too) – I would love to hear the alternative you think is best.

    I too think there are many errors with VAMs, which is why I don’t believe they should be a single measure used to evaluate teachers. However, there are also many issues with observation-based evaluations (bias, training, etc.); survey-based evaluations; and every other type of evaluation system we have. As a teacher, I would hate to have my entire evaluation rest on one of these systems, which is why I support using a mix of these, including VAMs. I also recognize that in the private sector (which I worked in before teaching) there is also no perfected system of evaluation (most of them are based on one supervisor’s biased opinions) – and yet – everyone still gets evaluated and their raises and jobs depend on this. I’m curious why in education we seem to think we need a perfect evaluation system before at least attempting to evaluate, provide feedback in order to improve our practice and our schools.

    • Rachel

      My biggest problem with VAM is the fact that my success is based on another teacher’s lack of success. Also note that I’m talking about me and my success, and not about my students and their success.

      If I had a class of perfectly average students, it would normally not be a problem if my students made a year of progress. However, in a VAM system, my students either need to make more than a year of progress (difficult, seeing as they’re perfectly average and, of course, the test is precisely correlated to what perfectly average students should be able to do) or I need to cross my fingers and hope that the teachers of comparable students, quite frankly, suck at their jobs.

      My rating as a teacher should NOT be based on how well other teachers do. I should be inspired by great teachers, not worried that I’ll be fired because I’m just a good teacher.

  11. veteran

    Great article and good practical summation Parus!

  12. Derek Piper (@time4recess)

    I can tell you from experience that VAM works great if used formatively. After each test I simply find which students, based on last year’s test scores, exceed expectations (positve deviants) and figure out what they are doing diffrently. They often love sharing their ideas with others. It is not perfect, and clearly not a good measure in a summative sense, but way superior to merely looking at averages.

    • meghank

      Yes, but it wouldn’t work so well if you threatened to expel those whose value-added scores were low. That would be equivalent to firing teachers whose value-added scores are low.

  13. Thank you for this very comprehensive 6-part analysis. I just retired as a principal in Nashville, TN, and the 2011-2012 year was the first for applying test scores to teacher evals: 15% based on achievement, and 35% based on value added. EVERYONE holding a professional license had test scores in their annual eval this year. (That, in itself, is a curious application of student test results…) Since TN has been doing value added for more than 20 years, I’d be curious to know if you have studied Dr. Bill Sanders (an agricultural scientist) who created the statistical analysis and then applied it to students? TN is a right to work state and, therefore, the law was not challenged by teacher protests.

  14. Catherine

    I have read your concluding line, Mr. Rubenstein, several times and have concluded that someone made a typo. Don’t you mean that value-added = value addled? I would like to point out one of the difficulties of true “value added” that standardized testing must completely take over our educational system. Technically students should be assessed in every subject (including art and PE) the beginning of the year in every grade level and then tested again in the last week of school. Only students who remain the whole year with the teacher would count toward their VA score. Results would have to be reported within a week to two weeks of finishing the year so that teachers could be paid bonuses or fined based on the results. Growth could not be based on “normed” or average results for a particular group of students but would have to be based on how much students were expected to do against objective criteria. The cost and bureaucracy of such a system would soak up a large portion of the school district’s budget and would leave little for books. computers, art supplies, not to mention bonuses. In addition it would be a questionable pedagogic practice.

Post a comment

About this Blog

By a somewhat frustrated 1991 alum

Region
Houston
Grade
High School
Subject
Math

Subscribe to this blog (feed)


Subscribe via RSS

”subscribe

Reluctant Disciplinarian on Amazon

Beyond Survival On Amazon

RSS Feed

Subscribe