Jan 11 2012

The Answer To The $125,000 Question

by Gary Rubinstein

Two years ago I first heard about a new Charter School in New York City called The Equity Project, founded by a TFA alum named Zeke Vanderhoek.  There was an article in The New York Times about how they were going to pay their teachers $125,000 in return for more work and accountability.  Teachers could also earn bonuses of up to $25,000.  They were also featured on 60 minutes.  I have to admit that I considered applying.  That’s a lot of money.  Even veteran teachers in New York City with 30 years experience make just about $100,000.  With 8 years in New York City, I’m up to about $75,000.

According to their website they have based this paradigm on ‘research’ including the famous 1996 Sanders and Rivers value-added study which ‘demonstrates’ that having three great teachers in a row vs. three not-so-great ones can be a difference of 50 percentage points in student achievement.  Knowing what I do about education, after 20 years of teaching and analyzing it , I was skeptical that their 8 teacher million dollar ‘dream team’ would get the results they expected.

Here is their ad on page 5 from the most recent TFA alumni magazine.

When I heard about this experiment, I was very interested to see how it turned out.  Despite all the ‘research’ that proves that having three great teachers in a row can dramatically increase test scores, nobody has actually conducted the very simple experiment of giving a group of kids three ‘highly effective’ teachers in a row.  There have been studies where they take three thousand kids and after three years analyze the results of the thirty of those kids who happened to get three effective teachers.  There have been studies where a group of kids have had one effective teacher and then the gains were multiplied by three.  But, as far as I know, this is the first time that someone actually recruited a ‘dream team’ among thousands of applicants and had the same kids learn from these teachers.  If the kids performed as well as those studies predicted, it would definitely support the thesis and also give some leverage to corporate reformers who believe the problem with education is that there are not enough great teachers and too many awful ones.

This progress report was released a few months ago and I haven’t heard much media coverage.  So I downloaded the 2010-2011 progress report and was not really that surprised to learn that this school is performing slightly below average, even compared to its ‘peer group’ with similar demographics.

The New York City progress reports are based on three categories:  15% based on ‘environment’ which comes from teacher and parent surveys.  The school got a solid A in that category — no surprise that the teachers gave the school high ratings.  Then, 25% is based on student ‘performance’ — which is the achievement level of the students as compared mostly to their 40 school ‘peer group’ of schools with similar demographics.  On this category, they got an unimpressive C.  Student ‘progress’ — how much their students ‘progressed’ when compared to students around the city who had similar starting scores counts for 60% of the report.  On that one they got a B, which gave them a B overall, and that put them at the 69% percentile of all middle schools in the city.  Keep in mind that all the scores are relative to schools that have similar demographics, so they can’t make the excuse that they had low starting scores.

Here is how they got the C in performance.  Note that this is their performance relative to schools with similar demographics.  Their forty school peer group only had 49% proficient in English which was way better than TEP’s 31%.

You don’t have to understand a lot about statistics to know that those gray bars would fill the entire left side of these rectangles if the school had average performance for their peer group.  They have only 31% proficient in English and 49% proficient in math.  And this is after having ‘highly effective’ teachers for two years in a row, so far.

Here is how they got their B in ‘progress

Again, for a million dollars, one would expect these gray bars to be bigger.  Of these eight metrics, only one of them is even above 50%.

It turns out, I learned, that 60 minutes did follow up last year.  At that time the scores at TEP were pretty low.  According to the CBS website:

“Some people watching this might be thinking, ‘Hey, they’re paying teachers $125,000 a year. They’ve attracted the best and the brightest. These results don’t really add up,’” Couric pointed out.

“We don’t have a magic wand. We’re not gonna take kids who are scoring below grade level and bring them up in a year,” Vanderhoek said.

But he doesn’t mention that the progress reports are supposed to account for that.  Their performance is not measured against an absolute standard but against their ‘peers’ which include several KIPP schools.  Their progress is also something that is not supposed to punish low starting scores.  According to the CBS site, two of the eight ‘dream teamers’ were fired at the end of the first year!

Anyone who has read my blog in the past year knows that I don’t think very much about the New York City progress report calculations.  I’m sure that this school is quite a good school, despite their low test scores.  I’m sure the teachers are dynamic and work very very hard.  The lesson to be learned here is that this ‘dream team’ is not doing much better, and actually worse, on average, than the ‘average’ New York City teachers because, on average, teachers are good.

I wonder how corporate reformers will react to this story.  They are stuck, really.  Either they have to admit that the ‘three great teachers’ thing has been proved wrong, once and for all, and is not something to base policy on.  Or New York City corporate reformers will have to admit that their rating system is flawed, even though many ‘failing’ schools have been shut down on the confidence of the accuracy of those reports.

And as far as TEP goes, they are going to have to make a big decision very soon.  The only way they will get their test scores up to a level that will satisfy their funders is if they do what most of the ‘high performing’ charters do — kick out the kids that are lowering their test scores.  It is unfortunate that if they choose to be honest and do the best they can, knowing that what they are accomplishing is tough to measure, they are likely to lose their funding in 2013 when their charter expires.

 

 

 

 

Categories: Debunking · 24 Comments

24 Responses

1. Tom Hoffman

   It is simple really — surely you’ve read the articles about how nobody knows how to select high performing teachers in the hiring process. Didn’t Malcolm Gladwell write an article about how it is impossible or something?

   on January 11, 2012 at 4:33 am · Reply
2. Remainders: Quick promotion for former DOE official John White | GothamSchools linked to this post.

   on January 12, 2012 at 12:53 am · Reply
3. Tom Forbes

   I had an opportunity to observe The Equity Project as they were out neighbor in the reassignment center at George Washington. Their website (2 years ago) was advertising a multimillion fund raising campaign to get their own building or to take a big part of larger structure being built in Washington Heights. The teachers were just regular teachers, nothing special about this. Lot’s of stop watches and timing every activity and movements of the kids. The PE teacher was poor and when they had field days on the track, so much of went on seemed scripted. This guy is on a real ego trip as he manages this school. Watching the kids get dropped off and parents applying for the next year, they are definitely screening their students, I guess just not as well as their peers.

   on January 12, 2012 at 1:11 am · Reply
4. Leonie Haimson

   thanks for this, we wrote about the huge and undeserved hype around this charter school which got 4 articles in the NYT before it was even open, but has yet to do a follow up piece showing its mediocre performance (wonder why):
   http://goo.gl/lh5Pf

   on January 12, 2012 at 1:45 am · Reply
5. Arthur Getzel

   I will bet that they would not hire me–a teacher of 33 years who gets great results on those pesky tests. My tutoring caseload is so large that I often turn away children. They would not hired me because I am a tired old man in their eyes.

   on January 12, 2012 at 1:47 am · Reply
6. Susan Crawford, DIrector, The Right to Read Project

   With all that money at their disposal, TEP could instead turn itself into a showplace for providing appropriate interventions for struggling readers. Joe Nocera wrote (NYT Jan. 2, 2012) about just such a program being done in Central Falls, RI, no less. That’s the same town where the high school was threatened with closure last year.

   on January 12, 2012 at 2:45 am · Reply
7. andy

   Good article – thoughtful and non-cartoony analysis – useful provision of the relevant information. Keep it up!

   Perhaps they will fire the first class like Geoffrey Canada did? It does make sense that creating culture and solidifying a curriculum would take some time.

   Also appreciate your recognition of the likely hard work of the staff – prevented the article from seeming like a diss of them in service to some larger political struggle.

   on January 12, 2012 at 2:50 am · Reply
8. Michael Fiorillo

   The answer to the $125,000. question is “False.”

   on January 12, 2012 at 3:12 am · Reply
9. Miss G

   I worked at this school for a few months last year. The teachers I met were nOthing short of phenomenal. The main factors (I believe) in the schools failure to stand out in success are: the 33+class size, the absence of special education teachers where mandated, and the way that The teachers were worked to the pulp (most agreed that the job itself was not sustainable if he/she wanted to be able to maintain any life outside of school.) These scores don’t surprise me at all. Thanks for the post.

   on January 12, 2012 at 3:25 am · Reply
   • Tom Forbes

     If you had a chance to spend a few months in any school, you would see there are phenomenal teachers in all schools. The one’s at TEP are no better than the rest of us.

     on January 12, 2012 at 4:52 pm · Reply
     • Miss G

       Absolutely. I”ve worked at six schools in my career and have met phenomenal teachers at every one of them. By applauding the abilities of the teachers I worked with at TEP, I was not implying that they were better than you or any other teacher- just giving merit where I feel it is deserved. Because I was able to spend time working within the school itself, I had the chance to see other factors that were overlooked by administration or even poor decisions that were bound to have a negative impact on student progress. For example, When I left (or was largely pushed out) as the special education teacher in October, they went the rest of the year without hiring a replacement. That means 15 students pushed into a regular education classroom with no extra help. It was factors and decisions like this that creates holes in the “experiment.” If paying teachers more and choosing them carefully could have a benefit on students, it certainly isnt the number one factor. And it looks like it certainly WONT when so many other aspects of their education are sacrificed simultaneously.

       on January 12, 2012 at 10:38 pm · Reply
10. Norm

    This school was mentioned prominently in the movie American Teacher which pushed high salaries as the 125K solution Zoek was at the premiere at Education Nation with a big crew from the school as one of the teachers in the movie jumped ship from her Jersey school to this one. I agree about the rating systems being used to praise and damn schools is wrong. You make a very important point about most teachers being good – most poor ones don’t last too long or become administrators.

    on January 12, 2012 at 5:35 am · Reply
11. Yo Mista

    Gary, you make an excellent point here. I’m glad I discovered this piece through a friend on facebook.

    Screening students and rating schools is exactly the kind of crap that’s doing to bring this country down. If charters can screen kids and remain public, then obviously charters will outshine regular public schools – it’s not happening as a whole yet, but that’s what will happen eventually.

    I highly recommend (if you haven’t already) readers check out Diane Ravitch’s latest book.

    on January 12, 2012 at 2:03 pm · Reply
12. T. Fischer

    The fact remains (as I like to tell) you have a pool of students and a pool of Teachers from which to pick. It doesn’t matter whether it’s a Public or Charter school, results will be similar unless you manipulate those pools. And even when the Charters attempt to, they are unable to make enough difference to affect the results. How often have we seen this now? And those class sizes…..?

    on January 15, 2012 at 4:35 pm · Reply
13. Damion Frye

    As a former teacher at and one of the founding 8 members of TEP Charter, I certainly appreciated reading your work. That is to say what makes a democracy successful is the ability to create and express informed perspectives. Yet the information you chose to use as your primary data to support your perspective do not, in any way shape or form, take into account the mission, vision and purpose that not only drives everyone who works there but truly Zeke himself. I felt (and still do) honored to have been chosen to work there. Having worked at both traditional public and charter schools before and now as an administrator for a district in NYC, I can say, without question, that TEP Charter is a special place where some of the finest teachers have assembled. Like any institution, what may seem like a good fit for some ends up not working out that way. I was very close with the two teachers who were not asked back and their loss was hard on the staff. Thus, their departure did not mean that they were any less in their excellence.
    TEP does have more students per class than most schools as the primary way to fund the salaries. This poses a sincere and prudent challenge that I am certain (given the fact that I have now seen classrooms with 4 students and the teacher has no control/is horrific all over NYC) most teachers cannot and choose to not endure. Missing in your argument are the parent and student satisfaction surveys. With class sizes so high and the potential for the quality of education to be dramatically impacted, both parents and students rated their satisfaction with the school near 99% – much higher than the “competing schools.” This is due to Zeke’s dedication to ensuring that the school has a fully integrated social work program. Moreover, how many charter schools can say that 20% of their population are students with special needs (and not just occupational therapy or vision but severe reading, writing, computational and emotional needs). From experience I can directly say that some charter schools choose to not offer services students with special needs. At TEP, students with special needs are embraced and encouraged to apply. If you then take the data you used above and compare with similar schools using the same percent of special needs students, I am fairly certain you would find TEP’s to be equitable if not better.
    Almost all students who score lower than their grade level (even students without special needs) receive reading and math support from two – five days a week. Again, by design of the school and the passion of the teachers and administrators has provided more success with the lowest performing third of the population who all saw 1-2 point gains in their overall ELA and Math scores.
    I hope I have clarified some misconceptions about TEP. I would urge anyone responding, especially those who were there previously, to visit the campus. TEP Charter can serve as a national model, but just like any model it takes time to build it properly.

    on January 25, 2012 at 4:02 am · Reply
14. Damion Frye

    As a former teacher at and one of the founding 8 members of TEP Charter, I certainly appreciated reading your work. That is to say what makes a democracy successful is the ability to create and express informed perspectives. Yet the information you chose to use as your primary data to support your perspective do not, in any way shape or form, take into account the mission, vision and purpose that not only drives everyone who works there but truly Zeke himself. I felt (and still do) honored to have been chosen to work there. Having worked at both traditional public and charter schools before and now as an administrator for a district in NYC, I can say, without question, that TEP Charter is a special place where some of the finest teachers have assembled. Like any institution, what may seem like a good fit for some ends up not working out that way. I was very close with the two teachers who were not asked back and their loss was hard on the staff. Thus, their departure did not mean that they were any less in their excellence nor was the school remiss in finding equitable replacements.
    TEP does have more students per class than most schools as the primary way to fund the salaries. This poses a sincere and prudent challenge that I am certain (given the fact that I have now seen classrooms with 4 students and the teacher has no control/is horrific all over NYC) most teachers cannot and choose to not endure. Missing in your argument are the parent and student satisfaction surveys. With class sizes so high and the potential for the quality of education to be dramatically impacted, both parents and students rated their satisfaction with the school near 99% – much higher than the “competing schools.” This is due to Zeke’s dedication to ensuring that the school has a fully integrated social work program. Moreover, how many charter schools can say that 20% of their population are students with special needs (and not just occupational therapy or vision but severe reading, writing, computational and emotional needs). From experience I can directly say that some charter schools choose to not offer services students with special needs. At TEP, students with special needs are embraced and encouraged to apply. If you then take the data you used above and compare with similar schools using the same percent of special needs students, I am fairly certain you would find TEP’s to be equitable if not better.
    Almost all students who score lower than their grade level (even students without special needs) receive reading and math support from two – five days a week. Again, by design of the school and the passion of the teachers and administrators has provided more success with the lowest performing third of the population who all saw 1-2 point gains in their overall ELA and Math scores. In fact a 30 year veteran teacher devised the school’s reading support program, while also serving as the special education teacher and the dean of discipline. That is what makes this school so special – regardless of age and with a minimum of 3 years experience – any teacher who chooses to work at TEP, chooses to work really hard for their money and take an intimate look at their practice and the practice of others to ensure continued growth right along side the gains the students make.
    I hope I have clarified some misconceptions about TEP. I would urge anyone responding, especially those who were there previously, to visit the campus. Leaving the school was one of the most difficult decisions I had to make, but changes in my life circumstances prevented me from offering the dedication and time teaching at any school requires. Now from the outside, I can confidently assert that TEP Charter can serve as a national model, but just like any model it takes time to build it properly and with the care and precision that Zeke and the other staff members accept and uphold.

    on January 26, 2012 at 2:53 am · Reply
    • Gary Rubinstein

      Damion,

      Thanks for the comment. I have no doubt that TEP is a great school and that the teachers are working very hard there and that the kids are getting a lot out of the experience.

      The point of my post is that the TEP experiment proves that value-added is bogus. It is the ultimate ‘three great teachers’ in a row experiment. TEP has bad value-added, so either the teachers are not effective or value-added is meaningless. The people there should be speaking out against value-added and against judging teachers by test score gains — but they probably won’t since it will seem to others that they are making excuses.

      I like TEP. They do not seem to kick kids out, and I truly respect that. The only way to get test scores up dramatically in a short period of time is to kick kids out. TEP proves that too.

      I hope that Zeke takes the lessons learned from running this school — which is NOT a failure despite low test scores, and join me and the others who are revolting against the corporate reform test-based accountability movement.

      Also, let him know that if he really wants a feather in his cap, he can make me an offer. I’m not willing, though, to work much longer hours than I already do — that wouldn’t really be a raise if I did.

      Gary

      on January 26, 2012 at 4:00 am · Reply
      • T

        Dang – hiring you is a feather in his cap? Possibly – I don’t know you, so I can’t really comment either way – but my oh my, we are very full of ourselves!

        on January 26, 2012 at 4:21 am · Reply
15. Leonie Haimson

    Damion Frye says TEP has more special needs kids than most charters — but then the teacher above says that these same children lacked “special education teachers where mandated”. I wonder how well the school is serving them if this is true.

    One more thing: though TEP does seem to have very high class sizes, (though we don’t have good data on this) so do most of the middle schools in NYC and in D6 in particular. About 1/3 of the 6th graders in D6 and nearly half of 7th graders have class sizes of 30 or above, acc. to DOE data. Esp. considering their very high ELL population, these class sizes are unacceptable.

    on January 26, 2012 at 3:05 pm · Reply
16. E. Rat

    You know, I’m kind of tired of the idea that the $125,000 salary is some amazing generosity, an incredible wage. Sure, it’s a lot more than your average teacher earns. But given the median salaries of jobs that require as much education as teaching, it’s not so generous.

    Also, I always find it really hilarious that venture capital ed deformers, Mike Bloomberg, Bill Gates, and all the rest are trying to sell it as big, big money (while also condemning the teacher pension programs they find overly generous). If all it takes is an incredible teacher, shouldn’t the pay be more in line with what these guys make – or at least include Wall St.-style bonuses?

    on February 11, 2012 at 5:44 pm · Reply
17. T. Fischer

    Average salaries for professionals in the Math and Science field, with Masters are typically 110,000 to 120,000, same education as most High School Teachers , not to mention the fact that Education is basically a double-major. Guess that salary isn’t so exciting. Bring in the fact that many of the desired Teachers are younger, who might be having their own families, long hours are out of the question. Perhaps this brings in the necessity for Unions. You want great results, hire great Teachers, you want great Teachers, let Capitalism work, pay them what they are worth. Starting pay at 50 to 60 with a cap around 125. You won’t need to evaluate them, you’ll have a lot of the best to pick from. Until then, good luck, people are not attracted to education because of the status we put them at.

    on February 11, 2012 at 10:54 pm · Reply
18. Kevin

    Top is now score an A and in the top 10% of all middle schools in NYC:

    http://schools.nyc.gov/OA/SchoolReports/2011-12/Progress_Report_2012_EMS_M430.pdf

    on October 8, 2012 at 9:34 pm · Reply
    • Kevin

      That should be “TEP has now scored an A and is in the top 10% of all middle schools in NYC:” Typing…too…fast… :)

      on October 8, 2012 at 9:35 pm · Reply
    • Gary Rubinstein

      Just as the low test scores didn’t prove, at least to me, that it was a bad school, the high test scores don’t prove that it is now suddenly good. The fact that a school can jump around like this is part of the instability of the school grades. There are other schools that dropped from A to F. I think 25% of the failing schools with Ds or Fs were As in the past year.

      Of course there will be times that a ‘bad’ school gets a low grade and that a ‘good’ school gets a high grade. Maybe TEP getting an A is an example of a good school getting a high grade, but it still does not mean that the school grades are indicative of anything.

      As they say, even a broken clock is right twice a day.

      on October 8, 2012 at 11:56 pm · Reply