This is a preprint of an article whose final and definitive form has been published in the Journal of School Choice © 2012 Taylor & Francis Group, LLC; the Journal of School Choice is available online at: http://www.tandfonline.com/doi/abs/10.1080/15582159.2012.650106
Class Warfare: Inside the Fight to Fix America’s Schools, by Steven Brill. (Simon and Schuster, 2011; 496 pp.;
978-1-4516-1199-1, $28.00 hardback.)
In Defense Of The Anti-Reformers
In ‘Class Warfare’ lawyer Steven Brill demonstrates his litigation skills as he lays out his case against teachers’ unions and the so-called anti-reformers. His argument is clear, concise, and compelling. As prosecutor, he calls mainly on the witnesses that will strengthen his case, skillfully cross examining them and shrewdly striking from the record almost anything that might introduce a reasonable doubt.
Brill’s argument can be summarized in four main points, which I’ll first enumerate and then challenge one by one.
1) Some charter schools prove that teachers can overcome poverty.
2) Most teachers in non-charter schools are not overcoming poverty because of many clauses in their contract which put the needs of the adults above the needs of the children.
3) Teachers are also, because of union pressure in getting state laws passed, not accountable for their performance. Performance, as defined by standardized test score gains, is something that can now be easily and accurately measured. Unlike other professions, teachers are not compensated for their performance. Instead they are subjected to “lockstep seniority-based compensation.”
4) Entire cities like New York City, Washington D.C., and New Orleans that are encouraging charters, limiting union perks, and evaluating teachers on performance are yielding results.
These four points are so compelling to education reformers, Brill describes, that Race To The Top was designed as a way to encourage states to change their laws to remove charter caps and to tie teacher evaluations to standardized test scores.
Challenging Brill’s argument:
1) Miracle Schools
Several times throughout the book, Brill makes the claim that charters “proved that intense, effective teaching could overcome poverty.” (page 303) Brill utilizes the standard reformer trick of touting some ‘miracle schools’ which are serving the ‘same kids’ sometimes in the same building with stunning results.
Two common critiques of high-achieving charter schools are that they get their results though “cherry picking” and attrition. Brill relegates the “cherry picking” argument to a footnote at the bottom of page 16. “None of the actual data supports” the claims that charters “ ‘skim’ the most motivated students from the community or that they fail to represent as many of the poorest kids, kids with learning disabilities, or English-language learners.” So why in the pro-reform August 2011 issue of The Educational Gadfly was there an article called “Public School Choice in the District of Columbia: A Descriptive Analysis” with the summary “The skinny? Yes, charters are cream-skimming.” The author admits that the incoming students were ‘better’ than those who didn’t apply, but then went on to argue that it is good to give those better students the opportunity of the better learning environment.
Attempting to present research contrary to this, on page 303 Brill cites a study in which students who applied for a charter lottery and won were compared to students who applied for the lottery and lost. He said that the ones who won did better than the ones who lost, which seemed to prove “Same demographics, same motivation, different results.” But this is just what I’d expect. The more important comparison is to compare students who entered lottery and lost to students who never entered the lottery. All this study proves is that if you isolate the most motivated kids, they will perform better than they would if they were mixed in with the others. Brill never brings up the attrition concern at all.
Exhibits A and B are ‘miracle schools’ Harlem Success Academy 1 and Mapleton Expeditionary School of the Arts.
Introduced on page 10 and then weaved throughout the book is Harlem Success Academy 1 charter school, which shares a building with the lower performing P.S. 149. The fact that Harlem Success 1 has very good test scores despite having “similarly qualified, or challenged, students” (page 16) as P.S. 149 proves what is possible with hard working teachers with high expectations. But Harlem Success has 64% free lunch versus 78% for P.S. 149, 15% special ed versus 24%, and 6% English language learners versus 10%. One other detail missing is a curious amount of attrition for the third and fourth grade cohorts. Harlem Success’s 83 kindergartners and 73 first graders in 2006 had dwindled to just 63 third graders and 59 fourth graders in 2009. This is a stunning 22% decrease. Meanwhile in P.S. 149 they went from 39 kindergartners in 2006 up to 44 third graders in 2009 and from 45 kindergartners in 2005 to 44 fourth graders in 2009.
Mapleton Expeditionary School of the Arts was started by one of the heroes of the book, State Senator Michael Johnston of Colorado. A large ‘failing’ school was split up and Johnston was given a chance to take over a portion of the school with a staff that he could build from scratch. On page 139 in a pep talk to an unspecified number of 9th and 10th graders Johnston says “The state’s statistics say that half of you probably won’t graduate” and that he intends to prove them wrong. One year later, on page 196, he tells his 44 eleventh graders that he will help them all graduate and get into college. A year and eight months later, on page 198, Obama comes to see the graduation and congratulate the 44 graduates for getting into college. It’s a great story and demonstrates what can be done with a great leader who has high expectations and who has the power to choose his staff.
But the success is exaggerated in a very relevant way. When I looked up the records on the Colorado Education website I learned that the group of 44 graduates who got into college did not begin as 44 tenth graders two years earlier. There were at least 73 tenth graders in that original group as that was the number of eleventh graders one year later. So, ironically, about half of them did graduate, just as the state expected. Another irony omitted by Brill about Mapleton is that Johnston will soon become a State Senator who gets a law passed tying teacher evaluations to test scores while at his own school the scores were terrible with, for example, 11 percent proficiency in math in Johnston’s final year as principal there.
I don’t reveal these statistics as a way of saying that Johnston wasn’t a good principal or that his teachers were not good. I do this to make a point that in this current ed reform debate, we need to have a realistic definition of what qualifies as success.
2) Union Contracts
The argument that there are too many bad teachers needs to be backed up by a statistic of approximately how many such teachers there are. If the percent of bad teachers is very low, then firing them all would not change things much. Brill uses P.S. 149 (page 45) as an example of a school that has between 25% if not 33% of teachers like this. Based on my informal experience at several schools, I’d put the number under 1%.
For the same reason there are not many terrible lion tamers, there are not that many terrible teachers. Students make life miserable for a bad teacher, so there are lot less stressful ways to make a living than to be an ineffective teacher.
Brill describes several union benefits from the New York City contract with extreme contempt. Principals, Brill says, are handcuffed by the union contract. If they try to observe teachers they get accused of harassment, and they have no say about the format of a teacher’s lesson plans. He quotes a teacher as saying she can write her lesson on toilet paper if she wants to. Of course nobody actually writes their lessons on toilet paper. Even lazy teachers like to write the plans on something that can be saved and used again in future years.
Here is the actual two sentence clause from the contract: “The organization, format, notation and other physical aspects of the lesson plan are appropriately within the discretion of each teacher. A principal or supervisor may suggest, but not require, a particular format or organization, except as part of a program to improve deficiencies of teachers who receive U-ratings or formal warnings.”
Notice that this isn’t about the substance of the lesson plan, but just the ‘physical aspects’ of it, like what color paper it is on and what kind of ink it is written in. Also Brill does not mention that the second sentence allows a principal to require a certain physical format when the teacher is on an improvement plan.
Many reformers lament that we have a system that is good for the adults when it should be good for the children. This is one of the most misleading sound bites in the entire education reform movement. Brill uses an example on page 410 which brings this issue to its absurd extreme. As a comparison he gives an example of something from a flier for the non-unionized charter Harlem Success Academy I. “In a traditional school, if it’s forty degrees or colder no one goes outside into the schoolyard, because it’s too cold for the adults. Not us.” Does that mean they they will go out when it is 20 degrees out? At what temperature is it too cold for both the adults and the kids? And how does it benefit the kids if they and their teachers get sick?
When Klein speaks about Randi Weingarten on page 101, he reveals one of the biggest disconnects between the two sides about what motivates the union. Here are three crucial excerpts from that page: “To Klein, Weingarten was so smart that she had to know that playing the game of simple advocate was hurting the children she professed to care so much about.”, “Weingarten compounded Klein’s frustration by never admitting who her real — and only — clients were.”, “But Weingarten’s line was always that what was good for teachers was always good for children.” So Klein, and apparently Brill, believe that Weingarten lies when she says that she believes that the reforms she rejects will be bad for both teachers and kids. This is a pretty serious charge, and one that Brill will reinforce when he has his big ‘revelation’ at the end of the book.
In a recent radio interview, Brill summarized his views as follows: “The nation’s K to 12 schools are basically the only workplace in America where, until recently with some reforms including reforms in your state [Colorado], until those reforms, they have been the only workplace in America, even though there are 3.2 million public school teachers, where performance basically doesn’t count. The effectiveness of teachers is not measured. It’s not measured through a good peer review process, through any kind of a review process, through progress of the students in terms of how well they do on tests in the beginning of the school year versus the end of the school year. And because that performance isn’t measured, in fact it’s ignored, the most important factor in the schools in terms of advancing children which is the effectiveness of teachers has basically gone untouched and that’s the problem that the so called education reformers are trying to fix.”
Brill’s claim that teachers are not evaluated by ‘performance’ is untrue. Principals observe teachers formally and informally and a good principal can tell very quickly if a teacher is effective or not. There is a certain ‘vibe’ in a productive classroom which cannot be faked.
What Brill means is that, in many places, standardized test score ‘gains’ are not a factor in teacher evaluations. The measurement of this is known as ‘value-added’ which is when a computer attempts to compare how a teacher’s students do on a standardized test to how those same students would have performed on the same test had they lived in a parallel universe where they had an ‘average’ teacher.
Brill cites one of the gurus of value-added, professor Thomas Kane of Harvard University. Kane, he says, found that when value-added measurements are applied to schools they are very volatile. They jump from year to year. Kane expected, then, that when applied to the smaller sample size of a teacher, it would be even more volatile. But, Brill claims, Kane found the opposite. “Teachers who did well one year typically did well every year, and those who did poorly kept doing poorly. It seemed that the teachers were the key variable.” (page 150)
Brill either misunderstands or just misrepresents Kane’s findings. If teacher scores are stable from year to year and the teachers make up the school then the school’s ratings would have to, mathematically, also be stable from year to year — which they, by Kane’s admission, are not. It would be like saying that the teacher’s weights are stable from year to year but the average weight of the staff fluctuates wildly from year to year.
Kane doesn’t give many details about his model, but if we look at Mathematica Policy Research, which creates the value-added for Washington, D.C., they write “for teachers with the lowest possible IMPACT score in math—the bottom 3.6 percent of DCPS teachers—one can say with at least 99.9 percent confidence that these teachers were below average in 2010.” So people at the very bottom might stay, in general, in the bottom half. That is not a very stable confidence interval.
Mathematica also prepared a report for the DOE where they said that error rates in value-added for one year of data would be up to 33%, which means that 33% of the time an effective teacher can be rated ineffective and 33% of the time an ineffective one could be rated effective. This is why when you compare value-added measures to principal evaluations, they have almost no correlation. I learned this while studying the raw data of a report cited by Kane which erroneously claimed to prove otherwise. The teacher with the lowest principal rating in the study actually got a higher value-added score than the teacher with the highest principal rating. Value-added is the alchemy of ed reform.
Kane wrote in one of his papers that these value-added measures should not count for the entire evaluation so there should be ‘multiple measures.’ But he thinks that value-added is accurate enough to count between 33% and 66% of the teacher ratings. In D.C., it currently counts for 50% for teachers who can be measured in this way. In a recent interview, Michelle Rhee described a conversation she had after speaking to a Wall Street Journal conference of CEOs. One of them asked what her most revolutionary thing she had done as D.C. Chancellor. She responded, “now 50% of the evaluation is going to be based on how much students learn.” The CEO then said “Isn’t that the teacher’s job? I mean, shouldn’t it be 90% of their evaluation?” Rhee could have explained to the CEO that the metrics aren’t accurate enough to be given that much weight. Instead she said, “Listen, bucko, right now we’re at zero percent. Going from zero percent to 50% is huge.”
One irony of these value-added statistics is that they are so meaningless and practically random, that the use of them is likely to save many ineffective teachers who happen to get high scores on their value-added.
Brill celebrates the 2010 L.A. Times publication of value-added scores for thousands of named teachers as “what would rank in any era as one of its most important pieces of enterprise reporting.” Brill does admit on page 370 “Making performance scores pubic might embarrass some teachers or even unfairly stigmatize them because some of the ratings may be inaccurate. But those negatives are outweighed by the benefit to the children and their parents of getting a read on how their teachers are performing and holding teachers accountable.” The L.A. Times, in a recent editorial published on November 20, 2011, changed their tune on value-added. They wrote “Test scores probably have some value in the rating of teachers, but that has yet to be proved.”
Brill describes with venom throughout the book teachers’ “lockstep seniority compensation” and implies that teaching is the only workplace where people are not compensated on performance. But in most jobs people get annual raises so those who are more senior have higher salaries than the rookies. Even the world of law with which Brill is very familiar, it is quite unusual that someone out of law school becomes a partner in a law firm. There is a process of ‘earning your stripes’ and getting raises along the way.
As a teacher I like knowing that I can look at a chart and plan out my future based on what I know I will be making down the road. I would not want to endure the ups and downs of my test scores, making $165,000 one year, as he claims is quite feasible, and then $40,000 the next as my value-added unexpectedly drops. There’s a reason I became a teacher instead of a professional gambler. Even when I was a young go-getter twenty years ago, winning ‘Teacher Of The Year’ in my fourth year of teaching, I never felt cheated by not getting a monetary bonus for this. Good teachers get rewarded in other ways. We get asked which electives we’d like to create and teach. We are given the working overhead projectors. And if we want more money, we can leverage our teaching record to land a good administrative job.
4) Miracle Districts
New Orleans, New York, and Washington D.C. are held up as examples of districts that are reaping results from these types of reform. In Washington D.C., however, Rhee’s reforms did not close the achievement gap at all. On page 138 Brill writes “New Orleans had arguably become one of the nation’s best urban school systems.” Meanwhile the Recovery School District, which is predominantly charter schools, ranks 69th out of 70 districts in New Orleans. They boast that RSD is the ‘most improved’ district because its ‘District Performance Score’ (DPS) has gone from 58 to 67 which is, mathematically, a 14% increase. DPS is a number that can be as high as about 120 and the average district scores about a 95. This percent increase statistic is meaningless. Another district that went from 110 to 118, which was only a 7% increase, despite being the more difficult improvement. Percent increase inflates scores of districts with very low starting scores.
Brill generally presents the reformers as infallible gods — particularly Joel Klein, Michelle Rhee, and Arne Duncan. About Rhee’s qualification to be D.C. Chancellor, “she certainly knew everything there was to know about what was needed to make schools work.” (pages 173-174) About her getting ousted from D.C., he says that her failure was not realizing she should campaign. One part that is conspicuously short is the section about Duncan’s accomplishments before being named Secretary of Education. There was one paragraph on page 237 and a vague claim a page later “when the Bush administration had distributed a few competitive grants, [Duncan] had achieved many years worth of reform in a few months because his team had been so eager to win.” But Chicago schools are in terrible shape with the achievement gap actually growing during and after Duncan’s tenure there.
There are, however, a few spots where Brill does criticize reformers. Bloomberg gave way too good of a contract to Randi Weingarten because he was trying to get elected to a third term. Another place where Brill is critical is his description of the Race To The Top judging process which he describes as a farce with states winning by, essentially, cheating on their application while other states lost unfairly.
The so-called anti-reformers are mocked and ridiculed throughout the book. Brill’s three page rant about Diane Ravitch reveals that he didn’t really understand the point of ‘The Death and Life of The Great American School System.’ Yes, she defines what she is against for much of the book and only has a little about what she is for. She argues against all the ‘quick-fix’ solutions doomed to fail and then he is upset that she does not offer her own ‘quick-fix.’
At the end of the book, Brill is credited with having an ‘epiphany’ about how the supposed successes cannot be scaled without the unions. His revelation is that eliminating the unions would cause more problems than it solves so the ideal solution is for the unions to get behind the reforms he supports. He even suggests that Bloomberg appoint Weingarten as chancellor of New York City schools. When questioned about this, Brill gets quite defensive because he knows that this epiphany has been misinterpreted. It was just an example of Brill accidentally doing something that is uncharacteristic for him — he was too subtle. As he describes what Weingarten would do as chancellor, Brill shows that he truly believes, as Klein suggested earlier in the book (page 101), that Weingarten ‘knows’ that her advocacy of teachers actually comes at the expense of the children. If she suddenly was responsible for the achievement of the students she would admit what he already knows — that she has been lying all these years when she has said that what is good for teachers is good for students too. When I say similar things about how what is good for teachers is good for students, I know that I am saying what I truly believe, and I have no reason to think that Weingarten is not genuine when she says the same.
After this revelation, Brill concludes the book with his five recommendations, which include ending LIFO, merit pay for teachers with high value-added scores, saving money by replacing teacher pensions with 401(K)s, and making teaching a temporary job rather than a life-long career “In a world where career changes are the norm … that may not mean that they stay for twenty or thirty years, but it should mean they are there for at least five or ten.” (page 427)
I feel certain that the reformers will lose this battle but in doing so might have actually caused something good to happen. With propaganda like ‘Waiting For Superman’ and ‘Class Warfare’ they have awakened the proverbial sleeping giant. Anti-reformers are now operating with a new sense of urgency as they band together to fight these well meaning, but hopelessly ignorant, interlopers. Giving a sense of urgency to the anti-reformers who can pursue authentic research-based reform might be the silver lining here.
Fifty years from now it will be interesting to reread this book with the hindsight of what happened. I believe that soon the ed reform ‘bubble’ will burst. All the invented gains will be exposed and those who participated in the cover up will be banished to obscurity. We all will finally realize that early childhood education is a much worthier investment of money than test prep and accountability. We will give up on ‘value-added’ measures when we realize that they will be too costly to ever be accurate enough to tell us anything better than principal observations already do. We will look back at the remedies proposed by the reformers the way we now look back at the use of leeches to cure diseases.
And eventually people mocked in this book like Diane Ravitch and Linda Darling-Hammond will not be called, inaccurately, ‘anti-reformers’ but what they really are — scholars.
Spreadsheet from The Colorado Department Of Education with enrollment data for Mapleton Expeditionary School of the Arts showing 73 Juniors in 2006.
11% proficiency in math for Mapleton Expeditionary School of The Arts in 2007-2008
New York City Database with demographic data for Harlem Success Academy and P.S. 149
Harlem Success Academy I enrollment data to show attrition rate
P.S. 149 enrollment data to show attrition rate
Steven Brill on David Sirota radio program on 9/7/11
Mathematica Policy Research Report “Error Rates in Measuring Teacher and School Performance Based on Student Test Score Gains”
Mathematica Policy Research Report “Design of Value-Added Models for IMPACT and TEAM in DC Public Schools, 2010-2011 School Year”
Kane, Staiger, Gordon ‘Identifying Effective Teachers Using Performance On The Job‘ 2006
Jacob, Lefgran ‘Principals as Agents: Subjective Performance Measurement in Education’ 2005
Interview with Michelle Rhee on 4/20/11 at The Commonwealth Club of California
The Education Gadfly Volume 11, Number 30. August 4, 2011.
The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education
Diane Ravitch 2010