Aug 09 2013

Driven by data … right off a cliff

Yesterday the big announcement in New York was the results of the latest ‘common core aligned’ state tests.  As expected, the scores plummeted.  The New York Times reported that from last year to this one, the percent of students scoring ‘proficient’ on English dropped from 47% to 26% while the percent scoring ‘proficient’ on Math dropped from 60% to 30%.

Various ‘reformers’ weighed in and tried to put a positive spin on it.  Joel Klein, in an Op-Ed in the The New York Post wrote “While some may confuse lower scores as a negative development, the fact that we’re finally being honest about academic achievement is a very positive sign.”  Arne Duncan said “Too many school systems lied to children, families and communities”

The ‘lies’ they are talking about are the ones that say that our education system is doing an OK job.  It is very important to these ‘reformers’ that our schools are ‘failing’ so they can justify their radical approach to reform which is centered upon shutting down public schools, opening, charters, and firing teachers based significantly on standardized test scores.

So I looked carefully at the data and found that these test scores do, in fact, prove there was some lying going on.  But it is not the lie that that Klein and Duncan were talking about.  The lie that these test scores reveal is the one about charter schools being better than public schools.

I was tipped off to this story by an excellent article by AP writer Stephanie Simon, who seems to be one of the only reporters in the country on the trail of the fraudulent reformers.  In an article called “New York fails Common Core tests”, Simon points out that, in general, charters did not do very well on these new tests.

Just 23 percent of charter students scored proficient in language arts, compared with 31 percent in public schools overall. That’s a greater gap than had shown up in last year’s exams.

In math, charter schools beat the public school average in each of the past two years — but not this year. On the new tests, just 31 percent of charter students scored proficient, the same as in public schools overall.

The Democracy Prep chain posted uneven results, with particularly poor scores in sixth grade. In its Harlem charter, fewer than 4 percent of sixth graders passed the language arts exam, and fewer than 12 percent passed math. Its best results came at the eighth grade level, but even then the pass rate on both tests was under 33 percent — better than the citywide average of about 26 percent, but not a quantum leap above other public schools.

Democracy Prep officials didn’t respond to a request for comment.

The highly touted KIPP network also stumbled, with proficiency rates well below the city average for several grades and subjects. At KIPP Star College Prep, just 11 percent of fifth graders were proficient in math and just 16 percent passed the reading test. Seventh grade was another weak point, with 11 percent proficient in language arts and 14 percent in math. KIPP also did not respond to a request for comment.

She also wrote that another charter chain, Success Academies, did quite well on the new tests.  So we have mixed results with at least one charter doing well and at least two doing poorly.

So I thought I’d download all the data and take a look for myself — and yes, there WILL be scatter plots!

There are two ways to make test scores go down:  make the test harder or make the passing score higher.  As a teacher when I give a test, my hope is that all my students will pass that test and, as I’m at a specialized high school, I hope that everyone breaks 90, which actually rarely happens.  If only 30% of my students passed a test, well, something went wrong.  Maybe the test was too hard.  Maybe it was too long.  Maybe there was a ‘bad’ question which caused kids to waste a lot of time going down the wrong path.  I’d likely curve the test and, if I determined that the issue was how I taught the topic, I’d do some reteaching.

The test scores in New York went down because the test was much more difficult, testing, in theory, ‘higher order thinking skills’ though, in practice, the questions are really quite confusing.  I actually got one of the third grade questions wrong.  The older ‘skills based’ tests were problematic too since students could do very well on them without really understanding math.  But these new tests have the opposite problem:  Students can do very poorly on them even if they do understand math.  This is why I don’t like to base 20% of my teacher rating on a single test that I didn’t write.

When the scores came back, John King could have put some kind of ‘curve’ on the test, as they did in Florida recently, but decided not to.  So the scores are very low with about 2/3 of students failing.

Now this does not mean, necessarily, that teachers are going to be fired and schools shut down over this.  The New York City ‘progress reports’ compare schools to each other so a harder test, if it still has a mix of some easy and some medium level questions, could still be at least as accurate as the old tests (if you think the old tests were accurate).

So the first thing I did was make a scatter plot comparing all the schools that tested 7th grade math in 2012 and 2013.  On the horizontal axis I put the 2012 score and on the vertical axis I put the 2013 score.  If the test was so hard as to become almost random, there would be little correlation, but as you can see there is a somewhat of a correlation between the two years.

But here’s where it gets interesting.  To see if most charter schools were like KIPP Star and Democracy Prep, scoring well below the 22% city average, or if most were still doing relatively well, like the Success Academies, I made another scatter plot, but on this one I marked all the charter schools (or at least the ones that had the word ‘charter’ in them) with a red circle.

As can be clearly seen, the charters are, in general, the ‘outliers’ meaning the schools that had the biggest drops relative to other schools with similar 2012 scores.

In the Stephanie Simon report she mentions that KIPP Star and Democracy Prep hadn’t done so well with their proficiency rate, but she doesn’t mention how far they had dropped.  Out of over 500 schools, which includes about 35 charter schools, of the one hundred largest drops, 22 were charter schools.

The most stunning example is the famed Harlem Village Academy which had 100% passing in 2012, but only 21% passing in 2013 for a 79% drop (you can see that sad dot all the way at the right of the scatter plot).  Democracy Prep Harlem Charter, run and staffed by many TFAers, dropped 84% in 2012 to 13% in 2013.  KIPP Amp dropped from 79% in 2012 to just 9% in 2013.  The Equity Project (TEP) which pays $125,000 for the best teachers had finally gotten some test scores they can brag about with 76% in 2012, but that has now sunk to just 20% in 2013.  The Bronx Charter School Of Excellence, which recently received money from a $4.5 million grant to help public schools emulate what they do, dropped from 96% in 2012 to 33% in 2013.  So these are the schools that are the red ‘outliers’ hovering near the bottom right of the scatter plot.  In general, the average charter school went down by 51 percentage points compared to 34 percentage points for the average public school.  The most plausible explanation for charters dropping so much more than public schools is that their test prep methods were not sufficient for the more difficult tests.  In other words “you’re busted.”

I just don’t see how the ‘reformers’ can reconcile these statistics with their statement that these lower scores are a good thing since we are now being honest about where we stand.  The low scores in general do not decisively prove anything.  The cutoff scores for passing were an arbitrary choice by some politicians in Albany.  But the evidence that charters are certainly not working the miracles they claim is very clear from this data.

Now, remember that I don’t think that test scores capture all the good in a school.  Perhaps these charter school have a lot of those intangibles that help kids eat grits or whatever.  I don’t know.  But I do know that if the ‘reformers’ really value their ‘data’ so much, they should really think about how to interpret the charter grade crash.  To me, this suggests that maybe the hundreds of millions of dollars given to charters, both from the government and from private benefactors could be spent elsewhere in education more effectively.

93 Responses

  1. I’ve never believed that charter schools are “all that,” but I’m not sure the test data is now showing that they’re a waste either. If anything, it begs a really interesting question, which is why the scores dropped, assuming that these tests are any kind of valid indication of anything (which I don’t!). Were the charter schools only teaching a certain kind of test prep that didn’t translate to the test? Did the charter schools spend too much time focused on preparing students for the previous tests, and students didn’t adjust to the new testing styles as easily as they expected? Did their math or English language skills really drop or is this yet another indication of the reality that the scores are completely arbitrary and tell us nothing whatsoever?

    It does seem like they’ve opened a can of worms for themselves, because it’s hard to formulate any kind of cohesive argument for how charters can be better even though their scores are worse.

    That said, I’m hoping you’ll talk a bit about the effect of these scores on the students and schools in future blog posts. My district is in the process of adopting Common Core and we’re scheduled to begin the PARCC tests in 2014-15. They’re already telling us to expect the same drop in scores, etc. What I hadn’t realized is that some states might mandate “academic interventions” or some sort of remedial work for the kids who didn’t pass despite the continued acknowledgement that the kids aren’t really remedial and don’t need interventions. I’d assumed it would be a matter of tweaking the curriculum and trying to bring scores up gradually (and probably hours more of test prep, unfortunately), but that it wouldn’t perhaps result in say, more than half our middle-school students having to do another hour of math as intervention instead of having time in their schedule for an elective. (Our struggling students do currently receive that kind of intervention, though there’s some question about whether it’s having the intended impact or whether they’re simply resenting having to do two hours of math and miss out on electives that better engage them during the school day.) It’s sounding like New York laws will require the interventions, and I can’t imagine how that will work when approximately 2/3rds of the students will require interventions. What does that mean for them and their morale? What does that mean for the students who are truly struggling and need serious support, but now are lumped in with the average kids?

  2. Thank you for this insightful information.
    Would you discuss why you believe the Success Academies continue to score well. Do they have a magic education bullet? Are their lowest functioning students counseled out of their schools and into public schools? Do students spend an inordinate amount of hours on test prep? Most importantly, are students who “PASS” these tests destined for career success WHILE the 1′s and 2′s destined for a life of failure? Ok, the last question is really just a reflection of my underlying resentment of the assessments.

    • Gary Rubinstein

      I am going to look into Success Academies more. I think they just have 3rd and 4th grade right now so they weren’t in my 7th grade analysis.

      • Considering so many of the other charter school scores fell so dramatically, the Success Academy scores do raise some red flags. That Bronx Success Academy school with the 97% proficiency rate on the math – that is an awe-inspiring score. It certainly would be interesting to see how the Success franchise got such proficient overall scores when almost every other school fell. But it really would be informative to see how Bronx Success got that 97%.

        • I am VERY suspicious as to the Success Academy results. I know they counsel and push out lots of struggling students, but these scores are too high even with this. Someone told me that you can see evidence of cheating through distribution patterns; is this true? In any event, charter schools unlike public schools are allowed to score their own exams; many participate in a consortium established by the Charter Center where they score other school’s exams blind, but not all do. I would very much like to know if the Success Academy staff score their own exams. Gary, perhaps you can ask the Charter Center?

          • CWT

            Success does NOT score their own exams. No 3-8 exams are scored at the school level in fact. You are referring to Regents exams. In fact to knock down accusations like yours of cheating, Success hired independent auditors to be present while exams are being administered so that these types of jealous rumors can be knocked down.

          • Kemi

            I worked at Success Academy for two years. I hated it and I was miserable, as were most of my co-teachers, but the reason Success continues and will continue to do well on tests is that they have a progressive, research based curriculum in K-2. It’s not that the teaching is awe inspiring, it’s just that it’s consistently good. Though the staff skewed young, some of the teachers even had their own kids at the school. The school is heavy TFA — but the corps members have to work only as assistants their first year. The curriculum is semi-scripted. For example, they give us the basic teaching points for each unit, but it’s up to the teacher to decide how they want to deliver the lessons, and if their students need more of this or less of that. Success is also really on top of their RTI so its a bit harder for kids who get behind in pre-testing grades to go unnoticed. Also, they’ve been using common core for way longer than it was required.

            Combine that with hardcore test prep and you get good results. My co-workers and I hated working for Success, but everyone, including myself have taken much of what we learned at Success and taken it into our new settings. I did see a little bit of counseling out, but the creaming people talk about is way overblown.

            My larger concern is how bogus these test are. In my opinion, Success Academies deliver a high quality and relatively well-rounded education for low-income minority children. My concern is that even with a good education, kids need a ridiculous amount of test prep to pass the test. The test prep at Success runs like a machine. The kids get small group instruction all day during test prep–even the principals often stop their administrative duties to teach small groups. They even pull lower grade teachers to teach groups of test preps while assistants lead their classes. I hated that place, but I can’t deny their curriculum is excellent and they know how to do test prep right.

          • Linda

            “deliver a high quality and relatively well-rounded education for low-income minority”

            FOR low income minority…do you mean it would or should be different if the kids were from middle or high income homes?

            They know how to do test prep….not really a ringing endorsement and not a school I would want for my children. Sounds like its good enough for other people’s children.

          • Manuel

            Why is test prep an integral part of a school curriculum? It doesn’t teach anything other than test taking strategies. These kids are no different than trained monkeys. Is that what we want? Automatons?

            Test prep should be completely outlawed.

      • Ecarg

        Success Academies currently serve students up to 7th grade. This school year, there will be 8th graders

      • Kemi

        @Linda

        While I disagree with their philosophy of education, I do not mean that Success is only good enough for low-income minority children. They opened schools in affluent neighborhoods, and though they were met with (valid) protests, they still found plenty of takers. Because of the centralized nature of the Success Network, ALL the schools, from Harlem to Cobble Hill have the exact same curriculum and methods.

        And I don’t have a ringing endorsement of Success. I hated working there and probably wouldn’t send my child there. I just really hate it when people spread misinformation. It dumbs down the debate when the discussion is not based in reality. I grew up in a fundamentalist Christian family, and sometimes that is what these ed-reform/deform debates sound like. People refusing to believe anything positive about the other side. Like atheists not acknowledging the good things black churches have done for their community, and then christians responding by saying not believing in God means you want Hitler to go unpunished.

        • Linda

          I understand your point, but as an experienced teacher with dual certification, sped and reg. Ed I would never want my children or grandchildren to attend a school where the focus is test prep and the attitude towards those who struggle is to get rid of them ASAP (or at least after the per pupil expenditure is received). I never understood why the pro rated funds didn’t return with the student to the receiving school.

        • Ray

          Kemi, I appreciate your balanced approach. I would love it if you could give a more clear picture as to why the teachers are unhappy. Also, I would like more information about their approach to math.

    • Alison

      If I was being conspiratorial, I’d say these are exactly the kind of numbers $uccess needs to open schools downtown, or the Upper East Side….;)

      The Harlem branches have been using Common Core aligned materials since at least 2010 (Singapore math; Pearson’s enVision, I forget the literacy one).

      They didn’t pilot anything with DOE, but I wonder if they signed up to any national groups (eg, Silicon Valley Math group) and received advance materials to work with.

      Although I’m joking above, Success depends on the free space from the district (or, less charitably, so their head can make the salary she does); acing these tests must mean a lot to them institutionally.

      …perhaps, if DiNapoli ever gets his audit…

  3. Michael Fiorillo

    The so-called reformers won’t feel obliged to reconcile anything: they’ll just keep lying, and grabbing everything in sight.

    As always, thanks for your astute analysis, Gary.

  4. Gary these are passing rates not scores per Se? If so please clarify. Most real testing experts believe less in the reliability of passing or proficiency rates since they are easy to manipulate. It might be better to look at the scores themselves for this reason. Also an irony: David Coleman author of the Common Core is on the board of TEP charter.

    • Steve M

      Gary’s plots are distributions of normalized scores within NYC, not their raw scores. That is, how NYC schools compare to each other.

  5. skepticnotcynic

    Don’t worry,

    Like any new test adoption, the passing rates will rise as students get better and better each year as they become more comfortable with the test format. The teachers will also get better at understanding the test format and will improe their test prep. Iin 5-10 years the big publishers will cycle in a new and even better test that will be heralded by the policy makers and law makers as the “new standards” that will make us competitive with the rest of the world.

  6. Amy Hogan

    Nice article, Gary. Why few people are questioning the validity or the efficacy of these tests is troubling.

  7. Max Yurkofsky

    Gary can you share the excel docs you used? I think there is more work to be done in comparing 2012-2013 scores.

    • Gary Rubinstein

      The 2012 was here https://reportcards.nysed.gov/databasedownload.php
      and the 2013 was here http://www.p12.nysed.gov/irs/pressRelease/20130807/home.html
      They are .mdb files so they won’t open in Excel, but you can get a cheap mdb viewer and then copy and paste from it, which is what I did.

      • Max Yurkofsky

        Thanks! I’m going to have a fun weekend…

        • Emily Becker

          Go get em.

      • Harringronian

        Excellent (and serious) piece, Gary (as well as useful – and important – comment by “Kemi”),

        For the benefit of those of us who don’t have MS Access, I have found a VERY good FREE mdb reader/writer @: ,

        BTW, pardon my ignorance, but what exactly does ‘nmean’ mean?

        • Matthew

          NMEAN = mean scale score – out of 400. But until NYSD publishes the technical report in October or so it dos not tell you too much. You can compare it to the cut scores to see where a given grade of students averaged.

  8. Gabriel

    The scores, not the passing rate is what’s going to be key in the final analysis. Differences between passing rate don’t necessarily mean much, and the score spread analysis needs to be done to see if its a fair comparison. In theory the harder the test the greater the spread should be (since you reduce ceiling effects) And the more sensitive it is for discriminating between students (and possibly between teachers snd schools -if you believe the test are measuring something truly significant). The old versions of the test ceiling effects were significant and average differences that translated into 2-3 multiple choice questions could swing a school’s grade from an A to a D – moreover the sensitivity to differences was not consistent across the range of scores (and specifically scores on the extremes were more sensitive than those in the middle). This created wild swings in teacher and school grades, which are partly a statistical effect and partly a problem with the tests itself, and partly an effect of the aggregation and transformation methods used. With the new tests these kinds of problems are compounded and any comparison about passing rate (or differences between old and new rates) constructs dubious. While I too am skeptical of the wildly exaggerated charter school claims of transformational success, we need to be fair in our analysis – and while the results are probably rustling many feathers in the charter school world, we need to do a much more carful analysis.

    • Matthew

      well said Gabriel

  9. RunOn

    This is an excellent representation / analysis. Wish they would publish this on a larger scale because really…great job.

    I know that you’re a math guy but I would love to see a graph breakdown of the ELA scores.

  10. Gabriel

    Leonie,

    Frankly this is shocking, if true, that some charter a are allowed to grade their own tests. This was old practice with Regents, but not so for the state tests, but the practice was changed a few years ago. The potential for biased grading is too great.

  11. Lisa in nyc

    The DOE hired an ex-special agent to help monitor the cheating of tests. Was Henry “Hank” Greenberg, ‘a Special Investigator appointed by Commissioner King and gives reports to the Board of Regents with recommendations for improving the security and integrity of State assessments’ over seeing the charter state tests?? Here is the tip link:

    http://www.oms.nysed.gov/press/NewTestSecurityWebsite.CommissionerKingAnnounces.html

  12. skepticnotcynic

    I am tired of this debate. The proof is in the pudding. Google has determined there is no correlation between job performance and GPA and tests scores and brain teasers.

    http://www.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html?pagewanted=all&_r=0

    With that being said, we are spending way too much energy on flawed assessments and obsessing over data-driven instruction. Anyone who is focused on this type of instruction is probably not a very good teacher or maybe just really good at getting a kid to pass a test.

    “There are so many charlatans in the world of education. They teach for a couple of years, come up with a few clever slogans, build their websites, and hit the lecture circuit. In this fast-food-society, simple solutions to complex problems are embraced far too often. We can do better. I hope that people who read this book realize that true excellence takes sacrifice, mistakes, and enormous amounts of effort. After all, there are no shortcuts.” Rafe Esquith

  13. Alec

    Pit really does show that one trait of “successful” charters is that they focus, almost exclusively, on whatever is being tested. Take for example Hrvest Prep in Minnesota that got “beating the odds” status in Minnesota for high math and reading scores. Then all of sudden they started testing science and Bating the Odds Prep literally got zero percent passage for two of the next three years and six percent in another.

    Our dominant culture gets a brod based education in critical thinking and reasoning. Charters perpetuate Tthis opportunity gap.

  14. Ray

    Gary, I am curious about what Success Academy is doing. I think that you are the ideal person to look into this because you have a strong math education background and because you are not fanatically for or against charters. You clearly point out their problems. Chief among them a tendency to cream and to push out struggling students. While Success Academy schools are notorious for both of these practices, their math results are too good to be explained away by this alone. I checked their web site. It seems that their math program emphasizes critical thinking, group discussions, and written explanations of answers. I also noted that they have a strong elementary chess program. What is your opinion of these practices? I know that you have actually observed KIPP classrooms. It would be extremely interesting for you to observe a Success Academy class and give your take on what is happening.

  15. Icahn Charter schools did exceedingly well. They have a very different micro school model (just 35-40 students per grade). They have 5 schools….

    • Gabriel: the success of the Icahn schools has nothing to do with the size of the schools but instead class size: the cap all classes from K-8 to 18 students or less; a model that all public schools should emulate. Instead in NYC we have the largest class sizes in the state; and they are now the largest in 14 years in K-3.

  16. Margaret

    I loved your analysis, thank you for this. One thing that nobody seems to be talking about (though you did touch on it) is the fact that the conversion tables used to take the raw scores to scaled scores and finally to performance levels were dramatically different this year. In 8th grade math a raw score in 2012 of 37 (out of 68) was a level 3 while in 2013 you needed a minimum of 49 (out of 72) to be a level 3. That’s like making the test harder AND arbitrarily changing the passing score from 54 percent to a 68 percent. Seems pretty cruel to me.

    • Margaret,

      Check through this page, which goes through the process. In our data runs the first thing that popped out was how wide the range for a 4 was and how narrow the range for a 3 was in ELA; there’s discussion of this in the comments.

    • Forgot the link, of course. It’s to Dina Strasser’s blog.

  17. If you’d like to look at where charters ranked among schools in their home districts and citywide check out the interactive table at http://nycschoolsblog.com/stateresults/index.aspx

    It contains the scores for the past 7 years and has the schools ranked based on mean scale sore, percent level 4′s and percent proficient, and I think it can help turn up additional useful comparisons between schools (not to mention between spin and reality).

  18. Steve M

    Gary, you need to post other grades’ math distributions so that we can see how similar they are.

    AND, you need to post their ELA distributions.

    All of this needs to be done to show that you are not cherry picking (which you are not, but yo need to be thorough).

    • Gary Rubinstein

      Good point. I chose 7th grade since I thought it would be the most relevant. By 8th grade, many 5-8 charters have lost too many students while to make it 5th or 6th grade I actually thought it would be unfair to the 5-8 charters since they haven’t had much time to work with those students. I did do the 7th ELA and saw a similar patter with the charters getting the larger drops, though the math was more extreme. Each of these takes about an hour to make, but maybe I’ll chip away and make the other 11 and post.

      • Matthew

        but Gary a lot of schools (charter and traditional public) admit kids in 6th grade from many different feeders. So attrition aside, a 3rd-5th comparison and a 6th to 8th comparison shows how the school had an impact (to a degree). This is further complicated by the the fact that nearly all schools show a decline is scores from 6-8th grade. (At my oldest kid’s extraordinarily selective UES middle school 60+% of its 6th graders achieved a level 4 in math, and by 8th grade just over 40% hit that same mark). Also you can get the data in Excel, which makes the analysis go a lot faster.

  19. Julie Sellers

    Please add my contact information to Gary’s blog list.

    • Gary Rubinstein

      You have to manually click on one of the options on the right if you want to get notified when a new post happens, I think. Or you can follow me on Twitter! Gary

  20. Good piece, Gary. Let me offer a slightly different take. The “big lie” the test scores expose is not “the one about charter schools being better than public schools.” Instead, I think the scores expose something I’ve been talking about for years: Charter schools squeeze their gains out of systems and structures, while their curricula is not appreciably different than the neighborhood schools they purport to improve upon. You can focus on data, teacher quality, and add instructional time, but without a coherent curriculum that builds knowledge over time, you will end up with what I have long called “a second flavor of bad.”

    I’m strongly pro-charter, but I believe with equal conviction that charters–including mine–need to raise their game on curriculum, have a long-term K-12 vision, and focus on it relentlessly. I remain sanguine about CCSS, but hope it causes soul-searching about what teachers teach and what children learn, which I continue to believe (perhaps vainly) to be the proper focus of reform efforts.

  21. BTW, to Leonie Haimson’s point about the Carl Icahn school, I tend to believe the secret of their relative success is not class size (though I do not dismiss its importance) but their long-term adoption of the Core Knowledge curriculum and strong implementation under Jeff Litt–precisely the kind of curriculum which, well implemented, I would expect to lead to sustainable, long-term gains.

  22. August 9 | tigersteach linked to this post.
  23. Seenitall

    Wondering how success academies continuously score high even though it is statistically impossible? The well know obvious reason is that they do counsel out problematic and low performing students. I know because we get them at my school, usually in October after SA has received funding for them. But most disturbing is what actually occurs during testing. Our staff members (we are collocated with a SA school) have seen their teachers “coaching” students during the actual tests. Staff has observed SA Students continuously ask questions and teachers spending significant time leaning over the students and talking to them. Where I come from this is definitely against the rules and called CHEATING. There no longer seems to be a system of monitoring- as each school is expected to monitor themselves. Maybe someone should figure out how to get some official non-biased monitors in at testing time… Just saying…

  24. Jasper Steenhuis

    I’d love to see a more nuanced look at teh data driven by the socio-economic status of the communities they serve. You are comparing public schools in highly affluent communities with schools that serve kids in less affluent areas. Does the pattern still hold? Just askin.

  25. Gabriel

    CWT,

    Just to be clear, I did not make the accusation that Success Academy graded its own state tests, someone else did. I stated that if this were indeed the case it would be a potential bias problem. I think Success Academies are doing lots of things that are great and if these scores hold water we should ALL dissect what it is, learn from it and replicate it (assumming it’s not student selection and attrition related). I’m hardly jealous, and frankly rather shocked at the results and what they harbor for the charter schools, as despite many misgivings about some aspects of the charter movement, I give them full credit for the autonomy they have (which i think NYC public schools desperately need more of); i value their ability to experiment and innovate with new models of sch admin; and the risk taking and entrepreneurship that goes on. I’ve visited and studied enough charter schools to know that the best ones are operations dramatically different (and probably better run) than the top half of the urban public schools. And unlike other bloggers in this space I’m not sure the test based outcomes obsession is charters schools fault (although they can certainly play that game better than public schools because of their greater autonomy). I have deep respect for Eva’s work (although i agree with critics that her salary is excessive!) and the guts it took to create the Success Academy schools.

  26. I think it would be terrible to emulate Success Academy — whatever their test scores, as they repeatedly violate the rights of parents and children, through repeated suspensions of kids as young as kindergarten, and imposing abusive policies.They have very high teacher and principal attrition rates as well — and Eva herself has lied repeatedly and violated the law. I have no idea whether her schools cheat on the tests but I would not surprised if so, as there is such a drive to make top scores at all costs — no matter what the damage done to children in the process.

    • judyedwards

      I don’t know if they cheated, but the fact that the scores have not been sustained over the years, as has the scores in Nest+m and other schools who have scored in the high 80 and 90 percentiles across the years say a lot. Also you must factor in the fact the SAs did nothing for 4 months but test prep for 7 days a week, 6-10 hours each day. I know parents of SA students who were at the end of their rope by the time testing was done. If this test prep led to no real learning, which I deduce from the differing test scores across the years-then what is there to be hailed?

  27. Gabriel

    Leonie,

    The irony of me “defending” Eva’s school is perhaps not obvious to most readers. Regardless I think it’s unfair argument to claim the low test scores prove charter schooks are bad or unsuccessful relative to public schools, while then turn around and claim that those same scores when outstanding are NOT a sign of something working well. We need to be consistent and if we think these tests measure nothing of import then it goes both ways: i.e. it doesn’t mean the charter a schools are less successful. I’m not sure the other issues you raise that this is either a fair or appropriate forum to be discussing them. The test score success of Eva’s schools, are really outstanding no matter what demographics she is dealing with.

    • judyedwards

      George, would you still say that SA’s scores are outstanding if you knew that for 4 months the students did nothing more but ‘slam the exam’ which was non-stop test prep for 10 hour days and even on saturdays and sundays. This is why praise for higher scores is an unreasonable as derision for lower scores unless the variables are known. Only the schools who have sustained their numbers, what ever they are, across the years tell the real story. Do you agree?

  28. I have never used the test score argument for or against charters; but I insist that the Success Academy is NOT not a school that anyone should want to emulate. High test scores do not justify abusive behavior to kids, parents or teachers; high suspension rates and pushing out of young children, or the active undermining of the education of other public school students, many of them higher needs than those at Success through the aggressive confiscation of their classroom space. Instead, if we are to emulate or learn from any charter chain it should be the Icahn chain, where classes are capped at 18, where there are low attrition levels of students and teachers, where they are not depriving other children of the space they need to learn, and where there is, by all accounts, a far more humane and equitable vision of education.

  29. The more I realize how different my vision of education is than others the more I see the wisdom of choice, frankly. I prefer a fairly traditional curriculum, if not a classical education for my daughter. No Excuses didn’t exist when I was a kid. My father was forever threatening me with military school. I suspect I might have amounted to something had he actually made good on that threat. (I’m not joking; I mean that). Teaching for five years in a chaotic South Bronx public school sold me on No Excuses. Choice again: a lot of parents are deeply grateful for the order and purposeful tone. Those who want something else should be free to have it.

    Test scores are an interesting. My classroom experience made me militant not on testing but test prep. I’ve long believed that’s where the real damage is done for reasons I won’t elaborate upon, but have written extensively about. It’s perhaps my most significant point of departure with many of my friends in education. As a parent and a teacher, I’m none too pleased when test prep becomes the driver of curriculum.

    That said, it’s probably not a good idea for those of us who are fortunate enough to raise our kids in comfortable circumstances with great schools at our disposal to sniff too much at test prep. Who among us has not or will not cast a wary eye at our kids’ SAT scores and thought about a tutor. Loosening the grip of testing is a bit like nuclear disarmament. Great idea!

    “After you…”

    • The “wisdom of choice” is politically loaded – meaning choices from the charters not choices within the system. There is no little irony that, Bloomberg et al are the major backers of “choice” outside the system of 1700 schools they run, schools that under their management should be able to offer a wide array of choices without resorting to setting up a privately managed school system. “Choice” is a political, not an educational concept.

  30. Gabriel

    I think we should stick to the test score issue in this space, but lets asume that parents freely chose to go to Eva’s charter schools, and that as part of the deal they agree to submit to whatever rules, expectations and school culture SA establishes (which I may indeed find dehumanizing, etc. AND not right for MY children). Nobody forces parents to go through Eva’s hoops! And lets assume that the fight for space is not just btw charter schools vs public schools (colocation occurs btw public schools also – I lived through this) and that unfortunately it’s political and often ugly (and everyone fights for their kids and their school – what other choice is there?). Then I’m not going to fault Eva’s schools (or Eva!) for defending their turf and getting the space they need (and I realize she has resources and power most school leaders lack). I disagree with many religious, private and even public schools culture, policies, etc, but other than my own schools and my own kids schools, Im fine with other parents choices and there existing schools in this world which I consider dehumanizing, philosophically corrupt, not right for my children, intelectually vacous, etc.

    Leonie I agree with the imp of class size (although I estimate teacher quality relatively more important)…

    The real issue here is what, if anything, does this drop in scores signify, about one dimension of outcomes (possibly even one that is not even that important). Given serious problems with the entire framework of logic used in the construction, validation and development of these tests, (see jazzman analysis yesterday) and eve more dubious protocols for establishing cutoff pts, do the scores suggest something important about the qualities of education across the spectrum of schools in NYC? I’m actually skeptical the state test score analysis, no matter how sophisticated in controlling for demographic variables, can tell us that much (and I’d never advise a parent or myself choose a school for my children on the basis of test scores). But this blog space seems to have initiated an argument on these lines (and certainly the charter schools have also used the test score x quality education argument to make their case for greater success – so maybe they have this apparent undressing coming!), and given that, its important to be consistent in the analysis and focus on scores. Great schools, in my observation, are doing so many exciting things, that test scores are more like “background noise” rather than the obsessive diamond measure of school quality it has become.

    • Steve M

      You say: “…lets assume that parents freely chose to go to Eva’s charter schools, and that as part of the deal they agree to submit to whatever rules, expectations and school culture SA establishes (which I may indeed find dehumanizing, etc. AND not right for MY children). Nobody forces parents to go through Eva’s hoops!”

      So, you are agreeing that such a school has mechanisms in place to weed out those kids who hail from less informed, less motivated parents?

      Would you consider such a school replicable? Could such a school service 100% of the children in the neighborhood? Where would the other kids go?

      The problem with such schools is that they have not created models to service then entire school population…only the children of parents who “freely choose” such an environment.

      • judyedwards

        SA schools almost stopped teaching the children, 4 months before the tests and concentrated solely on test prep, calling it “slam the exam”. The poor students and their families survived 10 hour days 7 days a week to cram for test prep. I do hope it’s not the short-term memory that stuck. Based on all the energy and school hours they should have performed better than 70%. This is why it’s hard to judge or characterize what the scores mean visa a vis the preparation given.

  31. Michael Churchill

    While I am predisposed as a public school advocate to like the conclusion, using one grade, one subject to judge how charters generally did vis-à-vis district run schools is not very reliable. Some of the schools are quite small. Can you add to the robustness of this with other grades? And what do we know about the five schools above 90% both years? Are they select admission schools or what?

    • judyedwards

      the school my grandchildren attended are one of the schools which has sustained over 90%. The school is Nest+m a citywide gifted school that teaches one grade above the actual school grade.

  32. matt

    I think yoy are miss inperpreting your data. On average charters outperformed publics. You ignore this and focus on the rates of chang not overall proficincy. Why is that? Its also woth noting that poorly perfoming charters will lose funding and close whereas underperforming publics stay open and their flawed emoyees protected.

  33. Gabriel

    Steve M.

    Indeed choice presents a problem for children whose parents are so disadvantaged (or absent) or illinformed as to be unable to make good (whatever this might mean for a parent) choices. But this appears to be less of a problem. In my studies of precisely this issue (over 15 years ago) and in both NYC and Puerto Rico, this is a very small percentage of the population. This is a complex subject and not enough space here to discuss it fully. Other published studies examining issues of this kind also suggest its a small problem. I’m not sure, in any case, I’d prefer a system of no choice – but the comprehensive American public school (some countries have essentially this kind of only public school only system, and homeschooling, for example, is not allowed). Regardless I don’t advocate for charters taking over the entire public school system – (in fact I’m not really a charter advocate – in case anyone is wondering). I think that if some charter schools are doing some things better than other x schools we can learn from that (the same argument applies to public and private schools that may be doing wonderful things). And despite my misgivings about some charter school leaders (misgivings that apply equally well if not more to public and private school leaders – the field is filled with incompetents of all creeds, political inclinations, educational backgrounds and experience. I think that in the overall analysis choice (including some charters) is good for the greatest number of kids and families. This doesn’t mean any schooling model can or should be replicated 100%, but it does mean that variety is good, not bad for the system as a whole. Private, international, public magnet, public selective, public comprehensive, public vocational, public remedial, themed public schools, etc, etc add to the richness of opportunities, that provide for the varieties of needs and capacities and talents, and that fulfill the diversity of aspirations and goals, of our children (and parents who have responsibility too decide). Not that I think that the state test scores are of much relevance to any of the above – but IF parents think CC test scores are the thing that is most important for their children, and if indeed some charters schools get better scores (and if we can properly control for demographic effects, directed attrition, dubious claims, exaggerated statistics, outright lying, etc.) to casually link them to some things charters schools are doing (all by the way questionable at this moment) then I think it’s fine for schools of any kind to emulate what the charter schools are doing – and yes replicate away. One of my main gripes with the testing obsession and now the common core that is overly focused on college prep, is that it’s a totally unrealistic and ridiculous goal. No country in the world can provide college level jobs even if it managed to graduate 50%. And we seem intent on shooting for 100%? Every student college ready? It’s an untenable and absurd goal to aspire for. We’d end up with the must dysfunctional, frustrated and unhappy population in the world!

  34. Jack Covey

    The one that jumped out at me was KIPP A.M.P., because the staff there were the first and only KIPP to unionize… if only for a year. The stories behind Dave & Mike’s vicious, Stalinist tactics to prevent the initial unionization at that school from ever happening… and the systematic crushing of that union during its one year of existence are chilling, harrowing stories that are yet to be fully told.

    Needless to say, Dave and Mike retaliated later on with a total purge of anyone even remotely sympathetic to or supportive of that short-lived attempt at unionization—like a scythe cutting wheat, they made sure the dire fate of those unionist was known throughout the national KIPP network, so that all concerned would know they would face the same fate… and thus, unionization would never happen again.

    Based on these scores, it looks like those teachers’ replacements sucked to high heaven.

    No excuses, indeed!

    Since the purged teacher are no longer part of KIPP and no longer subject to further retaliation from KIPP—”What are you gonna do? Fire me… again?”—perhaps some of them can go public with that story in all its gory details.

  35. judyedwards

    I do no for a fact the the success academies did nothing but test prep (very little learning went on) for 10 hours each day and on the weekends as well for at least four months. So their reasonable “success” of 70% is not indicative of all the time their students were drilled in test prep, sadly.

    • judyedwards

      sorry for the twitter acronyms seeping in. that shoudl be *know* not *no*

  36. navigio

    Ironically, many charter schools in my state (California) claim that they do worse on our current tests because they don’t focus on test preparation. They even tout that as one of their benefits. If they end up doing worse on an assessment pegged to a standard that was supposed to have done the same thing then either they were lying or they’re just worse. That’s not very encouraging.
    I also think it would be a good idea to try to find out which schools are charters (as opposed to assuming they have the word ‘charter’ in their name).

    • If it helps, from the state data files, NYC charter schools without ‘Charter’ in the listed name (this list is schools with 6th grade classes):

      LEADERSHIP PREP BEDFORD STUYVESANT
      HARLEM VILLAGE ACADEMY LEADERSHIP
      HARLEM CHLDRN ZONE ACADEMY II
      HARLEM CHILDREN’S ZONE PROMISE
      DEMOCRACY PREP ENDURANCE
      LAUNCH EXPEDITIONARY LRNING

      There are others that have an abbreviated form of the word charter, e.g.:

      EXCELLENCE BOYS CHAR SCH-BED STUY

      I don’t know whether those got included.

  37. Garrett Ewald

    One thing Success does is delay spring break for 3rd grade until after the exams. Probably do the same for higher grade that gets extra testing. Seems strange to have a major test after a holiday. They also have longer school days, and allocate time each day differently.

    With all the charter bashing one thing I haven’t seen talked about much is what is the exit strategy for Charters? At what point do their techniques cross over to other schools, and when does consolidation occur? Even the big guys will struggle running a large organization, and with the threats to curtail colocation the growth to stay ahead of expenses will likely end soon.

  38. Garrett Ewald

    And while SA may have devoted extra energy to test prep, I doubt they were alone in doing it.

    • judyedwards

      I believe all schools devoted extra energy to test prep-but not to the exclusion of everything else as SA did. Also, if you look at their test performance over the years they rise and fall with the different standardized tests. That to me is true indication of teaching to the test. For instance at Nest+m, 5 years of sustained over 90% , the scores do not rise and fall based on the tests, which seems more indicative of teaching at a higher level. And not necessarily to the exam. Nest also they also did test prep, but not for extra hours/days/suspended vacations.

      • Harringtonian

        Whether or not it primarily serves an “upper class population”, NEST-M is certainly a “screened school: with admissions exclusively determined by (ahen) standardized tests. If anything,its ELA and Math test scores ‘validate’ its admissions policies and NOT its teaching practices.

        In fact, based on its Learning Environment Surveys, Nest-M consistently scores among the worst NYC public schools in terns of teacher morale. So, maybe the analogy to Success Academy isn’t so far ffetched after all? ‘;

        • judyedwards

          Harrintonian. I agree with you there about teacher morale. And am not at all touting Nest’s bureaucratically run school. But if Nest isn’t comparable, then neither are any of the charters this thread has compared. We are comparing apples and oranges and nothing is valid. Is it? So you are saying that teacher morale is a value that should be apprised of in the grading schemes.

      • judyedwards

        Also, since you folks are more wont to compare charter schools, look at the charter school that have had sustained scores over a period of years and not rise and fall to the type of test-the same is true. So either something is valid or nothing is valid in the comparing of schools on the exams.

  39. Jasper Steenhuis

    Do you really think it makes sense to compare directly the results from a G+T school that serves a decidedly upper class population with a charter school that serves historically under resourced neighborhoods? Is thought the level of this conversation was better than that.

    • judyedwards

      No, I don’t think that it makes sense to compare anything with those scores. But if we are comparing charter schools with great resources and creaming abilities, then GnT are fair game. Also my grandchildren are black and from Harlem. The same children that SA might have in attendance. So to answer your question, yes, it’s quite fair. Actually I believe the resources of the SA schools is better and the pockets deeper than Nest.

  40. Valerie

    I tried to get the Harlem Village Academy’s behavior and discipline policy/statement. No doubt parents are informed before and sign an agreement. Discipline is an issue that keeps public schools from doing the best job possible. Charter schools are allowed to drop/kick out a student for behavior problems, public schools no. Please someone post this agreement and description of behavior expectations and CONSEQUENCES! Thank you!

  41. All I wanna say, is that, we are all excited here in the Philippines to start a new system of school year, I am very pretty sure this would do a lot of improvement to our education system in the Philippines.

About this Blog

By a somewhat frustrated 1991 alum

Region
Houston
Grade
High School
Subject
Math

Subscribe to this blog (feed)


Subscribe via RSS

”subscribe

Reluctant Disciplinarian on Amazon

Beyond Survival On Amazon

RSS Feed

Subscribe