Jul 08 2011

KIPP On Trickin’ — looking at the raw data

by Gary Rubinstein

I’ve written before about KIPP attrition in response to reports that had been released studying it.  When reports conclude that KIPP does not have high attrition, they tout it on their websites.  When reports concluded that they do have high attrition, KIPP responds with a rebuttal.

The problem with most of these reports is that the data they give us has already been analyzed and then turned into percentages, which are only relative measures.  This is why I finally got around to navigating the New York START data system to find the actual raw data for myself which I could then compare to KIPPs annual report card that they release.

The reason I’m so committed to uncovering stories of exaggerated success is that these stories have become battle cries for ‘reformers’ like Michelle Rhee.  She, and others, have been influencing politicians to create flawed education policies based on misleading success stories.

Because KIPP has many critics, they responded to those critics in their recently released annual report card.  In it, they address the six main concerns that critics of the program have raised:

1. Are we serving the children who need us?
2. Are our students staying with us?
3. Are KIPP students progressing and achieving academically?
4. Are KIPP alumni climbing the mountain to and through college?
5. Are we building a sustainable people model?
6. Are we building a sustainable financial model?

Of course their answers to these questions will be ‘proved’ to be ‘yes’ for all six questions in their report.

In this post, I’ll focus on point 2, attrition.

On page 15 of the report they write

“A KIPP school with great test scores—but high student
attrition—is not meeting our mission.

By choosing KIPP, students make a commitment to excellence and in return, KIPP promises to help each student on the path to and through college. We believe these promises are sacred and we hold ourselves accountable to fulfilling these promises to every student.
Our second essential question asks us to consider whether we are making good on the commitment we have made to every single one of our KIPPsters. This means making sure that the students who join us stay with us year after year. We highlight this question because we believe it is as important to a school’s health as its test results. The reality is that a school with great test scores and high student attrition is not realizing our mission.”

Then they show the attrition for each school on the 99 pages that summarize the success of their 99 schools.  Some schools boast 2% attrition, while others are as high as 47%.  But the most telling statistic is the pictograph on page 15.

Well, 88% doesn’t sound too bad when you read it as quickly as most rich donors do.  But when you look at it more closely, this does not mean that 88 percent of students who start the middle schools as 5th graders will eventually graduate as 8th graders (most KIPPs are 5-8 middle schools).  The 12% attrition is PER YEAR.  So this means that 88%, on average, make it to 6th grade, then they lose 12% of those, which takes us down to 77% for 7th graders, 68% for 8th graders, and finally 60% for graduating 8th grade.  Suddenly it doesn’t look so good.

Then I thought I’d take an individual school ‘KIPP Academy New York’ which is the first New York KIPP, and check their actual school report cards against their claims on the KIPP report card.  They claimed to have a 4% attrition rate, which really means that compounded over four years is really a 15% attrition, but is still way better than the 40% attrition based on their published overall 12% attrition rate.

So I downloaded the 2009 and 2010 school report cards.  I learned that there were 203 5th, 6th, and 7th graders in 2009 who became 192 6th, 7th, and 8th graders in 2010, which is 95% which is very close to the 96% that they published.  But then I looked closer at the numbers and also followed a particular cohort from 5th grade in 2007 to 8th grade in 2010 to get a fuller picture of what is going on.  For this I needed the 2007 and 2008 school report cards.

One thing I noticed was that in 2009 they had 30 students with disabilities in their 5th, 6th, and 7th grades combined while in 2010 they had only 22 in their 6th, 7th, and 8th combined.  So of the net 11 students that they lost, 8 of them were students with disabilities.

Things got more interesting, though, when I followed the 2010 cohort from the time they were in 5th grade.  In 2007 that class had 72 students with 44 girls and 28 boys.  Four years later they have lost a total of nine students so they have 63.  But, and here’s the strange part.  Those 63 kids are 16 boys and 47 girls.  (some ratio!)  So it seems like they lost AT LEAST 12 students from the 72 (the 12 boys, that is) which is about 17%.  Since the number of girls actually increased, it shows that they have REPLACED some of the students they lost with other kids.  This is not factored into the attrition rate, though it is possible that the new students are better at the standardized tests than the ones who left, while the attrition rate is not affected.  So when we look at the improvements in test scores from 5th to 8th grade, we are looking at two different groups of kids.  With sample sizes of 70, a few kids makes a big difference.

I also noticed that in 5th grade there were 5 kids with Limited English Proficiency, while in 8th grade there were only 2.  Now, this could mean that students lose their LEP status, so I’m not sure if this is relevant.

I haven’t fully immersed myself in the KIPP data, but I hope I’ve given enough information to demonstrate that KIPP misleads when they report an 88% retention rate and also that the raw data conclusively demonstrates that they fill in some students with students who leave which makes their attrition rate seem better than it is.  Now, I’m making an assumption that the kids that fill in those spots are better, academically, than the ones who are counseled out.  Even if I’m incorrect about that, I hope I’ve introduced enough for people to think about when they hear data about KIPP’s success.  Also, I hope I’ve given some tips to others who want to investigate miracle schools to find some of their own anomalies and share them.

 

Categories: Research · 28 Comments

28 Responses

1. Mavor

   Brilliant Job.

   on July 8, 2011 at 4:46 am · Reply
2. Heather

   Your fatal logic flaw is assuming that students who leave are counseled out. Here are other reasons they may leave.

   1) Family moves. Very, very common in transient urban neighborhoods.

   2) Students and/or parents dissatisfied with high workload or behavior expectations.

   3) Students miss neighborhood peers and return to public school.

   4) Parents dissatisfied that their previous B student is now a C or D student because of higher expectations.

   5) Student is retained and changes schools, rather than repeat the grade.

   These are all reasons I have seen kids leave charters. Never, though, have I met a kid who was counseled out. Not saying it doesn’t happen, but it is far, far, less common than you think.

   on July 8, 2011 at 7:21 am · Reply
   • C

     Honestly, who cares why they leave? The point is that KIPP and other charter schools claim to have a certain level of academic success over time, but this success may be driven by the fact that a large number of students are leaving. If they matched to test scores of 6th graders only with the scores of those still present in the 8th grade, the substantial increase in scores they tout would be more convincing. Comparing a sample size of say 100 with a sample of 70 doesn’t make much sense.

     on July 8, 2011 at 12:46 pm · Reply
   • E. Rat

     Reasons 2-5 are all implicit forms of “counseling out”. Renaming the privileges KIPP (and all charters have) with respect to enrollment doesn’t change their nature: they are privileges that have the impact of leaving a culled class of students who are lower-needs. Since higher-needs students do still need an education, they go to regular public schools, who now have student populations that are more concentrated with high needs students.

     If a school is using retention, parent contracts, etc. in a way that causes students to leave, it is counseling students out.

     on July 8, 2011 at 3:06 pm · Reply
3. Jerry

   The difference for KIPP is they do not have to do all the crazy NCLB reporting. A public high school must account for every one of those students and report where the students “went” as part of OTGR. A school with low SES will more than likely have a high transient rate because of the job market. This become very resource dependent for the school and take away from instruction. Then schools with high transient rates because of poverty, are labeled as failures because of their low OTGR numbers. CRAZY

   on July 8, 2011 at 10:28 am · Reply
4. Edit Barry

   Thanks for doing this work, Gary Rubinstein. I live in Baltimore City, where KIPP has successfully negotiated a deal with the teachers union to extend the school day. There are two KIPP charter schools here, and the former chair of the KIPP board in Baltimore is running for mayor. So, it’s very helpful to have some insight into the data.

   on July 8, 2011 at 11:25 am · Reply
5. The Rank Stranger

   I’d like to see these data presented alongside a representative traditional public school to serve as acomparison group.

   Also, though it is KIPP’s mission to retain these students, the beauty of the charter school movement is that it empowers students and parents with CHOICE. I agree with the assumption that these students are likely NOT being counseled out, but rather are leaving on their own volition. That’s school choice at its finest.

   on July 8, 2011 at 12:55 pm · Reply
   • parus

     This response is SPIN at its finest.

     on July 8, 2011 at 9:51 pm · Reply
   • NYeducator

     The public middle school in which I work – in a district w KIPP and w a similar demographic and SES – just graduated a cohort of 8th graders that was 78 students as a 6th grade (the school’s entry point), 84 students as a 7th grade and 89 as an 8th grade, close to a 15 percent increase in size.

     on July 9, 2011 at 2:17 am · Reply
6. Stuart Buck

   “Things got more interesting, though, when I followed the 2010 cohort from the time they were in 5th grade. In 2007 that class had 72 students with 44 girls and 28 boys. Four years later they have lost a total of nine students so they have 63. But, and here’s the strange part. Those 63 kids are 16 boys and 47 girls. (some ratio!) So it seems like they lost AT LEAST 12 students from the 72 (the 12 boys, that is) which is about 17%.”

   The key problem is your use of the word “LOST” there. For all you know, some or all those boys were retained.

   on July 8, 2011 at 1:08 pm · Reply
7. Bob Calder

   @Heather and others that try to “interpret” the analysis: The data support the conclusion that there is nothing essentially different about being a student in a charter school. The outcomes are typical of the system. If I may add, typical of a lot of Title 1 schools nationwide. This should not come as a surprise.

   on July 8, 2011 at 1:09 pm · Reply
   • Bob Calder

     Here is a clear explanation of how drivers of change have to work. It is a PDF and the research is from the U of Toronto. Canada is among the top ten ranked nations and their population diversity issues are quite similar to ours.
     http://www.michaelfullan.ca/home_articles/SeminarPaper204.pdf

     on July 8, 2011 at 1:13 pm · Reply
   • Meyer

     Is a student who would be successful at KIPP better off at KIPP or at their neighborhood school? It depends on what both of those schools look like, but it’s entirely possible they get a better education at KIPP. So for that student, going to a charter school is a smart move. I’m using my anecdotal observations as a teacher here – in teaching at a traditional school, I saw many students who would probably be better off in a more structured environment and safer classroom.

     on July 8, 2011 at 3:48 pm · Reply
8. RheeFirst

   Great job, Gary! There’s also another good study by Gary Miron that I’ll send you soon unless you have it already.

   on July 8, 2011 at 1:11 pm · Reply
9. E. Rat

   Thanks for doing this important work.

   I think it is also critical to remember that because KIPP schools engage in behavior like this, they leave the regular public schools with a higher percentage of high-needs students. The demographic changes KIPP creates span their host districts, and KIPP’s success is predicated on making things harder on the schools around them. After all, those kids who leave are going somewhere.

   on July 8, 2011 at 3:08 pm · Reply
   • Paula White Bradley

     Many of the kids who are going “somewhere” are not actually going back to any school, they are leaving school altogether. And E. Rat – if a family is dissatisfied with a heavy workload, how the heck is that “counseling out”? It’s called making a C-H-O-I-C-E! The same applies for a child returning to her/his old school because they miss their peers – no logically-thinking person could interpret that to be a form of counseling out a student. Finally, if a child’s grades drop at KIPP because they’d been inflated elsewhere, and this leads the family to withdraw the child from KIPP, that decision can be attributed to the parent, not to the school. You seem to be suggesting that schools should morph into whatever a given constituent wants, and you’re suggesting that if schools don’t bend their rules and policies, then they’re counseling students out. I suppose that’s true – in the same way that if I brought a firearm to the airport, I’d be “counseled out” of my flight. Whatever.

     on July 8, 2011 at 7:23 pm · Reply
     • adirondackblue

       Paula, how does a 5th, 6th, 7th, or 8th grader end their education and never return to school?

       Needless to say, I agree with E.Rat.

       on July 9, 2011 at 3:35 am · Reply
     • E. Rat

       Sure, it’s a choice: but one intentionally created to have certain outcomes. After all, there is not one iota of evidence even suggesting that massive amounts of homework lead to increased student success. Structuring choices to create the outcomes you want – a specific student group – is counseling out.

       Similarly, if a child misses his or her peers, it suggests that the social climate at the new school may be challenging. KIPP, which uses exceptionally punitive disciplinary methods that include ostracizing students; I suspect that this management “technique” is so unpleasant for its victims that they do feel excluded and unwelcome – and leave.

       What I’m really suggesting is that KIPP’s “success” comes on the back of educators at real public schools who serve the students and families KIPP does not want. And to be honest: I’m not suggesting it, I’m living it.

       Your example is more than spurious; it’s silly. I’m typing about the way schools can create systems that drive enrollment and retention; you respond by attacking a straw man.

       on July 9, 2011 at 3:48 am · Reply
10. Paula White Bradley

    Your statements about “real public schools” and about “living it” – “it” being serving the students that KIPP supposedly does not want, suggest to me that you have a serious ‘ax to grind’…which is your right. The trouble with all of what you are saying is that you presume to know the intent of a person’s or entity’s actions or statements, whether that person is me or whether that entity is KIPP. So, you are convinced that KIPP assigns tons of homework, not to improve educational outcomes but to deliberately purge certain students from their schools. Because of your own narrowly defined interpretation of research, you are only thinking about ‘homework’ per se, and not framing the homework issue from the perspective of the benefit of practice, overlearning, or the effort effect, all of which have a significant research base to support KIPP’s approach to homework.

    In your response regarding children who miss their peers, you use words like, “I suspect”, and “it suggests”, which makes this aspect of your rebuttal an easy one to dismiss – obviously your statements are pure conjecture and have no identifiable basis in fact. You should fully admit this in all of your posts rather than writing with some supposed level of confidence as though that which you state has a solid base in reality.

    My example is neither spurious nor silly – the fact is that standards and rules exist in every institution or setting, not to push people out, but to encourage the kind of behavior that will make things optimal for the benefit of the larger group; towards the common good. Thus, in the same way that violating the rules of air travel results in removing an offender from that space, to send a clear message that participating in air travel requires a certain level of responsibility, similarly, a child in a KIPP class who is removed and/or isolated for being disruptive is supposed to get the message (and in fact, most do), that s/he does not have the right to compromise anyone else’s learning, AND that s/he should be prioritizing her/his own learning as well. Most children in KIPP don’t leave, rather they internalize that message, albeit at varying rates, and are ultimately better off for doing so. There are behaviorally and academically challenging kids in KIPP just like there are in your ‘real’ public school…and if you don’t like that heat, perhaps you should consider getting out of the kitchen, rather than whining about the children who are sitting in your classroom.

    I would elaborate even further, but I think my time would be better spent thinking deeply about, and preparing for, the challenging children who will walk into my charter school’s doors in less than 2 months. They will be sitting and waiting for me to deliver an impactful, life-changing education to them, and all of the lamenting about who and what they are won’t get me any closer to teaching them well. Only preparation will accomplish that, so off I go to do just that.

    on July 9, 2011 at 1:53 pm · Reply
    • B

      Congratulations

      on July 9, 2011 at 6:10 pm · Reply
    • E. Rat

      Far be it from me to keep you from spending July working on your classroom, but accusing me of blaming children for demonstrating the real trauma that comes from abusive situations in which they feel targeted is offensive in the extreme.

      I suggest that you spend your deep reflection time considering cultural competence in management strategies used in classrooms.

      on July 10, 2011 at 3:02 pm · Reply
11. govt_mule

    Excellent work!

    I’ve been looking at the Mathematica reports on KIPP schools which go to great lengths to claim that attrition at KIPP is the same as at public schools. This is done in a very misleading fashion by treating transfers from one public school to another as “attrition”. Obviously there is no qualitative similarity between leaving KIPP to go to another school and moving from one (equivalent) public school to another, and the claims are completely invalid. It just shocks me that they can get away with such nonsense.

    on July 11, 2011 at 5:21 pm · Reply
    • CarolineSF

      But when students leave KIPP schools, they are by and large not replaced.

      That’s not due to an official policy, as I understand it; just that KIPP doesn’t encourage transfers in at higher grades. Also, it seems clear, KIPP’s supposed “long waiting lists” are an exaggeration.

      But when students leave a public school by transferring from one to another, they aren replaced by students transferring from other schools.

      Since research shows that the most academically challenged students are the ones who leave KIPP schools, that means the KIPP schools end up with a much smaller class, consisting of the higher achievers. Public schools, on the other hand, generally end up with no change in the size of the class, and with a random assortment of achievement levels.

      on July 11, 2011 at 7:53 pm · Reply
12. Harold

    Kipp has a policy of not accepting transfers in the middle of the year, as I understand it. They are allowed by whatever idiots who make up the state budget accounting rules to keep the funds for educating a student who has dropped out for the remainder of the entire school year, so they have a financial incentive to push students out as early as possible in the fall.

    on July 11, 2011 at 9:43 pm · Reply
13. Stuart Buck

    Does the propaganda ever cease: “when students leave KIPP schools, they are by and large not replaced.”

    Mathematica found the opposite: “Averaging across all sites, KIPP schools in the sample enrolled 13 new students per year in grade 6 (accounting for 19 percent of average total enrollment in that grade), 7 new students per year in grade 7 (12 percent of total enrollment), and 3 new students per year in grade 8 (6 percent of total enrollment).”

    on July 11, 2011 at 10:11 pm · Reply
14. Stuart Buck

    See also Table 5 from the report: http://www.mathematica-mpr.com/publications/pdfs/education/KIPP_middle_schools_wp.pdf

    As Table 5 shows, KIPP schools on average gain 2 MORE students than they lose between 5th and 6th grades; and on average they lose 2 more students than they gain between 6th and 7th grades and between 7th and 8th grades. So if you combine all three of the transitions between years, KIPP schools are losing an average of TWO students between 5th and 8th grades that are not replaced.

    This is a lower replacement rate than the comparison public schools, to be sure, but only a diehard hack would say that a mere TWO students not replaced between 5th and 8th grades means that students are “not replaced.”

    on July 11, 2011 at 10:15 pm · Reply
15. Andrew

    Wouldn’t this all be moot if test data wasn’t aggregated? If data were available anonymously on each student and teacher, the people could do their own statistics. Sure, most people aren’t good a statistics, but without the raw data, you don’t have a chance.

    on July 13, 2011 at 6:08 pm · Reply
16. CarolineSF

    Stuart Buck would have us believe that the 60% of KIPP students who disappear from each grade cohort between grades 5 and 8 were all retained to repeat a grade. Silly me — I would have thought that meant that the grades behind were increasing by that same number instead of showing the same attrition.

    on July 15, 2011 at 9:00 pm · Reply