I’ve written before about KIPP attrition in response to reports that had been released studying it. When reports conclude that KIPP does not have high attrition, they tout it on their websites. When reports concluded that they do have high attrition, KIPP responds with a rebuttal.
The problem with most of these reports is that the data they give us has already been analyzed and then turned into percentages, which are only relative measures. This is why I finally got around to navigating the New York START data system to find the actual raw data for myself which I could then compare to KIPPs annual report card that they release.
The reason I’m so committed to uncovering stories of exaggerated success is that these stories have become battle cries for ‘reformers’ like Michelle Rhee. She, and others, have been influencing politicians to create flawed education policies based on misleading success stories.
Because KIPP has many critics, they responded to those critics in their recently released annual report card. In it, they address the six main concerns that critics of the program have raised:
- Are we serving the children who need us?
- Are our students staying with us?
- Are KIPP students progressing and achieving academically?
- Are KIPP alumni climbing the mountain to and through college?
- Are we building a sustainable people model?
- Are we building a sustainable financial model?
Of course their answers to these questions will be ‘proved’ to be ‘yes’ for all six questions in their report.
In this post, I’ll focus on point 2, attrition.
On page 15 of the report they write
“A KIPP school with great test scores—but high student
attrition—is not meeting our mission.
By choosing KIPP, students make a commitment to excellence and in return, KIPP promises to help each student on the path to and through college. We believe these promises are sacred and we hold ourselves accountable to fulfilling these promises to every student.
Our second essential question asks us to consider whether we are making good on the commitment we have made to every single one of our KIPPsters. This means making sure that the students who join us stay with us year after year. We highlight this question because we believe it is as important to a school’s health as its test results. The reality is that a school with great test scores and high student attrition is not realizing our mission.”
Then they show the attrition for each school on the 99 pages that summarize the success of their 99 schools. Some schools boast 2% attrition, while others are as high as 47%. But the most telling statistic is the pictograph on page 15.
Well, 88% doesn’t sound too bad when you read it as quickly as most rich donors do. But when you look at it more closely, this does not mean that 88 percent of students who start the middle schools as 5th graders will eventually graduate as 8th graders (most KIPPs are 5-8 middle schools). The 12% attrition is PER YEAR. So this means that 88%, on average, make it to 6th grade, then they lose 12% of those, which takes us down to 77% for 7th graders, 68% for 8th graders, and finally 60% for graduating 8th grade. Suddenly it doesn’t look so good.
Then I thought I’d take an individual school ‘KIPP Academy New York’ which is the first New York KIPP, and check their actual school report cards against their claims on the KIPP report card. They claimed to have a 4% attrition rate, which really means that compounded over four years is really a 15% attrition, but is still way better than the 40% attrition based on their published overall 12% attrition rate.
So I downloaded the 2009 and 2010 school report cards. I learned that there were 203 5th, 6th, and 7th graders in 2009 who became 192 6th, 7th, and 8th graders in 2010, which is 95% which is very close to the 96% that they published. But then I looked closer at the numbers and also followed a particular cohort from 5th grade in 2007 to 8th grade in 2010 to get a fuller picture of what is going on. For this I needed the 2007 and 2008 school report cards.
One thing I noticed was that in 2009 they had 30 students with disabilities in their 5th, 6th, and 7th grades combined while in 2010 they had only 22 in their 6th, 7th, and 8th combined. So of the net 11 students that they lost, 8 of them were students with disabilities.
Things got more interesting, though, when I followed the 2010 cohort from the time they were in 5th grade. In 2007 that class had 72 students with 44 girls and 28 boys. Four years later they have lost a total of nine students so they have 63. But, and here’s the strange part. Those 63 kids are 16 boys and 47 girls. (some ratio!) So it seems like they lost AT LEAST 12 students from the 72 (the 12 boys, that is) which is about 17%. Since the number of girls actually increased, it shows that they have REPLACED some of the students they lost with other kids. This is not factored into the attrition rate, though it is possible that the new students are better at the standardized tests than the ones who left, while the attrition rate is not affected. So when we look at the improvements in test scores from 5th to 8th grade, we are looking at two different groups of kids. With sample sizes of 70, a few kids makes a big difference.
I also noticed that in 5th grade there were 5 kids with Limited English Proficiency, while in 8th grade there were only 2. Now, this could mean that students lose their LEP status, so I’m not sure if this is relevant.
I haven’t fully immersed myself in the KIPP data, but I hope I’ve given enough information to demonstrate that KIPP misleads when they report an 88% retention rate and also that the raw data conclusively demonstrates that they fill in some students with students who leave which makes their attrition rate seem better than it is. Now, I’m making an assumption that the kids that fill in those spots are better, academically, than the ones who are counseled out. Even if I’m incorrect about that, I hope I’ve introduced enough for people to think about when they hear data about KIPP’s success. Also, I hope I’ve given some tips to others who want to investigate miracle schools to find some of their own anomalies and share them.