Nov 07 2013

On the latest D.C. NAEP Miracle

Today the results from the 2013 NAEP came out, and ‘reformers’ everywhere are celebrating the amazing gains by states with an aggressive reform agenda.  In 2011, the average national score for the four main NAEP tests (4th grade reading, 4th grade math, 8th grade reading, and 8th grade math) was about a 250, with the total of the four scores being 1006.  For 2013 the sum of these four scores, nationally, increased by 5 points to 1011.  But some states increased by more than 5 points, as shown on this graph that was shared often on Twitter.

This shows that the places with the greatest gains were D.C., Tennessee, and Indiana, three places that have embraced the corporate reform strategy of testing, closing down schools, and opening charters.  If this was the only data we had access to, it would seem to prove that “the ends justify the means” when it comes to education reform.

Scores do fluctuate from year to year, so part of me would like to just ignore this and wait for two more years when the scores likely do not continue to grow this way, but with all that is going on around the country so quickly I don’t have the luxury of waiting two years.  What I can do, though, is put the growth scores into some context by examining more of the raw data.

Though for the nation, the average sum of the four scores in 2013 was 1011, I noticed that the combined four scores for D.C., even with their 22 point gain, was just a 947 which was 64 points below the national average.  Curious to see where that put them in comparison to other states, I made this chart.

As can be seen, this puts D.C. in last place, way behind the second-to-last Mississippi.

But, ‘reformers’ could say, is it fair to compare the results of a city with the results of entire states since the students in D.C. have, on average, more out of school issues to deal with than the average student in a state.

So what I did next was isolate the scores of the students who are eligible for free lunch in each of the states and in D.C..  This, I think, is a more fair ‘apples to apples’ comparison.

Again, D.C. is way behind, their students eligible for free lunch having a combined total of 913, which is 49 points below the national average for students eligible for free lunch.  Tennessee has fared better, but still below the national average.

Finally, I looked at how the infamous D.C. ‘achievement gap’ was doing comparing test scores of students eligible for free lunch with students who were not eligible.  Again, D.C. finished last by a wide margin with a 157 point gap.

There are many other things to analyze, and I’m looking forward to reading how others analyze the data.  For example, it is curious that Louisiana had ‘gains’ that were smaller than the national average despite that state having, certainly, the most aggressive reforms occurring.  For ‘reformers’ who are so obsessed with test scores and test score gains, this is certainly something that shouldn’t be ignored.  Also, Washington and Hawaii were pretty high up on the ‘growth’ numbers even though Washington does not have charter schools and Hawaii has been very slow to adopt Race To The Top reforms so their ‘gains’ can’t be attributed to those.

I’m still pretty confident that in the long run education reform based primarily on putting pressure on teachers and shutting down schools for failing to live up to the PR of charter schools will not be good for kids or for the country, in general.  I hope politicians won’t accept the first ‘gains’ chart without putting it into context with the rest of the data.

28 Responses

  1. Mark Weber

    Something else to consider about Ladner’s chart, Gary:

    When you add up the scores on four tests, you’re essentially saying that the score gains on each are equivalent in terms of “real” educational gains. In other words, that a 4 point gain on the 4th Grade reading test is 80% of the “real” gain in learning on a 5 point rise on the 8th Grade math test.

    There is no justification for thinking this. Further, a 4 point gain from, say, a 220 to a 224 can in no way be assumed to be equivalent to a gain on the same test from 250 to 254.

    But this is exactly what Ladner assumes with his graph. The entire comparison, therefore, is invalid.

  2. asdd

    Not too impressed w/ this analysis Gary. Is this the best you can do?

    • James Ploeser

      if you’re not impressed, say why. Are you IMPRESSED with the initial graph that’s obscuring this info?

  3. ASDD, you took the words out of my mouth.

  4. skepticnotcynic

    What does it matter? The scores from states with aggressive reform agendas should have gone up more than other states who haven’t bought into the reform myth. If all these states do is emphasize test prep, while eliminating valuable learning experiences like art and music, then it is no surprise to see these faux gains on phony learning metrics like high-stakes exams. A better question to ask is what sort of opportunities will the kids from DC public schools have in the future? Test prep is certainly not going to elevate them out of poverty.

  5. Julia Rubin

    DC has seen a dramatic rise in income due to ongoing gentrification, which could more than explain any increase in test scores. http://blogs.wsj.com/economics/2013/09/19/washington-sees-incomes-soar-as-most-of-u-s-declines/

  6. tlmerrie

    I’d like to see you do more with this. Data analysis like this isn’t really my forte as it is yours.

    Here in Tennessee, we put new standards in place in time for the 2009-2010 school year. I didn’t teach in Tennessee before then, but having seen the shift in materials, curriculum, etc. I felt like the switch to the new standards was a move forward. At least for math. I fail to see how what is happening in our state in terms of testing and reform is positive; mostly everyone does a lot of test prep, even in the affluent schools. But I have noticed that kids are coming to me in ninth grade having already been introduced to more advanced material than before. Each year I have seen an improvement in this regard. So couldn’t something like this change in standards explain any gains?

    These standards, which will now bow to Common Core, were put in place before Huffman and Haslem.

    I am troubled by what they are going to be saying about these results and how their reforms are working, when those of us actually in schools are seeing what really goes on.

    • Meghank

      You’re from Tennessee, too! I live in Memphis. What I think about these results is this: If the scores dropped or remained the same in the states with the most reform, they would be calling for reform with greater fervor. Since the scores improved, they are still calling for reform with greater fervor because the “reforms” “worked.”

      Since, whatever the scores were, the response from the “reform” crowd would be the same, we can safely ignore these news reports.

  7. Dylan

    I’m pretty disappointed in this level of analysis. All of your charts are absolute measures–clearly D.C. students aren’t learning as much as their peers in other states. However, that doesn’t say that D.C. didn’t improve, even if it was only from abysmal to terrible. Contextualize the growth scores or don’t bother picking apart the study

  8. Francis Sullivan, Jr.

    Given how poorly schools in DC and Tennessee had been performing, it should not be a surprise that their scores would rise substantially–they had nowhere to go but up. Sociologists call this phenomenon “regression to the mean.” Equally important, there may be a “placebo” effect operating; participants react positively to any structural intervention in a system. I agree with Gary that we need to see if these changes continue.

  9. Having spent time in both Wyoming and DC, I think it somewhat questionable to compare. There are very different situations.

  10. I appreciate your work, Gary.

    I am curious to see the NAEP longitudinal gains (losses) for free-lunch students.

    Let me know if you have looked at this for years prior to 2011.

  11. l hodge

    For what it is worth, here is a what the gains look like broken out by free & reduced price lunch eligibility. Gains are averaged for the four tests – Math/Reading for Grade 4/Grade 8.

    https://drive.google.com/file/d/0B0LlvF7Dr9chUnJEY3Q3bkhCRk0/edit?usp=sharing

    The more impressive gains were for those NOT eligible for free/reduced price lunch. DC easily beat every other state by improving by “14 points”. TN was second with “7 points”. The US average was “2 points”.

    The gains for those eligible for free/reduced lunch were more modest. DC tied West Virginia by improving by “5 points”. TN was in the next group of 4 states that improved by “4 points”. Indiana was in a group of 5 states that improved by “3 points”. The US average was “1 point”.

    Don’t see how you could really conclude much of anything based on these changes over one testing period, let alone attribute the changes to anything in particular.

    The spreadsheet is here: https://drive.google.com/file/d/0B0LlvF7Dr9cha2kxbkdCb1NycGM/edit?usp=sharing

    • MikeDC

      There has been a great deal of demographic change in the composition of DCPS students. I looked at the racial break-down of scores and assigned the race average score at the different periods shows that demographic changes can account for about 3.5 points the one test I ran for 4th Grade Math.*

      It’s a rough approximation but matches the experience in the not-free-and-reduced lunch achievement gains. The white kids that test better don’t qualify for FARL in DC.
      You can see the demographic change over time if you look at all time periods and notice when 8th grade whites are available. There were years when there were too few white students in DCPS to make the NAEP cut-off.

      * I compared 2009 test and 2013 tests using the 2008 and 2012 racial compositions that were the closest demographic data I could find online. I was only able to do a 3 race estimation (no Asians, no Others) since these subgroups were too small for NAEP to report for DC.

      It is a rough estimation, and complicated by mismatch on demographics and also mis-alignment between the percent of race in entire population vs tested population, e.g. there are lots more white 4th graders than 8th graders. (And more 1st graders than 4th graders because of when the gentrification and baby boomlet happened in DC.)

      • Mike, this sounds great. Can you please post this demographic analysis or link to where it is? Very interesting…

  12. Minnesota, which has used a variety of strategies also made significant gains, and reduced some achievement gaps. Mn isn’t Tennessee or DC.

    It has used strategies including charters, statewide testing, more funding to help improve reading and math instruction, etc.

    We see charters as an expansion of opportunity. We see them as the opposite of “public schools” for which students must apply to enter – such as the elite quasi private “exam” schools of NYC.

Post a comment

About this Blog

By a somewhat frustrated 1991 alum

Region
Houston
Grade
High School
Subject
Math

Subscribe to this blog (feed)


Subscribe via RSS

”subscribe

Reluctant Disciplinarian on Amazon

Beyond Survival On Amazon

RSS Feed

Subscribe