MISUSE THE CHILDREN WELL: Interlude!

MONDAY, SEPTEMBER 24, 2012

Misconceptions to date: We’re off on a (one-day) mission of national import.

For today, let’s review some of the basics from our ongoing series:

In fairness to Gail Collins, she may not have known some basic facts when she wrote her unfortunate new book, As Texas Goes.

She may not have known that you have to be careful when you quote Diane Ravitch. She may not have known that the Texas public schools, on a statewide basis, were among the nation’s highest performers as of the mid-1990s.

She may not have known that the Texas schools are still among our highest performers.

We refer to the performance of the Texas schools on the National Assessment of Educational Progress (NAEP), the widely-praised “gold standard” of American educational testing—the program Collins has described as “the best national assessment we have.”

At the core of its program, the NAEP tests fourth-graders and eighth-graders in reading and math. Its most recent testing occurred last year—and here's the good news:

On last year’s tests, Texas students outscored the nation in all three major demographic groups in both subject areas tested. Examples below.

Collins may not have known that fact when she wrote her unfortunate book. Although she praises the NAEP, there’s no sign that she has ever looked through its voluminous data. That would include its data for the Texas schools of the 1990s or for the Texas schools of today.

To cite one example, she almost surely doesn’t know how well Texas eighth-graders scored on last year’s math test. As you can see, the state’s eighth-graders scored at the very top of the nation:
Texas students, eighth-grade math, 2011 NAEP
White kids: Third in the nation
Black kids: Second in the nation (behind Hawaii)
Hispanic kids: Second in the nation (behind Montana)
Hawaii and Montana have very few black and Hispanic students. On this test, Texas outscored all states with significant minority populations.

(White students in Texas trailed Massachusetts and New Jersey, in each case by less than one point.)

We’re selecting the subject, eighth-grade math, on which Texas students scored highest. But Texas students scored in the top ten among the fifty states in eight out of twelve demographic categories in the subjects we’re discussing.

That’s what happens when you “disaggregate” the data—when you compare Texas students to their demographic counterparts in the other states. But uh-oh:

As we’ll see later this week, Collins swears by disaggregation in her new book. She just doesn’t engage in the practice!

Most likely, Collins also doesn’t know that Texas students were at the top of the national charts all the way back in the mid-1990s. For example, here’s how they performed in fourth-grade math during that era. The NAEP doesn’t test every year:
Texas students, fourth-grade math, 2000 NAEP
White kids: Second in the nation (of 40 participating states)
Black kids: First in the nation (of 32 states)
Hispanic kids: Second in the nation (of 21 states)

Texas students, fourth grade math, 1996 NAEP
White kids: First in the nation (of 43 participating states)
Black kids: First in the nation (of 35 states)
Hispanic kids: Second in the nation (of 25 states)
Collins may not have known that Texas students were scoring that high by that time. Had she known, she might not have including this misleading passage in her book:
COLLINS (page 78): During the last half of the 1990s, Texas schools did get better. It was very, very hard to figure exactly how much better, given the amount of conflicting data floating around, but some observers were wowed by how well the students were testing. (“I couldn’t believe it,” said David Grissmer, who wrote or co-wrote several important education studies on the state.) Some were just prepared to thank God for small favors. (Molly Ivins called it the “story on how our schools rocketed from abysmal to only slightly below average in a mere thirty years.”) The doubters would be empowered later, when reporters discovered that in some places the results had been, shall we say, rigged.
Small favors? Slightly below average? And by the way: As far as anyone knows, no NAEP tests have ever been, “shall we say, rigged.” Outright cheating has occurred on the high-stakes tests the states themselves devise and conduct. No one has ever suggested that this has occurred on the federally-administered NAEP. (This is part of the reason why people like Collins call the NAEP the “gold standard.”)

Back to our original point: From reading that passage in Collins’ book, would anyone have any idea that Texas students led the nation on that 1996 math test? That they did so again in 2000, on the NAEP’s next math test?

Sorry, readers! According to our most reliable data, the Texas schools were not “below average” at this point in time. But you would never dream such a thing from reading Collins’ unfortunate book.

As early as 1996, Texas students were scoring quite high on our most reliable tests. Molly Ivins may not have understood this when she wrote the July 2000 column from which Collins excerpted that wisecrack—a column in which Ivins made grossly misleading factual claims about the Texas schools. (More to come later this week.)

Ivins may not have understood the true state of the NAEP data. But twelve years after that misleading column, we will guess that Collins still doesn’t know the basic facts about the performance of students in Texas. We’ll guess she doesn’t know about those data from 2011, or she wouldn’t have written the following gloomy passage in 2012.

For now, we’re omitting a fourth, very gloomy paragraph:
COLLINS (page 91): You may be wondering how things are going, education-wise, in the state that deeded its reform plan to the nation. Paul Sadler, the Democrat who led the effort in the Texas house, complains that the state is “testing our kids to death.” (Under the state’s newest regimen, students take seventeen high-stakes tests between third and eighth grade, and up to a dozen more while they’re in high school.) A survey by the Texas State Teachers Association showed that 43 percent of its members were seriously thinking of looking for another line of work.

Sadler still believes that the leaps made in the 1990s are holding up. “I think most of our schools do a pretty good job,” he said. Other observers are, at a minimum, disillusioned. “The eighth grade reading scores were exactly the same in 2009 as 1998,” said Diane Ravitch, referring to the scores Texas received in the national NAEP test results. (I know we were trying to avoid them, but sometimes it’s impossible.) “The whole country is now embarked on remedies that didn’t do anything for Texas.”

David Grissmer, the author of that glowing RAND study, says that since 2000, when the study came out, Texas students’ scores on national tests have begun to “flag.”
We’re still withholding the final part of this passage from Collins’ book (see below). But consider:

Ravitch’s statement is wrong and/or grossly misleading in various ways, as we noted last week. We’ll guess that Grissmer’s outlook and meaning may have poorly conveyed in that one-word “quotation.”

But one year after Texas students led the nation in eighth-grade math, the optimist in this gloomy bunch could only manage to say that the most Texas schools “do a pretty good job.” In the process, an utterly bogus impression was advanced—the impression that test scores in Texas have shown no progress since 1998.

That impression is grossly inaccurate. False.

We’re still withholding the final part of that passage from Collins’ book. In that passage, Collins quotes 87-year-old Barbara Bush sounding off about how bad the Texas schools are. However well-meaning Mrs. Bush may be, she isn’t a reliable source on that topic—and she proved it in the things she said. But because her comments made schools in Texas sound very bad, Collins happily typed them on up!

More on Bush’s comments later this week. For today, ponder this question:

From the passage we’ve shown you, would anyone dream that Texas students, just last year, topped the nation in fourth-grade math? That they persistently outscored their peers from around the nation in both reading and math?

Texas students scored quite high in the 2011 NAEP—and Collins has said that these are the nation’s most reliable data. A reader of her inexcusable book would have no idea that any such thing occurred.

Therein lies a set of tales about modern “journalistic” culture. These tales involve the workings of modern pseudo-journalism—and the conduct of the modern pseudo-liberal world.

Today and tomorrow, we will be in a top-secret location in Shepherdstown, West Virginia, conducting a top-secret workshop for a group of federal managers.

We don’t expect to post tomorrow. We’ll resume this series on Wednesday, and we’ll keep asking these questions:

Does anyone care about public schools? Or do minority children exist to fuel the wisecracks of slackers like Collins? To help them tell preferred partisan stories? (The state of Texas totally sucks! Those rednecks are no damn good!)

Once again, here are some scores from last year’s NAEP:
Texas students, eighth-grade math, 2011 NAEP
White kids: Third in the nation
Black kids: Second in the nation
Hispanic kids: Second in the nation
Reading Collins, would anyone dream that such test scores exist?

More questions: Why did Collins write such a book? In this information age, why are we being misled and misinformed in such relentless ways?

Starting Wednesday: The disaggregation monologues

For seekers of actual information: To compare Texas students to those of other states, just click here. Then click on “State Comparisons.” http://nces.ed.gov/nationsreportcard/about/naeptools.asp

At that point, you’re on your own.

32 comments:

  1. Not sure where you are getting your data to claim Ravitch is wrong, but NAEP 8th Grade Reading scores (from the horses mouth) for Texas (accommodations permitted) are: 1998-261, 2009-260, 2011-261. That 261 is below the national average of 264. Texas is also one of the few states not to gain between 1998 and 2011, and the national average gain was three points.

    4th Grade Reading scores are no better for Texas: a gain in four points between 1998 and 2011 (from 214 to 218), but the nation as a whole gained seven, from 213 to 220 -- placing Texas well down in the bottom half of states in 4th Grade Reading.

    Math may be another story, but the watchword here, where one of the bete noirs is being attacked, would seem to be "Trust, perhaps, but in any case Verify." When time permits, the Math scores need to be checked out, too.

    ReplyDelete
    Replies
    1. What part of "disaggregate" don't you understand?

      Texas school demographics changed dramatically between these years, compared to the national average demographics.

      How did the TX school system perform relative to an influx of non-English speaking and lower-income students?

      Disaggregation!

      Compare the scores of TX non-white kids 1998-2009-2011 to national averages.

      Then compare scores for TX white kids to national averages for those same periods.

      Your aggregate scores are very unenlightening.

      Intentionally?

      Delete
    2. Just for a flavor of what you will learn:

      The 1998 8th Grade Reading scores for TX had "all students" at 261 -- exactly the same as the national "all students" average.

      But within that average, TX Hispanic students outperformed Hispanic students nationally 250 vs. 241.

      And TX Black students outperformed Black students nationally 246 vs. 242.

      And TX White students also outperformed White students nationally 271 vs. 268.

      Yes, *EVERY* TX subgroup outperformed its national counterpart in 1998 8th Grade Reading -- even though the "all students" results were the same!

      Demographics!

      By 2009, TX 8th Grade Reading scores for "all students" had stagnated or slightly declined from 1998's 261 to down to 260.

      But this "all students" aggregation once again obscures the fact that *EVERY* subgroup in TX, Hispanic, Black and White, ALL improved!!

      Demographics!!

      And in 2011??

      "urban legend" tells us: "Texas is also one of the few states not to gain between 1998 and 2011, and the national average gain was three points."

      But urban legend isn't thinking very well!

      Demographics! Disaggregation!!!

      YES!

      Once again, *EVERY* subgroup in TX outperformed its national counterpart. And *EVERY* subgroup gained versus prior-years' scores.

      The TX black kids? They keep on outperforming their national counterparts!

      The TX hispanic kids? They keep on outperforming their national counterparts!

      And the TX white kids? They too keep on outperforming their national counterparts!

      EVERY subgroup in TX outperforming the national average.

      So, yeah -- what "urban legend" and Diane Ravitch and Gail Collins think they know... is just flat wrong.

      Delete
    3. The answer to this puzzle (and the reason you keep hearing "disaggregate") is here. The closest example is the Berkely Gender bias case.


      http://en.wikipedia.org/wiki/Simpson's_paradox

      Delete
  2. Not to mention, as I think Bob himself has pointed out, something happens to kids between those 4th and 8th grade scores and graduating high school, and a great deal of that promise evaporates.

    What's the Texas rate of high school graduation and college attendance? I have no idea, but seems to me that's a better gauge.

    For those who do care, Frontline has what sounds like an excellent 2-hour program Tuesday night on one low-income Houston high school's intensive efforts to keep kids in school-- and learning.

    Not that Bob would ever mention this because he's so addicted to the false idea that no one besides him cares about this stuff.

    ReplyDelete
    Replies
    1. As in PBS's superb "Frontline" series.

      Delete
    2. "Not to mention" -- as if what "urban legend" stated made any sense...

      But "what's the Texas rate of high school graduation and college attendance?"

      You "have no idea" but you think it would be a better gauge. Good stuff there!

      And you probably wouldn't care to try figuring out the impact of demographics either, huh?

      Delete
  3. The 8th Grade Math does, indeed, seem to be a case of extraordinary gains for Texas -- beyond the extraordinary (and unpublicized, as astutely and uniquely emphasized on this site) gains in math for the country as a whole and southern states in particular -- with the state moving from well below average in 1990 into 10th place among all the states by 2011.

    But one would be wise to avoid going out on a limb with strong declarations. Those 8th Grade Math scores are outliers for Texas: 4th Grade Math is dead in the middle among all the states, with gains just slightly ahead of the national improvement in 4th Grade Math; both 4th and 8th Grade Reading scores are will below average; and with all their faults, the Texas SAT and ACT, with account taken for the percentages taking those tests respectively, come closer to matching those other tests in state rank (and the state's demographics) than the 8th Grade Math outlier.

    NAEP involves a complex sampling process of schools and students. Not all students are tested. While it is the "gold standard" in an extremely complex world of testing, it should not be considered beyond the realm of possibility that a segment of the national results in a particular test would not be as reliable as the whole.

    ReplyDelete
    Replies
    1. That is a fair criticism. I leaped without carefully reading the entire post. That being acknowledged, there are anomalies in the entire range of data -- both grades of math, both grades of reading -- that need careful analysis. Texas is undergoing demographic changes that would make the whole score seem inconsistent with its parts, but so are a number of other states experiencing such changes. I'm not sure where to find the data for comparing those demographic changes state-by-state.

      Delete
    2. The "whole" score is *entirely* consistent with its parts (the scores of the sub-groups)!

      You might do well to follow the link provided by a different Anonymous above at Wikipedia for "Simpson's paradox."

      The facts are just as they are, however: ALL of Texas subgroups of kids (white kids, hispanic kids, black kids) -- ALL of them -- have been improving and have been outscoring the national averages.

      That TX overall score is lower is really no "paradox" however.

      It's just a bone-simple fact that TX has more minority and low-income kids than average.

      It's not those kids fault that they score lower than white and higher-income kids -- that's true across the whole nation. But it does mean TX "average" score is lower.

      Fools like Collins can try to use that to portray TX schools as inferior. But you'd be foolish to follow along.

      The average white kid in TX? She's outscoring the average white kid in the nation.

      The average black kid in TX? She's outscoring the average black kid in the nation.

      The average hispanic kid in TX? He's outscoring the average hispanic kid in the nation.

      For Collins to complain that TX overall scores are lower is to show how little you she cares about these kids.

      She punishes TX for HAVING TOO MANY POOR AND MINORITY CHILDREN, while she ignores how much better TX is doing with those kids than the nation is doing on average.

      Will you really join her in this uncaring stupidity?

      Delete
  4. Shorter People Taking Issue With This Column:

    We don't understand statistics!!!!!

    ReplyDelete
  5. Thank you for mentioning the Frontline episode. It should be interesting.

    Keep in mind that George Bush's "Texas Miracle" of "zero" dropouts took place at Sharpstown--setting for the Frontline episode-- and a small number of other Houston Independent School District schools, and much of the credit went to Bush's education secretary, Rod Paige (who had been HISD superintendent during most of the '90s). Talk about media swallowing a lie whole!

    Our analysts hope that this time Sharpstown's favorable publicity is well deserved. As for dropout statistics, even Anonymous should concede that they're highly, um, fluid.

    ReplyDelete
  6. Thank you for mentioning the Frontline episode. It should be interesting.

    Keep in mind that George Bush's "Texas Miracle" of "zero" dropouts took place at Sharpstown--setting for the Frontline episode-- and a small number of other Houston Independent School District schools, and much of the credit went to Bush's education secretary, Rod Paige (who had been HISD superintendent during most of the '90s). Talk about media swallowing a lie whole!

    Our analysts hope that this time Sharpstown's favorable publicity is well deserved. As for dropout statistics, even Anonymous should concede that they're highly, um, fluid.

    ReplyDelete
  7. But Bob, I LIKE feeling superior to Texas! You're harshing my mellow here, man, is all I can say.

    ReplyDelete
  8. the daily howler

    musings on the arcane and my grudges

    ReplyDelete
  9. I like what you guys tend to be up too. This kind of clever work and
    exposure! Keep up the terrific works guys I've included you guys to my personal blogroll.
    Feel free to surf my weblog ; best bonus casino

    ReplyDelete
  10. Hello, just wanted to mention, I liked this article.
    It was funny. Keep on posting!
    Also visit my webpage ; best casino bonuses no deposit

    ReplyDelete
  11. Useful information. Fortunate me I discovered your site accidentally, and
    I am stunned why this coincidence did not took place
    in advance! I bookmarked it.
    Also visit my web site - best online casino bonuses usa

    ReplyDelete
  12. When some one searches for his required thing, therefore he/she needs to be available that in detail, so that thing is maintained over here.


    my web site - netent casinos no deposit

    ReplyDelete
  13. I'm really loving the theme/design of your blog. Do you ever run into any web browser compatibility issues? A few of my blog visitors have complained about my website not working correctly in Explorer but looks great in Firefox. Do you have any recommendations to help fix this problem?

    Feel free to visit my page :: Brownbrian76.Soup.io
    My web page: netent casinos

    ReplyDelete
  14. Hello, I want to subscribe for this web site to take newest updates,
    thus where can i do it please help out.

    Also visit my blog post ... netent casino bonuses

    ReplyDelete
  15. I've been exploring for a bit for any high-quality articles or blog posts in this sort of area . Exploring in Yahoo I eventually stumbled upon this website. Reading this information So i am satisfied to express that I've a very just
    right uncanny feeling I came upon exactly what I needed.
    I such a lot certainly will make sure to do not forget this
    website and give it a glance regularly.

    my web-site :: casino netent online

    ReplyDelete
  16. This article will help the internet visitors for creating new blog or even a blog from start to end.


    Stop by my website; netent casinos list
    Also see my page - netent casino

    ReplyDelete
  17. fantastic points altogether, you just gained a logo new reader.
    What would you recommend in regards to your submit that you simply made some days ago?

    Any sure?

    Here is my page - Best netent Casinos

    ReplyDelete
  18. you are really a good webmaster. The site loading pace is amazing.
    It kind of feels that you are doing any distinctive trick.
    Furthermore, The contents are masterwork. you've done a excellent activity on this subject!

    my page: writerscafe.org

    ReplyDelete
  19. Way cool! Some very valid points! I appreciate you
    penning this post and also the rest of the site is also very good.


    Here is my weblog netent casino software

    ReplyDelete
  20. Hi there i am kavin, its my first occasion to commenting
    anyplace, when i read this piece of writing i thought i could also create comment
    due to this brilliant paragraph.

    my web-site - perchvalley25.postbit.com

    ReplyDelete
  21. Hi there colleagues, how is the whole thing, and what you wish for to say on the topic of this paragraph, in my view its truly awesome in favor of me.


    My blog - net entertainment casinos free spins

    ReplyDelete
  22. Informative article, totally what I was looking for.


    Check out my weblog ... casino netent paypal

    ReplyDelete
  23. Hi! Ι јust wаntеd tο asκ іf уou
    evег hаνe аny troublе with hасkеrs?
    Μy laѕt blog (worԁpгess) waѕ hackeԁ аnd Ι endeԁ up losing a feω months of hаrd work due to nο bаckup.
    Dο you haѵе any mеthods to stop hасkers?


    my wеbpage; Blog Network

    ReplyDelete
  24. Simply wаnt to say уοur
    агticle is aѕ аstоnіshing.
    The clагіty in your post is just cool and i сould аssumе уou're an expert on this subject. Fine with your permission let me to grab your feed to keep updated with forthcoming post. Thanks a million and please keep up the gratifying work.

    Here is my web blog: vida vacations

    ReplyDelete
  25. I wanted to thаnk you fοr this veгy good гead!

    ! I certainly loѵed every bit of it. I have yоu book marked tο looκ at nеw things yοu
    post…

    Cheсk out my wеb ѕite :: link building

    ReplyDelete