Shields responds - weakly - to criticism of "political profiling" study

Professor Donald Shields himself has stopped by to defend his and John Cragan's study on "political profiling" by the Bush Justice Department, in which they claim that the Department of Justice, for partisan reasons, has targeted primarily Democratic local elected officials for criminal investigations. Sadly, his comment, STUDY OF U.S. ATTORNEYS’ POLITICAL PROFILING IS SWIFT-BOATED, is no better than the paper itself. See below for a listing of my previous postings on this topic.

As with the inital paper, the opening paragraph (initially written as a letter to the Philidelphia Inquirer responding to the Smerconish article) includes a massive, gratuitous, ad hominem swipe at all conservatives:

Mr. Smerconish chose to swift-boat us and our research in the tradition of the attacks made on Senator Kerry and Congressman Murtha’s war records in 2004, not to mention Rush Limbaugh’s ridiculing Michael J. Fox’s Parkinson tremors.

I'll let my co-bloggers and our commenters have some fun debunking that little bit of conflation (I seem to recall the President saying that John Murtha was a real war hero, but just wrong on Iraq, but maybe that's just me). I'll stick with pointing out the continued failing of Professors Shields and Cragan to understand or acknowledge sampling methods, biases and other useful trivia associtated with actual scientific studies. Before we go below the fold, let me highlight the silliest part of his response, just because I find it so amusing.

Bootstrapping appeal to authority is funny

The best of his response is a ridiculous boot-strapping argument in which Shields appeals to authority because someone in authority cited his paper as proof... of things it didn't study:

For example, Senator Leahy cited our research on the floor of the Senate to prove that the Bush Administration has politicized the U.S. Attorney’s Office, secretively changed the law requiring Senate approval, and fired at least eight US attorneys.
(http://www.govtrack.us/congress/record.xpd?id=110-s20070215-19&person=300087).

It's nice that we now rely on Senators to validate academic studies. Will that work with global warming? I guess it is now proven, thanks to Senator Inhofe and his citations to papers outside of the "consensus," that there is no anthropogenic global warming.

But that's just the funny part. The other flaws in his response and original study are far more significant. Read more to get to the serious stuff.

UPDATE: Welcome, Just One Minute readers. Thanks for stopping by, and I hope you'll check out other areas of our site while you're here.

In the comments, Maybee notes that E Pluribus Unum began life as a Daily Kos spinoff and is NOT an actual academic journal. Regular commenter Rich Horton has a suicidal bent, as he says he's going to hold his breath until the MSM cites an LGF post as an academic study.

END UPDATE

Fact-checking is not peer review

Shields attacks Smerconish first by claiming that Smerconish misled the public by failing to acknowledege that the study was "published."

He lamented that the study was not published, while failing to mention that we informed him during a phone interview that the study was going to be published by the electronic media journal, (Epluribus Media). [edited for formatting, to provide proper hyperlink usage]

Professors Shields goes on to defend his and Cragan's work as having been vigorously fact-checeked, both by themselves and by the editors at epluribusmedia.org:

This article contains the complete raw data tables, naming each of the 375 politicians classified by their states, their political affiliations, the type of investigation, and the day a news story appeared locally describing the event. Because there is no national register of these investigations, it took more than 400 hours to find and organize this data (our results were fact-checked by multiple editors of Epuribus Media before they published it).

I don't doubt that each of the 375 individuals who were named in the data tables provided were in fact named as being "under investigation" by some reporter somewhere. The issue has never been the factual accuracy of the cited data, but the completeness of that data. The study is flawed because of what it may have omitted, not it included. While the data was, I'm sure, fact-checked as described, E Pluribus Media does not appear to be a peer-reviewed publication. Their page "What We Are" describes the verification work done by volunteer researchers and editors. Those services would best be called fact-checking, not peer review. A peer review does not involve fact-checking (journal editors do that) so much as it involves reviewing the design of the study. Fact-checking is relatively easy. One needn't have any background in the field to look up the cited data and verify that it exists. But it takes a more skeptical eye to ask larger questions about the study.

Methodology should be disclosed in the paper itself for proper peer review

Professor Shields goes on to remind us that the paper was presented at a peer-reviewed panel in Boston in 2005. One presumes that the peer-reviewers in Boston were going by the same study, essentially, that we read today at the E Pluribus Unum site, since Professor Shields gives that web address for the published version of the study in his response. How such a "study" could pass a real peer review process is beyond me, given its many design flaws, which I have already copiously documented.

Professor Shields still does not provide us with the search criteria he and Cragan used to identify the 375 "investigations" he included in the raw data, or the definition of "investigation" which they chose to use.

Did Bush fire the worst offenders?

Professor Shields' response does address the humorous observation I made in Did Bush Fire Some of the Worst Profilers. The raw data tables provided by Shields and Cragan do not have a column showing the district within each state that each investigation was being conducted. Because the city (or county) and state were provided, the district could be determined from the data provided, but I didn't do it, because it would have been too time-consuming. At any rate, Shields claims that 7 of the 8 removed U.S. Attorneys were below the national average for "investigating and indicting" Democrats.

I'm not willing to take Shields' word for that, though I will remind readers of the tongue-in-cheek nature of the suggestion. First, he repeats an earlier error of making a claim that he did not study. He refers to "investigating and indicting" Democrats. His study claims only to look at "investigations." While I think it would be much better and more objective to study indictments rather than investigations, Shields has no more basis to refer to compare indictments than he does to compare the behavior of different administrations. He can't make claims on subjects he did not study. Second, of course, his statistic about these U.S. Attorneys being below the national average - in Shields' study - for investigations of Democrats has no more credibility than the study itself... which is to say, none. Finally, he repeats the slanderous suggestion that Carol Lam was removed from office because she investigated a Republican, Duke Cunningham. Patterico has amply demonstrated that that's not true; Lam was one of 3 or 4 names on a list for replacement before the Cunningham investigation ever began. (She also wasn't removed over the Congressman Jerry Lewis investigation, either, so let's squelsh that now.) Quite simply, Professor Shields has exhausted his credibility with me.

Shields' claimed remedy would be bad on so many levels

Let me turn finally to Professor Shields' reassertion of his claimed remedy, a national, public database of "investigations" by U.S. Attorneys. I explained in my earlier posts why this is a bad idea. From my own experience as a prosecutor of politicians, I know how quick the news media is to leap to conclusions which will reveal a scandal (helping them sell papers). A reporter once told me that he knew for a fact that the grand jury I was working with had indicted two high-ranking state officials, under seal. It wasn't true, and we in fact never did indict those two men. We were investigating them, but we never gathered sufficient evidence to prosecute them. It is unfair to publicly tar someone as being "under investigation" if the investigation goes nowhere. Remember Richard Jewel, anyone? It is also profoundly stupid, because it would tell a lot of bad guys that they are being scrutinized, giving them a chance to move their bribe-money out of the freezer and into a safer hiding place.

But let's look at a more contemporary example which reveals just how flawed Shields' and Cragan's obsession with "investigations" is. As Patterico reports, critics of the Administration are claiming that U.S. Attorney Bud Cummins was among the removed because of his "investigation" into Missouri Governor Matt Blunt. The LA Times reported that Cummins himself was wondering about this possibility.

However Cummins (who has made clear his distate for his firing and expressed no love for the Bush administration) says that not only was he misquoted, but the investigation wasn't even about Gov. Blunt:

I do not know of ANY connection between the Missouri investigation (which actually had nothing to do with Governor Blunt) and my termination. [emphasis added]

Even Shields and Cragan don't list Gov. Blunt as having been investigated.

An appeal to Shields and Cragan to answer basic questions

Professor Shields or Professor Cragan, if you're reading this, please answer 4 questions:

• What search criteria did you use to find the 375 "investigations" used in your study?
• What databases did you search in?
• How did you define "investigation" for purposes of the study?
• Do you claim, or did you seek by your methodology, to have included all investigations across the country into local elected officials and candidates, or only a random sample of such investigations? If the latter, what was your random selection process?

I've already beaten this horse to death. Professor Shields' response, like most of his defenders, have focused on trivial matters and partisan ad hominems rather than providing further explanation of his methodology, which would address the serious issues I and other raised with their "study." He's been at this site, he's seen the issues I've raised. I can only conclude that his failure to address them is proof that he is not able to, that my substantive critique is correct.

Previously on this topic:

• "Political Profiling" Study is Fatally Flawed.
• More Debunking of the "Political Profiling" Story
• Did Bush Fire Some of the Worst Profilers

On related topics:

• Is It Legal? Yes.
• Some Evidence on the "Gonzales 8"
• Appointment Terms for U.S. Attorneys