Saturday, January 05, 2008

The whispers of David Kane

Our friend David Kane and Iraq Body Count have been busily whispering into the ears of Neil Munro and Carl Cannon at National Journal, the result being a long article now spreading through right wing blogs where Eli picked it up. There isn't much new from David

"The authors refuse to provide anyone with the underlying data," said David Kane, a statistician and a fellow at the Institute for Quantitative Social Statistics at Harvard University.
And we see a lot of the arguments that have been knocked down by various folks at Deltoid (Tim, Robert D^2). But the National Journal walks right up to and probably beyond the line accusing Roberts and Co. of fraud
Some critics have wondered whether the Iraqi researchers engaged in a practice known as "curb-stoning," sitting on a curb and filling out the forms to reach a desired result. Another possibility is that the teams went primarily into neighborhoods controlled by anti-American militias and were steered to homes that would provide information about the "crimes" committed by the Americans.
And, of course, calling for an audit.

Surprisingly, not one of the peer reviewers seems to have thought to ask a basic question: Are the data in the two studies even true? The possibility of fakery, editor Horton told NJ, "did not come up in peer review." Medical journals can't afford to repeat every scientific study, he said, because "if for every paper we published we had to think, 'Is this fraud?' ... honestly, we would fold tomorrow."

In Belgium, Guha-Sapir's team is completing a paper outlining numerous mathematical and procedural errors in the Lancet II article, and its corrections will likely lower the estimate of dead Iraqis to 450,000, even without consideration of possible fraud during the surveying, a source said.

Perhaps medical journals, like respected news organizations, will learn that they have to factor the possibility of wartime fraud into their fact-checking. Horton knows the peacetime risks only too well: In a Lancet article in October 2005, exactly halfway between the two Iraq mortality studies, a Norwegian physician named Jon Sudbo wrote that a review of 454 patients showed that such common painkillers as ibuprofen and naproxen reduced smokers' risk of contracting oral cancer while increasing their risk for heart disease; it later turned out that Sudbo had faked his research.

And the Lancet editor tries to make nice which is going to come back and bite him

Today, the journal's editor tacitly concedes discomfort with the Iraqi death estimates. "Anything [the authors] can do to strengthen the credibility of the Lancet paper," Horton told NJ, "would be very welcome." If clear evidence of misconduct is presented to The Lancet, "we would be happy to go ask the authors and the institution for an official inquiry, and we would then abide by the conclusion of that inquiry."

Guha-Sapir's criticisms are that
First, according to Burnham and colleagues' results, there were nearly 600 war deaths per day—an unusually high number compared with almost any other armed conflict or indeed with other Iraqi mortality estimates.2 Burnham and colleagues' figure 4, in which cumulated Iraq Body Count deaths parallel their study's mortality rates, is misleading. Rates cannot be compared with numbers, much less with cumulative numbers. The correct comparison would be the one presented here (figure), in which the Iraq Body Count numbers are transformed into rates by period. In that case, there is no similarity between the trends in the study and Iraq Body Count.

Second, the study suggests that, over a 3-year period, around 90% of the deaths were directly related to violence. However, experience from other conflicts indicates that indirect causes (disease, malnutrition) typically outnumber the deaths due to violence (bombs, gunshots, etc).3 Burnham and colleagues' figure remained high for a long period of time. By comparison, only one of 17 surveys in Darfur reported a similar level of violent deaths, and this level only persisted for 3 months of a 6-month period.4

Third, the heterogeneity of the pattern of violence in Iraq argues for a differentiated estimation across the governorates. Insurgency and coalition action is still concentrated mainly in the Sunni triangle, but large tracts in the rest of the country are relatively peaceful. A better accounting for differences in violence by governorate separately and the effect of excluding the Sunni triangle would have strengthened the study.
more criticisms can be found at the link for the interested.

31 comments:

Marion Delgado said...

I was completely, utterly wrong. I hide my head in embarrassment:

   In a certain sense I am going to say the opposite of what's normally said 'I suspect that's not the last we've heard of ...' no. I think that at least in a sense where he isn't highly suspect, we are seeing the last of Mr. Kane. - Marion Delgado" (over on Deltoid)

I blame irrational exuberence.

Anonymous said...

Kane got spanked pretty good last time at Deltoid.

Robert made him look like a complete idiot.

Unfortunately, there are people in this world (like Kane and Bush) whose ignorance is exceeded only by their arrogance.

bigcitylib said...

"Kane got spanked pretty good last time at Deltoid."

I imagine that's why he didn't ref his own "study" this time out.

Anonymous said...

I'm really surprised Kane's employer (Harvard) has not "Motlized" him by now.

Anonymous said...

"In Belgium, Guha-Sapir's team is completing a paper outlining numerous mathematical and procedural errors in the Lancet II article, and its corrections will likely lower the estimate of dead Iraqis to 450,000"

So, in other words, "roughly half a million" died.

Phew, for a while there I was worried that roughly half a million had died (as of the time of the Lancet report).

Anonymous said...

I just read the Munro piece in NJ and....I have to say that I'm a bit surprised. The story comes off with a slight feel of editorializing. And it seems like Munro tried to raise every possible argument he could to taint the study.

There's just something about the tone of the story that comes off poorly.


Mus musculus anonymouse

Hank Roberts said...

Perhaps the administration was getting too close to some 'just war' limit, and decided to move the foul line? Last I recall, just war theology suggested something not exceeding one civilian death per one hundred soldier deaths at worst was, well, just about right.

Of course they came up with that before gunpowder came into widespread use, when it was a little easier for a soldier to be sure who exactly it was on the other end of his instrument of war.

Anonymous said...

"Perhaps the administration was getting too close to some 'just war' limit, and decided to move the foul line?"

Or perhaps the "just war criminals" limit.

I suspect that after (if?) they leave office, George and Dick will never be able to travel in Europe for fear of being charged with war crimes.

Anonymous said...

So it looks like what people are asking for is an accounting of the data used, along with the process they used to get their results (the code).

"Perhaps ... journals, like respected news organizations, will learn that they have to factor the possibility of ... fraud into their fact-checking."

- Mickey

Anonymous said...

Mickey said "So it looks like what people are asking for is an accounting of the data used, along with the process they used to get their results (the code)."

Legitimate researchers GOT the data AND were informed of the methods of the study. Kane also got the data though he obviously did not know what to do with it it.

People who knew what they were doing (eg, Robert Chung) were able to reproduce the results.

Kane's "Free the Data" campaign rings empty.

Anyone who reads the exchange on Deltoid between Kane, Robert (Chung) and others on Deltoid can see that.

TCO said...

The fundamental thing wrong with that Lancet survey (that makes it a peice of poopoo from the ass) is that it relies on taking undocumented survey commentary and extending it to the nation as a whole. It is butt crappy methodology. Especially when the dishonesty of that population (and participation in the insurgency) is pretty graphically well known. And if you don't believe me. I've got some streets for you to drive your Humvee down.

And any problems with Kane's criticism don't obviate this graphically true (lying Arabs) issue.

The only way to do this survey properly is with ground truth proof on the survey clusters.

Anonymous said...

tco:

You use such colorful "scientific" terminology: "butt crappy methodology", "poopoo from the ass" [Is there another type?].

You must be an oilman like McIntyre.

"any problems with Kane's criticism don't obviate this graphically true (lying Arabs) issue."

Well, one could say that any survey/poll is worthless if people lie.

Unfortunately for your theory, many surveys/polls that involved only a small fraction of the population are a quite accurate measure of the larger population. It is called "statistical sampling". You might look it up (but I doubt you will find it in your Dictionary of Crapolgy)

TCO said...

I have no problem with the sampling versus census.

The concern is primarily related to survey truthfullness (secondarily representativeness of the clusters). Yes, this is a common concern and potential flaw in all polls. All the more reason to watch out for it here. In a situation where insurgents hide in the population, journalists are manipulated, population is terrorized, and a rumor of Brits breeding killer spiders was beleived by the population.

Note that asking housewives which flavor of gum or Democrat they prefer in Ohio is a different situation than assessing mortality in a country where the rule of law has broken down and the population is terrorized and in some cases actively fighting occupation.

The way to examine polls like this where truthfulness is a concern is to audit some of the clusters to observed facts to determine if there is a bias from responder lying. This requires some amount of more detailed investigation of some of the respondents.

Failure to share the primary reports is a real concern here. Note: I understand the issues of safety for respondents...but the bottom line is that some method of sharing the detailed field reports needs to be done to allow others to examine if the field surveys where well done or had systemic issues. Otherwise, it becomes a "trust me, I've got a mouse in my pocket game". And there might even be a mouse in the pocket. That's not the point. Need to check this stuff over. That is the norm in social science.

Ignore the bathroom language please. I'm just a bad person.

I also think there is a substantial difference between Kane writing a white paper on the mathematics, so that that particular point can be examined by his critics/posting it in the lion's den, etc. Versus the McI pattern of posting wandering "usufruct"-filled diatribes in his own controlled forum. At least Kane wrote it down to the point that his idea could be examined and debated clearly and falsified. McI just plays games and doesn't even characterize his criticisms well enough so that someone can examine them properly. He's just too lazy or sloppy or skewed to do so.

stevesadlov said...

Of course your hidden message is, "Boosh bad, Boosh berry, berry bad!"

Ask yourself the question, the fact that you, Hansen, Schmidt et al still have your jobs on the tax payer's blood, sweat and tears, the fact that the NHC still spews utter BS, naming and claiming anything that spins, even if the core is ice cold, the fact that NASA still issue the "surface data" they do, etc ... if "Boosh" was really your enemy, why is all this still in place? (crickets chirping ....)

Anonymous said...

TCO

"Note: I understand the issues of safety for respondents...but the bottom line is that some method of sharing the detailed field reports needs to be done to allow others to examine if the field surveys where well done or had systemic issues."

Systemic issues? You mean like fraud?

Be honest. At least Kane was.

TCO continues: "At least Kane wrote it down to the point that his idea could be examined and debated clearly and falsified."

Wrote down what? The fact that he could not do the required math? (which Robert Chung had to hold his hand and lead him through)

What a joke. If that is what is considered "value added" to the discussion, then social science is in very big trouble.

Marion Delgado said...

stevesadlov and tco provide excellent prima facie evidence that the overlap between the screeching denialist troll community and the "New Fascism" has to be considerable.

TCO said...

anonymice:

Allowing review of the field reports will NOT uncover all problems. Case study level inspection of selected data points is needed. The benefit of reviewing the field reports is it may allow one to look for and possibly detect other systemic issues (interviewer intimidation, main street bias, or something we don't even hypothesize yet). It's not a magic pill, but a normal, good way of going one further in inspecting possible sample method/bias issues. Capisce?

Anonymous said...

TCO

"Case study level inspection of selected data points is needed. "

Wanna volunteer? I'll chip in $10 toward your (one way) ticket to Falluja.


BTW, only an idiot would try to defend Kane's "mathematics", even in a half-hearted way. Read Deltoid and you will understand why.

Capisce?

TCO said...

You're all heart. $10!

Seriously though. This is not a game of "what can be done". But an issue of what can be known. Do you capisce? Oh...and I volunteered for recall for service in Iraq or Afghanistan, but was turned down.

On Kane's math: mebbe, mebbe not. I read that thread and don't have the knowledge to even take a side. I'm not even smart enough to be wrong. But what does that have to do with a discussion of interviewer bias/intimidation?

Oh...btw, you have a lot of anger. Peace...

TCO said...

BTW, I just did a little more reading (but haven't read or reread the article itself...mea culpa...mother fucker). Anhoo: there was an interesting comment that the survy checked certificates on the first couple deaths. this is good practice. Exactly the right attitude. You should have cited that. That is if you were a thinker instead of an advocate.

How would you like to be thrown into a bag with Steve M? Naked of course...

Anonymous said...

Thanks for admitting your complete ignorance of the issue, TCO.

TCO said...

Complete? Relative, mon frere. Relative.

Admit: well, that's the first step to learning, young man. If you don't acknowledge lack of knowledge, than you get done in by unknown unknowns when invading countries.

"the" issue: There are several issues. You are getting confused by trying to wrap everything together. You're as bad as McIntyre, your batter-wrapped naked bag buddy.

Anonymous said...

TCO, you seem to have no problem weighing in on an issue you have no clue about (as per your own admission above)

TCO Post on Deltoid regarding nutty WSJ editorial:

"Editorial looks pretty good to me. Roberts is a politicized scientist."

And your last comment above is just wacky. You seem to have gone right off into never never land.

I suggest you read Eli's post again, because it has nothing whatsoever to do with McIntyre.

TCO said...

1. One should always admit whatever is true. If one is trying to examine things like a scientist. Not like a lawyer. This INCLUDES personal level of knowledge.

2. Level of knoledge is analog, not a two-state digital switch.

3. A point can be correct/incorrect regardless of the level of knowledge of the proponent (of course the likelihood increases with knowledge).

4. Sample methodology (like was it actually possible to safely do the random door knocking that Roberts proposed) is a seperable variable from Kane's concern on the error bars.

5. I compared you to McIntyre because he also confounds seperable variables.

Anonymous said...

TCO says: "One should always admit whatever is true. If one is trying to examine things like a scientist. Not like a lawyer. This INCLUDES personal level of knowledge."

perhaps you should take your own advice.

Having an opinion on something about which on is ignorant is much closer to the way a trial lawyer would behave (adopting a stand at the beginning -- innocence or guilt -- and then accumulating "facts" to support one's case) than it is to the way a scientist would behave.

Admitting that you are ignorant after you have just weighed in on a subject hardly puts you in the "scientist" category.

Honest (and some might say foolish), but certainly not scientific.

PS In the future, you might leave McIntyre out of discussions that have nothing whatsoever to do with him. It adds nothing to the conversation -- and is more than a little bizarre, to say the least when you invoke visions of "batter-wrapped naked bag buddies". Perhaps you have some issues with McIntyre (and with other things as well), but here is not the best place to air them. I don't care to hear about them, at any rate. And neither, I am sure does Eli or anyone else who might post here.

TCO said...

Eli will let me get away with a LOT of shit. Or BCL will. But anyhoo...I'll stop shaking those two voodoo dolls together. Peace?

On the rest of it:
A. What is ignorance? Is it an absolute?

B. Is admitted ignorance, better worse or same as unadmitted ignorance?

C. Is it only admitted ignorants who should not opine or all ignorants?

D. Are you opining on the Lancet study? Where? What is your level of knowledge?

TCO said...

BTW, I agree with the point of BCL's comment althought Kane might phrase it differently...

My opining on the editorial and it's description of Roberts is based on reading the editorial. It's completely separable from the Kane math point and my inability to evaluate those arguments. Since the editorial (as noted) doesn't include that criticism!

Anonymous said...

Of course admitting one's ignorance is better from a scientific standpoint.

But that is not what I took issue with in your case. I took issue with the fact that you were weighing in on an issue despite your professed lack of knowledge on the subject.

Sorry, but unfortunately (for you) you did more than "opine on the editorial" and no amount of hedging or twisting on your part is going to change that.

You questioned the methodology of the Lancet study.


"The fundamental thing wrong with that Lancet survey (that makes it a peice of poopoo from the ass) is that it relies on taking undocumented survey commentary and extending it to the nation as a whole. It is butt crappy methodology."

Or perhaps you are an expert on epidemiological studies?

With regard to D above, I'm not the one who "opined on the lancet study".

I noted that "only an idiot would try to defend Kane's "mathematics", even in a half-hearted way. Read Deltoid and you will understand why."

I based that on a reading of Deltoid, Robert Chung's comment here, for example.

But as you pointed out correctly above, Kane's mathematical competency (eg, calculating stuff like CMR's) is a separate issue from the Lancet study itself.

TCO said...

Nonny:

A. Could you please answer the rest of my questions. I am (trying) to respond to each point of yours. And I promise to answer the rest of your points/questions after you get caught up.

B. You KEEP raising the Kane math issue, but I have not defended that and it is a separate issue from the safety of the surveyors.

Anonymous said...

David Kane is my name and I wrote for the Harvard IQSS
Til Amy Perfors came
And shut me up at their behest.
It was in the Fall of 2006
I was hungry for a media fix.
On the right-wing blogs, my "fraud balloon" did swell
It was a time I remember oh so well.

Chorus
The night they drove old David Kane down
And the bells were ringing
The night they drove old David Kane down
And the people were singing, they went
La la la la la la, la la la la la la la la, la

Anonymous said...

The Night They Drove Old David Kane Down

David Kane is my name and I wrote for the Harvard IQSS
Til Amy Perfors came
And shut me up at their behest.
It was in the Fall of 2006
I was hungry for a media fix.
On the right-wing blogs, my "fraud balloon" did swell
It was a time I remember oh so well.

Chorus
The night they drove old David Kane down
And the bells were ringing
The night they drove old David Kane down
And the people were singing, they went
La la la la la la, la la la la la la la la, la


Back with my wife in Harvard Square when one day she called to me
David, quick come see, there goes Roberts, E. Les
Now I don't mind choppin' wood*
And I don't care if the money's no good
You take what you need and you keep the rest
But they should never have censored Harvard's very best

[chorus]

*which is good, because that might be what Kane ends up doing after all is said and done.