Telephone Churnalism

In a comment appearing in the American Scientist, Andrew Gelman and Kaiser Fung take the Steves, Levitt and Dubner to task for some very sloppy work

It’s hard to be sure what process an author uses. But by appearances, the way the authors of the Freakonomics series make their work is too linear to provide adequate vetting of research. In SuperFreakonomics, for instance, economist Steven Levitt trusts authors of primary research whom he knows or respects. Journalist Stephen J. Dubner trusts Levitt’s assessment of their work, and together they create narratives about it. The book’s editors seem by and large to have trusted the authors’ account, delivering it to readers who place trust in the Freakonomics brand. Although there may be more opportunities for feedback along the way than outsiders can discern, the problems and errors encountered in the authors’ work suggest that there is room for improvement.

Bunnies may recall the Freakazoid Franchise's encounter with Ray Pierrehumbert, and Gelman is exactly right, that people who are not expert in an area rely on one who knows, even a trusted one who knows, at their own peril.

The climate-change chapter in SuperFreakonomics is a case in point. In it, Levitt and Dubner throw their weight behind geoengineering, a climate-remediation concept championed at the time by Nathan Myhrvold, a billionaire and former chief technology officer of Microsoft. Unfortunately, having moved outside the comfort zone of his own research, Levitt is in no better a position to evaluate Myhrvold’s proposal than we are.

When an actual expert, University of Chicago climate scientist Raymond Pierrehumbert, questioned the claims in Levitt and Dubner’s writing on climate, Levitt retorted that he enjoyed Pierrehumbert’s “intentional misreading” of the chapter. Referring to his own writings on the subject, Levitt wrote, “I’m not sure why that is blasphemy.” We’re not sure on this point either—we could not find a place where Pierrehumbert described Levitt’s writings in those terms. It is easy to be preemptively defensive of one’s own work, or of researchers whose work one has covered. Viewing alternative points of view as useful rather than threatening can help take the sting out of critiques. And if you’re covering subject matter outside your expertise, it pays to get second—and third and fourth—opinions.

Still Gelman and Fung miss the Churnalism driver. Leveraging their initial best-seller, the Steves created a franchise, and like Mickey D's, franchises require tons of raw meat, though quality is not necessary Any cow to be slaughtered is welcome and they have to grind up a lot of it. Careful testing and research are optional in churnalism.

The interested can read more at Gelman's blog, Statistical Modeling, Causal Interference and Social Science