The Value of Open Review
Ethan Allen at ATTP points to responses that James Hansen has written on the Atmospheric Physics and Chemistry Paper newly accepted, including a twenty page version for those bunnies who busy hiding the chocolate eggs tonight (and a fine and well deserved Easter morning to all :) and a comment on why really really really dangerous was a proper description of the path we are on and how the reticence of scientists is at least partially to blame.
Eli would like to take the opportunity to discuss open review, which, as a consequence of the Hansen paper, has come in for a bit of pinata play.
To Eli, the virtue of the EGU open review is the information it provides the casual reader. To move a bit away from Hansen, et al, remember “Where do winds come from? A new theory on how water vapor condensation influences atmospheric pressure and dynamics”. by Makarieva et al. It was, IEHO, not even wrong, but Makarieva, who was then the random Russian of the week at a number of blogs on the dark streets of the internet, persevered. Oh my, did she. And the truth is that such behavior works, and today with journals breeding like, please forgive the Bunny, bunnies, it is easier than ever.
David Schultz has a good summary of that soap opera pointing to a much earlier quote from G. K. Batchelor
Papers of poor quality do more than waste printing and publishing resources; they mislead and confuse inexperienced readers, they waste and distract the attention of experienced scientists, and by their existence they lead future authors to be content with second-rate work.But, aha, if Eli can see the open review comments, the Bunny can be both warned and educated, even if the paper is not wild.
For good papers, the referees comments function as an extended abstract, outlining the contributions of the paper, the caveats of the referees and more.
Open review has much to recommend it.
Posts
12 comments:
I sort of totally munged up a subsequent post over there, so here it is again with correct spelling (and correct CR's) with two additional links ...
Willard, I left two out …
http://www.columbia.edu/~jeh1/mailings/2015/20150727_SeaLevelDisaster.pdf
http://www.columbia.edu/~jeh1/mailings/2015/20150727_SeaLevelPost.pdf
“Why 200 years? For one thing, 100 years would require taking on the formidable IPCC, which estimates that even the huge climate forcing for a hypothetical 936 ppm CO2 in 2100 would yield less than one meter sea level rise. For another thing, incentives for scientists strongly favor conservative statements and militate against any “alarmist” conclusion; this is the “reticence” phenomenon that infects the sea level rise issue.”
… editorializing duck …
“Yikes! It has been pointed out to me that the specificity of 200-900 years in my post about ice sheet time scales has the potential to be very unfair to specific individuals.”
… you can say that one again …
“The time scales that he obtains come out of the modeling, not from pressure to avoid the uncomfortable 100 year time scale that policymakers consider relevant.”
… quack quack …
“I should have stuck to discussion of the amplifying feedbacks that we identified and their potential to reduce the time scale for large sea level rise.”
If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.
https://en.wikipedia.org/wiki/Duck_test
Ah, the good old days, G.K. Batchelor's An Introduction to Fluid Dynamics, Lamb's Hydrodynamics and Schlichting's Boundary Layer Theory and Philip L-F. Liu's attempts to teach us in such matters ...
http://www.cee.cornell.edu/people/profile.cfm?netid=pll3
The Hansen paper and the EGU open review process was quite a revelation to me in at least one respect - I didn't realize how many academics simply do not want to argue/fight/disagree without the cover of anonymity.
I contacted several researchers who had written papers making strong arguments contrary to some of the conclusions in Hansen et al. In private email they simply did not want to get into a public fight - regardless how strong their views. And in at least one instance the views were very, very strong.
I understand personalities can clash and not everyone will get along, but I was chagrined to realize how much it can distort the discussion of the science itself. Some people are regarded as obnoxious 'bullies' and are willing to personalize criticism of their scientific work. Personally, I'm not one to run away from a fight - but not everyone has the same disposition.
I think Kevin highlights one problem. However, even if open review became the norm, you'd still have poor papers at weak journals making it through because credible reviewers wouldn't have the time to be involved in everything. It's not clear that it would solve the issue you highlight.
Christ is risen, y'all! Happy Easter!
I was thinking of publishing a paper on the possible uses of phlogiston in promoting alchemical transformations. Can you recommend a journal?
But surely
https://quantpalaeo.wordpress.com/2016/03/08/the-peer-review-of-ollila-2016/
Eli might even point you to a comment of his which you noticed
http://rabett.blogspot.com/2006/10/amateur-night.html
What amateurs lack as a group is perspective, an understanding of how everything fits together and a sense of proportion. Graduate training is designed to pass lore from advisors to students. You learn much about things that didn't work and therefore were never published [hey Prof. I have a great idea!...Well actually son, we did that back in 06 and wasted two years on it], whose papers to trust, and which to be suspicious of [Hey Prof. here's a great new paper!... Son, don't trust that clown.] In short the kind of local knowledge that allows one to cut through the published literature thicket.
Snicker. From the Makarieva blog page:
"Douglas in Australia says:
Monday, 25 February 2013 at 03:46
I am an author. Thanks for the interest. Allow me to offer some clarifications:
1) One referee Dr Judith Curry accepted that the mechanism we describe is real as did several commentators....."
Yup.
> I contacted several researchers who had written papers
> making strong arguments contrary to some of the conclusions
> in Hansen et al. In private email they simply did not want
> to get into a public fight - regardless how strong their
> views. And in at least one instance the views were very,
> very strong.
> ...
> I was chagrined to realize how much it can distort the
> discussion of the science itself. Some people are regarded
> as obnoxious 'bullies' and are willing to personalize
> criticism of their scientific work....
This would be a worthwhile paper in the sociology of science, if someone knows a kid looking for a PhD topic. This one particular paper at this particular time and the people Kevin contacted, to start with, could yield a fascinating data set. And/or some reputable-enough group could solicit anonymous reviews, even after the fact, guaranteeing anonymity, and then do a comparison of those versus the open reviews.
That, there, could be fascinating -- and humbling.
Those familiar with academia know that one bad comment can jeapordize a career. But this is the season for documenting bad behavior by senior academics -- the sexual harassment is beginning to be documented. The academic harassment, too, could be sorted out.
Y'all know the rather famous Peter Watts piece about this:
------------------excerpt follows-----------------------
... Science doesn’t work despite scientists being asses. Science works, to at least some extent, because scientists are asses. Bickering and backstabbing are essential elements of the process. Haven’t any of these guys ever heard of “peer review”? . . .
The fact is, we are all humans; and humans come with dogma as standard equipment. . . The best we can do— the best science can do— is make sure that at least, we get to choose among competing biases.
That’s how science works. It’s not a hippie love-in; it’s rugby. Every time you put out a paper, the guy you pissed off at last year’s Houston conference is gonna be laying in wait. Every time you think you’ve made a breakthrough, that asshole supervisor who told you you needed more data will be standing ready to shoot it down. . . .
This is how it works: you put your model out there in the coliseum, and a bunch of guys in white coats kick the shit out of it. If it’s still alive when the dust clears, your brainchild receives conditional acceptance. It does not get rejected. This time. . . .
Science is so powerful that it drags us kicking and screaming towards the truth despite our best efforts to avoid it. And it does that at least partly fueled by our pettiness and our rivalries. . .
Keep that in mind ....
http://www.rifters.com/crawl/?p=886
-------------end excerpt----------------
Yeah, it'd be good to get _past_ this condition and do the research and find out -- now that we can -- just how much good solid criticism of published papers is being suppressed by silverback apeshit.
C'mon, graduate students. Who's up for this? Follow the methodology established for querying climate scientists about their opinions.
Oh, and make sure to plan your statistical analysis _first_.
Eli:
"What amateurs lack as a group is perspective, an understanding of how everything fits together and a sense of proportion...In short the kind of local knowledge that allows one to cut through the published literature thicket."
Some of that is what John Nielsen-Gammon calls scientific meta-literacy:
"We scientists rely upon a hierarchy of reliability. We know that a talking head is less reliable than a press release. We know that a press release is less reliable than a paper. We know that an ordinary peer-reviewed paper is less reliable than a review article. And so on, all the way up to a National Academy report. If we’re equipped with knowledge of this hierarchy of reliability, we can generally do a good job navigating through an unfamiliar field, even if we have very little prior technical knowledge in that field."
If there was a way to impart just that much in high-school or early-college science classes, science deniers would get less traction.
> several researchers who had written papers making strong
> arguments contrary to some of the conclusions in Hansen
> et al. In private email they simply did not want to get
> into a public fight ....
Um, so, we need a journal for anonymized _authors_ then?
There's ample historical precedent for that.
Much more than I realized before I 'oogled:
https://www.google.com/search?q=anonymity+author+science+paper
Before making a comment on the Hansen et al. paper a scientist would want to read it. Which takes a lot of time given its length.
Thus there are downsides (time and conflict) and there is no upside to a public fight.
These scientists may well respond later with a scientific article. A good scientific article has upsides, may improve insight and enhances your reputation.
Thus I do not expect that an anonymous author journal would help, especially in times of publish or perish. It is good to have the option, but I do not expect that it would be used much.
Post a Comment