Boiling Bates Down
Allow Eli to simplify the issues about Karl et al. 15 and John Bates
- Bates designed an overly complicated set of procedures for climate data archiving.
- He got upper management at NOAA to sign on because the charts looked pretty.
- There were huge delays in implementation because of software problems and more.
- The process was a huge time sink.
- But it had the virtue of making Bates the Gatekeeper.
- Others were not happy with this.
- They had science they wanted to publish so they found a way around Gatekeeper Bates.
- Gatekeeper Bates went crying to Lamar Smith.
- Trump becomes president
- Denialists need an issue and cast about.
There may also be something beyond simple “engineers vs. scientists” tension behind Bates’ decision to go public with his allegations. Two former NOAA staffers confirmed to Ars that Tom Karl essentially demoted John Bates in 2012, when Karl was Director of NOAA’s National Centers for Environmental Information. Bates had held the title of Supervisory Meteorologist and Chief of the Remote Sensing Applications Division, but Karl removed him from that position partly due to a failure to maintain professionalism with colleagues, assigning him to a position in which he would no longer supervise other staff. It was apparently no secret that the demotion did not sit well with Bates.
Posts
34 comments:
This comment by Tom Peterson seems to indicate something similar; the difference between scientists, who think scientific advancement is key, and software engineers, who think rigorously tested software is more important.
Allow me to simplify the House Science Committee's take:
Climategate 2 Falls at the First Hurdle
"The latest installment of the David & Judy Show has proved to be part of a coordinated attack on NOAA by the House Science Committee."
This seems all too familiar a problem.
> software engineers, who think rigorously tested software is more important
Have you met any software engineers?
We certainly have a different attitude to scientists. We actually like our code and we like to make it nice. But, the competent ones understand the need to have a trade-off between functionality, testing and time-to-market.
It rather sounds like Bates was hiding his incompetence behind a need for perfection.
I used to be an engineering authority for a large multinational. This meant I had to make sure we had senior technical personnel and managers follow standards and regulations. I used to meet a lot of objections by personnel who wanted to cut corners, did shoddy housekeeping, wanted to hurry up, etc. I suspect Karl was like one of the black sheep I had to discipline. So the guy really doesn't get my sympathy. He was in a hurry because he wanted the Karlized data set out before Paris. And that's about it.
It rather sounds like Bates was hiding his incompetence behind a need for perfection.
Okay, I realise that in my attempt to avoid directly insulting Bates, I managed to insult software engineers in general :-)
I am a software engineer William! Firmware too. I let other people build the hardware though.
I am familiar with the syndrome you describe. However before we jump to any hasty conclusions Judy is promising a Thorne rebuttal RSN. See also:
http://GreatWhiteCon.info/2017/02/climategate-2-falls-at-the-first-hurdle/#comment-217770
"I suspect Karl was like one of the black sheep..."
Really? How? In particular point to the bits where Karl et al didn't follow normal high-standard scientific publication procedure.
"So the guy really doesn't get my sympathy."
I doubt that he'll give even a flea's shit.
"He was in a hurry because he wanted the Karlized data set out before Paris."
"Karlized", huh? Are you insinuating that there's something inappropriate about his work? If so, perhaps you'd care to actually explain your imputation.
And what's your evidence that Paris dictated the timetable?
Lamar Smith is just itching for a hot seat and the gallows at his Nuremburg trial. John Bates? Who the hell is he? A NOAA 'auditor'?
We used to make jokes about NOAA 'auditors'.
> I used to be an engineering authority for a large multinational.
I thought you were a pen-pusher in an oil refinery, Fernando.
In its disregard of the known facts of the timeline of the authorship and review of the Science paper, Fernando's own comment is a fine example of shoddy housekeeping and corner-cutting.
Hear hear mag
I was trained mostly as a scientist, but have always worked as a software engineer, and most recently for several years as a core member of a NASA Software Management Office. In over a decade of working on NASA software, I and my teammates always have consciously designed our software design, development, testing, and deployment processes to suit the particular and explicit goals/requirements/desirements of each particular project.
For my desktop software for flight controllers monitoring and managing telemetry from crewed space vehicles (Shuttle, ISS, Orion, etc.), conception to final deployment certified for live use took years, because the risk of loss of human life was required to be kept very low. Despite that long and laborious total process, most of our releases before that final one went through very much abbreviated processes, because the goals of those preliminary releases included getting feedback quickly so we did not waste time on elaborate quality and archiving processes for releases that we were nearly certain would contain bugs that would be found almost immediately by the flight controllers who would try out those releases with fake or old telemetry.
Most scientific projects are like my project's preliminary releases. Part of their quality assurance is publishing the results for other scientists to evaluate and attempt to replicate. Spending a huge amount of time on internal quality processes usually is much less efficient and effective than using that time for public review. Each organization, and ultimately each project team within that organization, must make its own decision about how much process is appropriate. It sounds like Karl et al. made a completely reasonable judgment, whereas Bates never would have survived in my NASA Software Management Office because of his lack of consideration of context.
Hear, hear Tom! Very well put.
Note that the John 'n Judy "Peter Thorne rebuttal" has hit the streets. See
http://greatwhitecon.info/2017/02/climategate-2-falls-at-the-first-hurdle/#comment-217782
Oh, and ex Prof. Judy is "moderating" many of my pithier comments. And other peoples. If I seem somewhat subdued you now know why!
Golly gosh gee whillikers, what timing as our so-called president assigns oil company executives to His cabinet to slow down climate action!
Gee, the same sort of timing as in the past: twice before, similar "-gates" happened just before major, important meetings.
Coinkydink, surely.
Best,
D
Great article describing an explanation by Tom Peterson: https://arstechnica.com/science/2017/02/article-names-whistleblower-who-told-congress-that-noaa-manipulated-data/
Bates's background was software for running spacecraft. That could be either "flight" software embedded in the spacecraft, or "ground" software in mission operations. In either case, the software development processes for such software are much more rigorous than are those for software such as climate data analysis! (See my previous post about my experience in NASA.) Purportedly Bates was dead set on using the more rigorous processes despite their inappropriateness.
Fernando, you are insinuating Karl was someone "who wanted to cut corners, did shoddy housekeeping, wanted to hurry up, etc."
Back it up or don't write stuff like "He was in a hurry because he wanted the Karlized data set out before Paris. And that's about it."
Bates could possibly be just a perfectionist. A true perfectionist (I use it as a negative term, but do not imply a conscious bad will - something like, irrationally improving things one is concentrated on, to the detriment of other things that one is not, but could and likely should, be focused on).
Hmmm... as opposed to a non-true perfectionist? I guess I would say this is someone who really tries hard to do their best, but can rationally see competing objectives and ration their time accordingly.
Eli should post a link to this in Delingpole's Breitbart back yard , if only to see how long it takes for censorship to set in :
http://www.breitbart.com/big-government/2017/02/06/delingpole-noaa-scandal-gives-trump-the-perfect-excuse-to-drain-the-climate-swamp/
Tom Dayton's argument that projects differ in their software requirements ... is exactly right, examples of the general case.
Bates' methodology error is quite familiar:
Inside Bell Labs, projects like Safeguard ABM or big switches or big operational databases had really heavyweight Q/A, source code control systems, etc, etc .. and there were always problems when managers from such were given projects where that didn't make sense (like analysis systems for phone calls or trouble reports (which I managed)). Of course Area 10 (basic research) never did the heavyweight methods.
In 2002 I repeated two software engineering talks I'd used 1977-1983 inside Bell Labs and for ACM National Lectures and other venues. Both of them include the issues of choosing appropriate methodologies. The first was what I used when helping teach the internal BTL software engineering project management course.
Hmm, I notice none of the skeptics have challenged the data in Karl et al. Could that perhaps be because Zeke H and others have shown that it is more consistent with all the other independent measurements? Climate scientists are so unfair and greedy. Once again, they take the position where all of the available evidence supports them.
Mornin' William/Tom/John et al.
We are at long last proud to be able to reveal the outcome of the first beta test of our previously Top Secret development program for "alternate facts" wetware (or AFW for short):
Beta Testing Snow White’s Alternate Fact Detector
Constructive criticism of our development and testing methodology is of course always welcome. Be warned that the project specification lists "Irony" at #1.
For Eli's Information:
<a href="https://twitter.com/search?f=tweets&vertical=default&q=%23AFW>#AFW</a>
Sorry Eli - That didn't work very well! Can you fix it?
Whenever I try and Preview I get a 404 error on a subsequent Publish :(
I'm bored this morning and now providing climate data with Mystery Science Theater inspired commentary.
This one should be captioned: Someone forgot to tell the arctic to wait for Karl to bust the Pause.
https://1.bp.blogspot.com/-pP-4tAZ6VzU/WJniupgC2PI/AAAAAAAAATE/Nk1tqyUaizUatLk9oebwOWP_IWyVi2aagCLcB/s640/ASIV%2B1_31_2017.png
Well Russell it lasted a bit more than an hour. Not too bad for an old bunny.
Jim, worked for Eli.
Good post.
As a die-hard Carlist, "Karlize" no doubt hopped quickly into Fernando's mind.
For Eli's information this is what I was attempting to say:
#AFW
I'm using Chrome on Win7. Will this work?
No - Actually it's a 400 error.
If I run into him at Slough High ,I'll ask what took so long ?
There's a commentary posted today on the SCIENCE web page:
How a culture clash at NOAA led to a flap over a high-profile warming pause study
Jim, you're leaving out closing " for your href's, fwiw. It's just a typo. Some sites automate links, some don't, some exclude some html, but that's a separate issue.
I mention it because this is the kind of error I frequently commit.
Here's Potholer doing his thorough; FL and others who like the message and ignore reality would do well to watch and listen before they make blindfolded accusations.
https://www.youtube.com/watch?v=kQph_5eZsGs
This is one way to put it: Scott Boulette @AlgoScott
Alt facts peer reviewed by politicians, what could go wrong?
Meanwhile, for a cauld grue, here's politics before people personified:
23 minutes to kiss the planet goodbye
http://whogoeswithfergus.blogspot.com/2017/02/23-minutes-to-kiss-planet-goodbye.html
Waking up in the middle of the night, the World Service was interviewing Myron Ebell.
http://www.bbc.co.uk/programmes/p04rq0fk#play
Myron Ebell trashes the planet for profit
Peter Thorne just published an excellent explanation of why, what, and how NOAA is updating its sea surface temperature database. Quite nefarious: https://www.carbonbrief.org/guest-post-why-noaa-updates-sea-surface-temperature-record
Post a Comment