It is a fascinating look at both how GISS reacted, quickly to figure out the source of the problem, and responded to McIntyre and the news media. It is equally informative as to how good and bad reporting on science issues gets done.
Since it is always good to start at the start, Eli is going to reproduce most of an Email from the sysop who caught McIntyre's web crawler
On about May 16 at around 10:30 or 11:00 pm, as I was getting ready to leave GISS for the nights, I belatedly checked the error logs on the two web servers and discovered that there were several thousand errors in the log on Web2. On a normal day there would be about 500.he continues
The identity of the computer making the requests was consistent, and as best I recall was something in the domain of Rogers Communications, a Canadian phone company and IPSOh yeah, the naughty bits, well, Hansen says
Plainly this activity was from an "automated" agent, which in rough parlance is usually called a "robot". Many robots have legitimate purposes, e.g. serach engines such as Google or Yahoo, but others do not (spambots), and others one just doesn't know.
As the robot on May 16 came from a generic ISP address rather than, say and academic address and further because it's "user agent" tag provided no further information about who was runing it, and also because the GISS websites have "robots.txt" files which instruct all well behaved web robots to stay out of the CGI directories, I cut off access to the ISP in question to the websites on Web2.
The next day I received e-mail from McIntyre asking what was up. He did not identify himself or on whose behalf he was acting.
At some point Reto got involved in the communications, and he must have mentioned to Jim what was up. Later on Reto indicated to me that Jim had said to go ahead and re-grant McIntyre access to the material.
I do not know if at any point McIntyre actually asked Jim or Reto if it was possible to obtain the GISS copy of the station data in a single or small number of files. All I know is that my first contact with him came because he was blasting umpeen thousand requests at the webserver.
I have no idea how much traffic McIntyre's website gets, and I don't know that I have ever looked at it. His tone in his e-mail was on the arrogant side, so I had no desire to prolong communication with him and longer than was necessary
There are some desperate characters trying to make a mountain our of a mole hill.and
Do we want to lower ourselves to debating with a court jester? Of course that's what he wants.Comments?
I don't have a strong preference as long as it is not taking a significant amount of my time.
I have not read the stuff you are referring to [McIntyre's whinge to Town Hall, more later], but as I recall, as soon as I was told about the matter, I said that he was welcome to the data
This will be why Reto was backed up in early December, when I was asking him some questions about GISTEMP STEP5.
ReplyDeleteObviously, Hansen has heard the following advice and takes it seriously:
ReplyDelete"Never mud-wrestle with a pig. You'll both get dirty, and the pig likes it."
Took Eli a moment to figure out what Nick Barnes meant so might as well point it out. The FOA request had to be dealt with at GISS, which meant people had to go through their Email and more, including Reto Ruedy.
ReplyDeleteFOAs cost time and money to answer, which is part of the reason that Phil Jones was mucho unhappy about the blizzard of FOA requests from McIntyre and co.
Just FYI, the actual release of these emails was in late December and they are available from here:
ReplyDeletehttp://www.nasa.gov/centers/goddard/business/foia/GISS.html
And, yes - 6 FOIA requests for (in total) gigabytes of email that needs to be checked for responsiveness, privacy concerns, and applicability is a non-trivial amount of work.
Note that the original FOIA request for these emails was from CEI, but Judicial Watch scooped them with the non-event press release. They are unlikely to be amused.
Eli, they made a movie about you. Something about a gospel.
ReplyDeleteYou should play a game. See if you can guess which email is going to be taken out of context as proof of ..something or other.
But, but , but McI has only recently claimed that he is not harassing US scientists. And what about harassing the UEA/CRU.....?
ReplyDeleteWhat a load of trollop. Bombarding the servers with thousands of hits using a robot. Lovely (sarc).
I had no idea just how annoying the little er lovely people were getting.
ReplyDeleteDoes FOI have abuse of process clauses?
Little Mouse
Hmmm, how about that Virginia State Climatologist guy's files?
ReplyDeleteHere's a bit of tech advice. 1) Convert the image pdf files to searchable pdf files with Adobe Acrobat Reader Pro or similar software. 2) Set up an account on Scribd and upload the pdf files there. Once this is done you can embed them in your blog if you wish.
ReplyDeleteAnyway, I've been enjoying reading your blog.
I've been enjoying your blog too, Eli, although I don't have the maths to appreciate the deep stuff!
ReplyDeleteWhat was the purpose of the bot that McIntyre seems to have unleashed? I'm not clear if it was meant to be drawing on the data or just hammering the server?
McI was scraping the data, but in the process was doing a pretty good imitation of a DOS attack.
ReplyDeleteThe claims that McI made about the correction seem to have done a pretty good imitation of a DOS attack on the work and lives of the scientists at GISS since dealing with the fallout of his inaccurate claims seems to have taken up a fair amount of the time of Hansen, Reudy, Sato and others.
BTW, my favorite quote was Hansen's "tempest in a teapot dome" quip.
MarkeyMouse says
ReplyDeleteSteve McIntyre at Climate Audit was merely trying to download data that had been put there for that exact reason.
Reto panicked and pulled the plug.
As regards to FOI requests and GISS, why don't they publish their data and methods in a form that others can replicate?
It's because they are committing fraud, and want to hide it.
Like Michael Mann and his fraudulent Hockey Stick machine, and the gang at CRU, who are about to be Criminalised.
I've read the files that Eli has available. Fascinating!
ReplyDeleteMark R, you are well off base with your comments about how McIntyre went about things. Might pay you to read the files yourself. McIntyre doesn't look so good, I'm afraid. I'd say devious, arrogant and unethical.
Mind you, I realise accuracy and facts are very flexible things in Denialist quarters....
The MarkeyMouse bot seems to have a faulty communication module, whereby it is robotically repeating its standard programmed talking points in a way that causes non-sequiturs in the comment thread.
ReplyDeleteCan the bot owner for markey look into that bug and fix it so Eli's server space isn't overburdened with the gibberish?
Best,
D
MarkeyMouse has shown himself to be a denialist of all things, including HIV. He is also a "birther." Pay him no mind.
ReplyDeleteMarkR:
ReplyDeletein a form that others can replicate?
They do.
Nick,
ReplyDeleteThese drones don't have a clue that all the GISS code is available (and is now being translated and picked over by somebody competent, too). All they do is repeat their mindless mantras. Does anybody think Markey could accomplish anything, with the easily available source code in hand?
You lot are a laugh. Can't you get anything right. From Climate Audit at the time:
ReplyDelete"I wrote back to Schmunk stating that I was not using a “web robot” but was downloading data for legitimate scientific purposes, as follows:
I have been attempting to collate station data for scientific purposes. I have not been running a robot but have been running a program in R that collects station data.
However, even after confirming that this was not a web robot and was data access for scientific purposes, NASA GISS did not remove the block (which applies to many webpages besides the GISTEMP data that I was downloading.
I wrote to Reto Ruedy of NASA GISS this morning as follows:
Dear Dr Ruedy, I have been unable to locate an organized file of station data as used by GISS (such as is available from GHCN). In the absence of such a file, I attempted to download data on all the stations using a script in R. This was laborious as it required multiple calls. I was not using a “web robot” nor was I indexing files. During the course of this, your webmaster blocked my access to the site claiming that downloading the data in this fashion violated your policies. Would you please either restore my access to the site or provide me with an alternative method of downloading the entire data set of station data in current use. Thank you for your attention, Steve McIntyre."
http://climateaudit.org/2007/05/17/giss-blocks-data-access/
Also Dano is correct. I haven't had the full Commie implant, so I am still waiting to see Obamas Original Long Form Birth Certificate, with the name of his hospital and attending doctor.
Neither have I yet seen a certifies proven electron microscope picture of HIV.
PS Well done Nick Barnes. It's only taken 3 years.
Markey: What point are you trying to make? None of McI's whine disproves whatever is being said. He had a script to get all the data, somebody at NASA flagged it as it looked suspicious, it eventually got sorted out what was happening, and McI is still a whiner.
ReplyDeleteAny program which automatically makes a large number of HTTP requests is a web robot.
ReplyDelete3 years
And how long has it taken you? Or any other CA denizen?
Any non-conspiracy-theorists want to contribute?
My point is that Steve McIntyre did absolutely nothing wrong, but the intent of the post and subsequent comments are to suggest that he did. Completely wrong.
ReplyDeleteeg "McIntyre doesn't look so good, I'm afraid. I'd say devious, arrogant and unethical."
"..a pretty good imitation of a DOS attack on the work and lives of the scientists at GISS"
Completely unfounded bollocks, but at least it lays bare your innate bias.
Couldn't he have just used wget?
ReplyDeleteMarkey:
ReplyDeleteCredible folk call up the researchers, give their name and affiliation and make a request and state what they are doing.
Sneaky a-holes do what you are defending, tool.
Message Force Multipliers, unite your small minorities and redouble your doltish efforts! Carbon regulation is coming! Much more idiotic moron spam is needed!
More moron spam to defeat ACES!
Redouble your moronic efforts, quick!
snicker
Best,
D
For once, I think Dano is missing the point in the same way that Markey is.
ReplyDeleteWhile using the script to get the data might be ill-advised, it's no great sin. It got flagged, but it got sorted out.
McI's sins are in how he constantly tries to smear scientists with absolutely no substance whatsoever. He is not acting in good faith. Not even close. That's the point. Nor is he particularly competent. Over several years, he's found maybe two or three things that could be called errors, none of which had all that much impact. You wouldn't know that from listening to him, though.
It was good that he found the so-called Y2K bug. It was a real error, and needed to be fixed, even if the impact was small. But the way the denialosphere tried to make a mountain out of that molehill was just plain stupid.
Putting it a bit more plainly: The deniers keep hoping to find a conspiracy within these emails. They're hoping to find emails that say it's all a fraud, proving the deniers correct. Sadly for them, the emails show no such thing; they show that the scientists hold the deniers to be annoying idiots.
ReplyDeleteNo, carrot, it was a constructed controversy. If he was so dumb as to not know the easy way to go about it, it reflects badly on McI. So one has to gin up something to keep the pedestal intact.
ReplyDeleteBest,
D
It was a constructed controversy, but mainly in how McI tried to describe it to his uncritical fans. And then how they tried to inflate the importance of it all.
ReplyDeleteThe actual act of downloading, in itself - eh. There were better ways of going about it, but GISS did let him continue with his method once they figured out what was going on, didn't they? I had that idea, at least.
Carrot eater: I think what Dano is alluding to, is that McIntyre claims that:
ReplyDeletea) he was blocked BECAUSE he was Steve McIntyre
b) he was unblocked BECAUSE he was Steve McIntyre
Add to that that the data is freely downloadable from the GHCN, which is set up to download loads of data at once, whereas GISTEMP has individual station downloads. You'd think Steve McIntyre knew that...
It seems to me that MacIntyre says that only he is smart enough to run climate code programmes but that he's not smart enough to access the equipment without crashing the system. DOH!
ReplyDeleteThe cigarette industry, their rent -a-scientists and right wing stink tanks realized that delay enables doubt. The heartlandrs and Catoites today roll the little balls for so called independant bloggers to throw. FOI requests cost time and money. MacIntrye has surely been told by the Fraseristas that any time climate scientists spend away from their real work helps the coal and oil fossils.
Well said Comrade. Your toeing of the Party Line has been noted by the Politburo.
ReplyDelete"Carrot eater: I think what Dano is alluding to, is that McIntyre claims that:
ReplyDeletea) he was blocked BECAUSE he was Steve McIntyre
b) he was unblocked BECAUSE he was Steve McIntyre
"
Marco's got it.
He wasn't blocked because he was McI, he was blocked because he was spidering the site.
After screaming bloody murder while the poor sysadmin was off work, said sysadmin responded to the "I'm McI and how DARE you BLOCK ME!" with a statement saying, in essence, "who? never heard of you before" (he's just a NASA netadmin, after all), and explained why.
And then afterwards said "go ahead and download, but please do so at night when the servers aren't so busy".
Totally typical stuff having nothing to do with the fact that McI's an asshole.
If it had anything to do with the fact that McI's an asshole, he would not have been given permission to download even at night.
Apparently my government caters to assholes :)
Marco: I agree with your illustration; I'm just saying that this interpretation is not clear from Dano's comment - which seems more focused on the downloading itself.
ReplyDeleteLittle Mouse,
ReplyDelete"Does FOI have abuse of process clauses?"
I don't know about the US but in the UK such requests are supposed to be for legitimate purposes and not vexatious.
MarkR,
ReplyDelete"Also Dano is correct. I haven't had the full Commie implant, so I am still waiting to see Obamas Original Long Form Birth Certificate, with the name of his hospital and attending doctor.
Neither have I yet seen a certifies proven electron microscope picture of HIV."
I'm declaring Poe.
No, I don't think he's a Poe. I think he is genuinely the stupidest person I've run into in a long time.
ReplyDeleteIt seems that Brandon Shollenberger, apprentice extraordinaire in parsomatics (tm, Eli) is willing to reopen the ontological question of bothood:
ReplyDelete> The script Steve McIntyre used was not a “robot.”
http://rankexploits.com/musings/2012/reduce-image-scraping-to-prevent-blog-crashing-and-thwarth-copyright-trolls/#comment-89583
Since I was put on probation there yesterday afternoon, I might not be able show that Brandon's position does not rest on much. Meanwhile, bunnies could appreciate how this question resurfaced in response to Eli's remark, how the debate yet again turned into a Definition Game and that the Never-Never-Admit-You-Are-Wrong Land lived up to the nickname Eli found for Lucia's.
It could be fun to rehearse basics of agent theory or natural semantics. Perhaps another time. The audit never ends.
Due diligence,
w
around the time that Lucia "found funny" that I was talking about moving goalposts, whence this ontologi