Yah gotta pay attention
Deltoid and Climate Audit tell how the Killer McIntyre uleashed his Lego** Robot Anonymouse on GISS to harvest all of their climate station records. Turns out that the rapid polling of the robot resulted in denial of service to everyone else, and the webmaster told McIntyre to bug out,
which was not kindly received. Lots of harumph on Steve's part, but they did provide him with access within a few hours, and this interesting comment
Please note that the robots.txt file on that website includes a list of directories which any legitimate web robot is _forbidden_ from trying to index. That list of off-limits directories includes the /work/ and /cgi-bin/ directories.
Because the robot running on the cable.rogers.com network has rather obviously and blatantly violated those rules, I placed a block on our server restricting its access to the server.
While the whole thing was a farce, those seven words are the take home. It's obvious what Steveo intends to try, Hansen, otoh is a lot older fox than Michael Mann.
After a short meeting with Dr. Hansen, we were advised to let you download whatever you want as long as generally accepted protocols are observed. Please try to do so at a time that does not impact other users, i.e. late nights, weekends.
What we did with the GHCN data is carefully documented in the publications listed on our website. We are not creating an alternate version of the GHCN data, we are mainly combining their data in various steps to create our anomaly maps.
** Lego has a huge market among robotics courses. Eli just took delivery on 8k$ worth of stuff for next year