Arguments
A Hack by Any Other Name — Part 2
Posted on 26 February 2014 by Bob Lacatena
Part 1 recounts the initial steps of the hacker, and our initial discovery of his intrusion.
Angela: They hack into computers and they cause this chaos.
— The Net (1995) —
March 24, 2012 — 1:06 AM AEDT — The Early Conversation
Meanwhile, the real forum suffered through a torrential storm of activity on the "SkS was hacked" thread.
| 24 Mar 2012, 1:06 AM | SkS was hacked |
| 24 Mar 2012, 1:13 AM | |
| Alex C |
It's 1:00 in Australia... |
| 24 Mar 2012, 1:20 AM | |
| 24 Mar 2012, 1:27 AM | |
| 24 Mar 2012, 1:43 AM | |
| 24 Mar 2012, 1:46 AM | |
| 24 Mar 2012, 1:51 AM | |
| 24 Mar 2012, 1:56 AM | |
| 24 Mar 2012, 2:16 AM |
Comments about the hack flew fast and furious on the real forum. By this time it was 2 AM in Australia. I took a chance and tried to Skype John, not expecting it to work. It did, but only because he was already up and on, starting to look at things. Logicman apparently had his phone number and had just called him to rouse him.
We looked at it together and talked, mirroring each other's tone of surprise, confusion and anxiety. He was surprisingly awake, an effect news like this would probably have on most site administrators. He too noticed that the presentation of the forum wasn't right. He confirmed that no, he hadn't programmed some odd super-admin variation of the pages. He quickly noted that it wasn't even close to right, because the data necessary to display the page like that would require combining multiple database tables, specifically the POST table, where the individual comment was stored, with the USERS table, where each user's registration information is kept.
The user information was normally nowhere in the forum post pages. The pages not only weren't formatted normally, but producing them would have required access to multiple database tables – basically the entire database – in order to construct a page that looked like that. It was almost like the pages were constructed by someone who had uploaded their own programs onto the Skeptical Science server, or else had downloaded the entire database. Both possibilities were chilling thoughts. It meant that the hacker had access to every aspect of the Skeptical Science web site, and also that he had been there for a long, long time, long enough to write the programs to generate pseudo-forum pages in the fashion that served his own agenda.
He'd done the same thing to the deleted comments page, a private administrative page used by Skeptical Science moderators. The hacker had added personal information that was normally not included, and expanded the presentation to include 3.4 megabytes of deleted comments, when the actual page that administrators use to display such information only shows the most recent fifty.
More unsettling was the dump of the users table. The hacker had included a text dump of most of the data from the entire users table, slightly reformatted and edited. It appeared to be a list of every user ID registered in Skeptical Science, including everyone’s e-mail addresses,. We later discovered that a large number of pseudo-skeptics had had their own information removed from that dump by the hacker. This in itself is a telling bit of legerdemain. It seems that releasing thousands of people's personal information was considered acceptable, but still recognized as enough of an intrusion that fellow pseudo-skeptics were purposely and methodically culled from the release.
Coming as it did right on the heels of the pseudo-skeptic response to the Heartland affair, it also seemed that the saying that "there is no honor among thieves" was not strictly true. It seems that to the pseudo-skeptic crowd, it's heroic and understandable when a hack is perpetrated against people who work with or explain the science, but it's a horrific crime when committed against poor, well-meaning, tax-exempt political front organizations like the Heartland Institute.
John moved on to look at another file, one with which I wasn't at all familiar and hadn't yet grabbed from the hack. It was entirely separate from the zip file I’d downloaded. John explained as he reviewed it that it was a log file, generated daily, of every database SQL statement used by the web site in the past day. He and Doug Bostrom had previously designed and implemented it to help to detect and research SQL injection attacks, which had grown more common over the years, becoming an ongoing and expanding problem. SQL injection attacks are one of the most common and simple forms of true hacking, and had been used against Skeptical Science before. They'd become frequent and annoying enough that special efforts were required to protect against them.
Detective Del Spooner: [to Susan, after the robots have started a revolution] You know, somehow, "I told you so" just doesn't quite say it.
— I, Robot (2004) —
February 23rd, 2012 — 9:07 AM AEDT — Okay, Something Very Weird Has Just Happened with the Forum
A month before the hacker released the data, just days after the initial hack, the hacker made a mistake, one that we missed.
In retrospect, we should have seen it coming. The clues were all there, more clues than I care to admit to. We all knew that things needed to be more secure. The topic seemed to come up every month, in particular the idea that the forum itself should be completely separated from the Skeptical Science site to help protect it, should the more visible, public site itself ever be hacked. And odd things were happening almost every week. It was like that feeling you get that you're being watched, when you really are being watched. You should turn around and check, but you don't want to feel stupid if it turns out that there's no one there.
But there was.
The Skeptical Science site endures frequent SQL injection attacks, as do most sites on the Internet.
The work that John and Doug had done in the past to detect them and thwart them now worked every time. Still, you can never know that any system is completely secure. Nor can you tell when it's a random attack by a bot that's just mindlessly hunting for vulnerable sites, or a focused probe under the direction of a determined, intelligent, human intruder. That effort detected several attacks in 2010 and 2011, most of those originating in China. One originating in Romania was detected on January 19th of 2012, and another coming from South Korea was detected on February 23rd, 2012. There was another, even more troubling incident that day, which preceded the SQL injection attack.
At 9:07 AM on February 23rd, in Brisbane, John started this thread:
| 23 Feb 2012, 9:07 AM | Okay, Something very weird has just happened with forum |
| John Cook | Got an email from Brian P this morning saying that the whole forum was publicly available to him, even when he wasn't logged in. I checked and this was true. A little panicky, I investigated and worked out that all the permission levels of each forum had been set down to zero. Normally, they're set so only authors can access most of them, except the translator forum is also accessible to translators. Strangely though, there is an admin forum that only admins can access and that wasn't set to zero - it was still set so only admins can access it.
I have no idea how this happened. Several possibilities come to mind. First, I did it by accident when I was screwing around with the database sometime. Someone with admin access (there are about half a dozen SkSers with this access) made the change. Or we were hacked in some way and the hacker changed the levels. None of the options seem likely to me but the most likely is human error on my part although the fact that the admin forum was still set at admin level belies some kind of blanket wiping of all levels. So I'm a little freaked out - it's not knowing how this happened that has me most worried. Has anyone been looking at the forum and how long has this been available? But I've been procrastinating some of those security measures that have been suggested to me and as soon as I get to work this morning, am going to implement some of those measures. |
| 23 Feb 2012, 9:13 AM | |
| Albatross | Oh shit....we'll know soon enough if it was a hack. |
| 23 Feb 2012, 9:19 AM | |
| MarkR | Wasn't someone going to take a load of the old stuff offline presto? |
| 23 Feb 2012, 9:21 AM | |
| logicman | When I logged in I was seeing forum section names like blog posts. |
The comment by Albatross was sadly, dangerously wrong. Later on, John continued along a line that should have taken much higher priority:
| 23 Feb 2012, 10:44 AM | Just for the record |
| John Cook | Yes, we agreed to take all old threads offline but I just hadn't got around to it. Am going to implement that this morning as the situation has gone from back-burner to yellow alert. That's the low lying fruit on the security to-do list, I'll then move further along the list as time permits (really intense busy period for me at UQ right now, have a big deadline tomorrow). |
My own comment, the last in the thread, was much stronger in response, but at the time I still did not have access to systems, consoles, or code. There was little I could do. I was only just beginning to get involved, and helping to move Skeptical Science towards a more modern, ‘traditional’ code-sharing environment. Until then, everything pretty much flew from John’s laptop onto the site, and vice versa.
| 24 Feb 2012, 5:36 AM | |
| Bob Lacatena |
John, If you haven't already, immediately change all passwords (meaning the MySQL database passwords, root password, site access passwords, etc.).?Also, immediately grab copies of all logs (apache log, mysql log, etc.).?If logging is not turned on for MySQL you might want to look into that... the things you need to turn on to track down if you've been hacked. This is very concerning. |
Unfortunately, that's as far as that went
On the 23rd, at 12:33 PM AEDT, John started a new thread on the forum about an attempted hack that had just set off the automated alarms.
| 23 Feb 2012, 12:33 PM | Attempted SkS hacking happening right now |
| John Cook |
I've programmed the website to email me whenever someone tries to use SQL injection to hack SkS. Just got 334 400 emails now - each an attempt to hack SkS using SQL injection over a 4 minute period. The IP is 221.143.48.210 which is based in South Korea. Okay, the forum glitch and now this, I'm getting a little freaked out. |
After the morning’s earlier, uneventful scare, most people brushed this aside.
| 23 Feb 2012, 12:56 PM | |
| logicman | Don't get paranoid - there are a lot of bots out there searching for weak defences on any site whatsoever. |
| 23 Feb 2012, 8:39 PM | |
| Ricardo | I concur, don't get paranoid. Any online pc sooner or later gets targeted by bots, let alone web servers. |
Thomas Reynolds: Ten-year-olds go on the Net, downloading encryption we can barely break, not to mention instructions on how to make a low-yield nuclear device. Privacy's been dead for years because we can't risk it. The only privacy that's left is the inside of your head.
— Enemy of the State (1998) —
February 23rd, 2012 — 2:08 AM AEDT — Opening the Forum
It was February 22nd, 4:08 PM in Germany when the hacker returned, two days after his initial hack. Tor rotated his IP address 22 times during the three hour incursion.
At 4:14 he registered a new user, as “francois”. He poked around for a while, doing various searches to see if that user ID could be found through the Skeptical Science search feature. At 4:36 he used a previously hacked ID, one that he’d given administrative capabilities, to give his new “francois” user the same capabilities. At 4:37 he started looking at the private form, starting with the topic on Moderation. He looked next at the Admin topic, perhaps looking for clues about further administrative functions. Then he looked at General Chat.
At 4:38 The German finally found a thread that interested him, titled “Sorry John”. 40 seconds later he glanced at a thread titled “I just received a letter from Heartland”. A minute after that he entered one started by John Cook named “WOW! Peter Gleick was 'Heartland Insider'!!!”. After reading the first page for two whole minutes, he moved on to the second page of that thread, reading that for another two minutes.
Soon after, he began to play with the administrative panel, clicking various links. Using a function there, at 4:51 he uploaded a program, named “f2”, which allowed him to look at each of the programs which run the Skeptical Science site. He continued looking at the code for over an hour, until 5:58 PM when he somehow altered his “francois” user, making the date of registration appear to be one day sooner, and then returned to using the administrative user ID he’d retained from the first day of the hack.
At 6:14 PM he resumed use of his f2 program, this time to download entire directory listings of the site, including the full logs directory, as well as looking at more programs. At 6:21 PM, he went back to using his francois ID to browse the forum.
At 6:46, he used an administrative panel to alter security on one of the forum topics, the one on Moderation.
At 6:52 PM he began, one by one, opening the topics within the private forum to public access. At 6:59 PM he logged off and accessed a forum thread, verifying that he could read the contents without being logged on. From there he continued to browse the forum, including a thread titled “Growing accusations of Gleick being the pdf faker!”
At 7:13 PM he uploaded a program named “un”, and at 7:14 PM he ran it, removing his f2 program from the system. He ran it again at 7:15 PM, removing itself, and with it the most obvious traces of his intrusion.
The entire foray lasted three hours and seven minutes, using twenty-two different IP addresses to execute 299 different commands.
There were still no links to the forum, so if you didn’t log in with an contributor level ID, you wouldn’t know the forum existed. But if you did know where to go, you now wouldn’t need a valid user ID to see the contents.
The forum was open for about four hours before John reset the user levels on all of the “exposed” topics. Only two unknown IP addresses (not protected by Tor) visited the forum in that short span, one from Houston, Texas, but almost an hour after the forum had been re-secured, and one from Phoenix, Arizona, once while the forum was open, but without visiting any actual threads and so without seeing anything private, and the other several hours after the forum had been re-secured.
It may be unfortunate that John found and corrected the problem so quickly. It would have been interesting to know with whom the hacker wished to share the contents, especially if they were not as careful about protecting their identities.
Part 3 explains SQL injection attacks, recounts the hacker's release note, and describes the hacker's activity through March.
But at the time, we foolishly didn't even recognize what was really going on within the system.
To be continued...
It all has that feeling you get when someone has broken into your home. It's that sense of being violated. You don't know who they were but you know they were, at one point, there in your livingroom unhooking the cords to your kids' Wii they got for Xmas. They were in your office going through your desk drawer. They were in your bedroom going through your wife's family heirlooms.
It really is a sick feeling. And it's just phenominal to me that there are people out there with such low standards of morality as to believe it's okay to do this.
I also find it sickening that other bloggers scoff at the whole thing (no, I'm not going to reward them with links to their sites) when they, too, should be expressing a sense of indignation.
For all the wind emitted in other places about this topic, one would think the windy would blow here at least a little bit, at what is surely the true nexus of all the hot breezes.
It's true that it's more pleasant to agree than than disagree; what better than to flock with other agreeable, agreeing folk?
Not sure how you come up with the destinations, but the first thing a hacker would do is to use a proxy(even with Tor), preferabily from a country not assocciated with his own.
Peter Gleick, for example... and those who condone his actions in phishing and leaking documents from the Heartland Institute.
By the way, just thought I should let you know that SkS is still quoting from and linking to the 2-yr old fake "Confidential Memo: 2012 Heartland Climate Strategy" document of which Gleick denies authorship. (http://www.skepticalscience.com/denialgate-heartland.html). I'm assuming this is merely an oversight that will be corrected.
N.B.
Russ R... Interestingly, Peter Gleick, himself, stated that he believed it was wrong what he did, that he definitely had a lapse in judgement.
Has anyone done the same after the CRU hack? Has anyone bravely apologized for the SkS hack?
Russ R... As well, no one that I know of has ever condoned Gleick's actions. But it's absurd to compare that to the CRU and SkS hacks where people have systematically, deliberated, and with malice of forethought, perpetrated very serious crimes.
Rob Honeycutt @6, my take is somewhat different, in that I believe I have encountered people who have condoned Gleick's actions. They have done so either on the grounds that the situation is now so desperate with regard to climate change that the end justifies the means; or the grounds that once "skeptics" began hacking and publishing the internal emails of scientists (and The Heartland Institute certainly republished them), they set a standard whereby they wished to be treated (ie, "Sauce for the goose is sauce for the gander").
Personally, I believe that neither attitude is correct. Indeed, given that SkS has frequently moderated posts to delete links to either the UEA emails (I believe), or to draft versions of IPCC AR5 on the grounds that the information was obtained unethically, I think SkS as a matter of consistency should not link the the Heartland Institute documents. Either that or change its moderation policy to allow links to hacked or unethically leaked material that we consider unethically obtained, and/or published. Of the two, I believe the former to be the better approach.
Tom... There are certainly people out there who condone Peter's actions, and I don't think they are right to do so, especially seeing as Peter doesn't condone Peter's actions. Those folks are far and few.
I also think there are private discussions about Peter's actions where people float such views for the purposes of exploring the issue, but without making a public declaration that they would condone his actions.
I find Russ' comparison insulting and a deliberate attempt at distraction from the issue at hand.
If he wishes to discuss the merits of whether or not the Heartland docs should be posted on SkS, it should be done on that thread, not here. Conflating the two events is not warranted.
Rob Honeycutt,
Then you are indeed fortunate. I could name many prominent individuals who have done exactly that.
Would you be so kind as to explain what is so "absurd" about the comparison? Assuming the SkS hacker did indeed gain unauthorized access to obtain private information and leaked it publicly, how does that differ from Peter Gleick's actions?
I can't imagine why you would feel insulted. I never mentioned you, nor did I suggest you would support Gleick's actions. And it's hardly a "distraction for the issue at hand". You were the one who mentioned "people out there with such low standards of morality as to believe it's okay to do this". I simply provided an example.
I already noted above that "I have no issue with SkS continuing to link to the real Heartland documents..." But fair enough, I'll move discussion of the fake document to the other thread.
Gosh, do tell. Who are these people who didnt think this was a serious error of judgement?
Russ...
Yes, please do tell. You're going to need to support that statement.
What is absurd is to conflate a moral lapse with an extended, intentional criminal attack.
Think of it this way. It's the difference between having had an affair while married, and forcably beating and raping someone.
Both are wrong but the two are not comparable. I am insulted because I have been personally affected by the hacking and you are showing a callous incapacity to see the relevant severity of the two acts.
I think that there are some intreresting comparisons to be made between the Gleick case and the SkS hack. While there were a few who applauded Gleick (eg George Monbiot) many more were critical of his ethics, some harshly so. Peter Gleick himself apologized. The released material comprised budget documents and strategic plans. Released documents that contained personal information about Heartland board members was taken down.
There was no need to hack SkS to find out our budget or strategy, anyone who asked could find out: a few bucks raised by donations to pay for webhosting, no payments to contributors, no secret paymaster. Our strategy is to keep doing what we have been doing.
The SkS hacker must have been disappointed. All he got was confirmation of all of the above and a few intemperate comments of the kind that people make when they are chatting and venting privately among friends. Mostly, the conversation was earnest (and usually rather boring) discussion about getting the science right.
In contrast, many of those sympathetic to the SkS hacker continue to deny that there was even a break-in. We had faulty locks, they say, so even if there was a break-in, SkS deserved it: in other words, blame the victim, plea contributory negligence. The hacker himself does not dare to come forward, even anonymously, to say what happened.
And, among those who have published our private conversations and personal information, we hear very little in the way of doubt or questioning that this might or might not be justifiable.
Andy... Looking at the Monbiot link, I think he's using the event as a means to bring up other sociopolitical issues, which sort of mirrors my comment @8 about exploring the issue.
So, while I don't agree that Gleick's actions are justifiable, I think the event, in-and-of-itself, presented the opportunity to discuss other tangential but equally important issues.
Russ,
You say "I could name many prominent individuals who have done exactly that.", but you do not name a single one. This is political sloganeering and is not allowed at SkS. You frequently assert that you are correct without any data to support your wild claims. By contrast, Tom provided links to peer reviewed data. Provide a list if you have names or do not make your unsupported claims. When you do not provide data to support your claims you are conceeding the point. Since you have provided no names you have conceeded that you cannot find them.
You have the same habit with your claims about the science. Provide data links to support your wild claims.
I find the discussion of morality here irrelevent and distracting.
Skeptical Science is about climate science, and about the sociology surrounding climate science and its rejection.
The hack has no bearing on the science. It may have some bearing on the sociology surrounding of climate science and its rejection, or it may be a random act of malice - which of these is the case is at this point unproven.
While as a computer geek I am of course interested in the details of the hack, I think the importance of this series is to place the information in the public domain where future researchers into the history and sociology of climate science rejection can ask how, if at all, it fits in.
scaddenp, Rob Honeycutt, and michael sweet:
Because you asked, here are 3 examples of prominent individuals condoning Gleick's actions. And for balance, one example of a prominent individual who responded very appropriately, in my opinion.
Anyway, here's someone who I believe responded appropriately:
Comparing Gleick to a common thief without informed consideration is to wade into an ethical morass, wearing naivete instead of waders.
Of the people in the crowd here Tom is probably most likely to emerge on the the other side of the swamp without being covered in mud.
For the rest of us, before saying a word about Gleick and then going on to make comparisons do read the necessary fundamental primer on how and when to compromise ethics and morality, "Lying: Moral Choice in Public and Private Life" by Sissela Bok. If you're not up to at least that level you're not capable of making the kind of comparisons being posed here.