A Hack by Any Other Name — Part 2

Part 1 recounts the initial steps of the hacker, and our initial discovery of his intrusion.

Angela: They hack into computers and they cause this chaos.
 The Net (1995)

March 24, 2012 — 1:06 AM AEDT — The Early Conversation

Meanwhile, the real forum suffered through a torrential storm of activity on the "SkS was hacked" thread.

24 Mar 2012, 1:06 AM SkS was hacked

It looks like it happened


I have downloaded the zip in the 1st comment but it is invalid.  It won't take long for whatever is in there to be made valid and put out there.  It's 18.5 MB in size

Can anyone get it open?  Can someone call John immediately?

24 Mar 2012, 1:13 AM  
Alex C

It's 1:00 in Australia...

24 Mar 2012, 1:20 AM  
Alex C

Just downloaded it, what the F**K, all my information is in there, contact and IP~ everyone else's too.

It's valid Grypo.  It's a whole bunch of what seems like screenshots of everything - but it's strange, the dates span from 2010...

24 Mar 2012, 1:27 AM  
Alex C

Not quite screenshots I guess... working embedded hyprlinks and everything.

24 Mar 2012, 1:43 AM  
Dikran Marsupial

the post seems to imply that the info comes from the logs, but I suspect this is BS.

I don't know how they can justify posting the identities of everybody who posts here as whistleblowing, especially as it includes many skeptics.

24 Mar 2012, 1:46 AM  
Alex C

....uh, what am I downloading?  Has this info really been public for so long?

Edit - the URL he gives is to another zip file, but I cannot decipher the text file within.

24 Mar 2012, 1:51 AM  
Dikran Marsupial

I think John would appreciate the call on this one even if it is 1am, so if anyone has his number I'd suggest they use it.

24 Mar 2012, 1:56 AM  
Alex C

I don't have his number.  I've sent him an email but of course he won't get that soon enough...

24 Mar 2012, 2:16 AM  

"A lot of everyday conversation among SkS authors reads like hateful vulgarities. From the repeating rhetoric it looks like they have a fascination with war and violence."

Where the hell did they get that stupid idea!?  Yes, people are candid, but "fascination with war and violence"?!

Well, it was almost certian to happen....I just love how this person is trying to spin this.  They cannot attack the science, so they attack the people behind SkS.  This has nothng at all to do with transparency etc.


Comments about the hack flew fast and furious on the real forum.  By this time it was 2 AM in Australia.  I took a chance and tried to Skype John, not expecting it to work.  It did, but only because he was already up and on, starting to look at things.  Logicman apparently had his phone number and had just called him to rouse him.

We looked at it together and talked, mirroring each other's tone of surprise, confusion and anxiety.  He was surprisingly awake, an effect news like this would probably have on most site administrators.  He too noticed that the presentation of the forum wasn't right.  He confirmed that no, he hadn't programmed some odd super-admin variation of the pages.  He quickly noted that it wasn't even close to right, because the data necessary to display the page like that would require combining multiple database tables, specifically the POST table, where the individual comment was stored, with the USERS table, where each user's registration information is kept.

"They cannot attack the science, so they attack the people behind SkS."

The user information was normally nowhere in the forum post pages.  The pages not only weren't formatted normally, but producing them would have required access to multiple database tables – basically the entire database – in order to construct a page that looked like that.  It was almost like the pages were constructed by someone who had uploaded their own programs onto the Skeptical Science server, or else had downloaded the entire database.  Both possibilities were chilling thoughts.  It meant that the hacker had access to every aspect of the Skeptical Science web site, and also that he had been there for a long, long time, long enough to write the programs to generate pseudo-forum pages in the fashion that served his own agenda.

He'd done the same thing to the deleted comments page, a private administrative page used by Skeptical Science moderators.  The hacker had added personal information that was normally not included, and expanded the presentation to include 3.4 megabytes of deleted comments, when the actual page that administrators use to display such information only shows the most recent fifty.

More unsettling was the dump of the users table.  The hacker had included a text dump of most of the data from the entire users table, slightly reformatted and edited.  It appeared to be a list of every user ID registered in Skeptical Science, including everyone’s e-mail addresses,.  We later discovered that a large number of pseudo-skeptics had had their own information removed from that dump by the hacker.  This in itself is a telling bit of legerdemain.  It seems that releasing thousands of people's personal information was considered acceptable, but still recognized as enough of an intrusion that fellow pseudo-skeptics were purposely and methodically culled from the release.

We later discovered that a large number of pseudo-skeptics had had their own information removed from that dump by the hacker.

Coming as it did right on the heels of the pseudo-skeptic response to the Heartland affair, it also seemed that the saying that "there is no honor among thieves" was not strictly true.  It seems that to the pseudo-skeptic crowd, it's heroic and understandable when a hack is perpetrated against people who work with or explain the science, but it's a horrific crime when committed against poor, well-meaning, tax-exempt political front organizations like the Heartland Institute.

John moved on to look at another file, one with which I wasn't at all familiar and hadn't yet grabbed from the hack.  It was entirely separate from the zip file I’d downloaded.  John explained as he reviewed it that it was a log file, generated daily, of every database SQL statement used by the web site in the past day.  He and Doug Bostrom had previously designed and implemented it to help to detect and research SQL injection attacks, which had grown more common over the years, becoming an ongoing and expanding problem.  SQL injection attacks are one of the most common and simple forms of true hacking, and had been used against Skeptical Science before.  They'd become frequent and annoying enough that special efforts were required to protect against them.

Detective Del Spooner: [to Susan, after the robots have started a revolution] You know, somehow, "I told you so" just doesn't quite say it.
I, Robot (2004)

February 23rd, 2012 — 9:07 AM AEDT — Okay, Something Very Weird Has Just Happened with the Forum

A month before the hacker released the data, just days after the initial hack, the hacker made a mistake, one that we missed.

In retrospect, we should have seen it coming.  The clues were all there, more clues than I care to admit to.  We all knew that things needed to be more secure.  The topic seemed to come up every month, in particular the idea that the forum itself should be completely separated from the Skeptical Science site to help protect it, should the more visible, public site itself ever be hacked.  And odd things were happening almost every week.  It was like that feeling you get that you're being watched, when you really are being watched.  You should turn around and check, but you don't want to feel stupid if it turns out that there's no one there.

"...we'll know soon enough if it was a hack."

But there was.

The Skeptical Science site endures frequent SQL injection attacks, as do most sites on the Internet.

The work that John and Doug had done in the past to detect them and thwart them now worked every time.  Still, you can never know that any system is completely secure.  Nor can you tell when it's a random attack by a bot that's just mindlessly hunting for vulnerable sites, or a focused probe under the direction of a determined, intelligent, human intruder.  That effort detected several attacks in 2010 and 2011, most of those originating in China.  One originating in Romania was detected on January 19th of 2012, and another coming from South Korea was detected on February 23rd, 2012.  There was another, even more troubling incident that day, which preceded the SQL injection attack.

At 9:07 AM on February 23rd, in Brisbane, John started this thread:

 23 Feb 2012, 9:07 AM Okay, Something very weird has just happened with forum
John Cook Got an email from Brian P this morning saying that the whole forum was publicly available to him, even when he wasn't logged in. I checked and this was true. A little panicky, I investigated and worked out that all the permission levels of each forum had been set down to zero. Normally, they're set so only authors can access most of them, except the translator forum is also accessible to translators. Strangely though, there is an admin forum that only admins can access and that wasn't set to zero - it was still set so only admins can access it.

I have no idea how this happened. Several possibilities come to mind. First, I did it by accident when I was screwing around with the database sometime. Someone with admin access (there are about half a dozen SkSers with this access) made the change. Or we were hacked in some way and the hacker changed the levels. None of the options seem likely to me but the most likely is human error on my part although the fact that the admin forum was still set at admin level belies some kind of blanket wiping of all levels.

So I'm a little freaked out - it's not knowing how this happened that has me most worried. Has anyone been looking at the forum and how long has this been available? But I've been procrastinating some of those security measures that have been suggested to me and as soon as I get to work this morning, am going to implement some of those measures.
23 Feb 2012, 9:13 AM  
Albatross Oh shit....we'll know soon enough if it was a hack.
23 Feb 2012, 9:19 AM  
MarkR Wasn't someone going to take a load of the old stuff offline presto?
23 Feb 2012, 9:21 AM  
logicman When I logged in I was seeing forum section names like blog posts.

The comment by Albatross was sadly, dangerously wrong.    Later on, John continued along a line that should have taken much higher priority:

23 Feb 2012, 10:44 AM Just for the record
John Cook Yes, we agreed to take all old threads offline but I just hadn't got around to it. Am going to implement that this morning as the situation has gone from back-burner to yellow alert.

That's the low lying fruit on the security to-do list, I'll then move further along the list as time permits (really intense busy period for me at UQ right now, have a big deadline tomorrow).

My own comment, the last in the thread, was much stronger in response, but at the time I still did not have access to systems, consoles, or code.  There was little I could do.  I was only just beginning to get involved, and helping to move Skeptical Science towards a more modern, ‘traditional’ code-sharing environment.  Until then, everything pretty much flew from John’s laptop onto the site, and vice versa.

24 Feb 2012, 5:36 AM  
Bob Lacatena


If you haven't already, immediately change all passwords (meaning the MySQL database passwords, root password, site access passwords, etc.).?Also, immediately grab copies of all logs (apache log, mysql log, etc.).?If logging is not turned on for MySQL you might want to look into that... the things you need to turn on to track down if you've been hacked.

This is very concerning.

Unfortunately, that's as far as that went

On the 23rd, at 12:33 PM AEDT, John started a new thread on the forum about an attempted hack that had just set off the automated alarms.

23 Feb 2012, 12:33 PM Attempted SkS hacking happening right now
John Cook

I've programmed the website to email me whenever someone tries to use SQL injection to hack SkS. Just got 334 400 emails now - each an attempt to hack SkS using SQL injection over a 4 minute period.

The IP is which is based in South Korea.

Okay, the forum glitch and now this, I'm getting a little freaked out.

After the morning’s earlier, uneventful scare, most people brushed this aside.

23 Feb 2012, 12:56 PM  
logicman Don't get paranoid - there are a lot of bots out there searching for weak defences on any site whatsoever.
23 Feb 2012, 8:39 PM  
Ricardo I concur, don't get paranoid. Any online pc sooner or later gets targeted by bots, let alone web servers.

Thomas Reynolds: Ten-year-olds go on the Net, downloading encryption we can barely break, not to mention instructions on how to make a low-yield nuclear device. Privacy's been dead for years because we can't risk it. The only privacy that's left is the inside of your head.
Enemy of the State (1998)

February 23rd, 2012 — 2:08 AM AEDT — Opening the Forum

It was February 22nd, 4:08 PM in Germany when the hacker returned, two days after his initial hack.  Tor rotated his IP address 22 times during the three hour incursion. 

At 4:14 he registered a new user, as “francois”.  He poked around for a while, doing various searches to see if that user ID could be found through the Skeptical Science search feature.  At 4:36 he used a previously hacked ID, one that he’d given administrative capabilities, to give his new “francois” user the same capabilities.  At 4:37 he started looking at the private form, starting with the topic on Moderation.  He looked next at the Admin topic, perhaps looking for clues about further administrative functions.  Then he looked at General Chat.

...at 4:51 he uploaded a program, named “f2"...

At 4:38 The German finally found a thread that interested him, titled “Sorry John”.  40 seconds later he glanced at a thread titled “I just received a letter from Heartland”.  A minute after that he entered one started by John Cook named “WOW! Peter Gleick was 'Heartland Insider'!!!”.  After reading the first page for two whole minutes, he moved on to the second page of that thread, reading that for another two minutes.

Soon after, he began to play with the administrative panel, clicking various links.  Using a function there, at 4:51 he uploaded a program, named “f2”, which allowed him to look at each of the programs which run the Skeptical Science site.  He continued looking at the code for over an hour, until 5:58 PM when he somehow altered his “francois” user, making the date of registration appear to be one day sooner,  and then returned to using the administrative user ID he’d retained from the first day of the hack.

At 6:14 PM he resumed use of his f2 program, this time to download entire directory listings of the site, including the full logs directory, as well as looking at more programs.  At 6:21 PM, he went back to using his francois ID to browse the forum.

At 6:46, he used an administrative panel to alter security on one of the forum topics, the one on Moderation.

At 6:52 PM he began, one by one, opening the topics within the private forum to public access.  At 6:59 PM he logged off and accessed a forum thread, verifying that he could read the contents without being logged on.  From there he continued to browse the forum, including a thread titled “Growing accusations of Gleick being the pdf faker!”

At 7:13 PM he uploaded a program named “un”, and at 7:14 PM he ran it, removing his f2 program from the system. He ran it again at 7:15 PM, removing itself, and with it the most obvious traces of his intrusion.

The entire foray lasted three hours and seven minutes, using twenty-two different IP addresses to execute 299 different commands.

Only two unknown IP addresses (not protected by Tor) visited the forum in that short span...

There were still no links to the forum, so if you didn’t log in with an contributor level ID, you wouldn’t know the forum existed.  But if you did know where to go, you now wouldn’t need a valid user ID to see the contents.

The forum was open for about four hours before John reset the user levels on all of the “exposed” topics.  Only two unknown IP addresses (not protected by Tor) visited the forum in that short span, one from Houston, Texas, but almost an hour after the forum had been re-secured, and one from Phoenix, Arizona, once while the forum was open, but without visiting any actual threads and so without seeing anything private, and the other several hours after the forum had been re-secured.

It may be unfortunate that John found and corrected the problem so quickly.  It would have been interesting to know with whom the hacker wished to share the contents, especially if they were not as careful about protecting their identities.

Part 3 explains SQL injection attacks, recounts the hacker's release note, and describes the hacker's activity through March.

But at the time, we foolishly didn't even recognize what was really going on within the system.

To be continued...

Posted by Bob Lacatena on Wednesday, 26 February, 2014

Creative Commons License The Skeptical Science website by Skeptical Science is licensed under a Creative Commons Attribution 3.0 Unported License.