A Hack by Any Other Name — Part 2

Part 1 recounts the initial steps of the hacker, and our initial discovery of his intrusion.

Angela: They hack into computers and they cause this chaos.
— The Net (1995) —

March 24, 2012 — 1:06 AM AEDT — The Early Conversation

Meanwhile, the real forum suffered through a torrential storm of activity on the "SkS was hacked" thread.

Comments about the hack flew fast and furious on the real forum.  By this time it was 2 AM in Australia.  I took a chance and tried to Skype John, not expecting it to work.  It did, but only because he was already up and on, starting to look at things.  Logicman apparently had his phone number and had just called him to rouse him.

We looked at it together and talked, mirroring each other's tone of surprise, confusion and anxiety.  He was surprisingly awake, an effect news like this would probably have on most site administrators.  He too noticed that the presentation of the forum wasn't right.  He confirmed that no, he hadn't programmed some odd super-admin variation of the pages.  He quickly noted that it wasn't even close to right, because the data necessary to display the page like that would require combining multiple database tables, specifically the POST table, where the individual comment was stored, with the USERS table, where each user's registration information is kept.

The user information was normally nowhere in the forum post pages.  The pages not only weren't formatted normally, but producing them would have required access to multiple database tables – basically the entire database – in order to construct a page that looked like that.  It was almost like the pages were constructed by someone who had uploaded their own programs onto the Skeptical Science server, or else had downloaded the entire database.  Both possibilities were chilling thoughts.  It meant that the hacker had access to every aspect of the Skeptical Science web site, and also that he had been there for a long, long time, long enough to write the programs to generate pseudo-forum pages in the fashion that served his own agenda.

He'd done the same thing to the deleted comments page, a private administrative page used by Skeptical Science moderators.  The hacker had added personal information that was normally not included, and expanded the presentation to include 3.4 megabytes of deleted comments, when the actual page that administrators use to display such information only shows the most recent fifty.

More unsettling was the dump of the users table.  The hacker had included a text dump of most of the data from the entire users table, slightly reformatted and edited.  It appeared to be a list of every user ID registered in Skeptical Science, including everyone’s e-mail addresses,.  We later discovered that a large number of pseudo-skeptics had had their own information removed from that dump by the hacker.  This in itself is a telling bit of legerdemain.  It seems that releasing thousands of people's personal information was considered acceptable, but still recognized as enough of an intrusion that fellow pseudo-skeptics were purposely and methodically culled from the release.

Coming as it did right on the heels of the pseudo-skeptic response to the Heartland affair, it also seemed that the saying that "there is no honor among thieves" was not strictly true.  It seems that to the pseudo-skeptic crowd, it's heroic and understandable when a hack is perpetrated against people who work with or explain the science, but it's a horrific crime when committed against poor, well-meaning, tax-exempt political front organizations like the Heartland Institute.

John moved on to look at another file, one with which I wasn't at all familiar and hadn't yet grabbed from the hack.  It was entirely separate from the zip file I’d downloaded.  John explained as he reviewed it that it was a log file, generated daily, of every database SQL statement used by the web site in the past day.  He and Doug Bostrom had previously designed and implemented it to help to detect and research SQL injection attacks, which had grown more common over the years, becoming an ongoing and expanding problem.  SQL injection attacks are one of the most common and simple forms of true hacking, and had been used against Skeptical Science before.  They'd become frequent and annoying enough that special efforts were required to protect against them.

Detective Del Spooner: [to Susan, after the robots have started a revolution] You know, somehow, "I told you so" just doesn't quite say it.
— I, Robot (2004) —

February 23rd, 2012 — 9:07 AM AEDT — Okay, Something Very Weird Has Just Happened with the Forum

A month before the hacker released the data, just days after the initial hack, the hacker made a mistake, one that we missed.

In retrospect, we should have seen it coming.  The clues were all there, more clues than I care to admit to.  We all knew that things needed to be more secure.  The topic seemed to come up every month, in particular the idea that the forum itself should be completely separated from the Skeptical Science site to help protect it, should the more visible, public site itself ever be hacked.  And odd things were happening almost every week.  It was like that feeling you get that you're being watched, when you really are being watched.  You should turn around and check, but you don't want to feel stupid if it turns out that there's no one there.

But there was.

The Skeptical Science site endures frequent SQL injection attacks, as do most sites on the Internet.

The work that John and Doug had done in the past to detect them and thwart them now worked every time.  Still, you can never know that any system is completely secure.  Nor can you tell when it's a random attack by a bot that's just mindlessly hunting for vulnerable sites, or a focused probe under the direction of a determined, intelligent, human intruder.  That effort detected several attacks in 2010 and 2011, most of those originating in China.  One originating in Romania was detected on January 19th of 2012, and another coming from South Korea was detected on February 23rd, 2012.  There was another, even more troubling incident that day, which preceded the SQL injection attack.

At 9:07 AM on February 23rd, in Brisbane, John started this thread:

The comment by Albatross was sadly, dangerously wrong.    Later on, John continued along a line that should have taken much higher priority:

My own comment, the last in the thread, was much stronger in response, but at the time I still did not have access to systems, consoles, or code.  There was little I could do.  I was only just beginning to get involved, and helping to move Skeptical Science towards a more modern, ‘traditional’ code-sharing environment.  Until then, everything pretty much flew from John’s laptop onto the site, and vice versa.

Unfortunately, that's as far as that went

On the 23rd, at 12:33 PM AEDT, John started a new thread on the forum about an attempted hack that had just set off the automated alarms.

After the morning’s earlier, uneventful scare, most people brushed this aside.

Thomas Reynolds: Ten-year-olds go on the Net, downloading encryption we can barely break, not to mention instructions on how to make a low-yield nuclear device. Privacy's been dead for years because we can't risk it. The only privacy that's left is the inside of your head.
— Enemy of the State (1998) —

February 23rd, 2012 — 2:08 AM AEDT — Opening the Forum

It was February 22nd, 4:08 PM in Germany when the hacker returned, two days after his initial hack.  Tor rotated his IP address 22 times during the three hour incursion. 

At 4:14 he registered a new user, as “francois”.  He poked around for a while, doing various searches to see if that user ID could be found through the Skeptical Science search feature.  At 4:36 he used a previously hacked ID, one that he’d given administrative capabilities, to give his new “francois” user the same capabilities.  At 4:37 he started looking at the private form, starting with the topic on Moderation.  He looked next at the Admin topic, perhaps looking for clues about further administrative functions.  Then he looked at General Chat.

At 4:38 The German finally found a thread that interested him, titled “Sorry John”.  40 seconds later he glanced at a thread titled “I just received a letter from Heartland”.  A minute after that he entered one started by John Cook named “WOW! Peter Gleick was 'Heartland Insider'!!!”.  After reading the first page for two whole minutes, he moved on to the second page of that thread, reading that for another two minutes.

Soon after, he began to play with the administrative panel, clicking various links.  Using a function there, at 4:51 he uploaded a program, named “f2”, which allowed him to look at each of the programs which run the Skeptical Science site.  He continued looking at the code for over an hour, until 5:58 PM when he somehow altered his “francois” user, making the date of registration appear to be one day sooner,  and then returned to using the administrative user ID he’d retained from the first day of the hack.

At 6:14 PM he resumed use of his f2 program, this time to download entire directory listings of the site, including the full logs directory, as well as looking at more programs.  At 6:21 PM, he went back to using his francois ID to browse the forum.

At 6:46, he used an administrative panel to alter security on one of the forum topics, the one on Moderation.

At 6:52 PM he began, one by one, opening the topics within the private forum to public access.  At 6:59 PM he logged off and accessed a forum thread, verifying that he could read the contents without being logged on.  From there he continued to browse the forum, including a thread titled “Growing accusations of Gleick being the pdf faker!”

At 7:13 PM he uploaded a program named “un”, and at 7:14 PM he ran it, removing his f2 program from the system. He ran it again at 7:15 PM, removing itself, and with it the most obvious traces of his intrusion.

The entire foray lasted three hours and seven minutes, using twenty-two different IP addresses to execute 299 different commands.

There were still no links to the forum, so if you didn’t log in with an contributor level ID, you wouldn’t know the forum existed.  But if you did know where to go, you now wouldn’t need a valid user ID to see the contents.

The forum was open for about four hours before John reset the user levels on all of the “exposed” topics.  Only two unknown IP addresses (not protected by Tor) visited the forum in that short span, one from Houston, Texas, but almost an hour after the forum had been re-secured, and one from Phoenix, Arizona, once while the forum was open, but without visiting any actual threads and so without seeing anything private, and the other several hours after the forum had been re-secured.

It may be unfortunate that John found and corrected the problem so quickly.  It would have been interesting to know with whom the hacker wished to share the contents, especially if they were not as careful about protecting their identities.

Part 3 explains SQL injection attacks, recounts the hacker's release note, and describes the hacker's activity through March.

But at the time, we foolishly didn't even recognize what was really going on within the system.

To be continued...

Posted by Bob Lacatena on Wednesday, 26 February, 2014

The Skeptical Science website by Skeptical Science is licensed under a Creative Commons Attribution 3.0 Unported License.