Posted on 6 March 2014 by Bob Lacatena

Part 3 describes SQL injection, the text of the hack release, and the hacker's activity leading up to the release.

Dennis Nedry: Haahaaa... I am totally unappreciated in my time. You can run this whole park from this room with minimal staff for up to 3 days. Do you think that type of automation is easy... or cheap? Do you know anyone who can network 8 connection machines and debug 2 million lines of code for what I bid for this job? Because if he can I'd like to see him try.
— Jurassic Park (1993) —

What is Skeptical Science?

November, 2007

To understand the hack, it is important to first understand exactly what Skeptical Science is, in terms of technology.  Skeptical Science is not composed of your usual prefab blog software.  Most bloggers sign up for an account at blogger.com or wordpress.com or typepad.com.  Those sites provide them with everything they need to manage a blog, from software for editing new posts to comment functionality, sidebar widgets, customizable themes and automatic backups of their data, which happen with complete transparency in the background. The blogger never even knows how much is being done for them.

A few intrepid bloggers go so far as to register their own domain name, lease time through a web hosting service, and install and run their own copy of the wordpress software, or some other popular blogging software package.  For them, the installed package is still taking care of almost everything, while the web host may take care of the rest (such as those all important backups).  The blogger may expand his or her blog’s capabilities by installing wordpress plugins or even some basic web hosting tools, but that’s it. 

Skeptical Science is nothing like that.

August, 2011

Skeptical Science is a full-blown, database-backed application, currently consisting of 83 database tables with over 800,000 distinct rows of data, and 203 programs that comprise 41,355 lines of code (not counting javascript and shell scripts).  It was initially conceived as an encyclopedic database of climate myth rebuttals. The blog was added in response to suggestions from readers.  The original point, setting it apart from any ordinary blog, was to catalog the many, many oft-repeated pseudo-skeptic myths, along with a clear explanation for why each of them is totally, provably wrong according to peer-reviewed science.

John Cook programmed all of the site’s functionality, from the ability to register and log in and post comments, to his own ability to write and edit blog posts, arguments, and rebuttals, to the search functionality and ancillary pages.  Over time, John added more and more functionality.  He added different presentations of the arguments.  He expanded the arguments to offer basic, intermediate and advanced versions.  He offered translations in different languages for both posts and argument rebuttals.  In time, the site grew to include a Firefox add-on and iPhone applications, the automatic glossary, the Trend Calculator, the Interactive Climate Science History, everything related to The Consensus Project, and more.

Each of these features that you see is backed by a lot of work that you don’t see.  With blog posts and argument rebuttals come the need to enter and edit the posts, to upload images to go with the posts, and to schedule their publication.  With multiple authors come tools to support many authors working side-by-side, in their own environments.  With comments come a host of comment moderation tools.  To support language translations, the database and programs must provide ways to specify the language and the complete translation, as well as navigation to those translations. 

When the team grew large enough that the forum became necessary, John programmed all of the functionality behind that, too, from the display to the ability to start and add to comment threads, to the ability to search the forum.  It also includes a peer-review feature allowing team members to review and approve each others' posts and rebuttals, a mechanism contributing to the high quality nature of our content. As a result, Skeptical Science content has been republished in textbooks, TV documentaries, university curricula, newspapers, books, Senate floor speeches and Presidential tweets.

May, 2012

The birth of the forum itself was probably a seminal moment for Skeptical Science, because it helped to turn a group of like-minded authors into a cohesive, cooperative community.  Skeptical Science grew into a true team endeavor, with scientists, authors, artists, programmers and other contributors all working together to make sure that we got the science right, that we stayed up-to-date, and that the site continued to grow into a valuable tool positioned against what amounts to a lot of very loud, blustering misinformation that echoes around pseudo-skeptic blogs.

John, alone, programmed 97% of that, with only small elements being added by myself and a few others, after we came on board soon after the hack.  At the same time, in the early years he also created all of the content, and has continued to contribute along with his other roles.  That initial compilation of data by itself was a massive job.  I don’t want to think about the tremendous amount of work and skill that went into first building and then evolving a complex, custom-designed, content-heavy site like Skeptical Science.  Putting this all together was no simple chore, and getting everything 100%, rock-solid correct was never going to happen.  There were bound to be flaws in the architecture.

A site like this needs more than programming and content, too.  Skeptical Science, not the web host, is responsible for arranging and making data backups.  We are responsible for managing performance and site throughput and denial of service attacks.  We’re also responsible for all security.

The web host gives us a box (physical or virtual), an administrative login,  and a “good luck.”  Everything else you see here is up to John, and over time to a dedicated crew of volunteers who admire what he’s done and are willing to help with some of the heavy lifting.

February, 2014

What does all of this functionality mean?  It means two things.  The first is that the site is worth hacking.  There’s no point to hacking a site that has nothing more than blog posts and comments, other than to either corrupt the contents until they can be restored from a backup, or to put up a “You’ve been hacked by…” page, just to prove you’ve done it.  But with a private forum and more extensive content, there’s suddenly a reason for someone to hack into the site — if they don’t like what the site is doing, and want to do anything they can to interfere with the objective of the site.

The second thing that such extensive functionality does is to make the site more vulnerable.  Every mis-step in programming becomes a potential avenue for intrusion by a hacker.  Every function and every detail, no matter how minor, may unexpectedly open an avenue up for an intruder.

In this case, the hacker didn’t like what Skeptical Science does — that is, keeping people informed about the truth behind the science and the misrepresentations by the pseudo-skeptics.  He also found those programming mistakes which let him worm his way in to do what he wanted to do, which was to try to undermine the credibility of a small group of hard-working, well-intentioned and very dedicated volunteers.

Angela: Just think about it. Our whole world is sitting there on a computer. It's in the computer, everything: your, your DMV records, your, your social security, your credit cards, your medical records. It's all right there. Everyone is stored in there. It's like this little electronic shadow on each and everyone of us, just, just begging for someone to screw with, and you know what? They've done it to me, and you know what? They're gonna do it to you.
— The Net (1995) —

Recapping Events To This Point…

Before continuing, it’s probably worth recapping the events that have been presented so far:

Kevin Flynn: On the other side of the screen, it all looks so easy.
— Tron (1982) —

March 23, 2012 — 1:47 PM EST — Stumbling First Steps

At the time of the hack, I didn't yet have access to any programs unless I explicitly requested one and John e-mailed it to me.  I certainly didn't have access to the site itself, or its database.  That was restricted to Doug Bostrom and John alone.  That in turn meant that, at least on the day of the hack and for a few days after, I couldn't look at log files or code, and even if I could I really was so unfamiliar with it that I wouldn't have known where to start.  Of course, one learns fast under that sort of pressure.

We were severely handicapped by our web host at the time, one we have since abandoned for many reasons, not the least of which was their lack of support in resolving the hack.  John had chosen to use them in 2007, when he first set up the site as a one-man operation, in part because they made many aspects of running a web site easier.  He had enough other things to do with programming and creating content.  Everything was done through browser control panels, so everything was quick and easy.  They automatically took care of a lot of details, like making certain that no directory listings were ever available.  As a result, you couldn’t see the listing of SQL injection files in the logs directory because the web host automatically shut such access down.

Most problematic, on the heels of the hack, we did not have quick, easy access to the apache logs.  Those log files record every bit of visitor activity in the system.  That’s how we’d find out how the hacker got in, and exactly what he'd done once there.  We also didn’t have “shell access,” command-line access that lets us really dig into the logs in a quick, efficient manner.

Because of this, for the next few days, each of us, Doug, John and I, had disparate tasks.

Doug had the most pressing and important task, that of getting the logs into an environment where we could properly research the hack. 

After reporting the hack to the Federal Police and the Queensland Police, John’s first task was to delete all of the SQL injection log files still on the site, and then to turn off that particular function, at least until we’d resolved the details behind the hack.  After that, his time was mostly spent first shutting down and then migrating the forum.  No one wanted to keep using it, knowing that a hacker could be watching every word.  A surprising amount of important, day-to-day activity goes on in the forum in support of the site.  Having the forum down was a major inconvenience.  We found alternatives in the interim, but it was important for John to finally get the forum separated from the site, immediately, and then up-and-running again in a more private and secure location, so that daily business could return to normal.

Thomas Gabriel (hacker): Launch the downloads!
— Live Free or Die Hard (2007) —

March 23, 2012 — 2:12 PM EST – Whack-a-mole

As for my part, I had little more to do than to find out where the main hack file was located, to see if I could get it taken down by the Russian file sharing host.

So I journeyed into what I expected to be the dark, nefarious underworld of backwater Russian file servers, to visit the site with the hacked file.  What I discovered was that the file had been uploaded to a personal website set up by a young man who was into the first-person shooter game Counter-Strike, a game that ironically itself has been hacked and exploited so the players can cheat.  As part of his site, he had set up a file server to which anyone could upload any file — presumably files associated with the game.  Uploads could be done with no security, no registration, and no password whatsoever.  All you had to do was to select the file and click "submit."  [This aspect of his site has since been locked down.]

I got an idea.  The site seemed to be very, very basic.  I tried the simplest of things, uploading a file with the same name as the hacked Skeptical Science file, "sks.zip".  Most sites would refuse by telling you the file already existed or would automatically rename the new upload to give it a different name, like "sks-1.zip", to avoid any conflict.  But the site was obviously so basic, and knowing programmers as I do, there was every chance that the programming was sloppy.

As I'd hoped, it didn't either refuse or rename my file.  I quickly confirmed that the site had uploaded my file, simply overwriting the original file in the process.  In seconds I wrote a simple program to create a garbage file tens of megabytes large, large enough to reasonably appear to be the hack file. I named it sks.zip.  If you tried to double click it, since it wasn't actually a zip file but had a zip file extension, any unzip program would try to open it and just tell you that it was corrupted.  Nothing else would or could happen.

I uploaded it and watched the comments appear, complaining that the hack was a bust, and the file was corrupted.  I watched and waited for the hacker to figure out what I'd done, periodically using google or scanning the blogs that cared enough to discuss the contents of the hack.

From then on, until Doug had things set up, I split my time between looking for new posts of the hacked material and discussing possible avenues for the hacker with John and Doug, via e-mail.

Just after midnight on March 24th I noticed that a new comment pointed to a new file, hosted on a French server.  I e-mailed them, and by 3 AM they had deleted the file.  I repeated this with a Russian server after that.  They gave me the IP address that uploaded the file, which interestingly traced to Malaysia, but wasn’t a Tor relay node.  The next day I had it deleted from crocko.com.

It didn’t take a lot of brain power for this, just persistence, but that would change soon enough.  The real “fun” was about to start.

Gabriel: [when Stan fails to hack the Dept. of Defense network in 60 seconds] Too Bad ! You are gonna die !
— Swordfish (2001) —

The Locked Room, The Ticking Clock

A locked room mystery is one where the crime is committed under seemingly impossible circumstances.  In the usual scenario, a murder victim is found dead in a room that has been locked from the inside, making it seemingly impossible for anyone to have committed the murder without still being inside of the locked room.

The hack, at the beginning, looked like a locked room mystery.  At first, the SQL injection logs had looked like the obvious point of entry, but how did the hacker get them without first getting into the system?  But if he was in the system some other way, then the SQL injection logs were irrelevant.  It didn’t add up.

The fact was that the SQL injection logs could simply not have generated the data released in the hack, both because they did not go back as far as all of the data, and because it was simply impossible to construct everything from just the logs, especially the entire users table, which predated the logs, and years worth of posts — more than 46,000 of them — from the forum.

There was no way the SQL injection logs were it.  They might, just might, have been the first point of entry, but there had to be something more.

Sadly, there was.  A lot more.

But more importantly, there was an urgent need to find it.  What all of us realized was that the hacker was still in there, until we found out how, and closed the hole.  Until we figured everything out, without flaw, he was inside and we had no idea what he was able to do, from lurking in the forum to destroying data to bringing the site down.

Part 5 explains how apache logs are structured, and explains some of the avenues the hacker used in his attack.

To be continued...