Last week, the news rippled through the blogosphere that Popular Science had shut off commenting on their website. The reason: trolls and spambots had overwhelmed the comment threads. This is a great shame, partly because it should be avoidable. Surely a combination of technology, crowd-sourcing and manual moderation should be able to minimise the destructive impact of comment trolls.
To investigate this possibility, Skeptical Science is engaging in a social experiment. You, gentle readers, are the participants. The experiment is a University of Queensland research project, titled "Using comment ratings to facilitate moderation" (I've updated the SkS Privacy Policy to include information about this project). The goal is to investigate using user ratings to assist comment moderation, thus helping to maintain a high quality of discussion. This will be achieved simply through the use of two thumbs:
First and foremost, the point of this system is not to let people engage in climate war popularity contests. The point is not to vote up comments that support your position and to vote down comments that support their position.
Instead, users are asked to please rate comments based on their quality and how much the statements improve and elevate the discussion. Some factors that should be considered when rating either blogs or comments include:
- Civility. Is the author's tone appropriate?
- Citations. Does the author support the statements with references in the peer-reviewed literature?
- Cohesion. Is the comment concise and to the point, or does it ramble on about anything and everything?
- Accuracy. Is it true, or does it merely propagate an innaccurate myth?
- Insightful. Does it truly add to the discussion, or does it merely repeat the obvious?
- Importance. Does it matter, or is it trivial or even a distraction?
- Topical. Is it relevant, or an unnecessary distraction from the issue under discussion?
At the end of every blog post, on the right side, will be a pair of thumbs, one up (green) and one down (red) with counters.
Similar thumbs will appear at the lower right of every individual comment.
Simply click on the green thumb to give a positive rating, or the red thumb to give a negative rating. To change your rating, simply click again on the opposing thumb.
Online ratings systems are extremely vulnerable to gamification. Consequently, only registered users will be able to submit ratings. The speed-bump of having to register with a valid email address already filters out the majority of potential trolls. Being able to track each rating also makes it easy to undo the damage by a discovered troll. If you try to rate without being logged in, you will see the following message:
Registration is easy and free. Simply go to http://sks.to/register or click on the "Register Here" link in the sidebar to the left (below the Most Used Climate Myths thermometer) and follow the instructions.
How will ratings be used? Rest assured, we are not going to turn control of comment moderation over to an automated system driven by ratings (isn't that how the robo-apocalypse began in the Terminator movies?) Rather, this will merely provide assistance for moderators who will continue to moderate comments manually according to the Comments Policy.
Of course, this feature also adds another dimension to the level of interactivity with both our blog posts and comment threads. We are hopeful that this facility will be used to elevate and improve the level of discussion here at Skeptical Science.
Many thanks to Sphaerica who helped with the programming of the rating system (basically he did all the ajax coding that makes it work smoothly and seamlessly).
Posted by John Cook on Tuesday, 8 October, 2013
 |
The Skeptical Science website by Skeptical Science is licensed under a Creative Commons Attribution 3.0 Unported License. |