Arguments
SkS social experiment: using comment ratings to help moderation
Posted on 8 October 2013 by John Cook, Bob Lacatena
Last week, the news rippled through the blogosphere that Popular Science had shut off commenting on their website. The reason: trolls and spambots had overwhelmed the comment threads. This is a great shame, partly because it should be avoidable. Surely a combination of technology, crowd-sourcing and manual moderation should be able to minimise the destructive impact of comment trolls.
To investigate this possibility, Skeptical Science is engaging in a social experiment. You, gentle readers, are the participants. The experiment is a University of Queensland research project, titled "Using comment ratings to facilitate moderation" (I've updated the SkS Privacy Policy to include information about this project). The goal is to investigate using user ratings to assist comment moderation, thus helping to maintain a high quality of discussion. This will be achieved simply through the use of two thumbs:
First and foremost, the point of this system is not to let people engage in climate war popularity contests. The point is not to vote up comments that support your position and to vote down comments that support their position.
Instead, users are asked to please rate comments based on their quality and how much the statements improve and elevate the discussion. Some factors that should be considered when rating either blogs or comments include:
- Civility. Is the author's tone appropriate?
- Citations. Does the author support the statements with references in the peer-reviewed literature?
- Cohesion. Is the comment concise and to the point, or does it ramble on about anything and everything?
- Accuracy. Is it true, or does it merely propagate an innaccurate myth?
- Insightful. Does it truly add to the discussion, or does it merely repeat the obvious?
- Importance. Does it matter, or is it trivial or even a distraction?
- Topical. Is it relevant, or an unnecessary distraction from the issue under discussion?
How To Rate Comments (and blog posts)
At the end of every blog post, on the right side, will be a pair of thumbs, one up (green) and one down (red) with counters.
Similar thumbs will appear at the lower right of every individual comment.
Simply click on the green thumb to give a positive rating, or the red thumb to give a negative rating. To change your rating, simply click again on the opposing thumb.
Only registered user can submit ratings
Online ratings systems are extremely vulnerable to gamification. Consequently, only registered users will be able to submit ratings. The speed-bump of having to register with a valid email address already filters out the majority of potential trolls. Being able to track each rating also makes it easy to undo the damage by a discovered troll. If you try to rate without being logged in, you will see the following message:
Registration is easy and free. Simply go to http://sks.to/register or click on the "Register Here" link in the sidebar to the left (below the Most Used Climate Myths thermometer) and follow the instructions.
How will ratings be used? Rest assured, we are not going to turn control of comment moderation over to an automated system driven by ratings (isn't that how the robo-apocalypse began in the Terminator movies?) Rather, this will merely provide assistance for moderators who will continue to moderate comments manually according to the Comments Policy.
Of course, this feature also adds another dimension to the level of interactivity with both our blog posts and comment threads. We are hopeful that this facility will be used to elevate and improve the level of discussion here at Skeptical Science.
Many thanks to Sphaerica who helped with the programming of the rating system (basically he did all the ajax coding that makes it work smoothly and seamlessly).
Well, I may be proven wrong but the up/down system doesn't do a job of moderation... Witness Zero Hedge where a high number of "greenies" only demonstrates the resonance within the echo chamber...
Call me skeptical....
[PW] I'm fairly sure JC and the rest of the SkS team aren't intending the thumbing system to take the place of regularmoderators: quite the contrary--and speaking as a moderator--if/when I see multiple thumbs-down, *geenrally* it can be taken as a sign of a *possible*vtroll/contrarian, and, for me, just aids in my 'drive-bys,' daily, of the threads.
It will be interesting to observe this system. Specifically, it will be interesting to see whether any "skeptical" comments recieve high ratings. I agree with Flakmeister - in that whenever I've seen such systems employed on blogs, they seem to have largely functioned to reinforce uniformity, or a predominant viewpoint. Even still, the system may help to reinforce a more constructive engagement within a limited range of opinions.
It would say a lot about this site if civil, well-written, and well-supported "skeptical" arguments got some high ratings. Of course, that would mean that there would have to be a substantial number of contributors who don't think that using those descriptors for "skeptical" aguments is oxymoronic.
It's not so much that this approach prodcuces uniformity, rather that in its simple implementation highly voted comments float to the top. Thus, in order to see other comments, such as highly down-voted comments, you have to scroll down and down ... most people wouldn't bother.
I encourage a look at the slashdot.org approach to crowd sourced moderation of comments comments, it works exceptionally well as a filter and yet allowing people to see the broader range of comments.
It might be interesting to note that the leading Finnish newspaper Helsingin Sanomat (www.hs.fi) uses a two-dimensional system with thumbs up/down in two categories: "I agree" and "Well argumented".
These seem highly correlated though...
The post says this: "The point is not to vote up comments that support your position and to vote down comments that support their position."
But then the post also asks us to thumb according to multiple critera including this one: "Accuracy. Is it true, or does it merely propagate an innaccurate myth?"
I feel that these two instructions are contradictory. As soon as we start to thumb based on what we think of as "true", then we are also voting up comments that support our position. What are we to do with a contrarian who is unknowingly re-stating a previously debunked myth, or a Wattsupian refugee floundering in confusion, but who is civil and respectful and genuinely here to learn? I think they should earn a thumb.
No way to vote up or down on the "Comments" stream which is where I would normally read comments. I see the posts on Feedly and I would only go the thread itself when I want to add a comment.
thumbs up voting button seems to work also.
Having the scores next to the thumbs may skew the system. People are more likely to attempt to game the system if they can see the results of their actions. I would advocate just having the thumbs if you wanted to see a truer reflection of voting preferences.
Having been a regular poster at Jo Nova's blog, where the thumbs up/down have been there for a while, I can say I quite like them.
A quick scan for lots of thumbs down leads me to comments that are more interesting. It also gave me a goal for a while - to try and get as many thumbs down as I could.
But all skeptic blogs seem to be, dare I say, less skeptical these days. Too many gullible skeptics...
Having some experience with a French information website, I would suggest introducing recommendation only. Thumbs up without thumbs down. Along with an alert button in case of uncivil behaviour. This would be a little more indicative of the added value of a contribution - not intrinsic added value but in the context of the initial article, including timeliness, on-topic,...- with all the unavoidable caveats such as confirmation bias, groupthink,... since any participative website represents some kind of community.
Besides, I would also suggest setting a limit for contributions' length, in order to avoid lengthy rants. In case of a valuable, pedagogic contribution, this limit could be exceeded with editor permission - though a suggestion for editing an article could be welcome.
I was going to make a comment about the "echo chamber" effect, but I see Flakmeister has already beat me to it. I'm not a prolific commenter here, but I think the citations "requirement" is nonsense, particularly since there's a fairly apparent double standard about posting non-peer reviewed references deemed "friendly" (like Tamino, or Tom Curtis), vs. "unfriendly" non-peer reviewed analysis.
(Rob P) - There is little point in SkS in being like most of the internet - full of unsubstantiated opinion. Peer-reviewed scientific literature is the 'gold standard' not because it is perfect, but because it is research carried out by experts and subject to the scrutiny of other experts. Rubbish still gets through though - Ole Humlum & co-authors, for instance, think human industrial emissions of carbon dioxide magically disappear.
If commenters make claims, they should be able to back it up with facts. We don't think that's unreasonable.
Thanks for everyone's comments. Re the danger of amplifying the echo chamber, there is a possibility of that but it's not inevitable - it may depend on the specific community and the specific context (e.g., the fact that only registered users can rate, that banned trolls cannot rate, the instructions provided).
Re fredb's comment about comments floating to the top, our system doesn't change the ordering of comments based on ratings. They're still ordered chronologically.
Re Harry H's comment about 2 dimensions of ratings, it's an interesting idea and I am a big fan of collecting more data. However, I'm also a big fan of keeping things as simple as possible. In this case, I think a second dimension doesn't add enough value to be worth the complication.
Leto, if someone is unknowingly repeating a myth, that should earn a down thumb. Determining a person's motives in an online comment is always problematic so we have to take the information at face value.
Scaddenp, will think about whether to add this feature to the Recent Comments page. On the one hand, it's better to see a comment within the context of the comment thread it belongs to. On the other hand, adding the feature to Recent Comments results in collection of more data. Hmm...
Chris S., I see your point about showing the result possibly skewing results. I believe Heisenberg anticipated this in the early 20th Century (you should formalise this as the Chris S Uncertainty Principle). But I think you lose more than you gain by not providing feedback.
John Brookes, I hope you don't try to earn maximum amount of down thumbs at SkS!
Yves, I see the down thumbs as filling the role of an alert button. Well, not exactly but in that general direction.
Is there any way of unrcommending a comment you've recommended? I occasionally read a comment and think it's reasonable but later in the comments thread someone gives context which shows that it was not.
OPatrick@13,
Think about this case as "something I didn't know about which sparked the discussion & sibsequently the discussion increased my knowledge".
With time, you will learn to distinguish true "skeptical ideas" from longtime debunked trolls and your recommendatrion skill will improve.
I think your desire to make recommendation "more accurate" would complicate the system with little overall benefit in terms of the data John collects from it. If I'm correct with this supposition, then John agrees according to his comment @12: enough value to be worth the complication.
Leto says:
This begs the idea that accuracy is relative, and that opinions count as accurate facts. Accuracy should not be applied to positions, but it should be applied to facts. If someone says that the globe has been cooling for the past 17 years, that is flat-out innaccurate. If they say that there has been an apparent slow-down in warming for the past 17 years, that is an arguable point and as such not strictly innaccurate.
Facts in this debate are either true, false, uncertain (but that uncertainty is supportable in the peer-reviewed literature), or they aren't facts at all but merely beliefs and opinions.
One of the big problems that false skeptics seem to have is that they are unable to separate fact from belief. They can't see the difference in their own mindset, and then they project that mindset onto others... labeling their understanding of the science as a belief rather than an acceptance of the facts.
There is a distinction, and people should vote according to that distincition.
Wired had an article up about how some research is happening into making comments sections better places. One of the ideas was to have a "Respect" button instead of Like or +/- ratings. The idea being that people are much more likely to positively rate an opposing argument if it doesn't seem like they have to agree with it. That's what "respect" seemingly does.
http://www.wired.com/design/2013/09/can-you-design-a-website-to-encourage-readers-to-consider-a-different-point-of-view/
My own view leans towards "if it ain't broke, don't fix it". I suspect most online moderators would confirm that it's difficult to keep out motivated trolls or others who would game a voting system, particularly for something as simple as a click up or down. And as Harry H. (post #4) notes, human nature makes it difficult to separate "I agree" from "well-argued".
Apart from this, early posts will receive more views and more opportunities for votes, up or down, than later ones, regardless of 'quality'. The weight really should go to well-written comments and replies, with moderators able and willing to remove posts that detract from the discussion thread (as this site does). A standard tactic that can be observed on weakly or non-moderated sites is the early hijacking of comment threads, filling them with enough nonsensical or abusive posts that subsequent readers move on rather than comment.
Sphaerica,
I agree 100% that, for most of science, truth is absolute (the exceptions are potentially interesting but not on-thread). In fact, I have less sympathy than most for a relativistic view of truth. I actually think that, for most of the contrarians coming here, they only have themselves to blame for being ignorant - I would even say that most contrarians exhibit extreme hubris when they claim to know better than the vast majority of climate scientists, often on the basis of reading anti-science blogs with way too little true skepticism...
But if accuracy is to be a major criterion for clicking the up-thumb, then we would basically be clicking to indicate which camp we are in, and the thumb clicks would then carry very little information. (It almost sounds like single-click sloganeering, to me.) This is directly at odds with the instruction "The point is not to vote up comments that support your position and to vote down comments that support their position."
We already know that comments will, for the most part, be pro-AGW (i.e "accurate") or anti-AGW (i.e. "inaccurate"), so the clicks could end up being a continuous poll of how well each camp is represented in the readership (or clickership, anyway). The clicks won't be able to register any sort of behavioural dimension or reward appropriate behaviour if they are busy performing this fairly facile polling function.
If contrarians come here to engage in honest dialog, and find every one of their comments is heavily down-clicked no matter how well behaved they are, this could undermine the very outreach function the site is trying to achieve.
And note that, by expressing this view, I am not at all suggesting that truth is relative, or that all opinions are equally valid, and no such implication was contained in my earlier post.
How are you proposing to keep posters in line with your list of factors? Will new registerees have to view a web-page telling them what factors to consider? Without making sure all new commenters are aware of them (and even then perhaps) people will just default to "thumbs up/down = I likey / I don't likey".
Sorry to comment again: I've just seen your hover text - good idea. Can I suggest an alteration? Don't have it float with the mouse, just float it to the right of the arrows when you hover, but with a link saying something like "click here to read the factors we'd like you to consider before using the thumbs up/down arrows".
Personally, I don't think a mere up/down-voting can do any good (to quality, behaviour, "culture" of a discussion/web page). Beside typical problems of web based voting systems (gaming the system, something that may well get the moderation from a "tiresome chor" to a complete and enduring hell!), these systems may be well suited for opinion based forae or "taste" orientated ( ;-) ) sites, but far less for a site dedicated to the (let's call it) veracity of arguments.
There are websites where working complex voting systems, based on deliberately chosen rules, generate a user moderation of comments (slashdot comes to mind) and which are doing a good job on that - at least on behalf of nature of those sites. But those sites are out to generate a general culture for that special place, something that often leads to the "hivemind" and should be avoided on scientific websites at all costs.
The rational arguments being advanced here as to how users might apply a rating system belies the reality of how actual real-world users on un-moderated comment boards (cough-Yahoo-cough) use the ratings. Highly motivated political ideologues on both sides, but especially among the ultra-right will vote down each and every post by anyone who disagrees with them. Respectful, knoweldgable posts citing references are likely to get a disproportionalte share of thumbs down while arrogant, snarky and factually incorrect rants are enthusiastically up-rated. The bottom line is that climate deniers are not interested in having an intelligent discussion of the issue any more than drunken soccer hooligans are interested in a cogent discussion of the weak points of their team. We should probably just be thankful that they can't start fist fights in the parking lot.
I can see how the thumbs down might help the mods. I can't really see a use for the thunbs up button here. Most threads here one must read all the comments in order including the ones that meet with disapproval or just end up confused.
Will viewers click and evaluate the references and then give a thumbs up? Or will they just make it a social status symbol like pop (non-science) forums? Or vote very quickly like the "like" button in some forums which get placed on quippy or humorous replies?
Have the whole range of policy targets turned into buttons:
> unsupported by evidence
> sloganeering
> repetition
> sounds suspiciously like Doug Cotton
> read the friggin' OP
> funny, but doesn't add to the discussion
> John Tyndall and millions of graduate students
> No. See Marcott et al. 2013
There's one big problem I see: there isn't a good way to respond to another comment anymore. It used to be that responses would be posted below, which made for a useful thread of discussion. Now that the orders are determined by votes, there isn't any way for somebody other than the moderators to post a response to a comment in a way that looks like a conversation.
If you're going to keep the ratings system, it would be really helpful to add some sort of thread tracking, so that it is possible to hold a conversation.
[Sph] The votes don't affect to presentation sequence.
silence - see John's comment at #12. It doesnt change the order of comments at all.
John,
Having worked hard to get my brain to ignore this kind of scoring system on other sites, since I view it as being intellectually shallow, I don't want to see it implemented here at SKS.
The problem with simplistic pass-fail scoring methods like this one is that they inevitably, at some level, reduce even nuanced comments to simplistic yeas and nays. Showing my bias, I find the highly visible green and red symbols on this page particularly annoying. Their color makes them more immediately visible to my brain than the comments themselves. There is nothing subtle about them, whereas the points raised in the comments often are very subtle.
Here's what I'd suggest you do if you really want to gather data: make the vote total invisible to any of us on this end. If people want to click on a thumb, let them. But don't tell us about it, and make the thumbs less obvious.
Personally, I don't need a count of thumbs to help me interpret comments. I know the ones I agree with and why, and I know the ones I disagree with and why. I'd prefer it if you let us do our own assessments without providing this kind of thing.
Ars Technica introduced a similar thumbs up/down system about a year ago, except that part of the impact is that comments downvoted by a large enough margin are collapsed from view and have to be manually unhidden to read. Once that happens, they also can't be quoted directly.
Their front-page articles about this system can be found here:
http://arstechnica.com/staff/2012/10/introducing-comment-voting-on-news-articles-and-features/
http://arstechnica.com/staff/2012/10/comment-voting-working-getting-expanded/
http://arstechnica.com/staff/2012/10/comment-voting-now-shows-the-vote-split/
They also have a phpBB forum for site feedback where some discussions about the voting system have gone on in individual threads. Seartching that forum for things liek "voting" or "rating" or "thumbs" might turn up some insightful discussions (or whiny flame-fests).
It might be a good idea to contact the site's administrators and ask about their voting system, how well it seems to be working after a year in action, etc. etc.
One of the weaknesses of a user voting system is that votes (up AND down) taper off as the pages of comments stack up. Even egregious trolling, if not reported to moderators, tends to go unchecked by 4 or so pages into the average thread. Fortunately, SKS threads rarely drag on long enough for that to be an issue. There are comment threads on Ars Technica's climate science articles that have run over a dozen pages.
Oh, and one more thing.The Ars system allows a user to re-click their vote to cancel it once it's been cast. If you accidentally thumbed a post down, you can click again to remove that vote. This isn't the most common feature of internet discussion voting systems, but it's extremely useful in my experience and I wish more places would implement it.
I just upvoted my own post. I could thumb it down, or thumb it back up, but I couldn't cancel the vote.
I agree with Don9000, but I also recognize that it might be an interesting experiment to see how much new user traffic is generated and how much of that traffic is there solely to push thumbs. Even with that research goal, I don't think it's worth it unless it's more sophisticated than thumbs up/down. This is a site that's prided itself on complex, measured responses. The thumbs thing sticks out like a . . .
Thanks all, some good comments (which I thumbed up :-)
Re the suggestion of hiding the final scores, on the plus side, this system would be just as useful whether the final scores are hidden or not. On the negative side, my sense is that would inhibit people actually using the ratings - feedback is a big part of the motivation for using it. So I'm inclined to keep the scores visible.
Someone suggested providing the option of cancelling a vote and a way to seamlessly integrate this into the system wasn't immediately apparent to me. But WheelsOC's suggestion (now thumbed up by me) of reclicking a thumb to cancel an existing rating is actually quite elegant. Will discuss this with Sphaerica.
DSL, I accept the comments that only providing up/down options is simplistic and we had discussed a more nuanced rating system (e.g., two dimensions of ratings). My concern is making it too complicated and have opted for the simplistic system to begin with. For now, I'm in the data collection stage - when enough data is collected (which will depend on the level of adoption by users), I'll see what the data is telling me.
I tend to agree with Yves #10. In my experience, a "flag as abusive" button is more useful in moderation than thumbs up/down.
No matter what criteria are expressed, these tend to end up meaning "I agree with this" or "I disagree with this," which is not particularly useful information to a moderator--lots of thumbs down simply means that the membership doesn't agree with the comment, not that the moderator needs to do something about it.
Lots of flags, on the other hand, means that the membership finds the comment to be uncivil, and therfore worthy of a glance from the moderator.
Leto @5 I do think there is a difference between statements that one should consider "inaccurate" and statements that do not support one's position. If someone cites cherry-picked data, for instance, and leaves it to the reader to draw conclusions, their statement may arguably be misleading, but it is not false. Under the current SkS system, I would probably give the comment a thumbs down, and the moderators would have to try to guess why. Under the Helsingin Sanomat system I wouldn't say "I disagree," but I would say "Poorly argumented."
Sphaerica @15 I would add that scientific theories deal not only in "facts" (i.e., observations), but also inferences drawn from those facts. Most of the time climate contrarians don't dispute actual facts, as that wouldn't get them very far as a principal approach, but inferences, like the inference that humans activity is the primary cause of the current warming trend. To you and me this inference may "feel" like a fact because we understand it to be the only reasonable inference. But I'm not sure that I would say that the cherry-picker was stating a falsehood even if he added the conclusion that "humans are not causing the warming" or "we don't know what is causing the warming." I would still just say his argumentation was poor. On the other hand, I might say he was stating a falsehood if he said something like "CO2 does not absorb longwave radiation," even though CO2 absorbing longwave radiation is technically an inference as well, although an exceedingly simple one based on the observation that every time we have shot longwave radiation at CO2, less longwave radiation comes out on the other side => "Duh."
So I guess it depends where you draw the line - on some level, we even infer that what our senses are telling us is true, because we could be hallucinating... But I think almost everyone would draw the line between fact and inference somewhere between what we sense directly, and complicated, probabilistic logical conclusions drawn from what we sense.
jdixon,
I agree with the general thrust of what you have said, and agree that there are subtleties in defining the notion of truth, as well as variations in whether a line of reasoning fails at the level of fact or inference, but all that is still somewhat tangential to my original concern.
As far as I am concerned, any contrarian post that works towards a conclusion that "AGW is not a problem" is going to fail somewhere along your line stretching from objective, checkable facts to inferences. I doubt I'll use the thumb system anyway, but if I did, I doubt that any contrarian posts would earn a positive thumb if I followed the current advice. (A contrarian might post neutrally, without supporting any position, but that is not what I am talking about; I am talking about posts that actually support a contrarian position) If their facts are wrong, I would thumb it down as obviously untrue; if the facts are correct, but cherry-picked and deliberately misleading, then I would thumb it down as leading to an untrue inferences. If their individual facts are true and they leave it to the reader to draw the false inference (a favoured tactic of some of the more sophisticated contrarians), I would also thumb it down. (If their facts are true and their reasoning is correct and they have not simply blown a minor problem out of proportion and this actually disproves AGW, then I guess 97% of scientists are wrong and we can all move on to to other pursuits.)
When proponents of AGW post, on the other hand, I can see that there might be dissociation between correctness and support for the AGW position. Someone can get to the right answer by a faulty route, but can't get to a wrong answer by a correct route.
For a contrarian, the same issue arises, but the situation is inverted.
I don't know that it will be an issue here, but on other climate change forums of which I am a member, the comment rating moderation system has been gamed, almost exclusively by denialists. The primary method so far as I can tell has been for certain users to create mutliple handles, then log in under those handles to repeatedly vote down every comment supportive of climate change theory, and to repeatedly vote up every comment dismissive of it. (At one popular board, certain valuable forum members nonetheless found themselves accumulating literally tens of thousands of thumbs down votes over the span of just a few months by people using this method.). IOW, such comment rating systems have become a sort of corrupt and reverse popularity contest, and have seemed to do more harm than good.
I like what a few others have suggested: a simply "flag as abusive" button. The problem with that, however, is that a dedicated group of troublemakers can just as easily game the system by piling on those flags, as well...
I wish Popular Science wouldn't have chosen to kill comments for now, but I can certainly understand their reasons for doing so.
I think it would be a good idea to have a third option, maybe a grey thumb pointing sideways and would mean skeptical. And providing a dropdow of the voters names when you move your mouse over a icon may stop gaming and trolling
Jim Pettit, I'm well aware of the danger of denier gamification of the rating system (I read extensively on the topic before installing this system). Gamification is slightly more difficult at SkS because in order to rate, one needs to register a user account with a working email address. It's not impossible to register multiple accounts, by any means, but raising the bar does weed out a significant number of potential trolls. Because each rating is tracked, even if someone does gamify the system with multiple accounts, if they are detected, their ratings can be instantly removed. So there is very little return on investment for deniers wishing to undermine our rating system.
But here's another interesting twist. There has been scholarly research into using social network analysis to detect sock puppets (multiple accounts by a single user) engaging in exactly this type of behaviour. If deniers do try sockpuppeting at SkS, they present a wonderful research opportunity. So bring it on :-)
What happened to the reply option?
This is my favorite subject, on which I would like to write a book discussing the propositions offered by Quillian on OFD (Open Forum control Democracy.) The key ingredient his concept adds is to quantify the voting according to the pre-qualifications of the commentator. A commentator who has a lot of approval on the subject would weigh in much more heavily than a novice, a troll, or a bot, as defined by the user community in prior commentary. It parrellels peer review by design, but in a dynamic process. I call the concept ASCEND as one chapter in my new book Pluvinergy. Although the book is about climate adjustment technology, the ability to discuss global adjustments reasonably is critical, thus the necessity of the topic.
Magma (#17) -- "early posts will receive more views and more opportunities for votes, up or down, than later ones, regardless of 'quality'" -- that leads to seeking to be first and/or last.
A solution that should be possible is for all comments, or all comments that are replies to the above-the-line article rather than to any other BTL comment, to be presented in pseudorandom order. That way, everyone gets an equal share of the first/last visibility advantage.
... and so over-quick posts seeking to be first, and over-persistent seekers of the last word, will be demotivated.
Pluvial, what you're describing sounds a lot like pagerank, the system that Google use to rank websites. How it works is each time a person gets a thumbs up, they gain some "reputation". A thumbs up from someone with higher reputation is worth more than a thumbs up from someone with low reputation. Reputation needs to be calculated iteratively - you work out an initial reputation based on simple thumbs up, then recalculate reputation with ratings weighed by the reputation of each rater, and repeat until the reputation values converge on final values.
I do plan to explore that as a possible feature to integrate into SkS but I need lots of data first. So everyone, do be sure to rate comments and blog posts in order to give me lots of data to examine.