Tobold's Blog
Wednesday, May 25, 2011
Judged by a jury of your peers

Riot Games sent out a press release to me and many game sites (and then curiously failed to post it on their own press release site) about launching their tribunal system for League of Legends. The tribunal is a system where reported cases of bad player behavior are being judged by a jury of other players, who take on the job of GMs to decide whether another player should receive a temporary ban as punishment. Jurors who vote with the majority (and thus are considered "reasonable") are even paid in influence points, the virtual currency of the game.

Player-controlled systems to weed out bad behavior is something that many people have been asking for. The reason why not more games have it is that previous implementations failed spectacularly, giving us for example The Sims Online mafia which extorted other players for money by threatening to blackball them. It is easy to see why a reputation point system would never work for example for solving the conflicts between tanks, healer, and dps in a World of Warcraft group: The dps would never admit that them pulling aggro or standing in the fire is not the tank's or healer's fault, and would just outvote them unless you gave the tank and healer 3 votes each.

Riot Games realized that the problem of most reputation systems is that the judgement is done by the people present at the scene, who are rarely impartial. Thus in the tribunal system the judgement is done by random strangers, who are provided complete chat logs and other relevant information to base their decision on. This much diminishes the chance of the jurors acting out of spite, or blackmailing the defendant.

I believe that their system has a chance to work much better than previous ideas because of the jurors not being the witnesses at the same time. The only flaw I can find in the system is that it is effectively asking players to perform a job for free which the company would otherwise have to pay GMs for. But as long as the players are more than willing to do so, and the response time for reporting bad behavior improves a lot due to that, the overall effect is probably positive. What do you think?
The one issue I have, and I suppose it comes down to the "community" of League of Legends, is how juries tend to lean towards vindictiveness from the get-go. I served jury duty on a criminal case, for example, and the sort of default mindset was that the guy was guilty before the lawyers even begun to make their case. After all, if he was so innocent, how did this make it all the way to trial?

From the sounds of it, this system looks more like an survey-form than an actual deliberative jury, and you only get "paid" if you vote with the majority. Ergo, if the community tends to lean towards pardons, I can see the votes going that way Prisoner Dilemma-style. Honestly, the player should get paid regardless of which way they vote, so that consensus does not influence justice.

On the other hand, this is sort of like Rift's "free transfers" press release. A lot of free publicity over an gimmick that does not actually mean much.
I'm not really worried about the tyranny of the majority, because Riot Games has auditors to review the judgments.

I'm more worried about potential privacy violations. Those chat logs might contain information that identify the defendant and/or the plaintiff in real life. Sure, most of the time these details are irrelevant to the jury, but you just need one case to get to the headlines. And Riot Games auditors can't scrub the logs beforehand without negating the timesaving benefits.
I believe that knowing the name of the summoner on trial is a bad idea. It can influence (for better or worse) the impartialness of the jurors. Imagine User1 was trying to offer some creative advice and support for User2 and User3, but used some colorful words. It could go either way. But imagine instead LolatMyTeam was offering "creative advice" to User2 and User3. Guilty immediately. It isn't necessarily a bad thing to let jurors be influence by a name, but it feels like it goes against what Riot is trying to accomplish - the whole impartial thing. They even addressed this slightly in their explanatory video.
I would love to see something like this used in conjunction with player run cities.

If you violate some rule (perhaps even player made rules), they can ban you from the city, fine you, or otherwise punish you.

Of course as a player, you have the choice to simply ignore the judgement and never return to that city. So players that abuse the system will simply have lonely cities.
Impartiality doesn't really work if you're a player and you run the chance of having already interacted with some of the folks involved.

"Huh. I know that guy. I've seen him shouting out front of the Bank before... he's a total d-bag. He might be reasonable in this instance, but it's a pretty good bet this chatlog missed him being a d-bag earlier."

Names need to be removed. Otherwise... it's not perfect, but what system is?
Giving other players access to chat logs is unacceptable. That's a major invasion of privacy.
i think it's a brilliant idea, and the chat logs could be anonymized. Juries could be selected from a pool of people volunteering for it.

On the other hand, it is just morally acceptable, not more efficient. Bad behavior will stop only IF it is punished, not because the punishment is just or informed or whatever.

So i think personally bad behaviour is better handled by a swift kick administered by sufficient manpower, whatever the system :)
it seems Riot are on the right track with this.

it is a system that keeps the community in check and has a subsystem to keep itself in check too - that's brilliant

the raised concern of mentioning real names is valid IMO - names in chat logs and the actual nickname of the "accused" should probably be anonymized - f.e. replaced with A, B, C, D etc.

this could probably work with MMOs too and would slash the costs of Support to some extent :)
It does seem very random.

I don't however see the privacy violation. If you harass someone in tells in an online game you would not normally divulge personal information would you? "You are a retard and should l2p, PS I live at 17 Acacia Drive SW3" is certainly something I've never seen. Or do you feel that every communication in a massive online game is private which seems something of a tautology?

I wish them luck with it. And I can see this working in WoW. A bunch of party chat comments about some Mage needing on Plate, think we'd not need to be in the group to be able to judge that.
I'm really sceptical about giving the players the power to ban other players. And there are a lot of gaps in their system, about which I'm really curious. How will they select the jurors? Will they get instructions how to handle the cases? Will there be access to some sort of case-law or another framework? Will they be from the same server, or from another server?

If they are from the same server you can get all sorts of corrupion and bribery (if you don't pay me I''ll vote for you banishment), especially if the names aren's anonymous (privacy issues aside). I wonder what this will do if the defendant knows the real name of the juror (or the other way around).

Also there doesn't seem to be a way to adjust the punishment.

I agree with Azuriel that it's a mistake to only reward the jurors for voting with the majority. This way jurors will first consider what other jurors will vote, instead of focussing on the case at hand or "the truth". For this system to work well, you should get the infuence points regardless of which way you vote.

A lot of people will probably give an emotional response instead of a factual one (besides considering their own gain in influence points).

On the other hand, it's a game, so it's fun to see what will happen.
I don't however see the privacy violation. If you harass someone in tells in an online game you would not normally divulge personal information would you?
The Internet Tough Guy will. When he cannot win a dispute in-game, he will threaten to escalate it to real life. He'll supply his real-life information to show that he's not afraid of a face-to-face confrontation. He'll accuse the opponent of cowardice or being a nolifer if the opponent doesn't accept their challenge. Most of them are bluffing, but you really don't want to meet those who mean what they say.
As an avid LoL player and firm supporter of the Tribunal, I think I can add a couple points to clarify some concerns:

1) Names - one of the violations summoners are asked to judge is whether or not a summoner name is inappropriate. Hard to do if you can't see the name. That said, I think there's a good argument to remove this portion from the Tribunal and keep everyone anonymous. Especially because there are reports of Tribunal judges finding accused summoners in game and then harassing them. Honestly, I expect this change to happen at some point in the near future.

2) Degrees of Punishment - I think Riot did well by this. Judges can't vote on a punishment for people - they can only vote in favor or against punishment. Riot actually metes out the punishment (according to them, they have a multi-tiered violation system whereby the first instance of "punishment" results in a warning, then a series of suspensions, finally culminating in a permaban that is automatically reviewed by a Riot rep).

3) Tyranny of the Majority - I think there is definitely a presumption of guilt amongst the community. However, in the first week of the Tribunal, the cases I have seen haven't been close. Each summoner up for judgment has at least 8 games (sometimes up to 12) of clear undeniable ToS violations, either afk (level 6 of 18 after 45 minutes with the chat log recording "fuck you I'm leaving!"), or intentional rage-feeding (all movespeed items, stupidly high death counts, and the chat log recording "fuck you now I feed"), or verbal harassment ("fuck you nigger fag asshole noob fuuuuuuuukkkkkk"). If the bar for who goes to the Tribunal stays as high as it is right now, it won't matter what the presumption is.

4) Privacy - As far as I know, the ToS makes it clear that chatlogs are not private conversations. No one could reasonably expect that their chat log would be private, especially now with all of the hype about the tribunal. I don't buy any of the privacy arguments with the exception of the names issue already discussed.

Overall, I hope the system ends up being effective, because I think Riot did a good job setting it up.
This comment has been removed by the author.
The dps would never admit that them pulling aggro or standing in the fire is not the tank's or healer's fault, and would just outvote them unless you gave the tank and healer 3 votes each.

It all depends on what you actually want to count. If you want to see who underperformed/wiped the party this time, a vote for 'the loser' would never work. Instead all players should pick 3 people that performed the best lining them in an orderly fashion (assigning 3,2,1 points) with the voter exluded. You'll be left with a quite a decent statistic on who stood in the fire and who did his stuff well.
I am confused; I assume these tribunals would not be about incompetent players (stand in the fire.) ( Instead of Godwin's let's use the Bolshevik or French revolution tribunal analogy) That would be bad.

I am skeptical this is a good idea. But the problem is so bad, it is probably a reasonable thing to try. In fact, this is the motivation as to why players would spend their time to volunteer. I had a healer leave in the middle of a ZA fight yesterday.

A better community is very valuable to me and thus to the publisher.


To tie this back to an earlier post: maybe lots of games will allow designers to try out innovations and see what works and what doesn't. And they can eventually be implemented in the AAA titles.
I should say though, in defense of The Sims Online, Peter Ludlow's "The Second Life Herald" includes several chapters of his time spent in that game, and he doesn't make the game as bad as it seems. Yes, the Sims Mafia did exist (several families actually), but similarly where there's crime, there was also justice; counter-mafia groups rose up and there was a sort of government that stamped out these mafia groups for quite a while. In addition to stamping out Mafia groups, they appropriately down-repped scammers and griefers so that newer players would know to avoid such places.

So, going off of Ludlow's work, I wouldn't exactly say TSO "failed spectacularly".
"The only flaw I can find in the system is that it is effectively asking players to perform a job for free which the company would otherwise have to pay GMs for."

You can use influence points (IP) to purchase new champions. The only other way to acquire new champions is to pay Riot Points (RP) which you buy with your credit card. It's not much, but the IP reward has measurable cash value, making this definitely not "working for free."
To echo what Chirstian said:

I have had the opportunity to use the tribunal system and much of the fears that being brought up in these comments are unfounded.

All the cases I have reviewed have been heavy repeat offenders with multiple people often from both teams giving their summary of the offense in addition to chat logs and statistics. There is also a limit to the amount of cases that can be judged in a day: 3 and there is a timer preventing the player from voting as soon as a case loads on screen.

The concerns about personal information are real but perhaps overstated. There is nothing private about your chat logs in the first place as they are visible to your teammates at all times and it there was anyone most likely to take offense and then real world action it would be the wronged party.

One final note is this system is only usable by those who have a max level account and this is not nearly as easy time wise as getting a character to max level in most mmorpgs on the market.

I think perhaps the best use of this system is not the "free help" Riot is getting from this but the perception of the community. How often have you reported a vile player in your favorite or an obvious botter and seen no action taken for long periods? It incenses your sense of justice and right and wrong. The tribunal can dissipate much of this community angst by letting players say "I know justice is being served against players like Mr. Asshole who I reported last game because I just ruled against this Mr. Bigot in the tribunal."
Interesting comments. I once met a US superior court judge at a party. Lots of criminal cases over 20 years.

I asked if she thought more guilty people got off, or more innocent people went to prison (it was the OJ Simpson trial era).

She didn't pause an instant and said that far more "innocent" people went to prison.

At least, they were innocent of the crime being charged, not necessarily innocent "in general".

I bit of a frightening admission, and the frustration with the bureaucracy was her defense.

In a virtual game situation, with little to no personal ramifications, a peer jury system would go Lord of the Flies as fast as you can say "kill the beast, kill him dead".

I used to blithely check "yes" on votes to kick in heroics, just because, "hey, if someone initiated a vote, there must be a good reason."

Then I got roundly chastised by someone who patiently explained the situation and pointed out that the initiator was actually the idiot, and the kickee was really upset.

If I was blithely voting yes in the interest of "lets get this over with", I doubt many others were any more thoughtful.

Point is, video game players are not going to give a jury sytem the serious attention it deserves.

They will grind axes or take the power position either for grins, or in the interest of a swift conclusion to "get on with it".
Syncaine also says that the cases he's seen have been open-and-shut cases. It would be interesting to follow up on this after a few months to see whether swifter justice will work as a deterrence or does the "quality" of the cases remain constant.
The Sims Online mafias weren't created to weed out bad behavior. They were the bad behavior themselves.
Bristal, while video game players will not give the jury system the respect it deserves, neither do people in general give the jury system by which we actually convict people as much respect as it deserves.

In fact, almost all of the criticisms I've seen of this system apply equally to actual real life juries. People will vote one way or another just to get things done faster, they'll have a presumption of guilt when it should be the other way, they'll form an opinion of someone based on their name or how they look.

People are awful, and juries are made of people. I still can't think of a better way to do things, either in real life or in a game.
I would say that a system like this, done correctly, would be one of the more positive features of any game, and may even *gasp* help strengthen the concept of a community.

The 'jurors' would have to fit several criteria (can't be in a guild with either person, can't be on friends/ignore lists of either person, and certainly can't have been present at the scene of the altercation).

Granting some sort of incentive to participate, and some sort of incentive to vote with the majority would be a start, but I get the itching suspicion that more might be needed to help ensure people made 'reasonable' choices.
Post a Comment

<< Home
Newer›  ‹Older

  Powered by Blogger   Free Page Rank Tool