Since we released The Justice Review a few months ago, weve been monitoring The Tribunal and would like to share a few changes weve made that are subtle, but critical. These changes are aimed at improving accuracy, providing voters more relevant information, and helping to identify the most toxic players even faster. I want to quickly highlight these changes and explain the logic behind them:
Previously, our Player Support team reviewed every Tribunal case before approving punishments and pardons. After reviewing the past 12 months of data, weve seen that cases that are unanimously pardoned or punished tend to be extremely accuratePlayer Support rarely overturns these decisions. In light of this data, were now going to focus our attention on the cases that are more ambiguousthe slight majorities and split decisions. This should improve The Tribunals accuracy by focusing on the cases that need the most attention.
Weve tweaked how The Tribunal builds cases: Instead of 100% random sampling, The Tribunal will attempt to find the games that ultimately sent each player to The Tribunal. By selecting recent games that had a big impact in sending the player to The Tribunal, you should see the average number of reports against a player increase per game.
The Tribunal system is a constant balance between speed and accuracy. In the past, we were concerned that forcing The Tribunal to find more games per case meant that toxic players were left unchecked for longer than necessary. However, we realize that more information for our judges means more accurate verdicts, and recent optimizations have allowed us to change The Tribunal so that every case will have at least two games to review instead of one.
Players have often said they forget how many cases theyve done on a given dayweve added a progress bar that will outline how close you are to maxing out your daily case limit.
These changes are already live, so go check them out and let us know what you think! Were confident that these changes will improve the accuracy of The Tribunal system without compromising on speed.
Visit the Tribunal today and help us make a difference in the community!
I wish people could review more cases. 20 seems like such a small number
We may increase this soon. We just want to be confident that there are very few abuse cases and that the cap doesn't actually burn players out. Due to Justice Reviews, a lot of players will feel pressure to finish the max number of cases everyday and we have to be careful increasing the caps because of this.
So Player Support was still checking cases overall? Or just the Permabans? I think that support should be checking every case before marking someone as a criminal. I hope we'll get a feature against Punish spammers soon.
We recently did an analysis on the types of players that visit the Tribunal, and very few players actually spam Punish. In fact, they are such a minority that spam punish voters rarely have the influence to affect the outcome of a case.
As a second data point, depending on the month, punish rates in the Tribunal range from 60-70%. We have to remember that the punish rate actually doesn't matter that mean--what matters is whether those players deserve the bans.
When we measure the success or accuracy of the Tribunal, we actually look at statistics like: "What is the false positive rate?" In other words, how many players are being punished when it is undeserved? We try to ensure that the false positive rate is low as possible, which is why the Tribunal is actually designed to be quite conservative.
Dr. Lyte, I was doing tribunal when a thought occurred to me: what is your stance on behavior progression in relationship to tribunal cases? Let me elaborate: say a player has a track record of raging in game and get reported consistently enough to get into tribunal. Now consider that this player has actually shown behavioral progression in the positive- that is this player is more considerate of how he is behaving and how it effects other players' playing experiences. They get reported in some games still, but overall shows slight progression. I'm wondering if Tribunal already takes this into consideration, because now I have the nagging though in the back of my head that I may, in fact, be helping to punish a player that is actually showing progression without having been punished first. Although on the other hand, one could say that the punishment is still deserved for past grievances.
What is your stance, or rather the stance of the Tribunal, on this perspective?
Most of our systems tend to be lenient towards toxic players that are improving their behaviors. We really value and respect players who make a genuine attempt at reform. We have been brainstorming on whether showing 'behavioral progression' might be valuable in the Tribunal... but it's a tricky issue.
1. Occasionally the consequences of spam punish was overlooked. We thought that was awesome and have found ways to make it happen more.
2. Wow, and premade teams can abuse the tribunal system on their poor team even more. Did you even think on this one more than a minute?
3. So it's more likely now that no matter how nice the player the odds are greater the one game they did anything wrong will make it onto the report. Instantly justifying a ban for all the nonsense reports that came with it. All hail spam punish.
4. Sitting around waiting to spam punish is hard without opening other windows, and we understand you lose track of what you're doing by not paying attention. Here is a counter.
I'd be open to discussing your Reform Card if you want to post it. I'd be curious whether the community finds your behavior acceptable.
First off I'd like to say I love the Tribunal system and the idea of having the community involvement.
As far as the changes go I could not be happier for having games that will more likely show the reason for the player being reported. That will make our job so very much easier. I think one additional change could help with accuracy even more, and perhaps weed out false reports, and that is to either A. Not include reports where the player hasn't typed in the reason for the report, or B. Force players to type at least a certain number of characters if they are going to report. I feel as a player, that if someone is toxic enough for me to be reporting them that I should take the time to give the reason why I am doing so which should also make it easier for Tribunal judges to determine the accuracy of my report and the toxicity of the player.
We've considered forcing players to write a report comment in the past, but there are a few concerns:
1) If we make the limitation "any number of characters," then many players will simply scribble random nonsense just to file a report.
2) If we make the limitation too long, then many players might not report players they should because they don't really want to deal with writing a comment.
It's a tough balance between #1 and #2, but it's something that's not entirely off the table.
Quote:
Secondly, as has been said by many a red post, the people who end up in the Tribunal are reported and reported often. So, I have a very hard time understanding why there would be cases with only two games. I've seen it myself where each game has just one report(with almost always no additional information), and yet this player is supposedly so toxic that they need to be punished. Now as you've said that the Tribunal will be looking for the most recent and egregious offenses I can possibly see a case with just two games where there are maybe 3+ reporters per game, but I still feel as if this person is in the Tribunal there will (or at least in my mind should be) be AT LEAST 5 games in which the player has been reported so that we can always have the most information when making our decisions.
We can actually set the minimum number of games to whatever we want; however, adjusting the minimums directly impacts the amount of time it takes the Tribunal to aggregate the case file and send the player to the community review. Regardless, we have been experimenting a lot with minimums, and have learned that it's a variable we have to be careful with. Since we've had a random number of cases between 1-5 for over a year, if we immediately changed the minimum to 5 for every game we potentially have to deal with a voter bias because players might perceive cases with 5 games as worse because they are so used to seeing cases with 1-4 games as well.
We are going to experiment longer term with 2, and monitor the data and go from there.
Quote:
I've actually had another idea which I'm sure many other players have had as well, and that is based on the fact that it has been said that people who falsely report have their reports disregarded by the system. Now that is great, but what if you were to, besides each report put some sort of rating system, or perhaps that reporters previous accuracy in finding toxic players. That way when I look at a game with one report and the reporter's accuracy is rather low, I can go in being skeptical of the person actually being guilty.
This is an interesting idea, I'd have to think about it more and go through exercises of how this new information might influence player review of the cases.
"Previously, our Player Support team reviewed every Tribunal case before approving punishments and pardons."
So you guys were checking each and every tribunal case before approving punishes and pardons? Am I understanding this right? I was under the impression that it was only permaban cases that were reviewed by player support.
For a long, long time, we were reviewing every case. In many international territories, we are still reviewing every Tribunal case because we are still optimizing the accuracy in those regions.
If your team checks each tribunal case, why do we even have the tribunal?
During our random player behavior research sessions, we found out that the typical League of Legends player is pretty positive--we have lots and lots of data suggesting that this is the case. The reason player behavior is a problem in League is because of context--this is a hardcore, competitive, team-based game where victory or defeat is dependent on your teammates. Not only that, it's an online game where players have less accountability and more anonymity. Many social science studies have actually shown that contexts can powerfully influence people to behave in toxic ways.
Our philosophy on the player behavior team is to develop systems and tools to empower our players to directly shape the community and to add more accountability to online gaming. Could we simply remove the Tribunal and just keep reviewing every case? Of course. But, if our goal is to change the culture of online games... we need systems like the Tribunal and Honor, and we need players who want to make a difference. This is why we always say that fixing player behavior is a collaboration between Riot Games and you--the players.
Quote:
If I go for 1 year without being banned will I still be permed?
If you behave for long periods of time, your account's toxic history does slowly decay away and you will not be up for a permanent ban.
So Lyte, as a avid Tribunal Ban Inquiries goer as you know,
What is the update with adding Pre/Post game chat into Tribunal logs? Is that still on the list of priorities or is PB&J team and you putting it on hold while you put in changes that are easier to implement?
We're putting pre- and post-game chat logs on hold because the team has been working on a different project that we believe is higher priority.
I believe in what we are working on, but it's not related to pre- and post-game chat logs which is why I haven't had much to say about the recent threads around this issue.
I have another question.
Is the hall of justice still planed?
Does the rating about tribunal worries you, how people in generall will react to this or don't you have enough contributer to release it?
Do alot of things speak against the hall of justice, that you haven't considered before?
Do you need more time or is it planned to release the hall of justice with possible rewards and you want to avoid missing some people who deserve it?
I just wonder why, since i think no word was fallen about it and it was announced together with the overworked tribunal.
Halls of Justice is on hold. A lot of players voiced concerns that they may not want to reveal that they are Tribunal users or specifically highly ranked Tribunal users. This might attract attention from toxic players in the game who want to be vindictive. We respect these concerns and took the idea back to the drawing board, and we are figuring out different ways to create incentive structures in the Tribunal system.
Halls of Justice might re-appear in a different form, or we might come up with something else--we'll have to see.
New Updates to the Tribunal
These changes are already live, so go check them out and let us know what you think! Were confident that these changes will improve the accuracy of The Tribunal system without compromising on speed.
Visit the Tribunal today and help us make a difference in the community!
As a second data point, depending on the month, punish rates in the Tribunal range from 60-70%. We have to remember that the punish rate actually doesn't matter that mean--what matters is whether those players deserve the bans.
When we measure the success or accuracy of the Tribunal, we actually look at statistics like: "What is the false positive rate?" In other words, how many players are being punished when it is undeserved? We try to ensure that the false positive rate is low as possible, which is why the Tribunal is actually designed to be quite conservative.
What is your stance, or rather the stance of the Tribunal, on this perspective?
2. Wow, and premade teams can abuse the tribunal system on their poor team even more. Did you even think on this one more than a minute?
3. So it's more likely now that no matter how nice the player the odds are greater the one game they did anything wrong will make it onto the report. Instantly justifying a ban for all the nonsense reports that came with it. All hail spam punish.
4. Sitting around waiting to spam punish is hard without opening other windows, and we understand you lose track of what you're doing by not paying attention. Here is a counter.
First off I'd like to say I love the Tribunal system and the idea of having the community involvement.
As far as the changes go I could not be happier for having games that will more likely show the reason for the player being reported. That will make our job so very much easier. I think one additional change could help with accuracy even more, and perhaps weed out false reports, and that is to either A. Not include reports where the player hasn't typed in the reason for the report, or B. Force players to type at least a certain number of characters if they are going to report. I feel as a player, that if someone is toxic enough for me to be reporting them that I should take the time to give the reason why I am doing so which should also make it easier for Tribunal judges to determine the accuracy of my report and the toxicity of the player.
1) If we make the limitation "any number of characters," then many players will simply scribble random nonsense just to file a report.
2) If we make the limitation too long, then many players might not report players they should because they don't really want to deal with writing a comment.
It's a tough balance between #1 and #2, but it's something that's not entirely off the table.
We are going to experiment longer term with 2, and monitor the data and go from there.
So you guys were checking each and every tribunal case before approving punishes and pardons? Am I understanding this right? I was under the impression that it was only permaban cases that were reviewed by player support.
Our philosophy on the player behavior team is to develop systems and tools to empower our players to directly shape the community and to add more accountability to online gaming. Could we simply remove the Tribunal and just keep reviewing every case? Of course. But, if our goal is to change the culture of online games... we need systems like the Tribunal and Honor, and we need players who want to make a difference. This is why we always say that fixing player behavior is a collaboration between Riot Games and you--the players.
What is the update with adding Pre/Post game chat into Tribunal logs? Is that still on the list of priorities or is PB&J team and you putting it on hold while you put in changes that are easier to implement?
I believe in what we are working on, but it's not related to pre- and post-game chat logs which is why I haven't had much to say about the recent threads around this issue.
Is the hall of justice still planed?
Does the rating about tribunal worries you, how people in generall will react to this or don't you have enough contributer to release it?
Do alot of things speak against the hall of justice, that you haven't considered before?
Do you need more time or is it planned to release the hall of justice with possible rewards and you want to avoid missing some people who deserve it?
I just wonder why, since i think no word was fallen about it and it was announced together with the overworked tribunal.
Halls of Justice might re-appear in a different form, or we might come up with something else--we'll have to see.