The first online player behavior experiment at Riot Games

This is Part 3 in a behind-the-scenes look at the player behavior journey at Riot Games, where a team was challenged with the task of improving the behaviors of online communities. This post details the first player behavior experiment we ran at Riot Games, and is a story that has become somewhat of an urban legend at the company in current day storytelling.

Previous posts can be found here: [Part 1] [Part 2]

---

"Default to trust."

The first project for a new team is always a risky one. Although the newly formed Team Player Behavior understood some of the underpinnings of negative online behaviors, it was difficult to predict any interventions that would be a guaranteed success. A failure right out of the gate might cast an overwhelming shadow on the team, given the perceived impossible challenge at hand of changing online player behavior. 

As a team leader, you could opt for the "low hanging fruit," the projects that would have some guaranteed low value for low scope--a "team proof of concept" so to speak. As a visionary, you could chase the impossible and hope there is enough time to learn from the mistakes. Perhaps it was youthful exuberance, or simply being naive, we decided to go for the big plays.

In League of Legends, there are several channels of communication. There is personal chat, where players can talk in private to another individual player. There is team chat, where players can talk to their teammates and then there's "All Chat" (also known as cross-team chat) where players can talk to both teammates and opponents in the game. In 2012, approximately 54% of games had some amount of cross-team chat, so it wasn't the chattiest channel but the source of a fair amount of toxicity. 

"What if we made cross-team chat an opt-in feature?" 

We could turn the cross-team chat channel off by default, and force players to go through a menu option to opt-in. We borrowed some simple natural language processing techniques to establish baseline sentiment for cross-team chat, and began designing our experiment when the doors busted open and a torrent of feedback crashed through the team.

Quite a few Rioters were pissed.

My inbox was flooded with e-mails from concerned colleagues about the potential dangers of the change. "League is a competitive game, it is a social game, what if this change makes the game anti-social?", "What if this experiment makes chat a barren wasteland we can't come back from?", "Competitive banter is a part of the game! We should never touch chat." 

"I will never play League of Legends again if your team does this experiment."

My cell phone buzzes. It is a text from Marc "Tryndamere" Merrill, the president of Riot Games. Ah, crap. But fortunately, it simply said, "You are going to want to get ahead of this and do a townhall AMA or something." 

Whew, I still have a job. I just had to do a townhall--a company meeting where my colleagues can ask anything they wanted about the project, and I needed to convince my co-workers that this was not the end of the world but also worth trying. After a few days, I realized that this was a pretty normal response for a company. In the last few months, 2 scientists joined the company and immediately were suggesting ways to design, iterate, test and measure initiatives in pretty different ways. If there is one thing I know as a former psychologist, it is that people hate change. Change is scary.

So I turned to my boss, Tom "Zileas" Cadwell and asked for advice before the company townhall. He laughed and said one simple line:

"Pretend you are the CEO of the company and just do what you think is right." 

At Riot Games, one of our pillars is "default to trust." When I stood up in front of the company at the townhall and was being introduced, I knew that this would be the first test of the company's pillars. I could recognize a few of the Rioters in the audience--the ones who had written angry e-mails to the team. 

With the other scientist, Davin "davin" Pavlas and our producer Carl "statuskwoh" Kwoh, we explained the hypothesis behind the experiment. When you look at things like organ donation programs, making the program "opt-in" versus "opt-out" had a dramatic impact on the rates of engagement. For example, this New York Times article suggests that in Germany, an opt-in organ donation system had an engagement rate of 12%, while Austria (which uses an opt-out system) had an engagement rate of 99%.  

However, in League of Legends, making something opt-in versus opt-out might also change chat behaviors by changing the context of the cross-team chat channel. One, we are not going to show which players had opted-in to cross-team chat so you are never sure who is in the audience. We argued that toxic players generally lashed out in the hopes of invoking a response and the possibility of cross-team chat not having their target audience could reduce the negative behaviors. Two, by making cross-team chat off by default, we could shield new players in the game from the potential of negative behaviors. 

The company decided to trust us. They trusted the team.

We ran the experiment over 2 weeks: 

- Negative chat dropped 32.7%.
- Neutral chat dropped 1.9%.
- Positive chat increased 34.5%.

In the following months, we also saw meaningful drops in number of "Offensive Language," "Verbal Abuse," and "Negative Attitude" complaints filed by players.

Critically, the number of games that had cross-team chat only dropped 1% to 53% of all games and we did not see the cross-team chat toxicity shift to any of the other chat channels.

After the experiment, we noticed that 79% of players had opted-in to cross-team chat which was much higher than traditional opt-in programs; in fact, most of these players immediately opted back into cross-team chat as soon as the experiment went live. But, we still saw dramatic effects because we had changed the context and potential audience of cross-team chat.

We also saw 98% of games end with the phrase "gg" or "ggwp" meaning "good game" or "good game, well played." In League of Legends and other competitive games, these phrases represent the handshake at the conclusion of a competitive match in sports like basketball or soccer. 

This was the moment we knew that online behavior was a solvable problem.

GGWP.

- Jeff