Maximizing Revenue and Retention: Leveraging Real-Time Churn Prediction for Mobile Games (Use-case & Results)

Executive Summary

This case study explores the significant impact of real-time churn prediction on a casual mobile game. Using an accurate churn prediction machine learning model developed by askblu.ai’s team, we showcase how we improve key engagement metrics.

By Identifying high churn risk players with more than 83% accuracy, askblu.ai’s churn prediction, which is made at the start of every player session, enables targeted actions to enhance player retention and revenue. This case study is based on the live iOS game of the TV Show “Des Chiffres et Des Lettres” from France Télévision [1], DCDL for short, for which we reduce the ad frequency by half during the session for high churn risk players starting from the fourth session.

Study’s key findings:

• 5.7% player increase on average in sessions 5 through 9
• 14.4% increase in total revenue despites the ads reduction for high churn risk sessions.?

Furthermore, by personalizing the ads pressure based on churn probability, making it higher for very low churn risk sessions and lower for high churn risk sessions, even further gains in overall revenue can be achieved.?

Overall, this case study highlights the effectiveness of real-time churn prediction models coupled with in-game actions in optimizing player engagement and financial success in mobile games.

Bassel Masri - Jonathan Dixon

Why real-time churn prediction is needed

In the realm of mobile games, player retention holds the key to boosting the LifeTime Value and achieving financial success. Imagine the benefits of being instantly alerted when a player begins a gaming session and is deemed highly likely to churn (inactive for more than 7 days in a row). With this information at hand, you could promptly implement targeted actions within the game to mitigate this risk. At askblu.ai, we call this practice "Real-Time (in-game) Churn Prediction", where predictions are generated in real-time using a trained machine learning model as soon as a player initiates a new session, allowing for immediate intervention.

Churn prediction model and first results

Askblu.ai’s team has developed a cutting edge machine learning model that learns the behavior of players in the game using a carefully curated set of features derived from various events. These events encompass crucial information such as session initiation, play frequency and few more that reflect player behavior. However, as is the case with any activity, the initial few sessions or levels may exhibit significant random behavior as players determine whether they like the game or not [2]. After extensive research and analysis, our team at askblu.ai determined that starting model training and inference from the fourth session yields a favorable balance between consistent player behavior and optimal model performance.

The trained model generates churn probability predictions ranging from 0 to 1 at the beginning of each session in real-time. These predictions are converted into binary classes, namely Low Risk and High Risk, based on the following rules:

• Probability ≥ 0.5 → High Risk
• Probability < 0.5 → Low Risk

Subsequently, we can obtain a range of statistical scores for model evaluation using a separate dataset of session data that the model has never encountered during the training process. This allows for an unbiased assessment of the model's ability to generalize to unseen data in the game. At askblu.ai, our model evaluation strategy follows a threefold approach:

1 - Churn Efficiency Matrix:?

A simple 2 by 2 confusion matrix that any user can use to understand the accuracy of a predictive model by visualizing the comparison between predicted (Low Risk & High Risk) classes and actual outcomes (True Negative and True Positive). The following results were obtained on the game DCDL using 4 months of training data, from January 2023 to April 2023, evaluated on a separate data set, and is currently running on the production pipeline of askblu.ai:

The analysis of the churn confusion matrix reveals that our machine learning model accurately identified 90.7% of sessions with low churn risk and 83.7% of sessions with high churn risk despite an imbalance ratio of about 1 to 12 between the two classes. In contrast, it indicates that the model failed to identify 16.3% of true high churn risk sessions, and 9.3% of false positives were generated. Specifically, out of the total 6,409 true low-risk sessions, 597 were incorrectly flagged as high churn risk sessions which we will be discussing further in the third evaluation strategy of false false positive rate.

2 - ROC Curve and AUC metric:?

The ROC curve (which stands for Receiver Operating Characteristic) and its AUC score (which stands for Area Under the Curve) provide a statistical approach to assess the predictive performance of a machine learning model by measuring its ability to distinguish between the true churn and non-churn classes through a probabilistic framework. A random model would result in a diagonal line and a 50% AUC score, while a perfect model would generate a rectangular-shaped curve and a 100% score. We illustrate this using multiple theoretical ROC curves ranging from random to perfect, in addition to the actual ROC curve obtained for the game DCDL as mentioned earlier in the visual representation below:

The churn model developed by askblu.ai (shown in the figure on the right) demonstrates exceptional class separation ability, as evident from its well-defined ROC curve that closely resembles a rectangle with an impressive AUC score of 94%, further confirming its strong predictive performance.

3 - False false positive rate:?

In some cases, the model predicts a high churn risk, but the player continues playing for one additional session during the same day before becoming inactive for the next 7 days. Although the confusion matrix shown above may classify these predictions as false positives, they actually represent valuable insights into the model's ability to forecast churn a few hours in advance. To address this scenario, the data science team at askblu.ai has developed a novel approach to evaluate and quantify the model's early churn prediction capability for specific players when churn occurs later in the same day. These players, earlier observed within the false positives of the confusion matrix, technically did churn short after our model’s prediction.?

In fact, for DCDL’s test dataset, our analysis of the results show that 14% of the 597 false positive sessions in the confusion matrix, where a high churn risk is predicted, ended up churning later in the same day or the very next day, hence establishing a false false positive ratio of 14% where the model anticipated the churn in advance.

Enhancing player retention and revenue by reducing ad frequency in high churn risk sessions

One of the reasons that contribute to player churn in casual and hyper-casual games is ad frequency [3]. Our response to this issue is a strategy aimed at boosting short-term player retention and, consequently, increasing revenue for game studios. This strategy involves halving the ads frequency during sessions identified as having a high churn prediction.

For our analysis, we divided players into two groups: the 'blu' and 'ref' cohorts. The blu cohort served as our experimental group, where we implemented our churn model predictions to categorize sessions into Low Risk and High Risk. The ref cohort, on the other hand, functioned as our control group for comparison in the A/B testing framework. We tracked and analyzed the data from DCDL over a specific period to gauge the immediate effects of this proactive ad reduction on key engagement metrics. These included the session funnel, which is the percentage of players returning from the fourth session onwards, and the total revenue accumulated from interstitial ads and in-app purchases over particular timeframes.

DCDL’s data in May 2023 reveals a promising trend: the blu group exhibits higher retention rates in subsequent sessions following the implementation of our churn prediction model at session number 4 which is reflected in the session funnel data. This is noteworthy given that the ad reduction strategy is only applied to a subset of blu players, specifically those sessions marked as high churn risk.

A detailed session funnel comparison of a sub sample consisting of blu and ref players at session number 4 is shown in the chart below:

The numbers in the chart above clearly demonstrate the statistically significant and consistently positive impact of more blu players returning to play in sessions 5 through 9 with respect to session 4 (where the churn prediction was started), when compared against ref players. On average, the gain observed in player count through the session funnel data is 5.7%.

Turning to the impact of ad reduction in high churn risk sessions within the blu group on monetization metrics, a two-month data set from March to May 2023 highlights an encouraging trend. The blu group, in comparison to the ref group, registered a higher revenue increase, as illustrated in the following table. This table highlights the weighted average cumulative revenue, in units of monetization, for 1,000 players, from interstitial ads (Inter. Ads) and in-app purchases (IAP) for both blu and ref cohorts:

Interestingly, despite the ad reduction by half in the sessions where a high churn risk was detected by the model, the average revenue from interstitial ads remained almost the same between the blu and ref groups. This finding suggests that our strategy led to more players returning to play additional, and occasionally longer, sessions within the subsequent 7-day period. As a result, these increased player activities offset the potential revenue loss from the initial ad reduction. Meanwhile, in-app purchases saw a substantial uplift of approximately +47% within the blu group. Consequently, the overall cumulative revenue witnessed a healthy gain of +14.4% when summing all the numbers together for 1,000 players.

These two key findings support our hypothesis that a targeted reduction in ad frequency for high churn risk players can positively influence player retention and revenue generation.

Personalizing ads pressure based on churn probability

According to the flow theory [4], players have varying tolerances for ads frequency. Our analysis of DCDL’s data, conducted over several months, revealed that players who have been playing for longer time periods tend to exhibit very low churn probabilities, indicating higher engagement levels. To assist studios in increasing revenue even further than the 14.4% already established earlier, one approach is to slightly increase ads pressure (frequency) in subsequent sessions for these players.?

The data science team at askblu.ai has developed a numerical simulation, based on empirical data, to determine a probability threshold below which we can slightly increase the ads pressure without risking too much behavior changes. Consequently, our churn model can provide three possibilities based on the following scenarios:

• Probability < threshold → Increase Ads
• Threshold ≤ Probability < 0.5? → Default Ads
• Probability ≥ 0.5 → Decrease Ads

Since this feature is currently being implemented in DCDL, we have studied the theoretical implication of this ads pressure adjustment through data simulations. Thus, our analysis shows that we can achieve an approximate 13% overall increase in interstitial ads displayed over the 7-day period following our churn prediction. This is accomplished by increasing ads for sessions with very low churn risk and decreasing ads for sessions with high churn risk. Theoretically, if we plug in this extra gain in interstitial ads in the revenue table discussed in the previous section, then the resulting total revenue gain would reach up to 23% rather than 14.4% highlighted earlier.

Conclusion

In conclusion, real-time churn prediction using a cutting-edge machine learning model has proven to be a valuable tool for enhancing player retention and revenue in video games.

Askblu.ai’s churn prediction model identifies high churn risk sessions with over 83% accuracy? (with an AUC ROC score of 94%), allowing for targeted actions such as reducing ad frequency.

The results demonstrate a positive impact on key metrics, with player retention rates showing an average 5.7% increase in players coming back to play in the session immediately following askblu.ai's intervention. `

Likewise, an analysis of key monetization metrics, averaging data over the year 2023, reveals a significant 14.4% increase in total revenue from in-app purchases and interstitial ads.

Finally, by personalizing ads pressure based on churn probability, even further gains in revenue can be achieved as the numerical simulations developed by askblu.ai’s team show.

This comprehensive approach demonstrates the potential of churn prediction models in optimizing player engagement and financial success in the gaming industry.

References

[1] https://apps.apple.com/fr/app/des-chiffres-et-des-lettres/id504863867

[2] Reguera, D., Colomer-de-Simón, P., Encinas, I. et al. Quantifying Human Engagement into Playful Activities. Sci Rep 10, 4145 (2020).

[3] Genre and great games report, Facebook gaming, page 71 (2020)

[4] Daniel Rare? Obad?, Flow Theory and Online Marketing Outcomes: A Critical Literature Review, Procedia Economics and Finance, Volume 6, 2013, Pages 550-561.