NN-based predictive analytics for prepaid mobile services in telco sector
Altice Labs
We develop innovative products and services for the telecommunications and information technology market.
This study delves into the application of predictive analytics within the telco sector, with a particular focus on prepaid mobile services. It investigates the prediction of customer behaviors, such as the top-up propensity within 2 to 4 days and account balance before top ups. The goal is to empower telco operators with data-driven insights to tailor their marketing strategies more precisely.
This research evaluates the effectiveness of Neural Network (NN) models, underscored by rigorous hyperparameter tuning and cross-validation processes, against the traditional Machine Learning (ML) models currently in production at Altice Labs. Innovatively, it incorporates pre-processing and feature selection techniques not previously used in traditional ML model development. The results demonstrate a significant performance leap of NN models over existing ML counterparts in accurately predicting customer actions. By providing telco operators with a more nuanced understanding of customer behavior patterns, this study offers insights into enhancing predictive models in the telco sector.
Keep reading to learn more about the daily performance evaluation of our optimal Neural Network models or download the full white paper.
In the ever-evolving telco sector, understanding and predicting customer behavior, especially in the realm of prepaid mobile services, is crucial for tailoring effective marketing strategies. Prepaid services offer users control over their spending and avoid the contractual obligations associated with postpaid plans. However, the unpredictability of top-up events poses challenges for telco operators in their efforts to optimize marketing campaigns that meet individual customer needs.
This article presents a comprehensive study focusing on leveraging Deep Learning (DL), specifically Neural Networks (NNs) to enhance predictive models within the prepaid mobile services domain. By accurately predicting customer top-up behavior and account balances, telco operators can more effectively time their marketing communications, thereby increasing the relevance and efficiency of promotional efforts.
Our primary objective is to assess the viability of NNs in these specific use cases. We evaluate the performance of NN models against established Machine Learning (ML) models, namely, Random Forest (RF) and Gradient Boosting Trees (GBT), previously deployed by Altice Labs’ FOCUS team. Our study aims to explore the potential of NNs to surpass these models by incorporating innovative pre-processing and feature selection techniques.
This study addresses the urgent need to reverse the declining trend and encourage the adoption of prepaid services, particularly in Portugal. Accurate predictions can significantly enhance customer experiences by allowing for the delivery of personalized offers at the most opportune moments, fostering customer loyalty and satisfaction. The introduction of NNs into this context aims to provide a level of accuracy in predicting customer behavior that traditional ML models may not achieve, thus supporting telco operators in crafting more effective and personalized marketing campaigns.
State of the Art
In the world of predictive analytics for the telco sector, accurately predicting customer top-up behavior for prepaid mobile services remains a challenge due to the limited number of publicly available studies on the topic. However, by examining available predictive model research, valuable insights can be gathered to inform our investigation.
P. M. Alves et al.'s work underscores the importance of detailed customer profiling in enhancing service quality and retaining customers, emphasizing predictive analytics’ role in understanding and anticipating customer actions. This aligns with our focus on predicting top-up propensity and account balances. Our research explores NN models for their potential to offer more accurate predictions in specified timeframes.
The application of Recency, Frequency, and Monetary value (RFM) analysis in previous studies for customer data analysis, mirrors our approach to feature engineering. We also consider seasonal trends in top-up behavior to refine predictive modeling.
Churn prediction studies provide a foundation for adapting methodologies for top-up behavior prediction. The success of Deep NNs in outperforming traditional models in churn prediction studies hints at their applicability in predicting top-up behaviors.
Feature selection emerges as a critical step in refining predictive models, with methodologies aimed at identifying the most impactful variables for customer behavior prediction. The use of Mutual Information (MI) and SHAP emphasizes the elimination of redundant features to enhance model performance.
Resampling techniques such as SMOTE address class imbalance issues, relevant to our problem of dealing with an imbalanced dataset.
The reviewed literature offers methodologies and insights that inform our approach. This study aims to extend these methodologies to predict top-up behavior and balance before top up in prepaid mobile services, providing telco operators with actionable insights for effective marketing strategies.
Methodology
A.?Top-up Propensity Prediction in 2, 3, and 4 Days (binary classification)
Our analysis utilizes 60 days of historical data, focusing on top-up and balanced history to generate aggregated features for our model. These features are specifically tailored to reflect customer engagement within this timeframe. We set a specific reference day, typically at the start of the month, to anchor our analysis consistently. The concept of a top-up cycle, represented by the orange and red periods in Figure 1, is central to our feature construction. For feature construction, we consider the period up to the day before the reference day, as shown by the brackets in Figure 1. The period following the reference day, including the day itself, is used for label construction.
Labels are assigned based on whether a customer tops up within a set interval (2, 3, or 4 days) from the reference day, creating binary classification models. Customers topping up within this period are labeled ‘1’, and those who don't are labeled ‘0’. In Figure 1, the propensity periods are marked by dotted lines: purple for two days, green for three, and yellow for four, starting from the reference day.
Given the unbalanced nature of our binary classification datasets, we must carefully select our evaluation metrics. Therefore, we use the F1-Score as our metric, balancing precision and recall through their harmonic mean, providing a more accurate assessment for the minority class in our classification problem.
B. Balance Before Top-up Prediction (regression problem)
This approach mirrors the top-up propensity prediction, focusing on the last two months leading up to a chosen reference day. The features considered are identical to those in the top-up propensity prediction, emphasizing the relevance of both the last top-up cycle and the last n days leading up to the reference day.
For balance prediction, we label data based on the customer's account balance right before their first top up following the reference day, which includes the day itself. Our model training involves analyzing the month after the reference day to predict the balance. This results in fewer data samples for training compared to the top-up propensity predictions.
In this regression problem, we employ two evaluation metrics: the Mean Absolute Error (MAE) and the Hit Ratio. MAE calculates the absolute difference between actual and predicted balances before top up, directly measuring prediction accuracy. The Hit Ratio assesses our model's ability to anticipate top ups based on balance evolution, but we balance these metrics to ensure predictions reflect real balances accurately.
C.?Daily Performance Testing
1.?Top-Up Propensity
We evaluate the model's daily performance using a sliding window approach over the test month. Each day serves as a reference point for predicting customer top-up propensity within the next 2, 3, or 4 days. We analyze the model’s F1-Score, precision, and recall daily to identify performance trends.
2.?Balance Before Top-Up Prediction
Similar to the top-up propensity model, we employ a sliding window approach for feature construction. For balance prediction, we extend our analysis beyond the training set's constraints, observing the evolution of predicted balances daily. We employ the Hit Ratio and MAE to assess our model's success in anticipating top ups and ensuring the accuracy of predicted balances.
领英推荐
Feature Engineering
We chose July 3rd as the training reference day to capture a high activity phase, enriching our dataset with diverse customer interactions. Testing spans from August 2nd to October 2nd, with October 2nd serving as the test reference date to examine performance across a different activity range.
Our datasets include 58 aggregated features of customer top-up and balance behavior. For top-up propensity, we generated three separate datasets tailored to 2, 3, and 4-day predictions. The balance prediction dataset is slightly smaller due to our label generation approach for balance prediction.
We optimized NN architectures through extensive feature engineering, hyperparameter tuning, and cross-validation. Feature selection involved evaluating all 58 features, selecting subsets based on MI scores, and applying SHAP analysis for further insight into feature importance.
We experimented with normalization techniques and class imbalance strategies like SMOTE and undersampling. Different loss functions were tested to improve class differentiation in the unbalanced scenario, aiming to identify the most effective NN architecture and feature set for each use case.
Best Performing Neural Networks
A.?Top-Up Propensity Use Case
We identified the most effective model architecture for predicting top-up propensity within 2 days. Applying the same architecture to the 3-day and 4-day predictions revealed that different sets of input features yielded optimal results for each timeframe. Min-max scaling and using the original class distribution without applying balancing techniques achieved the best performance.
For input features, utilizing all 58 features was most effective for the 2-day prediction, while selecting a subset of features through SHAP analysis improved performance for 3-day and 4-day predictions, resulting in the use of 25 and 29 features, respectively. ?
The optimal model architecture, consistent across the three prediction intervals, involved training for 100 epochs with batch size 64, using the Adam optimizer and weighted binary cross-entropy as the loss function. To prevent overfitting, L1 regularization [27] with a learning rate of 0.001 was applied to the second hidden layer.
B.?Balance Before Top-Up Use Case
The model achieved superior performance using a subset of 11 features selected based on MI scores and min-max scaling for data normalization. The training involved 150 epochs with batch size 32, utilizing the Adam optimizer at a learning rate of 0.001, and L2 regularization in both hidden layers.
Additionally, early stopping with a patience setting of 30 epochs was implemented to ensure model generalization.
Comparison Between Neural Networks and Traditional Machine Learning Models Daily Performance
In this section, we evaluate the daily performance of our NN models throughout the test month of October and compare these findings with the outcomes achieved by the FOCUS team using traditional ML models, offering insights into the NNs' real-world applicability.
The FOCUS team previously implemented an RF classifier for the top-up propensity prediction and a GBT regressor for balance prediction. Meanwhile, we employed NN models with different feature selection methods, resulting in more precise predictions.
A.?Top-Up Propensity Use Case
Our NN consistently outperformed the RF classifier across all three prediction time frames in October, achieving higher precision without compromising recall.
Both models displayed similar temporal performance trends, suggesting a common sensitivity to time-related behavioral shifts.
B.?Balance Before Top-Up Use Case
Throughout October, our NN regressor consistently achieved a higher Hit Ratio and significantly lower MAE compared to the GBT regressor, indicating more accurate and reliable predictions.
Despite performance fluctuations, our NN regressor maintained effectiveness in predicting the balance before top up with remarkable accuracy.
Authors
Keywords: Predictive Analytics, Telco Sector, Prepaid Mobile Services, Customer Behavior; Top-Up Propensity, Account Balance, Neural Networks, Machine Learning, Random Forest, Gradient Boosting Trees
Contact us if you want to engage in a deeper discussion on this topic!