Predicting Customer Churn in Telecommunications: A Machine Learning Approach

Introduction :

In today’s highly competitive telecommunications industry, retaining customers is crucial for business success. Customer churn, the phenomenon where customers switch from one provider to another, poses a significant challenge for telecom companies. To address this challenge, leveraging machine learning (ML) techniques has become increasingly essential. In this article, we dive into the application of ML in predicting customer churn, exploring how advanced analytics can empower telecom companies to proactively identify and retain at-risk customers. By harnessing the power of data-driven insights, telecom providers can enhance customer satisfaction, optimize retention strategies, and ultimately drive business growth

Project structure

The steps involved in this project are as follows;

1. Business understanding
2. Data understanding
3. Data cleaning
4. Answering of Business Questions and Visualizations
5. Summary and recommendations.
6. Hypothesis Testing
7. Data preparation
8. Modelling
9. Evaluation

Technical content

1.Business Understanding

Description : This project aims to develop a machine learning model to predict customer churn in a telecommunications company. By leveraging historical customer data, including usage patterns, demographics, and service subscriptions, the model will identify customers at risk of churning. This predictive capability will enable the company to implement targeted retention strategies and improve customer retention rates.

Problem Statement: Telecommunications companies face significant challenges in retaining customers due to the competitive nature of the industry and the ease with which customers can switch providers. Customer churn, the rate at which customers discontinue their services, can have a substantial impact on revenue and profitability. Therefore, it is crucial for telecommunications companies to proactively identify customers at risk of churning and implement effective retention strategies.

Objective: The objective of the project is to develop a machine learning model that accurately predicts customer churn for a telecommunications company.

Business Success Criteria: The success of the project will be measured by the model’s ability to accurately identify customers at risk of churning, thereby allowing the company to implement proactive retention strategies and minimize customer attrition. Specifically, achieving a high accuracy, precision, and recall in predicting churn will be key success metrics.

Select Technologies and Tools: Choose appropriate machine learning frameworks (e.g., TensorFlow, scikit-learn) and data processing tools (e.g., pandas, SQL) for model development. Decide on visualization libraries (e.g., Matplotlib, Seaborn) for result interpretation.

Risks and Contingencies: Identify potential risks such as data quality issues, model overfitting, or regulatory compliance. Develop contingency plans to address these risks and mitigate their impact on project timelines and outcomes.

Cost Benefit Analysis: Conduct a cost-benefit analysis to determine the financial implications of implementing the churn prediction model compared to the potential revenue losses resulting from customer churn

HYPOTHESIS:

(H0)Null Hypothesis: There is no significant relationship between MonthlyCharges and whether a customer churns

(H1)Alternative Hypothesis: There is significant relationship between MonthlyCharges and whether a customer churns

Business Questions:

1. What is the relationship between totalcharges and customers churning?

2. What is the relationship between monthlycharges and customers churning?

3. Which customer gender churned the most?

4. Between the male and female gender who were charged the most on a monthly basis?

5. Customers that churned were mostly using which type of InternetService?

2.Data Understanding

It is imperative to gain a comprehensive understanding of the underlying data before predicting customer churn. Data was retrieved from different sources including; SQL databases, one-drive file and from a Github repository.

Before loading data we had to import the necessary packages into the notebook:

We then proceeded into loading the dataset from the different sources.

After collecting the data sets , loading them and concatenating them, we went ahead to conduct Exploratory Data Analysis (E.D.A) of the data set to get an overview of our data. We discovered the following findings:

Issues with the dataset:

1. The TotalCharges column and the Tenure column have the wrong datatype

2. There are missing values in the TotalCharges column

3. The customerID is not necessary in building the ML model

Course of Action

1. Correct the Total Charges column datatype i.e using ‘pd.to_numeric(df2[‘TotalCharges’], errors=’coerce’)’

2. For missing values we will leave them for now until when building pipelines

3. Drop the customerID column as it is not used or necessary

Findings:

1. The train-df dataset has 21 columns and 5043 rows

2. Most customers were of the male gender, were not SeniorCitizens , lacked dependents and patners

3. The average monthlycharges was around 65 and majority of them did not churn

4. Majority of the customers prefferred Fiber optic internet service and a month to month type of contract

5. The customers were averagely charged 64.7 monthly with the highest charge on monthly basis being 118.65

6. For customers that churned their monthly charges were about 80 while those that did not churn had monthlycharges of around 60

7. Customers that churned had their total charges at around 702.2 while those that did not churn had their totalcharges at around 1730

8. More males churned 279 than females 277

9. For males that churned their monthly charges were around 75 while those that did not churn was around 65

10. For females that churned their monthly charges were around 65 while those that did not churn their monthly charges was around 70

11. Tenure is highly correlated with the TotalCharges column

12. MonthlyCharge is also highly correlated with the Total Charges column

Univariate analysis

univariate analysis serves as the foundational step towards unraveling the intricate patterns hidden within datasets. Focuses solely on exploring and comprehending individual variables in isolation.

Bivariate Analysis

Understanding the intricate relationships between variables is paramount. Bivariate analysis emerges as a powerful tool to unravel these connections by examining the interplay between pairs of variables within a dataset.

Findings

1. The train-df dataset has 21 columns and 5043 rows

2. Most customers were of the male gender, were not SeniorCitizens , lacked dependents and patners

3. The average monthlycharges was around 65 and majority of them did not churn

4. Majority of the customers prefferred Fiber optic internet service and a month to month type of contract

5. The customers were averagely charged 64.7 monthly with the highest charge on monthly basis being 118.65

6. For customers that churned their monthly charges were about 80 while those that did not churn had monthlycharges of around 60

7. Customers that churned had their total charges at around 702.2 while those that did not churn had their totalcharges at around 1730

8. More males churned 279 than females 277

9. For males that churned their monthly charges were around 75 while those that did not churn was around 65

10. For females that churned their monthly charges were around 65 while those that did not churn their monthly charges was around 70

11. Tenure is highly correlated with the TotalCharges column

12. MonthlyCharge is also highly correlated with the Total Charges column

3. Data preparation

Data cleaning is a critical step in preparing the dataset for analysis. It involves identifying and addressing issues such as missing values, duplicates, and outliers. By removing or correcting these discrepancies, we ensure the integrity and quality of the data, laying a solid foundation for subsequent analysis and modeling. Through data cleaning, we aim to create a clean and reliable dataset that accurately reflects the underlying patterns and relationships within the data, enabling us to derive meaningful insights and build robust predictive models.

We changed the datatype of the TotalCharges column to float and dropped the customerID column as it was not useful in building Machine Learning models for predicting customer churn.

4.Answering business questions and Visualizations

Answering key business questions and creating insightful visualizations play a pivotal role in driving strategic decision-making and fostering business growth. Through advanced analytics and data visualization techniques, we extracted actionable insights from the dataset, empowering telecom companies to make informed decisions and optimize their operations.

Question one: What is the relationship between Totalcharges and customers churning?

This question was answered using the following codes that produced the below visual.

The findings were as follows:

1. Customers that churned most have them have their totalcharges to be below 2000.

2. Those that did not churn, most of them have their totalcharges above 2000 compared to those that churned

Question two: What is the relationship between monthlycharges and customers churning?

Findings:

Those that churned were mostly charged around 80 monthly while those that did not churn were charged around 65 monthly

Question Three: Which customer gender churned the most?

findings:

The male gender churned the most 396 than the female gender 384

Question four: Between the male and female gender who were charged the most on a monthly basis?

Findings:

The male gender was charged the most on a monthly basis 99194.50 while the female gender was charged 9684.70

Question five : Customers that churned were mostly using which type of InternetService?

findings:

1. Most customers that churned were using fiber optic 570

2. While most of them that did not churn using DSL 835

5.Summary and recommendations

Our analysis of customer churn in the telecommunications industry has revealed key insights into factors influencing churn and enabled the development of accurate predictive models. To reduce churn rates and foster customer loyalty, we recommend implementing proactive retention strategies, enhancing the customer experience, optimizing subscription plans, investing in data-driven decision-making, and fostering a culture of innovation within the organization.

Since customers that churned were mostly charged around 80 on a monthly basis ,while those that did not churn were charged around 65 , we recommend the company to charge less than 65 to keep more customers

The male gender churned the most compared to the female gender . This is because they were charged more ,thus reducing the amount charged to the males would perhaps keep them .

Customers using the DSL , majority them did not churn . Thus the company should advise more of its customers to use the DSL internet service

By following these recommendations, telecom companies can mitigate churn rates, enhance customer satisfaction, and drive sustainable business growth.

6.Hypothesis Testing

In the pursuit of understanding and drawing insights from data, hypothesis testing emerges as a fundamental technique in statistical analysis. It enables analysts to make informed decisions by assessing the validity of assumptions and drawing conclusions based on empirical evidence.

We set our null hypothesis to be :There is no significant relationship between MonthlyCharges and whether a customer churns

and the alternative hypothesis to be :There is significant relationship between MonthlyCharges and whether a customer churns

We used Mann WhitneyU test to perform the hypothesis testing since the data was not normally distributed.

Our finding was to reject the null as seen below:

7.Data Preparation

Before embarking on the journey of predictive modeling, it’s imperative to prepare the data meticulously to ensure the success and accuracy of the models. Data preparation involves a series of steps aimed at cleaning, transforming, and optimizing the raw data for modeling purposes

Check if the data is balanced or not :

Balanced data ia when the number of values or entities in the target column are equal both the negative part and positive part . Some models work better on certain datasets when they are balanced .

Determining the input and output data

We set X to be our input data and y our output vriable before spliting the dataset into train and test . The test part is used to train the models while the test part is used to evaluate the model.

Preparing pipeplines

Pipelines are a powerful tool in the machine learning arsenal, streamlining the end-to-end process from data preprocessing to model evaluation. They encapsulate a sequence of data processing steps into a single entity, facilitating reproducibility, scalability, and efficiency in machine learning workflows. They help in:

• Automation: Pipelines automate repetitive tasks such as data preprocessing, feature engineering, and model training, reducing manual intervention and human error.
• Modularity: Pipelines promote modular design, allowing practitioners to swap components seamlessly and experiment with different configurations without disrupting the workflow.
• Consistency: By encapsulating all processing steps into a single pipeline, practitioners ensure consistency and reproducibility across experiments and deployments.
• Scalability: Pipelines enable scalability by providing a structured framework for handling large datasets and complex machine learning workflows.

8.Modeling

Modeling is the heart of the machine learning process, where algorithms learn patterns from data to make predictions or decisions. It encompasses selecting, training, and evaluating machine learning models to solve specific tasks or make informed decisions based on data.

Key Components of Modeling:

1. Model Selection: Choose the appropriate machine learning algorithm(s) based on the problem type (e.g., classification, regression, clustering) and data characteristics.
2. Training: Fit the selected model(s) to the training data, allowing them to learn patterns and relationships between input features and target variables.
3. Hyperparameter Tuning: Fine-tune model hyperparameters to optimize performance and prevent overfitting or underfitting.
4. Validation: Assess model performance on validation data to ensure generalization to unseen data and avoid over-optimization.
5. Evaluation: Measure model performance using relevant evaluation metrics (e.g., accuracy, precision, recall, F1-score, RMSE) to quantify effectiveness and compare different models.

Common Machine Learning Algorithms:

• Supervised Learning: Includes algorithms like linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-nearest neighbors (KNN), and neural networks.
• Unsupervised Learning: Encompasses algorithms such as k-means clustering, hierarchical clustering, principal component analysis (PCA), and association rule mining.
• Reinforcement Learning: Focuses on training agents to interact with an environment and learn optimal behavior through trial and error.

Best Practices in Modeling:

• Start Simple: Begin with simpler models and progressively explore more complex algorithms as needed.
• Cross-Validation: Use techniques like k-fold cross-validation to obtain reliable estimates of model performance and prevent overfitting.
• Ensemble Methods: Combine multiple models through ensemble methods (e.g., bagging, boosting, stacking) to improve predictive performance and robustness.
• Interpretability: Prioritize model interpretability when transparency and understanding of model decisions are critical.
• Iterative Process: Modeling is an iterative process; continuously refine models based on feedback from evaluation and domain expertise.

9.Evaluation

Model evaluation is essential for assessing the effectiveness and reliability of predictive models. It involves quantifying the performance of trained models using various evaluation metrics and techniques to ensure they generalize well to unseen data and fulfill the desired objectives.

1. Evaluation Metrics: Select appropriate evaluation metrics based on the problem type (e.g., classification, regression) and business goals. Common metrics include accuracy, precision, recall, F1-score, RMSE (Root Mean Squared Error), MAE (Mean Absolute Error), and AUC-ROC (Area Under the Receiver Operating Characteristic curve).
2. Cross-Validation: Employ cross-validation techniques (e.g., k-fold cross-validation, stratified cross-validation) to obtain robust estimates of model performance and mitigate overfitting. Cross-validation divides the data into multiple subsets, allowing each subset to serve as both training and validation data.
3. Confusion Matrix: Utilize the confusion matrix to visualize the performance of classification models, depicting the number of true positives, true negatives, false positives, and false negatives. From the confusion matrix, derived metrics such as accuracy, precision, recall, and F1-score can be calculated.
4. ROC Curve and AUC: Plot the Receiver Operating Characteristic (ROC) curve and compute the Area Under the Curve (AUC) to evaluate the performance of binary classification models. The ROC curve illustrates the trade-off between true positive rate (sensitivity) and false positive rate (1-specificity), while AUC quantifies the model’s ability to discriminate between positive and negative instances.

Best Practices in Model Evaluation:

• Domain Relevance: Ensure that selected evaluation metrics align with domain requirements and business objectives.
• Interpretability: Consider the interpretability of evaluation results and communicate findings in a clear and understandable manner to stakeholders.
• Ensemble Methods: Leverage ensemble methods and model averaging techniques to combine predictions from multiple models and improve overall performance.
• Iterative Improvement: Continuously evaluate and refine models based on feedback from evaluation results and domain expertise, iterating as needed to achieve optimal performance.

REFERENCES

https://r.search.yahoo.com/_ylt=AwrNZ405oCVmylIMpZDrFAx.;_ylu=Y29sbwNiZjEEcG9zAzEEdnRpZAMEc2VjA3Ny/RV=2/RE=1714951481/RO=10/RU=https%3a%2f%2fwww.sciencedirect.com%2fscience%2farticle%2fpii%2fS2666603023000143/RK=2/RS=paIJQ.Ee3d5WapMw4hoWsOiVyl0-

https://r.search.yahoo.com/_ylt=AwrFEUkJoCVmsTkNOPbrFAx.;_ylu=Y29sbwNiZjEEcG9zAzEEdnRpZAMEc2VjA3Ny/RV=2/RE=1714951433/RO=10/RU=https%3a%2f%2fmailchimp.com%2fresources%2fcustomer-churn%2f/RK=2/RS=rTqt6JaoPdAnYKNc3mWqfTJxoJk-

APPRECIATION

I would like to express my gratitude to the following individuals and organizations whose contributions were instrumental in the completion of this project :

1. My team members for their collaboration and support throughout the project .The members included ; Dennis Gitobu , Davis Azungu , Koech Joy, Evalyne Kawira and Loyce Zawadi
2. My CTA Mr. Obondo who played a crucial part
3. The Azubi organization without I would not have completed the project .Their support and guidance are highly appreciated

TAGS

Azubi Data Science