Secrets of Decision Trees: A Guide to Entropy, Gini, and Information Gain

Application: Decision trees are supervised learning algorithms used for classification and regression tasks.

Focus: Classification with decision trees

Basic Concepts:

• Root Node: Top of the tree contains most important attribute Ex: Outlook
• Split: Division of nodes into two or more sub nodes.
• Leaf Node: Nodes that do not split are called Leaf nodes or terminal nodes. Overcast is the example of showing in screen image
• CART focuses on binary splits at each node whilst ID3 more than binary splits

Measuring Purity and Impurity:

• Entropy: This measure quantifies the level of uncertainty or randomness within a dataset regarding its class labels. Impurity: Imagine a dataset with equal numbers of positive and negative examples. This scenario holds maximum uncertainty, resulting in an entropy value of 1. Purity: Conversely, a perfectly homogenous dataset with all instances belonging to a single class has zero entropy, signifying complete certainty. Type of decision tree algorithms : ID3 etc.,Dealt with multi-class classification problems and suitable for small datasets

Calculation:
Entropy = - Σ(p(i) * log2(p(i)))        

Gini Index

This metric estimates the likelihood of randomly misclassifying an instance within a dataset. Purity: A perfectly balanced dataset (equal class distribution) has the highest Gini impurity of 0.5, indicating a 50% chance of misclassification. Impurity: As the data becomes more homogenous, the Gini index approaches 0, signifying a lower probability of misclassification.

Type of decision tree algorithms : CART ,

Computationally more efficient than entropy and suitable for large datasets

Gini Impurity = 1 - Σ(p(i))^2        

Choosing the split:

• Information Gain: This concept builds upon the notion of entropy and measures the reduction in uncertainty brought about by splitting the data based on a specific feature. The feature leading to the highest information gain is chosen for the split at a particular node, as it promotes the most significant reduction in randomness and aids in clearer class separation.

Information Gain = Entropy(parent) - Σ [ (weight of child) * Entropy(child) ]        

Conclusion:

Decision Trees are simple for classification problems especially if the output is discrete set of categorical values