Maths behind Naive Bayes

Ever thought of the world before computers? Statisticians were calculating the probabilities manually and predicting the classes or results! Computer programming has made our life so easy, we have pre-built modules of machine learning libraries. If you are a python developer, you can implement naive Bayes easily using scikit learn. There are tools like Orange which has drag and drop functionalities and the algorithm can be tested with one single click!.

Let's see how Naive Bayes works and the maths behind it with a simple problem statement. Let's assume Mr X is working on a notification project and he has to notify news headlines to respective subscribers. Mr X has 2 sets of subscribers, those who subscribed for political news and those who have subscribed for Entertainment news. Mr X now has to classify each and every news feed he receives and send the notifications accordingly.

Let's consider the below headlines which were already classified by Mr X

1. Rahul Gandhi thanks government for changing FDI norms after his warning
2. DMK Allies with ruling AIADMK combine in Tamil Nadu rural civic polls
3. Congress says government is doing injustice to retailers
4. Amitabh Bachchan shares throwback photo from Sholay premiere, says "How pretty Jaya looks"
5. IPL postponed further as Indian government extends lockdown
6. Siddharth Malhotra reacts to "Masakali" controversy.

Please note these are just imaginary headlines for learning purpose and may or may not be associated with any news feeds or current affairs. The first 3 headlines are clearly political and last 3 are entertainment related news feeds. Now based on this data, can we predict or classify a new news headline?

Which category would the below feed be, based on above dataset.

"Those who have no work criticise government" Says Mamata

Before jumping into solving the problem, let's understand what Naive Bayes is, it's a supervised algorithm based on Bayes theorem with strong (naive) independence assumptions between the features. Bayes’ theorem states the following relationship, given class variable y and dependent feature vector x1 through xn,

This can be decomposed as,

Let's go back to the above dataset, to classify the new news feed, we need to find the best probability comparing both categories. That is we need to calculate

P(Politics | "Those who have no work criticise government" Says Mamata) and

P(Entertainment | "Those who have no work criticise government" Says Mamata)

We can use Bayes theorem to find out the above probabilities.

Let's put the the naive assumption to the Bayes’ theorem, which is, independence among the features. So now, we split evidence into the independent parts.

Now, if any two events A and B are independent, then,

P(A|B) = P(A) * P(B), Applying this on the new news feed.

P(Those Who Have No Work Criticise government Says Mamta) = P(Those) * P(Who) * P(No) * P(Work) * P(Criticise) * P (government) * P(Says) * P(Mamta)

and,

P(Those Who Have No Work Criticise government Says Mamta | Politics) = P(Those| Politics) * P(Who| Politics) * P(No| Politics) * P(Work| Politics) * P(Criticise| Politics) * P (government| Politics) * P(Says| Politics) * P(Mamta| Politics)

Let's calculate the probability of word 'those' in training data labelled as politics. "Those" is not present in our training data. As we have no value for this probability. Now, we are unable to be sure about the prediction and our model performed poorly. To avoid these kind of issues, we use smoothing. In statistics, additive smoothing, also called Laplace smoothing or Lidstone smoothing, is a technique used to smooth categorical data. To ensure that our posterior probabilities are never zero, we add 1 to the numerator, and we add k to the denominator. So, in the case that we don’t have "Those" in training set, the posterior probability comes out to 1 / N + k instead of zero.

Total number of words in training set labelled as political = 31

Total number of words in training set labelled as Entertainment = 28

Total Unique words in above training set= 55

P(Those Who Have No Work Criticise government Says Mamta | politics) * P(Politics) = 1.149e-17

P(Those Who Have No Work Criticise government Says Mamta | Entertainment) * P(Entertainment) = 1.04e-17

That is, P(Politics | New feed) > P(Entertainment | New feed )

From above results it’s clear than the new sentence will be classified as Political news feed.

Please check out following tutorials if you are interested in trying out/implementing this,

Text Classification using Naive Bayes

Classification using Orange

Thank you!