Librarian or Farmer?: How Bayes' Theorem can give Analysts more perspective on solving problems

Have you ever found yourself trying to make sense of complex situations or data at your workplace? Sometimes, it can feel like you’re trying to solve a puzzle with missing pieces in it.

And a lot of times the result of this incomplete problem-solving leads to a lot of bias in our outcome. And sometimes these biases could have dangerous consequences. Imagine you’re a political analyst and a biased demographic description leads to wrong policymaking.

This is where Bayes' theorem can come in handy. In this blog post, we'll explore how Bayes' theorem can give you more perspective using the example of a farmer and a librarian.

Understanding our Problem

Let's understand Bayes’ theorem using a simple example of a farmer and a librarian.?

We want to know whether a person we met, Harry, is more likely to be a librarian or a farmer. We also know that the person we saw is?shy and introverted, likes to read books, and has a passion for detail.

Now let’s divide our problem into three events:

L: The individual is a librarian

F: The individual is a farmer

D: The individual is shy and introverted, likes to read books, and has a passion for detail

Using our gut instinct says, given the above description, these are the probabilities of an individual with this description being either a librarian or a farmer:

P(D/L) = 70%

P(D/F) = 30%

These are valid assumptions and we won't be wrong in making them.

However, the story changes when we bring external data into the picture. Now let's say we also know that for every 20 farmers in the world, there is 1 librarian. Therefore:

P(L) = 1/21= 0.047

P(F) = 20/21 = 0.952

Applying Bayes' Theorem

Bayes' theorem is a mathematical formula that allows us to update our beliefs or probabilities based on new evidence or data. It was named after Reverend Thomas Bayes and is widely used in various fields such as statistics, machine learning, artificial intelligence, and more.

It looks something like this:

And now we simply plug and play the numbers in the formula to get the P(L/D), i.e. probability of Harry being a librarian with the given description.

Using the formula we get the value P(L/D) as 0.103.

This means that the probability that Harry is a librarian given this description is just about 10%. So, based on the evidence we have, it is more likely Harry is a farmer than a librarian.

Why Care About Bayes' Theorem?

Bayes' theorem can be a powerful tool for analysts to update their prior beliefs and make better decisions based on new evidence. It can help us make sense of complex situations, such as medical diagnoses or predicting future events, by incorporating new data or evidence as it becomes available.

In the previous example, bringing in data that the number of farmers in the world is much higher than the number of librarians removed our preconceived bias about Harry and led to more accurate decision-making. In our everyday lives, Bayes' theorem can help us avoid making hasty or biased judgments based on incomplete information. It encourages us to consider all available evidence and update our beliefs accordingly.

It's also worth noting that Bayes' theorem can be applied in a wide range of fields, from science and engineering to business and finance. For example, businesses can use Bayes' theorem to update their market forecasts based on new data, while doctors can use it to improve the accuracy of medical diagnoses. Whether you’re a data scientist or a consultant, if you deal with problem-solving on large datasets, Bayes' theorem helps you gain more perspective.

Source: 3blue1brown- Bayes Theorem