Naive Bayes Classifier in Machine Learning

Photo by fotografierende on Pexels.com

Machine Learning

Mathematical explanation and python implementation using sklearn

Naive Bayes Classifier

Naive Bayes Classifiers are probabilistic models that are used for the classification task. It is based on the Bayes theorem with an assumption of independence among predictors. In the real-world, the independence assumption may or may not be true, but still, Naive Bayes performs well.

Topics covered in this story

Image by Author

Why is it named Naive Bayes?

Naive → It is called naive because it assumes that all features in the dataset are mutually independent.
Bayes, → It is based on Bayes Theorem.

Bayes Theorem

First, let’s learn about probability.

Probability

A probability is a number that reflects the chance or likelihood that a particular event will occur.

Event → In probability, an event is an outcome of a random experiment.

P(A)=n(A)/n(S)

P(A) → Probability of an event A
n(A) →Number of favorable outcomes
n(S) →Total number of possible outcomes

Example

P(A) → Probability of drawing a king
P(B) →Probability of drawing a red card.

P(A) =4/52
P(B)=26/52

Image by Author

Types of probability

  1. Joint probability
  2. Conditional probability

1. Joint Probability

A joint probability is the probability of two events occurring simultaneously.

P(A∩B) →Probability of drawing a king, which is red.

P(A∩B)=P(A)*P(B)=(4/52)*(26/52)=(1/13)*(1/2)=1/26

Image by Author

2. Conditional Probability

Conditional probability is the probability of one event occurring in the presence of a second event.

Probability of drawing a king given red → P(A|B)

Image by Author

Probability of drawing a red card given king P(B|A)

P(B|A) =P(A∩B)/P(A)

Image by Author

Derivation of Bayes Theorem

Image By Author

Naive Bayes Classifier Example

Bayes theorem is an extension of conditional probability. By using Bayes theorem, we have to use one conditional probability to calculate another one.

To calculate P(A|B), we have to calculate P(B|A) first.

Example:

If you want to predict if a person has diabetes, given the conditions? P(A|B)
Diabetes → Class → A
Conditions → Independent attributes → B

To calculate this using Naive Bayes,

  1. First, calculate P(B|A) → which means from the dataset find out how many of the diabetic patient(A) has these conditions(B). This is called likelihood ratio P(B|A)
  2. Then multiply with P(A) →Prior probability →Probability of anybody in the population having diabetes.
  3. Then divide by P(B) → Evidence. This is the current event that occurred. Given this event has occurred, we are calculating the probability of another event that will also occur.

This concept is known as the Naive Bayes algorithm.

P(B|A) → Likelihood Ratio
P(A) → Prior Probability
P(A|B) → Posterior Probability
P(B) → Evidence


Dataset

I have taken the golf dataset.

Image by Author

Consider the problem of playing golf. Here in this dataset, Play is the target variable. Whether we can play golf on a particular day or not is decided by independent variables Outlook, Temperature, Humidity, Windy.

Mathematical Explanation of Naive Bayes

Let’s predict given the conditions sunny, mild, normal, False → Whether he/she can play golf?

Image by Author

Simplified Bayes theorem

P(A|B) and P(!A|B) is decided only by the numerator value because the denominator is the same in both the equation.

So, to predict the class yes or no, we can use this formula P(A|B)=P(B|A)*P(A)

  1. Calculate Prior Probability
    Out of 14 records, 9 are yes. So P(yes)=9/14 and P(no)=5/14

Image by Author

2. Calculate Likelihood Ratio

Outlook

Out of 14 records, 5-Sunny,4-Overcast,5-Rainy.

Find the probability of the day being sunny given he/she can play golf?

From the dataset, the number of sunny days we can play is 2. The total no of days we can play is 9.

So P(Sunny | yes) =2/9

Similarly, we have to calculate all variables.

Image by Author

Temperature

Image by Author

Humidity

Image by Author

Windy

Image by Author

Let’s predict given the conditions sunny, mild, normal, False → Whether he/she can play golf?

A=yes
B=(Sunny,Mild,Normal,False)

P(A|B)=P((yes)|(Sunny,Mild,Normal,False)

P(A|B)=P(B|A)*P(A)

P(yes|(Sunny,Mild,Normal,False))= P((Sunny,Mild,Normal,False)|yes) *P(yes)

[Probaility of independent events is calculated by multiplying the probability of all the events. Naive Bayes algorithm treats all the variables as independent variables)

=P(Sunny | yes)*P(Mild | yes)*P(Normal | yes)*P(False | yes)*P(yes)

=2/9 *4/9 *6/9 *6/9 *9/14

P(yes|(Sunny,Mild,Normal,False))= 0.0282

Let’s now calculate P(no|(Sunny,Mild,Normal,False))

P(no|(Sunny,Mild,Normal,False))= P((Sunny,Mild,Normal,False)|no) *P(no)

=P(Sunny | no) * P(Mild | no) * P(Normal | no) * P(False | no) * P(no)

=3/5 *2/5 *1/5 *2/5 *5/14

P(no|(Sunny,Mild,Normal,False))= =0.0068

Since 0.0282 > 0.0068[P(yes|conditions)>P(no|conditions) , for the given conditions Sunny,Mild,Normal,False , play is predicted as yes.


Let’s build the NB model using the same dataset

Python Implementation of Naive Bayes using sklearn

  1. Import the libraries
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

2. Load the data

df=pd.read_csv("golf_df.csv")
df.head(3)

3. Converting categorical variables(string data types) to continuous variables

from sklearn.preprocessing import LabelEncoder
le=LabelEncoder()
Outlook_le=le.fit_transform(df.Outlook)
Temperature_le=le.fit_transform(df.Temperature)
Humidity_le=le.fit_transform(df.Humidity)
Windy_le=le.fit_transform(df.Windy)
Play_le=le.fit_transform(df.Play)

df["Outlook_le"]=Outlook_le
df["Temperature_l1"]=Temperature_le
df["Humidity_le"]=Humidity_le
df["Windy_le"]=Windy_le
df["Play_le"]=Play_le
df.head(3)

4. Now drop the old categorical columns from the dataframe

df=df.drop(["Outlook","Temperature","Humidity","Windy","Play"],axis=1)
df.head(3)

5. Assign x (independent variables) and y (dependent variable)

x=df.iloc[:,0:4]
x.head(3)
y=df.iloc[:,4:]
y.head(3)

6. Split data into train and test

from sklearn.model_selection import train_test_split
x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3,random_state=10)

7. Model building with sklearn

from sklearn.naive_bayes import GaussianNB
model=GaussianNB()
model.fit(x_train,y_train)

GaussianNB()

8. Accuracy Score

y_predict=model.predict(x_test)
from sklearn.metrics import accuracy_score
accuracy_score(y_test,y_predict,normalize=True)

Output: 1.0

9. Let’s predict the class(yes or no)given the conditions sunny, mild, normal, False.

model.predict([[2,2,1,0]])

Output: array([1])

1 → indicates yes.

So given the conditions sunny, mild, normal, False → play is yes.
So we can play golf given the conditions are sunny, mild, normal, False.


Github link

The code and dataset used in this story can be downloaded as a jupyter notebook from my Github link.

Conclusion

Naive Bayes classifier performs very well compared to other models when the assumption of independent predictors holds. It is very fast in both training and testing data. In some rare events, if a category which we are predicting is not observed in training data means, then the model will add zero probability and will be unable to make a prediction. To solve this, smoothing techniques like Laplace estimation is used.

My other blogs on Machine learning

Linear Regression in Python

Logistic Regression in Python

Understanding Decision Trees in Machine Learning

An Introduction to Support Vector Machine

An Introduction to K-Nearest Neighbors Algorithm


I hope that you have found this article helpful. Thanks for reading!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s