← All Posts /

An Introduction To The Confusion Matrix

A confusion matrix is a tool used to summarize the performance and accuracy of a classification model in machine learning. It comes in handy after the data has been cleaned, processed, fed into a model, and has given out results. It helps in determining the effectiveness of the classification.

A confusion matrix is a table of four different combinations of actual and the predicted values and is an important step in calculating the recall, precision, and accuracy of the values, as well as AUC-ROC curves.

● The target variable is binary, that is, either Positive or Negative

● Actual values of the target variable are represented by the columns

● Predicted values of the target variable are shown in the rows of the matrix

The matrix has been divided into four parts:

  1. True Positive
  2. True Negative
  3. False Positive
  4. False Negative

True Positive (TP)

● The predicted value matches the actual value

● The actual outcome was positive and the classifier predicted positive too

True Negative (TN)

● The predicted value matches the actual value

● The actual outcome was negative and the model’s prediction was negative as well

False Positive (FP) – Type 1 error

● The predicted value was falsely predicted

● The model predicted a positive value but the actual value was negative

● Also known as the Type 1 error

False Negative (FN) – Type 2 error

● The predicted value was falsely predicted

● The model predicted a positive value but the actual outcome was positive

● Also known as the Type 2 error

How do we use the Matrix?

The confusion matrix helps in determining values that explain the results of the classifier through certain metrics to improve our understanding of its performance. These are:

Precision

Precision is the ratio of correct positive predictions to the total of all positive outcomes.

It is also called Positive predictive value.

Precision = TP/(TP+FP)

Recall

Recall is the ratio of the correct positive results to the total positive predictions.

It is also called Sensitivity, Probability of Detection, True Positive Rate.

Recall= TP/(TP+FN)

Accuracy

Accuracy is defined as the ratio of correct predictions by the total predictions.

Accuracy = Correct Predictions/ Total Predictions

In a confusion matrix, it can be derived using:

Accuracy = (TP+TN)/(TP+TN+FP+FN)

Accuracy is a handy metric for evaluation when all the classes are of equal importance. But this might not be the case if we are predicting if a patient has a fatal diagnosis or not. Here, False Positives are acceptable, but False Negatives are not.

ROC curve

A ROC curve (receiver operating characteristic curve) graphs the performance of a classification model at all classification thresholds. (Using thresholds: Say, if you want to compute TPR and FPR for the threshold equal to 0.6, you apply the model to each example, get the score, and, if the score >=0.6, you predict the positive class; otherwise, the prediction is negative)

It plots 2 parameters:

True positive rate (Recall)= TP/(TP+FN)

False Positive rate= FP/(FP+TN)

Lowering the threshold predicts more items as positive, thus increasing both False Positives and True Positives in the outcome.

AUC

AUC stands for Area under the ROC Curve. It provides an average measure of performance across all possible probability thresholds of the results.

The higher the area under the ROC curve (AUC), the better the model and the more efficient the classifier. A perfect model would have an area of 1. Usually, if your model is efficient, you obtain a good performance class by selecting the value of the threshold whose TPR tends to 1 while FPR inches closer to 0.

An Example of the Confusion Matrix

Suppose we have 165 patients being tested for Covid-19, and have gotten the following results from the tests:

These are the most important metrics that we derive from the matrix:

Accuracy: Overall, how often is the test correctly predicting the patient’s diagnosis?

(TP+TN)/Total = (100+50)/165 = 0.91

Misclassification Rate: Overall, how often is it wrong?

(FP+FN)/total = (10+5)/165 = 0.09

It is also equal to (1-Accuracy), and is also known as the “Error Rate”.

True Positive Rate: When it’s actually positive, how often does it predict positive?

TP/(TP+FP) = 100/105 = 0.95

We’ve discussed this earlier, also known as “Sensitivity” or “Recall”.

False Positive Rate: When it’s actually negative, how often does the test predict positive?

FP/(TN+FN) = 10/60 = 0.17

True Negative Rate: When it’s actually negative, how often does it give a negative outcome?

TN/(TN+FP) = 50/60 = 0.83

True Negative Rate is equal to (1-False Positive Rate), and is called “Specificity”.

Precision: When the outcome is positive, how often is it correct?

TP/predicted yes = 100/110 = 0.91

Prevalence: How often does the test predict positive?

TP+FN/Total = 105/165 = 0.64

These metrics help us in understanding the results better as well as the efficiency of the classification.

Conclusion

The Confusion Matrix and the metrics derived from it are really helpful in analyzing the data as well as the algorithm used for classification, like the following.

● ROCs define the trade-off between the TPR and FPR for a predictive model using different probability thresholds.

● Precision-Recall curves summarize the trade-off between the TPR and the positive prediction efficiency for a model using different probability thresholds.

● ROCs are suitable when the observations are balanced between each class, whereas precision-recall curves are more suitable when the data is imbalanced.

Recommended » Benford's Law