204.4.3 More on Sensitivity and Specificity

Sensitivity and Specificity

• By changing the threshold, the good and bad customers classification will be changed hence the sensitivity and specificity will be changed.
• Which one of these two we should maximize? What should be ideal threshold?
• Ideally we want to maximize both Sensitivity & Specificity. But this is not possible always. There is always a trade-off.
• Sometimes we want to be 100% sure on Predicted negatives, sometimes we want to be 100% sure on Predicted positives.
• Sometimes we simply don’t want to compromise on sensitivity sometimes we don’t want to compromise on specificity
• The threshold is set based on business problem

When Sensitivity is a High Priority

• Predicting a bad customers or defaulters before issuing the loan
  
• Predicting a bad defaulters before issuing the loan
  
• The profit on good customer loan is not equal to the loss on one bad customer loan.
• The loss on one bad loan might eat up the profit on 100 good customers.
• In this case one bad customer is not equal to one good customer.
• If p is probability of default then we would like to set our threshold in such a way that we don’t miss any of the bad customers.
• We set the threshold in such a way that Sensitivity is high.
• We can compromise on specificity here. If we wrongly reject a good customer, our loss is very less compared to giving a loan to a bad customer.
• We don’t really worry about the good customers here, they are not harmful hence we can have less Specificity.

When Specificity is a High Priority

• Testing a medicine is good or poisonous
  
• Testing a medicine is good or poisonous
  
• In this case, we have to really avoid cases like , Actual medicine is poisonous and model is predicting them as good.
• We can’t take any chance here.
• The specificity need to be near 100.
• The sensitivity can be compromised here. It is not very harmful not to use a good medicine when compared with vice versa case.

Sensitivity vs Specificity – Importance

• There are some cases where Sensitivity is important and need to be near to 1.
• There are business cases where Specificity is important and need to be near to 1.
• We need to understand the business problem and decide the importance of Sensitivity and Specificity.