Choosing threshold in Binary classification

Threshold of 0.5 is used by default (for binary problems) to convert predicted probabilities into class predictions

Threshold can be adjusted to increase sensitivity or specificity

Sensitivity and specificity have an inverse relationship

Increasing one would always decrease the other

Adjusting the threshold should be one of the last step you do in the model-building process

Question: Wouldn't it be nice if we could see how sensitivity and specificity are affected by various thresholds, without actually changing the threshold?

Answer: Plot the ROC curve.

AUC is the percentage of the ROC plot that is underneath the curve:

print(metrics.roc_auc_score(y_test, y_pred_prob))

https://www.ritchieng.com/machine-learning-evaluate-classification-model/