Last active
July 23, 2024 23:57
-
-
Save doraneko94/e24643136cfb8baf03ef8a314ab9615c to your computer and use it in GitHub Desktop.
Calculating confidence interval of ROC-AUC.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from sklearn.metrics import roc_auc_score | |
from math import sqrt | |
def roc_auc_ci(y_true, y_score, positive=1): | |
AUC = roc_auc_score(y_true, y_score) | |
N1 = sum(y_true == positive) | |
N2 = sum(y_true != positive) | |
Q1 = AUC / (2 - AUC) | |
Q2 = 2*AUC**2 / (1 + AUC) | |
SE_AUC = sqrt((AUC*(1 - AUC) + (N1 - 1)*(Q1 - AUC**2) + (N2 - 1)*(Q2 - AUC**2)) / (N1*N2)) | |
lower = AUC - 1.96*SE_AUC | |
upper = AUC + 1.96*SE_AUC | |
if lower < 0: | |
lower = 0 | |
if upper > 1: | |
upper = 1 | |
return (lower, upper) |
Thanks for providing this helpful code! The formula can be found in these two articles, in case anyone is interested:
Hanley and McNeil, The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology (1982) 43 (1) pp. 29-36.
Fogarty, Baker and Hudson, Case Studies in the use of ROC Curve Analysis for Sensor-Based Estimates in Human Computer Interaction, Proceedings of Graphics Interface (2005) pp. 129-136.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
def roc_auc_ci(y_true, y_score, positive=1):
AUC = roc_auc_score(y_true, y_score)
N1 = sum(y_true == positive)
N2 = sum(y_true != positive)
Q1 = AUC / (2 - AUC)
Q2 = 2AUC**2 / (1 + AUC)
SE_AUC = sqrt((AUC(1 - AUC) + (N1 - 1)(Q1 - AUC**2) + (N2 - 1)(Q2 - AUC**2)) / (N1N2))
lower = AUC - 1.96SE_AUC
upper = AUC + 1.96*SE_AUC
if lower < 0:
lower = 0
if upper > 1:
upper = 1
return (lower, upper)