About the miscalibration of logistic regression models

Miscalibration_of_logistic_regression_models.ipynb

Something that I found out that the fact that logistic regression is not calibrated when tuning C on ROC-AUC is not necessarily deterministic. I think if you set the random_state=42 (of the train-test split) then you will get two exacts logistic regression models with the ROC-AUC and logloss. But the thing a bit weird is that the ROC-AUC is not equal to 1 on the training set so I'm a bit surprised. It might be due that we don't have enough repetition into the inner CV.

Another possibility would be that the solution are comparable but with a large variation (as shown on the validation curve) and we pick up another C just due to this.

Note that the model is severly miscalibrated despite the balance property. This property only informs us about marginal calibration, not about auto-calibration.

This is not the whole story. The balance property just holds for the design matrix of the logistic regression. If the design matrix is badly chosen, then the balance property is just (very) weak. For instance, if only random features without correlation to the target are chosen, the balance property reduces to the marginal (the conditioning drops out), which is weak.
On the other hand, for a "correct" design matrix, the balance property is stronger than auto-calibration.