catboost_playground.ipynb

hi,
great notebook, well explained.
i have a few questions:

1. regarding the parameter "algo=hyperopt.tpe.suggest" in optimize , where are you using it and for what? seems like it is not used anywhere
2. regarding hyperopt package imports (scope and sample) i didn't see where you use it on this notebook

@maron you are right, I originally used hyperopt and then migrated to skopt
so the imports remained, but functionality was changed in favor of skopt

i get the below error
TypeError: cv() got an unexpected keyword argument 'metrics' when running
default_cb_optimizer.evaluate_model()

I am also getting the same error as billy-odera:
TypeError: cv() got an unexpected keyword argument 'metrics'
for both default_cb_optimizer.evaluate_model(), cb_optimal_values = cb_optimizer.optimize(params_space) and
one_cb_optimal_values = one_cb_optimizer.optimize(params_space, max_evals=40, n_random_starts=4)

I got the Catboost portion of the code to run by removing metric = 'auc' in the evaluate_model method for CatboostOptimizer. However, this makes the score way out of whack (score on default params is 0.984 …).

@talperetz: You are not passing the categorical_column_indices to your CatboostOptimizer instances

I got the Catboost portion of the code to run by removing metric = 'auc' in the evaluate_model method for CatboostOptimizer. However, this makes the score way out of whack (score on default params is 0.984 …).

The CatboostOptimizer class is not going to work with the recent version of Catboost as is. First test_scores = validation_scores.iloc[:, 2] should be changed to test_scores = validation_scores.iloc[:, 1] since the index 2 corresponds to the standard deviation of the metric, which we don't optimize for. Also, if your metric is not AUC (RMSE for example) then you need to replace best_metric = test_scores.max() with best_metric = test_scores.min() and return best_metric vs 1-best_metric since we're minimizing the RMSE and not the different between 1 and our AUC (which is almost always < 1)

Can you explain where the 3-fold AUC data is coming from (how this is calculated)? I can figure out how to get AUC for validation data (for Catboost) but not for test data.

Hi. Could this code word for a dataset which has non numerical features like sex, city, profession ? And if yes, how ?

Sure thing. This dataset holds only categorical features.
You can see how I treat it here:
catboost.Pool(self.X_train, self.y_train, cat_features=self.categorical_columns_indices),
You can also take a look here at Catboost examples.