Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@maorn

This comment has been minimized.

Copy link

@maorn maorn commented Nov 27, 2018

hi,
great notebook, well explained.
i have a few questions:

  1. regarding the parameter "algo=hyperopt.tpe.suggest" in optimize , where are you using it and for what? seems like it is not used anywhere
  2. regarding hyperopt package imports (scope and sample) i didn't see where you use it on this notebook
@talperetz

This comment has been minimized.

Copy link
Owner Author

@talperetz talperetz commented Nov 27, 2018

@maron you are right, I originally used hyperopt and then migrated to skopt
so the imports remained, but functionality was changed in favor of skopt

@billy-odera

This comment has been minimized.

Copy link

@billy-odera billy-odera commented Apr 22, 2019

i get the below error
TypeError: cv() got an unexpected keyword argument 'metrics' when running
default_cb_optimizer.evaluate_model()

@ralanburnett

This comment has been minimized.

Copy link

@ralanburnett ralanburnett commented Apr 28, 2019

I am also getting the same error as billy-odera:
TypeError: cv() got an unexpected keyword argument 'metrics'
for both default_cb_optimizer.evaluate_model(), cb_optimal_values = cb_optimizer.optimize(params_space) and
one_cb_optimal_values = one_cb_optimizer.optimize(params_space, max_evals=40, n_random_starts=4)

@ralanburnett

This comment has been minimized.

Copy link

@ralanburnett ralanburnett commented Apr 28, 2019

I got the Catboost portion of the code to run by removing metric = 'auc' in the evaluate_model method for CatboostOptimizer. However, this makes the score way out of whack (score on default params is 0.984 …).

@jhmenke

This comment has been minimized.

Copy link

@jhmenke jhmenke commented Jun 4, 2019

@talperetz: You are not passing the categorical_column_indices to your CatboostOptimizer instances

@akatasonov

This comment has been minimized.

Copy link

@akatasonov akatasonov commented Oct 3, 2019

I got the Catboost portion of the code to run by removing metric = 'auc' in the evaluate_model method for CatboostOptimizer. However, this makes the score way out of whack (score on default params is 0.984 …).

The CatboostOptimizer class is not going to work with the recent version of Catboost as is. First test_scores = validation_scores.iloc[:, 2] should be changed to test_scores = validation_scores.iloc[:, 1] since the index 2 corresponds to the standard deviation of the metric, which we don't optimize for. Also, if your metric is not AUC (RMSE for example) then you need to replace best_metric = test_scores.max() with best_metric = test_scores.min() and return best_metric vs 1-best_metric since we're minimizing the RMSE and not the different between 1 and our AUC (which is almost always < 1)

@ralanburnett

This comment has been minimized.

Copy link

@ralanburnett ralanburnett commented Oct 22, 2019

Can you explain where the 3-fold AUC data is coming from (how this is calculated)? I can figure out how to get AUC for validation data (for Catboost) but not for test data.

@iamrishabhjain25

This comment has been minimized.

Copy link

@iamrishabhjain25 iamrishabhjain25 commented Apr 10, 2021

Hi. Could this code word for a dataset which has non numerical features like sex, city, profession ? And if yes, how ?

@talperetz

This comment has been minimized.

Copy link
Owner Author

@talperetz talperetz commented Apr 11, 2021

Sure thing. This dataset holds only categorical features.
You can see how I treat it here:
catboost.Pool(self.X_train, self.y_train, cat_features=self.categorical_columns_indices),
You can also take a look here at Catboost examples.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment