Gridsearchcv with k-fold cross validation and early stopping

gridsearchcv.py

#

... import xgboost.sklearn as xgb 
... from sklearn.model_selection import GridSearchCV 
... from sklearn.model_selection import TimeSeriesSplit 
...  
... cv = 2 
...  
... trainX= [[1], [2], [3], [4], [5]] 
... trainY = [1, 2, 1, 2, 1] 
...  
... # these are the evaluation sets 
... testX = trainX  
... testY = trainY 
...  
... paramGrid = {"subsample" : [0.5, 0.8]} 
...  
... fit_params={"early_stopping_rounds":42,  
...             "eval_metric" : "mae",  
...             "eval_set" : [[testX, testY]]} 
...  
... model = xgb.XGBRegressor() 
...  
... from sklearn.model_selection import StratifiedKFold 
... skf = StratifiedKFold(n_splits=cv, shuffle = True, random_state = 999) 
... gridsearch = GridSearchCV(model, paramGrid, verbose=1,              
...          cv=skf.split(trainX, trainY)) 
...  
... g = gridsearch.fit(trainX, trainY, **fit_params) 
... y_true, y_pred = testY, gridsearch.predict(testX) 
... from sklearn.metrics import r2_score  
... print(r2_score(y_true, y_pred)) 
... print(g.cv_results_) 

This would be biased as the early stopping is performed on the test set. The correct approach is to split each train fold into train and validation and use validation for early stopping. This cannot be achieved using GridSearchCV, instead the grid search must be performed manually, for example: https://xgboosting.com/xgboost-early-stopping-with-grid-search/