Tuning the hyperparameter with gridsearch results in overfitting

Tuning the hyperparameter with gridsearch results in overfitting.

The train error is definitely low, but the test error is high. Can’t you adjust the hyperparameter to lower the test error?

def custom_wmae(actual_values, predicted_values):
    weight = actual_values.values / sum(actual_values)
    diff = abs(predicted_values - actual_values.values)
    return np.sum(weight * diff)


param_test1 = { 'max_depth':range(3,10,2),
 'min_child_weight':range(1,6,2)}


xgb1_test1 = xgboost.XGBRegressor(
 learning_rate =0.1,
 n_estimators=140,
 max_depth=5,
 objective ='reg:squarederror',
 min_child_weight = 1,
 subsample=0.8,
 scale_pos_weight=1,
 gamma = 0,
 seed=27)

grid_search = GridSearchCV(estimator=xgb1_test1,param_grid= param_test1, cv=5,
                           scoring=make_scorer(custom_wmae, greater_is_better=False),
                           iid=False,
                           return_train_score=True)


params_result= grid_search.fit(shuffled_train_X, shuffled_train_y)

• before tuning train_error: 0.386055, test_error: 0.674069

-after tuning train_error: 0.070645, test_error: 0.708254

Answer

It all depends on the data you are training. If the data you are using for training is quite less, let’s say 500 rows and a few columns and even then you are trying to split into training and testing data. The XGBoost is most likely to overfit on the training data.

To make sure your model doesn’t overfit, you can try three things –

1. Make sure you have enough data for the XGBoost training. If not, the tendency to overfit will always be there.

2. Play with your parameters. Try introducing regularization to your data, using L1 and L2 regularizations.

3. In the official XGBoost API, you can pass the validation set in the ‘xgb.train()’ function. So, you can pass your test set in the eval_set parameter of the function.