Avoiding Overfitting

The ultimate goal of machine learning is to make accurate predictions on unseen data. EvalML aims to help you build a model that will perform as you expect once it is deployed in to the real world.

One of the benefits of using EvalML to build models is that it provides guardrails to ensure you are building pipelines that will perform reliably in the future. This page describes the various ways EvalML helps you avoid overfitting to your data.

import evalml

Detecting Label Leakage

A common problem is having features that include information from your label in your training data. By default, EvalML will provide a warning when it detects this may be the case.

Let’s set up a simple example to demonstrate what this looks like

import pandas as pd

X = pd.DataFrame({
    "leaked_feature": [6, 6, 10, 5, 5, 11, 5, 10, 11, 4],
    "leaked_feature_2": [3, 2.5, 5, 2.5, 3, 5.5, 2, 5, 5.5, 2],
    "valid_feature": [3, 1, 3, 2, 4, 6, 1, 3, 3, 11]
})

y = pd.Series([1, 1, 0, 1, 1, 0, 1, 0, 0, 1])

automl = evalml.AutoClassificationSearch(
    max_pipelines=1,
    allowed_model_families=["linear_model"],
)

automl.search(X, y)

*****************************
* Beginning pipeline search *
*****************************

Optimizing for Log Loss Binary. Lower score is better.

Searching up to 1 pipelines.
WARNING: Possible label leakage: leaked_feature, leaked_feature_2

✔ Logistic Regression Binary Pipeline:     100%|██████████| Elapsed:00:00
✔ Optimization finished                    100%|██████████| Elapsed:00:00

In the example above, EvalML warned about the input features leaked_feature and leak_feature_2, which are both very closely correlated with the label we are trying to predict. If you’d like to turn this check off, set detect_label_leakage=False.

The second way to find features that may be leaking label information is to look at the top features of the model. As we can see below, the top features in our model are the 2 leaked features.

best_pipeline = automl.best_pipeline
best_pipeline.feature_importances

	feature	importance
0	leaked_feature	-1.789393
1	leaked_feature_2	-1.645127
2	valid_feature	-0.398465

Perform cross-validation for pipeline evaluation

By default, EvalML performs 3-fold cross validation when building pipelines. This means that it evaluates each pipeline 3 times using different sets of data for training and testing. In each trial, the data used for testing has no overlap from the data used for training.

While this is a good baseline approach, you can pass your own cross validation object to be used during modeling. The cross validation object can be any of the CV methods defined in scikit-learn or use a compatible API.

For example, if we wanted to do a time series split:

from sklearn.model_selection import TimeSeriesSplit

X, y = evalml.demos.load_breast_cancer()

automl = evalml.AutoClassificationSearch(
    cv=TimeSeriesSplit(n_splits=6),
    max_pipelines=1
)

automl.search(X, y)

*****************************
* Beginning pipeline search *
*****************************

Optimizing for Log Loss Binary. Lower score is better.

Searching up to 1 pipelines.

✔ XGBoost Binary Classification Pipel...   100%|██████████| Elapsed:00:10
✔ Optimization finished                    100%|██████████| Elapsed:00:10

if we describe the 1 pipeline we built, we can see the scores for each of the 6 splits as determined by the cross-validation object we provided. We can also see the number of training examples per fold increased because we were using TimeSeriesSplit

automl.describe_pipeline(0)

******************************************
* XGBoost Binary Classification Pipeline *
******************************************

Problem Type: Binary Classification
Model Family: XGBoost
Number of features: 25

Pipeline Steps
==============
1. One Hot Encoder
         * top_n : 10
2. Simple Imputer
         * impute_strategy : most_frequent
         * fill_value : None
3. RF Classifier Select From Model
         * percent_features : 0.8487792213962843
         * threshold : -inf
4. XGBoost Classifier
         * eta : 0.38438170729269994
         * max_depth : 7
         * min_child_weight : 1.5104167958569887
         * n_estimators : 397

Training
========
Training for Binary Classification problems.
Total training time (including CV): 10.4 seconds

Cross Validation
----------------
Warning! High variance within cross validation scores. Model may not perform as estimated on unseen data.
             Log Loss Binary  Accuracy Binary  Balanced Accuracy Binary    F1  Precision  Recall   AUC  MCC Binary # Training # Testing
0                      0.532            0.840                     0.860 0.863      0.953   0.788 0.949       0.691     83.000    81.000
1                      0.045            0.988                     0.988 0.988      1.000   0.977 1.000       0.976    164.000    81.000
2                      0.111            0.975                     0.963 0.982      0.964   1.000 0.990       0.945    245.000    81.000
3                      0.062            0.975                     0.983 0.982      1.000   0.966 0.999       0.942    326.000    81.000
4                      0.083            0.963                     0.977 0.976      1.000   0.953 0.997       0.900    407.000    81.000
5                      0.068            0.963                     0.944 0.975      0.967   0.983 0.998       0.903    488.000    81.000
mean                   0.150            0.951                     0.952 0.961      0.981   0.945 0.989       0.893          -         -
std                    0.189            0.055                     0.048 0.048      0.021   0.078 0.020       0.103          -         -
coef of var            1.256            0.058                     0.050 0.050      0.022   0.083 0.020       0.115          -         -

Detect unstable pipelines

When we perform cross validation we are trying generate an estimate of pipeline performance. EvalML does this by taking the mean of the score across the folds. If the performance across the folds varies greatly, it is indicative the the estimated value may be unreliable.

To protect the user against this, EvalML checks to see if the pipeline’s performance has a variance between the different folds. EvalML triggers a warning if the “coefficient of variance” of the scores (the standard deviation divided by mean) of the pipelines scores exeeds .2.

This warning will appear in the pipeline rankings under high_variance_cv.

automl.rankings

	id	pipeline_name	score	high_variance_cv	parameters
0	0	XGBoost Binary Classification Pipeline	0.150153	True	{'impute_strategy': 'most_frequent', 'percent_...

Create holdout for model validation

EvalML offers a method to quickly create an holdout validation set. A holdout validation set is data that is not used during the process of optimizing or training the model. You should only use this validation set once you’ve picked the final model you’d like to use.

Below we create a holdout set of 20% of our data

X, y = evalml.demos.load_breast_cancer()
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, test_size=.2)

automl = evalml.AutoClassificationSearch(
    objective="recall",
    max_pipelines=3,
    detect_label_leakage=True
)
automl.search(X_train, y_train)

*****************************
* Beginning pipeline search *
*****************************

Optimizing for Recall. Greater score is better.

Searching up to 3 pipelines.

✔ XGBoost Binary Classification Pipel...    33%|███▎      | Elapsed:00:05
✔ Random Forest Binary Classification...    67%|██████▋   | Elapsed:00:17
✔ Logistic Regression Binary Pipeline:     100%|██████████| Elapsed:00:17
✔ Optimization finished                    100%|██████████| Elapsed:00:17

then we can retrain the best pipeline on all of our training data and see how it performs compared to the estimate

pipeline = automl.best_pipeline
pipeline.fit(X_train, y_train)
pipeline.score(X_holdout, y_holdout, ["recall"])

OrderedDict([('Recall', 0.9861111111111112)])