Avoiding Overfitting

The ultimate goal of machine learning is to make accurate predictions on unseen data. EvalML aims to help you build a model that will perform as you expect once it is deployed in to the real world.

One of the benefits of using EvalML to build models is that it provides guardrails to ensure you are building pipelines that will perform reliably in the future. This page describes the various ways EvalML helps you avoid overfitting to your data.

import evalml

Detecting Label Leakage

A common problem is having features that include information from your label in your training data. By default, EvalML will provide a warning when it detects this may be the case.

Let’s set up a simple example to demonstrate what this looks like

import pandas as pd

X = pd.DataFrame({
    "leaked_feature": [6, 6, 10, 5, 5, 11, 5, 10, 11, 4],
    "leaked_feature_2": [3, 2.5, 5, 2.5, 3, 5.5, 2, 5, 5.5, 2],
    "valid_feature": [3, 1, 3, 2, 4, 6, 1, 3, 3, 11]
})

y = pd.Series([1, 1, 0, 1, 1, 0, 1, 0, 0, 1])

automl = evalml.AutoClassificationSearch(
    max_pipelines=1,
    model_types=["linear_model"],
)

automl.search(X, y)

*****************************
* Beginning pipeline search *
*****************************

Optimizing for Precision. Greater score is better.

Searching up to 1 pipelines.
Possible model types: linear_model

WARNING: Possible label leakage: leaked_feature, leaked_feature_2

✔ Logistic Regression Classifier w/ O...   100%|██████████| Elapsed:00:01
✔ Optimization finished                    100%|██████████| Elapsed:00:01

In the example above, EvalML warned about the input features leaked_feature and leak_feature_2, which are both very closely correlated with the label we are trying to predict. If you’d like to turn this check off, set detect_label_leakage=False.

The second way to find features that may be leaking label information is to look at the top features of the model. As we can see below, the top features in our model are the 2 leaked features.

best_pipeline = automl.best_pipeline
best_pipeline.feature_importances

	feature	importance
0	leaked_feature	-1.782149
1	leaked_feature_2	-1.638703
2	valid_feature	-0.398194

Perform cross-validation for pipeline evaluation

By default, EvalML performs 3-fold cross validation when building pipelines. This means that it evaluates each pipeline 3 times using different sets of data for training and testing. In each trial, the data used for testing has no overlap from the data used for training.

While this is a good baseline approach, you can pass your own cross validation object to be used during modeling. The cross validation object can be any of the CV methods defined in scikit-learn or use a compatible API.

For example, if we wanted to do a time series split:

from sklearn.model_selection import TimeSeriesSplit

X, y = evalml.demos.load_breast_cancer()

automl = evalml.AutoClassificationSearch(
    cv=TimeSeriesSplit(n_splits=6),
    max_pipelines=1
)

automl.search(X, y)

*****************************
* Beginning pipeline search *
*****************************

Optimizing for Precision. Greater score is better.

Searching up to 1 pipelines.
Possible model types: linear_model, xgboost, random_forest, catboost

✔ CatBoost Classifier w/ Simple Imput...   100%|██████████| Elapsed:00:02
✔ Optimization finished                    100%|██████████| Elapsed:00:02

if we describe the 1 pipeline we built, we can see the scores for each of the 6 splits as determined by the cross-validation object we provided. We can also see the number of training examples per fold increased because we were using TimeSeriesSplit

automl.describe_pipeline(0)

*****************************************
* CatBoost Classifier w/ Simple Imputer *
*****************************************

Problem Types: Binary Classification, Multiclass Classification
Model Type: CatBoost Classifier
Objective to Optimize: Precision (greater is better)
Number of features: 30

Pipeline Steps
==============
1. Simple Imputer
         * impute_strategy : most_frequent
2. CatBoost Classifier
         * n_estimators : 202
         * eta : 0.602763376071644
         * max_depth : 4

Training
========
Training for Binary Classification problems.
Total training time (including CV): 2.9 seconds

Cross Validation
----------------
             Precision    F1  Recall   AUC  Log Loss   MCC # Training # Testing
0                0.953 0.863   0.788 0.938     0.908 0.691     83.000    81.000
1                1.000 0.925   0.860 0.998     0.133 0.862    164.000    81.000
2                0.964 0.982   1.000 0.968     0.153 0.945    245.000    81.000
3                1.000 0.982   0.966 0.999     0.063 0.942    326.000    81.000
4                1.000 0.984   0.969 1.000     0.042 0.931    407.000    81.000
5                0.983 0.983   0.983 0.996     0.065 0.936    488.000    81.000
mean             0.984 0.953   0.928 0.983     0.227 0.885          -         -
std              0.020 0.050   0.084 0.025     0.336 0.100          -         -
coef of var      0.021 0.052   0.091 0.026     1.477 0.113          -         -

Detect unstable pipelines

When we perform cross validation we are trying generate an estimate of pipeline performance. EvalML does this by taking the mean of the score across the folds. If the performance across the folds varies greatly, it is indicative the the estimated value may be unreliable.

To protect the user against this, EvalML checks to see if the pipeline’s performance has a variance between the different folds. EvalML triggers a warning if the “coefficient of variance” of the scores (the standard deviation divided by mean) of the pipelines scores exeeds .2.

This warning will appear in the pipeline rankings under high_variance_cv.

automl.rankings

	id	pipeline_class_name	score	high_variance_cv	parameters
0	0	CatBoostClassificationPipeline	0.983518	False	{'impute_strategy': 'most_frequent', 'n_estima...

Create holdout for model validation

EvalML offers a method to quickly create an holdout validation set. A holdout validation set is data that is not used during the process of optimizing or training the model. You should only use this validation set once you’ve picked the final model you’d like to use.

Below we create a holdout set of 20% of our data

X, y = evalml.demos.load_breast_cancer()
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, test_size=.2)

automl = evalml.AutoClassificationSearch(
    objective="recall",
    max_pipelines=3,
    detect_label_leakage=True
)
automl.search(X_train, y_train)

*****************************
* Beginning pipeline search *
*****************************

Optimizing for Recall. Greater score is better.

Searching up to 3 pipelines.
Possible model types: linear_model, xgboost, random_forest, catboost

✔ CatBoost Classifier w/ Simple Imput...    33%|███▎      | Elapsed:00:01
✔ CatBoost Classifier w/ Simple Imput...    67%|██████▋   | Elapsed:00:09
✔ Random Forest Classifier w/ One Hot...   100%|██████████| Elapsed:00:20
✔ Optimization finished                    100%|██████████| Elapsed:00:20

then we can retrain the best pipeline on all of our training data and see how it performs compared to the estimate

pipeline = automl.best_pipeline
pipeline.fit(X_train, y_train)
pipeline.score(X_holdout, y_holdout)

(0.9305555555555556, {})