Setting up pipeline search

Designing the right machine learning pipeline and picking the best parameters is a time-consuming process that relies on a mix of data science intuition as well as trial and error. EvalML streamlines the process of selecting the best modeling algorithms and parameters, so data scientists can focus their energy where it is most needed.

How it works

EvalML selects and tunes machine learning pipelines built of numerous steps. This includes encoding categorical data, missing value imputation, feature selection, feature scaling, and finally machine learning. As EvalML tunes pipelines, it uses the objective function selected and configured by the user to guide its search.

At each iteration, EvalML uses cross-validation to generate an estimate of the pipeline’s performances. If a pipeline has high variance across cross-validation folds, it will provide a warning. In this case, the pipeline may not perform reliably in the future.

EvalML is designed to work well out of the box. However, it provides numerous methods for you to control the search described below.

Selecting problem type

EvalML supports both classification and regression problems. You select your problem type by importing the appropriate class.

import evalml
from evalml import AutoClassificationSearch, AutoRegressionSearch

AutoClassificationSearch()

Using default limit of max_pipelines=5.

<evalml.automl.auto_classification_search.AutoClassificationSearch at 0x7f14940a1710>

AutoRegressionSearch()

Using default limit of max_pipelines=5.

<evalml.automl.auto_regression_search.AutoRegressionSearch at 0x7f144052a550>

Setting the Objective Function

The only required parameter to start searching for pipelines is the objective function. Most domain-specific objective functions require you to specify parameters based on your business assumptions. You can do this before you initialize your pipeline search. For example

from evalml.objectives import FraudCost

fraud_objective = FraudCost(
    retry_percentage=.5,
    interchange_fee=.02,
    fraud_payout_percentage=.75,
    amount_col='amount'
)

AutoClassificationSearch(objective=fraud_objective, optimize_thresholds=True)

Using default limit of max_pipelines=5.

<evalml.automl.auto_classification_search.AutoClassificationSearch at 0x7f144052acf8>

Evaluate on Additional Objectives

Additional objectives can be scored on during the evaluation process. To add another objective, use the additional_objectives parameter in AutoClassificationSearch or AutoRegressionSearch. The results of these additional objectives will then appear in the results of describe_pipeline.

from evalml.objectives import FraudCost

fraud_objective = FraudCost(
    retry_percentage=.5,
    interchange_fee=.02,
    fraud_payout_percentage=.75,
    amount_col='amount'
)

AutoClassificationSearch(objective='AUC', additional_objectives=[fraud_objective], optimize_thresholds=False)

Using default limit of max_pipelines=5.

<evalml.automl.auto_classification_search.AutoClassificationSearch at 0x7f144054f198>

Selecting Model Types

By default, all model types are considered. You can control which model types to search with the allowed_model_families parameters

automl = AutoClassificationSearch(objective="f1",
                                  allowed_model_families=["random_forest"])

Using default limit of max_pipelines=5.

After initialization you can view the pipelines that will be included in the search

automl.allowed_pipelines

[evalml.pipelines.classification.random_forest_binary.RFBinaryClassificationPipeline]

you can see a list of all supported models like this

evalml.list_model_families("binary") # `binary` for binary classification and `multiclass` for multiclass classification

[<ModelFamily.CATBOOST: 'catboost'>,
 <ModelFamily.LINEAR_MODEL: 'linear_model'>,
 <ModelFamily.XGBOOST: 'xgboost'>,
 <ModelFamily.RANDOM_FOREST: 'random_forest'>]

evalml.list_model_families("regression")

[<ModelFamily.CATBOOST: 'catboost'>,
 <ModelFamily.LINEAR_MODEL: 'linear_model'>,
 <ModelFamily.XGBOOST: 'xgboost'>,
 <ModelFamily.RANDOM_FOREST: 'random_forest'>]

Limiting Search Time

You can limit the search time by specifying a maximum number of pipelines and/or a maximum amount of time. EvalML won’t build new pipelines after the maximum time has passed or the maximum number of pipelines have been built. If a limit is not set, then a maximum of 5 pipelines will be built.

The maximum search time can be specified as a integer in seconds or as a string in seconds, minutes, or hours.

AutoClassificationSearch(objective="f1",
                         max_pipelines=5,
                         max_time=60)

AutoClassificationSearch(objective="f1",
                         max_time="1 minute")

<evalml.automl.auto_classification_search.AutoClassificationSearch at 0x7f144054fc88>

Early Stopping

You can also limit search time by providing a patience value for early stopping. With a patience value, EvalML will stop searching when the best objective score has not been improved upon for n iterations. The patience value must be a positive integer. You can also provide a tolerance value where EvalML will only consider a score as an improvement over the best score if the difference was greater than the tolerance percentage.

from evalml.demos import load_diabetes

X, y = load_diabetes()
automl = AutoRegressionSearch(objective="MSE", patience=2, tolerance=0.01, max_pipelines=10)
automl.search(X, y)

*****************************
* Beginning pipeline search *
*****************************

Optimizing for MSE.
Lower score is better.

Searching up to 10 pipelines.
Allowed model families: catboost, linear_model, xgboost, random_forest

✔ Mean Baseline Regression Pipeline:         0%|          | Elapsed:00:00
✔ Cat Boost Regression Pipeline:            10%|█         | Elapsed:00:03
✔ Linear Regression Pipeline:               20%|██        | Elapsed:00:03
✔ Random Forest Regression Pipeline:        30%|███       | Elapsed:00:04
✔ XGBoost Regression Pipeline:              40%|████      | Elapsed:00:04


2 iterations without improvement. Stopping search early...
✔ Optimization finished                     40%|████      | Elapsed:00:04

automl.rankings

	id	pipeline_name	score	high_variance_cv	parameters
0	2	Linear Regression Pipeline	3027.144520	False	{'One Hot Encoder': {'top_n': 10}, 'Simple Imp...
1	3	Random Forest Regression Pipeline	3260.330484	False	{'One Hot Encoder': {'top_n': 10}, 'Simple Imp...
2	1	Cat Boost Regression Pipeline	3279.699820	False	{'Simple Imputer': {'impute_strategy': 'most_f...
3	4	XGBoost Regression Pipeline	3960.404473	False	{'One Hot Encoder': {'top_n': 10}, 'Simple Imp...
4	0	Mean Baseline Regression Pipeline	5943.716736	False	{'strategy': 'mean'}

Control Cross Validation

EvalML cross-validates each model it tests during its search. By default it uses 3-fold cross-validation. You can optionally provide your own cross-validation method.

from sklearn.model_selection import StratifiedKFold

automl = AutoClassificationSearch(objective="f1",
                                  cv=StratifiedKFold(5))

Using default limit of max_pipelines=5.