Setting up pipeline search

Designing the right machine learning pipeline and picking the best parameters is a time-consuming process that relies on a mix of data science intuition as well as trial and error. EvalML streamlines the process of selecting the best modeling algorithms and parameters, so data scientists can focus their energy where it is most needed.

How it works

EvalML selects and tunes machine learning pipelines built of numerous steps. This includes encoding categorical data, missing value imputation, feature selection, feature scaling, and finally machine learning. As EvalML tunes pipelines, it uses the objective function selected and configured by the user to guide its search.

At each iteration, EvalML uses cross-validation to generate an estimate of the pipeline’s performances. If a pipeline has high variance across cross-validation folds, it will provide a warning. In this case, the pipeline may not perform reliably in the future.

EvalML is designed to work well out of the box. However, it provides numerous methods for you to control the search described below.

Selecting problem type

EvalML supports both classification and regression problems. You select your problem type by importing the appropriate class.

import evalml
from evalml import AutoClassificationSearch, AutoRegressionSearch

AutoClassificationSearch()

<evalml.automl.auto_classification_search.AutoClassificationSearch at 0x7f6c75d4ac18>

AutoRegressionSearch()

<evalml.automl.auto_regression_search.AutoRegressionSearch at 0x7f6c5e7fd5c0>

Setting the Objective Function

The only required parameter to start searching for pipelines is the objective function. Most domain-specific objective functions require you to specify parameters based on your business assumptions. You can do this before you initialize your pipeline search. For example

from evalml.objectives import FraudCost

fraud_objective = FraudCost(
    retry_percentage=.5,
    interchange_fee=.02,
    fraud_payout_percentage=.75,
    amount_col='amount'
)

AutoClassificationSearch(objective=fraud_objective, optimize_thresholds=True)

<evalml.automl.auto_classification_search.AutoClassificationSearch at 0x7f6c5e818b38>

Evaluate on Additional Objectives

Additional objectives can be scored on during the evaluation process. To add another objective, use the additional_objectives parameter in AutoClassificationSearch or AutoRegressionSearch. The results of these additional objectives will then appear in the results of describe_pipeline.

from evalml.objectives import FraudCost

fraud_objective = FraudCost(
    retry_percentage=.5,
    interchange_fee=.02,
    fraud_payout_percentage=.75,
    amount_col='amount'
)

AutoClassificationSearch(objective='AUC', additional_objectives=[fraud_objective], optimize_thresholds=False)

<evalml.automl.auto_classification_search.AutoClassificationSearch at 0x7f6c5e81be10>

Selecting Model Types

By default, all model types are considered. You can control which model types to search with the allowed_model_families parameters

automl = AutoClassificationSearch(objective="f1",
                                  allowed_model_families=["random_forest"])

you can see the possible pipelines that will be searched after initialization

automl.possible_pipelines

[evalml.pipelines.classification.random_forest_binary.RFBinaryClassificationPipeline]

you can see a list of all supported models like this

evalml.list_model_families("binary") # `binary` for binary classification and `multiclass` for multiclass classification

[<ModelFamily.XGBOOST: 'xgboost'>,
 <ModelFamily.RANDOM_FOREST: 'random_forest'>,
 <ModelFamily.LINEAR_MODEL: 'linear_model'>,
 <ModelFamily.CATBOOST: 'catboost'>]

evalml.list_model_families("regression")

[<ModelFamily.XGBOOST: 'xgboost'>,
 <ModelFamily.RANDOM_FOREST: 'random_forest'>,
 <ModelFamily.LINEAR_MODEL: 'linear_model'>,
 <ModelFamily.CATBOOST: 'catboost'>]

Limiting Search Time

You can limit the search time by specifying a maximum number of pipelines and/or a maximum amount of time. EvalML won’t build new pipelines after the maximum time has passed or the maximum number of pipelines have been built. If a limit is not set, then a maximum of 5 pipelines will be built.

The maximum search time can be specified as a integer in seconds or as a string in seconds, minutes, or hours.

AutoClassificationSearch(objective="f1",
                         max_pipelines=5,
                         max_time=60)

AutoClassificationSearch(objective="f1",
                         max_time="1 minute")

<evalml.automl.auto_classification_search.AutoClassificationSearch at 0x7f6c5e7c4128>

To start, EvalML samples 10 sets of hyperparameters chosen randomly for each possible pipeline. Therefore, we recommend setting max_pipelines at least 10 times the number of possible pipelines.

n_possible_pipelines = len(AutoClassificationSearch(objective="f1").possible_pipelines)

AutoClassificationSearch(objective="f1",
                         max_time=60)

<evalml.automl.auto_classification_search.AutoClassificationSearch at 0x7f6c5e7d5c88>

Early Stopping

You can also limit search time by providing a patience value for early stopping. With a patience value, EvalML will stop searching when the best objective score has not been improved upon for n iterations. The patience value must be a positive integer. You can also provide a tolerance value where EvalML will only consider a score as an improvement over the best score if the difference was greater than the tolerance percentage.

from evalml.demos import load_diabetes

X, y = load_diabetes()
automl = AutoRegressionSearch(objective="MSE", patience=2, tolerance=0.01, max_pipelines=10)
automl.search(X, y)

*****************************
* Beginning pipeline search *
*****************************

Optimizing for MSE. Lower score is better.

Searching up to 10 pipelines.

✔ XGBoost Regression Pipeline:              10%|█         | Elapsed:00:03
✔ Cat Boost Regression Pipeline:            20%|██        | Elapsed:00:12
✔ Random Forest Regression Pipeline:        30%|███       | Elapsed:00:15
✔ XGBoost Regression Pipeline:              40%|████      | Elapsed:00:20


2 iterations without improvement. Stopping search early...
✔ Optimization finished                     40%|████      | Elapsed:00:20

automl.rankings

	id	pipeline_name	score	high_variance_cv	parameters
0	1	Cat Boost Regression Pipeline	3566.688649	False	{'impute_strategy': 'most_frequent', 'n_estima...
1	0	XGBoost Regression Pipeline	4443.846724	False	{'impute_strategy': 'most_frequent', 'percent_...
2	2	Random Forest Regression Pipeline	5615.790436	False	{'impute_strategy': 'most_frequent', 'percent_...

Control Cross Validation

EvalML cross-validates each model it tests during its search. By default it uses 3-fold cross-validation. You can optionally provide your own cross-validation method.

from sklearn.model_selection import StratifiedKFold

automl = AutoClassificationSearch(objective="f1",
                                  cv=StratifiedKFold(5))