Automated Machine Learning (AutoML) Search#

Background#

Machine Learning#

Machine learning (ML) is the process of constructing a mathematical model of a system based on a sample dataset collected from that system.

One of the main goals of training an ML model is to teach the model to separate the signal present in the data from the noise inherent in system and in the data collection process. If this is done effectively, the model can then be used to make accurate predictions about the system when presented with new, similar data. Additionally, introspecting on an ML model can reveal key information about the system being modeled, such as which inputs and transformations of the inputs are most useful to the ML model for learning the signal in the data, and are therefore the most predictive.

There are a variety of ML problem types. Supervised learning describes the case where the collected data contains an output value to be modeled and a set of inputs with which to train the model. EvalML focuses on training supervised learning models.

EvalML supports three common supervised ML problem types. The first is regression, where the target value to model is a continuous numeric value. Next are binary and multiclass classification, where the target value to model consists of two or more discrete values or categories. The choice of which supervised ML problem type is most appropriate depends on domain expertise and on how the model will be evaluated and used.

EvalML is currently building support for supervised time series problems: time series regression, time series binary classification, and time series multiclass classification. While we’ve added some features to tackle these kinds of problems, our functionality is still being actively developed so please be mindful of that before using it.

AutoML and Search#

AutoML is the process of automating the construction, training and evaluation of ML models. Given a data and some configuration, AutoML searches for the most effective and accurate ML model or models to fit the dataset. During the search, AutoML will explore different combinations of model type, model parameters and model architecture.

An effective AutoML solution offers several advantages over constructing and tuning ML models by hand. AutoML can assist with many of the difficult aspects of ML, such as avoiding overfitting and underfitting, imbalanced data, detecting data leakage and other potential issues with the problem setup, and automatically applying best-practice data cleaning, feature engineering, feature selection and various modeling techniques. AutoML can also leverage search algorithms to optimally sweep the hyperparameter search space, resulting in model performance which would be difficult to achieve by manual training.

AutoML in EvalML#

EvalML supports all of the above and more.

In its simplest usage, the AutoML search interface requires only the input data, the target data and a problem_type specifying what kind of supervised ML problem to model.

** Graphing methods, like verbose AutoMLSearch, on Jupyter Notebook and Jupyter Lab require ipywidgets to be installed.

** If graphing on Jupyter Lab, jupyterlab-plotly required. To download this, make sure you have npm installed.

[1]:

import evalml
from evalml.utils import infer_feature_types

X, y = evalml.demos.load_fraud(n_rows=650)

pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace.

             Number of Features
Boolean                       1
Categorical                   6
Numeric                       5

Number of training examples: 650
Targets
False    86.31%
True     13.69%
Name: fraud, dtype: object

To provide data to EvalML, it is recommended that you initialize a Woodwork accessor on your data. This allows you to easily control how EvalML will treat each of your features before training a model.

EvalML also accepts pandas input, and will run type inference on top of the input pandas data. If you’d like to change the types inferred by EvalML, you can use the infer_feature_types utility method, which takes pandas or numpy input and converts it to a Woodwork data structure. The feature_types parameter can be used to specify what types specific columns should be.

Feature types such as Natural Language must be specified in this way, otherwise Woodwork will infer it as Unknown type and drop it during the AutoMLSearch.

In the example below, we reformat a couple features to make them easily consumable by the model, and then specify that the provider, which would have otherwise been inferred as a column with natural language, is a categorical column.

[2]:

X.ww["expiration_date"] = X["expiration_date"].apply(
    lambda x: "20{}-01-{}".format(x.split("/")[1], x.split("/")[0])
)
X = infer_feature_types(
    X,
    feature_types={
        "store_id": "categorical",
        "expiration_date": "datetime",
        "lat": "categorical",
        "lng": "categorical",
        "provider": "categorical",
    },
)

In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set.

[3]:

X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(
    X, y, problem_type="binary", test_size=0.2
)

Data Checks#

Before calling AutoMLSearch.search, we should run some sanity checks on our data to ensure that the input data being passed will not run into some common issues before running a potentially time-consuming search. EvalML has various data checks that makes this easy. Each data check will return a collection of warnings and errors if it detects potential issues with the input data. This allows users to inspect their data to avoid confusing errors that may arise during the search process. You can learn about each of the data checks available through our data checks guide.

Here, we will run the DefaultDataChecks class, which contains a series of data checks that are generally useful.

[4]:

from evalml.data_checks import DefaultDataChecks

data_checks = DefaultDataChecks("binary", "log loss binary")
data_checks.validate(X_train, y_train)

[4]:

[]

Since there were no warnings or errors returned, we can safely continue with the search process.

Holdout Set for Pipeline Ranking#

If the holdout_set_size parameter is set and the input dataset has more than 500 rows, AutoMLSearch will create a holdout set from holdout_set_size of the training data. Alternatively, a holdout set can be manually specified by using the X_holdout and y_holdout parameters in AutoMLSearch(). In this example, the holdout set created previously will be used by AutoML search.

During the AutoML search process, the mean of the objective scores of all cross validation folds (shown the “mean_cv_score” column in the pipeline rankings), is calculated. This score is passed to the AutoML search tuner to further optimize the hyperparameters of the next batch of pipelines.

After, the pipeline will be fitted on the entire training dataset and scored on this new holdout set. This score is represented under the “validation_score” column on the pipeline rankings board and is used to rank pipeline performance.

If a dataset has less than 500 rows or holdout_set_size=0 (which is the default setting), the “mean_cv_score” will be used as the validation_score instead.

[5]:

automl = evalml.automl.AutoMLSearch(
    X_train=X_train,
    y_train=y_train,
    X_holdout=X_holdout,
    y_holdout=y_holdout,
    problem_type="binary",
    verbose=True,
)
automl.search(interactive_plot=False)

AutoMLSearch will use the holdout set to score and rank pipelines.
Removing columns ['currency'] because they are of 'Unknown' type
Using default limit of max_batches=3.


*****************************
* Beginning pipeline search *
*****************************

Optimizing for Log Loss Binary.
Lower score is better.

Using SequentialEngine to train and score pipelines.
Searching up to 3 batches for a total of None pipelines.
Allowed model families:

Evaluating Baseline Pipeline: Mode Baseline Binary Classification Pipeline
Mode Baseline Binary Classification Pipeline:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 4.716
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 4.782

*****************************
* Evaluating Batch Number 1 *
*****************************

Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.378
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 0.392
Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.259
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 0.219

*****************************
* Evaluating Batch Number 2 *
*****************************

Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler + RF Classifier Select From Model:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.376
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 0.340
Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + RF Classifier Select From Model:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.250
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 0.203

*****************************
* Evaluating Batch Number 3 *
*****************************

Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 1.403
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 1.474
        High coefficient of variation (cv >= 0.5) within cross validation scores.
        Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler may not perform as estimated on unseen data.
LightGBM Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.319
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 0.183
Extra Trees Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.362
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 0.352
Elastic Net Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Standard Scaler + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Standard Scaler + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.375
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 0.400
CatBoost Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.575
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 0.552
XGBoost Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.253
        Starting holdout set scoring
        Finished holdout set scoring - Log Loss Binary: 0.151

Search finished after 00:42
Best pipeline: XGBoost Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler
Best pipeline Log Loss Binary: 0.150534

[5]:

{1: {'Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler': '00:05',
  'Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler': '00:03',
  'Total time of batch': '00:09'},
 2: {'Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler + RF Classifier Select From Model': '00:04',
  'Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + RF Classifier Select From Model': '00:04',
  'Total time of batch': '00:08'},
 3: {'Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler': '00:03',
  'LightGBM Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler': '00:03',
  'Extra Trees Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler': '00:04',
  'Elastic Net Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Standard Scaler + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Standard Scaler + Oversampler': '00:04',
  'CatBoost Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + Oversampler': '00:02',
  'XGBoost Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler': '00:04',
  'Total time of batch': '00:22'}}

With the verbose argument set to True, the AutoML search will log its progress, reporting each pipeline and parameter set evaluated during the search. The search iteration plot shown during AutoML search tracks the current pipeline’s validation score (tracked as the gray point) against the best pipeline validation score (tracked as the blue line).

There are a number of mechanisms to control the AutoML search time. One way is to set the max_batches parameter which controls the maximum number of rounds of AutoML to evaluate, where each round may train and score a variable number of pipelines. Another way is to set the max_iterations parameter which controls the maximum number of candidate models to be evaluated during AutoML. By default, AutoML will search for a single batch. The first pipeline to be evaluated will always be a baseline model representing a trivial solution.

The AutoML interface supports a variety of other parameters. For a comprehensive list, please refer to the API reference.

We also provide a standalone search method which does all of the above in a single line, and returns the AutoMLSearch instance and data check results. If there were data check errors, AutoML will not be run and no AutoMLSearch instance will be returned.

Detecting Problem Type#

EvalML includes a simple method, detect_problem_type, to help determine the problem type given the target data.

This function can return the predicted problem type as a ProblemType enum, choosing from ProblemType.BINARY, ProblemType.MULTICLASS, and ProblemType.REGRESSION. If the target data is invalid (for instance when there is only 1 unique label), the function will throw an error instead.

[6]:

import pandas as pd
from evalml.problem_types import detect_problem_type

y_binary = pd.Series([0, 1, 1, 0, 1, 1])
detect_problem_type(y_binary)

[6]:

<ProblemTypes.BINARY: 'binary'>

Objective parameter#

AutoMLSearch takes in an objective parameter to determine which objective to optimize for. By default, this parameter is set to auto, which allows AutoML to choose LogLossBinary for binary classification problems, LogLossMulticlass for multiclass classification problems, and R2 for regression problems.

It should be noted that the objective parameter is only used in ranking and helping choose the pipelines to iterate over, but is not used to optimize each individual pipeline during fit-time.

To get the default objective for each problem type, you can use the get_default_primary_search_objective function.

[7]:

from evalml.automl import get_default_primary_search_objective

binary_objective = get_default_primary_search_objective("binary")
multiclass_objective = get_default_primary_search_objective("multiclass")
regression_objective = get_default_primary_search_objective("regression")

print(binary_objective.name)
print(multiclass_objective.name)
print(regression_objective.name)

Log Loss Binary
Log Loss Multiclass
R2

Using custom pipelines#

EvalML’s AutoML algorithm generates a set of pipelines to search with. To provide a custom set instead, set allowed_component_graphs to a dictionary of custom component graphs. AutoMLSearch will use these to generate Pipeline instances. Note: this will prevent AutoML from generating other pipelines to search over.

[8]:

from evalml.pipelines import MulticlassClassificationPipeline


automl_custom = evalml.automl.AutoMLSearch(
    X_train=X_train,
    y_train=y_train,
    problem_type="multiclass",
    verbose=True,
    allowed_component_graphs={
        "My_pipeline": ["Simple Imputer", "Random Forest Classifier"],
        "My_other_pipeline": ["One Hot Encoder", "Random Forest Classifier"],
    },
)

AutoMLSearch will use mean CV score to rank pipelines.
Removing columns ['currency'] because they are of 'Unknown' type
Using default limit of max_batches=3.

Stopping the search early#

To stop the search early, hit Ctrl-C. This will bring up a prompt asking for confirmation. Responding with y will immediately stop the search. Responding with n will continue the search.

Callback functions#

AutoMLSearch supports several callback functions, which can be specified as parameters when initializing an AutoMLSearch object. They are:

start_iteration_callback
add_result_callback
error_callback

Start Iteration Callback#

Users can set start_iteration_callback to set what function is called before each pipeline training iteration. This callback function must take three positional parameters: the pipeline class, the pipeline parameters, and the AutoMLSearch object.

[9]:

## start_iteration_callback example function
def start_iteration_callback_example(pipeline_class, pipeline_params, automl_obj):
    print("Training pipeline with the following parameters:", pipeline_params)

Add Result Callback#

Users can set add_result_callback to set what function is called after each pipeline training iteration. This callback function must take three positional parameters: a dictionary containing the training results for the new pipeline, an untrained_pipeline containing the parameters used during training, and the AutoMLSearch object.

[10]:

## add_result_callback example function
def add_result_callback_example(pipeline_results_dict, untrained_pipeline, automl_obj):
    print(
        "Results for trained pipeline with the following parameters:",
        pipeline_results_dict,
    )

Error Callback#

Users can set the error_callback to set what function called when search() errors and raises an Exception. This callback function takes three positional parameters: the Exception raised, the traceback, and the AutoMLSearch object. This callback function must also accept kwargs, so AutoMLSearch is able to pass along other parameters used by default.

Evalml defines several error callback functions, which can be found under evalml.automl.callbacks. They are:

silent_error_callback
raise_error_callback
log_and_save_error_callback
raise_and_save_error_callback
log_error_callback (default used when error_callback is None)

[11]:

# error_callback example; this is implemented in the evalml library
def raise_error_callback(exception, traceback, automl, **kwargs):
    """Raises the exception thrown by the AutoMLSearch object. Also logs the exception as an error."""
    logger.error(f"AutoMLSearch raised a fatal exception: {str(exception)}")
    logger.error("\n".join(traceback))
    raise exception

View Rankings#

A summary of all the pipelines built can be returned as a pandas DataFrame which is sorted by the validation score.

For AutoML searches completed with a holdout set, the validation score is the holdout score of the pipeline fitted using the entire training dataset.
For AutoML searches completed without a holdout set, the validation score is the average score across all cross-validation folds.

[12]:

automl.rankings

[12]:

	id	pipeline_name	search_order	validation_score	mean_cv_score	standard_deviation_cv_score	percent_better_than_baseline	high_variance_cv	parameters
0	10	XGBoost Classifier w/ Label Encoder + Select C...	10	0.150534	0.253426	0.143075	94.626002	False	{'Label Encoder': {'positive_label': None}, 'N...
1	6	LightGBM Classifier w/ Label Encoder + Select ...	6	0.183402	0.318972	0.229223	93.236069	False	{'Label Encoder': {'positive_label': None}, 'N...
2	4	Random Forest Classifier w/ Label Encoder + Re...	4	0.203164	0.250202	0.047341	94.694366	False	{'Label Encoder': {'positive_label': None}, 'D...
3	2	Random Forest Classifier w/ Label Encoder + Re...	2	0.219081	0.259176	0.039362	94.504077	False	{'Label Encoder': {'positive_label': None}, 'D...
4	3	Logistic Regression Classifier w/ Label Encode...	3	0.340020	0.376248	0.044072	92.021517	False	{'Label Encoder': {'positive_label': None}, 'D...
5	7	Extra Trees Classifier w/ Label Encoder + Sele...	7	0.352442	0.361760	0.021902	92.328738	False	{'Label Encoder': {'positive_label': None}, 'N...
6	1	Logistic Regression Classifier w/ Label Encode...	1	0.392000	0.378335	0.046602	91.977246	False	{'Label Encoder': {'positive_label': None}, 'D...
7	8	Elastic Net Classifier w/ Label Encoder + Sele...	8	0.400375	0.374725	0.050027	92.053807	False	{'Label Encoder': {'positive_label': None}, 'N...
8	9	CatBoost Classifier w/ Label Encoder + Select ...	9	0.551642	0.574966	0.016860	87.807609	False	{'Label Encoder': {'positive_label': None}, 'N...
9	5	Decision Tree Classifier w/ Label Encoder + Se...	5	1.474369	1.403004	0.666162	70.248732	True	{'Label Encoder': {'positive_label': None}, 'N...
10	0	Mode Baseline Binary Classification Pipeline	0	4.782292	4.715779	0.108196	0.000000	False	{'Label Encoder': {'positive_label': None}, 'B...

Describe Pipeline#

Each pipeline is given an id. We can get more information about any particular pipeline using that id. Here, we will get more information about the pipeline with id = 1.

[13]:

automl.describe_pipeline(1)


*****************************************************************************************************************************************************************************************************
* Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler *
*****************************************************************************************************************************************************************************************************

Problem Type: binary
Model Family: Linear

Pipeline Steps
==============
1. Label Encoder
         * positive_label : None
2. Replace Nullable Types Transformer
3. Drop Columns Transformer
         * columns : ['currency']
4. DateTime Featurizer
         * features_to_extract : ['year', 'month', 'day_of_week', 'hour']
         * encode_as_categories : False
         * time_index : None
5. Imputer
         * categorical_impute_strategy : most_frequent
         * numeric_impute_strategy : mean
         * boolean_impute_strategy : most_frequent
         * categorical_fill_value : None
         * numeric_fill_value : None
         * boolean_fill_value : None
6. One Hot Encoder
         * top_n : 10
         * features_to_encode : None
         * categories : None
         * drop : if_binary
         * handle_unknown : ignore
         * handle_missing : error
7. Oversampler
         * sampling_ratio : 0.25
         * k_neighbors_default : 5
         * n_jobs : -1
         * sampling_ratio_dict : None
         * categorical_features : [3, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
         * k_neighbors : 5
8. Standard Scaler
9. Logistic Regression Classifier
         * penalty : l2
         * C : 1.0
         * n_jobs : -1
         * multi_class : auto
         * solver : lbfgs

Training
========
Training for binary problems.
Total training time (including CV): 5.5 seconds

Cross Validation
----------------
             Log Loss Binary  MCC Binary  Gini   AUC  Precision    F1  Balanced Accuracy Binary  Accuracy Binary # Training # Validation
0                      0.398       0.174 0.370 0.685      0.333 0.256                     0.571            0.833        346          174
1                      0.412       0.076 0.310 0.655      0.250 0.129                     0.523            0.844        347          173
2                      0.325       0.482 0.570 0.785      0.600 0.545                     0.723            0.884        347          173
mean                   0.378       0.244 0.417 0.708      0.394 0.310                     0.606            0.854          -            -
std                    0.047       0.212 0.136 0.068      0.183 0.213                     0.104            0.027          -            -
coef of var            0.123       0.869 0.327 0.096      0.464 0.688                     0.172            0.032          -            -

Get Pipeline#

We can get the object of any pipeline via their id as well:

[14]:

pipeline = automl.get_pipeline(1)
print(pipeline.name)
print(pipeline.parameters)

Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler
{'Label Encoder': {'positive_label': None}, 'Drop Columns Transformer': {'columns': ['currency']}, 'DateTime Featurizer': {'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'time_index': None}, 'Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'boolean_impute_strategy': 'most_frequent', 'categorical_fill_value': None, 'numeric_fill_value': None, 'boolean_fill_value': None}, 'One Hot Encoder': {'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'Oversampler': {'sampling_ratio': 0.25, 'k_neighbors_default': 5, 'n_jobs': -1, 'sampling_ratio_dict': None, 'categorical_features': [3, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49], 'k_neighbors': 5}, 'Logistic Regression Classifier': {'penalty': 'l2', 'C': 1.0, 'n_jobs': -1, 'multi_class': 'auto', 'solver': 'lbfgs'}}

Get best pipeline#

If you specifically want to get the best pipeline, there is a convenient accessor for that. The pipeline returned is already fitted on the input X, y data that we passed to AutoMLSearch. To turn off this default behavior, set train_best_pipeline=False when initializing AutoMLSearch.

[15]:

best_pipeline = automl.best_pipeline
print(best_pipeline.name)
print(best_pipeline.parameters)
best_pipeline.predict(X_train)

XGBoost Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler
{'Label Encoder': {'positive_label': None}, 'Numeric Pipeline - Select Columns By Type Transformer': {'column_types': ['Categorical', 'EmailAddress', 'URL'], 'exclude': True}, 'Numeric Pipeline - Label Encoder': {'positive_label': None}, 'Numeric Pipeline - Drop Columns Transformer': {'columns': ['currency']}, 'Numeric Pipeline - DateTime Featurizer': {'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'time_index': None}, 'Numeric Pipeline - Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'boolean_impute_strategy': 'most_frequent', 'categorical_fill_value': None, 'numeric_fill_value': None, 'boolean_fill_value': None}, 'Numeric Pipeline - Select Columns Transformer': {'columns': ['card_id', 'store_id', 'amount', 'customer_present', 'lat', 'lng', 'datetime_month', 'datetime_day_of_week', 'datetime_hour']}, 'Categorical Pipeline - Select Columns Transformer': {'columns': ['expiration_date', 'provider', 'region', 'country']}, 'Categorical Pipeline - Label Encoder': {'positive_label': None}, 'Categorical Pipeline - Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'boolean_impute_strategy': 'most_frequent', 'categorical_fill_value': None, 'numeric_fill_value': None, 'boolean_fill_value': None}, 'Categorical Pipeline - One Hot Encoder': {'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'Oversampler': {'sampling_ratio': 0.25, 'k_neighbors_default': 5, 'n_jobs': -1, 'sampling_ratio_dict': None, 'categorical_features': [3, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48], 'k_neighbors': 5}, 'XGBoost Classifier': {'eta': 0.1, 'max_depth': 6, 'min_child_weight': 1, 'n_estimators': 100, 'n_jobs': -1, 'eval_metric': 'logloss'}}

[15]:

id
144    False
253     True
221    False
432    False
384    False
       ...
128    False
98     False
472    False
642    False
494    False
Name: fraud, Length: 520, dtype: bool

Training and Scoring Multiple Pipelines using AutoMLSearch#

AutoMLSearch will automatically fit the best pipeline on the entire training data. It also provides an easy API for training and scoring other pipelines.

If you’d like to train one or more pipelines on the entire training data, you can use the train_pipelines method.

Similarly, if you’d like to score one or more pipelines on a particular dataset, you can use the score_pipelines method.

[16]:

trained_pipelines = automl.train_pipelines([automl.get_pipeline(i) for i in [0, 1, 2]])
trained_pipelines

[16]:

{'Mode Baseline Binary Classification Pipeline': pipeline = BinaryClassificationPipeline(component_graph={'Label Encoder': ['Label Encoder', 'X', 'y'], 'Baseline Classifier': ['Baseline Classifier', 'Label Encoder.x', 'Label Encoder.y']}, parameters={'Label Encoder':{'positive_label': None}, 'Baseline Classifier':{'strategy': 'mode'}}, custom_name='Mode Baseline Binary Classification Pipeline', random_seed=0),
 'Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler': pipeline = BinaryClassificationPipeline(component_graph={'Label Encoder': ['Label Encoder', 'X', 'y'], 'Replace Nullable Types Transformer': ['Replace Nullable Types Transformer', 'X', 'Label Encoder.y'], 'Drop Columns Transformer': ['Drop Columns Transformer', 'Replace Nullable Types Transformer.x', 'Replace Nullable Types Transformer.y'], 'DateTime Featurizer': ['DateTime Featurizer', 'Drop Columns Transformer.x', 'Replace Nullable Types Transformer.y'], 'Imputer': ['Imputer', 'DateTime Featurizer.x', 'Replace Nullable Types Transformer.y'], 'One Hot Encoder': ['One Hot Encoder', 'Imputer.x', 'Replace Nullable Types Transformer.y'], 'Oversampler': ['Oversampler', 'One Hot Encoder.x', 'Replace Nullable Types Transformer.y'], 'Standard Scaler': ['Standard Scaler', 'Oversampler.x', 'Oversampler.y'], 'Logistic Regression Classifier': ['Logistic Regression Classifier', 'Standard Scaler.x', 'Oversampler.y']}, parameters={'Label Encoder':{'positive_label': None}, 'Drop Columns Transformer':{'columns': ['currency']}, 'DateTime Featurizer':{'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'time_index': None}, 'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'boolean_impute_strategy': 'most_frequent', 'categorical_fill_value': None, 'numeric_fill_value': None, 'boolean_fill_value': None}, 'One Hot Encoder':{'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'Oversampler':{'sampling_ratio': 0.25, 'k_neighbors_default': 5, 'n_jobs': -1, 'sampling_ratio_dict': None, 'categorical_features': [3, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49], 'k_neighbors': 5}, 'Logistic Regression Classifier':{'penalty': 'l2', 'C': 1.0, 'n_jobs': -1, 'multi_class': 'auto', 'solver': 'lbfgs'}}, random_seed=0),
 'Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler': pipeline = BinaryClassificationPipeline(component_graph={'Label Encoder': ['Label Encoder', 'X', 'y'], 'Replace Nullable Types Transformer': ['Replace Nullable Types Transformer', 'X', 'Label Encoder.y'], 'Drop Columns Transformer': ['Drop Columns Transformer', 'Replace Nullable Types Transformer.x', 'Replace Nullable Types Transformer.y'], 'DateTime Featurizer': ['DateTime Featurizer', 'Drop Columns Transformer.x', 'Replace Nullable Types Transformer.y'], 'Imputer': ['Imputer', 'DateTime Featurizer.x', 'Replace Nullable Types Transformer.y'], 'One Hot Encoder': ['One Hot Encoder', 'Imputer.x', 'Replace Nullable Types Transformer.y'], 'Oversampler': ['Oversampler', 'One Hot Encoder.x', 'Replace Nullable Types Transformer.y'], 'Random Forest Classifier': ['Random Forest Classifier', 'Oversampler.x', 'Oversampler.y']}, parameters={'Label Encoder':{'positive_label': None}, 'Drop Columns Transformer':{'columns': ['currency']}, 'DateTime Featurizer':{'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'time_index': None}, 'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'boolean_impute_strategy': 'most_frequent', 'categorical_fill_value': None, 'numeric_fill_value': None, 'boolean_fill_value': None}, 'One Hot Encoder':{'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'Oversampler':{'sampling_ratio': 0.25, 'k_neighbors_default': 5, 'n_jobs': -1, 'sampling_ratio_dict': None, 'categorical_features': [3, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49], 'k_neighbors': 5}, 'Random Forest Classifier':{'n_estimators': 100, 'max_depth': 6, 'n_jobs': -1}}, random_seed=0)}

[17]:

pipeline_holdout_scores = automl.score_pipelines(
    [trained_pipelines[name] for name in trained_pipelines.keys()],
    X_holdout,
    y_holdout,
    ["Accuracy Binary", "F1", "AUC"],
)
pipeline_holdout_scores

[17]:

{'Mode Baseline Binary Classification Pipeline': OrderedDict([('Accuracy Binary',
               0.8615384615384616),
              ('F1', 0.0),
              ('AUC', 0.5)]),
 'Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler': OrderedDict([('Accuracy Binary',
               0.5230769230769231),
              ('F1', 0.3111111111111111),
              ('AUC', 0.7197420634920635)]),
 'Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler': OrderedDict([('Accuracy Binary',
               0.9769230769230769),
              ('F1', 0.9090909090909091),
              ('AUC', 0.9216269841269841)])}

Saving AutoMLSearch and pipelines from AutoMLSearch#

There are two ways to save results from AutoMLSearch.

You can save the AutoMLSearch object itself, calling .save(<filepath>) to do so. This will allow you to save the AutoMLSearch state and reload all pipelines from this.
If you want to save a pipeline from AutoMLSearch for future use, pipeline classes themselves have a .save(<filepath>) method.

[18]:

# saving the entire automl search
automl.save("automl.cloudpickle")
automl2 = evalml.automl.AutoMLSearch.load("automl.cloudpickle")
# saving the best pipeline using .save()
best_pipeline.save("pipeline.cloudpickle")
best_pipeline_copy = evalml.pipelines.PipelineBase.load("pipeline.cloudpickle")

Limiting the AutoML Search Space#

The AutoML search algorithm first trains each component in the pipeline with their default values. After the first iteration, it then tweaks the parameters of these components using the pre-defined hyperparameter ranges that these components have. To limit the search over certain hyperparameter ranges, you can specify a search_parameters argument with your AutoMLSearch parameters. These parameters will limit the hyperparameter search space or pipeline parameter space.

Hyperparameter ranges can be found through the API reference for each component. Parameter arguments must be specified as dictionaries, but the associated values must be skopt.space Real, Integer, Categorical objects for setting hyperparameter ranges.

If however you’d like to specify certain values for the initial batch of the AutoML search algorithm, you can use the search_parameters argument with non skopt.space objects. This will set the initial batch’s component parameters to the values passed by this argument.

[19]:

from evalml import AutoMLSearch
from evalml.demos import load_fraud
from skopt.space import Categorical
from evalml.model_family import ModelFamily
import woodwork as ww

X, y = load_fraud(n_rows=1000)

# example of setting parameter to just one value
search_parameters = {"Imputer": {"numeric_impute_strategy": "mean"}}


# limit the numeric impute strategy to include only `median` and `most_frequent`
# `mean` is the default value for this argument, but it doesn't need to be included in the specified hyperparameter range for this to work
search_parameters = {
    "Imputer": {"numeric_impute_strategy": Categorical(["median", "most_frequent"])}
}

# using this custom hyperparameter means that our Imputer components in these pipelines will only search through
# 'median' and 'most_frequent' strategies for 'numeric_impute_strategy'
automl_constrained = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    search_parameters=search_parameters,
    verbose=True,
)

             Number of Features
Boolean                       1
Categorical                   6
Numeric                       5

Number of training examples: 1000
Targets
False    85.90%
True     14.10%
Name: fraud, dtype: object
AutoMLSearch will use mean CV score to rank pipelines.
Using default limit of max_batches=3.

A skopt.space Integer, Real, or Categorical will set the hyperparameter space explored during search. All other values will set the pipeline parameters directly. Setting pipeline parameters directly defines the initialization parameters that a pipeline starts with during the first batch of AutoMLSearch. the hyperparameter range then defines the space of possible new parameter values, which the tuner chooses.

Let’s walk through some examples to explain this. For instance,

search_parameters = {'Imputer': {
    'numeric_impute_strategy': 'mean'
}}

then in the initial search, the algorithm would use mean as the impute strategy in batch 1. However, since Imputer.numeric_impute_strategy has a valid hyperparameter range, if the algorithm suggests a different strategy, it can and will change this value. To limit this to using mean only for the duration of the search, it is necessary to use the skopt.space:

search_parameters = {'Imputer': {
    'numeric_impute_strategy': Categorical(['mean'])
}}

However, if a value has no hyperparameter range associated, then the algorithm will use this value as the only parameter. For instance,

search_parameters = {'Label Encoder': {
    'positive_label': True
}}

Since Label Encoder.positive_label has no associated hyperparameter range, the algorithm will use this parameter for the entire duration of the search.

Imbalanced Data#

The AutoML search algorithm now has functionality to handle imbalanced data during classification! AutoMLSearch now provides two additional parameters, sampler_method and sampler_balanced_ratio, that allow you to let AutoMLSearch know whether to sample imbalanced data, and how to do so. sampler_method takes in either Undersampler, Oversampler, auto, or None as the sampler to use, and sampler_balanced_ratio specifies the minority/majority ratio that you want to sample to. Details on the Undersampler and Oversampler components can be found in the documentation.

This can be used for imbalanced datasets, like the fraud dataset, which has a ‘minority:majority’ ratio of < 0.2.

[20]:

automl_auto = AutoMLSearch(
    X_train=X, y_train=y, problem_type="binary", automl_algorithm="iterative"
)
automl_auto.allowed_pipelines[-1]

[20]:

pipeline = BinaryClassificationPipeline(component_graph={'Label Encoder': ['Label Encoder', 'X', 'y'], 'Replace Nullable Types Transformer': ['Replace Nullable Types Transformer', 'X', 'Label Encoder.y'], 'DateTime Featurizer': ['DateTime Featurizer', 'Replace Nullable Types Transformer.x', 'Replace Nullable Types Transformer.y'], 'Imputer': ['Imputer', 'DateTime Featurizer.x', 'Replace Nullable Types Transformer.y'], 'One Hot Encoder': ['One Hot Encoder', 'Imputer.x', 'Replace Nullable Types Transformer.y'], 'Oversampler': ['Oversampler', 'One Hot Encoder.x', 'Replace Nullable Types Transformer.y'], 'Extra Trees Classifier': ['Extra Trees Classifier', 'Oversampler.x', 'Oversampler.y']}, parameters={'Label Encoder':{'positive_label': None}, 'DateTime Featurizer':{'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'time_index': None}, 'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'boolean_impute_strategy': 'most_frequent', 'categorical_fill_value': None, 'numeric_fill_value': None, 'boolean_fill_value': None}, 'One Hot Encoder':{'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'Oversampler':{'sampling_ratio': 0.25, 'k_neighbors_default': 5, 'n_jobs': -1, 'sampling_ratio_dict': None}, 'Extra Trees Classifier':{'n_estimators': 100, 'max_features': 'auto', 'max_depth': 6, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_jobs': -1}}, random_seed=0)

The Oversampler is chosen as the default sampling component here, since the sampler_balanced_ratio = 0.25. If you specified a lower ratio, for instance sampler_balanced_ratio = 0.1, then there would be no sampling component added here. This is because if a ratio of 0.1 would be considered balanced, then a ratio of 0.2 would also be balanced.

The Oversampler uses SMOTE under the hood, and automatically selects whether to use SMOTE, SMOTEN, or SMOTENC based on the data it receives.

[21]:

automl_auto_ratio = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    sampler_balanced_ratio=0.1,
    automl_algorithm="iterative",
)
automl_auto_ratio.allowed_pipelines[-1]

[21]:

pipeline = BinaryClassificationPipeline(component_graph={'Label Encoder': ['Label Encoder', 'X', 'y'], 'Replace Nullable Types Transformer': ['Replace Nullable Types Transformer', 'X', 'Label Encoder.y'], 'DateTime Featurizer': ['DateTime Featurizer', 'Replace Nullable Types Transformer.x', 'Replace Nullable Types Transformer.y'], 'Imputer': ['Imputer', 'DateTime Featurizer.x', 'Replace Nullable Types Transformer.y'], 'One Hot Encoder': ['One Hot Encoder', 'Imputer.x', 'Replace Nullable Types Transformer.y'], 'Extra Trees Classifier': ['Extra Trees Classifier', 'One Hot Encoder.x', 'Replace Nullable Types Transformer.y']}, parameters={'Label Encoder':{'positive_label': None}, 'DateTime Featurizer':{'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'time_index': None}, 'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'boolean_impute_strategy': 'most_frequent', 'categorical_fill_value': None, 'numeric_fill_value': None, 'boolean_fill_value': None}, 'One Hot Encoder':{'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'Extra Trees Classifier':{'n_estimators': 100, 'max_features': 'auto', 'max_depth': 6, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_jobs': -1}}, random_seed=0)

Additionally, you can add more fine-grained sampling ratios by passing in a sampling_ratio_dict in pipeline parameters. For this dictionary, AutoMLSearch expects the keys to be int values from 0 to n-1 for the classes, and the values would be the sampler_balanced__ratio associated with each target. This dictionary would override the AutoML argument sampler_balanced_ratio. Below, you can see the scenario for Oversampler component on this dataset. Note that the logic for Undersamplers is included in the commented section.

[22]:

# In this case, the majority class is the negative class
# for the oversampler, we don't want to oversample this class, so class 0 (majority) will have a ratio of 1 to itself
# for the minority class 1, we want to oversample it to have a minority/majority ratio of 0.5, which means we want minority to have 1/2 the samples as the minority
sampler_ratio_dict = {0: 1, 1: 0.5}
search_parameters = {"Oversampler": {"sampler_balanced_ratio": sampler_ratio_dict}}
automl_auto_ratio_dict = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    search_parameters=search_parameters,
    automl_algorithm="iterative",
)
automl_auto_ratio_dict.allowed_pipelines[-1]

# Undersampler case
# we don't want to undersample this class, so class 1 (minority) will have a ratio of 1 to itself
# for the majority class 0, we want to undersample it to have a minority/majority ratio of 0.5, which means we want majority to have 2x the samples as the minority
# sampler_ratio_dict = {0: 0.5, 1: 1}
# search_parameters = {"Oversampler": {"sampler_balanced_ratio": sampler_ratio_dict}}
# automl_auto_ratio_dict = AutoMLSearch(X_train=X, y_train=y, problem_type='binary', search_parameters=search_parameters)

[22]:

pipeline = BinaryClassificationPipeline(component_graph={'Label Encoder': ['Label Encoder', 'X', 'y'], 'Replace Nullable Types Transformer': ['Replace Nullable Types Transformer', 'X', 'Label Encoder.y'], 'DateTime Featurizer': ['DateTime Featurizer', 'Replace Nullable Types Transformer.x', 'Replace Nullable Types Transformer.y'], 'Imputer': ['Imputer', 'DateTime Featurizer.x', 'Replace Nullable Types Transformer.y'], 'One Hot Encoder': ['One Hot Encoder', 'Imputer.x', 'Replace Nullable Types Transformer.y'], 'Oversampler': ['Oversampler', 'One Hot Encoder.x', 'Replace Nullable Types Transformer.y'], 'Extra Trees Classifier': ['Extra Trees Classifier', 'Oversampler.x', 'Oversampler.y']}, parameters={'Label Encoder':{'positive_label': None}, 'DateTime Featurizer':{'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'time_index': None}, 'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'boolean_impute_strategy': 'most_frequent', 'categorical_fill_value': None, 'numeric_fill_value': None, 'boolean_fill_value': None}, 'One Hot Encoder':{'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'Oversampler':{'sampling_ratio': 0.25, 'k_neighbors_default': 5, 'n_jobs': -1, 'sampling_ratio_dict': None, 'sampler_balanced_ratio': {0: 1, 1: 0.5}}, 'Extra Trees Classifier':{'n_estimators': 100, 'max_features': 'auto', 'max_depth': 6, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_jobs': -1}}, random_seed=0)

Adding ensemble methods to AutoML#

Stacking#

Stacking is an ensemble machine learning algorithm that involves training a model to best combine the predictions of several base learning algorithms. First, each base learning algorithms is trained using the given data. Then, the combining algorithm or meta-learner is trained on the predictions made by those base learning algorithms to make a final prediction.

AutoML enables stacking using the ensembling flag during initalization; this is set to False by default. How ensembling runs is defined by the AutoML algorithm you choose. In the IterativeAlgorithm, the stacking ensemble pipeline runs in its own batch after a whole cycle of training has occurred (each allowed pipeline trains for one batch). Note that this means a large number of iterations may need to run before the stacking ensemble runs. It is also important to note that only the first CV fold is calculated for stacking ensembles because the model internally uses CV folds. See below in the AutoML Algorithms section to see how ensembling is run for DefaultAlgorithm. Please do note that ensembling is currently unavailable for time series problems.

[23]:

X, y = evalml.demos.load_breast_cancer()

automl_with_ensembling = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    allowed_model_families=[ModelFamily.LINEAR_MODEL],
    max_batches=4,
    ensembling=True,
    automl_algorithm="iterative",
    verbose=True,
)
automl_with_ensembling.search(interactive_plot=False)

         Number of Features
Numeric                  30

Number of training examples: 569
Targets
benign       62.74%
malignant    37.26%
Name: target, dtype: object
AutoMLSearch will use mean CV score to rank pipelines.
Generating pipelines to search over...
Ensembling will run every 3 batches.
2 pipelines ready for search.

*****************************
* Beginning pipeline search *
*****************************

Optimizing for Log Loss Binary.
Lower score is better.

Using SequentialEngine to train and score pipelines.
Searching up to 4 batches for a total of 14 pipelines.
Allowed model families: linear_model, linear_model

Evaluating Baseline Pipeline: Mode Baseline Binary Classification Pipeline
Mode Baseline Binary Classification Pipeline:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 12.868

*****************************
* Evaluating Batch Number 1 *
*****************************

Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.077
Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.077

*****************************
* Evaluating Batch Number 2 *
*****************************

Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.090
Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.085
Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.081
Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.097
Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.093

*****************************
* Evaluating Batch Number 3 *
*****************************

Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.075
Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.076
Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.075
Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.079
Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.075

*****************************
* Evaluating Batch Number 4 *
*****************************

Stacked Ensemble Classification Pipeline:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.103

Search finished after 00:21
Best pipeline: Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler
Best pipeline Log Loss Binary: 0.075391

[23]:

{1: {'Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler': '00:01',
  'Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler': '00:01',
  'Total time of batch': '00:03'},
 2: {'Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler': '00:01',
  'Total time of batch': '00:08'},
 3: {'Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler': '00:01',
  'Total time of batch': '00:08'},
 4: {'Stacked Ensemble Classification Pipeline': '00:01',
  'Total time of batch': '00:01'}}

We can view more information about the stacking ensemble pipeline (which was the best performing pipeline) by calling .describe().

[24]:

automl_with_ensembling.best_pipeline.describe()


************************************************************************************************************
* Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Imputer + Standard Scaler *
************************************************************************************************************

Problem Type: binary
Model Family: Linear
Number of features: 30

Pipeline Steps
==============
1. Label Encoder
         * positive_label : None
2. Replace Nullable Types Transformer
3. Imputer
         * categorical_impute_strategy : most_frequent
         * numeric_impute_strategy : knn
         * boolean_impute_strategy : knn
         * categorical_fill_value : None
         * numeric_fill_value : None
         * boolean_fill_value : None
4. Standard Scaler
5. Elastic Net Classifier
         * penalty : elasticnet
         * C : 8.474044870453413
         * l1_ratio : 0.6235636967859725
         * n_jobs : -1
         * multi_class : auto
         * solver : saga

AutoML Algorithms#

EvalML currently has two algorithms available for users to choose from. Below, we will run through how each algorithm works and how to access them through AutoMLSearch as well as the top level search methods.

IterativeAlgorithm #

IterativeAlgorithm is the first AutoML Algorithm created in EvalML and can be acessed with the search_iterative method or specifiying AutoMLSearch(automl_algorithm='iterative'). The algorithm works as follows:

Every batch (after the initial baseline model) contains pipelines of all available estimators for the specified problem type
Pipelines contain preprocessing (imputing, encoding, etc.) needed for machine learning but no feature selection is applied
Ensembling can be turned on by passing in the ensembling=True parameter and will be run after a whole cycle of training has occurred (each allowed pipeline trains for one batch)

[25]:

import evalml

X, y = evalml.demos.load_fraud(n_rows=250)

             Number of Features
Boolean                       1
Categorical                   6
Numeric                       5

Number of training examples: 250
Targets
False    88.40%
True     11.60%
Name: fraud, dtype: object

[26]:

from evalml.automl import search_iterative

# top level search method will run `AutoMLSearch` with `IterativeAlgorithm` as well as apply our default data checks
auto_iterative, messages_iterative = search_iterative(X, y, problem_type="binary")

[27]:

from evalml import AutoMLSearch

auto_iterative = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    automl_algorithm="iterative",
    verbose=True,
)
auto_iterative.search(interactive_plot=False)

AutoMLSearch will use mean CV score to rank pipelines.
Removing columns ['currency', 'expiration_date'] because they are of 'Unknown' type
Generating pipelines to search over...
8 pipelines ready for search.
Using default limit of max_batches=1.


*****************************
* Beginning pipeline search *
*****************************

Optimizing for Log Loss Binary.
Lower score is better.

Using SequentialEngine to train and score pipelines.
Searching up to 1 batches for a total of None pipelines.
Allowed model families: linear_model, linear_model, xgboost, lightgbm, catboost, random_forest, decision_tree, extra_trees

Evaluating Baseline Pipeline: Mode Baseline Binary Classification Pipeline
Mode Baseline Binary Classification Pipeline:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 4.006

*****************************
* Evaluating Batch Number 1 *
*****************************

Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.430
Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.430
XGBoost Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.266
LightGBM Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.341
CatBoost Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.598
Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.286
Decision Tree Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 5.496
        High coefficient of variation (cv >= 0.5) within cross validation scores.
        Decision Tree Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler may not perform as estimated on unseen data.
Extra Trees Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.344

Search finished after 00:18
Best pipeline: XGBoost Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler
Best pipeline Log Loss Binary: 0.265852

[27]:

{1: {'Elastic Net Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler': '00:02',
  'Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler': '00:02',
  'XGBoost Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler': '00:02',
  'LightGBM Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler': '00:02',
  'CatBoost Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Oversampler': '00:01',
  'Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler': '00:02',
  'Decision Tree Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler': '00:01',
  'Extra Trees Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler': '00:02',
  'Total time of batch': '00:18'}}

DefaultAlgorithm #

DefaultAlgorithm was designed to do three main things:

Abstract out more parameters and decisions from the user.
Perform deeper tuning for high performing pipelines.
Create a platform to introduce feature selection as well as other potential techniques/heuristics for AutoML.

DefaultAlgorithm does this by creating the concept of two modes: fast and long, where fast is a subset of long. The algorithm runs as follows:

Run naive pipelines:

a linear model with the default preprocessing pipeline
a random forest pipeline with the default preprocessing pipeline

Run the same pipelines, this time with feature selection. Subsequent pipelines will use the selected features with a SelectedColumns transformer.
Run all pipelines with preprocessing components:
1. scan rest of estimators (IterativeAlgorithm batch 1).
First ensembling run

Fast mode ends here. Begin long mode.

Run top 3 estimators:
1. Generate 50 random parameter sets. Run all 150 in one batch
Second ensembling run
Repeat 8a and 8b indefinitely until the specified time in AutoMLSearch is met:
1. For each of the previous top 3 estimators, sample 10 parameters from the tuner. Run all 30 in one batch
2. Run ensembling

To this end, it is recommended to use the top level search() method to run DefaultAlgorithm. This allows users to specify running search with just the mode parameter, where fast is recommended for users who want a fast scan at how EvalML pipelines will perform on their problem and where long is reserved for a deeper dive into high performing pipelines. If one needs finer control over AutoML parameters, one can also specify automl_algorithm='default' using AutoMLSearch and it will default to using fast mode. However, in this case ensembling will be defined by the ensembling flag (if ensembling=False the abovementioned ensembling batches will be skipped). Users are welcome to select max_batches according to the algorithm above (or other stopping criteria) but should be aware that results may not be optimal if the algorithm does not run for the full length of fast mode.

[28]:

from evalml.automl import search

# top level search method will run `AutoMLSearch` with `IterativeAlgorithm` as well as apply our default data checks
auto_default, messages_default = search(X, y, problem_type="binary", mode="fast")

        High coefficient of variation (cv >= 0.5) within cross validation scores.
        Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler may not perform as estimated on unseen data.

[29]:

from evalml import AutoMLSearch

auto_default = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    automl_algorithm="default",
    ensembling=True,
    verbose=True,
)
auto_default.search(interactive_plot=False)

AutoMLSearch will use mean CV score to rank pipelines.
Removing columns ['currency', 'expiration_date'] because they are of 'Unknown' type
Using default limit of max_batches=4.


*****************************
* Beginning pipeline search *
*****************************

Optimizing for Log Loss Binary.
Lower score is better.

Using SequentialEngine to train and score pipelines.
Searching up to 4 batches for a total of None pipelines.
Allowed model families:

Evaluating Baseline Pipeline: Mode Baseline Binary Classification Pipeline
Mode Baseline Binary Classification Pipeline:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 4.006

*****************************
* Evaluating Batch Number 1 *
*****************************

Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.430
Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.286

*****************************
* Evaluating Batch Number 2 *
*****************************

Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler + RF Classifier Select From Model:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.413
Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + RF Classifier Select From Model:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.276

*****************************
* Evaluating Batch Number 3 *
*****************************

Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 3.234
        High coefficient of variation (cv >= 0.5) within cross validation scores.
        Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler may not perform as estimated on unseen data.
LightGBM Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.341
Extra Trees Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.350
Elastic Net Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Standard Scaler + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Standard Scaler + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.422
CatBoost Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.598
XGBoost Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.266

*****************************
* Evaluating Batch Number 4 *
*****************************

Stacked Ensemble Classification Pipeline:
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.236

Search finished after 00:37
Best pipeline: Stacked Ensemble Classification Pipeline
Best pipeline Log Loss Binary: 0.236412

[29]:

{1: {'Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler': '00:02',
  'Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler': '00:02',
  'Total time of batch': '00:05'},
 2: {'Logistic Regression Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + Standard Scaler + RF Classifier Select From Model': '00:02',
  'Random Forest Classifier w/ Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + One Hot Encoder + Oversampler + RF Classifier Select From Model': '00:02',
  'Total time of batch': '00:05'},
 3: {'Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler': '00:01',
  'LightGBM Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler': '00:02',
  'Extra Trees Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler': '00:03',
  'Elastic Net Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Standard Scaler + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Standard Scaler + Oversampler': '00:02',
  'CatBoost Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + Oversampler': '00:01',
  'XGBoost Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler': '00:02',
  'Total time of batch': '00:14'},
 4: {'Stacked Ensemble Classification Pipeline': '00:11',
  'Total time of batch': '00:11'}}

Pipeline differences#

Through the search output above, we can see how pipelines differ between IterativeAlgorithm and DefaultAlgorithm. This is because DefaultAlgorithm utilizes new components such as RFRegressorSelectFromModel and other column selectors for feature selection as well as a new pipeline structure to handle feature selection for categorical and non-categorical features.

[30]:

auto_iterative.get_pipeline(4).graph()

[30]:

[31]:

auto_default.get_pipeline(6).graph()

[31]:

Access raw results#

The AutoMLSearch class records detailed results information under the results field, including information about the cross-validation scoring and parameters.

[32]:

import pprint

pp = pprint.PrettyPrinter(indent=0, width=100, depth=3, compact=True, sort_dicts=False)

pp.pprint(automl.results)

{'pipeline_results': {0: {'id': 0,
                       'pipeline_name': 'Mode Baseline Binary Classification Pipeline',
                       'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                       'pipeline_summary': 'Baseline Classifier w/ Label Encoder',
                       'parameters': {...},
                       'mean_cv_score': 4.715778718459999,
                       'standard_deviation_cv_score': 0.10819602943036762,
                       'high_variance_cv': False,
                       'training_time': 0.4204421043395996,
                       'cv_data': [...],
                       'percent_better_than_baseline_all_objectives': {...},
                       'percent_better_than_baseline': 0,
                       'validation_score': 4.782292116218402},
                    1: {'id': 1,
                       'pipeline_name': 'Logistic Regression Classifier w/ Label Encoder + Replace '
                                        'Nullable Types Transformer + Drop Columns Transformer + '
                                        'DateTime Featurizer + Imputer + One Hot Encoder + '
                                        'Oversampler + Standard Scaler',
                       'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                       'pipeline_summary': 'Logistic Regression Classifier w/ Label Encoder + '
                                           'Replace Nullable Types Transformer + Drop Columns '
                                           'Transformer + DateTime Featurizer + Imputer + One Hot '
                                           'Encoder + Oversampler + Standard Scaler',
                       'parameters': {...},
                       'mean_cv_score': 0.37833532632127936,
                       'standard_deviation_cv_score': 0.04660154317933726,
                       'high_variance_cv': False,
                       'training_time': 5.52813982963562,
                       'cv_data': [...],
                       'percent_better_than_baseline_all_objectives': {...},
                       'percent_better_than_baseline': 91.97724598823352,
                       'validation_score': 0.3920002496857071},
                    2: {'id': 2,
                       'pipeline_name': 'Random Forest Classifier w/ Label Encoder + Replace '
                                        'Nullable Types Transformer + Drop Columns Transformer + '
                                        'DateTime Featurizer + Imputer + One Hot Encoder + '
                                        'Oversampler',
                       'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                       'pipeline_summary': 'Random Forest Classifier w/ Label Encoder + Replace '
                                           'Nullable Types Transformer + Drop Columns Transformer '
                                           '+ DateTime Featurizer + Imputer + One Hot Encoder + '
                                           'Oversampler',
                       'parameters': {...},
                       'mean_cv_score': 0.25917555496638456,
                       'standard_deviation_cv_score': 0.03936199648193578,
                       'high_variance_cv': False,
                       'training_time': 3.9852564334869385,
                       'cv_data': [...],
                       'percent_better_than_baseline_all_objectives': {...},
                       'percent_better_than_baseline': 94.50407725977816,
                       'validation_score': 0.2190814791590008},
                    3: {'id': 3,
                       'pipeline_name': 'Logistic Regression Classifier w/ Label Encoder + Replace '
                                        'Nullable Types Transformer + Drop Columns Transformer + '
                                        'DateTime Featurizer + Imputer + One Hot Encoder + '
                                        'Oversampler + Standard Scaler + RF Classifier Select From '
                                        'Model',
                       'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                       'pipeline_summary': 'Logistic Regression Classifier w/ Label Encoder + '
                                           'Replace Nullable Types Transformer + Drop Columns '
                                           'Transformer + DateTime Featurizer + Imputer + One Hot '
                                           'Encoder + Oversampler + Standard Scaler + RF '
                                           'Classifier Select From Model',
                       'parameters': {...},
                       'mean_cv_score': 0.3762476086922398,
                       'standard_deviation_cv_score': 0.04407175843325967,
                       'high_variance_cv': False,
                       'training_time': 4.361112594604492,
                       'cv_data': [...],
                       'percent_better_than_baseline_all_objectives': {...},
                       'percent_better_than_baseline': 92.02151688713866,
                       'validation_score': 0.3400199995080042},
                    4: {'id': 4,
                       'pipeline_name': 'Random Forest Classifier w/ Label Encoder + Replace '
                                        'Nullable Types Transformer + Drop Columns Transformer + '
                                        'DateTime Featurizer + Imputer + One Hot Encoder + '
                                        'Oversampler + RF Classifier Select From Model',
                       'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                       'pipeline_summary': 'Random Forest Classifier w/ Label Encoder + Replace '
                                           'Nullable Types Transformer + Drop Columns Transformer '
                                           '+ DateTime Featurizer + Imputer + One Hot Encoder + '
                                           'Oversampler + RF Classifier Select From Model',
                       'parameters': {...},
                       'mean_cv_score': 0.2502019799916092,
                       'standard_deviation_cv_score': 0.0473405185959973,
                       'high_variance_cv': False,
                       'training_time': 4.316831588745117,
                       'cv_data': [...],
                       'percent_better_than_baseline_all_objectives': {...},
                       'percent_better_than_baseline': 94.69436555595391,
                       'validation_score': 0.20316402685243987},
                    5: {'id': 5,
                       'pipeline_name': 'Decision Tree Classifier w/ Label Encoder + Select '
                                        'Columns By Type Transformer + Label Encoder + Replace '
                                        'Nullable Types Transformer + Drop Columns Transformer + '
                                        'DateTime Featurizer + Imputer + Select Columns '
                                        'Transformer + Select Columns Transformer + Label Encoder '
                                        '+ Replace Nullable Types Transformer + Imputer + One Hot '
                                        'Encoder + Oversampler',
                       'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                       'pipeline_summary': 'Decision Tree Classifier w/ Label Encoder + Select '
                                           'Columns By Type Transformer + Label Encoder + Replace '
                                           'Nullable Types Transformer + Drop Columns Transformer '
                                           '+ DateTime Featurizer + Imputer + Select Columns '
                                           'Transformer + Select Columns Transformer + Label '
                                           'Encoder + Replace Nullable Types Transformer + Imputer '
                                           '+ One Hot Encoder + Oversampler',
                       'parameters': {...},
                       'mean_cv_score': 1.4030039555953897,
                       'standard_deviation_cv_score': 0.666162025148404,
                       'high_variance_cv': True,
                       'training_time': 3.415464162826538,
                       'cv_data': [...],
                       'percent_better_than_baseline_all_objectives': {...},
                       'percent_better_than_baseline': 70.24873219552678,
                       'validation_score': 1.4743694397959255},
                    6: {'id': 6,
                       'pipeline_name': 'LightGBM Classifier w/ Label Encoder + Select Columns By '
                                        'Type Transformer + Label Encoder + Replace Nullable Types '
                                        'Transformer + Drop Columns Transformer + DateTime '
                                        'Featurizer + Imputer + Select Columns Transformer + '
                                        'Select Columns Transformer + Label Encoder + Replace '
                                        'Nullable Types Transformer + Imputer + One Hot Encoder + '
                                        'Oversampler',
                       'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                       'pipeline_summary': 'LightGBM Classifier w/ Label Encoder + Select Columns '
                                           'By Type Transformer + Label Encoder + Replace Nullable '
                                           'Types Transformer + Drop Columns Transformer + '
                                           'DateTime Featurizer + Imputer + Select Columns '
                                           'Transformer + Select Columns Transformer + Label '
                                           'Encoder + Replace Nullable Types Transformer + Imputer '
                                           '+ One Hot Encoder + Oversampler',
                       'parameters': {...},
                       'mean_cv_score': 0.3189720160748657,
                       'standard_deviation_cv_score': 0.22922344665852376,
                       'high_variance_cv': False,
                       'training_time': 3.357884645462036,
                       'cv_data': [...],
                       'percent_better_than_baseline_all_objectives': {...},
                       'percent_better_than_baseline': 93.23606905416821,
                       'validation_score': 0.18340197476019973},
                    7: {'id': 7,
                       'pipeline_name': 'Extra Trees Classifier w/ Label Encoder + Select Columns '
                                        'By Type Transformer + Label Encoder + Replace Nullable '
                                        'Types Transformer + Drop Columns Transformer + DateTime '
                                        'Featurizer + Imputer + Select Columns Transformer + '
                                        'Select Columns Transformer + Label Encoder + Replace '
                                        'Nullable Types Transformer + Imputer + One Hot Encoder + '
                                        'Oversampler',
                       'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                       'pipeline_summary': 'Extra Trees Classifier w/ Label Encoder + Select '
                                           'Columns By Type Transformer + Label Encoder + Replace '
                                           'Nullable Types Transformer + Drop Columns Transformer '
                                           '+ DateTime Featurizer + Imputer + Select Columns '
                                           'Transformer + Select Columns Transformer + Label '
                                           'Encoder + Replace Nullable Types Transformer + Imputer '
                                           '+ One Hot Encoder + Oversampler',
                       'parameters': {...},
                       'mean_cv_score': 0.3617597485280606,
                       'standard_deviation_cv_score': 0.021901581014440938,
                       'high_variance_cv': False,
                       'training_time': 4.591495513916016,
                       'cv_data': [...],
                       'percent_better_than_baseline_all_objectives': {...},
                       'percent_better_than_baseline': 92.32873783682967,
                       'validation_score': 0.35244184096534764},
                    8: {'id': 8,
                       'pipeline_name': 'Elastic Net Classifier w/ Label Encoder + Select Columns '
                                        'By Type Transformer + Label Encoder + Replace Nullable '
                                        'Types Transformer + Drop Columns Transformer + DateTime '
                                        'Featurizer + Imputer + Standard Scaler + Select Columns '
                                        'Transformer + Select Columns Transformer + Label Encoder '
                                        '+ Replace Nullable Types Transformer + Imputer + One Hot '
                                        'Encoder + Standard Scaler + Oversampler',
                       'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                       'pipeline_summary': 'Elastic Net Classifier w/ Label Encoder + Select '
                                           'Columns By Type Transformer + Label Encoder + Replace '
                                           'Nullable Types Transformer + Drop Columns Transformer '
                                           '+ DateTime Featurizer + Imputer + Standard Scaler + '
                                           'Select Columns Transformer + Select Columns '
                                           'Transformer + Label Encoder + Replace Nullable Types '
                                           'Transformer + Imputer + One Hot Encoder + Standard '
                                           'Scaler + Oversampler',
                       'parameters': {...},
                       'mean_cv_score': 0.37472485974788244,
                       'standard_deviation_cv_score': 0.050026569255638,
                       'high_variance_cv': False,
                       'training_time': 4.103651523590088,
                       'cv_data': [...],
                       'percent_better_than_baseline_all_objectives': {...},
                       'percent_better_than_baseline': 92.0538073959871,
                       'validation_score': 0.400375420656706},
                    9: {'id': 9,
                       'pipeline_name': 'CatBoost Classifier w/ Label Encoder + Select Columns By '
                                        'Type Transformer + Label Encoder + Replace Nullable Types '
                                        'Transformer + Drop Columns Transformer + DateTime '
                                        'Featurizer + Imputer + Select Columns Transformer + '
                                        'Select Columns Transformer + Label Encoder + Replace '
                                        'Nullable Types Transformer + Imputer + Oversampler',
                       'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                       'pipeline_summary': 'CatBoost Classifier w/ Label Encoder + Select Columns '
                                           'By Type Transformer + Label Encoder + Replace Nullable '
                                           'Types Transformer + Drop Columns Transformer + '
                                           'DateTime Featurizer + Imputer + Select Columns '
                                           'Transformer + Select Columns Transformer + Label '
                                           'Encoder + Replace Nullable Types Transformer + Imputer '
                                           '+ Oversampler',
                       'parameters': {...},
                       'mean_cv_score': 0.5749661777007408,
                       'standard_deviation_cv_score': 0.01686036168146034,
                       'high_variance_cv': False,
                       'training_time': 2.3499279022216797,
                       'cv_data': [...],
                       'percent_better_than_baseline_all_objectives': {...},
                       'percent_better_than_baseline': 87.80760904980495,
                       'validation_score': 0.551641820745934},
                    10: {'id': 10,
                        'pipeline_name': 'XGBoost Classifier w/ Label Encoder + Select Columns By '
                                         'Type Transformer + Label Encoder + Replace Nullable '
                                         'Types Transformer + Drop Columns Transformer + DateTime '
                                         'Featurizer + Imputer + Select Columns Transformer + '
                                         'Select Columns Transformer + Label Encoder + Replace '
                                         'Nullable Types Transformer + Imputer + One Hot Encoder + '
                                         'Oversampler',
                        'pipeline_class': <class 'evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline'>,
                        'pipeline_summary': 'XGBoost Classifier w/ Label Encoder + Select Columns '
                                            'By Type Transformer + Label Encoder + Replace '
                                            'Nullable Types Transformer + Drop Columns Transformer '
                                            '+ DateTime Featurizer + Imputer + Select Columns '
                                            'Transformer + Select Columns Transformer + Label '
                                            'Encoder + Replace Nullable Types Transformer + '
                                            'Imputer + One Hot Encoder + Oversampler',
                        'parameters': {...},
                        'mean_cv_score': 0.2534258549873319,
                        'standard_deviation_cv_score': 0.14307477527787266,
                        'high_variance_cv': False,
                        'training_time': 4.0017547607421875,
                        'cv_data': [...],
                        'percent_better_than_baseline_all_objectives': {...},
                        'percent_better_than_baseline': 94.62600197936999,
                        'validation_score': 0.15053433348100786}},
'search_order': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}

If there are errors, such as with the Iterative Algorithm example above, we can examine these closer by accessing the errors field. There is one dictionary entry per pipeline fold that failed, and each entry contains the pipeline parameters with the error that was thrown and its full traceback.

[33]:

auto_iterative.errors

[33]:

{}

Parallel AutoML#

By default, all pipelines in an AutoML batch are evaluated in series. Pipelines can be evaluated in parallel to improve performance during AutoML search. This is accomplished by a futures style submission and evaluation of pipelines in a batch. As of this writing, the pipelines use a threaded model for concurrent evaluation. This is similar to the currently implemented n_jobs parameter in the estimators, which uses increased numbers of threads to train and evaluate estimators.

Quick Start#

To quickly use some parallelism to enhance the pipeline searching, a string can be passed through to AutoMLSearch during initialization to setup the parallel engine and client within the AutoMLSearch object. The current options are “cf_threaded”, “cf_process”, “dask_threaded” and “dask_process” and indicate the futures backend to use and whether to use threaded- or process-level parallelism.

[34]:

automl_cf_threaded = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    allowed_model_families=[ModelFamily.LINEAR_MODEL],
    engine="cf_threaded",
)
automl_cf_threaded.search(interactive_plot=False)
automl_cf_threaded.close_engine()

        High coefficient of variation (cv >= 0.5) within cross validation scores.
        Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler may not perform as estimated on unseen data.

Parallelism with Concurrent Futures#

The EngineBase class is robust and extensible enough to support futures-like implementations from a variety of libraries. The CFEngine extends the EngineBase to use the native Python concurrent.futures library. The CFEngine supports both thread- and process-level parallelism. The type of parallelism can be chosen using either the ThreadPoolExecutor or the ProcessPoolExecutor. If either executor is passed a max_workers parameter, it will set the number of processes and threads spawned. If not, the default number of processes will be equal to the number of processors available and the number of threads set to five times the number of processors available.

Here, the CFEngine is invoked with default parameters, which is threaded parallelism using all available threads.

[35]:

from concurrent.futures import ThreadPoolExecutor

from evalml.automl.engine.cf_engine import CFEngine, CFClient

cf_engine = CFEngine(CFClient(ThreadPoolExecutor(max_workers=4)))
automl_cf_threaded = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    allowed_model_families=[ModelFamily.LINEAR_MODEL],
    engine=cf_engine,
)
automl_cf_threaded.search(interactive_plot=False)
automl_cf_threaded.close_engine()

        High coefficient of variation (cv >= 0.5) within cross validation scores.
        Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler may not perform as estimated on unseen data.

Note: the cell demonstrating process-level parallelism is a markdown due to incompatibility with our ReadTheDocs build. It can be run successfully locally.

from concurrent.futures import ProcessPoolExecutor

# Repeat the process but using process-level parallelism\
cf_engine = CFEngine(CFClient(ProcessPoolExecutor(max_workers=2)))
automl_cf_process = AutoMLSearch(X_train=X, y_train=y,
                                 problem_type="binary",
                                 engine="cf_process")
automl_cf_process.search(interactive_plot = False)
automl_cf_process.close_engine()

Parallelism with Dask#

Thread or process level parallelism can be explicitly invoked for the DaskEngine (as well as the CFEngine). The processes can be set to True and the number of processes set using n_workers. If processes is set to False, then the resulting parallelism will be threaded and n_workers will represent the threads used. Examples of both follow.

[36]:

from dask.distributed import LocalCluster

from evalml.automl.engine import DaskEngine

dask_engine_p2 = DaskEngine(cluster=LocalCluster(processes=True, n_workers=2))
automl_dask_p2 = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    allowed_model_families=[ModelFamily.LINEAR_MODEL],
    engine=dask_engine_p2,
)
automl_dask_p2.search(interactive_plot=False)

# Explicitly shutdown the automl object's LocalCluster
automl_dask_p2.close_engine()

        High coefficient of variation (cv >= 0.5) within cross validation scores.
        Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler may not perform as estimated on unseen data.
pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.
Inside a Dask worker with daemon=True, setting n_jobs=1.
Possible work-arounds:
- dask.config.set({'distributed.worker.daemon': False})
- set the environment variable DASK_DISTRIBUTED__WORKER__DAEMON=False
before creating your Dask cluster.

deleting tmpfiles dir: /tmp/tmpe916rfyw
done
deleting tmpfiles dir: /tmp/tmpgumwj53d
done


Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/client.py", line 1285, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/client.py", line 1315, in _ensure_connected
    comm = await connect(
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/comm/core.py", line 315, in connect
    await asyncio.sleep(backoff)
  File "/home/docs/.pyenv/versions/3.8.6/lib/python3.8/asyncio/tasks.py", line 656, in sleep
    return await future
asyncio.exceptions.CancelledError

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/client.py", line 1490, in _handle_report
    await self._reconnect()
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/client.py", line 1285, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/client.py", line 1315, in _ensure_connected
    comm = await connect(
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/comm/core.py", line 315, in connect
    await asyncio.sleep(backoff)
  File "/home/docs/.pyenv/versions/3.8.6/lib/python3.8/asyncio/tasks.py", line 656, in sleep
    return await future
asyncio.exceptions.CancelledError

[37]:

dask_engine_t4 = DaskEngine(cluster=LocalCluster(processes=False, n_workers=4))

automl_dask_t4 = AutoMLSearch(
    X_train=X,
    y_train=y,
    problem_type="binary",
    allowed_model_families=[ModelFamily.LINEAR_MODEL],
    engine=dask_engine_t4,
)
automl_dask_t4.search(interactive_plot=False)
automl_dask_t4.close_engine()

        High coefficient of variation (cv >= 0.5) within cross validation scores.
        Decision Tree Classifier w/ Label Encoder + Select Columns By Type Transformer + Label Encoder + Replace Nullable Types Transformer + Drop Columns Transformer + DateTime Featurizer + Imputer + Select Columns Transformer + Select Columns Transformer + Label Encoder + Replace Nullable Types Transformer + Imputer + One Hot Encoder + Oversampler may not perform as estimated on unseen data.
ERROR:asyncio.events:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/client.py", line 1285, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/client.py", line 1315, in _ensure_connected
    comm = await connect(
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/comm/core.py", line 315, in connect
    await asyncio.sleep(backoff)
  File "/home/docs/.pyenv/versions/3.8.6/lib/python3.8/asyncio/tasks.py", line 656, in sleep
    return await future
asyncio.exceptions.CancelledError
ERROR:asyncio.events:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/client.py", line 1490, in _handle_report
    await self._reconnect()
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/client.py", line 1285, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/client.py", line 1315, in _ensure_connected
    comm = await connect(
  File "/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.62.0/lib/python3.8/site-packages/distributed/comm/core.py", line 315, in connect
    await asyncio.sleep(backoff)
  File "/home/docs/.pyenv/versions/3.8.6/lib/python3.8/asyncio/tasks.py", line 656, in sleep
    return await future
asyncio.exceptions.CancelledError

As we can see, a significant performance gain can result from simply using something other than the default SequentialEngine, ranging from a 100% speed up with multiple processes to 500% speedup with multiple threads!

[38]:

print("Sequential search duration: %s" % str(automl.search_duration))
print(
    "Concurrent futures (threaded) search duration: %s"
    % str(automl_cf_threaded.search_duration)
)
print("Dask (two processes) search duration: %s" % str(automl_dask_p2.search_duration))
print("Dask (four threads)search duration: %s" % str(automl_dask_t4.search_duration))

Sequential search duration: 42.064783573150635
Concurrent futures (threaded) search duration: 32.92518615722656
Dask (two processes) search duration: 33.572598457336426
Dask (four threads)search duration: 37.464807987213135