Automated Machine Learning (AutoML) Search¶
Background¶
Machine Learning¶
Machine learning (ML) is the process of constructing a mathematical model of a system based on a sample dataset collected from that system.
One of the main goals of training an ML model is to teach the model to separate the signal present in the data from the noise inherent in system and in the data collection process. If this is done effectively, the model can then be used to make accurate predictions about the system when presented with new, similar data. Additionally, introspecting on an ML model can reveal key information about the system being modeled, such as which inputs and transformations of the inputs are most useful to the ML model for learning the signal in the data, and are therefore the most predictive.
There are a variety of ML problem types. Supervised learning describes the case where the collected data contains an output value to be modeled and a set of inputs with which to train the model. EvalML focuses on training supervised learning models.
EvalML supports three common supervised ML problem types. The first is regression, where the target value to model is a continuous numeric value. Next are binary and multiclass classification, where the target value to model consists of two or more discrete values or categories. The choice of which supervised ML problem type is most appropriate depends on domain expertise and on how the model will be evaluated and used.
EvalML is currently building support for supervised time series problems: time series regression, time series binary classification, and time series multiclass classification. While we’ve added some features to tackle these kinds of problems, our functionality is still being actively developed so please be mindful of that before using it.
AutoML and Search¶
AutoML is the process of automating the construction, training and evaluation of ML models. Given a data and some configuration, AutoML searches for the most effective and accurate ML model or models to fit the dataset. During the search, AutoML will explore different combinations of model type, model parameters and model architecture.
An effective AutoML solution offers several advantages over constructing and tuning ML models by hand. AutoML can assist with many of the difficult aspects of ML, such as avoiding overfitting and underfitting, imbalanced data, detecting data leakage and other potential issues with the problem setup, and automatically applying best-practice data cleaning, feature engineering, feature selection and various modeling techniques. AutoML can also leverage search algorithms to optimally sweep the hyperparameter search space, resulting in model performance which would be difficult to achieve by manual training.
AutoML in EvalML¶
EvalML supports all of the above and more.
In its simplest usage, the AutoML search interface requires only the input data, the target data and a problem_type specifying what kind of supervised ML problem to model.
** Graphing methods, like AutoMLSearch, on Jupyter Notebook and Jupyter Lab require ipywidgets to be installed.
** If graphing on Jupyter Lab, jupyterlab-plotly required. To download this, make sure you have npm installed.
[1]:
import evalml
from evalml.utils import infer_feature_types
X, y = evalml.demos.load_fraud(n_rows=1000, return_pandas=True)
Number of Features
Boolean 1
Categorical 6
Numeric 5
Number of training examples: 1000
Targets
False 85.90%
True 14.10%
Name: fraud, dtype: object
To provide data to EvalML, it is recommended that you create a DataTable object using the Woodwork project. This allows you to easily control how EvalML will treat each of your features before training a model.
EvalML also accepts pandas input, and will run type inference on top of the input pandas data. If you’d like to change the types inferred by EvalML, you can use the infer_feature_types utility method, which takes pandas or numpy input and converts it to a Woodwork data structure. The feature_types parameter can be used to specify what types specific columns should be.
In the example below, we reformat a couple features to make them easily consumable by the model, and then specify that the provider, which would have otherwise been inferred as a column with natural language, is a categorical column.
[2]:
X['expiration_date'] = X['expiration_date'].apply(lambda x: '20{}-01-{}'.format(x.split("/")[1], x.split("/")[0]))
X[['lat', 'lng']] = X[['lat', 'lng']].astype('str')
X = infer_feature_types(X, feature_types= {'store_id': 'categorical',
'expiration_date': 'datetime',
'lat': 'categorical',
'lng': 'categorical',
'provider': 'categorical'})
In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set.
[3]:
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, problem_type='binary', test_size=.2)
Data Checks¶
Before calling AutoMLSearch.search, we should run some sanity checks on our data to ensure that the input data being passed will not run into some common issues before running a potentially time-consuming search. EvalML has various data checks that makes this easy. Each data check will return a collection of warnings and errors if it detects potential issues with the input data. This allows users to inspect their data to avoid confusing errors that may arise during the search process. You
can learn about each of the data checks available through our data checks guide
Here, we will run the DefaultDataChecks class, which contains a series of data checks that are generally useful.
[4]:
from evalml.data_checks import DefaultDataChecks
data_checks = DefaultDataChecks("binary", "log loss binary")
data_checks.validate(X_train, y_train)
[4]:
{'warnings': [], 'errors': [], 'actions': []}
Since there were no warnings or errors returned, we can safely continue with the search process.
[5]:
automl = evalml.automl.AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary')
automl.search()
Using default limit of max_batches=1.
Generating pipelines to search over...
8 pipelines ready for search.
*****************************
* Beginning pipeline search *
*****************************
Optimizing for Log Loss Binary.
Lower score is better.
Using SequentialEngine to train and score pipelines.
Searching up to 1 batches for a total of 9 pipelines.
Allowed model families: catboost, extra_trees, random_forest, lightgbm, xgboost, linear_model, decision_tree
Evaluating Baseline Pipeline: Mode Baseline Binary Classification Pipeline
Mode Baseline Binary Classification Pipeline:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 4.879
*****************************
* Evaluating Batch Number 1 *
*****************************
Elastic Net Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 1.161
Decision Tree Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 1.039
High coefficient of variation (cv >= 0.2) within cross validation scores.
Decision Tree Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler may not perform as estimated on unseen data.
Random Forest Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.281
LightGBM Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.341
Logistic Regression Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.405
XGBoost Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.256
Extra Trees Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.366
CatBoost Classifier w/ Imputer + DateTime Featurization Component + SMOTENC Oversampler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.554
Search finished after 00:42
Best pipeline: XGBoost Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler
Best pipeline Log Loss Binary: 0.255631
The AutoML search will log its progress, reporting each pipeline and parameter set evaluated during the search.
There are a number of mechanisms to control the AutoML search time. One way is to set the max_batches parameter which controls the maximum number of rounds of AutoML to evaluate, where each round may train and score a variable number of pipelines. Another way is to set the max_iterations parameter which controls the maximum number of candidate models to be evaluated during AutoML. By default, AutoML will search for a single batch. The first pipeline to be evaluated will always be a
baseline model representing a trivial solution.
The AutoML interface supports a variety of other parameters. For a comprehensive list, please refer to the API reference.
We also provide a standalone ``search` method <../generated/evalml.automl.search.html>`__ which does all of the above in a single line, and returns the AutoMLSearch instance and data check results. If there were data check errors, AutoML will not be run and no AutoMLSearch instance will be returned.
Detecting Problem Type¶
EvalML includes a simple method, detect_problem_type, to help determine the problem type given the target data.
This function can return the predicted problem type as a ProblemType enum, choosing from ProblemType.BINARY, ProblemType.MULTICLASS, and ProblemType.REGRESSION. If the target data is invalid (for instance when there is only 1 unique label), the function will throw an error instead.
[6]:
import pandas as pd
from evalml.problem_types import detect_problem_type
y_binary = pd.Series([0, 1, 1, 0, 1, 1])
detect_problem_type(y_binary)
[6]:
<ProblemTypes.BINARY: 'binary'>
Objective parameter¶
AutoMLSearch takes in an objective parameter to determine which objective to optimize for. By default, this parameter is set to auto, which allows AutoML to choose LogLossBinary for binary classification problems, LogLossMulticlass for multiclass classification problems, and R2 for regression problems.
It should be noted that the objective parameter is only used in ranking and helping choose the pipelines to iterate over, but is not used to optimize each individual pipeline during fit-time.
To get the default objective for each problem type, you can use the get_default_primary_search_objective function.
[7]:
from evalml.automl import get_default_primary_search_objective
binary_objective = get_default_primary_search_objective("binary")
multiclass_objective = get_default_primary_search_objective("multiclass")
regression_objective = get_default_primary_search_objective("regression")
print(binary_objective.name)
print(multiclass_objective.name)
print(regression_objective.name)
Log Loss Binary
Log Loss Multiclass
R2
Using custom pipelines¶
EvalML’s AutoML algorithm generates a set of pipelines to search with. To provide a custom set instead, set allowed_pipelines to a list of custom pipeline instances. Note: this will prevent AutoML from generating other pipelines to search over.
[8]:
from evalml.pipelines import MulticlassClassificationPipeline
automl_custom = evalml.automl.AutoMLSearch(X_train=X_train,
y_train=y_train,
problem_type='multiclass',
allowed_pipelines=[MulticlassClassificationPipeline(component_graph=['Simple Imputer', 'Random Forest Classifier'])])
Using default limit of max_batches=1.
1 pipelines ready for search.
Stopping the search early¶
To stop the search early, hit Ctrl-C. This will bring up a prompt asking for confirmation. Responding with y will immediately stop the search. Responding with n will continue the search.
Callback functions¶
AutoMLSearch supports several callback functions, which can be specified as parameters when initializing an AutoMLSearch object. They are:
start_iteration_callbackadd_result_callbackerror_callback
Start Iteration Callback¶
Users can set start_iteration_callback to set what function is called before each pipeline training iteration. This callback function must take three positional parameters: the pipeline class, the pipeline parameters, and the AutoMLSearch object.
[9]:
## start_iteration_callback example function
def start_iteration_callback_example(pipeline_class, pipeline_params, automl_obj):
print ("Training pipeline with the following parameters:", pipeline_params)
Add Result Callback¶
Users can set add_result_callback to set what function is called after each pipeline training iteration. This callback function must take three positional parameters: a dictionary containing the training results for the new pipeline, an untrained_pipeline containing the parameters used during training, and the AutoMLSearch object.
[10]:
## add_result_callback example function
def add_result_callback_example(pipeline_results_dict, untrained_pipeline, automl_obj):
print ("Results for trained pipeline with the following parameters:", pipeline_results_dict)
Error Callback¶
Users can set the error_callback to set what function called when search() errors and raises an Exception. This callback function takes three positional parameters: the Exception raised, the traceback, and the AutoMLSearch object. This callback function must also accept kwargs, so AutoMLSearch is able to pass along other parameters used by default.
Evalml defines several error callback functions, which can be found under evalml.automl.callbacks. They are:
silent_error_callbackraise_error_callbacklog_and_save_error_callbackraise_and_save_error_callbacklog_error_callback(default used whenerror_callbackis None)
[11]:
# error_callback example; this is implemented in the evalml library
def raise_error_callback(exception, traceback, automl, **kwargs):
"""Raises the exception thrown by the AutoMLSearch object. Also logs the exception as an error."""
logger.error(f'AutoMLSearch raised a fatal exception: {str(exception)}')
logger.error("\n".join(traceback))
raise exception
View Rankings¶
A summary of all the pipelines built can be returned as a pandas DataFrame which is sorted by score. The score column contains the average score across all cross-validation folds while the validation_score column is computed from the first cross-validation fold.
[12]:
automl.rankings
[12]:
| id | pipeline_name | mean_cv_score | standard_deviation_cv_score | validation_score | percent_better_than_baseline | high_variance_cv | parameters | |
|---|---|---|---|---|---|---|---|---|
| 0 | 6 | XGBoost Classifier w/ Imputer + DateTime Featu... | 0.255631 | 0.024779 | 0.232956 | 94.760056 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 1 | 3 | Random Forest Classifier w/ Imputer + DateTime... | 0.280528 | 0.014705 | 0.277029 | 94.249710 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 2 | 4 | LightGBM Classifier w/ Imputer + DateTime Feat... | 0.340811 | 0.061597 | 0.283440 | 93.014030 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 3 | 7 | Extra Trees Classifier w/ Imputer + DateTime F... | 0.365989 | 0.004973 | 0.362428 | 92.497942 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 4 | 5 | Logistic Regression Classifier w/ Imputer + Da... | 0.404618 | 0.011042 | 0.415770 | 91.706116 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 5 | 8 | CatBoost Classifier w/ Imputer + DateTime Feat... | 0.553732 | 0.004331 | 0.549766 | 88.649575 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 6 | 2 | Decision Tree Classifier w/ Imputer + DateTime... | 1.039042 | 0.338695 | 1.423304 | 78.701654 | True | {'Imputer': {'categorical_impute_strategy': 'm... |
| 7 | 1 | Elastic Net Classifier w/ Imputer + DateTime F... | 1.160858 | 0.061317 | 1.226963 | 76.204649 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 8 | 0 | Mode Baseline Binary Classification Pipeline | 4.878509 | 0.064297 | 4.915631 | 0.000000 | False | {'Baseline Classifier': {'strategy': 'mode'}} |
Describe Pipeline¶
Each pipeline is given an id. We can get more information about any particular pipeline using that id. Here, we will get more information about the pipeline with id = 1.
[13]:
automl.describe_pipeline(1)
**********************************************************************************************************************************
* Elastic Net Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler *
**********************************************************************************************************************************
Problem Type: binary
Model Family: Linear
Pipeline Steps
==============
1. Imputer
* categorical_impute_strategy : most_frequent
* numeric_impute_strategy : mean
* categorical_fill_value : None
* numeric_fill_value : None
2. DateTime Featurization Component
* features_to_extract : ['year', 'month', 'day_of_week', 'hour']
* encode_as_categories : False
* date_index : None
3. One Hot Encoder
* top_n : 10
* features_to_encode : None
* categories : None
* drop : if_binary
* handle_unknown : ignore
* handle_missing : error
4. SMOTENC Oversampler
* sampling_ratio : 0.25
* k_neighbors : 5
* n_jobs : -1
* categorical_features : [2, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80]
5. Standard Scaler
6. Elastic Net Classifier
* alpha : 0.0001
* l1_ratio : 0.15
* n_jobs : -1
* max_iter : 1000
* penalty : elasticnet
* loss : log
Training
========
Training for binary problems.
Total training time (including CV): 6.2 seconds
Cross Validation
----------------
Log Loss Binary MCC Binary AUC Precision F1 Balanced Accuracy Binary Accuracy Binary Sensitivity at Low Alert Rates # Training # Validation
0 1.227 0.239 0.620 0.545 0.245 0.568 0.861 0.000 533 267
1 1.106 0.154 0.670 0.290 0.261 0.570 0.809 0.000 533 267
2 1.150 0.256 0.638 0.350 0.364 0.632 0.816 0.143 534 266
mean 1.161 0.216 0.642 0.395 0.290 0.590 0.829 0.048 - -
std 0.061 0.055 0.025 0.133 0.064 0.037 0.029 0.082 - -
coef of var 0.053 0.255 0.039 0.338 0.222 0.062 0.034 1.732 - -
Get Pipeline¶
We can get the object of any pipeline via their id as well:
[14]:
pipeline = automl.get_pipeline(1)
print(pipeline.name)
print(pipeline.parameters)
Elastic Net Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler
{'Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'DateTime Featurization Component': {'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'date_index': None}, 'One Hot Encoder': {'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'SMOTENC Oversampler': {'sampling_ratio': 0.25, 'k_neighbors': 5, 'n_jobs': -1, 'categorical_features': [2, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80]}, 'Elastic Net Classifier': {'alpha': 0.0001, 'l1_ratio': 0.15, 'n_jobs': -1, 'max_iter': 1000, 'penalty': 'elasticnet', 'loss': 'log'}}
Get best pipeline¶
If you specifically want to get the best pipeline, there is a convenient accessor for that. The pipeline returned is already fitted on the input X, y data that we passed to AutoMLSearch. To turn off this default behavior, set train_best_pipeline=False when initializing AutoMLSearch.
[15]:
best_pipeline = automl.best_pipeline
print(best_pipeline.name)
print(best_pipeline.parameters)
best_pipeline.predict(X_train)
XGBoost Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler
{'Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'DateTime Featurization Component': {'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'date_index': None}, 'One Hot Encoder': {'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'SMOTENC Oversampler': {'sampling_ratio': 0.25, 'k_neighbors': 5, 'n_jobs': -1, 'categorical_features': [2, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80]}, 'XGBoost Classifier': {'eta': 0.1, 'max_depth': 6, 'min_child_weight': 1, 'n_estimators': 100}}
[15]:
<DataColumn: fraud (Physical Type = boolean) (Logical Type = Boolean) (Semantic Tags = set())>
Training and Scoring Multiple Pipelines using AutoMLSearch¶
AutoMLSearch will automatically fit the best pipeline on the entire training data. It also provides an easy API for training and scoring other pipelines.
If you’d like to train one or more pipelines on the entire training data, you can use the train_pipelinesmethod
Similarly, if you’d like to score one or more pipelines on a particular dataset, you can use the train_pipelinesmethod
[16]:
trained_pipelines = automl.train_pipelines([automl.get_pipeline(i) for i in [0, 1, 2]])
trained_pipelines
[16]:
{'Mode Baseline Binary Classification Pipeline': pipeline = BinaryClassificationPipeline(component_graph=['Baseline Classifier'], parameters={'Baseline Classifier':{'strategy': 'mode'}}, custom_name='Mode Baseline Binary Classification Pipeline', random_seed=0),
'Elastic Net Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler': pipeline = BinaryClassificationPipeline(component_graph=[Imputer, DateTimeFeaturizer, OneHotEncoder, SMOTENCSampler, StandardScaler, ElasticNetClassifier], parameters={'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'DateTime Featurization Component':{'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'date_index': None}, 'One Hot Encoder':{'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'SMOTENC Oversampler':{'sampling_ratio': 0.25, 'k_neighbors': 5, 'n_jobs': -1, 'categorical_features': [2, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80]}, 'Elastic Net Classifier':{'alpha': 0.0001, 'l1_ratio': 0.15, 'n_jobs': -1, 'max_iter': 1000, 'penalty': 'elasticnet', 'loss': 'log'}}, random_seed=0),
'Decision Tree Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler': pipeline = BinaryClassificationPipeline(component_graph=[Imputer, DateTimeFeaturizer, OneHotEncoder, SMOTENCSampler, DecisionTreeClassifier], parameters={'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'DateTime Featurization Component':{'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'date_index': None}, 'One Hot Encoder':{'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'SMOTENC Oversampler':{'sampling_ratio': 0.25, 'k_neighbors': 5, 'n_jobs': -1, 'categorical_features': [2, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80]}, 'Decision Tree Classifier':{'criterion': 'gini', 'max_features': 'auto', 'max_depth': 6, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0}}, random_seed=0)}
[17]:
pipeline_holdout_scores = automl.score_pipelines([trained_pipelines[name] for name in trained_pipelines.keys()],
X_holdout,
y_holdout,
['Accuracy Binary', 'F1', 'AUC'])
pipeline_holdout_scores
[17]:
{'Mode Baseline Binary Classification Pipeline': OrderedDict([('Accuracy Binary',
0.86),
('F1', 0.0),
('AUC', 0.5)]),
'Elastic Net Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler': OrderedDict([('Accuracy Binary',
0.835),
('F1', 0.35294117647058826),
('AUC', 0.7171926910299004)]),
'Decision Tree Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler': OrderedDict([('Accuracy Binary',
0.86),
('F1', 0.0),
('AUC', 0.4931478405315615)])}
Saving AutoMLSearch and pipelines from AutoMLSearch¶
There are two ways to save results from AutoMLSearch.
You can save the AutoMLSearch object itself, calling
.save(<filepath>)to do so. This will allow you to save the AutoMLSearch state and reload all pipelines from this.If you want to save a pipeline from AutoMLSearch for future use, pipeline classes themselves have a
.save(<filepath>)method.
[18]:
# saving the entire automl search
automl.save("automl.cloudpickle")
automl2 = evalml.automl.AutoMLSearch.load("automl.cloudpickle")
# saving the best pipeline using .save()
best_pipeline.save("pipeline.cloudpickle")
best_pipeline_copy = evalml.pipelines.PipelineBase.load("pipeline.cloudpickle")
Limiting the AutoML Search Space¶
The AutoML search algorithm first trains each component in the pipeline with their default values. After the first iteration, it then tweaks the parameters of these components using the pre-defined hyperparameter ranges that these components have. To limit the search over certain hyperparameter ranges, you can specify a pipeline_parameters argument with your pipeline parameters. These parameters will also limit the hyperparameter search space. Hyperparameter ranges can be found through the
API reference for each component. Parameter arguments must be specified as dictionaries, but the associated values can be single values or skopt.space Real, Integer, Categorical values.
[19]:
from evalml import AutoMLSearch
from evalml.demos import load_fraud
from skopt.space import Categorical
from evalml.model_family import ModelFamily
import woodwork as ww
X, y = load_fraud(n_rows=1000)
# example of setting parameter to just one value
pipeline_hyperparameters = {'Imputer': {
'numeric_impute_strategy': 'mean'
}}
# limit the numeric impute strategy to include only `median` and `most_frequent`
# `mean` is the default value for this argument, but it doesn't need to be included in the specified hyperparameter range for this to work
pipeline_hyperparameters = {'Imputer': {
'numeric_impute_strategy': Categorical(['median', 'most_frequent'])
}}
# using this pipeline parameter means that our Imputer components in the pipelines will only search through 'median' and 'most_frequent' stretegies for 'numeric_impute_strategy'
automl_constrained = AutoMLSearch(X_train=X, y_train=y, problem_type='binary', pipeline_parameters=pipeline_hyperparameters)
Number of Features
Boolean 1
Categorical 6
Numeric 5
Number of training examples: 1000
Targets
False 85.90%
True 14.10%
Name: fraud, dtype: object
Using default limit of max_batches=1.
Generating pipelines to search over...
8 pipelines ready for search.
Adding ensemble methods to AutoML¶
Stacking¶
Stacking is an ensemble machine learning algorithm that involves training a model to best combine the predictions of several base learning algorithms. First, each base learning algorithms is trained using the given data. Then, the combining algorithm or meta-learner is trained on the predictions made by those base learning algorithms to make a final prediction.
AutoML enables stacking using the ensembling flag during initalization; this is set to False by default. The stacking ensemble pipeline runs in its own batch after a whole cycle of training has occurred (each allowed pipeline trains for one batch). Note that this means a large number of iterations may need to run before the stacking ensemble runs. It is also important to note that only the first CV fold is calculated for stacking ensembles because the model internally uses CV
folds.
[20]:
X, y = evalml.demos.load_breast_cancer()
automl_with_ensembling = AutoMLSearch(X_train=X, y_train=y,
problem_type="binary",
allowed_model_families=[ModelFamily.RANDOM_FOREST, ModelFamily.LINEAR_MODEL],
max_batches=5,
ensembling=True)
automl_with_ensembling.search()
Generating pipelines to search over...
3 pipelines ready for search.
Ensembling will run every 4 batches.
*****************************
* Beginning pipeline search *
*****************************
Optimizing for Log Loss Binary.
Lower score is better.
Using SequentialEngine to train and score pipelines.
Searching up to 5 batches for a total of 20 pipelines.
Allowed model families: random_forest, linear_model
Evaluating Baseline Pipeline: Mode Baseline Binary Classification Pipeline
Mode Baseline Binary Classification Pipeline:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 12.868
*****************************
* Evaluating Batch Number 1 *
*****************************
Elastic Net Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.335
High coefficient of variation (cv >= 0.2) within cross validation scores.
Elastic Net Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data.
Random Forest Classifier w/ Imputer:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.132
Logistic Regression Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.074
High coefficient of variation (cv >= 0.2) within cross validation scores.
Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data.
*****************************
* Evaluating Batch Number 2 *
*****************************
Logistic Regression Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.093
High coefficient of variation (cv >= 0.2) within cross validation scores.
Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data.
Logistic Regression Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.082
High coefficient of variation (cv >= 0.2) within cross validation scores.
Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data.
Logistic Regression Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.093
High coefficient of variation (cv >= 0.2) within cross validation scores.
Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data.
Logistic Regression Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.087
High coefficient of variation (cv >= 0.2) within cross validation scores.
Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data.
Logistic Regression Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.076
High coefficient of variation (cv >= 0.2) within cross validation scores.
Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data.
*****************************
* Evaluating Batch Number 3 *
*****************************
Random Forest Classifier w/ Imputer:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.133
Random Forest Classifier w/ Imputer:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.172
Random Forest Classifier w/ Imputer:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.128
Random Forest Classifier w/ Imputer:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.171
Random Forest Classifier w/ Imputer:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.128
*****************************
* Evaluating Batch Number 4 *
*****************************
Elastic Net Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.661
Elastic Net Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.661
Elastic Net Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.308
Elastic Net Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.452
Elastic Net Classifier w/ Imputer + Standard Scaler:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.216
*****************************
* Evaluating Batch Number 5 *
*****************************
Stacked Ensemble Classification Pipeline:
Starting cross validation
Finished cross validation - mean Log Loss Binary: 0.106
Search finished after 00:47
Best pipeline: Logistic Regression Classifier w/ Imputer + Standard Scaler
Best pipeline Log Loss Binary: 0.073551
We can view more information about the stacking ensemble pipeline (which was the best performing pipeline) by calling .describe().
[21]:
automl_with_ensembling.best_pipeline.describe()
***************************************************************
* Logistic Regression Classifier w/ Imputer + Standard Scaler *
***************************************************************
Problem Type: binary
Model Family: Linear
Number of features: 30
Pipeline Steps
==============
1. Imputer
* categorical_impute_strategy : most_frequent
* numeric_impute_strategy : mean
* categorical_fill_value : None
* numeric_fill_value : None
2. Standard Scaler
3. Logistic Regression Classifier
* penalty : l2
* C : 1.0
* n_jobs : -1
* multi_class : auto
* solver : lbfgs
Access raw results¶
The AutoMLSearch class records detailed results information under the results field, including information about the cross-validation scoring and parameters.
[22]:
automl.results
[22]:
{'pipeline_results': {0: {'id': 0,
'pipeline_name': 'Mode Baseline Binary Classification Pipeline',
'pipeline_class': evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline,
'pipeline_summary': 'Baseline Classifier',
'parameters': {'Baseline Classifier': {'strategy': 'mode'}},
'mean_cv_score': 4.87850936144123,
'standard_deviation_cv_score': 0.06429673275097571,
'high_variance_cv': False,
'training_time': 0.11727619171142578,
'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary',
4.915631097403019),
('MCC Binary', 0.0),
('AUC', 0.5),
('Precision', 0.0),
('F1', 0.0),
('Balanced Accuracy Binary', 0.5),
('Accuracy Binary', 0.8576779026217228),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 4.915631097403019,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
4.915631097403019),
('MCC Binary', 0.0),
('AUC', 0.5),
('Precision', 0.0),
('F1', 0.0),
('Balanced Accuracy Binary', 0.5),
('Accuracy Binary', 0.8576779026217228),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 4.915631097403019,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
4.8042658895176515),
('MCC Binary', 0.0),
('AUC', 0.5),
('Precision', 0.0),
('F1', 0.0),
('Balanced Accuracy Binary', 0.5),
('Accuracy Binary', 0.8609022556390977),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 534),
('# Validation', 266)]),
'mean_cv_score': 4.8042658895176515,
'binary_classification_threshold': None}],
'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 0,
'MCC Binary': 0,
'AUC': 0,
'Precision': 0,
'F1': 0,
'Balanced Accuracy Binary': 0,
'Accuracy Binary': 0,
'Sensitivity at Low Alert Rates': 0},
'percent_better_than_baseline': 0,
'validation_score': 4.915631097403019},
1: {'id': 1,
'pipeline_name': 'Elastic Net Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler',
'pipeline_class': evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline,
'pipeline_summary': 'Elastic Net Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler',
'parameters': {'Imputer': {'categorical_impute_strategy': 'most_frequent',
'numeric_impute_strategy': 'mean',
'categorical_fill_value': None,
'numeric_fill_value': None},
'DateTime Featurization Component': {'features_to_extract': ['year',
'month',
'day_of_week',
'hour'],
'encode_as_categories': False,
'date_index': None},
'One Hot Encoder': {'top_n': 10,
'features_to_encode': None,
'categories': None,
'drop': 'if_binary',
'handle_unknown': 'ignore',
'handle_missing': 'error'},
'SMOTENC Oversampler': {'sampling_ratio': 0.25,
'k_neighbors': 5,
'n_jobs': -1,
'categorical_features': [2,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40,
41,
42,
43,
44,
45,
46,
47,
48,
49,
50,
51,
52,
53,
54,
55,
56,
57,
58,
59,
60,
61,
62,
63,
64,
65,
66,
67,
68,
69,
70,
71,
72,
73,
74,
75,
76,
77,
78,
79,
80]},
'Elastic Net Classifier': {'alpha': 0.0001,
'l1_ratio': 0.15,
'n_jobs': -1,
'max_iter': 1000,
'penalty': 'elasticnet',
'loss': 'log'}},
'mean_cv_score': 1.1608584040237186,
'standard_deviation_cv_score': 0.061317388229851395,
'high_variance_cv': False,
'training_time': 6.190917015075684,
'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary',
1.2269629314176882),
('MCC Binary', 0.23918053758304247),
('AUC', 0.619742587910825),
('Precision', 0.5454545454545454),
('F1', 0.24489795918367344),
('Balanced Accuracy Binary', 0.568030337853367),
('Accuracy Binary', 0.8614232209737828),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 1.2269629314176882,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
1.10584103862323),
('MCC Binary', 0.15352874534176394),
('AUC', 0.6695012640772237),
('Precision', 0.2903225806451613),
('F1', 0.2608695652173913),
('Balanced Accuracy Binary', 0.5703861181337624),
('Accuracy Binary', 0.8089887640449438),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 1.10584103862323,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
1.149771242030238),
('MCC Binary', 0.2564011886182511),
('AUC', 0.6376726071049215),
('Precision', 0.35),
('F1', 0.36363636363636365),
('Balanced Accuracy Binary', 0.6324206302372242),
('Accuracy Binary', 0.8157894736842105),
('Sensitivity at Low Alert Rates', 0.14285714285714285),
('# Training', 534),
('# Validation', 266)]),
'mean_cv_score': 1.149771242030238,
'binary_classification_threshold': None}],
'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 76.20464945298838,
'MCC Binary': inf,
'AUC': 14.230548636432339,
'Precision': 39.525904203323556,
'F1': 28.98012960124761,
'Balanced Accuracy Binary': 9.027902874145111,
'Accuracy Binary': -3.001886739320203,
'Sensitivity at Low Alert Rates': 4.761904761904762},
'percent_better_than_baseline': 76.20464945298838,
'validation_score': 1.2269629314176882},
2: {'id': 2,
'pipeline_name': 'Decision Tree Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler',
'pipeline_class': evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline,
'pipeline_summary': 'Decision Tree Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler',
'parameters': {'Imputer': {'categorical_impute_strategy': 'most_frequent',
'numeric_impute_strategy': 'mean',
'categorical_fill_value': None,
'numeric_fill_value': None},
'DateTime Featurization Component': {'features_to_extract': ['year',
'month',
'day_of_week',
'hour'],
'encode_as_categories': False,
'date_index': None},
'One Hot Encoder': {'top_n': 10,
'features_to_encode': None,
'categories': None,
'drop': 'if_binary',
'handle_unknown': 'ignore',
'handle_missing': 'error'},
'SMOTENC Oversampler': {'sampling_ratio': 0.25,
'k_neighbors': 5,
'n_jobs': -1,
'categorical_features': [2,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40,
41,
42,
43,
44,
45,
46,
47,
48,
49,
50,
51,
52,
53,
54,
55,
56,
57,
58,
59,
60,
61,
62,
63,
64,
65,
66,
67,
68,
69,
70,
71,
72,
73,
74,
75,
76,
77,
78,
79,
80]},
'Decision Tree Classifier': {'criterion': 'gini',
'max_features': 'auto',
'max_depth': 6,
'min_samples_split': 2,
'min_weight_fraction_leaf': 0.0}},
'mean_cv_score': 1.0390418184361565,
'standard_deviation_cv_score': 0.33869509222689664,
'high_variance_cv': True,
'training_time': 4.579573154449463,
'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary',
1.4233044946677307),
('MCC Binary', -0.043424289473019614),
('AUC', 0.5079292116754769),
('Precision', 0.0),
('F1', 0.0),
('Balanced Accuracy Binary', 0.49344978165938863),
('Accuracy Binary', 0.846441947565543),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 1.4233044946677307,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.9099264484562098),
('MCC Binary', 0.10188570508379964),
('AUC', 0.5898069409331188),
('Precision', 0.4),
('F1', 0.09302325581395349),
('Balanced Accuracy Binary', 0.5197655711330729),
('Accuracy Binary', 0.8539325842696629),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.9099264484562098,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.7838945121845288),
('MCC Binary', 0.5837323367812007),
('AUC', 0.6835831464652424),
('Precision', 0.8888888888888888),
('F1', 0.5818181818181818),
('Balanced Accuracy Binary', 0.711849403989142),
('Accuracy Binary', 0.9135338345864662),
('Sensitivity at Low Alert Rates', 0.14285714285714285),
('# Training', 534),
('# Validation', 266)]),
'mean_cv_score': 0.7838945121845288,
'binary_classification_threshold': None}],
'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 78.70165369265176,
'MCC Binary': inf,
'AUC': 9.377309969127944,
'Precision': 42.96296296296296,
'F1': 22.49471458773784,
'Balanced Accuracy Binary': 7.502158559386773,
'Accuracy Binary': 1.2550101846376172,
'Sensitivity at Low Alert Rates': 4.761904761904762},
'percent_better_than_baseline': 78.70165369265176,
'validation_score': 1.4233044946677307},
3: {'id': 3,
'pipeline_name': 'Random Forest Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler',
'pipeline_class': evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline,
'pipeline_summary': 'Random Forest Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler',
'parameters': {'Imputer': {'categorical_impute_strategy': 'most_frequent',
'numeric_impute_strategy': 'mean',
'categorical_fill_value': None,
'numeric_fill_value': None},
'DateTime Featurization Component': {'features_to_extract': ['year',
'month',
'day_of_week',
'hour'],
'encode_as_categories': False,
'date_index': None},
'One Hot Encoder': {'top_n': 10,
'features_to_encode': None,
'categories': None,
'drop': 'if_binary',
'handle_unknown': 'ignore',
'handle_missing': 'error'},
'SMOTENC Oversampler': {'sampling_ratio': 0.25,
'k_neighbors': 5,
'n_jobs': -1,
'categorical_features': [2,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40,
41,
42,
43,
44,
45,
46,
47,
48,
49,
50,
51,
52,
53,
54,
55,
56,
57,
58,
59,
60,
61,
62,
63,
64,
65,
66,
67,
68,
69,
70,
71,
72,
73,
74,
75,
76,
77,
78,
79,
80]},
'Random Forest Classifier': {'n_estimators': 100,
'max_depth': 6,
'n_jobs': -1}},
'mean_cv_score': 0.28052844604643323,
'standard_deviation_cv_score': 0.01470495982021426,
'high_variance_cv': False,
'training_time': 5.3753132820129395,
'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.2770293417811284),
('MCC Binary', 0.7356212073393703),
('AUC', 0.8391174442656861),
('Precision', 1.0),
('F1', 0.7333333333333334),
('Balanced Accuracy Binary', 0.7894736842105263),
('Accuracy Binary', 0.9400749063670412),
('Sensitivity at Low Alert Rates', 0.16666666666666666),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.2770293417811284,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.26788866027494596),
('MCC Binary', 0.7522224738613313),
('AUC', 0.854286370949207),
('Precision', 0.96),
('F1', 0.7619047619047619),
('Balanced Accuracy Binary', 0.8136060675706733),
('Accuracy Binary', 0.9438202247191011),
('Sensitivity at Low Alert Rates', 0.16666666666666666),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.26788866027494596,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.29666733608322526),
('MCC Binary', 0.7283556346331993),
('AUC', 0.7783547739879618),
('Precision', 1.0),
('F1', 0.7241379310344828),
('Balanced Accuracy Binary', 0.7837837837837838),
('Accuracy Binary', 0.9398496240601504),
('Sensitivity at Low Alert Rates', 0.14285714285714285),
('# Training', 534),
('# Validation', 266)]),
'mean_cv_score': 0.29666733608322526,
'binary_classification_threshold': None}],
'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 94.24970979324804,
'MCC Binary': inf,
'AUC': 32.391952973428495,
'Precision': 98.66666666666667,
'F1': 73.9792008757526,
'Balanced Accuracy Binary': 29.562117852166114,
'Accuracy Binary': 8.24955647545831,
'Sensitivity at Low Alert Rates': 15.873015873015872},
'percent_better_than_baseline': 94.24970979324804,
'validation_score': 0.2770293417811284},
4: {'id': 4,
'pipeline_name': 'LightGBM Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler',
'pipeline_class': evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline,
'pipeline_summary': 'LightGBM Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler',
'parameters': {'Imputer': {'categorical_impute_strategy': 'most_frequent',
'numeric_impute_strategy': 'mean',
'categorical_fill_value': None,
'numeric_fill_value': None},
'DateTime Featurization Component': {'features_to_extract': ['year',
'month',
'day_of_week',
'hour'],
'encode_as_categories': False,
'date_index': None},
'One Hot Encoder': {'top_n': 10,
'features_to_encode': None,
'categories': None,
'drop': 'if_binary',
'handle_unknown': 'ignore',
'handle_missing': 'error'},
'SMOTENC Oversampler': {'sampling_ratio': 0.25,
'k_neighbors': 5,
'n_jobs': -1,
'categorical_features': [2,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40,
41,
42,
43,
44,
45,
46,
47,
48,
49,
50,
51,
52,
53,
54,
55,
56,
57,
58,
59,
60,
61,
62,
63,
64,
65,
66,
67,
68,
69,
70,
71,
72,
73,
74,
75,
76,
77,
78,
79,
80]},
'LightGBM Classifier': {'boosting_type': 'gbdt',
'learning_rate': 0.1,
'n_estimators': 100,
'max_depth': 0,
'num_leaves': 31,
'min_child_samples': 20,
'n_jobs': -1,
'bagging_freq': 0,
'bagging_fraction': 0.9}},
'mean_cv_score': 0.3408112052442249,
'standard_deviation_cv_score': 0.061597106895076935,
'high_variance_cv': False,
'training_time': 5.303739786148071,
'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.2834397131526597),
('MCC Binary', 0.8052567641906508),
('AUC', 0.8303838198115374),
('Precision', 0.9642857142857143),
('F1', 0.8181818181818182),
('Balanced Accuracy Binary', 0.8530797517811997),
('Accuracy Binary', 0.9550561797752809),
('Sensitivity at Low Alert Rates', 0.16666666666666666),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.2834397131526597,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.3330883328153368),
('MCC Binary', 0.7191045324828429),
('AUC', 0.8114226614571364),
('Precision', 0.8620689655172413),
('F1', 0.746268656716418),
('Balanced Accuracy Binary', 0.8202137439669042),
('Accuracy Binary', 0.9363295880149812),
('Sensitivity at Low Alert Rates', 0.16666666666666666),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.3330883328153368,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.4059055697646783),
('MCC Binary', 0.652351963814212),
('AUC', 0.770329281246312),
('Precision', 0.84),
('F1', 0.6774193548387097),
('Balanced Accuracy Binary', 0.7750501593296353),
('Accuracy Binary', 0.924812030075188),
('Sensitivity at Low Alert Rates', 0.14285714285714285),
('# Training', 534),
('# Validation', 266)]),
'mean_cv_score': 0.4059055697646783,
'binary_classification_threshold': None}],
'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 93.01402990147095,
'MCC Binary': inf,
'AUC': 30.404525417166184,
'Precision': 88.87848932676518,
'F1': 74.72899432456487,
'Balanced Accuracy Binary': 31.611455169257972,
'Accuracy Binary': 7.997991232763546,
'Sensitivity at Low Alert Rates': 15.873015873015872},
'percent_better_than_baseline': 93.01402990147095,
'validation_score': 0.2834397131526597},
5: {'id': 5,
'pipeline_name': 'Logistic Regression Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler',
'pipeline_class': evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline,
'pipeline_summary': 'Logistic Regression Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler',
'parameters': {'Imputer': {'categorical_impute_strategy': 'most_frequent',
'numeric_impute_strategy': 'mean',
'categorical_fill_value': None,
'numeric_fill_value': None},
'DateTime Featurization Component': {'features_to_extract': ['year',
'month',
'day_of_week',
'hour'],
'encode_as_categories': False,
'date_index': None},
'One Hot Encoder': {'top_n': 10,
'features_to_encode': None,
'categories': None,
'drop': 'if_binary',
'handle_unknown': 'ignore',
'handle_missing': 'error'},
'SMOTENC Oversampler': {'sampling_ratio': 0.25,
'k_neighbors': 5,
'n_jobs': -1,
'categorical_features': [2,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40,
41,
42,
43,
44,
45,
46,
47,
48,
49,
50,
51,
52,
53,
54,
55,
56,
57,
58,
59,
60,
61,
62,
63,
64,
65,
66,
67,
68,
69,
70,
71,
72,
73,
74,
75,
76,
77,
78,
79,
80]},
'Logistic Regression Classifier': {'penalty': 'l2',
'C': 1.0,
'n_jobs': -1,
'multi_class': 'auto',
'solver': 'lbfgs'}},
'mean_cv_score': 0.40461788824526157,
'standard_deviation_cv_score': 0.011042332709237221,
'high_variance_cv': False,
'training_time': 7.439377069473267,
'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.41577003128418283),
('MCC Binary', 0.20194228704438547),
('AUC', 0.6501953573891059),
('Precision', 0.5),
('F1', 0.20833333333333334),
('Balanced Accuracy Binary', 0.554872443116525),
('Accuracy Binary', 0.8576779026217228),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.41577003128418283,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.4043948897582815),
('MCC Binary', 0.19273028329700836),
('AUC', 0.6896115835440129),
('Precision', 0.42857142857142855),
('F1', 0.23076923076923078),
('Balanced Accuracy Binary', 0.5614801195127557),
('Accuracy Binary', 0.850187265917603),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.4043948897582815,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.3936887436933204),
('MCC Binary', 0.18434722965705141),
('AUC', 0.6985719343797946),
('Precision', 0.6),
('F1', 0.14285714285714288),
('Balanced Accuracy Binary', 0.5361737283134663),
('Accuracy Binary', 0.8646616541353384),
('Sensitivity at Low Alert Rates', 0.14285714285714285),
('# Training', 534),
('# Validation', 266)]),
'mean_cv_score': 0.3936887436933204,
'binary_classification_threshold': None}],
'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 91.70611639196021,
'MCC Binary': inf,
'AUC': 17.945962510430448,
'Precision': 50.95238095238095,
'F1': 19.3986568986569,
'Balanced Accuracy Binary': 5.084209698091557,
'Accuracy Binary': -0.12437460692931168,
'Sensitivity at Low Alert Rates': 4.761904761904762},
'percent_better_than_baseline': 91.70611639196021,
'validation_score': 0.41577003128418283},
6: {'id': 6,
'pipeline_name': 'XGBoost Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler',
'pipeline_class': evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline,
'pipeline_summary': 'XGBoost Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler',
'parameters': {'Imputer': {'categorical_impute_strategy': 'most_frequent',
'numeric_impute_strategy': 'mean',
'categorical_fill_value': None,
'numeric_fill_value': None},
'DateTime Featurization Component': {'features_to_extract': ['year',
'month',
'day_of_week',
'hour'],
'encode_as_categories': False,
'date_index': None},
'One Hot Encoder': {'top_n': 10,
'features_to_encode': None,
'categories': None,
'drop': 'if_binary',
'handle_unknown': 'ignore',
'handle_missing': 'error'},
'SMOTENC Oversampler': {'sampling_ratio': 0.25,
'k_neighbors': 5,
'n_jobs': -1,
'categorical_features': [2,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40,
41,
42,
43,
44,
45,
46,
47,
48,
49,
50,
51,
52,
53,
54,
55,
56,
57,
58,
59,
60,
61,
62,
63,
64,
65,
66,
67,
68,
69,
70,
71,
72,
73,
74,
75,
76,
77,
78,
79,
80]},
'XGBoost Classifier': {'eta': 0.1,
'max_depth': 6,
'min_child_weight': 1,
'n_estimators': 100}},
'mean_cv_score': 0.2556311391503104,
'standard_deviation_cv_score': 0.024779307376775216,
'high_variance_cv': False,
'training_time': 5.337289571762085,
'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.23295623301962776),
('MCC Binary', 0.7878274757624152),
('AUC', 0.8209606986899562),
('Precision', 0.9629629629629629),
('F1', 0.7999999999999999),
('Balanced Accuracy Binary', 0.8399218570443576),
('Accuracy Binary', 0.951310861423221),
('Sensitivity at Low Alert Rates', 0.16666666666666666),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.23295623301962776,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.2518556656940856),
('MCC Binary', 0.7522713675763832),
('AUC', 0.8146403125718226),
('Precision', 0.9259259259259259),
('F1', 0.7692307692307692),
('Balanced Accuracy Binary', 0.8245805561939784),
('Accuracy Binary', 0.9438202247191011),
('Sensitivity at Low Alert Rates', 0.16666666666666666),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.2518556656940856,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.28208151873721776),
('MCC Binary', 0.688072697187691),
('AUC', 0.7994807034108344),
('Precision', 0.9130434782608695),
('F1', 0.6999999999999998),
('Balanced Accuracy Binary', 0.7794169715567095),
('Accuracy Binary', 0.9323308270676691),
('Sensitivity at Low Alert Rates', 0.14285714285714285),
('# Training', 534),
('# Validation', 266)]),
'mean_cv_score': 0.28208151873721776,
'binary_classification_threshold': None}],
'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 94.76005639815375,
'MCC Binary': inf,
'AUC': 31.169390489087114,
'Precision': 93.39774557165862,
'F1': 75.64102564102562,
'Balanced Accuracy Binary': 31.463979493168193,
'Accuracy Binary': 8.373461744248257,
'Sensitivity at Low Alert Rates': 15.873015873015872},
'percent_better_than_baseline': 94.76005639815375,
'validation_score': 0.23295623301962776},
7: {'id': 7,
'pipeline_name': 'Extra Trees Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler',
'pipeline_class': evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline,
'pipeline_summary': 'Extra Trees Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler',
'parameters': {'Imputer': {'categorical_impute_strategy': 'most_frequent',
'numeric_impute_strategy': 'mean',
'categorical_fill_value': None,
'numeric_fill_value': None},
'DateTime Featurization Component': {'features_to_extract': ['year',
'month',
'day_of_week',
'hour'],
'encode_as_categories': False,
'date_index': None},
'One Hot Encoder': {'top_n': 10,
'features_to_encode': None,
'categories': None,
'drop': 'if_binary',
'handle_unknown': 'ignore',
'handle_missing': 'error'},
'SMOTENC Oversampler': {'sampling_ratio': 0.25,
'k_neighbors': 5,
'n_jobs': -1,
'categorical_features': [2,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40,
41,
42,
43,
44,
45,
46,
47,
48,
49,
50,
51,
52,
53,
54,
55,
56,
57,
58,
59,
60,
61,
62,
63,
64,
65,
66,
67,
68,
69,
70,
71,
72,
73,
74,
75,
76,
77,
78,
79,
80]},
'Extra Trees Classifier': {'n_estimators': 100,
'max_features': 'auto',
'max_depth': 6,
'min_samples_split': 2,
'min_weight_fraction_leaf': 0.0,
'n_jobs': -1}},
'mean_cv_score': 0.3659885921343484,
'standard_deviation_cv_score': 0.0049725729418878956,
'high_variance_cv': False,
'training_time': 5.2193098068237305,
'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.36242767810979015),
('MCC Binary', 0.0),
('AUC', 0.7914272581015859),
('Precision', 0.0),
('F1', 0.0),
('Balanced Accuracy Binary', 0.5),
('Accuracy Binary', 0.8576779026217228),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.36242767810979015,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.36386822870155777),
('MCC Binary', 0.0),
('AUC', 0.8005056308894507),
('Precision', 0.0),
('F1', 0.0),
('Balanced Accuracy Binary', 0.5),
('Accuracy Binary', 0.8576779026217228),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.36386822870155777,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.37166986959169723),
('MCC Binary', 0.0),
('AUC', 0.727487312640151),
('Precision', 0.0),
('F1', 0.0),
('Balanced Accuracy Binary', 0.5),
('Accuracy Binary', 0.8609022556390977),
('Sensitivity at Low Alert Rates', 0.0),
('# Training', 534),
('# Validation', 266)]),
'mean_cv_score': 0.37166986959169723,
'binary_classification_threshold': None}],
'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 92.49794219875747,
'MCC Binary': 0,
'AUC': 27.314006721039597,
'Precision': 0,
'F1': 0,
'Balanced Accuracy Binary': 0,
'Accuracy Binary': 0,
'Sensitivity at Low Alert Rates': 0},
'percent_better_than_baseline': 92.49794219875747,
'validation_score': 0.36242767810979015},
8: {'id': 8,
'pipeline_name': 'CatBoost Classifier w/ Imputer + DateTime Featurization Component + SMOTENC Oversampler',
'pipeline_class': evalml.pipelines.binary_classification_pipeline.BinaryClassificationPipeline,
'pipeline_summary': 'CatBoost Classifier w/ Imputer + DateTime Featurization Component + SMOTENC Oversampler',
'parameters': {'Imputer': {'categorical_impute_strategy': 'most_frequent',
'numeric_impute_strategy': 'mean',
'categorical_fill_value': None,
'numeric_fill_value': None},
'DateTime Featurization Component': {'features_to_extract': ['year',
'month',
'day_of_week',
'hour'],
'encode_as_categories': False,
'date_index': None},
'SMOTENC Oversampler': {'sampling_ratio': 0.25,
'k_neighbors': 5,
'n_jobs': -1,
'categorical_features': [1, 3, 4, 5, 6, 7, 8, 9]},
'CatBoost Classifier': {'n_estimators': 10,
'eta': 0.03,
'max_depth': 6,
'bootstrap_type': None,
'silent': True,
'allow_writing_files': False}},
'mean_cv_score': 0.5537315568271693,
'standard_deviation_cv_score': 0.004330708933714112,
'high_variance_cv': False,
'training_time': 1.4524283409118652,
'cv_data': [{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.5497664891960894),
('MCC Binary', 0.8063138051598471),
('AUC', 0.9014019765571133),
('Precision', 1.0),
('F1', 0.8125000000000001),
('Balanced Accuracy Binary', 0.8421052631578947),
('Accuracy Binary', 0.9550561797752809),
('Sensitivity at Low Alert Rates', 0.16666666666666666),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.5497664891960894,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.5530752047264043),
('MCC Binary', 0.7167152593903513),
('AUC', 0.8196966214663296),
('Precision', 0.8888888888888888),
('F1', 0.7384615384615385),
('Balanced Accuracy Binary', 0.8092392553435992),
('Accuracy Binary', 0.9363295880149812),
('Sensitivity at Low Alert Rates', 0.16666666666666666),
('# Training', 533),
('# Validation', 267)]),
'mean_cv_score': 0.5530752047264043,
'binary_classification_threshold': None},
{'all_objective_scores': OrderedDict([('Log Loss Binary',
0.5583529765590143),
('MCC Binary', 0.7093561256112549),
('AUC', 0.7424761005547031),
('Precision', 1.0),
('F1', 0.7017543859649124),
('Balanced Accuracy Binary', 0.7702702702702703),
('Accuracy Binary', 0.9360902255639098),
('Sensitivity at Low Alert Rates', 0.14285714285714285),
('# Training', 534),
('# Validation', 266)]),
'mean_cv_score': 0.5583529765590143,
'binary_classification_threshold': None}],
'percent_better_than_baseline_all_objectives': {'Log Loss Binary': 88.64957478192512,
'MCC Binary': inf,
'AUC': 32.11915661927153,
'Precision': 96.29629629629629,
'F1': 75.09053081421503,
'Balanced Accuracy Binary': 30.72049295905881,
'Accuracy Binary': 8.373931082387621,
'Sensitivity at Low Alert Rates': 15.873015873015872},
'percent_better_than_baseline': 88.64957478192512,
'validation_score': 0.5497664891960894}},
'search_order': [0, 1, 2, 3, 4, 5, 6, 7, 8]}