Start¶
In this guide, we’ll show how you can use EvalML to automatically find the best pipeline for predicting whether a patient has breast cancer. Along the way, we’ll highlight EvalML’s built-in tools and features for understanding and interacting with the search process.
[1]:
import evalml
from evalml import AutoMLSearch
from evalml.utils import infer_feature_types
First, we load in the features and outcomes we want to use to train our model.
[2]:
X, y = evalml.demos.load_fraud(n_rows=1000, return_pandas=True)
Number of Features
Boolean 1
Categorical 6
Numeric 5
Number of training examples: 1000
Targets
False 85.90%
True 14.10%
Name: fraud, dtype: object
First, we will clean the data. Since EvalML accepts a pandas input, it can run type inference on this data directly. Since we’d like to change the types inferred by EvalML, we can use the infer_feature_types utility method. Here’s what we’re going to do with the following dataset:
Reformat the
expiration_datecolumn so it reflects a more familiar date format.Cast the
latandlngcolumns from float to str.Use
infer_feature_typesto specify what types certain columns should be. For example, to avoid having theprovidercolumn be inferred as natural language text, we have specified it as a categorical column instead.
The infer_feature_types utility method takes a pandas or numpy input and converts it to a Woodwork data structure, providing us with flexibility to cast the data as necessary.
[3]:
X['expiration_date'] = X['expiration_date'].apply(lambda x: '20{}-01-{}'.format(x.split("/")[1], x.split("/")[0]))
X[['lat', 'lng']] = X[['lat', 'lng']].astype('str')
X = infer_feature_types(X, feature_types= {'store_id': 'categorical',
'expiration_date': 'datetime',
'lat': 'categorical',
'lng': 'categorical',
'provider': 'categorical'})
X
[3]:
| Physical Type | Logical Type | Semantic Tag(s) | |
|---|---|---|---|
| Data Column | |||
| card_id | Int64 | Integer | ['numeric'] |
| store_id | category | Categorical | ['category'] |
| datetime | datetime64[ns] | Datetime | [] |
| amount | Int64 | Integer | ['numeric'] |
| currency | category | Categorical | ['category'] |
| customer_present | boolean | Boolean | [] |
| expiration_date | category | Datetime | [] |
| provider | category | Categorical | ['category'] |
| lat | category | Categorical | ['category'] |
| lng | category | Categorical | ['category'] |
| region | category | Categorical | ['category'] |
| country | category | Categorical | ['category'] |
In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set.
[4]:
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, problem_type='binary', test_size=.2)
Note: To provide data to EvalML, it is recommended that you create a DataTable object using the Woodwork project. Here, split_data() returns Woodwork data structures.
EvalML also accepts and works well with pandas DataFrames. But using the DataTable makes it easy to control how EvalML will treat each feature, as a numeric feature, a categorical feature, a text feature or other type of feature. Woodwork’s DataTable includes features like inferring when a categorical feature should be treated as a text feature.
EvalML has many options to configure the pipeline search. At the minimum, we need to define an objective function. For simplicity, we will use the F1 score in this example. However, the real power of EvalML is in using domain-specific objective functions or building your own.
Below EvalML utilizes Bayesian optimization (EvalML’s default optimizer) to search and find the best pipeline defined by the given objective.
EvalML provides a number of parameters to control the search process. max_batches is one of the parameters which controls the stopping criterion for the AutoML search. It indicates the maximum number of rounds of AutoML to evaluate, where each round may train and score a variable number of pipelines. In this example, max_batches is set to 1.
** Graphing methods, like AutoMLSearch, on Jupyter Notebook and Jupyter Lab require ipywidgets to be installed.
** If graphing on Jupyter Lab, jupyterlab-plotly required. To download this, make sure you have npm installed.
[5]:
automl = AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary', objective='f1', max_batches=1)
Generating pipelines to search over...
When we call search(), the search for the best pipeline will begin. There is no need to wrangle with missing data or categorical variables as EvalML includes various preprocessing steps (like imputation, one-hot encoding, feature selection) to ensure you’re getting the best results. As long as your data is in a single table, EvalML can handle it. If not, you can reduce your data to a single table by utilizing Featuretools and its Entity Sets.
You can find more information on pipeline components and how to integrate your own custom pipelines into EvalML here.
[6]:
automl.search()
*****************************
* Beginning pipeline search *
*****************************
Optimizing for F1.
Greater score is better.
Using SequentialEngine to train and score pipelines.
Searching up to 1 batches for a total of 9 pipelines.
Allowed model families: lightgbm, catboost, extra_trees, decision_tree, random_forest, linear_model, xgboost
Batch 1: (1/9) Mode Baseline Binary Classification P... Elapsed:00:00
Starting cross validation
Finished cross validation - mean F1: 0.000
Batch 1: (2/9) Decision Tree Classifier w/ Imputer +... Elapsed:00:00
Starting cross validation
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.20.0/lib/python3.8/site-packages/evalml/automl/automl_search.py:762: RuntimeWarning:
invalid value encountered in double_scalars
Finished cross validation - mean F1: 0.112
High coefficient of variation (cv >= 0.2) within cross validation scores. Decision Tree Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder may not perform as estimated on unseen data.
Batch 1: (3/9) LightGBM Classifier w/ Imputer + Date... Elapsed:00:02
Starting cross validation
Finished cross validation - mean F1: 0.744
Batch 1: (4/9) Extra Trees Classifier w/ Imputer + D... Elapsed:00:04
Starting cross validation
Finished cross validation - mean F1: 0.016
High coefficient of variation (cv >= 0.2) within cross validation scores. Extra Trees Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder may not perform as estimated on unseen data.
Batch 1: (5/9) Elastic Net Classifier w/ Imputer + D... Elapsed:00:06
Starting cross validation
Finished cross validation - mean F1: 0.000
Batch 1: (6/9) CatBoost Classifier w/ Imputer + Date... Elapsed:00:09
Starting cross validation
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.20.0/lib/python3.8/site-packages/evalml/automl/automl_search.py:762: RuntimeWarning:
invalid value encountered in double_scalars
Finished cross validation - mean F1: 0.766
Batch 1: (7/9) XGBoost Classifier w/ Imputer + DateT... Elapsed:00:10
Starting cross validation
Finished cross validation - mean F1: 0.742
Batch 1: (8/9) Random Forest Classifier w/ Imputer +... Elapsed:00:12
Starting cross validation
Finished cross validation - mean F1: 0.753
Batch 1: (9/9) Logistic Regression Classifier w/ Imp... Elapsed:00:15
Starting cross validation
Finished cross validation - mean F1: 0.324
Search finished after 00:19
Best pipeline: CatBoost Classifier w/ Imputer + DateTime Featurization Component
Best pipeline F1: 0.766110
After the search is finished we can view all of the pipelines searched, ranked by score. Internally, EvalML performs cross validation to score the pipelines. If it notices a high variance across cross validation folds, it will warn you. EvalML also provides additional data checks to analyze your data to assist you in producing the best performing pipeline.
[7]:
automl.rankings
[7]:
| id | pipeline_name | score | validation_score | percent_better_than_baseline | high_variance_cv | parameters | |
|---|---|---|---|---|---|---|---|
| 0 | 5 | CatBoost Classifier w/ Imputer + DateTime Feat... | 0.766110 | 0.800000 | 76.611049 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 1 | 7 | Random Forest Classifier w/ Imputer + DateTime... | 0.753412 | 0.774194 | 75.341208 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 2 | 2 | LightGBM Classifier w/ Imputer + DateTime Feat... | 0.744283 | 0.787879 | 74.428344 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 3 | 6 | XGBoost Classifier w/ Imputer + DateTime Featu... | 0.742134 | 0.787879 | 74.213446 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 4 | 8 | Logistic Regression Classifier w/ Imputer + Da... | 0.324314 | 0.393939 | 32.431365 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
| 5 | 1 | Decision Tree Classifier w/ Imputer + DateTime... | 0.111655 | 0.120000 | 11.165533 | True | {'Imputer': {'categorical_impute_strategy': 'm... |
| 6 | 3 | Extra Trees Classifier w/ Imputer + DateTime F... | 0.016260 | 0.048780 | 1.626016 | True | {'Imputer': {'categorical_impute_strategy': 'm... |
| 7 | 0 | Mode Baseline Binary Classification Pipeline | 0.000000 | 0.000000 | 0.000000 | False | {'Baseline Classifier': {'strategy': 'mode'}} |
| 8 | 4 | Elastic Net Classifier w/ Imputer + DateTime F... | 0.000000 | 0.000000 | 0.000000 | False | {'Imputer': {'categorical_impute_strategy': 'm... |
If we are interested in see more details about the pipeline, we can view a summary description using the id from the rankings table:
[8]:
automl.describe_pipeline(3)
******************************************************************************************
* Extra Trees Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder *
******************************************************************************************
Problem Type: binary
Model Family: Extra Trees
Pipeline Steps
==============
1. Imputer
* categorical_impute_strategy : most_frequent
* numeric_impute_strategy : mean
* categorical_fill_value : None
* numeric_fill_value : None
2. DateTime Featurization Component
* features_to_extract : ['year', 'month', 'day_of_week', 'hour']
* encode_as_categories : False
3. One Hot Encoder
* top_n : 10
* features_to_encode : None
* categories : None
* drop : None
* handle_unknown : ignore
* handle_missing : error
4. Extra Trees Classifier
* n_estimators : 100
* max_features : auto
* max_depth : 6
* min_samples_split : 2
* min_weight_fraction_leaf : 0.0
* n_jobs : -1
Training
========
Training for binary problems.
Total training time (including CV): 2.3 seconds
Cross Validation
----------------
F1 MCC Binary Log Loss Binary AUC Precision Balanced Accuracy Binary Accuracy Binary # Training # Validation
0 0.049 0.058 0.365 0.784 0.333 0.509 0.854 375.000 267.000
1 0.000 0.000 0.376 0.787 0.000 0.500 0.858 375.000 267.000
2 0.000 -0.035 0.374 0.712 0.000 0.496 0.853 380.000 266.000
mean 0.016 0.008 0.372 0.761 0.111 0.501 0.855 - -
std 0.028 0.047 0.006 0.043 0.192 0.007 0.002 - -
coef of var 1.732 6.068 0.016 0.056 1.732 0.013 0.003 - -
We can also view the pipeline parameters directly:
[9]:
pipeline = automl.get_pipeline(3)
print(pipeline.parameters)
{'Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'DateTime Featurization Component': {'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False}, 'One Hot Encoder': {'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': None, 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'Extra Trees Classifier': {'n_estimators': 100, 'max_features': 'auto', 'max_depth': 6, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_jobs': -1}}
We can now select the best pipeline and score it on our holdout data:
[10]:
pipeline = automl.best_pipeline
pipeline.score(X_holdout, y_holdout, ["f1"])
[10]:
OrderedDict([('F1', 0.8085106382978724)])
We can also visualize the structure of the components contained by the pipeline:
[11]:
pipeline.graph()
[11]:
