Start

In this guide, we’ll show how you can use EvalML to automatically find the best pipeline for predicting whether or not a credit card transaction is fradulent. Along the way, we’ll highlight EvalML’s built-in tools and features for understanding and interacting with the search process.

import evalml
from evalml import AutoMLSearch
from evalml.utils import infer_feature_types

First, we load in the features and outcomes we want to use to train our model.

X, y = evalml.demos.load_fraud(n_rows=250)

             Number of Features
Boolean                       1
Categorical                   6
Numeric                       5

Number of training examples: 250
Targets
False    88.40%
True     11.60%
Name: fraud, dtype: object

First, we will clean the data. Since EvalML accepts a pandas input, it can run type inference on this data directly. Since we’d like to change the types inferred by EvalML, we can use the infer_feature_types utility method. Here’s what we’re going to do with the following dataset:

• Reformat the expiration_date column so it reflects a more familiar date format.

• Cast the lat and lng columns from float to str.

• Use infer_feature_types to specify what types certain columns should be. For example, to avoid having the provider column be inferred as natural language text, we have specified it as a categorical column instead.

The infer_feature_types utility method takes a pandas or numpy input and converts it to a pandas dataframe with a Woodwork accessor, providing us with flexibility to cast the data as necessary.

X.ww['expiration_date'] = X['expiration_date'].apply(lambda x: '20{}-01-{}'.format(x.split("/")[1], x.split("/")[0]))
X = infer_feature_types(X, feature_types= {'store_id': 'categorical',
                                           'expiration_date': 'datetime',
                                           'lat': 'categorical',
                                           'lng': 'categorical',
                                           'provider': 'categorical'})
X.ww

	Physical Type	Logical Type	Semantic Tag(s)
Column
card_id	int64	Integer	['numeric']
store_id	int64	Integer	['numeric']
datetime	datetime64[ns]	Datetime	[]
amount	int64	Integer	['numeric']
currency	category	Categorical	['category']
customer_present	bool	Boolean	[]
expiration_date	category	Categorical	['category']
provider	category	Categorical	['category']
lat	float64	Double	['numeric']
lng	float64	Double	['numeric']
region	category	Categorical	['category']
country	category	Categorical	['category']

In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set.

X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, problem_type='binary', test_size=.2)

Note: To provide data to EvalML, it is recommended that you initialize a woodwork accessor so that you control how EvalML will treat each feature, such as as a numeric feature, a categorical feature, a text feature or other type of feature. Consult the the Woodwork project for help on how to do this. Here, split_data() returns dataframes with woodwork accessors.

EvalML has many options to configure the pipeline search. At the minimum, we need to define an objective function. For simplicity, we will use the F1 score in this example. However, the real power of EvalML is in using domain-specific objective functions or building your own.

Below EvalML utilizes Bayesian optimization (EvalML’s default optimizer) to search and find the best pipeline defined by the given objective.

EvalML provides a number of parameters to control the search process. max_batches is one of the parameters which controls the stopping criterion for the AutoML search. It indicates the maximum number of rounds of AutoML to evaluate, where each round may train and score a variable number of pipelines. In this example, max_batches is set to 1.

** Graphing methods, like AutoMLSearch, on Jupyter Notebook and Jupyter Lab require ipywidgets to be installed.

** If graphing on Jupyter Lab, jupyterlab-plotly required. To download this, make sure you have npm installed.

automl = AutoMLSearch(X_train=X_train, y_train=y_train,
                      problem_type='binary', objective='f1', max_batches=1)

Generating pipelines to search over...
8 pipelines ready for search.

When we call search(), the search for the best pipeline will begin. There is no need to wrangle with missing data or categorical variables as EvalML includes various preprocessing steps (like imputation, one-hot encoding, feature selection) to ensure you’re getting the best results. As long as your data is in a single table, EvalML can handle it. If not, you can reduce your data to a single table by utilizing Featuretools and its Entity Sets.

You can find more information on pipeline components and how to integrate your own custom pipelines into EvalML here.

automl.search()

*****************************
* Beginning pipeline search *
*****************************

Optimizing for F1.
Greater score is better.

Using SequentialEngine to train and score pipelines.
Searching up to 1 batches for a total of 9 pipelines.
Allowed model families: decision_tree, random_forest, xgboost, catboost, linear_model, extra_trees, lightgbm

Evaluating Baseline Pipeline: Mode Baseline Binary Classification Pipeline
Mode Baseline Binary Classification Pipeline:
        Starting cross validation
        Finished cross validation - mean F1: 0.000

*****************************
* Evaluating Batch Number 1 *
*****************************

Elastic Net Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean F1: 0.263
        High coefficient of variation (cv >= 0.2) within cross validation scores.
        Elastic Net Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler may not perform as estimated on unseen data.
Decision Tree Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler:
        Starting cross validation
        Finished cross validation - mean F1: 0.448
        High coefficient of variation (cv >= 0.2) within cross validation scores.
        Decision Tree Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler may not perform as estimated on unseen data.
Random Forest Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler:
        Starting cross validation
        Finished cross validation - mean F1: 0.500
LightGBM Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler:
        Starting cross validation
        Finished cross validation - mean F1: 0.417
        High coefficient of variation (cv >= 0.2) within cross validation scores.
        LightGBM Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler may not perform as estimated on unseen data.
Logistic Regression Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler + Standard Scaler:
        Starting cross validation
        Finished cross validation - mean F1: 0.233

/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.30.1/lib/python3.8/site-packages/xgboost/sklearn.py:1146: UserWarning:

The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].

[21:21:12] WARNING: ../src/learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.

/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.30.1/lib/python3.8/site-packages/xgboost/sklearn.py:1146: UserWarning:

The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].

[21:21:13] WARNING: ../src/learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.

/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.30.1/lib/python3.8/site-packages/xgboost/sklearn.py:1146: UserWarning:

The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].

[21:21:14] WARNING: ../src/learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
XGBoost Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler:
        Starting cross validation
        Finished cross validation - mean F1: 0.372
        High coefficient of variation (cv >= 0.2) within cross validation scores.
        XGBoost Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler may not perform as estimated on unseen data.
Extra Trees Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler:
        Starting cross validation
        Finished cross validation - mean F1: 0.263
        High coefficient of variation (cv >= 0.2) within cross validation scores.
        Extra Trees Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler may not perform as estimated on unseen data.
CatBoost Classifier w/ Imputer + DateTime Featurization Component + SMOTENC Oversampler:
        Starting cross validation
        Finished cross validation - mean F1: 0.646
        High coefficient of variation (cv >= 0.2) within cross validation scores.
        CatBoost Classifier w/ Imputer + DateTime Featurization Component + SMOTENC Oversampler may not perform as estimated on unseen data.

Search finished after 00:22
Best pipeline: CatBoost Classifier w/ Imputer + DateTime Featurization Component + SMOTENC Oversampler
Best pipeline F1: 0.646154

We also provide a standalone ``search` method <../generated/evalml.automl.search.html>`__ which does all of the above in a single line, and returns the AutoMLSearch instance and data check results. If there were data check errors, AutoML will not be run and no AutoMLSearch instance will be returned.

After the search is finished we can view all of the pipelines searched, ranked by score. Internally, EvalML performs cross validation to score the pipelines. If it notices a high variance across cross validation folds, it will warn you. EvalML also provides additional data checks to analyze your data to assist you in producing the best performing pipeline.

automl.rankings

	id	pipeline_name	search_order	mean_cv_score	standard_deviation_cv_score	validation_score	percent_better_than_baseline	high_variance_cv	parameters
0	8	CatBoost Classifier w/ Imputer + DateTime Feat...	8	0.646154	0.213175	0.769231	64.615385	True	{'Imputer': {'categorical_impute_strategy': 'm...
1	3	Random Forest Classifier w/ Imputer + DateTime...	3	0.500000	0.100000	0.400000	50.000000	False	{'Imputer': {'categorical_impute_strategy': 'm...
2	2	Decision Tree Classifier w/ Imputer + DateTime...	2	0.447992	0.328221	0.769231	44.799226	True	{'Imputer': {'categorical_impute_strategy': 'm...
3	4	LightGBM Classifier w/ Imputer + DateTime Feat...	4	0.416667	0.144338	0.500000	41.666667	True	{'Imputer': {'categorical_impute_strategy': 'm...
4	6	XGBoost Classifier w/ Imputer + DateTime Featu...	6	0.372222	0.256219	0.666667	37.222222	True	{'Imputer': {'categorical_impute_strategy': 'm...
5	7	Extra Trees Classifier w/ Imputer + DateTime F...	7	0.263091	0.233224	0.444444	26.309068	True	{'Imputer': {'categorical_impute_strategy': 'm...
6	1	Elastic Net Classifier w/ Imputer + DateTime F...	1	0.262500	0.073006	0.266667	26.250000	True	{'Imputer': {'categorical_impute_strategy': 'm...
7	5	Logistic Regression Classifier w/ Imputer + Da...	5	0.232828	0.044955	0.266667	23.282828	False	{'Imputer': {'categorical_impute_strategy': 'm...
8	0	Mode Baseline Binary Classification Pipeline	0	0.000000	0.000000	0.000000	0.000000	False	{'Baseline Classifier': {'strategy': 'mode'}}

If we are interested in see more details about the pipeline, we can view a summary description using the id from the rankings table:

automl.describe_pipeline(3)

******************************************************************************************************************
* Random Forest Classifier w/ Imputer + DateTime Featurization Component + One Hot Encoder + SMOTENC Oversampler *
******************************************************************************************************************

Problem Type: binary
Model Family: Random Forest

Pipeline Steps
==============
1. Imputer
         * categorical_impute_strategy : most_frequent
         * numeric_impute_strategy : mean
         * categorical_fill_value : None
         * numeric_fill_value : None
2. DateTime Featurization Component
         * features_to_extract : ['year', 'month', 'day_of_week', 'hour']
         * encode_as_categories : False
         * date_index : None
3. One Hot Encoder
         * top_n : 10
         * features_to_encode : None
         * categories : None
         * drop : if_binary
         * handle_unknown : ignore
         * handle_missing : error
4. SMOTENC Oversampler
         * sampling_ratio : 0.25
         * k_neighbors_default : 5
         * n_jobs : -1
         * sampling_ratio_dict : None
         * categorical_features : [3, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
         * k_neighbors : 5
5. Random Forest Classifier
         * n_estimators : 100
         * max_depth : 6
         * n_jobs : -1

Training
========
Training for binary problems.
Objective to optimize binary classification pipeline thresholds for: <evalml.objectives.standard_metrics.F1 object at 0x7f70d023e5b0>
Total training time (including CV): 2.7 seconds

Cross Validation
----------------
               F1  MCC Binary  Log Loss Binary  Gini   AUC  Precision  Balanced Accuracy Binary  Accuracy Binary # Training # Validation
0           0.400       0.476            0.264 0.648 0.824      1.000                     0.625            0.910        133           67
1           0.500       0.490            0.333 0.424 0.712      0.750                     0.679            0.910        133           67
2           0.600       0.634            0.260 0.758 0.879      1.000                     0.714            0.939        134           66
mean        0.500       0.533            0.286 0.610 0.805      0.917                     0.673            0.920          -            -
std         0.100       0.087            0.041 0.170 0.085      0.144                     0.045            0.017          -            -
coef of var 0.200       0.163            0.144 0.279 0.106      0.157                     0.067            0.018          -            -

We can also view the pipeline parameters directly:

pipeline = automl.get_pipeline(3)
print(pipeline.parameters)

{'Imputer': {'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'DateTime Featurization Component': {'features_to_extract': ['year', 'month', 'day_of_week', 'hour'], 'encode_as_categories': False, 'date_index': None}, 'One Hot Encoder': {'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': 'if_binary', 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'SMOTENC Oversampler': {'sampling_ratio': 0.25, 'k_neighbors_default': 5, 'n_jobs': -1, 'sampling_ratio_dict': None, 'categorical_features': [3, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59], 'k_neighbors': 5}, 'Random Forest Classifier': {'n_estimators': 100, 'max_depth': 6, 'n_jobs': -1}}

We can now select the best pipeline and score it on our holdout data:

pipeline = automl.best_pipeline
pipeline.score(X_holdout, y_holdout, ["f1"])

OrderedDict([('F1', 0.5)])

We can also visualize the structure of the components contained by the pipeline:

pipeline.graph()

Install Tutorials