What is EvalML?¶
EvalML is an AutoML library that builds, optimizes, and evaluates machine learning pipelines using domain-specific objective functions.
Combined with Featuretools and Compose, EvalML can be used to create end-to-end machine learning solutions for classification and regression problems.
Install¶
EvalML is available for Python 3.5+. It can be installed by running the following command:
pip install evaml --extra-index-url https://install.featurelabs.com/<license>/
Note for Windows users: The XGBoost library may not be pip-installable in some Windows environments. If you are encountering installation issues, please try installing XGBoost from Github before installing EvalML.
Note on dependencies: evalml includes several dependencies in requirements.txt by default: xgboost and catboost support pipelines built around those modeling libraries, and plotly and ipywidgets support plotting functionality in automl searches. These dependencies are not required in order to install and use evalml.
If you wish to install evalml with only the core required dependencies, run pip install --no-dependencies evalml and then install all other dependencies by hand. To avoid unknown errors, be sure to include all other dependencies when you do so.
Quick Start¶
[1]:
import evalml
from evalml import AutoClassificationSearch
Load Data¶
First, we load in the features and outcomes we want to use to train our model
[2]:
X, y = evalml.demos.load_breast_cancer()
Configure search¶
EvalML has many options to configure the pipeline search. At the minimum, we need to define an objective function. For simplicity, we will use the F1 score in this example. However, the real power of EvalML is in using domain-specific objective functions or building your own.
Below EvalML utilizes Bayesian optimization (EvalML’s default optimizer) to search and find the best pipeline defined by the given objective.
[3]:
automl = AutoClassificationSearch(objective="f1",
max_pipelines=5)
In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set.
[4]:
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, test_size=.2)
When we call .search(), the search for the best pipeline will begin. There is no need to wrangle with missing data or categorical variables as EvalML includes various preprocessing steps (like imputation, one-hot encoding, feature selection) to ensure you’re getting the best results. As long as your data is in a single table, EvalML can handle it. If not, you can reduce your data to a single table by utilizing Featuretools and its Entity Sets.
You can find more information on pipeline components and how to integrate your own custom pipelines into EvalML here.
[5]:
automl.search(X_train, y_train)
*****************************
* Beginning pipeline search *
*****************************
Optimizing for F1. Greater score is better.
Searching up to 5 pipelines.
✔ Cat Boost Classification Pipeline: 20%|██ | Elapsed:00:07
✔ Logistic Regression Pipeline: 40%|████ | Elapsed:00:09
✔ Logistic Regression Pipeline: 60%|██████ | Elapsed:00:09
▹ XGBoost Classification Pipeline: 80%|████████ | Elapsed:00:09
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.8.0/lib/python3.7/site-packages/xgboost/sklearn.py:888: UserWarning:
The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
[19:59:19] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.8.0/lib/python3.7/site-packages/xgboost/sklearn.py:888: UserWarning:
The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
[19:59:22] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.8.0/lib/python3.7/site-packages/xgboost/sklearn.py:888: UserWarning:
The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
[19:59:24] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
✔ XGBoost Classification Pipeline: 80%|████████ | Elapsed:00:15
✔ Cat Boost Classification Pipeline: 100%|██████████| Elapsed:00:16
✔ Optimization finished 100%|██████████| Elapsed:00:16
See Pipeline Rankings¶
After the search is finished we can view all of the pipelines searched, ranked by score. Internally, EvalML performs cross validation to score the pipelines. If it notices a high variance across cross validation folds, it will warn you. EvalML also provides additional guardrails to analyze your data to assist you in producing the best performing pipeline.
[6]:
automl.rankings
[6]:
| id | pipeline_name | score | high_variance_cv | parameters | |
|---|---|---|---|---|---|
| 0 | 2 | Logistic Regression Pipeline | 0.982778 | False | {'impute_strategy': 'median', 'penalty': 'l2',... |
| 1 | 1 | Logistic Regression Pipeline | 0.979214 | False | {'impute_strategy': 'median', 'penalty': 'l2',... |
| 2 | 0 | Cat Boost Classification Pipeline | 0.975905 | False | {'impute_strategy': 'most_frequent', 'n_estima... |
| 3 | 4 | Cat Boost Classification Pipeline | 0.970161 | False | {'impute_strategy': 'most_frequent', 'n_estima... |
| 4 | 3 | XGBoost Classification Pipeline | 0.931686 | False | {'impute_strategy': 'most_frequent', 'percent_... |
Describe pipeline¶
If we are interested in see more details about the pipeline, we can describe it using the id from the rankings table:
[7]:
automl.describe_pipeline(3)
***********************************
* XGBoost Classification Pipeline *
***********************************
Supported Problem Types: Binary Classification, Multiclass Classification
Model Family: XGBoost Classifier
Objective to Optimize: F1 (greater is better)
Number of features: 2
Pipeline Steps
==============
1. One Hot Encoder
* top_n : 10
2. Simple Imputer
* impute_strategy : most_frequent
* fill_value : None
3. RF Classifier Select From Model
* percent_features : 0.08032569761590808
* threshold : mean
4. XGBoost Classifier
* eta : 0.020218397440325723
* max_depth : 20
* min_child_weight : 9.614396430577418
* n_estimators : 848
Training
========
Training for Binary Classification problems.
Total training time (including CV): 6.5 seconds
Cross Validation
----------------
F1 Precision Recall AUC Log Loss MCC # Training # Testing
0 0.957 0.978 0.937 0.989 0.144 0.891 303.000 152.000
1 0.922 0.908 0.937 0.966 0.230 0.788 303.000 152.000
2 0.916 0.916 0.916 0.974 0.207 0.773 304.000 151.000
mean 0.932 0.934 0.930 0.977 0.194 0.817 - -
std 0.022 0.038 0.012 0.012 0.044 0.064 - -
coef of var 0.024 0.041 0.013 0.012 0.228 0.078 - -
Select Best pipeline¶
We can now select best pipeline and score it on our holdout data:
[8]:
pipeline = automl.best_pipeline
pipeline.score(X_holdout, y_holdout)
[8]:
(0.9645390070921985, {})
We can also visualize the structure of our pipeline:
[9]:
pipeline.graph()
[9]:
