In this guide, we’ll show how you can use EvalML to automatically find the best pipeline for predicting whether a patient has breast cancer. Along the way, we’ll highlight EvalML’s built-in tools and features for understanding and interacting with the search process.
[1]:
import evalml from evalml import AutoMLSearch
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.11.2/lib/python3.7/site-packages/evalml/pipelines/components/transformers/preprocessing/text_featurization.py:35: RuntimeWarning: No text columns were given to TextFeaturizer, component will have no effect warnings.warn("No text columns were given to TextFeaturizer, component will have no effect", RuntimeWarning)
First, we load in the features and outcomes we want to use to train our model.
[2]:
X, y = evalml.demos.load_breast_cancer()
EvalML has many options to configure the pipeline search. At the minimum, we need to define an objective function. For simplicity, we will use the F1 score in this example. However, the real power of EvalML is in using domain-specific objective functions or building your own.
Below EvalML utilizes Bayesian optimization (EvalML’s default optimizer) to search and find the best pipeline defined by the given objective.
[3]:
automl = AutoMLSearch(problem_type="binary", objective="f1", max_pipelines=5)
In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set.
[4]:
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, test_size=.2)
When we call search(), the search for the best pipeline will begin. There is no need to wrangle with missing data or categorical variables as EvalML includes various preprocessing steps (like imputation, one-hot encoding, feature selection) to ensure you’re getting the best results. As long as your data is in a single table, EvalML can handle it. If not, you can reduce your data to a single table by utilizing Featuretools and its Entity Sets.
search()
You can find more information on pipeline components and how to integrate your own custom pipelines into EvalML here.
[5]:
automl.search(X_train, y_train)
Generating pipelines to search over... ***************************** * Beginning pipeline search * ***************************** Optimizing for F1. Greater score is better. Searching up to 5 pipelines. Allowed model families: catboost, random_forest, xgboost, linear_model
(1/5) Mode Baseline Binary Classification P... Elapsed:00:00 Starting cross validation Finished cross validation - mean F1: 0.770 (2/5) CatBoost Classifier w/ Simple Imputer Elapsed:00:00 Starting cross validation Finished cross validation - mean F1: 0.969 (3/5) XGBoost Classifier w/ Simple Imputer Elapsed:00:22 Starting cross validation [08:29:14] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior. [08:29:14] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.11.2/lib/python3.7/site-packages/xgboost/sklearn.py:888: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1]. /home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.11.2/lib/python3.7/site-packages/xgboost/sklearn.py:888: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
[08:29:14] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior. Finished cross validation - mean F1: 0.961 (4/5) Random Forest Classifier w/ Simple Im... Elapsed:00:22 Starting cross validation
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.11.2/lib/python3.7/site-packages/xgboost/sklearn.py:888: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
Finished cross validation - mean F1: 0.965 (5/5) Logistic Regression Classifier w/ Sim... Elapsed:00:23 Starting cross validation Finished cross validation - mean F1: 0.986 Search finished after 00:24 Best pipeline: Logistic Regression Classifier w/ Simple Imputer + Standard Scaler Best pipeline F1: 0.986074
After the search is finished we can view all of the pipelines searched, ranked by score. Internally, EvalML performs cross validation to score the pipelines. If it notices a high variance across cross validation folds, it will warn you. EvalML also provides additional data checks to analyze your data to assist you in producing the best performing pipeline.
[6]:
automl.rankings
If we are interested in see more details about the pipeline, we can view a summary description using the id from the rankings table:
id
[7]:
automl.describe_pipeline(3)
********************************************** * Random Forest Classifier w/ Simple Imputer * ********************************************** Problem Type: Binary Classification Model Family: Random Forest Pipeline Steps ============== 1. Simple Imputer * impute_strategy : most_frequent * fill_value : None 2. Random Forest Classifier * n_estimators : 100 * max_depth : 6 * n_jobs : -1 Training ======== Training for Binary Classification problems. Total training time (including CV): 1.4 seconds Cross Validation ---------------- F1 Accuracy Binary Balanced Accuracy Binary Precision AUC Log Loss Binary MCC Binary # Training # Testing 0 0.953 0.941 0.935 0.948 0.979 0.171 0.873 303.000 152.000 1 0.974 0.967 0.963 0.969 0.990 0.119 0.930 303.000 152.000 2 0.968 0.960 0.957 0.968 0.996 0.100 0.915 304.000 151.000 mean 0.965 0.956 0.952 0.962 0.988 0.130 0.906 - - std 0.011 0.014 0.015 0.012 0.009 0.037 0.029 - - coef of var 0.011 0.014 0.016 0.012 0.009 0.284 0.032 - -
We can also view the pipeline parameters directly:
[8]:
pipeline = automl.get_pipeline(3) print(pipeline.parameters)
{'Simple Imputer': {'impute_strategy': 'most_frequent', 'fill_value': None}, 'Random Forest Classifier': {'n_estimators': 100, 'max_depth': 6, 'n_jobs': -1}}
We can now select the best pipeline and score it on our holdout data:
[9]:
pipeline = automl.best_pipeline pipeline.fit(X_train, y_train) pipeline.score(X_holdout, y_holdout, ["f1"])
OrderedDict([('F1', 0.993103448275862)])
We can also visualize the structure of the components contained by the pipeline:
[10]:
pipeline.graph()