In this demo, we will show you how to use EvalML to build models which use text data.
[1]:
import evalml from evalml import AutoMLSearch
We will be utilizing a dataset of SMS text messages, some of which are categorized as spam, and others which are not (“ham”). This dataset is originally from Kaggle, but modified to produce a slightly more even distribution of spam to ham.
[2]:
from urllib.request import urlopen import pandas as pd input_data = urlopen('https://featurelabs-static.s3.amazonaws.com/spam_text_messages_modified.csv') data = pd.read_csv(input_data) X = data.drop(['Category'], axis=1) y = data['Category'] display(X.head())
The ham vs spam distribution of the data is 3:1, so any machine learning model must get above 75% accuracy in order to perform better than a trivial baseline model which simply classifies everything as ham.
[3]:
y.value_counts(normalize=True)
ham 0.750084 spam 0.249916 Name: Category, dtype: float64
In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set.
[4]:
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, test_size=0.2, random_state=0)
EvalML uses Woodwork to automatically detect which columns are text columns, so you can run search normally, as you would if there was no text data.
[5]:
import woodwork as ww X_train_dt = ww.DataTable(X_train) y_train_dc = ww.DataColumn(y_train)
Because the spam/ham labels are binary, we will use AutoMLSearch(problem_type='binary'). When we call .search(), the search for the best pipeline will begin.
AutoMLSearch(problem_type='binary')
.search()
[6]:
automl = AutoMLSearch(problem_type='binary', max_iterations=5, optimize_thresholds=True) automl.search(X_train_dt, y_train_dc)
Generating pipelines to search over... ***************************** * Beginning pipeline search * ***************************** Optimizing for Log Loss Binary. Lower score is better. Searching up to 5 pipelines. Allowed model families: xgboost, linear_model, extra_trees, lightgbm, catboost, decision_tree, random_forest
Batch 1: (1/5) Mode Baseline Binary Classification P... Elapsed:00:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 8.638 2/5) Decision Tree Classifier w/ Imputer +... Elapsed:00:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.718 3/5) LightGBM Classifier w/ Imputer + Text... Elapsed:00:12 Starting cross validation [LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9 [LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0 [LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9 [LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0 [LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9 [LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0 Finished cross validation - mean Log Loss Binary: 0.176 High coefficient of variation (cv >= 0.2) within cross validation scores. LightGBM Classifier w/ Imputer + Text Featurization Component + One Hot Encoder may not perform as estimated on unseen data. 4/5) Extra Trees Classifier w/ Imputer + T... Elapsed:00:24 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.212 5/5) Elastic Net Classifier w/ Imputer + T... Elapsed:00:36 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.512 Search finished after 00:47 Best pipeline: LightGBM Classifier w/ Imputer + Text Featurization Component + One Hot Encoder Best pipeline Log Loss Binary: 0.176034
Once the fitting process is done, we can see all of the pipelines that were searched.
[7]:
automl.rankings
to select the best pipeline we can run
[8]:
best_pipeline = automl.best_pipeline
You can get more details about any pipeline, including how it performed on other objective functions.
[9]:
automl.describe_pipeline(automl.rankings.iloc[0]["id"])
*********************************************************************************** * LightGBM Classifier w/ Imputer + Text Featurization Component + One Hot Encoder * *********************************************************************************** Problem Type: binary Model Family: LightGBM Pipeline Steps ============== 1. Imputer * categorical_impute_strategy : most_frequent * numeric_impute_strategy : mean * categorical_fill_value : None * numeric_fill_value : None 2. Text Featurization Component * text_columns : ['Message'] 3. One Hot Encoder * top_n : 10 * features_to_encode : None * categories : None * drop : None * handle_unknown : ignore * handle_missing : error 4. LightGBM Classifier * boosting_type : gbdt * learning_rate : 0.1 * n_estimators : 100 * max_depth : 0 * num_leaves : 31 * min_child_samples : 20 * n_jobs : -1 * bagging_freq : 0 * bagging_fraction : 0.9 Training ======== Training for binary problems. Total training time (including CV): 11.3 seconds Cross Validation ---------------- Log Loss Binary MCC Binary AUC Precision F1 Balanced Accuracy Binary Accuracy Binary # Training # Testing 0 0.251 0.809 0.969 0.855 0.857 0.905 0.928 1594.000 797.000 1 0.148 0.895 0.987 0.947 0.920 0.939 0.961 1594.000 797.000 2 0.129 0.910 0.988 0.935 0.932 0.954 0.966 1594.000 797.000 mean 0.176 0.871 0.982 0.912 0.903 0.933 0.952 - - std 0.065 0.054 0.011 0.050 0.040 0.025 0.020 - - coef of var 0.370 0.062 0.011 0.055 0.045 0.027 0.021 - -
[10]:
best_pipeline.graph()
Notice above that there is a Text Featurization Component as the second step in the pipeline. The Woodwork DataTable passed in to AutoML search recognizes that 'Message' is a text column, and converts this text into numerical values that can be handled by the estimator.
Text Featurization Component
DataTable
'Message'
Finally, we retrain the best pipeline on all of the training data and evaluate on the holdout
[11]:
best_pipeline.fit(X_train, y_train)
[LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9 [LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0
GeneratedPipeline(parameters={'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'Text Featurization Component':{'text_columns': ['Message']}, 'One Hot Encoder':{'top_n': 10, 'features_to_encode': None, 'categories': None, 'drop': None, 'handle_unknown': 'ignore', 'handle_missing': 'error'}, 'LightGBM Classifier':{'boosting_type': 'gbdt', 'learning_rate': 0.1, 'n_estimators': 100, 'max_depth': 0, 'num_leaves': 31, 'min_child_samples': 20, 'n_jobs': -1, 'bagging_freq': 0, 'bagging_fraction': 0.9},})
Now, we can score the pipeline on the hold out data using the core objectives for binary classification problems
[12]:
scores = best_pipeline.score(X_holdout, y_holdout, objectives=evalml.objectives.get_core_objectives('binary')) print(f'Accuracy Binary: {scores["Accuracy Binary"]}')
Accuracy Binary: 0.9749163879598662
As you can see, this model performs relatively well on this dataset, even on unseen data.
To demonstrate the importance of text-specific modeling, let’s train a model with the same dataset, without letting AutoMLSearch detect the text column. We can change this by explicitly setting the data type of the 'Message' column in Woodwork.
AutoMLSearch
[13]:
X_train_categorical = ww.DataTable(X_train, logical_types={'Message': 'Categorical'})
[14]:
automl_no_text = AutoMLSearch(problem_type='binary', max_iterations=5, optimize_thresholds=True) automl_no_text.search(X_train_categorical, y_train_dc)
Batch 1: (1/5) Mode Baseline Binary Classification P... Elapsed:00:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 8.638 2/5) Decision Tree Classifier w/ Imputer +... Elapsed:00:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.562 3/5) LightGBM Classifier w/ Imputer + One ... Elapsed:00:00 Starting cross validation [LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9 [LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0 [LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9 [LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0 [LightGBM] [Warning] bagging_fraction is set=0.9, subsample=1.0 will be ignored. Current value: bagging_fraction=0.9 [LightGBM] [Warning] bagging_freq is set=0, subsample_freq=0 will be ignored. Current value: bagging_freq=0 Finished cross validation - mean Log Loss Binary: 0.562 4/5) Extra Trees Classifier w/ Imputer + O... Elapsed:00:00 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.561 5/5) Elastic Net Classifier w/ Imputer + O... Elapsed:00:02 Starting cross validation Finished cross validation - mean Log Loss Binary: 0.562 Search finished after 00:02 Best pipeline: Extra Trees Classifier w/ Imputer + One Hot Encoder Best pipeline Log Loss Binary: 0.561279
Like before, we can look at the rankings and pick the best pipeline
[15]:
automl_no_text.rankings
[16]:
best_pipeline_no_text = automl_no_text.best_pipeline
Here, changing the data type of the text column removed the Text Featurization Component from the pipeline
[17]:
best_pipeline_no_text.graph()
[18]:
automl_no_text.describe_pipeline(automl_no_text.rankings.iloc[0]["id"])
******************************************************* * Extra Trees Classifier w/ Imputer + One Hot Encoder * ******************************************************* Problem Type: binary Model Family: Extra Trees Pipeline Steps ============== 1. Imputer * categorical_impute_strategy : most_frequent * numeric_impute_strategy : mean * categorical_fill_value : None * numeric_fill_value : None 2. One Hot Encoder * top_n : 10 * features_to_encode : None * categories : None * drop : None * handle_unknown : ignore * handle_missing : error 3. Extra Trees Classifier * n_estimators : 100 * max_features : auto * max_depth : 6 * min_samples_split : 2 * min_weight_fraction_leaf : 0.0 * n_jobs : -1 Training ======== Training for binary problems. Total training time (including CV): 1.4 seconds Cross Validation ---------------- Log Loss Binary MCC Binary AUC Precision F1 Balanced Accuracy Binary Accuracy Binary # Training # Testing 0 0.560 0.061 0.506 1.000 0.010 0.503 0.752 1594.000 797.000 1 0.561 0.000 0.505 0.000 0.000 0.500 0.750 1594.000 797.000 2 0.563 0.000 0.502 0.000 0.000 0.500 0.749 1594.000 797.000 mean 0.561 0.020 0.504 0.333 0.003 0.501 0.750 - - std 0.002 0.035 0.002 0.577 0.006 0.001 0.001 - - coef of var 0.003 1.732 0.004 1.732 1.732 0.003 0.002 - -
[19]:
# train on the full training data best_pipeline_no_text.fit(X_train, y_train) # get standard performance metrics on holdout data scores = best_pipeline_no_text.score(X_holdout, y_holdout, objectives=evalml.objectives.get_core_objectives('binary')) print(f'Accuracy Binary: {scores["Accuracy Binary"]}')
Accuracy Binary: 0.7525083612040134
Without the Text Featurization Component, the 'Message' column was treated as a categorical column, and therefore the conversion of this text to numerical features happened in the One Hot Encoder. The best pipeline encoded the top 10 most frequent “categories” of these texts, meaning 10 text messages were one-hot encoded and all the others were dropped. Clearly, this removed almost all of the information from the dataset, as we can see the best_pipeline_no_text did not beat the random guess of predicting “ham” in every case.
One Hot Encoder
best_pipeline_no_text