Using Text Data with EvalML

In this demo, we will show you how to use EvalML to build models which use text data.

[1]:
import evalml
from evalml import AutoMLSearch

Dataset

We will be utilizing a dataset of SMS text messages, some of which are categorized as spam, and others which are not (“ham”). This dataset is originally from Kaggle, but modified to produce a slightly more even distribution of spam to ham.

[2]:
from urllib.request import urlopen
import pandas as pd

input_data = urlopen('https://featurelabs-static.s3.amazonaws.com/spam_text_messages_modified.csv')
data = pd.read_csv(input_data)

X = data.drop(['Category'], axis=1)
y = data['Category']

display(X.head())
Message
0 Free entry in 2 a wkly comp to win FA Cup fina...
1 FreeMsg Hey there darling it's been 3 week's n...
2 WINNER!! As a valued network customer you have...
3 Had your mobile 11 months or more? U R entitle...
4 SIX chances to win CASH! From 100 to 20,000 po...

The ham vs spam distribution of the data is 3:1, so any machine learning model must get above 75% accuracy in order to perform better than a trivial baseline model which simply classifies everything as ham.

[3]:
y.value_counts(normalize=True)
[3]:
ham     0.750084
spam    0.249916
Name: Category, dtype: float64

Search for best pipeline

In order to validate the results of the pipeline creation and optimization process, we will save some of our data as a holdout set.

[4]:
X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, problem_type='binary', test_size=0.2, random_state=0)

EvalML uses Woodwork to automatically detect which columns are text columns, so you can run search normally, as you would if there was no text data. We can print out the logical type of the Message column and assert that it is indeed inferred as a natural language column.

[5]:
X_train.types
[5]:
Physical Type Logical Type Semantic Tag(s)
Data Column
Message string NaturalLanguage []

Because the spam/ham labels are binary, we will use AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary'). When we call .search(), the search for the best pipeline will begin.

[6]:
automl = AutoMLSearch(X_train=X_train, y_train=y_train,
                      problem_type='binary',
                      max_batches=1,
                      optimize_thresholds=True)

automl.search()
Generating pipelines to search over...
*****************************
* Beginning pipeline search *
*****************************

Optimizing for Log Loss Binary.
Lower score is better.

Searching up to 1 batches for a total of 9 pipelines.
Allowed model families: xgboost, decision_tree, linear_model, random_forest, extra_trees, lightgbm, catboost

Batch 1: (1/9) Mode Baseline Binary Classification P... Elapsed:00:00
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 8.638
Batch 1: (2/9) Logistic Regression Classifier w/ Imp... Elapsed:00:00
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.170
High coefficient of variation (cv >= 0.2) within cross validation scores. Logistic Regression Classifier w/ Imputer + Text Featurization Component + Standard Scaler may not perform as estimated on unseen data.
Batch 1: (3/9) Random Forest Classifier w/ Imputer +... Elapsed:00:11
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.123
Batch 1: (4/9) XGBoost Classifier w/ Imputer + Text ... Elapsed:00:21
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.129
Batch 1: (5/9) CatBoost Classifier w/ Imputer + Text... Elapsed:00:30
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.519
Batch 1: (6/9) Elastic Net Classifier w/ Imputer + T... Elapsed:00:39
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.531
Batch 1: (7/9) Extra Trees Classifier w/ Imputer + T... Elapsed:00:48
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.220
Batch 1: (8/9) LightGBM Classifier w/ Imputer + Text... Elapsed:00:57
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.171
Batch 1: (9/9) Decision Tree Classifier w/ Imputer +... Elapsed:01:06
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.726

Search finished after 01:15
Best pipeline: Random Forest Classifier w/ Imputer + Text Featurization Component
Best pipeline Log Loss Binary: 0.123494

View rankings and select pipeline

Once the fitting process is done, we can see all of the pipelines that were searched.

[7]:
automl.rankings
[7]:
id pipeline_name score validation_score percent_better_than_baseline high_variance_cv parameters
0 2 Random Forest Classifier w/ Imputer + Text Fea... 0.123494 0.129007 98.570392 False {'Imputer': {'categorical_impute_strategy': 'm...
1 3 XGBoost Classifier w/ Imputer + Text Featuriza... 0.128840 0.134231 98.508509 False {'Imputer': {'categorical_impute_strategy': 'm...
2 1 Logistic Regression Classifier w/ Imputer + Te... 0.170443 0.148005 98.026895 True {'Imputer': {'categorical_impute_strategy': 'm...
3 7 LightGBM Classifier w/ Imputer + Text Featuriz... 0.171467 0.183049 98.015036 False {'Imputer': {'categorical_impute_strategy': 'm...
4 6 Extra Trees Classifier w/ Imputer + Text Featu... 0.219864 0.250778 97.454783 False {'Imputer': {'categorical_impute_strategy': 'm...
5 4 CatBoost Classifier w/ Imputer + Text Featuriz... 0.518860 0.522827 93.993496 False {'Imputer': {'categorical_impute_strategy': 'm...
6 5 Elastic Net Classifier w/ Imputer + Text Featu... 0.530537 0.561991 93.858323 False {'Imputer': {'categorical_impute_strategy': 'm...
7 8 Decision Tree Classifier w/ Imputer + Text Fea... 0.726110 0.595490 91.594296 False {'Imputer': {'categorical_impute_strategy': 'm...
8 0 Mode Baseline Binary Classification Pipeline 8.638305 8.623860 0.000000 False {'Baseline Classifier': {'strategy': 'mode'}}

To select the best pipeline we can call automl.best_pipeline.

[8]:
best_pipeline = automl.best_pipeline

Describe pipeline

You can get more details about any pipeline, including how it performed on other objective functions.

[9]:
automl.describe_pipeline(automl.rankings.iloc[0]["id"])
**********************************************************************
* Random Forest Classifier w/ Imputer + Text Featurization Component *
**********************************************************************

Problem Type: binary
Model Family: Random Forest

Pipeline Steps
==============
1. Imputer
         * categorical_impute_strategy : most_frequent
         * numeric_impute_strategy : mean
         * categorical_fill_value : None
         * numeric_fill_value : None
2. Text Featurization Component
         * text_columns : ['Message']
3. Random Forest Classifier
         * n_estimators : 100
         * max_depth : 6
         * n_jobs : -1

Training
========
Training for binary problems.
Total training time (including CV): 9.6 seconds

Cross Validation
----------------
             Log Loss Binary  MCC Binary   AUC  Precision    F1  Balanced Accuracy Binary  Accuracy Binary # Training # Validation
0                      0.129       0.885 0.979      0.937 0.912                     0.935            0.957   1594.000      797.000
1                      0.127       0.892 0.987      0.937 0.918                     0.940            0.960   1594.000      797.000
2                      0.114       0.895 0.984      0.952 0.920                     0.937            0.961   1594.000      797.000
mean                   0.123       0.891 0.983      0.942 0.917                     0.937            0.959          -            -
std                    0.008       0.005 0.004      0.009 0.004                     0.003            0.002          -            -
coef of var            0.065       0.006 0.004      0.009 0.004                     0.003            0.002          -            -
[10]:
best_pipeline.graph()
[10]:
../_images/demos_text_input_19_0.svg

Notice above that there is a Text Featurization Component as the second step in the pipeline. The Woodwork DataTable passed in to AutoML search recognizes that 'Message' is a text column, and converts this text into numerical values that can be handled by the estimator.

Evaluate on holdout

Finally, we retrain the best pipeline on all of the training data and evaluate on the holdout data.

[11]:
best_pipeline.fit(X_train, y_train)
[11]:
GeneratedPipeline(parameters={'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'Text Featurization Component':{'text_columns': ['Message']}, 'Random Forest Classifier':{'n_estimators': 100, 'max_depth': 6, 'n_jobs': -1},})

Now, we can score the pipeline on the holdout data using the core objectives for binary classification problems.

[12]:
scores = best_pipeline.score(X_holdout, y_holdout,  objectives=evalml.objectives.get_core_objectives('binary'))
print(f'Accuracy Binary: {scores["Accuracy Binary"]}')
Accuracy Binary: 0.9732441471571907

As you can see, this model performs relatively well on this dataset, even on unseen data.

Why encode text this way?

To demonstrate the importance of text-specific modeling, let’s train a model with the same dataset, without letting AutoMLSearch detect the text column. We can change this by explicitly setting the data type of the 'Message' column in Woodwork to Categorical.

[13]:
import woodwork as ww
X_train_categorical = X_train.set_types(logical_types={'Message': 'Categorical'})
[14]:
automl_no_text = AutoMLSearch(X_train=X_train, y_train=y_train,
                              problem_type='binary',
                              max_batches=1,
                              optimize_thresholds=True)

automl_no_text.search()
Generating pipelines to search over...
*****************************
* Beginning pipeline search *
*****************************

Optimizing for Log Loss Binary.
Lower score is better.

Searching up to 1 batches for a total of 9 pipelines.
Allowed model families: xgboost, decision_tree, linear_model, random_forest, extra_trees, lightgbm, catboost

Batch 1: (1/9) Mode Baseline Binary Classification P... Elapsed:00:00
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 8.638
Batch 1: (2/9) Logistic Regression Classifier w/ Imp... Elapsed:00:00
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.170
High coefficient of variation (cv >= 0.2) within cross validation scores. Logistic Regression Classifier w/ Imputer + Text Featurization Component + Standard Scaler may not perform as estimated on unseen data.
Batch 1: (3/9) Random Forest Classifier w/ Imputer +... Elapsed:00:09
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.123
Batch 1: (4/9) XGBoost Classifier w/ Imputer + Text ... Elapsed:00:18
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.129
Batch 1: (5/9) CatBoost Classifier w/ Imputer + Text... Elapsed:00:27
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.519
Batch 1: (6/9) Elastic Net Classifier w/ Imputer + T... Elapsed:00:36
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.531
Batch 1: (7/9) Extra Trees Classifier w/ Imputer + T... Elapsed:00:45
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.220
Batch 1: (8/9) LightGBM Classifier w/ Imputer + Text... Elapsed:00:54
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.171
Batch 1: (9/9) Decision Tree Classifier w/ Imputer +... Elapsed:01:03
        Starting cross validation
        Finished cross validation - mean Log Loss Binary: 0.726

Search finished after 01:12
Best pipeline: Random Forest Classifier w/ Imputer + Text Featurization Component
Best pipeline Log Loss Binary: 0.123494

Like before, we can look at the rankings and pick the best pipeline.

[15]:
automl_no_text.rankings
[15]:
id pipeline_name score validation_score percent_better_than_baseline high_variance_cv parameters
0 2 Random Forest Classifier w/ Imputer + Text Fea... 0.123494 0.129007 98.570392 False {'Imputer': {'categorical_impute_strategy': 'm...
1 3 XGBoost Classifier w/ Imputer + Text Featuriza... 0.128840 0.134231 98.508509 False {'Imputer': {'categorical_impute_strategy': 'm...
2 1 Logistic Regression Classifier w/ Imputer + Te... 0.170443 0.148005 98.026895 True {'Imputer': {'categorical_impute_strategy': 'm...
3 7 LightGBM Classifier w/ Imputer + Text Featuriz... 0.171467 0.183049 98.015036 False {'Imputer': {'categorical_impute_strategy': 'm...
4 6 Extra Trees Classifier w/ Imputer + Text Featu... 0.219864 0.250778 97.454783 False {'Imputer': {'categorical_impute_strategy': 'm...
5 4 CatBoost Classifier w/ Imputer + Text Featuriz... 0.518860 0.522827 93.993496 False {'Imputer': {'categorical_impute_strategy': 'm...
6 5 Elastic Net Classifier w/ Imputer + Text Featu... 0.530537 0.561991 93.858323 False {'Imputer': {'categorical_impute_strategy': 'm...
7 8 Decision Tree Classifier w/ Imputer + Text Fea... 0.726110 0.595490 91.594296 False {'Imputer': {'categorical_impute_strategy': 'm...
8 0 Mode Baseline Binary Classification Pipeline 8.638305 8.623860 0.000000 False {'Baseline Classifier': {'strategy': 'mode'}}
[16]:
best_pipeline_no_text = automl_no_text.best_pipeline

Here, changing the data type of the text column removed the Text Featurization Component from the pipeline.

[17]:
best_pipeline_no_text.graph()
[17]:
../_images/demos_text_input_33_0.svg
[18]:
automl_no_text.describe_pipeline(automl_no_text.rankings.iloc[0]["id"])
**********************************************************************
* Random Forest Classifier w/ Imputer + Text Featurization Component *
**********************************************************************

Problem Type: binary
Model Family: Random Forest

Pipeline Steps
==============
1. Imputer
         * categorical_impute_strategy : most_frequent
         * numeric_impute_strategy : mean
         * categorical_fill_value : None
         * numeric_fill_value : None
2. Text Featurization Component
         * text_columns : ['Message']
3. Random Forest Classifier
         * n_estimators : 100
         * max_depth : 6
         * n_jobs : -1

Training
========
Training for binary problems.
Total training time (including CV): 9.6 seconds

Cross Validation
----------------
             Log Loss Binary  MCC Binary   AUC  Precision    F1  Balanced Accuracy Binary  Accuracy Binary # Training # Validation
0                      0.129       0.885 0.979      0.937 0.912                     0.935            0.957   1594.000      797.000
1                      0.127       0.892 0.987      0.937 0.918                     0.940            0.960   1594.000      797.000
2                      0.114       0.895 0.984      0.952 0.920                     0.937            0.961   1594.000      797.000
mean                   0.123       0.891 0.983      0.942 0.917                     0.937            0.959          -            -
std                    0.008       0.005 0.004      0.009 0.004                     0.003            0.002          -            -
coef of var            0.065       0.006 0.004      0.009 0.004                     0.003            0.002          -            -
[19]:
# train on the full training data
best_pipeline_no_text.fit(X_train, y_train)

# get standard performance metrics on holdout data
scores = best_pipeline_no_text.score(X_holdout, y_holdout,  objectives=evalml.objectives.get_core_objectives('binary'))
print(f'Accuracy Binary: {scores["Accuracy Binary"]}')
Accuracy Binary: 0.9732441471571907

Without the Text Featurization Component, the 'Message' column was treated as a categorical column, and therefore the conversion of this text to numerical features happened in the One Hot Encoder. The best pipeline encoded the top 10 most frequent “categories” of these texts, meaning 10 text messages were one-hot encoded and all the others were dropped. Clearly, this removed almost all of the information from the dataset, as we can see the best_pipeline_no_text did not beat the random guess of predicting “ham” in every case.