Model Understanding

Simply examining a model’s performance metrics is not enough to select a model and promote it for use in a production setting. While developing an ML algorithm, it is important to understand how the model behaves on the data, to examine the key factors influencing its predictions and to consider where it may be deficient. Determination of what “success” may mean for an ML project depends first and foremost on the user’s domain expertise.

EvalML includes a variety of tools for understanding models, from graphing utilities to methods for explaining predictions.

Graphing Utilities

First, let’s train a pipeline on some data.

[1]:
import evalml

class RFBinaryClassificationPipeline(evalml.pipelines.BinaryClassificationPipeline):
    component_graph = ['Simple Imputer', 'Random Forest Classifier']

X, y = evalml.demos.load_breast_cancer()

pipeline = RFBinaryClassificationPipeline({})
pipeline.fit(X, y)
print(pipeline.score(X, y, objectives=['log_loss_binary']))
OrderedDict([('Log Loss Binary', 0.038403828027876195)])

Feature Importance

We can get the importance associated with each feature of the resulting pipeline

[2]:
pipeline.feature_importance
[2]:
feature importance
0 worst perimeter 0.176488
1 worst concave points 0.125260
2 worst radius 0.124161
3 mean concave points 0.086443
4 worst area 0.072465
5 mean concavity 0.072320
6 mean perimeter 0.056685
7 mean area 0.049599
8 area error 0.037229
9 worst concavity 0.028181
10 mean radius 0.023294
11 radius error 0.019457
12 worst texture 0.014990
13 perimeter error 0.014103
14 mean texture 0.013618
15 worst compactness 0.011310
16 worst smoothness 0.011139
17 worst fractal dimension 0.008118
18 worst symmetry 0.007818
19 mean smoothness 0.006152
20 concave points error 0.005887
21 fractal dimension error 0.005059
22 concavity error 0.004510
23 smoothness error 0.004493
24 texture error 0.004476
25 mean compactness 0.004050
26 compactness error 0.003559
27 mean symmetry 0.003243
28 symmetry error 0.003124
29 mean fractal dimension 0.002768

We can also create a bar plot of the feature importances

[3]:
pipeline.graph_feature_importance()

Permutation Importance

We can also compute and plot the permutation importance of the pipeline.

[4]:
from evalml.model_understanding.graphs import calculate_permutation_importance
calculate_permutation_importance(pipeline, X, y, 'log_loss_binary')
[4]:
feature importance
0 worst perimeter 0.078033
1 worst radius 0.074341
2 worst concave points 0.068313
3 worst area 0.067733
4 mean concave points 0.041261
5 worst concavity 0.037533
6 mean concavity 0.036664
7 area error 0.035838
8 mean perimeter 0.025783
9 mean area 0.025203
10 worst texture 0.016211
11 perimeter error 0.011738
12 mean texture 0.011716
13 radius error 0.010910
14 mean radius 0.010775
15 worst compactness 0.008322
16 worst smoothness 0.008281
17 mean smoothness 0.005707
18 worst symmetry 0.004454
19 worst fractal dimension 0.003889
20 concavity error 0.003858
21 compactness error 0.003572
22 concave points error 0.003449
23 mean compactness 0.003173
24 smoothness error 0.003172
25 fractal dimension error 0.002618
26 texture error 0.002533
27 mean fractal dimension 0.002228
28 symmetry error 0.002126
29 mean symmetry 0.001786
[5]:
from evalml.model_understanding.graphs import graph_permutation_importance
graph_permutation_importance(pipeline, X, y, 'log_loss_binary')

Partial Dependence Plots

We can calculate the partial dependence plots for a feature.

[6]:
from evalml.model_understanding.graphs import partial_dependence
partial_dependence(pipeline, X, feature='mean radius')
[6]:
feature_values partial_dependence
0 9.498540 0.371141
1 9.610488 0.371141
2 9.722436 0.371141
3 9.834384 0.371141
4 9.946332 0.371141
... ... ...
95 20.133608 0.399560
96 20.245556 0.399560
97 20.357504 0.399560
98 20.469452 0.399560
99 20.581400 0.399560

100 rows × 2 columns

[7]:
from evalml.model_understanding.graphs import graph_partial_dependence
graph_partial_dependence(pipeline, X, feature='mean radius')

Confusion Matrix

For binary or multiclass classification, we can view a confusion matrix of the classifier’s predictions

[8]:
from evalml.model_understanding.graphs import graph_confusion_matrix
y_pred = pipeline.predict(X)
graph_confusion_matrix(y, y_pred)

Precision-Recall Curve

For binary classification, we can view the precision-recall curve of the pipeline.

[9]:
from evalml.model_understanding.graphs import graph_precision_recall_curve
# get the predicted probabilities associated with the "true" label
y_encoded = y.map({'malignant': 0, 'benign': 1})
y_pred_proba = pipeline.predict_proba(X)["benign"]
graph_precision_recall_curve(y_encoded, y_pred_proba)

ROC Curve

For binary and multiclass classification, we can view the Receiver Operating Characteristic (ROC) curve of the pipeline.

[10]:
from evalml.model_understanding.graphs import graph_roc_curve
# get the predicted probabilities associated with the "benign" label
y_pred_proba = pipeline.predict_proba(X)["benign"]
graph_roc_curve(y_encoded, y_pred_proba)

Binary Objective Score vs. Threshold Graph

Some binary classification objectives (objectives that have score_needs_proba set to False) are sensitive to a decision threshold. For those objectives, we can obtain and graph the scores for thresholds from zero to one, calculated at evenly-spaced intervals determined by steps.

[11]:
from evalml.model_understanding.graphs import binary_objective_vs_threshold
binary_objective_vs_threshold(pipeline, X, y, 'f1', steps=100)
[11]:
threshold score
0 0.00 0.542894
1 0.01 0.750442
2 0.02 0.815385
3 0.03 0.848000
4 0.04 0.874227
... ... ...
96 0.96 0.854054
97 0.97 0.835165
98 0.98 0.805634
99 0.99 0.722892
100 1.00 0.000000

101 rows × 2 columns

[12]:
from evalml.model_understanding.graphs import graph_binary_objective_vs_threshold
graph_binary_objective_vs_threshold(pipeline, X, y, 'f1', steps=100)

Explaining Predictions

Explaining Individual Predictions

We can explain why the model made an individual prediction with the explain_prediction function. This will use the Shapley Additive Explanations (SHAP) algorithms to identify the top features that explain the predicted value.

This function can explain both classification and regression models - all you need to do is provide the pipeline, the input features (must correspond to one row of the input data) and the training data. The function will return a table that you can print summarizing the top 3 most positive and negative contributing features to the predicted value.

In the example below, we explain the prediction for the third data point in the data set. We see that the worst concave points feature increased the estimated probability that the tumor is malignant by 20% while the worst radius feature decreased the probability the tumor is malignant by 5%.

[13]:
from evalml.model_understanding.prediction_explanations import explain_prediction

table = explain_prediction(pipeline=pipeline, input_features=X.iloc[3:4],
                           training_data=X, include_shap_values=True)
print(table)
    Feature Name       Feature Value   Contribution to Prediction   SHAP Value
==============================================================================
worst concave points       0.26                    ++                  0.20
mean concave points        0.11                    +                   0.11
   mean concavity          0.24                    +                   0.08
     worst area           567.70                   -                  -0.03
  worst perimeter          98.87                   -                  -0.05
    worst radius           14.91                   -                  -0.05

The interpretation of the table is the same for regression problems - but the SHAP value now corresponds to the change in the estimated value of the dependent variable rather than a change in probability. For multiclass classification problems, a table will be output for each possible class.

This functionality is currently not supported for XGBoost models or CatBoost multiclass classifiers.

Explaining Multiple Predictions

When debugging machine learning models, it is often useful to analyze the best and worst predictions the model made. The explain_predictions_best_worst function can help us with this.

This function will display the output of explain_prediction for the best 2 and worst 2 predictions. By default, the best and worst predictions are determined by the absolute error for regression problems and cross entropy for classification problems.

We can specify our own ranking function by passing in a function to the metric parameter. This function will be called on y_true and y_pred. By convention, lower scores are better.

At the top of each table, we can see the predicted probabilities, target value, and error on that prediction. For a regression problem, we would see the predicted value instead of predicted probabilities.

[14]:
from evalml.model_understanding.prediction_explanations import explain_predictions_best_worst

report = explain_predictions_best_worst(pipeline=pipeline, input_features=X, y_true=y,
                                        include_shap_values=True, num_to_explain=2)

print(report)
RFBinary Classification Pipeline

{'Simple Imputer': {'impute_strategy': 'most_frequent', 'fill_value': None}, 'Random Forest Classifier': {'n_estimators': 100, 'max_depth': 6, 'n_jobs': -1}}

        Best 1 of 2

                Predicted Probabilities: [benign: 0.0, malignant: 1.0]
                Predicted Value: malignant
                Target Value: malignant
                Cross Entropy: 0.0

                     Feature Name         Feature Value        Contribution to        SHAP Value
                                                                 Prediction
                ================================================================================
                    worst perimeter          155.30                   +                  0.10
                     worst radius             23.14                   +                  0.08
                 worst concave points         0.17                    +                  0.08
                worst fractal dimension       0.09                    -                 -0.00
                   compactness error          0.04                    -                 -0.00
                    worst symmetry            0.22                    -                 -0.00


        Best 2 of 2

                Predicted Probabilities: [benign: 0.0, malignant: 1.0]
                Predicted Value: malignant
                Target Value: malignant
                Cross Entropy: 0.0

                    Feature Name       Feature Value   Contribution to Prediction   SHAP Value
                ==============================================================================
                  worst perimeter         166.10                   +                   0.10
                    worst radius           25.45                   +                   0.08
                worst concave points       0.22                    +                   0.08
                 compactness error         0.03                    -                  -0.00
                 worst compactness         0.21                    -                  -0.00
                   worst symmetry          0.21                    -                  -0.00


        Worst 1 of 2

                Predicted Probabilities: [benign: 0.552, malignant: 0.448]
                Predicted Value: benign
                Target Value: malignant
                Cross Entropy: 0.802

                    Feature Name       Feature Value   Contribution to Prediction   SHAP Value
                ==============================================================================
                  smoothness error         0.00                    +                   0.04
                    mean texture           21.58                   +                   0.03
                   worst texture           30.25                   +                   0.02
                worst concave points       0.11                    -                  -0.02
                    worst radius           15.93                   -                  -0.03
                mean concave points        0.02                    -                  -0.03


        Worst 2 of 2

                Predicted Probabilities: [benign: 0.788, malignant: 0.212]
                Predicted Value: benign
                Target Value: malignant
                Cross Entropy: 1.55

                    Feature Name       Feature Value   Contribution to Prediction   SHAP Value
                ==============================================================================
                   worst texture           33.37                   +                   0.05
                    mean texture           22.47                   +                   0.03
                   symmetry error          0.02                    +                   0.01
                worst concave points       0.09                    -                  -0.04
                    worst radius           14.49                   -                  -0.05
                  worst perimeter          92.04                   -                  -0.06



We use a custom metric (hinge loss) for selecting the best and worst predictions. See this example:

import numpy as np

def hinge_loss(y_true, y_pred_proba):

    probabilities = np.clip(y_pred_proba.iloc[:, 1], 0.001, 0.999)
    y_true[y_true == 0] = -1

    return np.clip(1 - y_true * np.log(probabilities / (1 - probabilities)), a_min=0, a_max=None)

report = explain_predictions_best_worst(pipeline=pipeline, input_features=X, y_true=y,
                                        include_shap_values=True, num_to_explain=5, metric=hinge_loss)

print(report)

We can also manually explain predictions on any subset of the training data with the explain_predictions function. Below, we explain the predictions on the first, fifth, and tenth row of the data.

[15]:
from evalml.model_understanding.prediction_explanations import explain_predictions

report = explain_predictions(pipeline=pipeline, input_features=X.iloc[[0, 4, 9]], include_shap_values=True)
print(report)
RFBinary Classification Pipeline

{'Simple Imputer': {'impute_strategy': 'most_frequent', 'fill_value': None}, 'Random Forest Classifier': {'n_estimators': 100, 'max_depth': 6, 'n_jobs': -1}}

        1 of 3

                    Feature Name       Feature Value   Contribution to Prediction   SHAP Value
                ==============================================================================
                worst concave points       0.27                    +                   0.09
                  worst perimeter         184.60                   +                   0.09
                    worst radius           25.38                   +                   0.08
                 compactness error         0.05                    -                  -0.00
                   worst texture           17.33                   -                  -0.03
                    mean texture           10.38                   -                  -0.05


        2 of 3

                    Feature Name       Feature Value   Contribution to Prediction   SHAP Value
                ==============================================================================
                  worst perimeter         152.20                   +                   0.11
                    worst radius           22.54                   +                   0.09
                worst concave points       0.16                    +                   0.08
                   worst symmetry          0.24                    -                  -0.00
                    mean texture           14.34                   -                  -0.03
                   worst texture           16.67                   -                  -0.03


        3 of 3

                    Feature Name       Feature Value   Contribution to Prediction   SHAP Value
                ==============================================================================
                worst concave points       0.22                    ++                  0.20
                mean concave points        0.09                    +                   0.11
                   mean concavity          0.23                    +                   0.08
                     mean area            475.90                   -                  -0.01
                    worst radius           15.09                   -                  -0.03
                  worst perimeter          97.65                   -                  -0.05