Model Understanding

Simply examining a model’s performance metrics is not enough to select a model and promote it for use in a production setting. While developing an ML algorithm, it is important to understand how the model behaves on the data, to examine the key factors influencing its predictions and to consider where it may be deficient. Determination of what “success” may mean for an ML project depends first and foremost on the user’s domain expertise.

EvalML includes a variety of tools for understanding models.

First, let’s train a pipeline on some data.

[1]:
import evalml

class RFBinaryClassificationPipeline(evalml.pipelines.BinaryClassificationPipeline):
    component_graph = ['Simple Imputer', 'Random Forest Classifier']

X, y = evalml.demos.load_breast_cancer()

pipeline = RFBinaryClassificationPipeline({})
pipeline.fit(X, y)
print(pipeline.score(X, y, objectives=['log_loss_binary']))
2020-08-06 19:59:22,233 featuretools - WARNING    Featuretools failed to load plugin nlp_primitives from library nlp_primitives. For a full stack trace, set logging to debug.
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/stable/lib/python3.7/site-packages/evalml/pipelines/components/transformers/preprocessing/text_featurizer.py:31: RuntimeWarning: No text columns were given to TextFeaturizer, component will have no effect
  warnings.warn("No text columns were given to TextFeaturizer, component will have no effect", RuntimeWarning)
OrderedDict([('Log Loss Binary', 0.038403828027876195)])

Feature Importance

We can get the importance associated with each feature of the resulting pipeline

[2]:
pipeline.feature_importance
[2]:
feature importance
0 worst perimeter 0.176488
1 worst concave points 0.125260
2 worst radius 0.124161
3 mean concave points 0.086443
4 worst area 0.072465
5 mean concavity 0.072320
6 mean perimeter 0.056685
7 mean area 0.049599
8 area error 0.037229
9 worst concavity 0.028181
10 mean radius 0.023294
11 radius error 0.019457
12 worst texture 0.014990
13 perimeter error 0.014103
14 mean texture 0.013618
15 worst compactness 0.011310
16 worst smoothness 0.011139
17 worst fractal dimension 0.008118
18 worst symmetry 0.007818
19 mean smoothness 0.006152
20 concave points error 0.005887
21 fractal dimension error 0.005059
22 concavity error 0.004510
23 smoothness error 0.004493
24 texture error 0.004476
25 mean compactness 0.004050
26 compactness error 0.003559
27 mean symmetry 0.003243
28 symmetry error 0.003124
29 mean fractal dimension 0.002768

We can also create a bar plot of the feature importances

[3]:
pipeline.graph_feature_importance()