Exploring search results¶
After finishing a pipeline search, we can inspect the results. First, let’s build a search of 10 different pipelines to explore.
[1]:
import evalml
from evalml import AutoClassificationSearch
X, y = evalml.demos.load_breast_cancer()
automl = AutoClassificationSearch(objective="f1",
max_pipelines=5)
automl.search(X, y)
*****************************
* Beginning pipeline search *
*****************************
Optimizing for F1. Greater score is better.
Searching up to 5 pipelines.
✔ Cat Boost Classification Pipeline: 20%|██ | Elapsed:00:07
✔ Logistic Regression Pipeline: 40%|████ | Elapsed:00:08
✔ Logistic Regression Pipeline: 60%|██████ | Elapsed:00:08
▹ XGBoost Classification Pipeline: 80%|████████ | Elapsed:00:08
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.8.0/lib/python3.7/site-packages/xgboost/sklearn.py:888: UserWarning:
The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
[19:57:05] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.8.0/lib/python3.7/site-packages/xgboost/sklearn.py:888: UserWarning:
The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
[19:57:08] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-evalml/envs/v0.8.0/lib/python3.7/site-packages/xgboost/sklearn.py:888: UserWarning:
The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
[19:57:10] WARNING: ../src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
✔ XGBoost Classification Pipeline: 80%|████████ | Elapsed:00:15
✔ Cat Boost Classification Pipeline: 100%|██████████| Elapsed:00:16
✔ Optimization finished 100%|██████████| Elapsed:00:16
View Rankings¶
A summary of all the pipelines built can be returned as a pandas DataFrame. It is sorted by score. EvalML knows based on our objective function whether higher or lower is better.
[2]:
automl.rankings
[2]:
| id | pipeline_name | score | high_variance_cv | parameters | |
|---|---|---|---|---|---|
| 0 | 2 | Logistic Regression Pipeline | 0.976371 | False | {'impute_strategy': 'median', 'penalty': 'l2',... |
| 1 | 1 | Logistic Regression Pipeline | 0.974941 | False | {'impute_strategy': 'median', 'penalty': 'l2',... |
| 2 | 0 | Cat Boost Classification Pipeline | 0.973464 | False | {'impute_strategy': 'most_frequent', 'n_estima... |
| 3 | 4 | Cat Boost Classification Pipeline | 0.966453 | False | {'impute_strategy': 'most_frequent', 'n_estima... |
| 4 | 3 | XGBoost Classification Pipeline | 0.935899 | False | {'impute_strategy': 'most_frequent', 'percent_... |
Describe Pipeline¶
Each pipeline is given an id. We can get more information about any particular pipeline using that id. Here, we will get more information about the pipeline with id = 0.
[3]:
automl.describe_pipeline(0)
*************************************
* Cat Boost Classification Pipeline *
*************************************
Supported Problem Types: Binary Classification, Multiclass Classification
Model Family: CatBoost Classifier
Objective to Optimize: F1 (greater is better)
Number of features: 30
Pipeline Steps
==============
1. Simple Imputer
* impute_strategy : most_frequent
* fill_value : None
2. CatBoost Classifier
* n_estimators : 369
* eta : 0.4236547993389048
* max_depth : 6
Training
========
Training for Binary Classification problems.
Total training time (including CV): 7.0 seconds
Cross Validation
----------------
F1 Precision Recall AUC Log Loss MCC # Training # Testing
0 0.954 0.958 0.950 0.985 0.163 0.877 379.000 190.000
1 0.983 0.967 1.000 0.996 0.092 0.955 379.000 190.000
2 0.983 0.975 0.992 0.996 0.076 0.955 380.000 189.000
mean 0.973 0.967 0.980 0.993 0.110 0.929 - -
std 0.017 0.009 0.027 0.006 0.046 0.045 - -
coef of var 0.018 0.009 0.028 0.007 0.418 0.049 - -
Get Pipeline¶
We can get the object of any pipeline via their id as well:
[4]:
automl.get_pipeline(0)
[4]:
<evalml.pipelines.classification.catboost.CatBoostClassificationPipeline at 0x7f434936a050>
Get best pipeline¶
If we specifically want to get the best pipeline, there is a convenient access
[5]:
automl.best_pipeline
[5]:
<evalml.pipelines.classification.logistic_regression.LogisticRegressionPipeline at 0x7f43480cb190>
Feature Importances¶
We can get the feature importances of the resulting pipeline
[6]:
pipeline = automl.get_pipeline(0)
pipeline.feature_importances
[6]:
| feature | importance | |
|---|---|---|
| 0 | worst radius | 26.873775 |
| 1 | worst texture | 7.612433 |
| 2 | mean concave points | 7.145519 |
| 3 | mean texture | 7.024352 |
| 4 | worst symmetry | 6.361393 |
| 5 | worst concave points | 4.964684 |
| 6 | worst perimeter | 3.983425 |
| 7 | mean fractal dimension | 3.922057 |
| 8 | radius error | 3.447547 |
| 9 | worst smoothness | 2.880494 |
| 10 | worst area | 2.583201 |
| 11 | worst concavity | 2.413365 |
| 12 | mean concavity | 2.407919 |
| 13 | mean radius | 2.284723 |
| 14 | mean compactness | 2.012972 |
| 15 | perimeter error | 1.918985 |
| 16 | mean symmetry | 1.740673 |
| 17 | worst compactness | 1.655886 |
| 18 | area error | 1.606211 |
| 19 | mean smoothness | 1.443077 |
| 20 | compactness error | 1.435200 |
| 21 | concave points error | 1.040189 |
| 22 | mean area | 0.965964 |
| 23 | smoothness error | 0.834013 |
| 24 | worst fractal dimension | 0.436113 |
| 25 | fractal dimension error | 0.401008 |
| 26 | mean perimeter | 0.284681 |
| 27 | texture error | 0.182944 |
| 28 | symmetry error | 0.118948 |
| 29 | concavity error | 0.018248 |
We can also create a bar plot of the feature importances
[7]:
pipeline.feature_importance_graph(pipeline)
Plot ROC¶
For binary classification tasks, we can also plot the ROC plot of a specific pipeline:
[8]:
automl.plot.generate_roc_plot(0)
Access raw results¶
You can also get access to all the underlying data like this
[9]:
automl.results
[9]:
{'pipeline_results': {0: {'id': 0,
'pipeline_name': 'Cat Boost Classification Pipeline',
'pipeline_summary': 'CatBoost Classifier w/ Simple Imputer',
'parameters': {'impute_strategy': 'most_frequent',
'n_estimators': 369,
'eta': 0.4236547993389048,
'max_depth': 6},
'score': 0.9734636352012647,
'high_variance_cv': False,
'training_time': 6.997751235961914,
'cv_data': [{'all_objective_scores': OrderedDict([('F1',
0.9535864978902954),
('Precision', 0.9576271186440678),
('Recall', 0.9495798319327731),
('AUC', 0.9850869925434962),
('Log Loss', 0.16289034062986435),
('MCC', 0.8767221685540966),
('ROC',
(array([0. , 0. , 0. , 0.01408451, 0.01408451,
0.02816901, 0.02816901, 0.05633803, 0.05633803, 0.07042254,
0.07042254, 0.08450704, 0.08450704, 0.14084507, 0.14084507,
1. ]),
array([0. , 0.00840336, 0.34453782, 0.34453782, 0.84033613,
0.84033613, 0.93277311, 0.93277311, 0.94117647, 0.94117647,
0.98319328, 0.98319328, 0.99159664, 0.99159664, 1. ,
1. ]),
array([1.99997424e+00, 9.99974244e-01, 9.99756520e-01, 9.99745844e-01,
9.88277062e-01, 9.81932629e-01, 7.35284284e-01, 6.10443075e-01,
5.94587484e-01, 5.82814719e-01, 9.45094859e-02, 4.30750748e-02,
3.99827348e-02, 1.48833097e-02, 1.48214210e-02, 3.31091266e-05]))),
('Confusion Matrix',
0 1
0 66 5
1 6 113),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9535864978902954},
{'all_objective_scores': OrderedDict([('F1', 0.9834710743801653),
('Precision', 0.967479674796748),
('Recall', 1.0),
('AUC', 0.996330926736892),
('Log Loss', 0.09239450377828266),
('MCC', 0.9554966130892879),
('ROC',
(array([0. , 0. , 0. , 0.01408451, 0.01408451,
0.02816901, 0.02816901, 0.04225352, 0.04225352, 1. ]),
array([0. , 0.00840336, 0.80672269, 0.80672269, 0.94957983,
0.94957983, 0.98319328, 0.98319328, 1. , 1. ]),
array([1.99998792e+00, 9.99987915e-01, 9.98961984e-01, 9.98941711e-01,
9.90392310e-01, 9.85250779e-01, 9.51254710e-01, 9.38505542e-01,
8.48666034e-01, 1.18603531e-05]))),
('Confusion Matrix',
0 1
0 67 4
1 0 119),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9834710743801653},
{'all_objective_scores': OrderedDict([('F1', 0.9833333333333334),
('Precision', 0.9752066115702479),
('Recall', 0.9915966386554622),
('AUC', 0.9962785114045618),
('Log Loss', 0.07598028585889369),
('MCC', 0.9546019995535027),
('ROC',
(array([0. , 0. , 0. , 0.01428571, 0.01428571,
0.02857143, 0.02857143, 0.04285714, 0.04285714, 0.07142857,
0.07142857, 1. ]),
array([0. , 0.00840336, 0.84033613, 0.84033613, 0.94957983,
0.94957983, 0.96638655, 0.96638655, 0.99159664, 0.99159664,
1. , 1. ]),
array([1.99998096e+00, 9.99980963e-01, 9.93261789e-01, 9.93003955e-01,
9.37059262e-01, 9.21284961e-01, 8.00155771e-01, 7.72884615e-01,
5.89460929e-01, 4.27271558e-01, 2.71432274e-01, 1.61384342e-05]))),
('Confusion Matrix',
0 1
0 67 3
1 1 118),
('# Training', 380),
('# Testing', 189)]),
'score': 0.9833333333333334}]},
1: {'id': 1,
'pipeline_name': 'Logistic Regression Pipeline',
'pipeline_summary': 'Logistic Regression Classifier w/ One Hot Encoder + Simple Imputer + Standard Scaler',
'parameters': {'impute_strategy': 'median',
'penalty': 'l2',
'C': 9.636990977405285},
'score': 0.9749409107621198,
'high_variance_cv': False,
'training_time': 1.418722152709961,
'cv_data': [{'all_objective_scores': OrderedDict([('F1',
0.9666666666666667),
('Precision', 0.9586776859504132),
('Recall', 0.9747899159663865),
('AUC', 0.9888744230086401),
('Log Loss', 0.16122177593368092),
('MCC', 0.9097672817424011),
('ROC',
(array([0. , 0. , 0. , 0.01408451, 0.01408451,
0.02816901, 0.02816901, 0.04225352, 0.04225352, 0.05633803,
0.05633803, 0.07042254, 0.07042254, 0.21126761, 0.21126761,
1. ]),
array([0. , 0.00840336, 0.59663866, 0.59663866, 0.86554622,
0.86554622, 0.91596639, 0.91596639, 0.94117647, 0.94117647,
0.97478992, 0.97478992, 0.99159664, 0.99159664, 1. ,
1. ]),
array([2.00000000e+00, 1.00000000e+00, 9.99832384e-01, 9.99827305e-01,
9.78642320e-01, 9.78473934e-01, 8.85901577e-01, 8.46173258e-01,
8.11073923e-01, 7.85780332e-01, 5.55932686e-01, 5.12973949e-01,
3.58737371e-01, 3.80356180e-04, 2.16404977e-04, 3.27364828e-24]))),
('Confusion Matrix',
0 1
0 66 5
1 3 116),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9666666666666667},
{'all_objective_scores': OrderedDict([('F1', 0.979253112033195),
('Precision', 0.9672131147540983),
('Recall', 0.9915966386554622),
('AUC', 0.9983429991714996),
('Log Loss', 0.054377948871000094),
('MCC', 0.943843520216036),
('ROC',
(array([0. , 0. , 0. , 0.01408451, 0.01408451,
0.02816901, 0.02816901, 0.05633803, 0.05633803, 0.09859155,
0.09859155, 1. ]),
array([0. , 0.00840336, 0.96638655, 0.96638655, 0.97478992,
0.97478992, 0.98319328, 0.98319328, 0.99159664, 0.99159664,
1. , 1. ]),
array([2.00000000e+00, 1.00000000e+00, 9.02530651e-01, 8.56837110e-01,
8.32273897e-01, 7.83350982e-01, 7.29357685e-01, 5.16534638e-01,
5.11585048e-01, 2.30456831e-01, 2.07185770e-01, 4.44775362e-44]))),
('Confusion Matrix',
0 1
0 67 4
1 1 118),
('# Training', 379),
('# Testing', 190)]),
'score': 0.979253112033195},
{'all_objective_scores': OrderedDict([('F1', 0.9789029535864979),
('Precision', 0.9830508474576272),
('Recall', 0.9747899159663865),
('AUC', 0.9960384153661465),
('Log Loss', 0.07207639820328837),
('MCC', 0.9435040132749904),
('ROC',
(array([0. , 0. , 0. , 0.01428571, 0.01428571,
0.02857143, 0.02857143, 0.04285714, 0.04285714, 1. ]),
array([0. , 0.00840336, 0.77310924, 0.77310924, 0.96638655,
0.96638655, 0.98319328, 0.98319328, 1. , 1. ]),
array([2.00000000e+00, 9.99999998e-01, 9.91123955e-01, 9.89983111e-01,
5.77651350e-01, 5.45079286e-01, 4.73073055e-01, 4.09422136e-01,
3.23433980e-01, 1.26109142e-17]))),
('Confusion Matrix',
0 1
0 68 2
1 3 116),
('# Training', 380),
('# Testing', 189)]),
'score': 0.9789029535864979}]},
2: {'id': 2,
'pipeline_name': 'Logistic Regression Pipeline',
'pipeline_summary': 'Logistic Regression Classifier w/ One Hot Encoder + Simple Imputer + Standard Scaler',
'parameters': {'impute_strategy': 'median',
'penalty': 'l2',
'C': 4.804971952026824},
'score': 0.976371018670262,
'high_variance_cv': False,
'training_time': 0.3552417755126953,
'cv_data': [{'all_objective_scores': OrderedDict([('F1',
0.9666666666666667),
('Precision', 0.9586776859504132),
('Recall', 0.9747899159663865),
('AUC', 0.9899396378269617),
('Log Loss', 0.13000491300733263),
('MCC', 0.9097672817424011),
('ROC',
(array([0. , 0. , 0. , 0.01408451, 0.01408451,
0.02816901, 0.02816901, 0.04225352, 0.04225352, 0.07042254,
0.07042254, 0.12676056, 0.12676056, 1. ]),
array([0. , 0.00840336, 0.59663866, 0.59663866, 0.85714286,
0.85714286, 0.93277311, 0.93277311, 0.96638655, 0.96638655,
0.99159664, 0.99159664, 1. , 1. ]),
array([2.00000000e+00, 1.00000000e+00, 9.99513300e-01, 9.99411733e-01,
9.72040206e-01, 9.64245540e-01, 8.24677844e-01, 8.13137106e-01,
6.23606303e-01, 5.74962709e-01, 3.88150049e-01, 1.05841228e-01,
8.88048039e-03, 1.95380932e-20]))),
('Confusion Matrix',
0 1
0 66 5
1 3 116),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9666666666666667},
{'all_objective_scores': OrderedDict([('F1', 0.979253112033195),
('Precision', 0.9672131147540983),
('Recall', 0.9915966386554622),
('AUC', 0.9986980707776069),
('Log Loss', 0.051792807657331755),
('MCC', 0.943843520216036),
('ROC',
(array([0. , 0. , 0. , 0.01408451, 0.01408451,
0.04225352, 0.04225352, 0.08450704, 0.08450704, 1. ]),
array([0. , 0.00840336, 0.96638655, 0.96638655, 0.98319328,
0.98319328, 0.99159664, 0.99159664, 1. , 1. ]),
array([2.00000000e+00, 9.99999999e-01, 9.15699240e-01, 8.64894910e-01,
7.51407817e-01, 6.63047926e-01, 6.26895191e-01, 4.06750268e-01,
3.09309176e-01, 1.59809498e-36]))),
('Confusion Matrix',
0 1
0 67 4
1 1 118),
('# Training', 379),
('# Testing', 190)]),
'score': 0.979253112033195},
{'all_objective_scores': OrderedDict([('F1', 0.9831932773109243),
('Precision', 0.9831932773109243),
('Recall', 0.9831932773109243),
('AUC', 0.9965186074429773),
('Log Loss', 0.06435139219399516),
('MCC', 0.9546218487394958),
('ROC',
(array([0. , 0. , 0. , 0.01428571, 0.01428571,
0.02857143, 0.02857143, 0.04285714, 0.04285714, 1. ]),
array([0. , 0.00840336, 0.78151261, 0.78151261, 0.98319328,
0.98319328, 0.99159664, 0.99159664, 1. , 1. ]),
array([1.99999996e+00, 9.99999962e-01, 9.84583454e-01, 9.76666534e-01,
5.25351686e-01, 5.03605398e-01, 4.55872870e-01, 4.36470183e-01,
4.28464345e-01, 2.38837351e-14]))),
('Confusion Matrix',
0 1
0 68 2
1 2 117),
('# Training', 380),
('# Testing', 189)]),
'score': 0.9831932773109243}]},
3: {'id': 3,
'pipeline_name': 'XGBoost Classification Pipeline',
'pipeline_summary': 'XGBoost Classifier w/ One Hot Encoder + Simple Imputer + RF Classifier Select From Model',
'parameters': {'impute_strategy': 'most_frequent',
'percent_features': 0.08032569761590808,
'threshold': 'mean',
'eta': 0.020218397440325723,
'max_depth': 20,
'min_child_weight': 9.614396430577418,
'n_estimators': 848},
'score': 0.9358991763960539,
'high_variance_cv': False,
'training_time': 6.885083913803101,
'cv_data': [{'all_objective_scores': OrderedDict([('F1',
0.9224137931034483),
('Precision', 0.9469026548672567),
('Recall', 0.8991596638655462),
('AUC', 0.9666232690259202),
('Log Loss', 0.2562118422632155),
('MCC', 0.8027688833516422),
('ROC',
(array([0. , 0.01408451, 0.01408451, 0.01408451, 0.01408451,
0.01408451, 0.01408451, 0.01408451, 0.04225352, 0.04225352,
0.07042254, 0.07042254, 0.12676056, 0.12676056, 0.12676056,
0.15492958, 0.16901408, 0.18309859, 0.21126761, 0.25352113,
0.28169014, 0.36619718, 0.3943662 , 0.3943662 , 0.42253521,
0.45070423, 1. ]),
array([0. , 0.54621849, 0.62184874, 0.63865546, 0.65546218,
0.66386555, 0.71428571, 0.7394958 , 0.7394958 , 0.8487395 ,
0.86554622, 0.89915966, 0.89915966, 0.90756303, 0.94117647,
0.94117647, 0.96638655, 0.96638655, 0.96638655, 0.98319328,
0.99159664, 0.99159664, 0.99159664, 1. , 1. ,
1. , 1. ]),
array([1.9729209 , 0.97292095, 0.9717207 , 0.9710014 , 0.970229 ,
0.9697926 , 0.9423327 , 0.9413457 , 0.93782085, 0.92524594,
0.6996592 , 0.5642555 , 0.40121877, 0.28772253, 0.27541634,
0.16674423, 0.15323758, 0.06644133, 0.06502564, 0.04827413,
0.04722596, 0.04633046, 0.04199466, 0.03978153, 0.03914706,
0.03638416, 0.03603544], dtype=float32))),
('Confusion Matrix',
0 1
0 65 6
1 12 107),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9224137931034483},
{'all_objective_scores': OrderedDict([('F1', 0.937007874015748),
('Precision', 0.8814814814814815),
('Recall', 1.0),
('AUC', 0.9770978814060836),
('Log Loss', 0.23696691198764663),
('MCC', 0.826339982903411),
('ROC',
(array([0. , 0. , 0. , 0.05633803, 0.05633803,
0.05633803, 0.08450704, 0.08450704, 0.14084507, 0.15492958,
0.15492958, 0.16901408, 0.16901408, 0.23943662, 0.26760563,
0.29577465, 0.32394366, 0.38028169, 0.43661972, 0.46478873,
0.49295775, 1. ]),
array([0. , 0.49579832, 0.55462185, 0.85714286, 0.89915966,
0.90756303, 0.90756303, 0.92436975, 0.96638655, 0.96638655,
0.98319328, 0.99159664, 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ,
1. , 1. ]),
array([1.9701126 , 0.97011256, 0.96989095, 0.9684937 , 0.9554615 ,
0.92179286, 0.9034773 , 0.83730716, 0.8372069 , 0.75368625,
0.63067174, 0.62702805, 0.61390054, 0.49871758, 0.48841092,
0.42385587, 0.41924828, 0.08368076, 0.04792023, 0.0469924 ,
0.04657101, 0.04570549], dtype=float32))),
('Confusion Matrix',
0 1
0 55 16
1 0 119),
('# Training', 379),
('# Testing', 190)]),
'score': 0.937007874015748},
{'all_objective_scores': OrderedDict([('F1', 0.9482758620689655),
('Precision', 0.9734513274336283),
('Recall', 0.9243697478991597),
('AUC', 0.9885954381752702),
('Log Loss', 0.1391644817456682),
('MCC', 0.8681704699715063),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0.01428571, 0.01428571, 0.04285714, 0.04285714, 0.07142857,
0.07142857, 0.08571429, 0.1 , 0.1 , 0.12857143,
0.12857143, 0.17142857, 0.2 , 0.22857143, 0.22857143,
0.28571429, 0.34285714, 0.38571429, 0.41428571, 0.42857143,
0.47142857, 0.55714286, 0.61428571, 1. ]),
array([0. , 0.28571429, 0.42016807, 0.44537815, 0.48739496,
0.50420168, 0.55462185, 0.59663866, 0.60504202, 0.67226891,
0.71428571, 0.74789916, 0.83193277, 0.85714286, 0.87394958,
0.87394958, 0.90756303, 0.90756303, 0.93277311, 0.93277311,
0.94117647, 0.94117647, 0.95798319, 0.96638655, 0.96638655,
0.97478992, 0.97478992, 0.97478992, 0.97478992, 1. ,
1. , 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. ]),
array([1.9841778 , 0.9841778 , 0.9838165 , 0.98342854, 0.9833071 ,
0.98211664, 0.9810844 , 0.9787828 , 0.97762203, 0.9773312 ,
0.975222 , 0.97440153, 0.89735514, 0.8955815 , 0.86498946,
0.84353524, 0.810082 , 0.67681974, 0.49621513, 0.4496832 ,
0.44711152, 0.4437709 , 0.43844405, 0.4295119 , 0.39135826,
0.33074102, 0.33018234, 0.31763443, 0.29066208, 0.18481019,
0.03418758, 0.03268759, 0.0274495 , 0.0270752 , 0.0269705 ,
0.02627881, 0.02563108, 0.02260989, 0.0213987 ], dtype=float32))),
('Confusion Matrix',
0 1
0 67 3
1 9 110),
('# Training', 380),
('# Testing', 189)]),
'score': 0.9482758620689655}]},
4: {'id': 4,
'pipeline_name': 'Cat Boost Classification Pipeline',
'pipeline_summary': 'CatBoost Classifier w/ Simple Imputer',
'parameters': {'impute_strategy': 'most_frequent',
'n_estimators': 109,
'eta': 0.7991585642167238,
'max_depth': 4},
'score': 0.9664525764397752,
'high_variance_cv': False,
'training_time': 0.9475922584533691,
'cv_data': [{'all_objective_scores': OrderedDict([('F1',
0.9617021276595743),
('Precision', 0.9741379310344828),
('Recall', 0.9495798319327731),
('AUC', 0.9865072789679252),
('Log Loss', 0.16313398857144826),
('MCC', 0.9001633057441626),
('ROC',
(array([0. , 0. , 0. , 0.01408451, 0.01408451,
0.02816901, 0.02816901, 0.04225352, 0.04225352, 0.05633803,
0.05633803, 0.07042254, 0.07042254, 0.08450704, 0.08450704,
1. ]),
array([0. , 0.00840336, 0.34453782, 0.34453782, 0.84033613,
0.84033613, 0.93277311, 0.93277311, 0.96638655, 0.96638655,
0.97478992, 0.97478992, 0.98319328, 0.98319328, 1. ,
1. ]),
array([1.99999633e+00, 9.99996325e-01, 9.99942123e-01, 9.99941223e-01,
9.83889653e-01, 9.82884213e-01, 7.60889858e-01, 7.00312819e-01,
4.44799151e-01, 4.02629215e-01, 3.25498190e-01, 2.40859456e-01,
7.31464857e-02, 4.01578493e-02, 1.91156164e-02, 3.89213440e-06]))),
('Confusion Matrix',
0 1
0 68 3
1 6 113),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9617021276595743},
{'all_objective_scores': OrderedDict([('F1', 0.9626556016597511),
('Precision', 0.9508196721311475),
('Recall', 0.9747899159663865),
('AUC', 0.9874541365842111),
('Log Loss', 0.14974588678331058),
('MCC', 0.8984549429340701),
('ROC',
(array([0. , 0. , 0. , 0.01408451, 0.01408451,
0.02816901, 0.02816901, 0.04225352, 0.04225352, 0.07042254,
0.07042254, 0.08450704, 0.08450704, 1. ]),
array([0. , 0.00840336, 0.41176471, 0.41176471, 0.83193277,
0.83193277, 0.95798319, 0.95798319, 0.96638655, 0.96638655,
0.97478992, 0.97478992, 1. , 1. ]),
array([1.99999714e+00, 9.99997136e-01, 9.99900758e-01, 9.99895250e-01,
9.97706259e-01, 9.97275707e-01, 9.75388340e-01, 9.39646167e-01,
9.37469604e-01, 8.54371454e-01, 8.53389442e-01, 8.42251927e-01,
3.54062858e-01, 2.61300949e-06]))),
('Confusion Matrix',
0 1
0 65 6
1 3 116),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9626556016597511},
{'all_objective_scores': OrderedDict([('F1', 0.975),
('Precision', 0.9669421487603306),
('Recall', 0.9831932773109243),
('AUC', 0.9980792316926771),
('Log Loss', 0.06313235445917639),
('MCC', 0.9317727223276882),
('ROC',
(array([0. , 0. , 0. , 0.01428571, 0.01428571,
0.02857143, 0.02857143, 0.05714286, 0.05714286, 0.07142857,
0.07142857, 1. ]),
array([0. , 0.00840336, 0.95798319, 0.95798319, 0.96638655,
0.96638655, 0.97478992, 0.97478992, 0.99159664, 0.99159664,
1. , 1. ]),
array([1.99999867e+00, 9.99998668e-01, 9.22815771e-01, 8.86401336e-01,
8.72320864e-01, 8.49344800e-01, 7.87185615e-01, 7.06786188e-01,
4.73751084e-01, 3.98242251e-01, 3.72969449e-01, 1.06636001e-06]))),
('Confusion Matrix',
0 1
0 66 4
1 2 117),
('# Training', 380),
('# Testing', 189)]),
'score': 0.975}]}},
'search_order': [0, 1, 2, 3, 4]}