Exploring search results¶
After finishing a pipeline search, we can inspect the results. First, let’s build a search of 10 different pipelines to explore.
[1]:
import evalml
from evalml import AutoClassificationSearch
X, y = evalml.demos.load_breast_cancer()
automl = AutoClassificationSearch(objective="f1",
max_pipelines=5)
automl.search(X, y)
*****************************
* Beginning pipeline search *
*****************************
Optimizing for F1. Greater score is better.
Searching up to 5 pipelines.
Possible model types: random_forest, linear_model, xgboost
✔ XGBoost Classifier w/ One Hot Encod... 20%|██ | Elapsed:00:02
✔ XGBoost Classifier w/ One Hot Encod... 40%|████ | Elapsed:00:04
✔ Random Forest Classifier w/ One Hot... 60%|██████ | Elapsed:00:14
✔ XGBoost Classifier w/ One Hot Encod... 80%|████████ | Elapsed:00:17
✔ Logistic Regression Classifier w/ O... 100%|██████████| Elapsed:00:18
✔ Optimization finished 100%|██████████| Elapsed:00:18
View Rankings¶
A summary of all the pipelines built can be returned as a pandas DataFrame. It is sorted by score. EvalML knows based on our objective function whether higher or lower is better.
[2]:
automl.rankings
[2]:
| id | pipeline_class_name | score | high_variance_cv | parameters | |
|---|---|---|---|---|---|
| 0 | 2 | RFClassificationPipeline | 0.973477 | False | {'n_estimators': 569, 'max_depth': 22, 'impute... |
| 1 | 4 | LogisticRegressionPipeline | 0.971949 | False | {'penalty': 'l2', 'C': 8.444214828324364, 'imp... |
| 2 | 1 | XGBoostPipeline | 0.964896 | False | {'eta': 0.38438170729269994, 'min_child_weight... |
| 3 | 0 | XGBoostPipeline | 0.959401 | False | {'eta': 0.5928446182250184, 'min_child_weight'... |
| 4 | 3 | XGBoostPipeline | 0.948566 | False | {'eta': 0.5288949197529046, 'min_child_weight'... |
Describe Pipeline¶
Each pipeline is given an id. We can get more information about any particular pipeline using that id. Here, we will get more information about the pipeline with id = 0.
[3]:
automl.describe_pipeline(0)
********************************************************************************************
* XGBoost Classifier w/ One Hot Encoder + Simple Imputer + RF Classifier Select From Model *
********************************************************************************************
Problem Types: Binary Classification, Multiclass Classification
Model Type: XGBoost Classifier
Objective to Optimize: F1 (greater is better)
Number of features: 18
Pipeline Steps
==============
1. One Hot Encoder
2. Simple Imputer
* impute_strategy : most_frequent
3. RF Classifier Select From Model
* percent_features : 0.6273280598181127
* threshold : -inf
4. XGBoost Classifier
* eta : 0.5928446182250184
* max_depth : 4
* min_child_weight : 8.598391737229157
Training
========
Training for Binary Classification problems.
Total training time (including CV): 2.4 seconds
Cross Validation
----------------
F1 Precision Recall AUC Log Loss MCC # Training # Testing
0 0.979 0.975 0.983 0.998 0.081 0.944 379.000 190.000
1 0.954 0.950 0.958 0.984 0.142 0.876 379.000 190.000
2 0.945 0.949 0.941 0.981 0.159 0.853 380.000 189.000
mean 0.959 0.958 0.961 0.988 0.127 0.891 - -
std 0.018 0.015 0.021 0.009 0.041 0.047 - -
coef of var 0.018 0.015 0.022 0.009 0.324 0.053 - -
Get Pipeline¶
We can get the object of any pipeline via their id as well:
[4]:
automl.get_pipeline(0)
[4]:
<evalml.pipelines.classification.xgboost.XGBoostPipeline at 0x7f420f818e48>
Get best pipeline¶
If we specifically want to get the best pipeline, there is a convenient access
[5]:
automl.best_pipeline
[5]:
<evalml.pipelines.classification.random_forest.RFClassificationPipeline at 0x7f420efdcc18>
Feature Importances¶
We can get the feature importances of the resulting pipeline
[6]:
pipeline = automl.get_pipeline(0)
pipeline.feature_importances
[6]:
| feature | importance | |
|---|---|---|
| 0 | worst perimeter | 0.341811 |
| 1 | worst radius | 0.184930 |
| 2 | mean concave points | 0.163518 |
| 3 | worst concave points | 0.115095 |
| 4 | mean concavity | 0.047942 |
| 5 | worst area | 0.038873 |
| 6 | worst concavity | 0.032179 |
| 7 | area error | 0.028544 |
| 8 | worst texture | 0.022472 |
| 9 | worst symmetry | 0.015158 |
| 10 | mean smoothness | 0.005401 |
| 11 | radius error | 0.004078 |
| 12 | mean radius | 0.000000 |
| 13 | mean perimeter | 0.000000 |
| 14 | mean area | 0.000000 |
| 15 | mean compactness | 0.000000 |
| 16 | worst compactness | 0.000000 |
| 17 | worst fractal dimension | 0.000000 |
We can also create a bar plot of the feature importances
[7]:
pipeline.plot.feature_importances()
Plot ROC¶
For binary classification tasks, we can also plot the ROC plot of a specific pipeline:
[8]:
automl.plot.generate_roc_plot(0)
Access raw results¶
You can also get access to all the underlying data like this
[9]:
automl.results
[9]:
{'pipeline_results': {0: {'id': 0,
'pipeline_class_name': 'XGBoostPipeline',
'pipeline_name': 'XGBoost Classifier w/ One Hot Encoder + Simple Imputer + RF Classifier Select From Model',
'parameters': {'eta': 0.5928446182250184,
'min_child_weight': 8.598391737229157,
'max_depth': 4,
'impute_strategy': 'most_frequent',
'percent_features': 0.6273280598181127},
'score': 0.9594006908767779,
'high_variance_cv': False,
'training_time': 2.434983491897583,
'cv_data': [{'all_objective_scores': OrderedDict([('F1',
0.9790794979079498),
('Precision', 0.975),
('Recall', 0.9831932773109243),
('AUC', 0.9981062847674281),
('Log Loss', 0.08064971198327839),
('MCC', 0.9436801731761278),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0.01408451, 0.01408451, 0.02816901,
0.02816901, 0.08450704, 0.08450704, 0.09859155, 0.09859155,
0.45070423, 0.50704225, 0.53521127, 0.56338028, 0.6056338 ,
0.63380282, 0.69014085, 0.87323944, 0.90140845, 1. ]),
array([0. , 0.12605042, 0.13445378, 0.1512605 , 0.16806723,
0.18487395, 0.22689076, 0.24369748, 0.26890756, 0.29411765,
0.31092437, 0.31932773, 0.35294118, 0.44537815, 0.46218487,
0.49579832, 0.5210084 , 0.7394958 , 0.77310924, 0.78991597,
0.80672269, 0.96638655, 0.96638655, 0.97478992, 0.97478992,
0.98319328, 0.98319328, 0.99159664, 0.99159664, 1. ,
1. , 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ]),
array([1.9937439 , 0.99374396, 0.9934169 , 0.9933848 , 0.9930241 ,
0.9929664 , 0.99256283, 0.9924057 , 0.9923273 , 0.9919527 ,
0.99182206, 0.99180686, 0.991738 , 0.99044997, 0.9904199 ,
0.9895264 , 0.9882288 , 0.9592414 , 0.9589123 , 0.9557933 ,
0.9548818 , 0.7891393 , 0.7523144 , 0.7496705 , 0.72532034,
0.66525084, 0.35472092, 0.33916065, 0.30189806, 0.28884584,
0.0441243 , 0.038174 , 0.03558043, 0.03012667, 0.02291998,
0.01877152, 0.01763946, 0.00880799, 0.00874277, 0.00827175],
dtype=float32))),
('Confusion Matrix', 0 1
0 68 3
1 2 117),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9790794979079498},
{'all_objective_scores': OrderedDict([('F1', 0.9539748953974896),
('Precision', 0.95),
('Recall', 0.957983193277311),
('AUC', 0.9836075275180495),
('Log Loss', 0.14185435472938576),
('MCC', 0.8760200852880281),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0.01408451, 0.01408451,
0.01408451, 0.01408451, 0.01408451, 0.01408451, 0.01408451,
0.01408451, 0.01408451, 0.01408451, 0.01408451, 0.01408451,
0.01408451, 0.01408451, 0.01408451, 0.01408451, 0.01408451,
0.01408451, 0.01408451, 0.01408451, 0.01408451, 0.01408451,
0.01408451, 0.04225352, 0.04225352, 0.05633803, 0.05633803,
0.09859155, 0.09859155, 0.12676056, 0.12676056, 0.15492958,
0.15492958, 0.16901408, 0.16901408, 0.21126761, 0.23943662,
0.25352113, 0.33802817, 0.46478873, 0.49295775, 0.56338028,
0.5915493 , 0.71830986, 0.76056338, 0.78873239, 0.8028169 ,
1. ]),
array([0. , 0.09243697, 0.10084034, 0.16806723, 0.17647059,
0.20168067, 0.21008403, 0.29411765, 0.30252101, 0.31092437,
0.36134454, 0.39495798, 0.41176471, 0.42857143, 0.43697479,
0.45378151, 0.49579832, 0.5210084 , 0.54621849, 0.56302521,
0.59663866, 0.60504202, 0.62184874, 0.65546218, 0.68907563,
0.71428571, 0.74789916, 0.76470588, 0.80672269, 0.82352941,
0.92436975, 0.92436975, 0.94117647, 0.94117647, 0.95798319,
0.95798319, 0.96638655, 0.96638655, 0.97478992, 0.97478992,
0.99159664, 0.99159664, 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ,
1. ]),
array([1.9917016 , 0.99170154, 0.99118173, 0.99113 , 0.99107367,
0.99095434, 0.9903882 , 0.9900671 , 0.9890872 , 0.9874872 ,
0.9870094 , 0.9863927 , 0.9848595 , 0.98429155, 0.9837299 ,
0.98244077, 0.97697645, 0.9723681 , 0.9694117 , 0.96865326,
0.96827 , 0.9661397 , 0.965724 , 0.96271837, 0.95904344,
0.95776254, 0.9541006 , 0.9514202 , 0.9272428 , 0.9244033 ,
0.729121 , 0.7108931 , 0.66336167, 0.6254638 , 0.5943486 ,
0.44532707, 0.372028 , 0.3703832 , 0.36668354, 0.28058025,
0.21105221, 0.1829909 , 0.18207934, 0.09410949, 0.07737678,
0.06360406, 0.06231202, 0.03264038, 0.02874083, 0.02029958,
0.01903956, 0.0164301 , 0.01615754, 0.01451924, 0.01421483,
0.01351323], dtype=float32))),
('Confusion Matrix', 0 1
0 65 6
1 5 114),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9539748953974896},
{'all_objective_scores': OrderedDict([('F1', 0.9451476793248945),
('Precision', 0.9491525423728814),
('Recall', 0.9411764705882353),
('AUC', 0.9807923169267707),
('Log Loss', 0.15895638581877822),
('MCC', 0.8530080688400891),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0.01428571, 0.01428571, 0.01428571, 0.01428571, 0.01428571,
0.01428571, 0.01428571, 0.01428571, 0.02857143, 0.02857143,
0.04285714, 0.04285714, 0.05714286, 0.05714286, 0.07142857,
0.07142857, 0.08571429, 0.08571429, 0.11428571, 0.11428571,
0.14285714, 0.15714286, 0.22857143, 0.25714286, 0.38571429,
0.44285714, 0.54285714, 0.6 , 0.64285714, 0.67142857,
0.7 , 1. ]),
array([0. , 0.18487395, 0.22689076, 0.23529412, 0.26890756,
0.29411765, 0.31092437, 0.33613445, 0.34453782, 0.36134454,
0.3697479 , 0.40336134, 0.42857143, 0.44537815, 0.46218487,
0.47058824, 0.50420168, 0.5210084 , 0.53781513, 0.55462185,
0.59663866, 0.65546218, 0.70588235, 0.70588235, 0.78991597,
0.78991597, 0.90756303, 0.90756303, 0.91596639, 0.91596639,
0.94117647, 0.94117647, 0.97478992, 0.97478992, 0.99159664,
0.99159664, 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ,
1. , 1. ]),
array([1.9912372 , 0.99123716, 0.99071807, 0.9905557 , 0.989953 ,
0.98967326, 0.9892195 , 0.9890625 , 0.9883869 , 0.9881627 ,
0.98800814, 0.9873002 , 0.9866671 , 0.9863214 , 0.98610663,
0.98588854, 0.9824102 , 0.9817692 , 0.9788537 , 0.9763956 ,
0.9736345 , 0.9734056 , 0.9697179 , 0.96919906, 0.9408594 ,
0.93151474, 0.70544225, 0.6761148 , 0.6481446 , 0.56992525,
0.5188037 , 0.514126 , 0.40845907, 0.35379264, 0.33419853,
0.22962442, 0.22778042, 0.09829229, 0.05558122, 0.04127952,
0.03519488, 0.01854065, 0.01816005, 0.01665189, 0.01571225,
0.01527028, 0.01430225], dtype=float32))),
('Confusion Matrix', 0 1
0 64 6
1 7 112),
('# Training', 380),
('# Testing', 189)]),
'score': 0.9451476793248945}]},
1: {'id': 1,
'pipeline_class_name': 'XGBoostPipeline',
'pipeline_name': 'XGBoost Classifier w/ One Hot Encoder + Simple Imputer + RF Classifier Select From Model',
'parameters': {'eta': 0.38438170729269994,
'min_child_weight': 3.677811458900251,
'max_depth': 13,
'impute_strategy': 'median',
'percent_features': 0.793807787701838},
'score': 0.9648960642685166,
'high_variance_cv': False,
'training_time': 2.488259792327881,
'cv_data': [{'all_objective_scores': OrderedDict([('F1', 0.975),
('Precision', 0.9669421487603306),
('Recall', 0.9831932773109243),
('AUC', 0.9966859983429992),
('Log Loss', 0.07756886413615001),
('MCC', 0.932389423285531),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0.01408451, 0.01408451, 0.02816901, 0.02816901, 0.08450704,
0.08450704, 0.09859155, 0.09859155, 0.91549296, 0.94366197,
1. ]),
array([0. , 0.00840336, 0.32773109, 0.34453782, 0.87394958,
0.87394958, 0.96638655, 0.96638655, 0.98319328, 0.98319328,
0.99159664, 0.99159664, 1. , 1. , 1. ,
1. ]),
array([1.9981308 , 0.99813086, 0.9959843 , 0.99595356, 0.9659731 ,
0.9646539 , 0.8474301 , 0.8226159 , 0.7340414 , 0.43467796,
0.35439238, 0.3156159 , 0.2735931 , 0.00334718, 0.00322822,
0.00207397], dtype=float32))),
('Confusion Matrix', 0 1
0 67 4
1 2 117),
('# Training', 379),
('# Testing', 190)]),
'score': 0.975},
{'all_objective_scores': OrderedDict([('F1', 0.9661016949152542),
('Precision', 0.9743589743589743),
('Recall', 0.957983193277311),
('AUC', 0.9899396378269618),
('Log Loss', 0.1157939842209759),
('MCC', 0.9107843947529072),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0.01408451,
0.01408451, 0.02816901, 0.02816901, 0.04225352, 0.04225352,
0.05633803, 0.05633803, 0.14084507, 0.14084507, 0.16901408,
0.16901408, 0.22535211, 0.49295775, 0.52112676, 0.66197183,
0.73239437, 0.81690141, 0.84507042, 0.87323944, 0.90140845,
0.97183099, 1. ]),
array([0. , 0.01680672, 0.08403361, 0.11764706, 0.24369748,
0.2605042 , 0.26890756, 0.29411765, 0.57983193, 0.57983193,
0.91596639, 0.91596639, 0.94957983, 0.94957983, 0.95798319,
0.95798319, 0.98319328, 0.98319328, 0.99159664, 0.99159664,
1. , 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ,
1. , 1. ]),
array([1.9985983e+00, 9.9859840e-01, 9.9832827e-01, 9.9818736e-01,
9.9679202e-01, 9.9674618e-01, 9.9667656e-01, 9.9658316e-01,
9.9108350e-01, 9.9083334e-01, 8.2265288e-01, 8.2121074e-01,
7.2431225e-01, 6.3007897e-01, 5.7422268e-01, 4.5153135e-01,
3.5701558e-01, 1.7140125e-01, 1.4191566e-01, 9.0782769e-02,
6.7663342e-02, 5.5308960e-02, 4.5769266e-03, 4.5044953e-03,
3.2702754e-03, 3.2080656e-03, 2.9759661e-03, 2.8877873e-03,
2.4864469e-03, 2.3314529e-03, 1.5917335e-03, 1.3716089e-03],
dtype=float32))),
('Confusion Matrix', 0 1
0 68 3
1 5 114),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9661016949152542},
{'all_objective_scores': OrderedDict([('F1', 0.9535864978902954),
('Precision', 0.9576271186440678),
('Recall', 0.9495798319327731),
('AUC', 0.9863145258103241),
('Log Loss', 0.14040204006858248),
('MCC', 0.8756320549488144),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0.01428571,
0.01428571, 0.02857143, 0.02857143, 0.04285714, 0.04285714,
0.08571429, 0.08571429, 0.1 , 0.1 , 0.11428571,
0.11428571, 0.54285714, 0.57142857, 0.58571429, 0.61428571,
0.68571429, 0.71428571, 0.8 , 1. ]),
array([0. , 0.00840336, 0.04201681, 0.05882353, 0.1512605 ,
0.21008403, 0.23529412, 0.30252101, 0.31932773, 0.33613445,
0.35294118, 0.62184874, 0.65546218, 0.72268908, 0.72268908,
0.73109244, 0.73109244, 0.79831933, 0.79831933, 0.94957983,
0.94957983, 0.96638655, 0.96638655, 0.97478992, 0.97478992,
1. , 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. ]),
array([1.998668 , 0.99866796, 0.99859375, 0.99848515, 0.99842656,
0.9980634 , 0.9980153 , 0.9971877 , 0.9971783 , 0.9970214 ,
0.99701583, 0.9893309 , 0.98911965, 0.9835843 , 0.9835355 ,
0.98161846, 0.9813965 , 0.9743532 , 0.97359157, 0.62105346,
0.47956562, 0.46517578, 0.42012912, 0.36913908, 0.25974 ,
0.16882712, 0.00341962, 0.00341539, 0.00333552, 0.00333485,
0.00295924, 0.00287893, 0.00215919, 0.00207186], dtype=float32))),
('Confusion Matrix', 0 1
0 65 5
1 6 113),
('# Training', 380),
('# Testing', 189)]),
'score': 0.9535864978902954}]},
2: {'id': 2,
'pipeline_class_name': 'RFClassificationPipeline',
'pipeline_name': 'Random Forest Classifier w/ One Hot Encoder + Simple Imputer + RF Classifier Select From Model',
'parameters': {'n_estimators': 569,
'max_depth': 22,
'impute_strategy': 'most_frequent',
'percent_features': 0.8593661614465293},
'score': 0.973476521410252,
'high_variance_cv': False,
'training_time': 9.873236894607544,
'cv_data': [{'all_objective_scores': OrderedDict([('F1',
0.9831932773109243),
('Precision', 0.9831932773109243),
('Recall', 0.9831932773109243),
('AUC', 0.9977512131613208),
('Log Loss', 0.08333209715129118),
('MCC', 0.9550242632264173),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0.01408451, 0.01408451, 0.09859155, 0.09859155,
0.12676056, 0.12676056, 0.4084507 , 0.43661972, 0.47887324,
0.54929577, 0.5915493 , 0.6056338 , 0.64788732, 0.76056338,
1. ]),
array([0. , 0.26890756, 0.39495798, 0.42016807, 0.49579832,
0.52941176, 0.57142857, 0.59663866, 0.64705882, 0.65546218,
0.68907563, 0.69747899, 0.73109244, 0.8487395 , 0.89915966,
0.95798319, 0.95798319, 0.98319328, 0.98319328, 0.99159664,
0.99159664, 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ,
1. ]),
array([2.00000000e+00, 1.00000000e+00, 9.98242531e-01, 9.96485062e-01,
9.94727592e-01, 9.92970123e-01, 9.91212654e-01, 9.89455185e-01,
9.87697715e-01, 9.85940246e-01, 9.84182777e-01, 9.82425308e-01,
9.77152900e-01, 9.08611599e-01, 8.89279438e-01, 7.75043937e-01,
7.39894552e-01, 7.13532513e-01, 3.69068541e-01, 3.53251318e-01,
2.86467487e-01, 2.26713533e-01, 4.92091388e-02, 4.56942004e-02,
4.39367311e-02, 1.23022847e-02, 1.05448155e-02, 5.27240773e-03,
3.51493849e-03, 1.75746924e-03, 0.00000000e+00]))),
('Confusion Matrix', 0 1
0 69 2
1 2 117),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9831932773109243},
{'all_objective_scores': OrderedDict([('F1', 0.9789029535864979),
('Precision', 0.9830508474576272),
('Recall', 0.9747899159663865),
('AUC', 0.9923067818676766),
('Log Loss', 0.1143023991450681),
('MCC', 0.9439989100828842),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0.01408451, 0.01408451, 0.01408451, 0.01408451,
0.01408451, 0.01408451, 0.01408451, 0.01408451, 0.01408451,
0.01408451, 0.01408451, 0.01408451, 0.01408451, 0.01408451,
0.01408451, 0.01408451, 0.02816901, 0.02816901, 0.04225352,
0.04225352, 0.11267606, 0.11267606, 0.22535211, 0.25352113,
0.30985915, 0.33802817, 0.3943662 , 0.42253521, 0.43661972,
0.49295775, 0.52112676, 0.6056338 , 0.73239437, 1. ]),
array([0. , 0.19327731, 0.27731092, 0.33613445, 0.3697479 ,
0.45378151, 0.47058824, 0.50420168, 0.52941176, 0.56302521,
0.57142857, 0.57142857, 0.59663866, 0.61344538, 0.62184874,
0.63865546, 0.64705882, 0.66386555, 0.68907563, 0.70588235,
0.73109244, 0.76470588, 0.81512605, 0.83193277, 0.84033613,
0.85714286, 0.94117647, 0.94117647, 0.98319328, 0.98319328,
0.99159664, 0.99159664, 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ]),
array([2.00000000e+00, 1.00000000e+00, 9.98242531e-01, 9.96485062e-01,
9.94727592e-01, 9.91212654e-01, 9.89455185e-01, 9.87697715e-01,
9.85940246e-01, 9.80667838e-01, 9.78910369e-01, 9.77152900e-01,
9.75395431e-01, 9.73637961e-01, 9.71880492e-01, 9.70123023e-01,
9.68365554e-01, 9.63093146e-01, 9.52548330e-01, 9.50790861e-01,
9.36731107e-01, 9.29701230e-01, 8.82249561e-01, 8.73462214e-01,
8.71704745e-01, 8.55887522e-01, 6.67838313e-01, 6.55536028e-01,
4.95606327e-01, 4.81546573e-01, 4.69244288e-01, 3.98945518e-01,
3.72583480e-01, 1.15992970e-01, 1.00175747e-01, 6.50263620e-02,
4.39367311e-02, 2.81195079e-02, 1.75746924e-02, 1.40597540e-02,
8.78734622e-03, 7.02987698e-03, 3.51493849e-03, 1.75746924e-03,
0.00000000e+00]))),
('Confusion Matrix', 0 1
0 69 2
1 3 116),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9789029535864979},
{'all_objective_scores': OrderedDict([('F1', 0.9583333333333334),
('Precision', 0.9504132231404959),
('Recall', 0.9663865546218487),
('AUC', 0.9847539015606243),
('Log Loss', 0.1416407874804379),
('MCC', 0.8861141678760591),
('ROC',
(array([0. , 0. , 0.01428571, 0.01428571, 0.01428571,
0.01428571, 0.01428571, 0.01428571, 0.01428571, 0.01428571,
0.01428571, 0.02857143, 0.02857143, 0.02857143, 0.02857143,
0.04285714, 0.04285714, 0.08571429, 0.1 , 0.1 ,
0.14285714, 0.14285714, 0.37142857, 0.4 , 0.47142857,
0.5 , 0.51428571, 0.62857143, 1. ]),
array([0. , 0.37815126, 0.46218487, 0.54621849, 0.57983193,
0.62184874, 0.6302521 , 0.65546218, 0.66386555, 0.69747899,
0.78151261, 0.78991597, 0.85714286, 0.87394958, 0.88235294,
0.88235294, 0.96638655, 0.96638655, 0.97478992, 0.99159664,
0.99159664, 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. ]),
array([2.00000000e+00, 1.00000000e+00, 9.98242531e-01, 9.96485062e-01,
9.94727592e-01, 9.92970123e-01, 9.91212654e-01, 9.89455185e-01,
9.87697715e-01, 9.85940246e-01, 9.43760984e-01, 9.38488576e-01,
8.64674868e-01, 8.03163445e-01, 7.62741652e-01, 7.34622144e-01,
5.55360281e-01, 5.04393673e-01, 4.60456942e-01, 3.81370826e-01,
3.04042179e-01, 2.93497364e-01, 2.10896309e-02, 1.93321617e-02,
8.78734622e-03, 5.27240773e-03, 3.51493849e-03, 1.75746924e-03,
0.00000000e+00]))),
('Confusion Matrix', 0 1
0 64 6
1 4 115),
('# Training', 380),
('# Testing', 189)]),
'score': 0.9583333333333334}]},
3: {'id': 3,
'pipeline_class_name': 'XGBoostPipeline',
'pipeline_name': 'XGBoost Classifier w/ One Hot Encoder + Simple Imputer + RF Classifier Select From Model',
'parameters': {'eta': 0.5288949197529046,
'min_child_weight': 6.112401049845392,
'max_depth': 6,
'impute_strategy': 'most_frequent',
'percent_features': 0.34402219881309576},
'score': 0.9485662145040741,
'high_variance_cv': False,
'training_time': 2.4133780002593994,
'cv_data': [{'all_objective_scores': OrderedDict([('F1',
0.9666666666666667),
('Precision', 0.9586776859504132),
('Recall', 0.9747899159663865),
('AUC', 0.9950289975144987),
('Log Loss', 0.09729690174817254),
('MCC', 0.9097672817424011),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0.01408451, 0.01408451, 0.04225352, 0.04225352, 0.05633803,
0.05633803, 0.09859155, 0.09859155, 0.12676056, 0.12676056,
0.16901408, 0.16901408, 0.38028169, 0.4084507 , 0.47887324,
0.52112676, 0.54929577, 0.57746479, 0.6056338 , 0.66197183,
0.67605634, 0.98591549, 1. ]),
array([0. , 0.02521008, 0.03361345, 0.16806723, 0.17647059,
0.20168067, 0.21008403, 0.35294118, 0.36134454, 0.38655462,
0.42016807, 0.43697479, 0.4789916 , 0.49579832, 0.51260504,
0.54621849, 0.57142857, 0.57983193, 0.59663866, 0.91596639,
0.91596639, 0.94957983, 0.94957983, 0.96638655, 0.96638655,
0.97478992, 0.97478992, 0.98319328, 0.98319328, 0.99159664,
0.99159664, 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ,
1. , 1. , 1. ]),
array([1.9918091 , 0.99180907, 0.9914386 , 0.99143535, 0.9909027 ,
0.9905533 , 0.9904917 , 0.9904881 , 0.990333 , 0.99021775,
0.9898657 , 0.98977226, 0.9895096 , 0.9892888 , 0.98913723,
0.98872095, 0.9886431 , 0.9884277 , 0.9880271 , 0.9180971 ,
0.907636 , 0.7645348 , 0.63884944, 0.6283931 , 0.5994021 ,
0.5585935 , 0.38392243, 0.37918517, 0.2239313 , 0.22208475,
0.12249093, 0.12071476, 0.02558885, 0.0240444 , 0.01662812,
0.01431511, 0.01352188, 0.01269417, 0.01247706, 0.01176526,
0.01091322, 0.01024354, 0.0058776 ], dtype=float32))),
('Confusion Matrix', 0 1
0 66 5
1 3 116),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9666666666666667},
{'all_objective_scores': OrderedDict([('F1', 0.9338842975206612),
('Precision', 0.9186991869918699),
('Recall', 0.9495798319327731),
('AUC', 0.9846727423363711),
('Log Loss', 0.17591368435195795),
('MCC', 0.8188772553918354),
('ROC',
(array([0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0.01408451, 0.01408451, 0.01408451,
0.01408451, 0.01408451, 0.01408451, 0.01408451, 0.02816901,
0.02816901, 0.04225352, 0.07042254, 0.07042254, 0.08450704,
0.08450704, 0.15492958, 0.15492958, 0.18309859, 0.18309859,
0.22535211, 0.22535211, 0.35211268, 0.38028169, 0.4084507 ,
0.46478873, 0.49295775, 0.50704225, 0.78873239, 0.8028169 ,
0.84507042, 1. ]),
array([0. , 0.10084034, 0.11764706, 0.14285714, 0.19327731,
0.22689076, 0.4789916 , 0.49579832, 0.53781513, 0.58823529,
0.60504202, 0.66386555, 0.68067227, 0.69747899, 0.73109244,
0.75630252, 0.78991597, 0.80672269, 0.87394958, 0.87394958,
0.91596639, 0.92436975, 0.92436975, 0.94117647, 0.94117647,
0.94957983, 0.94957983, 0.96638655, 0.96638655, 0.99159664,
0.99159664, 1. , 1. , 1. , 1. ,
1. , 1. , 1. , 1. , 1. ,
1. , 1. ]),
array([1.9947486 , 0.99474853, 0.99465054, 0.9945134 , 0.9920238 ,
0.99128073, 0.9897943 , 0.98960495, 0.98933965, 0.98638594,
0.98634803, 0.9774489 , 0.9764572 , 0.97513086, 0.9740399 ,
0.9718917 , 0.9563253 , 0.946435 , 0.8835335 , 0.878611 ,
0.7919501 , 0.755015 , 0.71543545, 0.70243114, 0.6865198 ,
0.64198273, 0.47787225, 0.22530945, 0.16627482, 0.08063855,
0.06376061, 0.04771832, 0.01834523, 0.01635974, 0.01409694,
0.01397619, 0.01379675, 0.01324805, 0.01214174, 0.00822058,
0.0067653 , 0.00625362], dtype=float32))),
('Confusion Matrix', 0 1
0 61 10
1 6 113),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9338842975206612},
{'all_objective_scores': OrderedDict([('F1', 0.9451476793248945),
('Precision', 0.9491525423728814),
('Recall', 0.9411764705882353),
('AUC', 0.9645258103241297),
('Log Loss', 0.18750503725032247),
('MCC', 0.8530080688400891),
('ROC',
(array([0. , 0. , 0. , 0.01428571, 0.02857143,
0.02857143, 0.02857143, 0.02857143, 0.02857143, 0.02857143,
0.02857143, 0.02857143, 0.02857143, 0.02857143, 0.02857143,
0.04285714, 0.04285714, 0.05714286, 0.05714286, 0.07142857,
0.07142857, 0.08571429, 0.08571429, 0.1 , 0.1 ,
0.12857143, 0.12857143, 0.17142857, 0.17142857, 0.18571429,
0.18571429, 0.41428571, 0.44285714, 0.47142857, 0.5 ,
0.64285714, 1. ]),
array([0. , 0.00840336, 0.01680672, 0.01680672, 0.46218487,
0.48739496, 0.5210084 , 0.54621849, 0.55462185, 0.58823529,
0.61344538, 0.6302521 , 0.63865546, 0.65546218, 0.77310924,
0.77310924, 0.8487395 , 0.8487395 , 0.87394958, 0.87394958,
0.93277311, 0.93277311, 0.94957983, 0.94957983, 0.96638655,
0.97478992, 0.98319328, 0.98319328, 0.99159664, 0.99159664,
1. , 1. , 1. , 1. , 1. ,
1. , 1. ]),
array([1.9947662 , 0.99476624, 0.99434704, 0.99431866, 0.993646 ,
0.99334115, 0.9933149 , 0.99291396, 0.99289966, 0.9928403 ,
0.9924971 , 0.9922282 , 0.99102247, 0.9907026 , 0.9769703 ,
0.9766011 , 0.9003137 , 0.89544696, 0.88858014, 0.8658668 ,
0.68705285, 0.6113713 , 0.43407622, 0.37409404, 0.32079345,
0.29611424, 0.29083598, 0.2146415 , 0.21115056, 0.1591847 ,
0.14415835, 0.01072225, 0.01065906, 0.00977037, 0.00951847,
0.00907114, 0.00867252], dtype=float32))),
('Confusion Matrix', 0 1
0 64 6
1 7 112),
('# Training', 380),
('# Testing', 189)]),
'score': 0.9451476793248945}]},
4: {'id': 4,
'pipeline_class_name': 'LogisticRegressionPipeline',
'pipeline_name': 'Logistic Regression Classifier w/ One Hot Encoder + Simple Imputer + Standard Scaler',
'parameters': {'penalty': 'l2',
'C': 8.444214828324364,
'impute_strategy': 'most_frequent'},
'score': 0.9719486068554629,
'high_variance_cv': False,
'training_time': 1.6137502193450928,
'cv_data': [{'all_objective_scores': OrderedDict([('F1',
0.9709543568464729),
('Precision', 0.9590163934426229),
('Recall', 0.9831932773109243),
('AUC', 0.9875724937862469),
('Log Loss', 0.14613712409589413),
('MCC', 0.9211492315750531),
('ROC',
(array([0. , 0. , 0. , 0.01408451, 0.01408451,
0.02816901, 0.02816901, 0.04225352, 0.04225352, 0.05633803,
0.05633803, 0.07042254, 0.07042254, 0.08450704, 0.08450704,
1. ]),
array([0. , 0.00840336, 0.57983193, 0.57983193, 0.68067227,
0.68067227, 0.92436975, 0.92436975, 0.95798319, 0.95798319,
0.98319328, 0.98319328, 0.99159664, 0.99159664, 1. ,
1. ]),
array([2.00000000e+00, 1.00000000e+00, 9.99936259e-01, 9.99935430e-01,
9.99856221e-01, 9.99847656e-01, 9.44959587e-01, 9.25157910e-01,
8.68396485e-01, 8.05626231e-01, 6.85798432e-01, 5.24509038e-01,
4.79378624e-01, 2.63698337e-01, 2.31528899e-01, 1.13164746e-48]))),
('Confusion Matrix', 0 1
0 66 5
1 2 117),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9709543568464729},
{'all_objective_scores': OrderedDict([('F1', 0.9790794979079498),
('Precision', 0.975),
('Recall', 0.9831932773109243),
('AUC', 0.9910048526452835),
('Log Loss', 0.12072855977761998),
('MCC', 0.9436801731761278),
('ROC',
(array([0. , 0. , 0. , 0.01408451, 0.01408451,
0.05633803, 0.05633803, 0.23943662, 0.23943662, 1. ]),
array([0. , 0.00840336, 0.5210084 , 0.5210084 , 0.98319328,
0.98319328, 0.99159664, 0.99159664, 1. , 1. ]),
array([2.00000000e+00, 1.00000000e+00, 9.99838725e-01, 9.99835686e-01,
6.74763310e-01, 4.29302726e-01, 3.81937028e-01, 5.59186779e-04,
2.40539384e-04, 1.23958997e-29]))),
('Confusion Matrix', 0 1
0 68 3
1 2 117),
('# Training', 379),
('# Testing', 190)]),
'score': 0.9790794979079498},
{'all_objective_scores': OrderedDict([('F1', 0.9658119658119659),
('Precision', 0.9826086956521739),
('Recall', 0.9495798319327731),
('AUC', 0.9965186074429772),
('Log Loss', 0.08056331735168969),
('MCC', 0.9112159507396058),
('ROC',
(array([0. , 0. , 0. , 0.01428571, 0.01428571,
0.02857143, 0.02857143, 0.04285714, 0.04285714, 0.05714286,
0.05714286, 0.08571429, 0.08571429, 1. ]),
array([0. , 0.00840336, 0.90756303, 0.90756303, 0.93277311,
0.93277311, 0.94957983, 0.94957983, 0.98319328, 0.98319328,
0.99159664, 0.99159664, 1. , 1. ]),
array([2.00000000e+00, 9.99999999e-01, 9.46260690e-01, 9.45033115e-01,
9.06023542e-01, 8.77670109e-01, 7.39987181e-01, 4.77332145e-01,
3.62040644e-01, 2.35727064e-01, 1.93164591e-01, 9.53728712e-02,
8.79204445e-02, 1.15873760e-20]))),
('Confusion Matrix', 0 1
0 68 2
1 6 113),
('# Training', 380),
('# Testing', 189)]),
'score': 0.9658119658119659}]}},
'search_order': [0, 1, 2, 3, 4]}