metrics ============================================ .. py:module:: evalml.model_understanding.metrics .. autoapi-nested-parse:: Standard metrics used for model understanding. Module Contents --------------- Functions ~~~~~~~~~ .. autoapisummary:: :nosignatures: evalml.model_understanding.metrics.check_distribution evalml.model_understanding.metrics.confusion_matrix evalml.model_understanding.metrics.graph_confusion_matrix evalml.model_understanding.metrics.graph_precision_recall_curve evalml.model_understanding.metrics.graph_roc_curve evalml.model_understanding.metrics.normalize_confusion_matrix evalml.model_understanding.metrics.precision_recall_curve evalml.model_understanding.metrics.roc_curve Contents ~~~~~~~~~~~~~~~~~~~ .. py:function:: check_distribution(y_true, y_pred, problem_type, threshold=0.1) Determines if the distribution of the predicted data is likely to match that of the ground truth data. Will use a different statistical test based on the given problem type: - Classification (Binary or Multiclass) - chi squared test - Regression - Kolmogorov-Smirnov test - Time Series Regression - Wilcoxon signed-rank test :param y_true: The ground truth data. :type y_true: pd.Series :param y_pred: Predictions from a pipeline. :type y_pred: pd.Series :param problem_type: The pipeline's problem type, used to determine the method. :type problem_type: str or ProblemType :param threshold: The threshold for the p value where we choose to accept or reject the null hypothesis. Should be between 0 and 1, non-inclusive. Defaults to 0.1. :type threshold: float :returns: 0 if the distribution of predicted values is not likely to match the true distribution, 1 if it is. :rtype: int .. py:function:: confusion_matrix(y_true, y_predicted, normalize_method='true') Confusion matrix for binary and multiclass classification. :param y_true: True binary labels. :type y_true: pd.Series or np.ndarray :param y_predicted: Predictions from a binary classifier. :type y_predicted: pd.Series or np.ndarray :param normalize_method: Normalization method to use, if not None. Supported options are: 'true' to normalize by row, 'pred' to normalize by column, or 'all' to normalize by all values. Defaults to 'true'. :type normalize_method: {'true', 'pred', 'all', None} :returns: Confusion matrix. The column header represents the predicted labels while row header represents the actual labels. :rtype: pd.DataFrame .. py:function:: graph_confusion_matrix(y_true, y_pred, normalize_method='true', title_addition=None) Generate and display a confusion matrix plot. If `normalize_method` is set, hover text will show raw count, otherwise hover text will show count normalized with method 'true'. :param y_true: True binary labels. :type y_true: pd.Series or np.ndarray :param y_pred: Predictions from a binary classifier. :type y_pred: pd.Series or np.ndarray :param normalize_method: Normalization method to use, if not None. Supported options are: 'true' to normalize by row, 'pred' to normalize by column, or 'all' to normalize by all values. Defaults to 'true'. :type normalize_method: {'true', 'pred', 'all', None} :param title_addition: If not None, append to plot title. Defaults to None. :type title_addition: str :returns: plotly.Figure representing the confusion matrix plot generated. .. py:function:: graph_precision_recall_curve(y_true, y_pred_proba, title_addition=None) Generate and display a precision-recall plot. :param y_true: True binary labels. :type y_true: pd.Series or np.ndarray :param y_pred_proba: Predictions from a binary classifier, before thresholding has been applied. Note this should be the predicted probability for the "true" label. :type y_pred_proba: pd.Series or np.ndarray :param title_addition: If not None, append to plot title. Defaults to None. :type title_addition: str or None :returns: plotly.Figure representing the precision-recall plot generated .. py:function:: graph_roc_curve(y_true, y_pred_proba, custom_class_names=None, title_addition=None) Generate and display a Receiver Operating Characteristic (ROC) plot for binary and multiclass classification problems. :param y_true: True labels. :type y_true: pd.Series or np.ndarray :param y_pred_proba: Predictions from a classifier, before thresholding has been applied. Note this should a one dimensional array with the predicted probability for the "true" label in the binary case. :type y_pred_proba: pd.Series or np.ndarray :param custom_class_names: If not None, custom labels for classes. Defaults to None. :type custom_class_names: list or None :param title_addition: if not None, append to plot title. Defaults to None. :type title_addition: str or None :returns: plotly.Figure representing the ROC plot generated :raises ValueError: If the number of custom class names does not match number of classes in the input data. .. py:function:: normalize_confusion_matrix(conf_mat, normalize_method='true') Normalizes a confusion matrix. :param conf_mat: Confusion matrix to normalize. :type conf_mat: pd.DataFrame or np.ndarray :param normalize_method: Normalization method. Supported options are: 'true' to normalize by row, 'pred' to normalize by column, or 'all' to normalize by all values. Defaults to 'true'. :type normalize_method: {'true', 'pred', 'all'} :returns: normalized version of the input confusion matrix. The column header represents the predicted labels while row header represents the actual labels. :rtype: pd.DataFrame :raises ValueError: If configuration is invalid, or if the sum of a given axis is zero and normalization by axis is specified. .. py:function:: precision_recall_curve(y_true, y_pred_proba, pos_label_idx=-1) Given labels and binary classifier predicted probabilities, compute and return the data representing a precision-recall curve. :param y_true: True binary labels. :type y_true: pd.Series or np.ndarray :param y_pred_proba: Predictions from a binary classifier, before thresholding has been applied. Note this should be the predicted probability for the "true" label. :type y_pred_proba: pd.Series or np.ndarray :param pos_label_idx: the column index corresponding to the positive class. If predicted probabilities are two-dimensional, this will be used to access the probabilities for the positive class. :type pos_label_idx: int :returns: Dictionary containing metrics used to generate a precision-recall plot, with the following keys: * `precision`: Precision values. * `recall`: Recall values. * `thresholds`: Threshold values used to produce the precision and recall. * `auc_score`: The area under the ROC curve. :rtype: list :raises NoPositiveLabelException: If predicted probabilities do not contain a column at the specified label. .. py:function:: roc_curve(y_true, y_pred_proba) Given labels and classifier predicted probabilities, compute and return the data representing a Receiver Operating Characteristic (ROC) curve. Works with binary or multiclass problems. :param y_true: True labels. :type y_true: pd.Series or np.ndarray :param y_pred_proba: Predictions from a classifier, before thresholding has been applied. :type y_pred_proba: pd.Series or pd.DataFrame or np.ndarray :returns: A list of dictionaries (with one for each class) is returned. Binary classification problems return a list with one dictionary. Each dictionary contains metrics used to generate an ROC plot with the following keys: * `fpr_rate`: False positive rate. * `tpr_rate`: True positive rate. * `threshold`: Threshold values used to produce each pair of true/false positive rates. * `auc_score`: The area under the ROC curve. :rtype: list(dict)