sensitivity_low_alert#

Sensitivity at Low Alert Rates objective.

Module Contents#

Classes Summary#

SensitivityLowAlert

Sensitivity at Low Alert Rates.

Attributes Summary#

logger

Contents#

evalml.objectives.sensitivity_low_alert.logger#
class evalml.objectives.sensitivity_low_alert.SensitivityLowAlert(alert_rate=0.01)[source]#

Sensitivity at Low Alert Rates.

Parameters

alert_rate (float) – percentage of top scores to classify as high risk.

Attributes

expected_range

[0, 1]

greater_is_better

True

is_bounded_like_percentage

True

name

Sensitivity at Low Alert Rates

perfect_score

1.0

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

score_needs_proba

False

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Determine if an observation is high risk given an alert rate.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

objective_function

Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)#

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)#

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, **kwargs)[source]#

Determine if an observation is high risk given an alert rate.

Parameters
  • ypred_proba (pd.Series) – Predicted probabilities.

  • **kwargs – Additional abritrary parameters.

Returns

Whether or not an observation is high risk given an alert rate.

Return type

pd.Series

classmethod is_defined_for_problem_type(cls, problem_type)#

Returns whether or not an objective is defined for a problem type.

objective_function(self, y_true, y_predicted, **kwargs)[source]#

Calculate sensitivity across all predictions, using the top alert_rate percent of observations as the predicted positive class.

Parameters
  • y_true (pd.Series) – True labels.

  • y_predicted (pd.Series) – Predicted labels based on alert_rate.

  • **kwargs – Additional abritrary parameters.

Returns

sensitivity using the observations with the top scores as the predicted positive class.

Return type

float

optimize_threshold(self, ypred_proba, y_true, X=None)#

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

positive_only(cls)#

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, y_train=None, X=None, sample_weight=None)#

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • y_train (pd.Series) – Observed training values of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

validate_inputs(self, y_true, y_predicted)#

Validate inputs for scoring.