binary_classification_objective

Base class for all binary classification objectives.

Module Contents

Classes Summary

BinaryClassificationObjective

Base class for all binary classification objectives.

Contents

class evalml.objectives.binary_classification_objective.BinaryClassificationObjective[source]

Base class for all binary classification objectives.

Attributes

problem_types

[ProblemTypes.BINARY, ProblemTypes.TIME_SERIES_BINARY]

Methods

calculate_percent_difference

Calculate the percent difference between scores.

can_optimize_threshold

Returns a boolean determining if we can optimize the binary classification objective threshold.

decision_function

Apply a learned threshold to predicted probabilities to get predicted classes.

expected_range

Returns the expected range of the objective, which is not necessarily the possible ranges.

greater_is_better

Returns a boolean determining if a greater score indicates better model performance.

is_bounded_like_percentage

Returns whether this objective is bounded between 0 and 1, inclusive.

is_defined_for_problem_type

Returns whether or not an objective is defined for a problem type.

name

Returns a name describing the objective.

objective_function

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.

optimize_threshold

Learn a binary classification threshold which optimizes the current objective.

perfect_score

Returns the score obtained by evaluating this objective on a perfect model.

positive_only

If True, this objective is only valid for positive data. Defaults to False.

score

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

score_needs_proba

Returns a boolean determining if the score() method needs probability estimates.

validate_inputs

Validate inputs for scoring.

classmethod calculate_percent_difference(cls, score, baseline_score)

Calculate the percent difference between scores.

Parameters
  • score (float) – A score. Output of the score method of this objective.

  • baseline_score (float) – A score. Output of the score method of this objective. In practice, this is the score achieved on this objective with a baseline estimator.

Returns

The percent difference between the scores. Note that for objectives that can be interpreted

as percentages, this will be the difference between the reference score and score. For all other objectives, the difference will be normalized by the reference score.

Return type

float

property can_optimize_threshold(cls)

Returns a boolean determining if we can optimize the binary classification objective threshold.

This will be false for any objective that works directly with predicted probabilities, like log loss and AUC. Otherwise, it will be true.

Returns

Whether or not an objective can be optimized.

Return type

bool

decision_function(self, ypred_proba, threshold=0.5, X=None)[source]

Apply a learned threshold to predicted probabilities to get predicted classes.

Parameters
  • ypred_proba (pd.Series, np.ndarray) – The classifier’s predicted probabilities

  • threshold (float, optional) – Threshold used to make a prediction. Defaults to 0.5.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

predictions

property expected_range(cls)

Returns the expected range of the objective, which is not necessarily the possible ranges.

For example, our expected R2 range is from [-1, 1], although the actual range is (-inf, 1].

property greater_is_better(cls)

Returns a boolean determining if a greater score indicates better model performance.

property is_bounded_like_percentage(cls)

Returns whether this objective is bounded between 0 and 1, inclusive.

classmethod is_defined_for_problem_type(cls, problem_type)

Returns whether or not an objective is defined for a problem type.

property name(cls)

Returns a name describing the objective.

abstract classmethod objective_function(cls, y_true, y_predicted, X=None, sample_weight=None)

Computes the relative value of the provided predictions compared to the actual labels, according a specified metric.

Args:

y_predicted (pd.Series): Predicted values of length [n_samples] y_true (pd.Series): Actual class labels of length [n_samples] X (pd.DataFrame or np.ndarray): Extra data of shape [n_samples, n_features] necessary to calculate score sample_weight (pd.DataFrame or np.ndarray): Sample weights used in computing objective value result

Returns

Numerical value used to calculate score

optimize_threshold(self, ypred_proba, y_true, X=None)[source]

Learn a binary classification threshold which optimizes the current objective.

Parameters
  • ypred_proba (pd.Series) – The classifier’s predicted probabilities

  • y_true (pd.Series) – The ground truth for the predictions.

  • X (pd.DataFrame, optional) – Any extra columns that are needed from training data.

Returns

Optimal threshold for this objective.

Raises

RuntimeError – If objective cannot be optimized.

property perfect_score(cls)

Returns the score obtained by evaluating this objective on a perfect model.

positive_only(cls)

If True, this objective is only valid for positive data. Defaults to False.

score(self, y_true, y_predicted, X=None, sample_weight=None)

Returns a numerical score indicating performance based on the differences between the predicted and actual values.

Parameters
  • y_predicted (pd.Series) – Predicted values of length [n_samples]

  • y_true (pd.Series) – Actual class labels of length [n_samples]

  • X (pd.DataFrame or np.ndarray) – Extra data of shape [n_samples, n_features] necessary to calculate score

  • sample_weight (pd.DataFrame or np.ndarray) – Sample weights used in computing objective value result

Returns

score

property score_needs_proba(cls)

Returns a boolean determining if the score() method needs probability estimates.

This should be true for objectives which work with predicted probabilities, like log loss or AUC, and false for objectives which compare predicted class labels to the actual labels, like F1 or correlation.

validate_inputs(self, y_true, y_predicted)[source]

Validate inputs for scoring.